55 Commits

Author SHA1 Message Date
3cd4cf98c3 chore: vendor @joan/procedural-glyph-engine for portable builds
Move @joan/procedural-glyph-engine from absolute path dep
(/private/tmp/orby-pkg) to vendored local dep (web/vendor/) so the
SPA builds in Docker and on other machines without the temp dir.

- Vendor Orby v5.0.0 into web/vendor/
- Switch package.json dep to file:../vendor
- Update Dockerfile to COPY vendor into build context
- Use npm install instead of npm ci (file: deps need install)
- Fix missing trailing newline in Dockerfile
2026-08-05 17:27:04 +02:00
38c472a118 0.26.0 — transport-aware classifier escalation + standalone-server monitoring override
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- classifyAndGate: escalate read-only commands on lxc: targets that
  touch /opt/, /etc/, /var/lib/ to config_mutation. The classifier
  scores command text only, not the SSH transport layer — SSH-ing into
  a container to read config is riskier than pct exec from the host.
- ontology: standalone-server monitoring override from inherited
  [ping, resource, updates] to [http]. VPS-like machines may not be
  SSH/ICMP-reachable from the scheduler; HTTP is the LCD liveness
  signal. Entities with full SSH can override per-entity.
2026-08-05 16:31:41 +02:00
1d0197da69 0.25.1 — get_health_summary destroyed filter + create_entity footgun doc
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- get_health_summary: filter out state=destroyed entities (was noise
  from 20+ destroyed test LXCs, deprecated services, etc.)
- create_entity: document the monitoring footgun in the tool description
  (creating a type=check entity does NOT wire a check_def; the correct
  path is update_entity_attributes with monitoring + url attributes)
2026-08-05 16:07:59 +02:00
0920c4cb6d 0.25.0 — DNS resolution check kind (KindDNS) + VPS monitoring fix
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Adds a new 'dns' semantic monitoring kind that probes whether a DNS name
resolves. Uses net.LookupNS (NS records) with fallback to net.LookupHost
(A/AAAA). Supports an explicit server config for split-horizon resolution.

Changes:
- seeds/ontology.yaml: dns-zone monitoring: none → [dns] (was deferred
  since 2026-06 with a comment 'no dns checker exists yet')
- seeds/inventory.yaml: host:netbird-vps monitoring: [http] (was none;
  VPS was invisible for 7 days during the 2026-07-29 outage)
- internal/checkdefaults/defaults.go: add KindDNS, buildKind case for 'dns'
  that creates a check_def at 5-minute intervals
- internal/scheduler/scheduler.go: add checkDNS probe + wire in executeCheck

The DNS checker catches stale/unreachable zones (e.g. matrix.hubris.network
pointing to a dead VPS IP). The VPS HTTP check probes the public endpoint
every 60s, closing the 7-day monitoring gap.
2026-08-05 15:31:50 +02:00
2254a07baf plan done: agent execution safety — move to done/, update index 2026-08-05 15:26:03 +02:00
1b9c761274 implements plan: agent execution safety — QEMU guest agent gate + health guard + policy docs
I — run pre-flights QEMU guest agent before queueing VM execution
  classifyAndGate now checks vm: targets for qemu_guest_agent attribute.
  If not_running/missing, returns immediate error instead of queuing forever.

II — policy.yaml: documented host-mutation classifier rule
  Added comment clarifying that host-level package/kernel mutations
  (apt-get install, dpkg, systemctl enable) always classify as
  config_mutation and thus need operator approval.

III — health attribute read-only in update_entity_attributes
  Strips scheduler-owned keys (health, last_check_at, last_check) from
  attribute updates with a clear message directing agents to
  get_health_summary / list_checks instead.

IV — Recorded discovered dependency edges
  vm:zimaos → depends-on → lxc:nfs-export (NFS /media/library mount)
  vm:zimaos → depends-on → host:strong (NFS /media/ludo-library mount)

Also updated the run tool description to mention both guardrails.
2026-08-05 15:25:14 +02:00
a126cfa710 0.24.0 — MCP tool improvements: type filter for get_relations, health filter for get_health_summary, live HTTP probe for ping_service
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
get_relations now accepts an optional 'types' (comma-separated) parameter
to filter relationship types — filters out the noisy exec/targets edges
that previously drowned useful host/provides edges.

get_health_summary now accepts an optional 'health' (comma-separated)
parameter to return only entities in specific health states (e.g.
'health=down,stale') instead of the full 100+ entity list.

ping_service now:
- Falls back to e.attributes->>'public_host' when 'url' is not set
  (covers LXCs that only have public_host in the graph)
- Performs a live HTTP HEAD probe against the resolved URL, returning
  the actual status code instead of just the scheduler's stale health
  state

Also: fixed matrix.hubris.network DNS record (was pointing to dead VPS),
pruned 6 dead graph edges, wired url attributes on 7 LXCs, added VPS
HTTP monitoring check, and resolved the 18k-occurrence unmonitored signal.

This session's audit is documented as
document:nomos/2026-08-05-dns-monitoring-improvements-for-strong-hosted-services.
2026-08-05 15:12:04 +02:00
86fa57b5cd plan: agent execution safety — QEMU guest agent gate + host-mutation guard
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-08-05 11:57:57 +02:00
8e97d589af scripts: ZimaOS NFS mount fix — both /media/library and /media/ludo-library 2026-08-05 08:31:04 +02:00
0dd8c28815 feat: MCP ping tool, tightened descriptions, and Hermes client docs
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Add  MCP tool — lightweight connectivity check returning server
  identity, no DB hit (resolves agent connection-test friction)
- Tighten 6 tool descriptions (get_relations, get_health_summary,
  query_metrics, get_trend, get_event_timeline, ping) to be searchable
  in the first 8-12 words
- Document Hermes MCP client setup in ADR-0012 with token security caveat
- Move completed plan to plans/done/
2026-08-05 00:21:56 +02:00
4e294b3630 0.22.0 — full MCP agent surface: 19 new tools, async run, session reliability
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Stage 1 — Foundation:
- Target validation: iptables + systemctl/docker target-type gates
- Async run for long-running commands (sleep/wait/poll loops)
- audit_log.session_id plumbing (SQL, sqlcgen, 18 call sites)

Stage 2 — External agent observe (11 tools):
- get_dashboard_summary, get_ontology, list_checks, list_executions
- get_knowledge_revisions, get_knowledge_duplicates, get_knowledge_orphans
- list_knowledge_tags, list_entity_sessions, find_entities_by
- 3 resource templates: oikos://entity/{slug}, knowledge/{id}, execution/{id}

Stage 3 — Nomos reliability:
- complete_task(success) refused without verification (upgraded from warn)
- sessionHasPlan excludes replaced steps (forces propose_plan after reopen)
- Bash syntax validation in run() (rejects literal \n, flag-space typos)
- Scope gate in SOUL.md (ask before pivoting to unrelated subsystem)

Stage 4 — External agent act (9 mutation tools):
- ack_signal, resolve_signal, mute_signal, cancel_execution
- update_check, delete_knowledge, restore_knowledge
- merge_knowledge, rename_knowledge_tag
2026-08-04 23:51:55 +02:00
85f0bb67fa docs(plans): move 2026-08-04 session audit plan to done (v0.21.0 shipped)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-08-04 23:43:28 +02:00
c3f478b8f8 v0.21.0: agent reliability overhaul — plan integrity, target validation, observability pipelines, learning loop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
P0 — stop the bleeding:
- prevent premature complete_task(success) when goal involves reachability
- validate run targets: block host-only commands (qm/pct/pvesh) on LXC/VM
- bump MCP client timeout 30s→120s to stop 'context deadline exceeded'

P1 — fix the plan system:
- add replaced_reason column to session_plan_steps (migration 030)
- track WHY steps are replaced (wrong_diagnosis/scope_change/superseded/etc)
- force fresh propose_plan on session resume (reopenSession marks old plan)

P2 — cognitive guardrails:
- SOUL.md scope-gate rule: ask before chasing unrelated subsystems
- auto-upsert knowledge entry on every session close

P3 — observability (all were empty/NULL):
- populate agent_activity.token_count from LLM usage (was always NULL)
- populate nomos_plan_executions linking executions to sessions
- write plan_completion_rate metric on task close

P4 — learning loop (all were empty/NULL):
- auto-classify every run call → classifications table (was 0 rows)
- auto-feedback on session close (was 0 rows)
2026-08-04 23:15:47 +02:00
1aaedf498a v0.20.0: thinking blocks, chat windows overhaul, scroll fix
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Backend:
- Add isThinking flag to agentEvent for text before tool calls
- Separate thinking from response text in runChatTurn and continue.go
- Persist thinking in a dedicated field in message content

Frontend:
- Add thinking field to MessageContent, ChatMessage, ChatTextEvent types
- Create ThinkingBlock.svelte — collapsible block with brain icon
- SSE handler moves text_delta content to thinking on isThinking flag
- Render thinking block between tools and response in ChatThread
- Fix chat window scroll reset on focus change (stable windowKeys order)
- Remove redundant #key id wrapper in WindowLayer
- Enlarge sidebar rail (24→32 default, 40→60 max)
- Remove glyph from sidebar, square graph at top
- Replace AgentTrace/ToolCallCard/UnifiedTimeline with TurnTrace/ToolLine
2026-08-04 22:42:53 +02:00
20adb89650 v0.18.0: MCP entity-graph CRUD, lifecycle validation, curl -o /dev/null fix
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- create_entity, set_entity_state, end_relationship MCP tools
- update_entity_attributes now triggers check derivation via EnsureEntityChecks
- shared db.EnsureEntityChecks + db.ValidateTransition hooks (HTTP + MCP parity)
- curl -o /dev/null now classified read_only (was config_mutation)
- db.ErrTransitionInvalid sentinel for HTTP error-type accuracy
- SOUL.md: capability escalation, self-grounding, exploration budget rules
- Runbook: oikos check lifecycle for agent self-knowledge
2026-08-04 08:52:08 +02:00
058f1afcdc fix(web): move composer working-strip into the message pane (stop clipping the input)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The "Working — message will queue" strip lived inside the sized input Pane, so
appearing/disappearing ate the textarea's fixed height and clipped it, forcing
a resize. Move it into the message Pane alongside the connection/error banners
— those correctly consume transcript space (flex-1) rather than the input's
fixed height. The input Pane is now stable whether or not a background turn is
running. Styling/idiom unchanged.

VERSION: 0.17.2 -> 0.17.3
2026-08-03 22:58:47 +02:00
2b73290994 fix(web): integrate composer "working/queued" strip into the terminal aesthetic
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The background-working hint was a plain muted text line bolted above the
textarea — misaligned with the input column and off-idiom. Restyle it as a
terminal status strip: spinner + uppercase fg "Working" label + muted detail,
hairline primary-tinted border (matching .trace.running), square, aligned to
the textarea's max-w-3xl column. Reads as part of the working state now.

VERSION: 0.17.1 -> 0.17.2
2026-08-03 22:56:14 +02:00
428f4fe945 docs(plans): add status notes missed by the rename (chat-full-polish, health-check-reality)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
git mv staged the pre-edit index content; the status edits to these two
files landed in the working tree but not the archive commit. Amending the
status now so the archived copies reflect Implemented.
2026-08-03 22:53:29 +02:00
195d45a0e9 docs(plans): reconcile plan statuses; archive 10 done plans
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Move ten completed plans from plans/ to plans/done/ and update the index:
- 2026-07-18 session-review-three-sessions, 2026-07-20 desktop-mascot,
  2026-07-20 session-review-ten-sessions, 2026-07-21 chat-full-polish,
  2026-07-29 health-check-reality-and-knowledge-graph,
  2026-07-30 session-review-plan-drift, and the four 2026-08-03 chat plans
  (changes-review, reliability-and-ux-audit, cyberspace-style-adoption,
  working-visibility).
- Refresh two stale statuses: cyberspace-style-adoption ("Draft" -> shipped as
  full replacement in v0.16.0/757ef2f) and health-check-reality ("ready for
  implementation" -> shipped across the v0.14.x-0.16.x check commits).
- .gitignore: ignore local tooling artifacts (.playwright-mcp/, config-screen.png).

No code change. index.md Active/Done tables now match the filesystem (no orphans).

VERSION: 0.17.0 -> 0.17.1
2026-08-03 22:52:25 +02:00
5b68bdc16c feat(nomos): chat working-visibility, message queue, generation-aware timeline
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Make background/long/desynced turns visible and queueable, fixing the four
symptoms that survived the v0.15.0 chat reliability pass.

F1 - status-driven working signal (workspace.ts taskWorking/currentWorking =
streaming OR status in {planning,executing}). Drives the chat trace, indicator,
and activity spinner so a turn with no live stream (background resume, a dropped
SSE, an idle-close mid long turn) still looks alive.

F2 - operator messages sent during an in-flight turn are now QUEUED and
auto-run when the gate frees, replacing the "still finishing a previous step...
send it again" rejection. Per-session in-memory FIFO (messagequeue.go, capped at
20) drained one-at-a-time under the turn gate; a `queued` SSE event drives a
"Queued" hint. drainQueued releases via a per-iteration deferred closure so a
runChatTurn panic can't deadlock the session's gate.

F3 - SSE keepalive (12s `:keepalive` comment) in handleChat so 20-40s
inter-iteration gaps no longer trip a proxy/browser idle close (the desync root
cause). All SSE writes serialized through one mutex.

F4 - generation-aware activity timeline (only the last propose_plan renders;
superseded ones collapse to one "Earlier plan revised" marker; step-attribution
follows only the current generation) + debounced plan refetch on lifecycle
events so a missed plan.proposed self-heals.

Verified against the last session (23da10db: 6m33s turn, operator "status"
deferred at 19:48:05). go test ./cmd/nomos/ green (new messagequeue tests);
web vitest 72/72 (new F4 generation tests); vite build clean.

VERSION: 0.16.0 -> 0.17.0
2026-08-03 22:34:14 +02:00
757ef2f34b feat(web): adopt cyberspace terminal aesthetic + dithered images
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Rebrand light/dark themes to the cyberspace.online look: warm cream-on-black
palette (light/dark are exact inverses), self-hosted JetBrains Mono + VT323,
square corners, border-driven surfaces with no soft shadows. Adds a terminal
design-system CSS layer (DOS double-border modals with hatched corner, fg focus,
inversion-on-hover), a theme-aware <RasterImage> (Atkinson-dithered canvas with
img fallback), and unifies desktop icons, taskbar, window controls, pills and
links under one idiom. Pins window titlebars to a fixed height and switches chat
auto-scroll off scrollIntoView to avoid titlebar reflow.

VERSION 0.15.1 -> 0.16.0
2026-08-03 22:03:24 +02:00
b27e1bf3ec fix(web): coerce chat composer draft to string (input.trim crash)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The Send button's disabled={!input.trim()} threw "trim is not a function" when
input was initialized from a non-string initialDraft (a Svelte 5 prop-init edge
where a null/undefined draft reached $state). Coerce at init so the composer
state is always a string.

VERSION: 0.15.0 -> 0.15.1
2026-08-03 16:00:01 +02:00
39e9227fdb feat(nomos): per-session turn serialization + chat reliability/UX fixes
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The agent could run two turns for one session at once (a reconnect resumed
while the live turn was still going), and their interleaved tool calls
corrupted the activity panel, fabricated a confusing "parallel/nested"
sequence, and made tasks feel stuck/never-ending. Several UX gaps compounded it.

Turn serialization (F1):
- turnGate: at most one in-flight turn per session. Background resume paths
  (continuation worker, idle sweep, answer-question, /resume, reconnect)
  skip non-blocking when busy; the live chat path waits briefly then bails
  cleanly instead of stacking a second turn.
- resumeSession returns whether it ran; continueSession marks an execution
  "continued" only after a real run (review P0) so a busy-skip can't lose a
  finished-execution result. Idle nudge bumps only after delivery (P1).

Connection state (F2/F3, web):
- humanize/bucket raw errors ("model connection dropped..."); one surface
  per drop; a terminal task.status event clears stuck streaming/disconnected
  state and dismisses the connection toast. Reconnect no longer spawns turns.

Streaming where you look (F4, web):
- live command output in the global activity timeline and in the inline
  tool card (auto-opened, tail-pinned) -- not just the per-window rail.

Other (web): artifact/knowledge deep links (F5); step-first stable
"thinking" headline (F6); stable chat layout, no empty->content reflow (F7);
lazy event sync (P2.2); reconnect skips a terminal session (P2.1).

VERSION: 0.14.2 -> 0.15.0
2026-08-03 15:42:10 +02:00
bb05f215c6 docs(plans): mark plan-drift & dead-activity-panel review done (467589d)
VERSION: 0.14.1 -> 0.14.2
2026-08-03 14:32:24 +02:00
467589d78a fix(nomos): generation-relative plan seq + real activity timestamps
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The plan recorded false history after a re-plan and the activity panel
showed fabricated, churning timestamps. Two bugs compounding on one event
stream.

Plan drift (P0.1):
- proposePlan seq is now 1..N per generation; (session,generation,seq) is
  the addressing key. The model's 1-based update_plan_step calls always map
  to the CURRENT plan after a re-plan, instead of resurrecting a superseded
  `replaced` row as done while the live work went unrecorded.
- updatePlanStep resolves against MAX(generation); a stale/out-of-range seq
  returns errPlanStepNotFound (never touches a superseded generation).
- getPlanSteps returns only the current generation by default; ?all=true
  keeps the audit/eval view (plan_generations assertion).
- completeTask auto-close scopes to the current gen, stamps started_at, and
  emits one plan.step.finished per closed step so the panel converges
  instead of freezing on "running" after completion (P1.1).
- propose_plan result enumerates step seqs; writeback detector matches
  "write back"/"writeback"/"upsert_knowledge" so a natural-language final
  step isn't doubled (P1.2).
- migration 029 renumbers existing seq per generation + unique index.

Activity panel (P0.2 / P1.1, web):
- computeActivityLog uses the real message created_at for tool calls; live
  entries fall back to wall-clock frozen on first sight, killing the 3s
  poll churn. Steps use real started_at.
- dropped plan-step events warn + count instead of a silent no-op.

Tests: TestProposePlan updated; + generation-relative-seq and auto-close
event-emission regression tests; + web activity purity/timestamp tests.

VERSION: 0.14.0 -> 0.14.1
2026-07-30 22:40:56 +02:00
e25e979757 chore(docker): ignore worktrees/git/node_modules from build context
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Every docker build sent the whole repo root as context, including every git
worktree under .claude/worktrees/ (200-300MB each) — that crossed 390MB of
cruft and starved mac-mini's disk mid-build on 2026-07-27 (873b00a). None of
it belongs in an image.
2026-07-30 00:10:07 +02:00
bc0ccb4cdc fix(seed): netbird-vps opts out of host monitoring (unreachable from lab; services cover it)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-30 00:04:28 +02:00
c9a00a9532 feat(checks): per-entity monitoring override; service:haos opts out
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
A `monitoring` attribute on an entity now overrides its type's declaration:
"none" opts out, a list overrides the kinds. service:haos uses it to opt out —
haos blocks SSH (no process probe can reach it) and the VM is already covered
by vm:haos's vm-status check, so the redundant process check only ever reported
false-down. vm:haos -> service:haos via provides confirms the coverage.
2026-07-29 23:47:43 +02:00
eb16796bf0 feat(checks): vm-status probe + matrix cert dial-by-name
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
VMs declared monitoring [ping], but many block ICMP and lack a guest agent
(haos), so ping was the wrong probe — a powered-on VM reported "down". Add a
vm-status check: `qm status <pve_id>` on the VM's Proxmox host, which tests
"powered on" without needing the VM's network at all. vm type monitoring is
now [vm-status].

matrix.hubris.network is a public hostname (federation) resolving to
netbird-vps, not served by the lab Caddy — so its cert-expiry check's
dial=caddy IP failed. Drop the dial for matrix; it dials by name (DNS ->
public) like wget already proved works.
2026-07-29 23:15:36 +02:00
a3914a1d41 fix(checks): process check is opt-in for url-fronted services
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The ontology's stated intent was "http when it has a url, else a process
check", but the implementation emitted BOTH for every url-service — so ~17
fronted services carried a redundant process check that, under worst-of
aggregation, let a fragile supplementary probe (wrong unit name, unreachable
host, no guest agent) veto two healthy http checks and report the service
"down" while it was up (authentik, zimaos, house, matrix, ...).

buildKind now emits a process check only for services WITHOUT a url, or when
an explicit probe_unit opts into binary-level depth. http is the canonical
service-liveness probe (tests the real endpoint through the TLS terminator);
the redundant process checks were removed.
2026-07-29 23:03:27 +02:00
8eb1ca2bac fix(seed): probe_unit for proxmox-ui/nextcloud/photos (real unit/container names)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-29 22:45:25 +02:00
e4104eb344 fix(checks): process_check matches docker containers + prefixed systemd units
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
process_check.sh ran `systemctl is-active <entity-name>`, but a service's name
is a logical label, not its unit/container name — matrix is matrix-synapse.service
+ element-web/mautrix-* containers, authentik is authentik-server/-worker
containers. So every multi-component or docker service reported "inactive"
while up (authentik, matrix, photos, house, arr-stack, …).

Resolve in order: exact systemd unit, a unit with the name as prefix
(matrix -> matrix-synapse.service), or a running docker container whose name
contains it. checkdefaults passes a declared probe_unit/systemd_unit/container
attribute when set, for precision.
2026-07-29 22:33:02 +02:00
0929c17cbb feat(mcp): discover_infra_drift — live Proxmox vs DB guest reconciliation
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The DB-only audit_knowledge_graph can't see guests running in Proxmox that
have no entity, or entities whose pve_id is no longer live — the drift that
the stray test LXCs were a symptom of. discover_infra_drift enumerates running
guests via pct/qm list on every proxmox host (over the same SSH/pct path the
checks use) and diffs against the DB: returns missing (live, no entity) and
ghost (DB, not live). Read-only.

Companion to audit_knowledge_graph; the skill now runs both and treats the
remaining checks (misplaced parent, undeployed scripts, seed drift) as manual.
2026-07-29 20:36:02 +02:00
6487032461 fix(remote): ignore polluted host attributes; audit surfaces them
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
resolveProxmoxHostSlug trusted attributes.host verbatim, so a value polluted
with prose — lxc:teddycloud carried host="hubris (confirmed via pct config…)" —
became a slug that never resolved, leaving its checks 'down' despite a correct
`hosts` edge. Treat an attribute containing whitespace/parens as invalid and
fall back to the canonical hosts edge.

The audit now reports `polluted_attrs` — entities whose routing-critical
attributes carry prose — so this class is visible instead of a silent
resolution failure.
2026-07-29 19:45:12 +02:00
fb6b6f9160 fix(scripts): also cascade relationships in orphan cleanup
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-29 19:40:13 +02:00
62c9fc5c86 fix(scripts): cascade entity_status/signals/metrics in orphan cleanup
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-29 19:39:52 +02:00
6007e922b4 fix(scheduler): lifecycle gate excluded NULL-state check targets
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The B1 target-state filter `tgt.state NOT IN ('deprecated','destroyed')`
evaluates to NULL (unknown) when a target's state is NULL, which the WHERE
clause treats as false — so freshly-seeded entities without an explicit state
(the 20 TLS certificates) were silently dropped from ListEnabledCheckDefs and
never monitored. Treat NULL state as active (only explicit deprecated/
destroyed is excluded): `tgt.state IS NULL OR tgt.state NOT IN (...)`.
2026-07-29 19:39:17 +02:00
2d8eb91b25 feat(cert): dial the TLS terminator directly so cert-expiry works from the container
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
checkCertExpiry now accepts a `dial` address and sets ServerName to the
hostname — it connects to the terminator's IP while SNI/cert-read use the
hostname. The scheduler container has no mesh interface and the host resolver
doesn't know the split-horizon zone, so *.hubris.network can't be dialed by
name from there; dialing Caddy's lab IP (reachable on the LAN) makes the probe
work. The builder passes through a cert entity's `dial` attribute.

Re-seed the 20 *.hubris.network certificate entities with dial=192.168.8.175
(Caddy) and uses-certificate edges; cert-expiry monitoring now has real data.
2026-07-29 19:33:45 +02:00
3d88f52988 fix(checks): disk_usage_check no longer hangs on a stuck mount
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
disk_usage_check.sh built its mount list with `df`, which blocks on a wedged
filesystem (stale NFS export, a stuck ZFS pool) — and that stalled the whole
check past the scheduler's 30s budget, leaving host:hubris:4 perpetually down.

Build the mount list from /proc/mounts (a read that never stats anything), and
bound every per-mount `df` with `timeout 8` so a single stuck mount is skipped
instead of hanging the probe. Degrades to plain `df` on hosts without
`timeout`//proc/mounts (macOS), whose local mounts don't hang.
2026-07-29 19:31:30 +02:00
9016c3a43b revert(seeds): drop TLS certificate entities (needs container reachability first)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The cert-expiry builder and checkCertExpiry probe are correct, but the
scheduler container can't reach *.hubris.network:443 — its DNS forwards to the
host resolver, which doesn't know the split-horizon zone, and overriding the
container DNS would break docker service-name resolution. Seeding the 20 cert
entities now produced 20 false-down certificates.

Keep the builder (committed), drop the entities + edges until the scheduler can
reach Caddy (extra_hosts mapping, or a SNI-dial enhancement) — then re-add them.
2026-07-29 18:49:37 +02:00
04775192c1 feat(checks): wire up TLS certificate expiry monitoring
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The ontology declared monitoring [cert-expiry] on the certificate type and a
working checkCertExpiry probe existed, but checkdefaults had no cert-expiry
builder and no certificate entities were seeded — so certificate expiry, a
real failure mode, was invisible.

Add a KindCertExpiry builder (dials the cert's hostname on :443 hourly, warns
at 30d / crit at 7d) and seed certificate entities for the 20 public
*.hubris.network routes plus uses-certificate edges from each ingress route.
2026-07-29 18:41:14 +02:00
a104cb4bb4 feat(remote): route service checks through their hosting compute entity
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
A service check used to bake its hosting LXC's lan_ip and SSH it directly as
root, which failed because the scheduler key is authorized on the Proxmox hosts
but not inside every guest — leaving all 8 service process checks 'down' even
after the guest routing and scripts were fixed.

ResolveExecTargetForCheck now, for a non-guest target, walks the
provides/runs-on/hosts edges to the compute entity that runs it and routes
through that: pct/qm exec if the host is a guest, direct SSH with the host's
correct user (workstation `user` attr) if it's a machine. The guest-resolution
path is shared via resolveGuest, and the scheduler no longer needs an
isMachine special case — one resolver handles guest, machine, and service.
2026-07-29 14:00:03 +02:00
72f0f46528 fix(scheduler): resolve guest routing when check_defs.target_type is blank
Older writeCheck inserts omitted target_type, so every seed-created check_def
had a NULL/empty target_type. checkSSHScript's IsGuest check then never matched,
and guest checks silently fell back to their baked (often mesh-only) address —
keeping them 'down' even after the pct-exec routing and deployed scripts were
in place. rclone stayed down for exactly this reason after the host-hop fix.

writeCheck now writes target_type, and checkSSHScript resolves the type from
the target_id when the column is blank (a runtime safety net for existing rows;
the seed rows were also backfilled in the live DB).
2026-07-29 13:44:14 +02:00
b87735a111 chore: graph view, dns-zone gap, fleet deploy/cleanup tooling
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Graph view: raise the node cap 500 -> 2000 and exclude execution/task audit
rows from the default whole-graph view so the cap is spent on actual topology
rather than ~380 cognition records that crowded out every host/lxc/service.

dns-zone monitoring [dns] -> none: no dns checker exists, so the declaration
only produced unresolvable `unmonitored` noise (requires ontology re-ingest;
coverageSweep now auto-clears the stale signals). Flip back to [dns] when a
checker lands.

Operator tooling: tools/deploy-checks.sh pushes check scripts into guests via
pct push (a pct-exec-routed check runs the script INSIDE the guest), wired
into the post-pull setup-checks hook so guests stay in sync on Proxmox hosts;
scripts/cleanup-orphan-checks.sh (dry-run by default) and
report-stray-test-lxcs.sh retire legacy cruft. VERSION 0.13.0 -> 0.14.0.

Plan: plans/2026-07-29-health-check-reality-and-knowledge-graph.md.
2026-07-29 13:37:27 +02:00
1540f74342 feat(audit): read-only knowledge-graph drift report + skill
Adds audit_knowledge_graph (MCP tool) and GET /api/v1/audit/drift (endpoint)
backed by a shared internal/audit package. One pass surfaces the structural
gaps an operator otherwise finds by accident: orphan check entities, checks
targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored
declared types, and live edges pointing at destroyed targets. Each finding
carries a suggested remediation runbook. Read-only and safe to run unattended.

Ships the knowledge-graph-audit skill (SKILL.md + seeded runbook) that
interprets the report and routes findings to the lifecycle runbooks.
2026-07-29 13:37:16 +02:00
c7729b2ef6 fix(scheduler): stop monitoring deprecated/destroyed targets
ListEnabledCheckDefs now LEFT JOINs the target entity and excludes rows whose
target is deprecated or destroyed, so retired things (secrets-issuance,
homelab-mcp, the dead secrets ingress route) stop generating permanent false
alarms instead of waiting for an operator to disable the check_def by hand.

coverageSweep's None() branch previously did nothing, so a type changed from
declared monitoring to `monitoring: none` (dns-zone) left its open
`unmonitored` signals lingering forever — a None() entity never gains a check,
so the hasCheck resolution path never fired. It now resolves those signals.
2026-07-29 13:37:08 +02:00
b8b4aa2aee feat(remote): route LXC/VM checks through the Proxmox host, not direct SSH
The scheduler SSHed each guest directly and assumed a deployed probe script
plus working root SSH at the guest's address — false for headless (nfs-export),
keyless (teddycloud), mesh-only (rclone), and macOS (mac-mini) targets, which
left 49 enabled checks stuck "down" on a healthy fleet.

Extract the MCP run tool's resolveExecTarget into a shared internal/remote
package and make it the single execution path for both the scheduler and MCP.
LXC/VM checks now host-hop via pct exec / qm guest exec through the owning
Proxmox host (no per-guest lan_ip, sshd, or authorized key needed); hosts and
workstations resolve their address and user live, so mac-mini's `user: dtoro`
is honored without a re-seed. Address preference now prefers public_ipv4 over
mesh, so netbird-vps is probeable from the scheduler container.

cpu_check.sh gains a real Darwin branch (it reported cpu_pct 0 before).
checkdefaults.resolveSSHUser reads the top-level `user` attribute too.
A machine-target resolution failure is now logged before falling back to baked
config, so a broken probe-config is distinguishable from a real outage.
2026-07-29 13:37:00 +02:00
c10f6920cd fix: blast radius walks dependency direction, and reachability survives no ICMP
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Two things the entity window redesign surfaced but deliberately left alone.

**blast_radius answered the wrong question.** It walked source→target for every
relationship type, but which end of an edge is the dependent differs per type:
"machine hosts container" means the target breaks, while "service depends-on
service" and "ingress routes-to service" mean the SOURCE breaks. Walking
everything forwards was right for hosts/provides and backwards for everything
else — and swept in 2,800+ documents/involves/targets edges of pure bookkeeping,
so the result contained tasks and executions that cannot break.

Direction is now declared per relationship type in seeds/ontology.yaml
(blast_direction: forward | backward | none), the same shape as the entity
types' monitoring: declaration, and defaults to none so an undeclared edge
contributes nothing rather than a confidently wrong answer.

It also needed a modelling fix: `routes-to` names an ingress's BACKEND, so
nothing recorded that all 21 public hostnames are terminated by caddy. A
`served-by` edge type now says so.

  pool:ludo-lvm    2 -> 23   (every container storing on it, then their services)
  lxc:caddy        4 -> 22   (service:caddy, then all 21 ingress routes)
  service:authentik      7   (what authenticates via it)

**Every ping check was reporting down.** Not a host:strong false positive: all
seven, including ws:mac-mini — the Docker host itself. The scheduler runs in
Docker on macOS, whose VM does not route ICMP to the LAN; loopback pings succeed
and every LAN ping fails. Under health aggregation each broken probe dragged its
entity to down.

The question the check exists to answer is "is it reachable", and ICMP is only
one way to ask it. checkPing now falls back to a TCP connect before concluding
anything, which restores an honest verdict for the four hosts that are genuinely
up while leaving the genuinely unreachable ones down.

TestBlastRadiusTerminatesOnCycles asserted the old direction (caddy=1,
authentik=2 — the cycle walked the wrong way); it now asserts the corrected
depths, and its exact-node-count check is relaxed because walking the right way
also surfaces the seed's own real dependents, which are correct answers.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 09:44:17 +02:00
ad29295c93 feat(web): make the entity window a triage surface, not a data dump
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The window rendered the same 13 collapsible sections for every entity, sorted
only by "does it have content". Audit trail carried the same visual weight as
Health, and the window answered "what data do we hold about X?" rather than
"what do I need to know, and what should I do?".

Measured against prod: host:hubris has 223 relations, 1,601 events, 2.7M metric
samples and 148 executions; an ingress route has three facts. Both got 13
identical headers. Expanding a host put ~540 interactive elements on screen.

- **A verdict header that never collapses.** Not just "down" but *why*:
  "ping failing · 5 of 6 checks passing". That line did not previously exist
  and could not have — checks rendered as configuration, never as results.
- **Sections composed per type.** A document has no checks, metrics or blast
  radius; a signal or execution is a record, not a thing. Infrastructure gets
  Status/Impact/Activity/Metrics/Reference, knowledge types lead with Content,
  records get a minimal view. Unknown types fall back to infrastructure so a
  new entity type is never a blank window.
- **Status replaces Monitoring**, showing each check's own verdict and when it
  last ran — the section that answers the header's "why".
- **Impact** finally calls /entities/{id}/blast-radius. The endpoint has existed
  since the first API and had no frontend caller anywhere, despite
  .agents/OIKOS.md naming blast radius as the reason the ontology exists. Its
  outgoing-edges-only limitation is stated in the UI rather than hidden.
- **Activity merges four lists** (executions, signals, events, agent activity)
  that were telling one story in four places.
- **Relations cap at 8 with a drill-in** — 540 interactive elements down to 126.
- **Ask Nomos** opens a task pre-scoped to what you are looking at, seeded with
  the verdict just computed, via an optional draft threaded through
  openNewTaskWindow -> NewTaskChat -> ChatThread.

Requires exposing check_defs.last_health/last_run_at through the API (the
columns landed with the health-aggregation work but were never surfaced).
Adding a fourth enum containing "unknown" made oapi-codegen disambiguate all
enum constants by type prefix, so metrics.go moves to gen.TrendDirection*.

Verdict derivation and type->section composition live in $lib/entityView.ts as
pure functions with 15 unit tests, including the host:strong case that
motivated this.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 09:24:28 +02:00
6ca6d5b352 fix(scheduler): derive entity health from all its checks, not the last one
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
host:strong logged 226 health.changed events in one hour, oscillating
down/healthy while the host was fine throughout. host:hubris did it 126 times.

runCheck wrote entity_status.health on every check completion, so an entity's
health was simply whichever of its checks finished most recently. A host with
six checks reported whichever facet happened to be sampled last, and one
failing probe alternating with five passing ones flapped forever. resolveSignal
forced "healthy" too, a second path by which one passing probe erased another
probe's genuine failure.

On this fleet the trigger is a known false positive: the scheduler's network
vantage point cannot ICMP host:strong, so its ping check fails while every
ssh-script check succeeds. Under last-writer-wins that single probe declared
the whole host down, twice a minute.

Each check now records its own verdict (check_defs.last_health, migration 027)
and the entity's health is the worst across its enabled checks. A failing probe
now degrades the entity honestly and *stably*, without erasing what the other
five report, and health.changed fires only when that aggregate actually moves.
Checks that have never run are ignored rather than counted as unknown, so
adding a check cannot drag a known-good entity down before it has a verdict.

Also declares service:oikos in the seed. The previous commit re-pointed the mcp
ingress at it, but the entity only ever existed in the production database — so
a fresh seed (a new install, or a DR restore) failed on an unresolvable edge.
Caught by seeding an empty database rather than a copy of prod, which is the
only way that class of bug shows up.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 21:33:48 +02:00
af450dac2a fix(web): break the effect feedback loop, and stop serving HTML as JavaScript
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Two unrelated console errors.

effect_update_depth_exceeded — mine, from the previous commit. The live-update
effects both read and wrote the same state: FleetMap's health patch builds a
new `graph` object every run, and EntityDetailContent's refreshExecutions()
assigns a fresh `executions` array. Svelte tracked those reads, so each write
re-triggered the effect, which wrote again, until it gave up. The effects now
depend on liveEvents alone and do their work inside untrack(). Applied to all
four live effects, including the two that happened to settle on their own —
relying on "applyHealthEvent returns the same reference when nothing changed"
to break a feedback loop is far too subtle to leave implicit.

SyntaxError: expected expression, got '<' — pre-existing, and unrelated to the
live-update work. index.html loads /wails/runtime.js unconditionally; that file
only exists inside the Wails desktop wrapper, which serves the same dist/ from
its own asset handler. In a browser it is missing, and the SPA fallback
answered it with index.html — so the browser parsed "<!doctype html>" as
JavaScript on every single page load. The web Caddyfile now returns a real 404
for /wails/*, and more generally serves asset extensions without the SPA
fallback: a missing .js or .css answered with HTML is always a confusing parse
error rather than an honest 404.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 21:22:03 +02:00
6ed9dc39e8 fix(compose): wait for the API to be healthy before starting nomos
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
nomos declared `depends_on: api: condition: service_started`, which only waits
for the container to exist. It came up while the API was still binding :8090,
failed its MCP initialize with "connection refused", exited 1, and crash-looped
for ~25 seconds on every single deploy. It always recovered on its own, which
is precisely why it went unnoticed.

service_healthy waits for the API to answer, so this needs api to declare a
healthcheck — wget is BusyBox's, already present in the alpine runtime image,
so nothing new is installed. /healthz pings the database, so "healthy" means
genuinely able to serve rather than merely listening.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 20:08:35 +02:00
cc8eae4979 perf(web): patch health in place instead of refetching, and reconnect the SSE stream
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Refetching everything on a health event was wasteful and churned the UI: one
container going degraded pulled down the entire fleet entity list (plus its
parent-grouping pass), or the whole fleet graph, to learn something the event
had already delivered.

health.changed / health.stale carry the new value in their payload, so the
views that hold the entity just patch it:

- Fleet table: patch the row. Only entity.* changes which entities exist, so
  only that still refetches.
- Fleet map: patch the node AND graph.health[id] — healthOf() reads the side
  map in preference to the node's own field, so patching only the nodes would
  have left the rendered colour unchanged.
- Entity detail: patch the open entity. Signals still need a read (the event
  says one was raised, not what the list now contains) but only the signals,
  not the entity and checks alongside them.

Shared in $lib/health.ts, which returns the original array when an event does
not apply so unrelated rows keep their identity and do not re-render. Note it
matches on entity_id, never data.slug: the scheduler emits health.changed with
entity_id = the observed entity but slug = the *check's* slug.

Separately, events.ts had no reconnect. onerror was empty on the assumption
the browser retries, but EventSource only does that for a transient failure --
once it reaches CLOSED (an HTTP error on connect, e.g. the API restarting
during a deploy) it stays closed forever. A single blip silently froze every
live surface in the app with nothing on screen to say so. Now reconnects with
capped exponential backoff, and exports eventsConnected so a future indicator
can show when the stream is down.

Verified against live prod: flipping lxc:apps health recoloured the map node
and moved its counts (30 healthy -> 29, 9 down -> 10) with ZERO network
requests.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 19:55:26 +02:00
4f706fa65f fix(web): keep health and status live everywhere they are shown
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The SSE stream already carried health.changed, health.stale, signal.raised,
signal.resolved and coverage.unmonitored, but two of the places that render
health never listened for them.

- The Fleet table refreshed only on entity.*, so its Health column sat at
  whatever it was when the page mounted while the map view beside it — which
  did listen — updated live. Health arrives on its own events, not entity.*.
  Coalesced on a 400ms timer because health.stale fires once per entity during
  a sweep, and refetching the whole fleet per event would mean a burst of
  identical requests.
- The entity detail window loaded health, signals and monitoring once on open
  and never again, so a window left on screen kept showing the health it had
  at mount. That is the same staleness this whole change set has been about,
  reproduced one window at a time. Now scoped by entity_id, and re-reads only
  what a health or signal event can actually change rather than re-running the
  full 11-request load().

Verified against live prod: flipping lxc:apps healthy -> degraded -> healthy
updated the Fleet table and an open detail window together, without a reload.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 15:05:06 +02:00
50e899e5ee fix(checks): stop process_check.sh emitting invalid JSON, and mint one kind
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
`systemctl is-active` prints the state AND exits non-zero when a unit is not
active, so `... || echo unknown` appended a second line: STATE became
"inactive\nunknown" and the script emitted a raw newline inside a JSON string.
The scheduler rejected all 14 process checks with "invalid character '\n' in
string literal".

Latent since the script was written — process checks never actually ran,
because checkdefaults wrote an `args` config the ssh-script checker ignored.
Passing args through finally executed them and exposed it.

- head -1 keeps the state, and the fallback only fires on empty output.
- Quotes are stripped from both the unit name and the state; either would
  break the hand-built JSON just as thoroughly.
- signalKind is now the constant "process" rather than "$SERVICE". Emitting
  the service name minted a distinct signal kind per service (kind=paperless,
  kind=qbit, …) — nothing an approval_rule can match, and it makes "how many
  process checks are failing?" unanswerable.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 14:41:56 +02:00
177 changed files with 40474 additions and 2605 deletions

View File

@@ -0,0 +1,87 @@
---
name: knowledge-graph-audit
risk_class: read_only
inputs: []
verification: "audit_knowledge_graph returns a report with summary.total_findings"
docs_update_checklist: []
---
# Knowledge-graph audit
Goal: validate that the knowledge graph (entities, relationships, checks) and
the monitoring built on it reflect live reality — without mutating anything.
Read-only. Run this before trusting health, blast-radius, or coverage answers,
and whenever something feels off (a healthy host reports `down`, a retired
service still alarms, the graph looks thin).
## 1. Run the drift report
Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
ranked list of findings, each with `{category, severity, count, entities,
evidence, suggested_runbook}`, plus a `summary` with totals by category.
The DB-side categories:
- **orphan_checks** — check entities with truncated/random slugs left by the
old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
`destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
(the scheduler already skips these, but the rows should be retired).
- **down_checks** — enabled probes reporting `down`. Remediation:
`service-health-check` (then check whether the failure is real or a
probe-config/routing problem — see step 3).
- **unknown_checks** — probes that ran but reported `unknown` (usually a
misconfigured or not-yet-deployed probe script).
- **unmonitored** — active entities whose type declares monitoring but have no
enabled `check_def`.
- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
## 2. Triage
`severity: critical` (down_checks) first. For each finding, read `evidence` and
open the entities with `get_entity` / `get_relations` to confirm the diagnosis
before acting — the report is a pointer, not a verdict.
## 3. Common probe-failure causes
A `down_checks` finding that is NOT a real outage is usually one of:
- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
exec` automatically when both are present.
- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
script flag. The scheduler resolves `user: dtoro` from the entity attribute.
## 4. What this audit does NOT cover (follow-ups)
Live-infrastructure discovery has its own tool — run **`discover_infra_drift`**
alongside this one. It compares running Proxmox guests (`pct`/`qm list` on every
proxmox host) against the DB graph and returns:
- **missing entities** — a guest running in Proxmox with no DB entity.
- **ghost entities** — a DB lxc/vm whose `pve_id` is no longer live.
Still manual until that machinery lands:
- **Misplaced parent** — compare each guest's actual Proxmox host against its
`hosts` edge (migrations leave these stale).
- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
- **Unmodeled certs** — now modeled; verify with `audit_knowledge_graph` /
the cert-expiry checks.
- **Seed drift** — run `oikos export` and `git diff seeds/` to find
runtime-created entities not in version control.
## 5. Acting on findings
This skill is read-only — make no changes here. Route each confirmed finding to
its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
proceed through the normal lifecycle/approval flow. Re-run the audit afterward
to confirm the finding cleared.
Docs-update checklist: none — the audit reads state; it changes nothing. If a
finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
in that remediation session.

12
.dockerignore Normal file
View File

@@ -0,0 +1,12 @@
# Every docker build in this repo previously sent the whole directory as
# build context — including every OTHER git worktree under .claude/worktrees/
# (each with its own web/node_modules, ~200-300MB apiece). That's what
# starved the mac-mini's disk mid-build on 2026-07-27 (SHA 873b00a): the
# context alone crossed 390MB of pure worktree cruft before the host ran out
# of space. None of this ever belonged in an image.
.claude/worktrees/
.git/
**/node_modules/
**/dist/
**/build/
*.log

4
.gitignore vendored
View File

@@ -24,3 +24,7 @@ cmd/desktop/build/
cmd/desktop/Oikos
desktop
/eval
# Local tooling artifacts (Playwright MCP session logs, stray screenshots)
.playwright-mcp/
config-screen.png

View File

@@ -1 +1 @@
0.13.0
0.26.0

View File

@@ -2284,6 +2284,23 @@ components:
type: boolean
version:
type: integer
last_health:
type: string
description: >-
This check's own most recent verdict. An entity's health is the
worst of these across its enabled checks, so this is what explains
*why* an entity is degraded. Null until the check first runs.
nullable: true
enum:
- healthy
- degraded
- down
- unknown
last_run_at:
type: string
format: date-time
description: When this check last executed. Null = never run.
nullable: true
CheckCreate:
type: object
required:

View File

@@ -0,0 +1,104 @@
# Oikos check lifecycle — how monitoring works
This runbook covers how Oikos health checks are derived, created, and wired so
an agent (Nomos) doesn't reverse-engineer source when asked to add monitoring to
an entity — the problem that stranded session `23da10db` (2026-08-03).
## Concepts
- **`check_defs`** (scheduler config, table `check_defs`): the row the scheduler
reads to know *what* to probe and *when*. One per check instance.
- **`check` entity** (type `check`, slug `check:<kind>:<target>:<n>`): the
knowledge-graph entity for that check. It carries attributes
(`check_type`, `target`, `port`, …) and `checks` edges to the probed target.
- **`monitoring` spec** on an entity type (`entity_types.monitoring_spec`): the
default list of check kinds (e.g. `[http, process]` for `service`).
- Per-entity override: set `monitoring` in the entity's attributes —
`"none"` for zero checks, `["http"]` to replace the type defaults.
- **`checkdefaults.Ensure`** (`internal/checkdefaults/defaults.go`): the
function that reads the monitoring spec, resolves host/port/URL from
attributes + relationships, and writes `check_defs` rows. Idempotent.
## When checks are derived
`checkdefaults.Ensure` runs in three situations (as of v0.17.1+):
1. **Seed/deploy ingest**`internal/db/seed.go:231`. Every entity gets its
default checks once on initial ingest.
2. **HTTP `POST /api/v1/entities` (create)**`ensureDefaultChecks` at
`internal/httpapi/impl.go:1012`. Creating an entity via the REST API derives
its checks in the same transaction.
3. **HTTP `PATCH /api/v1/entities` (patch)**`ensureDefaultChecks` at
`internal/httpapi/impl.go:1280`. Changing an entity's attributes (especially
`monitoring`) via the REST API regenerates its checks.
4. **MCP `create_entity`** — SAME hook. Creating an entity via the MCP tool
derives checks. (Added 2026-08-03; previously MCP had no create.)
5. **MCP `update_entity_attributes`** — SAME hook. Changing an entity's
`monitoring` attribute via MCP now regenerates checks. (Added 2026-08-03;
previously MCP updates silently skipped check derivation — the exact bug
that stranded the haos session.)
## Check slug grammar
```
check:<kind>:<target-type>:<target-name>:<n>
```
Examples: `check:http:service:jellyfin:0`, `check:vm-status:vm:haos:0`,
`check:cert-expiry:cert:house.hubris.network:0`.
## Adding monitoring to an entity
**If the entity already exists:**
```
update_entity_attributes(slug="service:haos", attributes={"monitoring":["http"]})
```
This regenerates checks via `checkdefaults.Ensure`. The result message tells you
how many checks were derived and whether any kinds were skipped (and why).
**If the entity does not exist yet (a new check, ingress, cert, etc.):**
```
create_entity(type="check", name="HAOS http check",
slug="check:http:service:haos:0",
attributes={"check_type":"http:service","target":"service:haos","port":"8123"})
```
This creates the entity AND derives its `check_defs`. Same for a new `ingress`
(`type=ingress`, monitoring `[http]`) or `cert` (`type=cert`,
monitoring `[cert-expiry]`).
**To remove monitoring:** set `monitoring:["none"]` or transition the entity
to a terminal lifecycle state (`set_entity_state``deprecated`/`destroyed`).
## Caveats
- **A service without a `url` attribute AND without a `probe_unit` gets no
process check** (the http check covers liveness; the process check would
be redundant without an opt-in `probe_unit`). The skip is logged.
- **A service whose address comes from a `hosts` edge** may produce no checks on
initial create because the edge doesn't exist yet — the next inventory ingest
(or a later `update_entity_attributes` after the edge is created) fills it in.
- **A `not found` error from `update_entity_attributes`** means the entity
doesn't exist — use `create_entity` instead.
- **`check_defs` has target columns** (`target_id`, `target_type`). A check
entity needs a `checks` relationship (`create_relationship(source=check:…,
target=service:…, type="checks")`) so the scheduler can resolve what to
probe. `create_entity` derives the check_def; `create_relationship` links
the check entity to its target in the graph.
## Related files
- `internal/checkdefaults/defaults.go``Ensure`, `Target`, `LogResult`
- `internal/httpapi/default_checks.go``ensureDefaultChecks` (HTTP hook)
- `internal/db/checks.go``db.EnsureEntityChecks` (shared hook)
- `internal/db/seed.go` — seed-time check derivation
- `internal/mcp/tools.go``create_entity`, `update_entity_attributes`
## Revision history
- **2026-08-03:** Created after session `23da10db` stranded for lack of entity-
creation tool and unawareness of check-derivation triggers. Covers the MCP
create_entity + update_entity_attributes regen paths added same day.

View File

@@ -2,12 +2,27 @@
# cpu_check.sh — CPU usage % and thermal temperature.
set -euo pipefail
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
if [ -z "$USAGE" ]; then
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
os=$(uname -s)
if [ "$os" = "Darwin" ]; then
# `top -l 1 -n 0` prints "CPU usage: X% user, Y% sys, Z% idle".
# Usage is 100 minus the idle figure that precedes the literal `idle`.
USAGE=$(top -l 1 -n 0 -s 0 2>/dev/null | awk '
/^CPU usage/ {
for (i = 1; i <= NF; i++) {
if ($i == "idle") { gsub(/%/, "", $(i - 1)); printf "%.1f", 100 - $(i - 1) }
}
}' || true)
else
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
if [ -z "$USAGE" ]; then
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
fi
fi
[ -z "$USAGE" ] && USAGE=0
TEMP=""
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)

22
checks/disk_usage_check.sh Normal file → Executable file
View File

@@ -2,18 +2,34 @@
# disk_usage_check.sh — disk usage and inode usage per mountpoint.
set -euo pipefail
MOUNTS=$(df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
# `timeout` caps each df so a single hung/stale mountpoint (a stale NFS
# export, a wedged ZFS pool) can't stall the whole check — that hung the
# scheduler's 30s budget on hubris. Available on Linux (coreutils); absent on
# Darwin, whose local mounts don't hang, so it degrades to an empty prefix.
TO=""
if command -v timeout >/dev/null 2>&1; then TO="timeout 8"; fi
# Build the mount list WITHOUT statting anything: reading /proc/mounts never
# blocks the way `df` does on a stuck filesystem, so the enumeration itself
# can't hang. Fall back to `df` on hosts without /proc/mounts (macOS).
if [ -r /proc/mounts ]; then
MOUNTS=$(awk '$1 ~ /^\// && $2 !~ /^\/(snap|dev|proc|sys|run|private)/ {print $2}' /proc/mounts || true)
else
MOUNTS=$($TO df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
fi
FIRST=1
echo -n '{"health":"healthy","metrics":{'
for m in $MOUNTS; do
LINE=$(df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
# Each df is bounded: a stuck mount times out and is skipped (LINE empty)
# rather than hanging the probe.
LINE=$($TO df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
if [ -z "$LINE" ]; then continue; fi
USED=$(echo "$LINE" | awk '{print $1}')
FREE=$(echo "$LINE" | awk '{print $2}')
PCT=$(echo "$LINE" | awk '{print $3}')
INODE_LINE=$(df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
INODE_LINE=$($TO df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
INODE_PCT=$(echo "${INODE_LINE:-0}" | sed 's/-/0/')
KEY=$(echo "$m" | sed 's|/|_|g' | sed 's|^_||')

View File

@@ -1,5 +1,17 @@
#!/usr/bin/env bash
# process_check.sh — systemd service liveness.
# process_check.sh — service liveness.
#
# A service entity's name is a logical label, rarely the literal systemd unit
# or container name. matrix = matrix-synapse.service + element-web/mautrix-*
# containers; authentik = authentik-server/-worker containers. So checking
# `systemctl is-active matrix` reports "inactive" for a healthy service.
#
# Resolution order, any hit = healthy:
# 1. exact systemd unit `systemctl is-active <name>`
# 2. a systemd unit with the name as prefix `<name>*.service`
# 3. a running docker container whose name contains <name>
# An explicit probe target overrides the label — see checkdefaults, which
# passes a `probe_unit`/`container`/`systemd_unit` attribute as $1 when set.
set -euo pipefail
SERVICE="${1:-}"
@@ -8,15 +20,31 @@ if [ -z "$SERVICE" ]; then
exit 0
fi
if ! command -v systemctl >/dev/null 2>&1; then
echo '{"health":"unknown","signalKind":"process-check","evidence":"systemctl not found"}'
exit 0
ok() { echo "{\"health\":\"healthy\"}"; exit 0; }
# 1. exact systemd unit
if command -v systemctl >/dev/null 2>&1; then
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null | head -1 || true)
[ "$STATE" = "active" ] && ok
# 2. prefix match: matrix -> matrix-synapse.service, house -> house.service, etc.
# --no-legend strips the header/footer so grep can see the unit rows; the
# pattern is a systemd unit glob.
if systemctl list-units --type=service --state=active --no-legend "$SERVICE*.service" 2>/dev/null \
| grep -q '\.service'; then
ok
fi
fi
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null || echo "unknown")
if [ "$STATE" = "active" ]; then
echo "{\"health\":\"healthy\"}"
else
echo "{\"health\":\"degraded\",\"signalKind\":\"$SERVICE\",\"evidence\":\"$SERVICE is $STATE\"}"
# 3. a running docker container whose name contains the label.
if command -v docker >/dev/null 2>&1; then
if docker ps --filter "status=running" --filter "name=$SERVICE" --format '{{.Names}}' 2>/dev/null \
| grep -q .; then
ok
fi
fi
STATE=${STATE:-inactive}
STATE=${STATE//\"/}
SAFE_SERVICE=${SERVICE//\"/}
echo "{\"health\":\"degraded\",\"signalKind\":\"process\",\"evidence\":\"$SAFE_SERVICE is $STATE (no active unit/container matched)\"}"

View File

@@ -57,6 +57,13 @@ type agent struct {
apiBase string // oikos HTTP API base, derived from NOMOS_MCP_URL, for chat-assent approvals
apiToken string // OIKOS_MCP_BEARER_TOKEN — api's combinedAuth requires it (no dev-open bypass)
httpClient *http.Client
// gate serializes turns per session (at most one in-flight turn per
// sessionID). See turngate.go and plan 2026-08-03 F1.
gate *turnGate
// queue holds operator messages that arrived while a turn was already
// running; they are auto-run when the gate frees (plan 2026-08-03 F2).
// See messagequeue.go.
queue *messageQueue
}
func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug string) (*agent, error) {
@@ -117,6 +124,8 @@ func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug
apiBase: apiBase,
apiToken: os.Getenv("OIKOS_MCP_BEARER_TOKEN"),
httpClient: &http.Client{Timeout: 15 * time.Second},
gate: newTurnGate(),
queue: newMessageQueue(),
}, nil
}
@@ -172,6 +181,11 @@ type agentEvent struct {
Data any `json:"data,omitempty"`
SessionID string `json:"session_id,omitempty"`
Iteration int `json:"iteration,omitempty"`
// IsThinking marks text/text_delta events that carry the model's internal
// reasoning (text produced before tool calls in the same iteration), as
// distinct from the final response text. The frontend renders these as
// collapsible thinking blocks separated from the response.
IsThinking bool `json:"is_thinking,omitempty"`
}
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
@@ -368,7 +382,12 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
var msg openai.ChatCompletionMessage
var acc openai.ChatCompletionAccumulator
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
// Capture token usage from this LLM response for activity logging.
// Previously always NULL — every agent_activity row had no token
// count. Now each tool call in this iteration gets the same total.
totalTokens := 0
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
acc = openai.ChatCompletionAccumulator{}
stream := a.provider.Chat.Completions.NewStreaming(ctx, params, a.reqOpts...)
for stream.Next() {
@@ -400,14 +419,19 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
msg = acc.Choices[0].Message
finishReason := acc.Choices[0].FinishReason
// Capture token usage from this iteration.
if acc.Usage.TotalTokens > 0 {
totalTokens = int(acc.Usage.TotalTokens)
}
if len(msg.ToolCalls) == 0 {
if isRefusalOrEmpty(msg.Content) {
if attempt < maxLLMRetries {
slog.Warn("nomos: empty or refusal response, retrying",
"session", sessionID, "iter", i+1, "attempt", attempt+1,
"content_len", len(msg.Content), "finish_reason", finishReason)
continue
}
if attempt < maxLLMRetries {
slog.Warn("nomos: empty or refusal response, retrying",
"session", sessionID, "iter", i+1, "attempt", attempt+1,
"content_len", len(msg.Content), "finish_reason", finishReason)
continue
}
// B.4: surface the real error context (finish_reason +
// refusal text) instead of a generic "empty response" —
// the operator can tell "content_filter — rephrase" from
@@ -456,7 +480,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
// led to each step. Emitting it lets the persist layer accumulate
// per-iteration reasoning into the row's text field.
if strings.TrimSpace(msg.Content) != "" {
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID})
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID, IsThinking: true})
}
slog.Info("nomos: tool calls", "count", len(msg.ToolCalls), "iter", i+1, "correlation", correlationID)
@@ -491,7 +515,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
slog.Warn("nomos: run retry cap hit — refusing dispatch",
"target", t, "failures", n, "session", sessionID)
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args,
tc.Function.Arguments, directive, 0, false, correlationID)
tc.Function.Arguments, directive, 0, false, correlationID, totalTokens)
emit(agentEvent{
Type: "tool_result",
Data: map[string]any{"name": tc.Function.Name, "result": directive, "id": tc.ID, "retry_capped": true},
@@ -542,7 +566,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
inputStr := string(inputJSON)
if callErr != nil {
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID)
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID, totalTokens)
// Retry cap: dispatch errors (e.g. MCP client timeout)
// count toward the cap too. A command that keeps timing
@@ -572,7 +596,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
}
resultJSON, _ := json.Marshal(result)
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID)
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID, totalTokens)
// Link any execution this tool queued/started back to this
// session, so the auto-continuation worker can feed its result

View File

@@ -76,16 +76,20 @@ func (a *agent) processIdleSweep(ctx context.Context) {
s := s
if s.CompletionNudges == 0 {
safego.Go("nomos:idle-nudge:"+s.ID, func() {
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
return
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
s.Goal, idleTaskThreshold)
note = a.store.enrichResumeNote(ctx, s.ID, note)
// P1: only count the nudge if it actually delivered. resumeSession
// skips (returns false) when a turn is already active; bumping the
// counter anyway would make the next sweep auto-close a merely-busy
// session as "unanswered."
if a.resumeSession(ctx, s.ID, note) {
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
}
}
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
s.Goal, idleTaskThreshold)
note = a.store.enrichResumeNote(ctx, s.ID, note)
a.resumeSession(ctx, s.ID, note)
})
continue
}
@@ -163,7 +167,9 @@ func (a *agent) processContinuations(ctx context.Context) {
continue
}
}
a.store.markContinued(ctx, p.ExecID) // stamp first: a failure here must not cause a re-continue loop
// markContinued now happens inside continueSession, AFTER resumeSession
// actually runs (P0). Pre-marking here consumed the item even when
// resumeSession skipped on a busy session, losing the result.
safego.Go("nomos:continue-session:"+p.SessionID, func() { a.continueSession(ctx, p) })
}
}
@@ -179,7 +185,18 @@ func (a *agent) processContinuations(ctx context.Context) {
// something new to poll for.
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
a.resumeSession(ctx, p.SessionID, buildContinuationNote(p))
// P0 (plans/2026-08-03-nomos-chat-changes-review.md): mark the execution
// continued ONLY after the turn actually ran. resumeSession skips (returns
// false) when another turn is already active for this session; marking
// before that — as the old code did — consumed the item (continued_at set,
// never re-queued by pendingContinuations) and silently lost the result.
// On a skip, leave it pending so the next worker tick retries once the
// active turn frees the permit.
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
return
}
a.store.markContinued(ctx, p.ExecID)
}
// resumeSession re-invokes the agent for a session with a system-injected note —
@@ -187,7 +204,32 @@ func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
// (handleAnswerQuestion) — persisting progress LIVE (a placeholder row updated
// in place as each tool call lands) so the frontend poller sees each step,
// instead of total silence until the whole resume concludes.
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
//
// F1 (plan 2026-08-03): this is the single entry point for EVERY background
// turn — the continuation worker, idle sweep, answer-question, /resume, and the
// empty-message reconnect all funnel through here. It acquires the session's
// turn permit non-blocking and SKIPS if a turn is already running. A duplicate
// resume while a turn (live or background) is active is exactly the
// interleaving that corrupted the activity panel and made tasks feel stuck.
//
// Returns whether the turn actually ran. Callers that mutate state before
// resuming (the continuation worker's markContinued, the idle sweep's nudge
// bump) MUST gate that mutation on a true return — otherwise a busy-skip leaves
// the state changed but the work undone (lost continuation / false auto-close).
// See plans/2026-08-03-nomos-chat-changes-review.md P0/P1.
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
if !a.gate.acquire(sessionID, 0) {
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
return false
}
// Release the gate, then drain any operator message that was queued while
// this background turn ran (plan 2026-08-03 F2). Queued messages are run as
// real user turns server-side; resumeSession itself never enqueues.
defer func() {
a.gate.release(sessionID)
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
}()
placeholder, _ := json.Marshal(map[string]any{
"role": "assistant",
"text": "",
@@ -200,6 +242,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
var toolCalls []map[string]any
var finalText, errText string
var finalThinking string
persist := func() {
if msgID == uuid.Nil {
@@ -212,6 +255,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
body, _ := json.Marshal(map[string]any{
"role": "assistant",
"text": text,
"thinking": finalThinking,
"tool_calls": toolCalls,
"auto": true, // marks this as an autonomous continuation, not an operator turn
})
@@ -242,15 +286,19 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
if attempt > 0 {
select {
case <-cctx.Done():
return
return true // a turn ran on an earlier attempt; consume, don't re-loop
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
}
}
toolCalls, finalText, errText = nil, "", ""
finalThinking = ""
// P3: accumulate per-iteration reasoning instead of overwriting
// (same fix as main.go's chat handler). Without this, a resumed
// turn's intermediate thinking is lost on reload.
// (same fix as main.go's chat handler). Without this, a resumed
// turn's intermediate thinking is lost on reload.
var textParts []string
var thinkingParts []string
emit := func(ev agentEvent) {
if ev.Type == "tool_use" || ev.Type == "tool_result" {
if m, ok := ev.Data.(map[string]any); ok {
@@ -277,8 +325,13 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
}
if ev.Type == "text" {
if t, ok := ev.Data.(string); ok && t != "" {
textParts = append(textParts, t)
finalText = strings.Join(textParts, "\n\n")
if ev.IsThinking {
thinkingParts = append(thinkingParts, t)
finalThinking = strings.Join(thinkingParts, "\n\n")
} else {
textParts = append(textParts, t)
finalText = strings.Join(textParts, "\n\n")
}
persist()
}
}
@@ -314,9 +367,10 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
// No placeholder was inserted (rare), save directly.
a.store.saveMessage(context.Background(), sessionID, "assistant", body)
}
return // do not call persist() again — already persisted above
return true // do not call persist() again — already persisted above
}
persist() // final state — same row, updated one last time with the concluding text
return true
}
// buildContinuationNote frames the finished execution for the model: what

View File

@@ -1,6 +1,11 @@
package main
import "testing"
import (
"context"
"testing"
"github.com/google/uuid"
)
func TestExtractExecutionIDs(t *testing.T) {
// Real tool-result phrasings that should yield an execution id.
@@ -37,3 +42,37 @@ func TestExtractExecutionIDs(t *testing.T) {
t.Errorf("expected de-dup to 1 id, got %v", ids)
}
}
// TestResumeSession_SkipsWhenBusy guards the P0 fix
// (plans/2026-08-03-nomos-chat-changes-review.md): resumeSession must skip —
// return false, body never executed — when a turn is already active for the
// session. continueSession relies on this so it only marks a continuation
// "continued" after a turn really ran (otherwise the result is lost: marked
// continued, never re-queued by pendingContinuations).
//
// A minimal agent with only a gate is enough: if the body ever ran, chatWith
// would dereference the nil provider and panic. Returning false cleanly proves
// the body was skipped.
func TestResumeSession_SkipsWhenBusy(t *testing.T) {
a := &agent{gate: newTurnGate()}
if !a.gate.acquire("sess", 0) {
t.Fatal("precondition: initial acquire should succeed on a free session")
}
ran := a.resumeSession(context.Background(), "sess", "note")
if ran {
t.Fatal("resumeSession must return false (skip) while a turn is active for the session")
}
}
// TestContinueSession_DefersWhenBusy guards the other half of P0: when the
// session is busy, continueSession defers (leaves the execution pending for the
// next worker tick) instead of running or marking it. It must return cleanly
// without reaching resumeSession's body (nil provider → panic) or markContinued.
func TestContinueSession_DefersWhenBusy(t *testing.T) {
a := &agent{gate: newTurnGate()}
if !a.gate.acquire("sess", 0) {
t.Fatal("precondition: initial acquire should succeed on a free session")
}
p := pendingContinuation{ExecID: uuid.New(), SessionID: "sess", Status: "completed"}
a.continueSession(context.Background(), p) // must not panic; must not run/mark
}

View File

@@ -319,8 +319,10 @@ func fetchTranscript(ctx context.Context, gateway, sid string) (transcript, sess
}
// Fetch the plan (steps with generation numbers) for the
// plan_generations assertion. A 404 or empty response is fine — a
// pure-DB Q&A with no propose_plan has no plan.
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan"); perr == nil {
// pure-DB Q&A with no propose_plan has no plan. ?all=true returns every
// generation so the assertion can count them (the default view returns
// only the current generation).
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan?all=true"); perr == nil {
if planResp.StatusCode == 200 {
pb, _ := io.ReadAll(planResp.Body)
_ = json.Unmarshal(pb, &t) // fills t.PlanSteps via "steps" field

View File

@@ -167,6 +167,147 @@ func sseEvent(w http.ResponseWriter, flusher http.Flusher, event agentEvent) {
flusher.Flush()
}
// runChatTurn is the shared core of an operator-initiated turn: insert an
// assistant placeholder, run a.chat with incremental persistence (so whatever
// happened before an abort is never lost), finalize the row, and derive a
// title. It is agnostic to the transport: `sink` receives every agent event
// for delivery (SSE for a live handleChat, a no-op for a queued turn that has
// no client attached — the frontend learns about those via the poller + the
// status-driven "working" signal). The caller MUST already hold the session's
// turn-gate permit.
func (a *agent) runChatTurn(pctx, ctx context.Context, sessionID, message string, sink func(agentEvent)) {
toolCalls := []map[string]any{}
// P3: accumulate per-iteration reasoning instead of overwriting with the
// final `text` event (see the original inline comment in handleChat).
var textParts []string
var thinkingParts []string
var finalText string
var finalThinking string
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
msgID, err := a.store.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
if err != nil {
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
}
persist := func() {
if msgID == uuid.Nil {
return
}
body, _ := json.Marshal(map[string]any{
"role": "assistant",
"text": finalText,
"thinking": finalThinking,
"tool_calls": toolCalls,
})
a.store.updateMessage(pctx, msgID, body)
}
a.chat(ctx, sessionID, message, func(ev agentEvent) {
if ev.Type == "tool_use" || ev.Type == "tool_result" {
if m, ok := ev.Data.(map[string]any); ok {
m["type"] = ev.Type
// One entry per tool call: tool_use creates it, tool_result
// merges the result into the same entry (matched by id).
id, _ := m["id"].(string)
if id != "" && ev.Type == "tool_result" {
for _, existing := range toolCalls {
if eID, _ := existing["id"].(string); eID == id {
for k, v := range m {
existing[k] = v
}
break
}
}
} else {
toolCalls = append(toolCalls, m)
}
}
persist() // live: survives even if the client disconnects right after
}
if ev.Type == "text" {
if t, ok := ev.Data.(string); ok && t != "" {
if ev.IsThinking {
thinkingParts = append(thinkingParts, t)
finalThinking = strings.Join(thinkingParts, "\n\n")
} else {
textParts = append(textParts, t)
finalText = strings.Join(textParts, "\n\n")
}
persist()
}
}
sink(ev)
})
// B.6: if the turn ended with no text and no tool calls (the model
// empty-response'd and all retries failed), delete the placeholder row
// instead of persisting an empty bubble.
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
a.store.deleteMessage(pctx, msgID)
} else {
persist() // final state — same row, updated one last time
}
// Title: prefer the goal once set; else the first assistant answer.
if finalText != "" && sessionID != "ephemeral" {
var goalTitle string
if sess, gerr := a.store.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
goalTitle = truncate(sess.Goal, 120)
}
title := goalTitle
if title == "" {
title = truncate(finalText, 80)
}
if title != "" {
a.store.updateSessionTitle(pctx, sessionID, title)
}
}
}
// drainAcquireWait is how long drainQueued blocks for a busy gate before
// re-queuing and deferring to the holder's own release-drain. A package var so
// tests can shorten it; in production it just needs to outlast the brief
// release→drain handoff window.
var drainAcquireWait = 5 * time.Second
// drainQueued runs every queued operator message for a session as its own turn,
// one at a time, under the turn gate. Called (in a goroutine) whenever a turn
// releases the gate — from handleChat (live) and resumeSession (background) —
// so a message queued while the agent was busy is acted on as soon as it's
// free, without the operator re-sending. See messagequeue.go (plan 2026-08-03
// F2).
//
// Each queued turn is persisted incrementally and has no SSE client (the
// browser detached after receiving the `queued` event); the frontend sees the
// result via the 3s poller and the status-driven "working" indicator.
func (a *agent) drainQueued(ctx context.Context, sessionID string) {
for {
msg, ok := a.queue.dequeue(sessionID)
if !ok {
return
}
// Block briefly for the gate. If a live turn grabbed it first, put the
// message back — that turn's release will drain it again. Never stack.
if !a.gate.acquire(sessionID, drainAcquireWait) {
a.queue.requeueFront(sessionID, msg)
return
}
slog.Info("nomos: running queued operator message", "session", sessionID)
pctx := context.Background()
// Run the turn inside a per-iteration closure so the gate release is
// deferred to the end of THIS turn (and runs even if runChatTurn
// panics — safego recovers the panic at the goroutine boundary, so a
// non-deferred release would be skipped and the session's permit held
// forever, deadlocking all future turns). A bare `defer release` in
// the loop would be wrong too: Go defers run at function exit, not
// iteration exit, so the gate would stay held across iterations.
func() {
defer a.gate.release(sessionID)
a.runChatTurn(pctx, ctx, sessionID, msg, func(agentEvent) {})
}()
}
}
func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", 405)
@@ -186,12 +327,22 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
return
}
// Empty message with an existing session = reconnect/resume. The
// frontend sends this after a dropped SSE stream to re-establish the
// connection and catch up on any auto-continuation work that happened
// while disconnected. Route into resumeSession so the agent sees a
// system note and reports current state.
// Empty message with an existing session = reconnect/resume. This path is
// defensive now — the frontend (post F2) recovers a dropped SSE via the
// poller + terminal task.status clearing, and no longer POSTs empty
// messages. If a client ever does, route into resumeSession so the agent
// reports current state — but SKIP a terminal session (done/failed/
// abandoned): there's nothing to resume, and running a "report state"
// turn there is just a spare turn the operator never asked for (P2.1).
if req.Message == "" && req.SessionID != "" {
if sess, err := st.getSession(context.Background(), req.SessionID); err == nil {
switch sess.Status {
case "done", "failed", "abandoned":
slog.Info("nomos: reconnect skipped — session already terminal", "session", req.SessionID, "status", sess.Status)
w.WriteHeader(202)
return
}
}
slog.Info("nomos: reconnect", "session", req.SessionID)
safego.Go("nomos:reconnect:"+req.SessionID, func() {
base := "[System: the operator's connection was re-established. The task may have progressed in the background.]"
@@ -200,7 +351,7 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
})
// Return 202 so the frontend doesn't try to consume an SSE stream
// from this POST — resumeSession writes to the DB directly and
// the poller (already running from handleDisconnect) picks it up.
// the poller picks it up.
w.WriteHeader(202)
return
}
@@ -217,6 +368,17 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
w.Header().Set("X-Accel-Buffering", "no") // disable proxy buffering
w.WriteHeader(200)
// All writes to w (events + the keepalive comment below) go through one
// mutex: http.ResponseWriter is NOT safe for concurrent use, and the
// keepalive ticker runs alongside the turn's event sink (plan 2026-08-03
// F3). Without this, interleaved writes corrupt the SSE stream.
var writeMu sync.Mutex
writeEvent := func(ev agentEvent) {
writeMu.Lock()
defer writeMu.Unlock()
sseEvent(w, flusher, ev)
}
ctx := r.Context()
sessionID := req.SessionID
@@ -268,111 +430,65 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
st.answerQuestion(pctx, sessionID, qid, req.Message)
}
sseEvent(w, flusher, agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
writeEvent(agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
toolCalls := []map[string]any{}
// P3: accumulate per-iteration reasoning instead of overwriting with
// the final `text` event. The agent loop emits a `text` event for each
// LLM iteration that produced text (intermediate reasoning before tool
// calls + the final answer). Without accumulation, only the last `text`
// survives in the persisted row — a reload shows the final summary but
// not the thinking that led to each tool call.
var textParts []string
var finalText string
// Incremental persistence, mirroring resumeSession's existing
// placeholder+update pattern (continue.go): insert a placeholder now,
// update the SAME row after every tool call, so whatever happened before
// an abort is never lost — only what hadn't happened yet is.
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
msgID, err := st.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
if err != nil {
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
}
persist := func() {
if msgID == uuid.Nil {
return
}
body, _ := json.Marshal(map[string]any{
"role": "assistant",
"text": finalText,
"tool_calls": toolCalls,
})
st.updateMessage(pctx, msgID, body)
// F1/F2 (plan 2026-08-03): serialize turns per session. The user message is
// already persisted above, so it is never lost. Wait briefly for a finishing
// background turn; if one is still running after that, QUEUE this message
// (don't reject it) and tell the client so it shows a "queued" state. The
// in-flight turn's release drains the queue (drainQueued) and runs it as a
// real turn server-side. This never stacks concurrent turns — the gate still
// guarantees one in-flight turn per session.
const turnWait = 5 * time.Second
if !a.gate.acquire(sessionID, turnWait) {
a.queue.enqueue(sessionID, req.Message)
slog.Info("nomos: turn already active, queued operator message", "session", sessionID)
writeEvent(agentEvent{Type: "queued", Data: sessionID, SessionID: sessionID})
writeEvent(agentEvent{Type: "done", Data: map[string]any{
"session_id": sessionID,
"queued": true,
}, SessionID: sessionID})
return
}
defer func() {
a.gate.release(sessionID)
// Run any message that was queued while this turn held the gate. In a
// goroutine so the HTTP response finishes without waiting on the next
// turn; the queued turn has no SSE client of its own.
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
}()
a.chat(ctx, sessionID, req.Message, func(ev agentEvent) {
if ev.Type == "tool_use" || ev.Type == "tool_result" {
if m, ok := ev.Data.(map[string]any); ok {
m["type"] = ev.Type
// One entry per tool call: tool_use creates it, tool_result
// merges the result into the same entry (matched by id).
// Before this fix, both events appended separate entries,
// doubling every tool call in the persisted transcript
// (confirmed pre-existing in d9cdcee1, v0.3.x era).
id, _ := m["id"].(string)
if id != "" && ev.Type == "tool_result" {
for _, existing := range toolCalls {
if eID, _ := existing["id"].(string); eID == id {
for k, v := range m {
existing[k] = v
}
break
}
}
} else {
toolCalls = append(toolCalls, m)
}
}
persist() // live: survives even if the client disconnects right after
}
if ev.Type == "text" {
// P3: accumulate. Each `text` event is one iteration's reasoning
// (or the final answer). Join with newlines so the persisted row
// reads as the full transcript of what the agent said, not just
// the last thing.
if t, ok := ev.Data.(string); ok && t != "" {
textParts = append(textParts, t)
finalText = strings.Join(textParts, "\n\n")
persist()
// F3 (plan 2026-08-03): keep the SSE alive during long turns. A turn can
// run for many minutes (provisioning chains, deep research); the model
// often takes 20-40s between tool iterations, and with nothing flushed in
// that gap a proxy/browser idle timeout silently closes the stream. The
// client then sees streaming=false while the server keeps working — the
// "I can't tell it's working" desync. An SSE comment line (":keepalive") is
// ignored by EventSource but resets idle timers.
keepDone := make(chan struct{})
go func() {
t := time.NewTicker(12 * time.Second)
defer t.Stop()
for {
select {
case <-keepDone:
return
case <-t.C:
writeMu.Lock()
fmt.Fprintf(w, ":keepalive\n\n")
flusher.Flush()
writeMu.Unlock()
}
}
sseEvent(w, flusher, ev)
}()
// Defer the close (not a statement after runChatTurn) so the goroutine
// exits even if runChatTurn panics — net/http recovers handler panics, so
// a non-deferred close would be skipped and the ticker would keep writing
// to a dead ResponseWriter forever.
defer close(keepDone)
a.runChatTurn(pctx, ctx, sessionID, req.Message, func(ev agentEvent) {
writeEvent(ev)
})
// B.6: if the turn ended with no text and no tool calls (the model
// empty-response'd and all retries failed), delete the placeholder row
// instead of persisting an empty bubble. The error event was already
// streamed to the frontend via the 'done with error=true' event, so the
// operator sees the error inline — an empty assistant bubble in the
// transcript adds nothing and looks like the agent is broken.
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
st.deleteMessage(pctx, msgID)
} else {
persist() // final state — same row, updated one last time with the concluding text
}
// Generate a meaningful title from the assistant's first answer
// instead of reusing the raw user message for every session.
// P2.9 (2026-07-20): prefer the goal as the title when one is set —
// the first assistant text is often a greeting or narrative that
// doesn't describe the task ("Hey! 👋 Nomos here, running on
// mac-mini:8092..."). The goal is the operator's actual intent.
// Sessions that never call set_goal (pure Q&A) fall back to the
// assistant text, which is still better than the raw user message.
if finalText != "" && sessionID != "ephemeral" {
var goalTitle string
if sess, gerr := st.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
goalTitle = truncate(sess.Goal, 120)
}
title := goalTitle
if title == "" {
title = truncate(finalText, 80)
}
if title != "" {
st.updateSessionTitle(pctx, sessionID, title)
}
}
}
func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
@@ -483,7 +599,8 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
if len(parts) == 2 && r.Method == http.MethodGet {
switch parts[1] {
case "plan":
steps, err := st.getPlanSteps(r.Context(), id)
all := r.URL.Query().Has("all") && r.URL.Query().Get("all") != "0" && r.URL.Query().Get("all") != "false"
steps, err := st.getPlanSteps(r.Context(), id, all)
if err != nil {
http.Error(w, err.Error(), 500)
return
@@ -701,7 +818,7 @@ func newMCPClient(baseURL, token string) (*mcpClient, error) {
c := &mcpClient{
baseURL: baseURL,
token: token,
http: &http.Client{Timeout: 30 * time.Second},
http: &http.Client{Timeout: 120 * time.Second},
}
resp, err := c.doRequest("initialize", map[string]any{

82
cmd/nomos/messagequeue.go Normal file
View File

@@ -0,0 +1,82 @@
package main
import (
"log/slog"
"sync"
)
// maxQueuedPerSession caps a session's queue. A held turn plus unbounded
// enqueues would grow memory without limit; an operator nudging a long
// autonomous turn realistically queues only a handful, so a generous cap is
// pure insurance. Overflow drops the newest enqueue and logs (the message is
// already persisted in the DB by handleChat before enqueue, so it isn't lost
// from the transcript — it just won't auto-run).
const maxQueuedPerSession = 20
// messageQueue holds operator messages that arrived while a turn was already
// running for a session. Plan 2026-08-03 (F2): instead of rejecting the
// operator's message with "Nomos is still finishing a previous step… send it
// again", the message is queued and auto-run when the in-flight turn releases
// the session's turn-gate permit.
//
// The queue only schedules WHEN a turn runs, not WHETHER the message is stored
// — handleChat persists the user message before acquiring the gate, so a queued
// message is already in the transcript; this just makes sure a turn eventually
// acts on it.
//
// Draining is strictly one-at-a-time under the turn gate (see drainQueued in
// main.go), so this cannot stack concurrent turns — the exact hazard the gate
// itself exists to prevent. Background resumeSession callers never touch this
// queue; they keep their non-blocking skip.
type messageQueue struct {
mu sync.Mutex
queue map[string][]string
}
func newMessageQueue() *messageQueue {
return &messageQueue{queue: map[string][]string{}}
}
// enqueue appends a message to the back of the session's FIFO. Returns false
// (and logs) if the session is already at maxQueuedPerSession — the caller's
// message is already persisted in the DB, so this only skips auto-running it.
func (q *messageQueue) enqueue(sessionID, msg string) bool {
q.mu.Lock()
defer q.mu.Unlock()
if len(q.queue[sessionID]) >= maxQueuedPerSession {
slog.Warn("nomos: message queue full; dropping auto-run for operator message", "session", sessionID, "cap", maxQueuedPerSession)
return false
}
q.queue[sessionID] = append(q.queue[sessionID], msg)
return true
}
// dequeue pops the next message from the front of the session's FIFO. Returns
// ok=false when empty.
func (q *messageQueue) dequeue(sessionID string) (string, bool) {
q.mu.Lock()
defer q.mu.Unlock()
xs := q.queue[sessionID]
if len(xs) == 0 {
return "", false
}
m := xs[0]
q.queue[sessionID] = xs[1:]
return m, true
}
// requeueFront pushes a message back to the front — used when a drainer popped
// a message but lost the race for the gate to a live turn; that turn's own
// release will drain it again.
func (q *messageQueue) requeueFront(sessionID, msg string) {
q.mu.Lock()
defer q.mu.Unlock()
q.queue[sessionID] = append([]string{msg}, q.queue[sessionID]...)
}
// peek reports the queued depth for a session (test/diagnostic helper).
func (q *messageQueue) peek(sessionID string) int {
q.mu.Lock()
defer q.mu.Unlock()
return len(q.queue[sessionID])
}

View File

@@ -0,0 +1,142 @@
package main
import (
"context"
"sync"
"testing"
"time"
)
func TestMessageQueue_FIFO(t *testing.T) {
q := newMessageQueue()
q.enqueue("s", "first")
q.enqueue("s", "second")
q.enqueue("s", "third")
want := []string{"first", "second", "third"}
for _, w := range want {
got, ok := q.dequeue("s")
if !ok || got != w {
t.Fatalf("dequeue = %q,%v want %q,true", got, ok, w)
}
}
if _, ok := q.dequeue("s"); ok {
t.Fatal("dequeue on drained queue should return ok=false")
}
}
func TestMessageQueue_RequeueFront(t *testing.T) {
q := newMessageQueue()
q.enqueue("s", "a")
q.enqueue("s", "b")
// Pop "a", then push it back to the front; "a" must come out before "b".
a, _ := q.dequeue("s")
q.requeueFront("s", a)
got, _ := q.dequeue("s")
if got != "a" {
t.Fatalf("after requeueFront, dequeue = %q want %q", got, "a")
}
got2, _ := q.dequeue("s")
if got2 != "b" {
t.Fatalf("next dequeue = %q want %q", got2, "b")
}
}
func TestMessageQueue_IsolatedPerSession(t *testing.T) {
q := newMessageQueue()
q.enqueue("s1", "one")
q.enqueue("s2", "two")
if got, _ := q.dequeue("s1"); got != "one" {
t.Fatalf("s1 = %q want one", got)
}
if got, _ := q.dequeue("s2"); got != "two" {
t.Fatalf("s2 = %q want two", got)
}
if q.peek("s1") != 0 || q.peek("s2") != 0 {
t.Fatal("both sessions should be drained")
}
}
func TestMessageQueue_Concurrent(t *testing.T) {
q := newMessageQueue()
const n = maxQueuedPerSession // stay under the cap so every enqueue lands
var wg sync.WaitGroup
for i := 0; i < n; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
q.enqueue("s", "m")
}(i)
}
wg.Wait()
if q.peek("s") != n {
t.Fatalf("peek = %d want %d (all enqueues must be counted)", q.peek("s"), n)
}
seen := 0
for {
if _, ok := q.dequeue("s"); !ok {
break
}
seen++
}
if seen != n {
t.Fatalf("drained %d want %d", seen, n)
}
}
func TestMessageQueue_CapsOverflow(t *testing.T) {
q := newMessageQueue()
for i := 0; i < maxQueuedPerSession; i++ {
if !q.enqueue("s", "m") {
t.Fatalf("enqueue #%d within cap should succeed", i)
}
}
if q.enqueue("s", "overflow") {
t.Fatal("enqueue past the cap should return false (dropped)")
}
if got := q.peek("s"); got != maxQueuedPerSession {
t.Fatalf("peek = %d want %d (overflow must not append)", got, maxQueuedPerSession)
}
}
// drainQueued on an empty queue must be a no-op: it returns immediately and
// never touches the gate (so the session stays free for the next turn).
func TestDrainQueued_NoOpOnEmpty(t *testing.T) {
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
a.drainQueued(context.Background(), "s")
if !a.gate.acquire("s", 0) {
t.Fatal("gate should be free after a no-op drain (drain must not hold it)")
}
a.gate.release("s")
}
// With a queued message but the gate held by another turn, drainQueued must
// re-queue the message and return WITHOUT running a turn (no store/provider → a
// real run would panic). This is the "never stack" property: a busy gate
// defers to the holder's own release-drain.
func TestDrainQueued_RequeuesWhenBusy(t *testing.T) {
prev := drainAcquireWait
drainAcquireWait = 10 * time.Millisecond
t.Cleanup(func() { drainAcquireWait = prev })
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
if !a.gate.acquire("s", 0) {
t.Fatal("precondition: hold the gate")
}
a.queue.enqueue("s", "queued-msg")
done := make(chan struct{})
go func() {
a.drainQueued(context.Background(), "s") // must not panic; must requeue
close(done)
}()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("drainQueued did not return promptly while the gate was busy")
}
if got := a.queue.peek("s"); got != 1 {
t.Fatalf("message should be re-queued while busy; peek = %d want 1", got)
}
a.gate.release("s")
}

View File

@@ -13,6 +13,7 @@ import (
"github.com/dtoro/oikos/internal/db/sqlcgen"
"github.com/dtoro/oikos/internal/observability"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
@@ -25,6 +26,13 @@ const maxToolResultSize = 4096
// The caller translates this into a directive tool result.
var errPlanInFlight = errors.New("plan already in flight")
// errPlanStepNotFound is returned by updatePlanStep when no step matches the
// given seq in the CURRENT (MAX) generation — either the seq is out of range,
// or (after a re-plan) the model addressed a stale 1-based number. seq is
// generation-relative, so this never resurrects a superseded generation's row.
// The caller translates it into a directive tool result (P0.1).
var errPlanStepNotFound = errors.New("plan step not found in current generation")
type store struct {
pool *pgxpool.Pool
}
@@ -799,10 +807,11 @@ func (s *store) setGoal(ctx context.Context, sessionID, goal string) error {
// Replace any prior plan steps (done/running/pending/...) as `replaced`.
// The rows are kept for the generation counter + audit trail; proposePlan
// excludes `replaced` from its in-flight check, so the next propose_plan
// takes the fresh-generation path.
// takes the fresh-generation path. replaced_reason records the cause
// (2026-08-04 plan-step integrity audit).
s.pool.Exec(ctx,
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
sessionID)
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
sessionID, "goal superseded")
if _, err := s.pool.Exec(ctx,
`UPDATE agent_sessions SET goal = $2, status = 'executing', title = $2, last_active_at = now() WHERE id = $1`,
sessionID, goal); err != nil {
@@ -842,6 +851,14 @@ func (s *store) reopenSession(ctx context.Context, sessionID string) bool {
s.pool.Exec(ctx,
`UPDATE agent_sessions SET status = 'executing', outcome = NULL, summary = NULL, last_active_at = now() WHERE id = $1`,
sessionID)
// Mark the prior plan's steps as replaced so the P1 plan-first gate in
// classifyAndGate forces a fresh propose_plan before any run. Without
// this, the agent could resume a session and call run against the old
// (completed) plan — exactly what caused the ZimaOS continuation to
// have 81 ad-hoc tool calls with zero plan structure (2026-08-04).
s.pool.Exec(ctx,
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
sessionID, "session reopened — awaiting new plan")
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.reopened", s.taskEntityPtr(ctx, sessionID),
"info", "nomos", sessionID, map[string]any{"prior_status": currentStatus})
return true
@@ -879,16 +896,15 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
}
defer tx.Rollback(ctx)
var startSeq int
var anyStarted bool
// `replaced` steps (from a prior plan generation superseded by a
// follow-up sub-task — see reopenSession) are excluded: they prove a
// prior plan was completed and superseded, not that a plan is in flight.
// Without this exclusion, reopenSession's `replaced` marking would be
// follow-up sub-task — see setGoal/reopenSession) are excluded: they
// prove a prior plan was completed and superseded, not that a plan is in
// flight. Without this exclusion, setGoal's `replaced` marking would be
// useless — propose_plan would still refuse on the follow-up.
if err := tx.QueryRow(ctx, `
SELECT COALESCE(max(seq), 0), COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&startSeq, &anyStarted); err != nil {
SELECT COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&anyStarted); err != nil {
return nil, err
}
if anyStarted {
@@ -898,35 +914,29 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
return nil, errPlanInFlight
}
// Fresh/revise: mark any prior PENDING steps as `replaced` (not DELETE).
// This preserves the rows for the generation counter (MAX(generation)+1
// below) and the plan_generations eval assertion. Without this, a first
// plan that was proposed but never executed (all pending) would be
// wiped, resetting the counter to 1 — making a follow-up's plan look
// like generation 1 instead of 2. `replaced` steps are excluded from
// the anyStarted check above, so they don't block the fresh proposal.
// The rows are kept for the generation counter (MAX(generation)+1 below)
// and the plan_generations eval assertion. `replaced` steps are excluded
// from the anyStarted check above, so they don't block this proposal.
// replaced_reason records the cause — required by the plan-step integrity
// gate (2026-08-04 session audit).
if _, err := tx.Exec(ctx,
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
sessionID); err != nil {
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
sessionID, "superseded by new plan generation"); err != nil {
return nil, err
}
// startSeq keeps the max(seq) from the query above: if prior steps
// exist (replaced or done), the new generation's steps start after them
// (no seq collisions across generations). If no rows exist (first plan),
// startSeq is 0 and the first step is seq 1.
// Resolve the generation number for this plan. Generation 1 is the
// initial plan; a genuine revise (which currently goes through the same
// fresh-start path above because all steps were pending) resets to 1
// since the DELETE wiped the prior rows. The column is wired here so a
// future explicit mid-flight revise path can increment it.
// nextGen: generation 1 for the first plan, MAX(generation)+1 for every
// revise/follow-up (prior rows were marked `replaced` above, not deleted,
// so the counter survives). seq is generation-relative — it resets to
// 1..N for this generation, so (session_id, generation, seq) is the
// addressing key and the model's 1-based update_plan_step always maps to
// the CURRENT plan after a re-plan (P0.1).
var nextGen int
if err := tx.QueryRow(ctx, `
SELECT COALESCE(MAX(generation), 0) + 1
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&nextGen); err != nil {
return nil, err
}
// After the DELETE above, no rows remain, so MAX(generation) is NULL →
// nextGen = 1. (Keep the query for the future revise path; it's cheap.)
out := make([]map[string]any, 0, len(steps))
for i, st := range steps {
@@ -934,7 +944,7 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
if st.TargetSlug != "" {
targetSlug = &st.TargetSlug
}
seq := startSeq + i + 1
seq := i + 1
var id uuid.UUID
if err := tx.QueryRow(ctx, `
INSERT INTO session_plan_steps (session_id, seq, title, detail, target_slug, generation)
@@ -973,10 +983,27 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
// be marked complete while an earlier step is still pending, preventing the
// agent from marking step 5 done before step 4 (observed in production: the
// agent rushed to close all steps in a final turn, in reverse order).
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID string) error {
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID, replacedReason string) error {
if s == nil || sessionID == "" || sessionID == "ephemeral" {
return nil
}
// Resolve the CURRENT generation: seq is generation-relative (1-based
// within the plan the model is working), so (session_id, MAX(generation),
// seq) is the addressing key. A re-plan's superseded generations have
// their own seq space and must never be touched by a follow-up's
// update_plan_step — that was the root cause of "the plan was off"
// (gen-1 `replaced` rows resurrected as `done` while gen-2 work went
// unrecorded). The MAX(generation) step is by construction the active
// plan, never `replaced`, so this can't resurrect a superseded row (P0.1).
var curGen int
if err := s.pool.QueryRow(ctx,
`SELECT COALESCE(MAX(generation), 0) FROM session_plan_steps WHERE session_id = $1`,
sessionID).Scan(&curGen); err != nil {
return err
}
if curGen == 0 {
return errPlanStepNotFound
}
stamp := ""
switch status {
case "running":
@@ -984,16 +1011,18 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
case "done", "failed", "skipped", "blocked", "replaced":
stamp = ", finished_at = now()"
}
// Completion ordering: for terminal states, check that no earlier step
// is still pending. Running steps can start out of order (the agent
// may dispatch parallel work), but completion must be sequential.
// Completion ordering, scoped to the CURRENT generation: for terminal
// states, no earlier step in THIS plan may still be pending. Running
// steps can start out of order (the agent may dispatch parallel work),
// but completion must be sequential. Earlier generations are superseded
// and irrelevant.
if status == "done" || status == "failed" || status == "skipped" || status == "blocked" {
var blockedBy int
if err := s.pool.QueryRow(ctx, `
SELECT COALESCE(MIN(seq), 0)
FROM session_plan_steps
WHERE session_id = $1 AND seq < $2 AND status = 'pending'`,
sessionID, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
WHERE session_id = $1 AND generation = $2 AND seq < $3 AND status = 'pending'`,
sessionID, curGen, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
return fmt.Errorf("cannot complete step %d — step %d is still pending", seq, blockedBy)
}
}
@@ -1004,12 +1033,33 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
var stepID uuid.UUID
var targetSlug *string
// stamp is a fixed literal from the switch above — never user input.
if err := s.pool.QueryRow(ctx, `
UPDATE session_plan_steps
SET status = $3, execution_id = COALESCE($4, execution_id)`+stamp+`
WHERE session_id = $1 AND seq = $2
RETURNING id, target_slug`, sessionID, seq, status, execPtr).Scan(&stepID, &targetSlug); err != nil {
return err
// status <> 'replaced' is defense-in-depth: MAX(generation) can't hold a
// replaced row, but if it ever could, this refuses the write instead of
// resurrecting it. No matching row → errPlanStepNotFound (stale/out-of-range seq).
if status == "replaced" && replacedReason != "" {
err := s.pool.QueryRow(ctx, `
UPDATE session_plan_steps
SET status = $4, execution_id = COALESCE($5, execution_id), replaced_reason = $6`+stamp+`
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr, replacedReason).Scan(&stepID, &targetSlug)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return errPlanStepNotFound
}
return err
}
} else {
err := s.pool.QueryRow(ctx, `
UPDATE session_plan_steps
SET status = $4, execution_id = COALESCE($5, execution_id)`+stamp+`
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr).Scan(&stepID, &targetSlug)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return errPlanStepNotFound
}
return err
}
}
// Anchor the event to the step's target entity when it has one, else the task.
entPtr := s.taskEntityPtr(ctx, sessionID)
@@ -1098,13 +1148,58 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
if outcome != "success" {
closeStatus = "skipped"
}
// Auto-close only the CURRENT generation's in-flight steps — superseded
// generations were already resolved when their plan was replaced. Stamp
// started_at so no `done` step is left with a NULL start time (P0.1 fix
// 5), and emit a plan.step.finished event per closed step so the panel
// converges instead of freezing on "running" after the task completes
// (P1.1: no bulk plan-step status write without a corresponding event).
type closingStep struct {
id uuid.UUID
seq int
targetSlug *string
}
var toClose []closingStep
if rows, qerr := s.pool.Query(ctx, `
SELECT id, seq, target_slug FROM session_plan_steps
WHERE session_id = $1
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
AND status IN ('pending', 'running')`, sessionID); qerr == nil {
for rows.Next() {
var cs closingStep
if err := rows.Scan(&cs.id, &cs.seq, &cs.targetSlug); err == nil {
toClose = append(toClose, cs)
}
}
rows.Close()
}
if _, err := s.pool.Exec(ctx, `
UPDATE session_plan_steps
SET status = $2, finished_at = COALESCE(finished_at, now())
WHERE session_id = $1 AND status IN ('pending', 'running')`,
SET status = $2,
started_at = COALESCE(started_at, now()),
finished_at = COALESCE(finished_at, now())
WHERE session_id = $1
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
AND status IN ('pending', 'running')`,
sessionID, closeStatus); err != nil {
slog.Warn("nomos: completeTask failed to auto-close in-flight steps", "session", sessionID, "error", err)
}
// Emit one plan.step.finished per closed step so the live panel advances
// (mirrors updatePlanStep's event). A bulk UPDATE that skips the event
// bus guarantees a stale panel — the rule is: no plan-step status change
// without a corresponding event.
taskEnt := s.taskEntityPtr(ctx, sessionID)
for _, cs := range toClose {
evEnt := taskEnt
if cs.targetSlug != nil && *cs.targetSlug != "" {
var tid uuid.UUID
if s.pool.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, *cs.targetSlug).Scan(&tid) == nil {
evEnt = &tid
}
}
_ = observability.Event(ctx, sqlcgen.New(s.pool), "plan.step.finished", evEnt, "info", "nomos", sessionID,
map[string]any{"step_id": cs.id.String(), "seq": cs.seq, "status": closeStatus})
}
// Clean up assent and destructive window keys from autonomy_settings.
s.pool.Exec(ctx, `DELETE FROM autonomy_settings
@@ -1148,9 +1243,141 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.status", entPtr, severity, "nomos", sessionID,
map[string]any{"status": status, "outcome": outcome, "summary": summary,
"cancelled_executions": cancelledCount, "blocker": blocker})
// Auto-persist knowledge so the graph learns from this session regardless
// of whether the agent remembered to call upsert_knowledge (2026-08-04
// session audit: only 2.4% of sessions called upsert_knowledge manually).
if outcome == "success" || outcome == "partial" {
autoUpsertKnowledge(ctx, s, sessionID, outcome, summary)
}
// Plan quality metric: compute step completion rate for the session's
// current plan generation. Tracked as a task attribute so the trend
// can be monitored over time (2026-08-04 session audit: 38% baseline).
writePlanCompletionRate(ctx, s, sessionID)
// Auto-feedback: create a feedback entry linking the session's outcome
// to its last execution, feeding the pattern-extraction pipeline that
// has been empty since launch (2026-08-04 session audit: 0 feedback rows).
if outcome == "success" || outcome == "partial" {
autoFeedback(ctx, s, sessionID, outcome, summary)
}
return nil
}
// autoUpsertKnowledge creates a knowledge entry for a completed session,
// capturing what was done and linking it to the entities involved. Called
// automatically from completeTask so every session leaves a trace, even if
// the agent forgot to call upsert_knowledge. Only fired for success/partial
// outcomes (failures don't have actionable discoveries).
func autoUpsertKnowledge(ctx context.Context, s *store, sessionID, outcome, summary string) {
var goal string
if err := s.pool.QueryRow(ctx,
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
sessionID).Scan(&goal); err != nil || goal == "" {
return
}
title := "Session " + sessionID[:8] + ": " + goal
if len(title) > 200 {
title = title[:200]
}
content := "## Outcome\n" + outcome + "\n\n## Summary\n" + summary
kind := "investigation"
slug := "investigation:nomos/" + sessionID
tags := []string{"nomos-session", "auto-generated"}
// Upsert the knowledge entity.
docID, _ := uuid.NewV7()
if err := s.pool.QueryRow(ctx, `
INSERT INTO entities (id, slug, type, name, attributes)
VALUES ($1, $2, $3, $4, '{}')
ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name, updated_at = now()
RETURNING id`, docID, slug, kind, title).Scan(&docID); err != nil {
slog.Warn("nomos: autoUpsertKnowledge entity insert", "session", sessionID, "error", err)
return
}
// Upsert the knowledge content.
if _, err := s.pool.Exec(ctx, `
INSERT INTO knowledge_entities (entity_id, title, content, source, tags, updated_at)
VALUES ($1, $2, $3, 'nomos-agent', $4, now())
ON CONFLICT (entity_id) DO UPDATE
SET title = EXCLUDED.title, content = EXCLUDED.content,
tags = EXCLUDED.tags, updated_at = now()`,
docID, title, content, tags); err != nil {
slog.Warn("nomos: autoUpsertKnowledge content insert", "session", sessionID, "error", err)
return
}
// Link to the task entity.
var taskEntID uuid.UUID
if s.pool.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`,
sessionID).Scan(&taskEntID) == nil && taskEntID != uuid.Nil {
s.pool.Exec(ctx, `
INSERT INTO relationships (source_id, target_id, type, attributes, valid_from)
SELECT $1, $2, 'involves', '{"by":"nomos","auto":true}'::jsonb, now()
WHERE NOT EXISTS (
SELECT 1 FROM relationships
WHERE source_id = $1 AND target_id = $2 AND type = 'involves' AND valid_to IS NULL)`,
taskEntID, docID)
}
slog.Info("nomos: auto-upserted knowledge for session",
"session", sessionID, "outcome", outcome, "slug", slug)
}
// writePlanCompletionRate computes the step completion rate for the current
// plan generation and writes it as a task entity attribute so the trend can
// be tracked. Baseline from 2026-08-04 audit: 38% (15/39 steps reached done).
func writePlanCompletionRate(ctx context.Context, s *store, sessionID string) {
var total, completed int
s.pool.QueryRow(ctx, `
SELECT COUNT(*), COALESCE(SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END), 0)
FROM session_plan_steps
WHERE session_id = $1
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
AND status <> 'replaced'`, sessionID).Scan(&total, &completed)
if total > 0 {
rate := float64(completed) / float64(total)
attrs, _ := json.Marshal(map[string]any{"plan_completion_rate": rate, "plan_steps_total": total, "plan_steps_completed": completed})
s.pool.Exec(ctx, `
UPDATE entities SET attributes = attributes || $2::jsonb, updated_at = now()
WHERE id = (SELECT entity_id FROM agent_sessions WHERE id = $1)`,
sessionID, string(attrs))
slog.Info("nomos: plan completion rate", "session", sessionID, "rate", fmt.Sprintf("%.0f%%", rate*100),
"completed", completed, "total", total)
}
}
// autoFeedback creates a feedback entry linking the session's outcome to its
// last execution, feeding the pattern-extraction pipeline that has been empty
// since launch. Only created for success/partial outcomes (failures don't
// have a specific execution to tie to).
func autoFeedback(ctx context.Context, s *store, sessionID, outcome, summary string) {
// Find the last execution linked to this session.
var execID uuid.UUID
if err := s.pool.QueryRow(ctx, `
SELECT pe.execution_id FROM nomos_plan_executions pe
WHERE pe.session_id = $1::uuid
ORDER BY pe.created_at DESC LIMIT 1`, sessionID).Scan(&execID); err != nil || execID == uuid.Nil {
return
}
fbID, _ := uuid.NewV7()
slug := "feedback:" + fbID.String()
if _, err := s.pool.Exec(ctx, `
INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'feedback', $3, '{}')`,
fbID, slug, "feedback for "+sessionID[:8]); err != nil {
slog.Warn("nomos: autoFeedback entity insert", "session", sessionID, "error", err)
return
}
_, err := s.pool.Exec(ctx, `
INSERT INTO feedback (entity_id, execution_id, outcome, observation, lesson, tags, created_at)
VALUES ($1, $2, $3, $4, $5, $6, now())`,
fbID, execID, outcome, summary, summary, []string{"nomos-session", "auto-generated", "session:" + sessionID[:8]})
if err != nil {
slog.Warn("nomos: autoFeedback insert", "session", sessionID, "error", err)
return
}
slog.Info("nomos: auto-feedback created for session", "session", sessionID, "outcome", outcome)
}
// blockerPatterns maps a substring (case-insensitive) to a structured blocker
// reason. Order matters — earlier patterns take precedence. These are the
// recurring failure signatures from the 2026-07-20 session audit. A
@@ -1240,6 +1467,53 @@ func (s *store) hadDiscovery(ctx context.Context, sessionID string) bool {
return count > 0
}
// sessionGoal returns the session's goal text, empty string if not found.
// Used by complete_task to check whether the goal involved a reachability
// verification before marking success.
func (s *store) sessionGoal(ctx context.Context, sessionID string) string {
if s == nil || sessionID == "" {
return ""
}
var goal string
s.pool.QueryRow(ctx,
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
sessionID).Scan(&goal)
return goal
}
// hadRecentVerification checks whether the session successfully verified
// reachability in recent turns — ping_service, or a run with curl/wget that
// returned successfully. Used by complete_task as a soft warning when the
// goal involved a reachability check but no recent verification occurred.
func (s *store) hadRecentVerification(ctx context.Context, sessionID string) bool {
if s == nil || sessionID == "" {
return true // fail safe: don't warn when we can't check
}
// Check for ping_service calls in the last 5 activity entries for this session.
var pingCount int
s.pool.QueryRow(ctx, `
SELECT COUNT(*) FROM (
SELECT 1 FROM agent_activity
WHERE session_id = $1 AND tool_name = 'ping_service' AND success = true
ORDER BY ts DESC LIMIT 5
) sub`, sessionID).Scan(&pingCount)
if pingCount > 0 {
return true
}
// Check for run calls with curl/wget that returned successfully.
var curlCount int
s.pool.QueryRow(ctx, `
SELECT COUNT(*) FROM (
SELECT 1 FROM agent_activity
WHERE session_id = $1
AND tool_name = 'run'
AND success = true
AND (input_summary LIKE '%curl%' OR input_summary LIKE '%wget%')
ORDER BY ts DESC LIMIT 10
) sub`, sessionID).Scan(&curlCount)
return curlCount > 0
}
// staleGoalSession is a goal-bearing task that's gone idle without reaching
// a terminal state — the idle-sweep worker's work list (fix 2+3 of
// plans/2026-07-11-task-completion-safety-net.md).
@@ -1347,15 +1621,23 @@ type planStep struct {
// getPlanSteps returns a task's plan in order — REST hydration for the context
// panel when it first opens a task (live events only carry deltas from then on).
func (s *store) getPlanSteps(ctx context.Context, sessionID string) ([]planStep, error) {
// By default only the CURRENT (MAX) generation is returned — the panel shows the
// live plan, not an archaeological record of every superseded generation. Pass
// all=true for the audit/eval view that needs every generation (the
// plan_generations assertion counts distinct generations across the full set).
func (s *store) getPlanSteps(ctx context.Context, sessionID string, all bool) ([]planStep, error) {
if s == nil {
return nil, nil
}
genFilter := ""
if !all {
genFilter = "AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)"
}
rows, err := s.pool.Query(ctx, `
SELECT id::text, seq, title, detail, status,
execution_id::text, target_slug,
started_at::text, finished_at::text, generation
FROM session_plan_steps WHERE session_id = $1 ORDER BY seq`, sessionID)
FROM session_plan_steps WHERE session_id = $1 `+genFilter+` ORDER BY generation, seq`, sessionID)
if err != nil {
return nil, err
}
@@ -1835,7 +2117,7 @@ func (s *store) resolveArgEntityID(ctx context.Context, args map[string]any) uui
// The (nullable) session_id column carries the conversation id. args is the
// tool call's own arguments, used to best-effort tag the row with the
// entity it acted on (see resolveArgEntityID).
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string) {
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string, tokenCount int) {
if s == nil || agentID == uuid.Nil {
return
}
@@ -1847,8 +2129,8 @@ func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, t
s.pool.Exec(ctx, `
INSERT INTO agent_activity
(agent_id, session_id, activity_type, tool_name, entity_id, input_summary, output_summary,
duration_ms, success, correlation_id)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
duration_ms, success, correlation_id, token_count)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
agentID, sessionID, "tool_call", toolName, entityIDArg, inputSummary, outputSummary,
durationMs, success, correlationID)
durationMs, success, correlationID, tokenCount)
}

View File

@@ -192,7 +192,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
}
// Mark step 1 as started.
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", ""); err != nil {
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
t.Fatalf("updatePlanStep: %v", err)
}
@@ -205,7 +205,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
}
// The original step 1 must be untouched — not erased, not appended to.
steps, err := s.getPlanSteps(ctx, sess.ID)
steps, err := s.getPlanSteps(ctx, sess.ID, false)
if err != nil {
t.Fatalf("getPlanSteps: %v", err)
}
@@ -217,7 +217,8 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
}
// Third call BEFORE anything runs on a fresh session: every step is
// still pending, so this must REPLACE, not refuse.
// still pending, so this must REPLACE (mark the prior plan `replaced`),
// not refuse. The new plan becomes generation 2.
sess2, err := s.createSession(ctx, "plan replace test")
if err != nil {
t.Fatalf("createSession: %v", err)
@@ -228,15 +229,146 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
if _, err := s.proposePlan(ctx, sess2.ID, []planStepInput{{Title: "Revised"}}); err != nil {
t.Fatalf("proposePlan (revise before execution): %v", err)
}
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID)
// Default (current generation) view: only the revised step.
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID, false)
if err != nil {
t.Fatalf("getPlanSteps: %v", err)
}
if len(revisedSteps) != 1 || revisedSteps[0].Title != "Revised" {
t.Fatalf("got %+v, want a single 'Revised' step (pre-execution revise must replace, not refuse)", revisedSteps)
t.Fatalf("got %+v, want a single 'Revised' step (current-generation view)", revisedSteps)
}
if revisedSteps[0].Generation != 1 {
t.Fatalf("revised step generation = %d, want 1 (fresh-start after DELETE resets generation)", revisedSteps[0].Generation)
if revisedSteps[0].Seq != 1 {
t.Fatalf("revised step seq = %d, want 1 (seq is generation-relative, resets to 1..N)", revisedSteps[0].Seq)
}
if revisedSteps[0].Generation != 2 {
t.Fatalf("revised step generation = %d, want 2 (prior pending plan is replaced, not deleted, so the counter increments)", revisedSteps[0].Generation)
}
// all=true audit view: both generations, the original marked `replaced`.
allSteps, err := s.getPlanSteps(ctx, sess2.ID, true)
if err != nil {
t.Fatalf("getPlanSteps(all): %v", err)
}
if len(allSteps) != 2 {
t.Fatalf("all=true got %d steps, want 2 (Original replaced gen1 + Revised gen2)", len(allSteps))
}
if allSteps[0].Title != "Original" || allSteps[0].Status != "replaced" || allSteps[0].Generation != 1 {
t.Errorf("gen1 step = %+v, want Original/replaced/gen1", allSteps[0])
}
if allSteps[1].Title != "Revised" || allSteps[1].Generation != 2 || allSteps[1].Seq != 1 {
t.Errorf("gen2 step = %+v, want Revised/gen2/seq1", allSteps[1])
}
}
// TestUpdatePlanStep_GenerationRelative is the P0.1 regression proof: after a
// re-plan, update_plan_step(seq=N) — using the 1-based number the model
// naturally carries — must address the CURRENT generation and never resurrect
// a superseded generation's `replaced` row. Before the fix, seq was globally
// increasing across generations, so seq=1 after a re-plan flipped the gen-1
// `replaced` step back to `running`/`done` while the real gen-2 work went
// unrecorded.
func TestUpdatePlanStep_GenerationRelative(t *testing.T) {
s := newTestStore(t)
ctx := context.Background()
sess, err := s.createSession(ctx, "gen-relative seq test")
if err != nil {
t.Fatalf("createSession: %v", err)
}
// Generation 1: two steps.
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
t.Fatalf("proposePlan #1: %v", err)
}
// Re-plan: setGoal marks the gen-1 plan `replaced`, proposePlan starts gen 2.
if err := s.setGoal(ctx, sess.ID, "follow-up sub-task"); err != nil {
t.Fatalf("setGoal: %v", err)
}
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "C"}, {Title: "D"}}); err != nil {
t.Fatalf("proposePlan #2: %v", err)
}
// The model addresses the new plan with 1-based seq. seq=1 must hit
// gen-2 "C", leaving gen-1 "A" (replaced) untouched.
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
t.Fatalf("updatePlanStep(seq=1, running): %v", err)
}
if err := s.updatePlanStep(ctx, sess.ID, 1, "done", "", ""); err != nil {
t.Fatalf("updatePlanStep(seq=1, done): %v", err)
}
all, err := s.getPlanSteps(ctx, sess.ID, true)
if err != nil {
t.Fatalf("getPlanSteps(all): %v", err)
}
byTitle := map[string]planStep{}
for _, st := range all {
byTitle[st.Title] = st
}
// gen-1 steps stay `replaced` — NOT resurrected to running/done.
if byTitle["A"].Status != "replaced" || byTitle["A"].Generation != 1 {
t.Errorf("A = %+v, want replaced/gen1 (a superseded row must never be touched)", byTitle["A"])
}
if byTitle["B"].Status != "replaced" || byTitle["B"].Generation != 1 {
t.Errorf("B = %+v, want replaced/gen1", byTitle["B"])
}
// gen-2 seq=1 advanced; seq=2 untouched.
if byTitle["C"].Status != "done" || byTitle["C"].Generation != 2 || byTitle["C"].Seq != 1 {
t.Errorf("C = %+v, want done/gen2/seq1 (the 1-based update must address the current generation)", byTitle["C"])
}
if byTitle["D"].Status != "pending" || byTitle["D"].Seq != 2 {
t.Errorf("D = %+v, want pending/seq2", byTitle["D"])
}
// Out-of-range seq must be refused (no current-gen step there).
if err := s.updatePlanStep(ctx, sess.ID, 99, "running", "", ""); !errors.Is(err, errPlanStepNotFound) {
t.Fatalf("updatePlanStep(seq=99) err = %v, want errPlanStepNotFound", err)
}
}
// TestCompleteTask_AutoCloseEmitsEvents is the P1.1 regression proof:
// completeTask's bulk auto-close of in-flight steps must emit one
// plan.step.finished event per closed step (so the live panel converges
// instead of freezing on "running" after the task completes) and must stamp
// started_at so no closed step is left un-timestamped (P0.1 fix 5).
func TestCompleteTask_AutoCloseEmitsEvents(t *testing.T) {
s := newTestStore(t)
ctx := context.Background()
sess, err := s.createSession(ctx, "auto-close events test")
if err != nil {
t.Fatalf("createSession: %v", err)
}
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
t.Fatalf("proposePlan: %v", err)
}
// A is running, B still pending at completion time.
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
t.Fatalf("updatePlanStep(1, running): %v", err)
}
if err := s.completeTask(ctx, sess.ID, "success", "done"); err != nil {
t.Fatalf("completeTask: %v", err)
}
// Every auto-closed step should now carry both a started_at and a
// finished_at (no NULL-started `done` step).
steps, err := s.getPlanSteps(ctx, sess.ID, true)
if err != nil {
t.Fatalf("getPlanSteps: %v", err)
}
for _, st := range steps {
if st.Status == "done" && st.StartedAt == nil {
t.Errorf("step %q done but started_at is NULL (P0.1 fix 5: stamp it)", st.Title)
}
}
// Exactly two plan.step.finished events — one per closed step (A and B).
var finished int
if err := s.pool.QueryRow(ctx,
`SELECT COUNT(*) FROM events WHERE type = 'plan.step.finished' AND correlation_id = $1`,
sess.ID).Scan(&finished); err != nil {
t.Fatalf("count events: %v", err)
}
if finished != 2 {
t.Fatalf("plan.step.finished events = %d, want 2 (one per auto-closed step)", finished)
}
}
@@ -265,7 +397,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
// A `run` call (discovery) — should set hadDiscovery, not hadEntityWriteback.
agentID := uuid.New()
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1")
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1", 0)
if !s.hadDiscovery(ctx, sess.ID) {
t.Fatal("hadDiscovery = false after a successful run call, want true")
}
@@ -278,7 +410,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
if err != nil {
t.Fatalf("createSession: %v", err)
}
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2")
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2", 0)
if s.hadDiscovery(ctx, sess2.ID) {
t.Fatal("hadDiscovery = true after a failed run call, want false (no facts learned)")
}
@@ -288,7 +420,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
if err != nil {
t.Fatalf("createSession: %v", err)
}
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3")
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3", 0)
if s.hadDiscovery(ctx, sess3.ID) {
t.Fatal("hadDiscovery = true after get_entity, want false (DB lookups are not discovery)")
}
@@ -298,12 +430,12 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
if err != nil {
t.Fatalf("createSession: %v", err)
}
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4")
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4", 0)
if !s.hadEntityWriteback(ctx, sess4.ID) {
t.Fatal("hadEntityWriteback = false after update_entity_attributes, want true")
}
// And the discovery+writeback combination (the conv3 scenario).
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5")
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5", 0)
if !s.hadDiscovery(ctx, sess4.ID) {
t.Fatal("hadDiscovery = false after run+writeback, want true")
}

View File

@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"log/slog"
"regexp"
"strings"
"time"
)
@@ -245,13 +246,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
// the seq-order enforcement (5.6) require it to be completed last,
// and D.1's complete_task gate enforces the actual calls. Together
// they close the loop structurally — neither relies on the agent
// reading SOUL.md.
// reading SOUL.md. The match is broadened past the literal tool
// names so a natural-language step ("Write back: update entity
// attributes…") isn't doubled by an auto-appended duplicate (P1.2).
hasWritebackStep := false
for _, st := range steps {
if strings.Contains(st.Title, "update_entity_attributes") ||
strings.Contains(st.Title, "create_relationship") ||
strings.Contains(st.Detail, "update_entity_attributes") ||
strings.Contains(st.Detail, "create_relationship") {
t := strings.ToLower(st.Title + " " + st.Detail)
if strings.Contains(t, "update_entity_attributes") ||
strings.Contains(t, "create_relationship") ||
strings.Contains(t, "upsert_knowledge") ||
strings.Contains(t, "write back") ||
strings.Contains(t, "writeback") {
hasWritebackStep = true
break
}
@@ -280,8 +285,19 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
// The writeback step is now always present (D.2 auto-appends it if
// the agent forgot), so the old advisory nudge is replaced by the
// structural gate: D.1 refuses complete_task without the actual
// update_entity_attributes/create_relationship calls.
result := fmt.Sprintf("Plan set (%d steps)%s. If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), appendedNote)
// update_entity_attributes/create_relationship calls. Enumerate the
// step seqs so the model knows exactly which numbers to address with
// update_plan_step (seq is 1-based within this plan — the addressing
// key, not a global counter).
var seqs strings.Builder
for i, p := range persisted {
if i > 0 {
seqs.WriteString("; ")
}
title := fmt.Sprint(p["title"])
fmt.Fprintf(&seqs, "%v=%s", p["seq"], title)
}
result := fmt.Sprintf("Plan set (%d steps): %s.%s Address them with update_plan_step(seq=N). If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), seqs.String(), appendedNote)
return result, true
case "update_plan_step":
@@ -291,7 +307,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
if seq <= 0 || status == "" {
return "error: update_plan_step needs seq (>=1) and status", true
}
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID); err != nil {
reason, _ := args["replaced_reason"].(string)
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID, reason); err != nil {
if errors.Is(err, errPlanStepNotFound) {
// The seq doesn't address a step in the CURRENT plan — most
// often a stale 1-based number the model carried across a
// re-plan, or an out-of-range seq. seq is generation-relative
// (1..N within the latest propose_plan), so a superseded
// generation's row is never touched (P0.1 fix 3). Direct the
// model instead of silently no-op'ing.
return fmt.Sprintf("Step %d is not in the current plan. seq is 1-based within your latest propose_plan (a re-plan resets it to 1..N, so an old step number no longer applies). The plan was not changed. Re-address with the correct 1-based seq, or if you've lost track, re-read the plan.", seq), true
}
return fmt.Sprintf("error updating step %d: %v", seq, err), true
}
return fmt.Sprintf("Step %d → %s. (Advance with update_plan_step + run; do not re-propose.)", seq, status), true
@@ -349,6 +375,18 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) && !a.store.hadEntityWriteback(ctx, sessionID) {
return "Refused: this session ran `run` against live targets (discovery) but did not call update_entity_attributes or create_relationship to persist what you learned. The knowledge graph will drift if you complete without writeback. Call update_entity_attributes for each entity you ran against (versions, states, counts, timestamps), and create_relationship for any edge you discovered, then call complete_task again. Outcome is held at 'executing' until you do.", true
}
// D.2: refuse success when the goal mentions a reachability/uptime
// check but no verification was done. The agent can't claim "X is
// reachable" based on a shell command alone — the proxy (Caddy) can
// return 200 for a terminal page (ttyd) or fallback while the actual
// dashboard is still down. Must call ping_service or run a successful
// curl before claiming success.
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) {
goal := a.store.sessionGoal(ctx, sessionID)
if mentionsReachability(goal) && !a.store.hadRecentVerification(ctx, sessionID) {
return "Refused: the goal involves a reachability or uptime check (\"make X reachable\", \"get X up\", etc.), but no ping_service call or successful curl/HTTP request against the target was detected. Caddy can return 200 for a terminal or fallback page while the actual service is still down — you must verify the service itself, not just the proxy. Call ping_service(target) or run a curl against the actual service URL, then call complete_task again. Outcome held until verified.", true
}
}
if err := a.store.completeTask(ctx, sessionID, outcome, summary); err != nil {
if errors.Is(err, errTaskAlreadyComplete) {
return "Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step (reconcile step states) or summarize the panel in your reply — do not re-execute the work.", true
@@ -365,6 +403,28 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
}
}
// reachabilityPatterns matches goal text that involves making something
// reachable/accessible/working. Used by complete_task to surface a soft
// warning when the session goal was about reachability but no verification
// occurred before marking success.
var reachabilityPatterns = []*regexp.Regexp{
regexp.MustCompile(`(?i)https?://[^\s]+`),
regexp.MustCompile(`(?i)\.hubris\.net\w+`),
regexp.MustCompile(`(?i)(un)?reachable`),
regexp.MustCompile(`(?i)(not?\s+)?(accessible|reachable|responding|resolving)`),
regexp.MustCompile(`(?i)diagnose\s+why`),
regexp.MustCompile(`(?i)(fix|restore|bring\s+back).*(accessible|reachable|online)`),
}
func mentionsReachability(goal string) bool {
for _, p := range reachabilityPatterns {
if p.MatchString(goal) {
return true
}
}
return false
}
// autoCompleteTrivialTask is the case-1 fix from
// plans/2026-07-11-task-completion-safety-net.md: a session that never
// called set_goal never framed itself as a structured task, so a turn that

90
cmd/nomos/turngate.go Normal file
View File

@@ -0,0 +1,90 @@
package main
import (
"sync"
"time"
)
// turnGate enforces at most one in-flight agent turn per session.
//
// Why this exists (plan 2026-08-03, F1): handleChat runs a turn in the HTTP
// request goroutine, and every "resume" path (the empty-message reconnect,
// the auto-continuation worker, the idle sweep, answer-question, the /resume
// endpoint) launches ANOTHER goroutine running a full turn. Nothing prevented
// two turns for the SAME session at once, so a network blip that triggered a
// reconnect would spawn a duplicate resumeSession while the original turn was
// still alive — their tool calls interleaved on the wire and in the persisted
// transcript, which is the root cause behind the "parallel/nesting/sequence
// is off" and "task didn't end / flaky" reports.
//
// Model: one permit (buffered-1 channel seeded with a single token) per
// session id. Acquiring consumes the token; releasing puts it back.
// - Background/best-effort callers (resumeSession and everything it backs)
// use a non-blocking acquire and SKIP when busy — a duplicate nudge while a
// turn is already running adds nothing, and the continuation/idle tickers
// will retry on their own.
// - The live chat path (an operator message) waits briefly for a finishing
// background turn, then bails with an actionable error if still busy — see
// handleChat.
//
// The permits map grows one entry per session id seen. For this single-agent
// homelab process that set is small and bounded by real sessions; cleanup is
// intentionally omitted (a sweep would race with acquire/release and the
// memory is negligible).
type turnGate struct {
mu sync.Mutex
permits map[string]chan struct{}
}
func newTurnGate() *turnGate {
return &turnGate{permits: make(map[string]chan struct{})}
}
// permit returns the single token-channel for sessionID, creating and seeding
// it on first use. Creation is guarded so two concurrent first-callers for the
// same id share one channel.
func (g *turnGate) permit(sessionID string) chan struct{} {
g.mu.Lock()
defer g.mu.Unlock()
ch, ok := g.permits[sessionID]
if !ok {
ch = make(chan struct{}, 1)
ch <- struct{}{}
g.permits[sessionID] = ch
}
return ch
}
// acquire takes the session's permit. With wait <= 0 it is non-blocking
// (returns false immediately if a turn is active). With wait > 0 it blocks up
// to wait for the permit, returning false on timeout. Every true return MUST
// be paired with exactly one release.
func (g *turnGate) acquire(sessionID string, wait time.Duration) bool {
ch := g.permit(sessionID)
if wait <= 0 {
select {
case <-ch:
return true
default:
return false
}
}
t := time.NewTimer(wait)
defer t.Stop()
select {
case <-ch:
return true
case <-t.C:
return false
}
}
// release returns the session's permit. Idempotent: a release with no matching
// acquire (or a double release) is a no-op rather than a blocking send.
func (g *turnGate) release(sessionID string) {
ch := g.permit(sessionID)
select {
case ch <- struct{}{}:
default:
}
}

114
cmd/nomos/turngate_test.go Normal file
View File

@@ -0,0 +1,114 @@
package main
import (
"sync"
"sync/atomic"
"testing"
"time"
)
func TestTurnGate_NonBlockingSkipsWhenBusy(t *testing.T) {
g := newTurnGate()
if !g.acquire("s1", 0) {
t.Fatal("first non-blocking acquire should succeed on a free session")
}
// A second non-blocking acquire (a background resume) must skip, not queue.
if g.acquire("s1", 0) {
t.Fatal("second non-blocking acquire should fail while a turn is active")
}
// A different session is independent.
if !g.acquire("s2", 0) {
t.Fatal("acquire on a different session should succeed")
}
g.release("s2")
g.release("s1")
// After release, the session is free again.
if !g.acquire("s1", 0) {
t.Fatal("acquire should succeed again after release")
}
g.release("s1")
}
func TestTurnGate_BlockingAcquireWaitsForRelease(t *testing.T) {
g := newTurnGate()
if !g.acquire("s1", 0) {
t.Fatal("first acquire should succeed")
}
got := make(chan bool, 1)
go func() { got <- g.acquire("s1", 2*time.Second) }()
select {
case <-got:
t.Fatal("blocking acquire should wait, not return before release")
case <-time.After(50 * time.Millisecond):
// expected: still waiting
}
g.release("s1")
select {
case ok := <-got:
if !ok {
t.Fatal("blocking acquire should succeed after release")
}
case <-time.After(time.Second):
t.Fatal("blocking acquire did not return after release")
}
g.release("s1")
}
func TestTurnGate_BlockingAcquireTimesOut(t *testing.T) {
g := newTurnGate()
g.acquire("s1", 0) // hold the permit
start := time.Now()
if g.acquire("s1", 60*time.Millisecond) {
t.Fatal("acquire should time out while permit is held")
}
if elapsed := time.Since(start); elapsed < 50*time.Millisecond {
t.Fatalf("acquire returned too fast (%v); expected to wait ~60ms", elapsed)
}
g.release("s1")
}
// TestTurnGate_SingleFlightConcurrent is the core F1 guarantee: many concurrent
// background acquirers on the SAME session, exactly one runs at a time. This is
// the property that prevents two turns interleaving tool calls.
func TestTurnGate_SingleFlightConcurrent(t *testing.T) {
g := newTurnGate()
const n = 50
var inFlight, maxInFlight int64
var runs int64
var wg sync.WaitGroup
wg.Add(n)
start := make(chan struct{})
for i := 0; i < n; i++ {
go func() {
defer wg.Done()
<-start
if !g.acquire("shared", 0) { // background-style: skip if busy
return
}
defer g.release("shared")
cur := atomic.AddInt64(&inFlight, 1)
for {
m := atomic.LoadInt64(&maxInFlight)
if cur <= m || atomic.CompareAndSwapInt64(&maxInFlight, m, cur) {
break
}
}
atomic.AddInt64(&runs, 1)
time.Sleep(2 * time.Millisecond)
atomic.AddInt64(&inFlight, -1)
}()
}
close(start)
wg.Wait()
if maxInFlight != 1 {
t.Fatalf("max in-flight turns = %d, want 1 (turns must not overlap)", maxInFlight)
}
if runs == 0 {
t.Fatal("expected at least one turn to run")
}
}

View File

@@ -1,5 +1,27 @@
:80 {
root * /srv
file_server
try_files {path} /index.html
# /wails/runtime.js is injected by the Wails desktop wrapper, which serves
# the same dist/ from its own asset handler. In a browser it does not
# exist, and the SPA fallback below answered it with index.html — so the
# browser parsed "<!doctype html>" as JavaScript and threw
# "SyntaxError: expected expression, got '<'" on every page load.
# Return a real 404 instead: the tag fails quietly, and the desktop app is
# unaffected because it never reaches this server.
handle /wails/* {
error 404
}
# Same reasoning for any other asset: a missing .js/.css/.map answered with
# HTML is always a confusing parse error rather than an honest 404. Only
# real routes should fall through to the SPA.
@asset path_regexp \.(js|mjs|css|map|json|png|jpg|svg|ico|woff2?)$
handle @asset {
file_server
}
handle {
file_server
try_files {path} /index.html
}
}

View File

@@ -8,7 +8,8 @@ FROM node:22-alpine AS builder
WORKDIR /build/web
COPY web/package.json web/package-lock.json ./
RUN npm ci
COPY web/vendor /build/vendor
RUN npm install --no-audit --no-fund
COPY VERSION ./
COPY web/ ./
RUN npm run build

View File

@@ -83,6 +83,18 @@ services:
command: ["api"]
stop_signal: SIGTERM
stop_grace_period: 30s
# Exists so nomos can wait for the API to actually answer rather than just
# for its container to exist — see nomos's depends_on below. wget is
# BusyBox's, already in the alpine runtime image, so this adds no
# dependency. /healthz pings the DB, so "healthy" means genuinely ready.
healthcheck:
test: ["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:8090/healthz"]
interval: 5s
timeout: 3s
retries: 10
# Migrations and seed run before this container, but the first bind can
# still take a moment; failures inside the start period don't count.
start_period: 10s
# Scheduler (Phase 3) — observe loop
scheduler:
@@ -139,7 +151,12 @@ services:
profiles: ["full"]
depends_on:
api:
condition: service_started
# service_started only waits for the container to exist, so nomos came
# up while the API was still binding :8090, failed its MCP initialize,
# exited 1, and crash-looped for ~25s on every single deploy. It always
# recovered, which is exactly why it went unnoticed. service_healthy
# waits for the API to actually answer.
condition: service_healthy
environment:
NOMOS_MCP_URL: http://api:8090/mcp
NOMOS_AGENT_SLUG: agent:nomos

View File

@@ -234,9 +234,33 @@ sequenceDiagram
---
**2026-07-08 — renamed to Nomos.** The Hermes agent gateway was renamed to
**2026-07-08 — renamed to Nomos.**
Nomos (from *oikonomos*, the steward of the oikos) under the
[Nomos resident agent plan](../../plans/2026-07-08-nomos-resident-agent.md),
### Hermes MCP client setup
To connect a Hermes Agent instance to oikos as a native MCP client, add to
`~/.hermes/config.yaml`:
```yaml
mcp_servers:
oikos:
url: "https://mcp.hubris.network/mcp"
headers:
Authorization: "Bearer <OIKOS_MCP_BEARER_TOKEN>"
timeout: 180
```
Run `/reload-mcp` in-session or restart Hermes. Tools appear as
`mcp__oikos__*`.
**Caveat:** Hermes stores the bearer token in plaintext in `config.yaml`
it does not support `${VAR}` interpolation in MCP server headers. Ensure
`security.redact_secrets: true` (default) so the token value is stripped
from tool output and logs. File an upstream feature request at
https://github.com/NousResearch/hermes-agent/issues for env-var
interpolation support.
N0 milestone. The gateway binary (`cmd/nomos`), Docker service, DB slug
(`agent:nomos`), and all referencing docs were updated. All architectural
principles in this ADR remain unchanged.

133
internal/audit/audit.go Normal file
View File

@@ -0,0 +1,133 @@
// Package audit produces read-only drift reports over the knowledge graph and
// monitoring state. It is the shared engine behind the
// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool.
//
// It surfaces the structural gaps an operator otherwise discovers only by
// accident: orphan check entities, checks targeting retired entities, probes
// stuck down/unknown, unmonitored declared types, and live edges pointing at
// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a
// follow-up that needs host-hop execution; these categories are pure DB
// queries, so the report is cheap, safe to run unattended, and testable.
package audit
import (
"context"
"github.com/dtoro/oikos/internal/db"
)
// Finding is one drift item the operator should look at.
type Finding struct {
Category string `json:"category"`
Severity string `json:"severity"` // info | warning | critical
Count int `json:"count"`
Entities []string `json:"entities"`
Evidence string `json:"evidence"`
SuggestedRunbook string `json:"suggested_runbook"`
}
// Summary tallies findings by category.
type Summary struct {
TotalFindings int `json:"total_findings"`
ByCategory map[string]int `json:"by_category"`
}
// Report runs every drift check and returns the findings plus a summary.
func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) {
specs := []struct {
finding Finding
query string
}{
{
Finding{Category: "orphan_checks", Severity: "warning",
Evidence: "check entities with truncated/random slugs (legacy shortSlug bug), no live target",
SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"},
`SELECT e.slug FROM entities e
WHERE e.type = 'check'
AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`,
},
{
Finding{Category: "dead_checks", Severity: "warning",
Evidence: "enabled check_defs whose target entity is deprecated/destroyed",
SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"},
`SELECT e.slug FROM check_defs cd
JOIN entities e ON e.id = cd.entity_id
JOIN entities tgt ON tgt.id = cd.target_id
WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`,
},
{
Finding{Category: "down_checks", Severity: "critical",
Evidence: "enabled checks reporting health=down",
SuggestedRunbook: "service-health-check"},
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
WHERE cd.enabled AND cd.last_health = 'down'`,
},
{
Finding{Category: "unknown_checks", Severity: "warning",
Evidence: "enabled checks that ran but reported health=unknown (likely misconfigured probe)",
SuggestedRunbook: "knowledge-graph-audit"},
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
WHERE cd.enabled AND cd.last_health = 'unknown'`,
},
{
Finding{Category: "unmonitored", Severity: "warning",
Evidence: "active entities whose type declares monitoring but have no enabled check_def",
SuggestedRunbook: "knowledge-graph-audit"},
`SELECT DISTINCT e.slug FROM signals sg
JOIN entities e ON e.id = sg.target_entity_id
WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`,
},
{
Finding{Category: "dangling_edges", Severity: "warning",
Evidence: "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets",
SuggestedRunbook: "lifecycle-destroy-node"},
`SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r
JOIN entities src ON src.id = r.source_id
JOIN entities tgt ON tgt.id = r.target_id
WHERE r.valid_to IS NULL
AND src.state NOT IN ('destroyed','deprecated')
AND tgt.state IN ('destroyed','deprecated')`,
},
{
Finding{Category: "polluted_attrs", Severity: "warning",
Evidence: "routing-critical attributes carrying prose (breaks resolution) — e.g. host='hubris (confirmed via pct…')",
SuggestedRunbook: "knowledge-graph-audit"},
`SELECT slug || ': host=' || (attributes->>'host') FROM entities
WHERE attributes->>'host' IS NOT NULL
AND (attributes->>'host') ~ '[ (]'`,
},
}
findings := make([]Finding, 0, len(specs))
summary := Summary{ByCategory: map[string]int{}}
for _, sp := range specs {
f := runFinding(ctx, pool, sp.finding, sp.query)
findings = append(findings, f)
summary.TotalFindings += f.Count
summary.ByCategory[f.Category] = f.Count
}
return findings, summary
}
const entityCap = 50
// runFinding runs a single-column slug query and folds the rows into a Finding.
func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding {
rows, err := pool.Query(ctx, query)
if err != nil {
f.Evidence = f.Evidence + " (query error: " + err.Error() + ")"
return f
}
defer rows.Close()
for rows.Next() {
var slug string
if err := rows.Scan(&slug); err != nil {
continue
}
f.Count++
if len(f.Entities) < entityCap {
f.Entities = append(f.Entities, slug)
}
}
return f
}

View File

@@ -0,0 +1,69 @@
package audit
import (
"context"
"testing"
"github.com/dtoro/oikos/internal/db"
"github.com/google/uuid"
)
// Integration tests against a real Postgres, guarded by
// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler).
func newAuditPool(t *testing.T) *db.Pool {
t.Helper()
base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
if base == "" {
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
}
return createTestDB(t, base)
}
func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) {
pool := newAuditPool(t)
ctx := context.Background()
// An orphan check entity (truncated random slug, the legacy bug shape).
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New())
// An active entity + a check_def on it stuck down.
target := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target)
checkE := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE)
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health)
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target)
// A deprecated entity still carrying an enabled check (dead_checks).
dep := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep)
depCheck := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck)
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep)
findings, summary := Report(ctx, pool)
byCat := map[string]int{}
for _, f := range findings {
byCat[f.Category] = f.Count
}
if byCat["orphan_checks"] < 1 {
t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"])
}
if byCat["down_checks"] < 1 {
t.Errorf("down_checks = %d, want >=1", byCat["down_checks"])
}
if byCat["dead_checks"] < 1 {
t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"])
}
if summary.TotalFindings < 3 {
t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings)
}
}

View File

@@ -0,0 +1,67 @@
package audit
import (
"context"
"fmt"
"math/rand"
"os"
"strings"
"testing"
"github.com/dtoro/oikos/internal/db"
"github.com/jackc/pgx/v5"
)
// createTestDB provisions a throwaway migrated database, same convention as
// internal/scheduler/coverage_test.go.
func createTestDB(t *testing.T, baseURL string) *db.Pool {
t.Helper()
ctx := context.Background()
admin, err := pgx.Connect(ctx, baseURL)
if err != nil {
t.Fatalf("connect admin: %v", err)
}
dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63())
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
admin.Close(ctx)
t.Fatalf("create test db: %v", err)
}
admin.Close(ctx)
at := strings.LastIndex(baseURL, "/")
testURL := baseURL[:at+1] + dbName
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
testURL += baseURL[at+q:]
}
pool, err := db.New(ctx, testURL)
if err != nil {
t.Fatalf("connect test db: %v", err)
}
if err := pool.Migrate(ctx); err != nil {
t.Fatalf("migrate: %v", err)
}
t.Cleanup(func() {
pool.Close()
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
admin.Close(ctx)
}
})
return pool
}
func getenvOrDefault(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
t.Helper()
if _, err := pool.Exec(ctx, q, args...); err != nil {
t.Fatalf("exec %s: %v", q, err)
}
}

View File

@@ -24,13 +24,16 @@ import (
// check_defs.kind values — one semantic kind can expand to several concrete
// checks (`resource` becomes four ssh-script rows).
const (
KindPing = "ping"
KindResource = "resource"
KindUpdates = "updates"
KindProcess = "process"
KindHTTP = "http"
KindCapacity = "capacity"
KindBackup = "backup-freshness"
KindPing = "ping"
KindResource = "resource"
KindUpdates = "updates"
KindProcess = "process"
KindHTTP = "http"
KindCapacity = "capacity"
KindBackup = "backup-freshness"
KindCertExpiry = "cert-expiry"
KindVMStatus = "vm-status"
KindDNS = "dns"
)
// defaultBackupMaxAge is how long a backup target may go without a new
@@ -108,6 +111,18 @@ func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (
attrs = map[string]any{}
}
// Per-entity override: an explicit `monitoring` attribute wins over the
// type declaration. A single entity can opt out (monitoring: none) or pick
// different kinds without introducing a new type — e.g. service:haos opts
// out because its VM is already covered by a vm-status check and the
// service can't be SSH-probed (haos blocks SSH).
if mo, ok := attrs["monitoring"]; ok {
mon = resolveMonitoringAttr(mo, mon)
if mon.None() {
return res, nil
}
}
// A service has no address of its own — it lives on the container that
// provides it. Fall back to the graph before giving up.
host := resolveHost(attrs)
@@ -146,6 +161,27 @@ func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (
return res, nil
}
// resolveMonitoringAttr turns an entity's `monitoring` attribute into a
// MonitoringResolution that overrides the type's declaration. Accepts the
// scalar "none" (or empty) to opt out, or a list of kind strings to override.
func resolveMonitoringAttr(v any, fallback ontology.MonitoringResolution) ontology.MonitoringResolution {
switch vv := v.(type) {
case string:
if vv == "none" || vv == "" {
return ontology.MonitoringResolution{Declared: true, Source: "attribute"}
}
case []any:
kinds := make([]string, 0, len(vv))
for _, k := range vv {
if s, ok := k.(string); ok && s != "" {
kinds = append(kinds, s)
}
}
return ontology.MonitoringResolution{Declared: true, Kinds: kinds, Source: "attribute"}
}
return fallback
}
// buildKind turns one declared semantic kind into concrete check_defs, or
// returns the reason it could not.
func buildKind(kind string, t Target, attrs map[string]any, host, user string, port int) ([]checkDef, string) {
@@ -201,12 +237,36 @@ func buildKind(kind string, t Target, attrs map[string]any, host, user string, p
if host == "" {
return nil, "no address on the entity or its host"
}
if t.Name == "" {
// A service's name is a logical label, not usually its systemd unit
// or container name (matrix = matrix-synapse.service + containers).
// Prefer an explicit probe target when declared; process_check.sh also
// matches a unit prefix or a docker container as a fallback.
unit := ""
for _, key := range []string{"probe_unit", "systemd_unit", "container"} {
if v, _ := attrs[key].(string); v != "" {
unit = v
break
}
}
// Ontology intent: "http when it has a url, else a process check." A
// url-fronted service is already liveness-probed via http (the real
// endpoint, through the TLS terminator); the process check is redundant
// and fragile (needs host access + the exact unit/container name), and
// under worst-of aggregation it lets a broken supplementary probe veto
// a working service. Emit it only for services WITHOUT a url, or when
// an explicit probe_unit opts into binary-level depth.
if unit == "" {
if httpURL(t, attrs) != "" {
return nil, "url present and no probe_unit; http check covers liveness"
}
unit = t.Name
}
if unit == "" {
return nil, "no name to check a process for"
}
// process_check.sh takes the unit name as $1 and reports "unknown"
// without it.
return []checkDef{ssh("process_check.sh", t.Name)}, ""
// process_check.sh takes the unit/container name as $1 and reports
// "unknown" without it.
return []checkDef{ssh("process_check.sh", unit)}, ""
case KindBackup:
// A backup target is checked from the machine that writes to it, so it
@@ -246,11 +306,79 @@ func buildKind(kind string, t Target, attrs map[string]any, host, user string, p
config: map[string]any{"url": url, "max_status": 500},
interval: 60,
}}, ""
case KindDNS:
// Resolve the entity's name via DNS to verify the zone is reachable.
// Uses the entity name (zone apex) or falls back to the slug.
name := t.Name
if name == "" {
name = strings.TrimPrefix(t.Slug, "zone:")
}
if name == "" {
return nil, "no name to resolve"
}
return []checkDef{{
kind: "dns",
config: map[string]any{"name": name},
interval: 300, // 5 min — DNS changes are rare; the cost of a miss
// is a stale IP, not a service outage.
}}, ""
case KindCertExpiry:
// The host whose cert to read (SNI / cert CN). Prefer an explicit
// `hostname` attribute, then `cn`, then a dotted name. Hourly: expiry
// changes once a day, but a renewal or mis-issued cert is worth
// noticing within the hour.
host := certHost(t, attrs)
if host == "" {
return nil, "no hostname / cn / dotted name to dial for the cert"
}
// `dial` is the TLS terminator's address to connect to (Caddy's lab
// IP), used when the hostname doesn't resolve/reach from the scheduler.
// Without it the probe can't reach *.hubris.network from a container
// with no mesh / split-horizon DNS.
dial, _ := attrs["dial"].(string)
config := map[string]any{"host": host, "warn_days": 30, "crit_days": 7}
if dial != "" {
config["dial"] = dial
}
return []checkDef{{
kind: "cert-expiry",
config: config,
interval: 3600,
}}, ""
case KindVMStatus:
// "Is the VM powered on" via `qm status` on its Proxmox host — the
// right reachability probe for a VM, since many block ICMP and lack a
// guest agent. checkVMStatus re-reads pve_id + host at runtime.
if _, ok := attrs["pve_id"]; !ok {
return nil, "no pve_id to run qm status"
}
return []checkDef{{
kind: "vm-status",
config: map[string]any{},
interval: 60,
}}, ""
}
return nil, "no builder for this kind yet"
}
// certHost works out the hostname to TLS-dial for a certificate's expiry.
func certHost(t Target, attrs map[string]any) string {
for _, key := range []string{"hostname", "cn", "san"} {
if v, ok := attrs[key].(string); ok && v != "" {
return v
}
}
// A dotted name is a hostname (hubris.network, media.hubris.network).
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
return t.Name
}
return ""
}
// writeCheck upserts one check_def and its backing check entity.
//
// The entity upsert MUST return the row's id. The previous version generated
@@ -299,14 +427,15 @@ func writeCheck(ctx context.Context, tx pgx.Tx, t Target, idx int, def checkDef)
// absent from the DO UPDATE below — a re-seed must not reset the schedule
// and re-herd everything.
tag, err := tx.Exec(ctx,
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled, last_run_at)
VALUES ($1, $2, $3, $4, $5, 30, true,
`INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
VALUES ($1, $2, $6, $3, $4, $5, 30, true,
now() - make_interval(secs => random() * $5::int))
ON CONFLICT (entity_id) DO UPDATE
SET target_id = EXCLUDED.target_id, kind = EXCLUDED.kind,
SET target_id = EXCLUDED.target_id, target_type = EXCLUDED.target_type,
kind = EXCLUDED.kind,
config = EXCLUDED.config, interval_s = EXCLUDED.interval_s,
updated_at = now()`,
checkID, t.ID, def.kind, configJSON, def.interval)
checkID, t.ID, def.kind, configJSON, def.interval, t.Type)
if err != nil {
return false, fmt.Errorf("upsert check_def %s: %w", checkSlug, err)
}
@@ -380,6 +509,12 @@ func resolveHost(attrs map[string]any) string {
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
return ip
}
// public_ipv4 before mesh: the scheduler container has no mesh interface,
// so a standalone-server reachable only by mesh IP (netbird-vps) is
// unprobeable even though a public IPv4 is available.
if ip, ok := attrs["public_ipv4"].(string); ok && ip != "" {
return ip
}
if mesh, ok := attrs["mesh"].(map[string]any); ok {
if nb, ok := mesh["netbird"].(map[string]any); ok {
if ip, ok := nb["ip"].(string); ok && ip != "" {
@@ -409,6 +544,13 @@ func resolveSSHUser(attrs map[string]any) string {
return u
}
}
// Workstations carry their login as a top-level `user` attribute
// (mac-mini: user: dtoro) rather than under ssh.user. Take it only when
// no explicit ssh.user was set, so a host that genuinely wants root still
// gets root.
if u, ok := attrs["user"].(string); ok && u != "" {
return u
}
return "root"
}

34
internal/db/checks.go Normal file
View File

@@ -0,0 +1,34 @@
package db
import (
"context"
"github.com/dtoro/oikos/internal/checkdefaults"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
// EnsureEntityChecks derives an entity's default check_defs from the
// monitoring spec of its type (resolving per-entity `monitoring` overrides).
//
// This is the single shared hook that keeps the check graph in sync with
// entity mutations. Both the HTTP create/patch handlers and the MCP
// entity-mutation tools (create_entity, update_entity_attributes) call it so
// that flipping an entity's `monitoring` attribute regenerates checks
// regardless of which surface made the change — previously only the HTTP
// path ran check derivation, so entities mutated via MCP silently produced no
// checks (see plans/2026-08-03-session-review-haos-monitoring-capability-gaps.md, A2).
func EnsureEntityChecks(ctx context.Context, tx pgx.Tx, id uuid.UUID, slug, entityType, name string, attrs []byte) (checkdefaults.Result, error) {
tree, err := LoadTypeTree(ctx, tx)
if err != nil {
return checkdefaults.Result{}, err
}
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
ID: id, Slug: slug, Type: entityType, Name: name, Attrs: attrs,
})
if err != nil {
return res, err
}
checkdefaults.LogResult(slug, entityType, res)
return res, nil
}

View File

@@ -306,7 +306,17 @@ func TestBlastRadiusTerminatesOnCycles(t *testing.T) {
pool := newTestPool(t)
seedAll(t, pool, seedsDir())
// Build a dependency cycle: gitea → caddy → authentik → gitea
// Build a dependency cycle: gitea → caddy → authentik → gitea.
//
// `depends-on` is declared blast_direction: backward — "A depends-on B"
// means B failing breaks A — so the blast radius of gitea walks the edges
// BACKWARDS: whoever depends on gitea is affected first. That is authentik
// (1 hop), then caddy which depends on authentik (2 hops).
//
// This test previously asserted caddy=1, authentik=2, which is the same
// cycle walked the wrong way round: blast_radius used to follow every edge
// source→target regardless of what the edge means, so it answered "what
// does gitea depend on" while being named for the opposite question.
cycle := []byte(`
version: 1
relationships:
@@ -342,14 +352,24 @@ relationships:
}
got[slug] = depth
}
want := map[string]int{"service:gitea": 0, "service:caddy": 1, "service:authentik": 2}
want := map[string]int{"service:gitea": 0, "service:authentik": 1, "service:caddy": 2}
for slug, depth := range want {
if got[slug] != depth {
t.Errorf("blast_radius[%s] = %d, want %d (full: %v)", slug, got[slug], depth, got)
}
}
if len(got) != len(want) {
t.Errorf("blast_radius returned %d nodes, want %d: %v", len(got), len(want), got)
// Deliberately not an exact node count. Walking the right way round also
// surfaces the real seed's own dependents of gitea (homelab-mcp and what
// depends on it), which are correct answers — the old exact-count
// assertion only held because the forward walk found nothing real.
// What matters here is that the cycle terminates rather than recursing.
if len(got) > 20 {
t.Errorf("blast_radius did not terminate sensibly: %d nodes: %v", len(got), got)
}
for slug, depth := range got {
if depth > 5 {
t.Errorf("blast_radius[%s] = %d, beyond the max_depth bound", slug, depth)
}
}
}

146
internal/db/lifecycle.go Normal file
View File

@@ -0,0 +1,146 @@
package db
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"github.com/dtoro/oikos/internal/db/sqlcgen"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
// ErrTransitionInvalid is a sentinel returned by ValidateTransition when the
// from→to pair is not a declared lifecycle transition or a precondition fails.
// Callers test with errors.Is to distinguish semantic validation failures
// (→ HTTP 409) from infrastructure errors (→ HTTP 500).
var ErrTransitionInvalid = errors.New("invalid lifecycle transition")
// ValidateTransition enforces an entity type's lifecycle: fromState → toState
// must be a declared transition, and every precondition it lists must hold. A
// type with no lifecycle defined allows any state. A no-op (fromState ==
// toState) passes immediately.
//
// Shared by the HTTP PATCH path and the MCP set_entity_state tool so both
// surfaces apply identical lifecycle rules — previously only the HTTP path
// validated transitions, so an agent changing state via MCP could skip the
// graph's retire/deprecate guardrails entirely.
func ValidateTransition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, fromState, toState string) error {
if toState == fromState {
return nil
}
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, entityType)
if err != nil {
if err == pgx.ErrNoRows {
return nil // no lifecycle defined → any state allowed
}
return err
}
var transitions map[string]map[string]json.RawMessage
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
return fmt.Errorf("parse lifecycle transitions: %w", err)
}
tos, ok := transitions[fromState]
if !ok {
return fmt.Errorf("%w: no transitions defined from %q", ErrTransitionInvalid, fromState)
}
trans, ok := tos[toState]
if !ok {
return fmt.Errorf("%w: %s → %s is not a declared lifecycle transition", ErrTransitionInvalid, fromState, toState)
}
var gate struct {
Requires []string `json:"requires"`
}
if err := json.Unmarshal(trans, &gate); err == nil {
for _, check := range gate.Requires {
if err := checkPrecondition(ctx, tx, entityID, entityType, check); err != nil {
return fmt.Errorf("%w: precondition %q not met: %w", ErrTransitionInvalid, check, err)
}
}
}
return nil
}
// checkPrecondition evaluates one mechanical precondition named by a lifecycle
// transition's `requires` list. Soft/operator-confirmed checks pass; unknown
// checks are skipped (operator intent overrides). Moved here from httpapi so
// both surfaces share one implementation.
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
switch check {
case "no-inbound-edges":
var count int
if err := tx.QueryRow(ctx,
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count); err != nil {
return err
}
if count > 0 {
return fmt.Errorf("%d inbound relationship edges remaining", count)
}
case "backups-verified", "secrets-revoked", "ingress-dns-removed":
var attrs string
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
return err
}
want := map[string]string{
"backups-verified": "backups_verified",
"secrets-revoked": "secrets_revoked",
"ingress-dns-removed": "ingress_dns_removed",
}[check]
if !strings.Contains(attrs, want) {
return fmt.Errorf("%s not recorded in entity attributes", want)
}
case "age-key-enrolled-if-needed":
if entityType == "workstation" {
var attrs string
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
return err
}
if !strings.Contains(attrs, "age_pubkey") {
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
}
}
case "mesh-joined-if-needed":
if entityType == "workstation" {
var attrs string
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
return err
}
if !strings.Contains(attrs, "mesh_ip") {
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
}
}
case "health-check-answering":
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
if err != nil || st.Health == "unknown" || st.Health == "down" {
h := "unknown"
if err == nil {
h = st.Health
}
return fmt.Errorf("health check not answering (status: %s)", h)
}
case "doc-page-complete":
var count int
if err := tx.QueryRow(ctx, `
SELECT count(*) FROM relationships r
JOIN entities ke ON ke.id = r.source_id
WHERE r.target_id = $1 AND r.valid_to IS NULL
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
entityID).Scan(&count); err != nil {
return err
}
if count == 0 {
return fmt.Errorf("no documentation linked to entity")
}
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
"ingress-live-if-public", "doc-page-stub", "un-deprecate-note", "write-off-note":
// Soft checks — always pass. Operator-confirmed via the transition
// request itself, or not mechanically enforceable.
default:
// Unknown preconditions are skipped (operator intent overrides).
}
return nil
}

View File

@@ -25,8 +25,8 @@ ON CONFLICT (actor, key) DO NOTHING;
-- name: InsertAuditEntry :exec
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
status_code, detail, source_ip, correlation_id)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
status_code, detail, source_ip, correlation_id, session_id)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11);
-- name: InsertEvent :one
INSERT INTO events (type, entity_id, severity, source, data, correlation_id)
@@ -63,12 +63,28 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
e.slug AS entity_slug
FROM check_defs cd
JOIN entities e ON e.id = cd.entity_id
LEFT JOIN entities tgt ON tgt.id = cd.target_id
WHERE cd.enabled = true
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
AND (cd.last_run_at IS NULL
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s));
-- name: MarkCheckRun :exec
UPDATE check_defs SET last_run_at = now() WHERE entity_id = $1;
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1;
-- name: WorstHealthForTarget :one
-- An entity is as healthy as its unhealthiest check. Checks that have not run
-- yet (last_health IS NULL) are ignored rather than counted as unknown, so a
-- newly added check does not drag a known-good entity down before it has
-- produced a verdict.
SELECT COALESCE(
(SELECT last_health FROM check_defs
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
ORDER BY CASE last_health
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
WHEN 'unknown' THEN 3 ELSE 4 END
LIMIT 1),
'unknown')::text AS health;
-- name: GetCheckDef :one
SELECT * FROM check_defs WHERE entity_id = $1;

View File

@@ -67,12 +67,20 @@ func IngestOntologySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*S
targetType, _ := rtMap["target"].(string)
cardinality, _ := rtMap["cardinality"].(string)
desc, _ := rtMap["description"].(string)
// Which end of the edge depends on the other; drives blast_radius().
// Absent means 'none' — an undeclared edge contributes nothing rather
// than silently producing a wrong dependency answer.
blastDirection, _ := rtMap["blast_direction"].(string)
if blastDirection == "" {
blastDirection = "none"
}
_, err := tx.Exec(ctx,
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description)
VALUES ($1, $2, $3, $4, $5, $6)
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description, blast_direction)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (name) DO UPDATE SET inverse = $2, source_type = $3,
target_type = $4, cardinality = $5, description = $6`,
name, nullableStr(inverse), sourceType, targetType, cardinality, desc)
target_type = $4, cardinality = $5, description = $6,
blast_direction = $7`,
name, nullableStr(inverse), sourceType, targetType, cardinality, desc, blastDirection)
if err != nil {
return nil, fmt.Errorf("relationship_type %s: %w", name, err)
}

View File

@@ -114,6 +114,8 @@ type CheckDef struct {
UpdatedAt time.Time
// When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.
LastRunAt *time.Time
// This check's own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target's enabled checks.
LastHealth *string
}
type Classification struct {
@@ -255,6 +257,20 @@ type KnowledgeEntity struct {
UpdatedAt time.Time
ContentHash *string
Search interface{}
EditedBy string
DeletedAt *time.Time
}
type KnowledgeRevision struct {
ID int64
EntityID uuid.UUID
Title string
Content string
Source *string
Tags []string
EditedBy string
VersionAt time.Time
RevisedAt time.Time
}
type Ledger struct {
@@ -364,6 +380,8 @@ type RelationshipType struct {
Cardinality string
Description *string
CreatedAt time.Time
// Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().
BlastDirection string
}
type RiskClass struct {
@@ -380,18 +398,19 @@ type SeedVersion struct {
}
type SessionPlanStep struct {
ID uuid.UUID
SessionID uuid.UUID
Seq int32
Title string
Detail string
Status string
ExecutionID *uuid.UUID
TargetSlug *string
StartedAt *time.Time
FinishedAt *time.Time
CreatedAt time.Time
Generation int32
ID uuid.UUID
SessionID uuid.UUID
Seq int32
Title string
Detail string
Status string
ExecutionID *uuid.UUID
TargetSlug *string
StartedAt *time.Time
FinishedAt *time.Time
CreatedAt time.Time
Generation int32
ReplacedReason *string
}
type SessionQuestion struct {

View File

@@ -99,7 +99,7 @@ func (q *Queries) ListLifecycleDefs(ctx context.Context) ([]LifecycleDef, error)
}
const listRelationshipTypes = `-- name: ListRelationshipTypes :many
SELECT name, inverse, source_type, target_type, cardinality, description, created_at FROM relationship_types ORDER BY name
SELECT name, inverse, source_type, target_type, cardinality, description, created_at, blast_direction FROM relationship_types ORDER BY name
`
func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType, error) {
@@ -119,6 +119,7 @@ func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType
&i.Cardinality,
&i.Description,
&i.CreatedAt,
&i.BlastDirection,
); err != nil {
return nil, err
}

View File

@@ -51,7 +51,7 @@ func (q *Queries) GetAutonomySetting(ctx context.Context, key string) (string, e
}
const getCheckDef = `-- name: GetCheckDef :one
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at FROM check_defs WHERE entity_id = $1
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at, last_health FROM check_defs WHERE entity_id = $1
`
func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef, error) {
@@ -69,6 +69,7 @@ func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef
&i.Enabled,
&i.UpdatedAt,
&i.LastRunAt,
&i.LastHealth,
)
return i, err
}
@@ -352,8 +353,8 @@ func (q *Queries) InsertApproval(ctx context.Context, arg InsertApprovalParams)
const insertAuditEntry = `-- name: InsertAuditEntry :exec
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
status_code, detail, source_ip, correlation_id)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
status_code, detail, source_ip, correlation_id, session_id)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
`
type InsertAuditEntryParams struct {
@@ -367,6 +368,7 @@ type InsertAuditEntryParams struct {
Detail []byte
SourceIp *string
CorrelationID *string
SessionID *uuid.UUID
}
func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryParams) error {
@@ -381,6 +383,7 @@ func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryPara
arg.Detail,
arg.SourceIp,
arg.CorrelationID,
arg.SessionID,
)
return err
}
@@ -695,7 +698,9 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
e.slug AS entity_slug
FROM check_defs cd
JOIN entities e ON e.id = cd.entity_id
LEFT JOIN entities tgt ON tgt.id = cd.target_id
WHERE cd.enabled = true
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
AND (cd.last_run_at IS NULL
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s))
`
@@ -1133,11 +1138,16 @@ func (q *Queries) ListSkills(ctx context.Context, status *string) ([]Skill, erro
}
const markCheckRun = `-- name: MarkCheckRun :exec
UPDATE check_defs SET last_run_at = now() WHERE entity_id = $1
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1
`
func (q *Queries) MarkCheckRun(ctx context.Context, entityID uuid.UUID) error {
_, err := q.db.Exec(ctx, markCheckRun, entityID)
type MarkCheckRunParams struct {
EntityID uuid.UUID
LastHealth *string
}
func (q *Queries) MarkCheckRun(ctx context.Context, arg MarkCheckRunParams) error {
_, err := q.db.Exec(ctx, markCheckRun, arg.EntityID, arg.LastHealth)
return err
}
@@ -1462,3 +1472,25 @@ func (q *Queries) UpsertSignal(ctx context.Context, arg UpsertSignalParams) (Sig
)
return i, err
}
const worstHealthForTarget = `-- name: WorstHealthForTarget :one
SELECT COALESCE(
(SELECT last_health FROM check_defs
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
ORDER BY CASE last_health
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
WHEN 'unknown' THEN 3 ELSE 4 END
LIMIT 1),
'unknown')::text AS health
`
// An entity is as healthy as its unhealthiest check. Checks that have not run
// yet (last_health IS NULL) are ignored rather than counted as unknown, so a
// newly added check does not drag a known-good entity down before it has
// produced a verdict.
func (q *Queries) WorstHealthForTarget(ctx context.Context, targetID *uuid.UUID) (string, error) {
row := q.db.QueryRow(ctx, worstHealthForTarget, targetID)
var health string
err := row.Scan(&health)
return health, err
}

View File

@@ -89,6 +89,7 @@ func (s *Server) CreateApprovalRule(ctx context.Context, req gen.CreateApprovalR
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
&id, "POST", "/api/v1/policy/approval-rules", "",
nil,
map[string]any{"action": req.Body.Action, "risk_class": req.Body.RiskClass}); auditErr != nil {
return nil, auditErr
}
@@ -148,6 +149,7 @@ func (s *Server) PatchApprovalRule(ctx context.Context, req gen.PatchApprovalRul
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
&id, "PATCH", "/api/v1/policy/approval-rules/"+req.Id, "",
nil,
map[string]any{"action": req.Body.Action}); auditErr != nil {
return nil, auditErr
}

View File

@@ -154,6 +154,7 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
if auditErr := observability.Audit(ctx, q, actorType, actor, "decide",
&id, "POST", "/api/v1/approvals/"+req.Id+"/decision", "",
nil,
map[string]any{"decision": status}); auditErr != nil {
return nil, auditErr
}

20
internal/httpapi/audit.go Normal file
View File

@@ -0,0 +1,20 @@
package httpapi
import (
"net/http"
"github.com/dtoro/oikos/internal/audit"
)
// serveAuditDrift returns a read-only DB-side drift report: orphan check
// entities, checks on retired targets, probes stuck down/unknown, unmonitored
// declared types, and dangling edges. Companion to the knowledge-graph-audit
// skill. Live-infra discovery (pct/docker/certs) is a follow-up.
func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) {
findings, summary := audit.Report(req.Context(), s.pool)
writeJSON(w, map[string]any{
"findings": findings,
"summary": summary,
"note": "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up",
})
}

View File

@@ -81,6 +81,7 @@ func (s *Server) PatchAutonomySettings(ctx context.Context, req gen.PatchAutonom
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
nil, "PATCH", "/api/v1/policy/autonomy", "",
nil,
map[string]any{"keys": keysOfMap(*req.Body)}); auditErr != nil {
return nil, auditErr
}

View File

@@ -22,7 +22,7 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
SELECT cd.entity_id, e.slug, cd.kind,
COALESCE(te.slug, '') AS target_slug, cd.target_type,
cd.config, cd.interval_s, cd.timeout_s, cd.zone, cd.enabled,
e.version
e.version, cd.last_health, cd.last_run_at
FROM check_defs cd
JOIN entities e ON e.id = cd.entity_id
LEFT JOIN entities te ON te.id = cd.target_id
@@ -43,10 +43,19 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
var c gen.Check
var targetSlug string
var configBytes []byte
// last_health is what turns a check list from configuration into an
// explanation: an entity's health is the worst of these, so this is
// the field that says which probe is responsible.
var lastHealth *string
if err := rows.Scan(&c.Id, &c.Slug, &c.Kind, &targetSlug, &c.TargetType,
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version); err != nil {
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version,
&lastHealth, &c.LastRunAt); err != nil {
return nil, err
}
if lastHealth != nil {
h := gen.CheckLastHealth(*lastHealth)
c.LastHealth = &h
}
if targetSlug != "" {
c.Target = &targetSlug
}
@@ -173,6 +182,7 @@ func (s *Server) CreateCheck(ctx context.Context, req gen.CreateCheckRequestObje
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
&id, "POST", "/api/v1/checks", "",
nil,
map[string]any{"kind": req.Body.Kind, "slug": slug}); auditErr != nil {
return nil, auditErr
}
@@ -260,6 +270,7 @@ func (s *Server) PatchCheck(ctx context.Context, req gen.PatchCheckRequestObject
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
&id, "PATCH", "/api/v1/checks/"+req.Id, "",
nil,
map[string]any{"enabled": updated.Enabled}); auditErr != nil {
return nil, auditErr
}
@@ -285,6 +296,15 @@ func checkDefToGen(cd sqlcgen.CheckDef) gen.Check {
if len(cd.Config) > 0 && json.Unmarshal(cd.Config, &config) == nil && len(config) > 0 {
c.Config = &config
}
// Carried through so toggling a check does not blank its verdict in the
// UI — the entity window renders last_health to explain which probe is
// responsible for an entity's health, and a patch response missing it
// would erase that until the next poll.
c.LastRunAt = cd.LastRunAt
if cd.LastHealth != nil {
h := gen.CheckLastHealth(*cd.LastHealth)
c.LastHealth = &h
}
return c
}

View File

@@ -3,31 +3,22 @@ package httpapi
import (
"context"
"github.com/dtoro/oikos/internal/checkdefaults"
"github.com/dtoro/oikos/internal/db"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
// ensureDefaultChecks derives an entity's default checks from the monitoring
// kinds its type declares.
// kinds its type declares. Thin wrapper over the shared db.EnsureEntityChecks
// hook so the HTTP create/patch paths and the MCP entity-mutation tools stay
// in lockstep.
//
// Note the ordering caveat: an entity created through the API usually has no
// edges yet, so a type whose address comes from its host (a service) will
// produce no checks on this pass. That gap is real and deliberately visible —
// coverageSweep reports it, and the next inventory ingest fills it in once
// the hosting edge exists.
// Note the ordering caveat (carried from db.LoadTypeTree / checkdefaults.Ensure):
// an entity created through the API usually has no edges yet, so a type whose
// address comes from its host (a service) will produce no checks on this pass.
// That gap is real and deliberately visible — coverageSweep reports it, and
// the next inventory ingest fills it in once the hosting edge exists.
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType, name string, attrsJSON []byte) error {
tree, err := db.LoadTypeTree(ctx, tx)
if err != nil {
return err
}
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
ID: entityID, Slug: slug, Type: entityType, Name: name, Attrs: attrsJSON,
})
if err != nil {
return err
}
checkdefaults.LogResult(slug, entityType, res)
return nil
_, err := db.EnsureEntityChecks(ctx, tx, entityID, slug, entityType, name, attrsJSON)
return err
}

View File

@@ -67,6 +67,7 @@ func (s *Server) CreateEntityType(ctx context.Context, req gen.CreateEntityTypeR
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
nil, "POST", "/api/v1/ontology/entity-types", "",
nil,
map[string]any{"name": req.Body.Name, "domain": req.Body.Domain}); auditErr != nil {
return nil, auditErr
}
@@ -148,6 +149,7 @@ func (s *Server) PatchEntityType(ctx context.Context, req gen.PatchEntityTypeReq
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
nil, "PATCH", "/api/v1/ontology/entity-types/"+req.Name, "",
nil,
map[string]any{"status": req.Body.Status}); auditErr != nil {
return nil, auditErr
}

View File

@@ -240,6 +240,7 @@ func (s *Server) RequestExecution(ctx context.Context, req gen.RequestExecutionR
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
&id, "POST", "/api/v1/executions", "",
nil,
map[string]any{"action": req.Body.Action, "target": req.Body.Target}); auditErr != nil {
return nil, auditErr
}
@@ -307,6 +308,7 @@ func (s *Server) CancelExecution(ctx context.Context, req gen.CancelExecutionReq
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, q, actorType, actor, "cancel",
&id, "POST", "/api/v1/executions/"+req.Id+"/cancel", "",
nil,
map[string]any{"status": "cancelled"}); auditErr != nil {
return nil, auditErr
}

View File

@@ -86,6 +86,14 @@ const (
CheckKindTcp CheckKind = "tcp"
)
// Defines values for CheckLastHealth.
const (
CheckLastHealthDegraded CheckLastHealth = "degraded"
CheckLastHealthDown CheckLastHealth = "down"
CheckLastHealthHealthy CheckLastHealth = "healthy"
CheckLastHealthUnknown CheckLastHealth = "unknown"
)
// Defines values for CheckCreateKind.
const (
CheckCreateKindCertExpiry CheckCreateKind = "cert-expiry"
@@ -273,10 +281,10 @@ const (
// Defines values for TrendDirection.
const (
Degrading TrendDirection = "degrading"
Improving TrendDirection = "improving"
Stable TrendDirection = "stable"
Unknown TrendDirection = "unknown"
TrendDirectionDegrading TrendDirection = "degrading"
TrendDirectionImproving TrendDirection = "improving"
TrendDirectionStable TrendDirection = "stable"
TrendDirectionUnknown TrendDirection = "unknown"
)
// Defines values for ListApprovalsParamsStatus.
@@ -462,7 +470,13 @@ type Check struct {
Id openapi_types.UUID `json:"id"`
IntervalS int `json:"interval_s"`
Kind CheckKind `json:"kind"`
Slug string `json:"slug"`
// LastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
LastHealth *CheckLastHealth `json:"last_health"`
// LastRunAt When this check last executed. Null = never run.
LastRunAt *time.Time `json:"last_run_at"`
Slug string `json:"slug"`
// Target Entity slug (instance-scoped)
Target *string `json:"target"`
@@ -477,6 +491,9 @@ type Check struct {
// CheckKind defines model for Check.Kind.
type CheckKind string
// CheckLastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
type CheckLastHealth string
// CheckCreate defines model for CheckCreate.
type CheckCreate struct {
Config *map[string]interface{} `json:"config,omitempty"`
@@ -8208,175 +8225,177 @@ func (sh *strictHandler) GetTrends(w http.ResponseWriter, r *http.Request, entit
// Base64 encoded, gzipped, json marshaled Swagger object
var swaggerSpec = []string{
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpCn6MpPsyLU/FFljO7FjraXJt6mRiwK7D0mM0EAPgKbEuFyV",
"X/sAW3nCPMlXuHY3iSabF1nOVP7YkhqNBs4FOPfzqZfyvOAMmJK9k0+9GeAMhPnx/ApP9f8ZyFSQQhHO",
"eie9DyB5KVJAcxCScIYmXKA3k8E7rNJZL+nJdAY51u+pRQG9k55UgrBp7/Pnz0mvwALnoNwHzkohuVj9",
"xPsC/1oCSs1jNBE8RxgVAuaElxIJkAVnEr6RiMG9GtlhvaRH9Lu/liAWvaTHcK4/Hh62LyvpnTNF1OJN",
"trqSn3568xJxgSQtp6gPx9NjdDPjUp3MyrEg8ubIf7bAalZ9lWS9pCfg15IIyHonSpTQZQWXtIwA3D5r",
"LOFOnuQ4HeSEkZsE3dD79CTFWbZoW49+d8sV/Sh4fkX025+igNVYaYB1wkWOVe+kl2EFA6VfTSLzvskg",
"L7gCli7+DIvV3Z5RAkwNpsBAYAUZuoXFCySgoHgh0R1RM8LQs+9mSIAqBUNqBogLMiUM00AaHgqWmKtF",
"1z4+0F+vrz/H92+BTdWsd/L02f+KLn1iaXwVQ1d4WpEp4QK9Or96gb57+gxxFvgkJzJ3PBJfXMVD2yDq",
"LcmJasMSNQ/rE2QwwSVVvZPvnyR6zyQv897Jsyf6N8Lsb0/D7glTMAVhPnTF19GD4ttTw2e9U4sxcx5c",
"AMsIm54WheBzTPWfUs4UMLM/XBSUpFjDfPiL1ID/VPvg/xQw6Z30/sewOs6G9qkchgnNJ5fobYbZFJBU",
"eArZC4RRDgoPsHsD3WGJUgGGEvtZielAr0hwetT7nPQuBB9TyNcstLAjfrfdgv28kfWeC8EF6n/48Qz9",
"8N33fzDLuCRThulPhYZ1djCo2Vlja3BfQtKP8Jg3WDydAlOnqSJzogyDF4IXIBSxSMbuychSw6ceME1z",
"P/cU53SUYkoNA2DJmaYS/e2UaAbqJb08LUae7kCmmJp99T6ukFbSw3oVI5JFmCbppVwIsC+7IaykFI8p",
"eI5beSUrhR2fyzXjA78kPTDHdtfpGwutzUJYUaqRLPMci0WnmXiptn1FgpRbgEKWaQpyHRjGnFPATA9W",
"/BbYKOWlJcfNcDNkYA+VDmuxQkvHu6c6Vn+2V7SSvRqlJEu0WZEVH/8CqdLfq59Nq3Rt2WuV3OwBMsKq",
"62It1Wfr39lMs26OcTcygPuCCJBbLdOSTBhblhauy8NuCcvqvA73kJbKMnXBKUkXg9QcxPp3rBQINjDI",
"aGfwAi8oxxGZzYhIGjdcQobs7Cgjk0k7yCr8CiJvRynFlrxXSd9JaKsPFFalrG+xsJeZpipDM5CZs4wR",
"84OFtRUT5/wWsugeZWkXtlYm1BJQuK9SzlIQTG4mjxg/ODnRkXIDGg6HYacNcmmQ+Dq2+VBS2Ip1cKk4",
"4/liRGEOtA5f/aS6Bgw/wBxEFI7uLPY3ThOW7/ACjQHhsVQCpwr1CZuBIEqijN+xoy6M1pELNhFXygsY",
"2bWuR7lWuQoQAzsW8TkIQTKQXdbqxNHYdRMjiTgtLKGlmnUT8s8MnTw+CeyLm/XM1AloUVCVGVHnTInF",
"diBKFRddr287eFn6MrdgL+npL2JlVeaFVOCVvKykLZDdRZgChQmtbaWCwGHEphzUjHebwmjKncQeY/YY",
"kaLbaHNMjlKeQUe55wCSTIXZwLhxKrOEeAlK6RlXSO3WauZwj/NCr7k3pXyM6bGm4BFOVexwK61SsJX0",
"MMe0jLNj91Pq1ujxdqb159DZDNLb1c2mnE1IxO7yV0yJ1XP0Uetuvwi9arTWybAm/Ha8F/TWxBzTkYxT",
"87L0NFOq0POk+t+MyFt9AYNQA3Mla3Bkgkw0lgorgUg5G9itxcWMNqlGYTGFDeJHnzCpMEthYM7IrNOF",
"aSduuZDd7Poh6ut/t5qZ5MC1/hMH5Rq6Snp/56yL1rFGcnJUUkNofUUVtXQg1LabsiLXdbQYzDxtelmT",
"5sLw3z95knydFLiJhtZTQtjg0+j+PObXY7qO5Fa8XXgb4S5o24SnyLWxjt4/xxappREycTah3SSx1J+k",
"K0PGFEs1EjgjVhsiCvK4ROX+gIXAi7gYcRA9uuMR7JTOkUFTBixddxCwMh9b6Fd2qhhiBaQ8z4EZPT6A",
"dV8dVPBSwbIYPLC3ck0UnnHaolQaq11nY88toZ0HV9y6wxEaF5rtbpsWwSVS2ah9Wp/CGWcK7lWE4o0B",
"aEIoyJG1QkSsChdYzSTiE2RGI33pidKsGJk3kZphhfzryRaEL4mjtuYHr0gOUuG8MNqeVvIZ3CtUcEqR",
"hh1IjfBuTCB5IS1pT9t3eCVKQGSCjvXo4wXO9XdSUmjYydrOYjY+TruAzowbfnssQZXFsZztDrPaNb6k",
"zHPGFWckRalFd3C/OKZNNsmTay9mQ0iXkAqw4vqK2CxXl/SGTYgkKaZImheRHoawsaGSMQWkOFIzIlFq",
"Zt8CDquSsIwu+yWWszHHIrusjMJLLOBUdDnyZqvoZWOUMwJyNF6MtIJjyBZnGdFbxfSiMefq60t2Oivm",
"GdOw1ECBDI0XyM6btGiG7uP+0j/wt53qtPrpuT4hhNvw0lT6mZ9pXKa3oOxk3w9ywkoFyF/hCcq5VJqp",
"9Bv6oqzjuokQO1H3ay4Y2DdQt5vXvxCjlmVeC7baA6HdTxfHfGIAg559N4shYgaYqoh0lcFU4AxarAEZ",
"v2uR+O18i/hDqTCNINysj4+lxqnZB6cZ6BMaW2+0EY6+kQjuC0g1LaTYChQxwbNkt6xldUuY80tNqs26",
"nVWzxNBpr3uLO5iDcAasXdHHC2CozzgbCJCcziE7cn7AVXz6z62samlrK5wdO2kC8uNbSiJnWJx2G+wc",
"g9g5E5zSD+6OXaG1GZfK+6easDlNVYkp8gPMlTcDBGY+wqYox+mMsCgD5yBnzrbUnPTSRtvo5+jNhREG",
"tIBqzq+5NVFYsalVqdoUTnInTxjcDSguFC+ONtqbzLTr4OZiMGJy1qgQZI4VjG5jsR+nr84Hl+dnH86v",
"Bn8+/9vg+PjYbJdyfXlmkIpF0bZXM3c5piSNT42n8FTPZ8doGjVTX76/uKxJOXHjjLu+R/Z+drJw2x3/",
"EyNagsD0tFQzd6WjNy87zWzlg61nd6/FiMrS28gTzMh4Y9d9wL1RkZiVU5B9cRNpLGEhWUF5HJztoIiT",
"mYoHFiglyLhUjXOsem0X5bG6blqvAMiQHZWguxkwx/AGdESinDOiuHX0BdtJh4PcXz4fD+d8Moq5uZsc",
"CJpbMou3SzNBL40ttikaG9eWY32RMMxSGJVMWZv/blP5I7f1mKus1bUIvV6Lab5jdEGbbWknY/d2jjdn",
"bHL3odl9NUeDnBvLaeeYVtdbg2/abOB4irXOa+hbf+AbicKLIxdQFPn0Rqy1Y6e5kpfWgCettgSIkgmk",
"i5TqhTjj3mhJdVjF45KuWEplPL9IizPB/wuV+bnbXdhEUjsCLuLRg6cKUTDcxoLIMCFAM4lyt8JCgASm",
"jnvJFrh7B8KEtM1XcNgFcV+Ec+OovjJWpArDVjFAfSUwk0ZorfYUF1daEHDlyKAFhqN6lGR9QX+6fP8X",
"dOlBtdF+13g5su2Ma+BGHxE58nQYNwdTvABRN/7loLC9QAW2JqlSaKxM+RyEQZ9R9qaMtEbSBEB3NfO1",
"IrTAQl/ent02GxcNTEdrnTKrkTUmMAjM/VkISE3Q48e9Dlx3ujrEeCg30RFW8nEtfW08ZUcrwbwPQTnB",
"3zHBVEYdQCuUdEgS2plk1h+3cTytR0iLP+Yg+NiVNqNH1NxF7C77jbaPrsAKP2BsRd2KUKMd3kt6d1h4",
"E70gSsvzcQ+EUWnjBk7ZXaAKISxB8LOGgWOBiYRsi8AJd3/XjAluiVHSCrGLW/nOavHIm0N0nCmj6/i0",
"4dPr/BbXYFN7Bpk+kN9u63jrzo4+gfOItHR5SyhF9inqr8pM9ok7LI5iEpMAaU7c/T18D+ihs4NrN+Nm",
"wK6T1MW+1BOJm3WRu83AWRt5tDaQ1tn6zPGj+XiyqP1sB08wsdEXennZiJfGEq5vOGr/Lrj+YTTG6a37",
"Tf84cu993MflWVtIRLLbOho3hOFu6wwNx1ergbM6xaqTVYDB9nqOirAElm1XZ53Gl1jRmpONczDnJggc",
"Musi6/PCGq2PesnW4Ur1ZL6Nl4Oba20c3SuBi9lfCdytwhCyKTQDINal2nxwCJQzUsRcMJUdqs1sv6Nx",
"KRKZGXGTkQwNrssnT55DsHU5jdTavAhLaZnB/7Y0acxHzkMN0Zg5xrMtgOPsfRGwKFGy1Oc8xZ3Z+lMo",
"xYVZ1IyoiP96WcbkNpDaYjCG9tcGBq2+VO+6aGxwyYUQ8HkovK1Y93Y8jlvjsAQ0I8BIrs9in6Gll+wC",
"vZT5QMMntaudba2xxcGwiwdTtqHqP27DFrfhMuQdACu/XBTuf2b8jmq+eU0i10pXOzVht5CNoly0MS5E",
"YHbbKXCrXahhpChil8hrMp1RMp1p1Jg8Xh9h0omvbOx451hzRRSFNTuu+DDjaZnbsBFRsjHnt8YaNAep",
"yLQte2qzvdkuIIbkt17Vf6mP7FWOqptio4aKrN0WuCW2FYicaClip5eDNTGiDXyaCJ6foE+Kn6BPDlTy",
"BP3McA7ZwLBqgo6Pjz9+/vx5o3ub+LQpc6+sGKtr64jB+x0oQdJLEA7Ckctm0aZ45ebdlihCSsuiTkkC",
"3/WS3tOZ/qclctBIg+suNjyfdmK/LfJBc3zfacqcsE7jtrEwhPyEpRIY+A5ZWCCfd7Dhs8vCpex0b4VL",
"d514dGUGRUMqFlYhcFQQcF4hMraICxsJu51xoygoAdkei90Mq21C878IlZwhyu9AoDEvWZZoia2wQSQw",
"t+/ZFOLh970ISptj4reyVuFKsXbINo7UYCDYS9wqKlivPPu1xAIzRVhbZPgWuaizRcHVDCT5u8098Iv3",
"Kc9LBktzgYQxH9tTwNdBczd3Z4OSarqvh1SDlFYwX1OL14Vx1so2LN9eS3lptfxBIbiI3BQ/EqDZwGT0",
"1cJxkB2O+t89e3bUHuVn3Hzx47lNc16CnZ0hjO9yqvh8nQ20E0s12CSUBK9DD495qU7GVMtjteCBUpDN",
"mrf5zFp3y4VWPeRaG8Y6p/a7Ny8TlHIBMkEC56N8nKCMyNvRdJwgUiRIQV5QE4yYm5i2BGmxnaQgo0GJ",
"XEbkxUtaTr0790Lw+5zfm8gwF3RVi1GI2jLiEWavyxyzgQCc6XMFOX9Ix8gv8116n578ApQuJoRt7yqn",
"92mC5nmCuEAZT29BmHoomLB6aHV3Z7kD3gYctwWUVem43ewHIRowanYKhjH05qWJMhA4vUWFXwZhU/3L",
"VICxv224JqLXcW9pCWu3fRk4cWnT+mSJ1Myag8CU2oMHkUlz4cHwubsFoMVZf1YKASxETbSGYEgFxTrJ",
"0ax7lIOUeNrNeTwhjMjZ/vbnh7BhhwBUUTLnETOKWcCDvNVqZsvlqqDoYA3Ro9aekmuTBRwzenxZ9MRm",
"adgmN5yzGx0flecvfli60nKBXTqbeMNpa2NH1kzQJqKay3tkypptox+QbKT4rrSzjBMLnaQyPruzsra2",
"TSjqFuXVGTHrbebt+Nj4XjeLXxwgm2AQj/NJscgIw3TJdc0ZDBQfcBOW7X7JMdPEo/+rnvnfzMONtvOY",
"5YNpoRT2C7FxlqROpUx622Zeb3w/Ho9RX1PzC0kD6lG8EXl75j2h8aSkUfXJCm3MIawqV+F/tIluIm85",
"XUMSK6Za5WxRrjbhsgU/cfisbiSyjBhwXAm2VVI2hv6OLl+vI3W8VIVUIwnAtvLWTygu1imDM06zUcbv",
"2L6xhNvWm6pCQ6wAP3Cm77hWv/W+KbkFuhiluOzI13mp9o6n5GlqhK71Bo8DhOlsFAUr06ELuMHprfcB",
"eOOC+Y7etlVTbc5QJQp93OWSr2LsbeLkIYpe+fJWtRggJxutwHuZTZaoJ8rJt4TSvWxqmwNxTB7tqLIc",
"dHyjc324bexjpdxTqF4TMGgz9Um2pcG/EDyFrBQx8dNHPWbICMJDGz8y9AEgQx8XVFDM0Ifngx+OVkKx",
"vd9uFCKX2qq5MH0OentkvuIL37yReihSPO7Crbuzj92Q56XWKGLu0xXVbeepLFwPMVdEG5Ih8CeeCNnd",
"XpoJPHGxCz6IIdhJBUyMTbamzG2IQ/bGUrE+7aSyZ+9mOl1JEQnG02AUrVig9Yy6dMrnchRqnmMWMZq8",
"4sFYZsrQuTi5Xry8pKtluMw4RIU6WTF3esZLPcCYmWRc6OqedyIgXlxN6W2o1hRYihfdS6xopb8ZWy3l",
"rJf4qjkmH5y1XLptd98WgI4Xvvn9k42lD9y6k4DuGJVceafUEgAZzzFdtEjTREAVVLZDBMmqwMk1x0Xt",
"rsQ45ihhgAUqBP/FfjpBf8iQjfjb7Ets95tKymOa01v7uQlRqACBMrzY2ikY3HQVtKKBGRLSUksoJiHF",
"VQsALECclrFcxfdOLRrOCdyBOEF6mJaebtH7Ny/P0J/+66oe70rY4PTiDfrXP/6JznCWLa7ZhIs7LLIB",
"LtUMEZNtBUzCgLBBBoWaJYhxmxfmrDdaQhOlmh0dXzNTDPrEmAVJiuw6bTKpLZheZZ72TZEvdGMCpW/0",
"u76guCEm82ZF7YaVTGlqI9O6mtcu+cGVJM8UF3w1qO3MFvAe6LsckN6sL7DyntxyiWY8B4rH6P3lMbrS",
"4uWEUNAb10O+/TZs8pqZXX77LeqbmuA4VQMjFx6doFfceAxAIKnKsURYAKpK2t8RNUMcF2Sgj70psOSa",
"2bxXifr+82dv3yRoUmqpBP30Rh5ZeBkw4xyQLCA9vmbX7IyzuUYnZzX55PnRyTUboHPrhdJf9wXD0U1b",
"efKbY/3KWyKVRKUEdPPJ3NFJvcvC5xu7eNeaocBTwqzDq+8OGmRKzqPvnyQox/fo2ZMnR2ben5jEE0AX",
"7y+vbPGTQqGbpXr8N6hvK/sXFC/QHWEZv7NvvyvNoYCEaz4hUYqFWKAbd9vdvECvzq9cTwCJbs6v8PQm",
"QRenV2evkY/fQDe+xP4N6rvi/L4ov/1MqLlTwez58+c/oJ+uzszzcxeUZJ7iLBMgpVnXuBldivrNLhEG",
"UVczQO/OLmw5kAlOAfWlEoBzM8Prq6uLBPHJhKQEU01Aly//fGRziEtmItEVuhnmaXFzzTirCGFMGBYL",
"hFmmB/NSGTuq4SVL15plXZDQC0RMBiWnEt0JXFyzip6sfoxMSg3ChtqlVrOyghOmpOVHSlJwrhjHZBc2",
"u1sf14I6xpQnw6HTvI+dJ3nossBrfsSeZbfTizc1qeWk9/T4yfETo+YWwHBBeie958dPjp9bJ/DMnHdD",
"c0gMcK3IvLs0rRGIcPYm6530/k8JYtGsR9/sQfJzvJlBrST4mtYLLe82aojvMEE9dGPtyzHJudrcMHTw",
"6DDW9XboMNL1bOkw0jam+PxxqcnDsydPtmpRsBRE6NWGTvpDE/URfaTeQGb7qmVmCZE7euXK8UtAwJSJ",
"4/qcVIJZfAsBZrVmELVIVttlAY1hhufE1MgwZnY8lcambcNMx8SbXe8HfuG2lmbvpGfFATPrMJROaeUk",
"fS2chlGdmChoHRUuD1cbvY15vBVn5ZN7V5z/zfFGaIryeGwRCGp/ftAEinCNQj0vVCWAtmGE4SeSfR6G",
"1iMa1k2K34Dg0FJK47hwASJNlnppujMENCRbfmGpkZKlJRMN80eeLfYgo/qmQ1qrZVPLpYvAmVHVjPGW",
"yN+W4jOv352eVQ0MrG7Ql4RNKQxKCQnyeVNOpB5IksHmMkVhG3FKbLZY+rwnI+7afeilWyQSkHKRQXaI",
"m8HiyoToAFvUYemg2wrQriwTvG6WacrMxvuvkcHMkG6yV72I+S7Sl62E/5CCV5vYx9kub64k8P1H6Nvv",
"Yqt6KDzi1aYX4cU91FcSvTy/PDs6BHubmbeX95o8azzI68W9MzukE9OuiF0daT/EdezArL6S+sqrtcy+",
"3xZl2wYGj0fUjiIOJKwZEkQZTAhz6S8VQbsKjxsktjbJysZAWWg9oli1EZUuVquTPPL0sJ+Ootcmjh8A",
"v3YmhB2O+/a0GWA5yLDCCfJmyj8cdcZ57PgyQvq+srkvD9MkIVM1ZkcKcm1CH5R0bFWbLyzJtlKObz25",
"P+XYmYzsSqxp1RHRroTSqIyy4cJbGtvNyhHqGHxpidNXxl+1dWzVEOA3d0k2+1s84m25RE4HOFbdjKA1",
"O6s4atlyBshFE/JSDvwTZNQypAQm9GhXe4jzSg1tCWODmmiyi68TKX2d4iS4uyTCDOEpoFtYFJiIxPXT",
"NX9vLzybGI9GLTm2EfXFG+kNx+gMUwrC1kvEVADOFmiG56C/4YtYMHPtMMj06dLIjjCBXtbB0TwVbEXj",
"M1+X/yFO82ax6S98oC9VbI61GzYjcmAqNNe2HkDbxIBlAWEHoG/7MYQRgztf3Phf//gnIlKW4GnI00+N",
"dsISKip3hNtC4rbZXYPCP0laTj8P06pJSDQM44PzMN7NSDpzvUBM/4/EutUs2Zqy0rbfhm9vgUybD0PE",
"UzIHhpT3NRovM0PeAWwafBivHWFSAc4Qn6ApUagoKY0R6StQzQYnK/dWbAuIM7pwi5NhcURW67JNpp8/",
"f/7DUUtzfdu5ZOu23x8fUkRpQCJ2Kru2IBlQhQ9As69AOTJI6zM7iOIKnNsSZ7KTVHtJy2nv88cIZcuq",
"a8l0bZHxgQkwMdZB84Z1Jme+8K6d9hu5cmK3E6ZvmPLgePcfiuD9slPvlQOpth5y65q8fFliyHwLmGGt",
"GE5UEn4FaqVfzAMibuVbMSu5H4P84vfH03sGA8FLlg2UIIUJqdOCT4gFcm3+keA8NyFBqMBT2MPHWi9o",
"06qC+ACTTWf4j4QqEKY+Qr1ZoivEJZEeDSzDTMm2w3tXC3vIGNz2xVC1des3fTnetS8usXs5tg9drR7O",
"TGDO0AXJxr7y656m938rLam9qNoX0o5cTT5K5MFOXaiYJyg7tXJVOxsSz/319lVaEhvF/b+wKdFTUbst",
"0dRKy8BWzji/wtO2Kd2woRnjJjyIDZKhFeVgA1U0LUh+8DCojO1q8JnTbOtNQCq10/W3mgddOeVMav3c",
"xH3aqhRac05xgVOjAvuI76MEeUOWm926G6vqtCaiJaIzNxVlfQz63QWXe0ynCFUIvm7aX6kD8oXpf7VG",
"RftJ58qyJk2E/FpC+ahsEraAMNKvlSqQbv/t/z1L0F/fJSjU+DhCZqAp2rEvP3njfZsUGkjvAc0fbceX",
"w5mrB/R42HkVahkshxjvdMkdwE2ynPbge3TUTx0sINJ1pNY5pmoXkMHkxTUjlMIU08YkNpYbfffkBy3X",
"mukG1fOjY3Rho/im+iPXzB6IWitdVK8+R31/ygW4HEXPO729Xc+6B/b31JvHfHH7YBuDOI9Pdbc+Foc4",
"h1FV3kIzSK1TzFIXmYOcWkPT5XpQdbluO8L+qMd9sMM6eZNMQk1DDwngeW5KIZK8zHsn30dSuR5am1gO",
"EixsslFLk9jOVZnaCiXZD2xd1maLCJ3JxNbZ9ai1du2pwMUMZcTVSDuEUdvnjPgPkok1BbmDfYIJlV/0",
"OF8laB+AJjdfyKHQSjeKFkB3DuarEuGiHNEbc8MsoZKLSe4zJgbz5OPhTc/7qNzri7zvTMf1aQ/hZHxp",
"gI5EfVrT9zx40voauiggRx49IvFa23YQqYdVLncbFS8XWHvA63P5UxHsXTT9kFC4Y8jt4wDyPac0XsTO",
"mDqXhf4vhcqaadp045Xr44fP5649a4cT58HzrqJW0VrZkP8EAD+ybXPufCaPZdqc20zeAwb9viZScWGc",
"3eBZYWdHxNxCy+SetroDL21qwCUwheyGjtE5Tmf2+99IdEOyG58VbXvgC36HSIb6AmSZwzUzB9nNWy0q",
"mxkGb17eHCXoxoxeelcDNUE3GVY4PPnT5fu/XDPzKrLQPkavAQs1Bqz0uZUbOGvOW6Cn38tj9EeQagCT",
"CRfGDUvMk3/945/XzNSVhgwVIAayHOudjkGgcTmZgEhQJngx4DQDqVwS9cXvj16YNOhX51fIweyaKY7G",
"OL2dkLgr/tLAtO2wanXhBAigQsCE3O/rsbFKVvViAwVrZ9jMtgrulQXHoKKg9glX/bCX58i9eAiz/9wT",
"kJ0T9S8vz4/2YY4qOmqtn64atmsu5INHyH8lGSn/XldHyBJ9xOujoq1DOcbq1Lp1HGDS4uy4mgGaYZZR",
"EMveiX6I4TM0eJTYGF7p/BRDX/wwuWaYZQiImoFAwIw13F0LoRpz34azukThI8RFLYLwmoXMQWd7Mz4Q",
"XwiiORNh6Ma3l7sJUX+nVHIE9+avPsjFRvQITsEEoNlwLDvd+7+8/Ru6wws7Ruotxq4C55E4r6cdf5Xu",
"w+V2cF/ahVhx3BpW8N4T1M9tiVKXQR6cWIcQsj4EAqpTnyPtBfrX//v/VZqqTWzQf3JUu1WIbS3+sBq7",
"2SFSo6WHM/l2wsch7GIBxK5t3O+Q66C52xm1lz2hiYShbQn5IFnfZ2bqx0flWeh6eQBXu5kLYeQP12Ho",
"xYnqdRd2zC/WZ7NoTzA+N48vAbK9jTlLooMprMRtrFwTen87ffcW1VpvrdZoZYpTPt3lVXtHbv3ikrwR",
"FpDU9hEm7yKHaIiicakv+IMcrj4hAEk9sd6NtDWtUtdC4OUffaf/lx/QELmSQD4Sr5EqtpAK8k7EY+z5",
"605V08Rzk7J2qbAInth+3Q979ALxnChjTLubaYHBehD6toVRW/Sd4HwnoX6Ng+jZBgdRYmqUUlNo0Uqs",
"ne313WuTSrUwtZ0mXOS91bC8pU6hv3DCvB9k5P6mxbeCFyU1El5osXrs+j0mXTbhPhPfQ6jJuNw5oeuu",
"HjJ+vWopG+FI8xDNzdO9GfKyHFtK1ZQ7J7LElPzd1XIzPVDR75DpgbqDeV8zXtXjtI3zfqQA6rVH64OB",
"tNmuNQJWO+CAscVmY65Vrp/WmvVNRzdEWKa3wsU+ZrxQaHsoAYu0HdKX5nHozdnNYPFrb1kJ2M8K8BXo",
"9o3upIfzv70m6hCK+o8lpQOTP2LRaYu8BiRXXuq+FwFkglzHzwaLhle2oqFPwfvRISarTku/NXS+Ne1n",
"K8AfomIHpUFuk0OPM2Qb3SLFozGqXdEY52bTaDbq2erO1EYzskVpN3jn3rlBnU6Wzm62Dvd8KJl7EFEl",
"Lqf5lpmR+ANcKl6LP2g2ULUtPnZLuN7FifeYR2ujR+3heNFOiyQcqGaiIVaTkTewc6I8UO6u13Bd0Ws7",
"Nd9XutgBUeQYSUNx27wP0yApwhIhWK77hI120LGO3LU4ki3XutLSaRNlNSAS/XRji11IMKDuMDc88uRi",
"zMq13C19mdeW6/9WW22NRP2aulOnjZhZDAIC4ib9ekgtwtKlMI/yUjnFIBjeNefgQTCH3s2AoSrCdsUa",
"Xk+kubLa5VecTKNX+JgJNZbY1xboefbkWQc6tEbyeqXPvY22SiswagYVJRvFxubs1wi6O7027TVRih1+",
"0tdxrNRPRNpxKX5bCDqt4e2vschQBhSUKQDPuEKyLAouTBX3makL77rpSgT3RNp6BaEfSEjhtzEFL59H",
"WKMWer4bZ3yR8HO9tEcMQW/jiFrhoUfiiFrBooD1KlRyH05wVYnXByJc+EHbyN57FNfcMaKgW/zDbymS",
"wLfRf7w4gkAaB4oiKCpS8/RMwXWS2yyJ+LcPWrMt3ukESTwBtUBzTOfgjt7LV384OkanocC3Ps6LurSz",
"Iupcftd2WF+EXvRf/qRukmRrqeVfSywwU6ZRVbQjz2rPq6rhf63ZVa2zlfEjhTExtfZxyywHhvsab4mr",
"KiEJe0ZCfVdvHtAQVbBFw+omOerOa0t3h4uzCclvJYVuxf0/lBRk7yuoS68Xcsg0CbOvA9eZR6JsqmaV",
"e3WHgKqzoFw1GhSEr70wcrf+JHLNIW0RJzxRIFbzuv3J97xVH2uA+ivVyOpr3EYnexQ2v7DxD04balAJ",
"6mclpoOIM3stzXRg64eugroflTywdvJvSh6GIhx7H5IwXFzlOmPkqRtzCUoRNn3cs765lgMe92F3hyi4",
"bheJpJsT9W8JpQN5R1Q6SxCDOYiBr7lqKtoc7XAlxGXaD5hIE+foF0EkqtMLhQz1nz15hn5XhUIeo7f8",
"DkzxI6Js+oNbOrqZUj7G9FhPN8KpOkHXPT6ZXPdutAaLMxtTabc08oPQLbgsCn/tkDyHjGAFdKG//uTo",
"xFxNNbDYUpxmHnSHXXwMZuuLjpjTJkaeu50bejv6EaYXDRpt8ww9nNz6dfLIqcGmTdhRgthQ7UcUksPx",
"6GndV6dsnJAvGmT2/scfNUsEgtzv/BRE3g5MwO8GafkDkbdnbtxjphT7ZRxSUCbyFnkYHEheFvU5tzwa",
"NXoaycj2kKRgVd/lgr1ZI826W3KOaSm5bcDL2mSdvWeqheFtZcheIsXvYhVsa24mYIfpOXTOMi3V1Kfu",
"S1DSloEZKW51FxPJQiS6hcLeCDOT17g42qUsR5sade5aVlofWqQQzapfEPVnBAQW6WwxwHdYwNELlGKR",
"EYapbds34SKFrE2RWk9zX4ciVV/j4zi3mgUQvkj/iQZFuoilneq/+J4D6y6FSzemc0YgHCzTPPTUZhPe",
"S3p3zlSU9FJBFEmjvcYfJA++SyOg35KZ3+L8Ea38nugOVbk40PB2rXhqPGJTaHB6+yD5M6fprYN5HOvr",
"d25fPVy7ktO0itDEDng7dippQC8vrXRzcPC9KxXU4HcIJ4Re66hkitCuFeBbe0Qud8SvZt6jieOXpQgN",
"4EAKrsTK1dXbQxCFAMnp/GHo4oOd+8Ck0Y7mFWR+FchzUKjwl2NWYkoXu6JPq6obhAY7pFtrTGt+GZmo",
"0/947x/yWtdYecxb3VLFoS51Mxvqm5QqFRLrChD20dFuLn077UO7HywqvmJfe0EYg2zkoBqvibjqbteY",
"aPW1f33edccQX71v3dCkafBE9K8eKTu60WsUPnRTdTjN/+pHfn0H2M4HUtjT/vhyU3nbj6ktaPG2/TG0",
"Z96/SceTXZOfrszorY+if7dkDrPNA5KOA9shGB1YhjDDdCGJK19Iqc/hMJXJI4lU2yR0PGQyld4KpKUx",
"3eipx4AFiNNSzXonP3/UGLfd2O2HS0F7J70hLshw/tTQg9vPatsml9zv8s5DXoEpOWtK4dRt581t2LSa",
"lTgU23kNQuu3pOptRaSts0w4S3ybo1rhKNfLaHXO8+1SHdx8vMq++BS3e5gtuuJCfYtq4wNqdOeMLijU",
"oKh6K7hOjUnwUkrUzyAlGQxxqmrTQr1E06eWuEuztCB66fOsNkM431bfrztgkqVQoyQ4x6qpnBdldaKQ",
"IelIw+UJV7a6Wo7jp2jqlUxsxrL5bkZU4qoPJihk43tMNbgsBu6CC7X6nqvk8Pnj5/8OAAD//2XM3suJ",
"7gAA",
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpClq7JlkI9f3Q5E1thM71lqafJuKXBTYfUhihAZ6ADQlxuWq",
"/NoH2MoT5km2cO1uEk02L7KcqfyxJTUaDZwLcO7nUy/lecEZMCV7p596M8AZCPPjxTWe6v8zkKkghSKc",
"9U57H0DyUqSA5iAk4QxNuEBvJoN3WKWzXtKT6QxyrN9TiwJ6pz2pBGHT3ufPn5NegQXOQbkPnJdCcrH6",
"ifcF/qUElJrHaCJ4jjAqBMwJLyUSIAvOJHwjEYMHNbLDekmP6Hd/KUEsekmP4Vx/PDxsX1bSu2CKqMWb",
"bHUlP/305iXiAklaTlEfjqfH6HbGpTqdlWNB5O2R/2yB1az6Ksl6SU/ALyURkPVOlSihywquaBkBuH3W",
"WMK9PM1xOsgJI7cJuqUP6WmKs2zRth797pYr+lHw/Jrotz9FAaux0gDrhIscq95pL8MKBkq/mkTmfZNB",
"XnAFLF38CRaruz2nBJgaTIGBwAoydAeLF0hAQfFConuiZoShZ9/PkABVCobUDBAXZEoYpoE0PBQsMVeL",
"rn18oL9eX3+OH94Cm6pZ7/S7Z/8ruvSJpfFVDF3jaUWmhAv06uL6Bfr+u2eIs8AnOZG545H44ioe2gZR",
"b0lOVBuWqHlYnyCDCS6p6p3+cJLoPZO8zHunz070b4TZ374LuydMwRSE+dA1X0cPim9PDZ/1Ti3GzHlw",
"CSwjbHpWFILPMdV/SjlTwMz+cFFQkmIN8+HPUgP+U+2D/1PApHfa+x/D6jgb2qdyGCY0n1yitxlmU0BS",
"4SlkLxBGOSg8wO4NdI8lSgUYSuxnJaYDvSLB6VHvc9K7FHxMIV+z0MKO+M12C/bzRtZ7IQQXqP/hx3P0",
"++9/+J1ZxhWZMkx/KjSss4NBzc4aW4P7EpJ+hMe8weLZFJg6SxWZE2UYvBC8AKGIRTJ2T0aWGj71gGma",
"+1tPcU5HKabUMACWnGkq0d9OiWagXtLL02Lk6Q5kiqnZV+/jCmklPaxXMSJZhGmSXsqFAPuyG8JKSvGY",
"gue4lVeyUtjxuVwzPvBL0gNzbHedvrHQ2iyEFaUayTLPsVh0momXattXJEi5BShkmaYg14FhzDkFzPRg",
"xe+AjVJeWnLcDDdDBvZQ6bAWK7R0vHuqY/Vv9opWslejlGSJNiuy4uOfIVX6e/WzaZWuLXutkps9QEZY",
"dV2spfps/TubadbNMe5GBvBQEAFyq2Vakgljy9LCdXnYHWFZndfhAdJSWaYuOCXpYpCag1j/jpUCwQYG",
"Ge0MXuAF5TgisxkRSeOGS8iQnR1lZDJpB1mFX0Hk3Sil2JL3Kuk7CW31gcKqlPUtFvYy01RlaAYyc5Yx",
"Yn6wsLZi4pzfQRbdoyztwtbKhFoCCvdVylkKgsnN5BHjBycnOlJuQMPhMOy0QS4NEl/HNh9KCluxDi4V",
"ZzxfjCjMgdbhq59U14DhB5iDiMLRncX+xmnC8h1eoDEgPJZK4FShPmEzEERJlPF7dtSF0TpywSbiSnkB",
"I7vW9SjXKlcBYmDHIj4HIUgGsstanTgau25iJBGnhSW0VLNuQv65oZOnJ4F9cbOemToBLQqqMiPqgimx",
"2A5EqeKi6/VtBy9LX+YW7CU9/UWsrMq8kAq8kpeVtAWyuwhToDChta1UEDiM2JSDmvFuUxhNuZPYY8we",
"I1J0G22OyVHKM+go9xxAkqkwGxg3TmWWEK9AKT3jCqndWc0cHnBe6DX3ppSPMT3WFDzCqYodbqVVCraS",
"HuaYlnF27H5K3Rk93s60/hw6n0F6t7rZlLMJidhd/oIpsXqOPmrd7RehV43WOhnWhN+O94LemphjOpJx",
"al6WnmZKFXqeVP+bEXmnL2AQamCuZA2OTJCJxlJhJRApZwO7tSgHUyzVaAaYqohx43pGJEo16L6RiN8z",
"lHOpkIAUmEJzEBlJ1TE6Y8hy7jcS2ZkQkUY0uedCKsQn+hcJCKeCS4n09epAZyeXCZIcKf0xItH9DCsE",
"DwXFhEn07f1s8S3C/hN6QAZTgTPIjtGfS0pRyRSh5nNmMjQh+qOiZPJYXxAebmZhBj7udf0jv9cndcnu",
"mP7pY4cr1MBLlMzRehNe/z0DZvdhl6IHIyvvhuX+FzIXlV6gXt9u0n2rLKqwmMIGobFPmFSYpTAwN1vW",
"ScyxE7eIUW52/RD19b9bzUxy4FprjTPAmtMg6f2dsy664hp51/F2jQ3rK6p4vMPx0ibfVIfMuhMkGOfa",
"tOnmSRGG//bkJHmCc6MDBW6iofWUEDb4XXR/HvPrMV1HciveLr1ldxe0bcJT5LJfR++fY4vUMiSZOEve",
"bvJz6u+/lSFje6LhjFgdlijI43Kw+wMWAi/iwt9BrB8dL05nKhgZNGXA0nUHASvzsYV+ZV2MIVZAyvMc",
"mLG+BLDuazkQvFSwrLwMrCxVU2BmnLaYAoyttbOJ7o7QzoMrbt3hCI2rOna3TTvuEqlstBlYT9A5Zwoe",
"VITijdluQijIkbUdRWxBl1jNpBY+zGikLz1RmhUj8yZSWtDwrydbEL4kjtqWpCWSg1Q4L4yOrgUSBg8K",
"FZxSpGEHUrVd+BGFo5CWtKftO7wWJSAyQcd69PEC5/o7KSk07GRtZzHLLKddQGfGDb89lqDK4ljOdodZ",
"7RpfMsFwxhVnJEWpRXdwmjmmTTZpAWsvZkNIV5AKsErWirIjV5f0hk2IJCmmSJoXkR6GsLF8kzEFpJy0",
"mprZt4DDqv4io8t+ieVszLHIripT/hILOMOKHHljY/SyMVIzATkaL0ZaLTVki7OM6K1ietmYc/X1Jeuq",
"FfOMQV9qoECGxgtk501a9Hn3cX/pH/jbTuFd/fRcnxDCbXhpKv3MzzQu0ztQdrIfBjlhpQLkr/CkofDo",
"i7KO6yZC7ETdr7ngFtlA3W5e/0KMWpZ5LVjYD4R2P10c84lVc559P4sholItm+AKWlh8AVodiz7xelz0",
"oVSYRhBu1sfHUuPU7IPTDPQJjVmlN34jtdIJqaaFFFuBIiZ4emVxM+Y6qpwrILPXvcWdVhWd2XFX9PEC",
"GOozzgYCJKdzyI6c93YVn/5zK6ta2toKZ8dOmoD8+JaSyBkWp90GO8cgdsEEp/SDu2NXaG3GpfJexSZs",
"zlJVYor8AGeqQGDmI2yKcpzOCIsycA5y5iyCzUmvbIyUfo7eXBphQAuo5vyaW8OSFZtalapNQUD38pTB",
"/YDiQvHiaKOV0Ey7Dm4uciYmZ40KQeZYweguFrFz9upicHVx/uHievCni78Ojo+PzXYp15dnBqlYFG17",
"NXOXY0rS+NR4Ct/p+ewYTaNm6qv3l1c1KSduUnPX98jez04Wbrvjf2JESxCYnpVq5q509OZlp5mtfLD1",
"7O61GFFZeht5ghkZH/q6D7g3KhKzcgqyL24ijSUsJCsoj4OzHRRxMlPxcBClBBmXqnGOVa/tojy2WTJr",
"VwBkzkqZoHtrr4OacTHnjChu3bPb2A795fPxcC5Do5ibuylqbDSLd/bWeyxRY4s7WxZzrC8ShlkKI2NZ",
"3T0EwR+5rcdc5WOoxVX2WhwqHWNC2mxLO7kotnOXOmOTuw/N7qs5GuTcWE47x7Q6TBt80+a5wFOsdV5D",
"3/oD30gUXhy5MLDIpzdirR07zZW8tAY8abUlQJRMIF2kVC/EGfdGS6rDKh6XdMVSKuOvR1qcCV57qMzP",
"3e7CJpLaEXAZj/k8U4iC4TYWRIYJAZpJlLsVFgIkMHXcS7bA3TsQJhBxvoLDLoj7IpwbR/W1sSJVGLaK",
"AeorgZk0Qmu1p7i40oKAa0cGLTAc1WNb6wv649X7P6MrD6qN9rvGy5FtZ1wDN/qIyJGnw7g5mOIFiLrx",
"LweF7QUqsDVJlUJjZcrnIAz6jLI3ZaQ1/ikAuquZrxWhBRb68vbsttm4aGA6WuuUWY2HMuFcYO7PQkBq",
"QlU/7nXgutPVIcZDuYmOsJKPa+lr4yk7WgnBfgzKCf6OCaYy6gBaoaRDktDOJLP+uI3jaT1CWvwxB8HH",
"rrQZPaLmLs562W+0fUwMVvgRI2LqVoQa7fBe0rvHwpvoBVFano97IIxKGzdwyu4CVQg8CoKfNQwcC0wk",
"ZFuEu7j7u2ZMcEuMklaION3Kd1aLIt8cWOVMGV3Hpw2fXue3uAab2jM0+JH8dltHyXd29AmcR6SlqztC",
"KbJPUX9VZrJP3GFxFJOYBEhz4u7v4XtED50dXLsZNwN2naQu9qWeSLSzi7duhjvbeLG14c/O1meOH83H",
"k0XtZzt4gomNvtDLy0a8NJZwfcNR+3fB9Q+jMU7v3G/6x5F77+M+Ls/aQiKS3dYx1CF4eltnaDi+Wg2c",
"1SlWnawCDLbXc1SEJbBsuzrrNL7EitacbJyDOTeh+5BZF1mfF9ZofdRLtg5Xqqdgbrwc3Fxrox9fCVzM",
"/kLgfhWGkE2hGQCxLkHqg0OgnJEi5oKp7FBtZvsdjUuReNqIm4xkaHBTnpw8h2DrchqptXkRltIyg/+y",
"NGnMR85DDdFIR8azLYDj7H0RsChRstRnqsWd2fpTKMWFWdSMqIj/elnG5Db83WIwhvbXBgatvlTvumhs",
"cMmFEPB5KLytWPcOHQkooBkBRnJ9Fvu8Or1kF+ilzAe2C4PsJv03jS0Ohl08mLINVf9xG7a4DZch7wBY",
"+eWicP8T4/dU881rErlWutqpCbuDbBTloo1xIQKzu06BW+1CDSNFEbtEXpPpjJLpTKPGZF/7CJNOfGUj",
"/jtnCCiiKKzZccWHGU/L3IaNiJKNOb8z1qA5SEWmbTlvm+3NdgExJL/1qv5LfWSvclTdFBs1VGTttsAt",
"sa1A5ERLETu9HKyJEW3g00Tw/BR9UvwUfXKgkqfobwznkA0Mqybo+Pj44+fPnze6t4lPdjP3yoqxuraO",
"GLzfgRIkvQLhIBy5bBZtildu3m2JIqS0LOqUJPB9L+l9N9P/tEQOGmlw3cWG59NO7LdFFm+OHzpNmRPW",
"adw2FoaQVbJUuATfIwsL5LNFNnx2WbiUne6tcOmuE4+uzaBoSMXCKgSOCgLOK0TGFnFpI2G3M24UBSUg",
"22Oxm2G1S2kOhErOEOX3INCYlyxLtMRW2CASmNv3bOL38IdeBKXNMfFbWatwpVg7ZBtHajAQ7CVuFRWs",
"V579UmKBmSKsLTJ8iwzi2aLgagaS/N3mHvjF+0T1JYOluUDCmI/tifvroLmbu7NBSTXd10OqQUormK+p",
"xevCOGvFNpZvr6VswlrWpxBcRG6KHwnQbGDyMGvhOMgOR/3vnz07ao/yM26++PHcpjkvwc7OEMZ3OVV8",
"vs4G2omlGmwSSoLXoYfHvFSnY6rlsVrwQCnIZs3bfGatu+VSqx5yrQ1jnVP73ZuXCUq5AJkggfNRPk5Q",
"RuTdaDpOECkSpCAvqAlGzE1MW4K02E5SkNGgRC4j8uIVLafenXsp+EPOH0xkmAu6qsUoRG0Z8Qiz12WO",
"2UAAzvS5gpw/pGPkl/kufUhPfwZKFxPCtneV04c0QfM8QVygjKd3IEwVG0xYPbS6u7PcAW8DjtsCyqok",
"6m72gxANGDU7BcMYevPSRBkInN6hwi+DsKn+ZSrA2N82XBPR67i3tIS1274KnLi0aX2yRCqdzUFgSu3B",
"g8ikufBg+NzdAtDirD8vhQAWoiZaQzCkgmKd5GjWPcpBSjzt5jyeEEbkbH/782PYsEMAqiiZ84gZxSzg",
"Qd5pNbPlclVQdLCG6FFrT8m1yQKOGT2+LHpiszRskxvO2Y2Oj8rzFz8sXUHAwC6dTbzhtLWxI2smaBNR",
"zeU9MsXottEPSDZSfFfaWcaJhU5SGZ/dWVlb2yYUdYvy6oyY9TbzdnxsfK+bxS8OkE0wiMf5pFhkhGG6",
"5LrmDAaKD7gJy3a/5Jhp4tH/Vc/8b+bhRtt5zPLBtFAK+4XYOEtSpwI0vW0zrze+H4/HqK+p+YWkAfUo",
"3oi8O/ee0HhS0qj6ZIU25hBWFRnxP9pEN5G3nK4hiRVTrXK2KFebcNmCnzh8VjcSWUYMOK5w3iopG0N/",
"R5ev15E6XqpCqpEEYFt56ycUF+uUwRmn2Sjj92zfWMJtq4RVoSFWgB8403dcq99635TcAV2MUlx25Ou8",
"VHvHU/I0NULXeoPHAcJ0NoqClenQBdzg9M77ALxxwXxHb9uqqTZnqBKFPu5yyVcx9jZx8hClynxRsloM",
"kJONVuC9zCZL1BPl5DtC6V42tc2BOCaPdlRZDjq+0bmq3zb2sVLuKVSvCRi0mfok29LgXwieQlaKmPjp",
"ox4zZAThoY0fGfoAkKGPCyooZujD88Hvj1ZCsb3fbhQil9pq8DB9Dnp7ZL7iC9+8kXooUjzuwq27s4/d",
"kOeV1ihi7tMV1W3nqSxcDzFXRBuSIfAnngjZ3V6aCTxxsQs+iCHYSQVMjE22psxtiEP2xlKxPu2ksmfv",
"ZjpdSREJxtNgFK1YoPWMunLK53IUap5jFjGavOLBWGaKB7o4uV68KKirQLnMOESF6mYxd3rGSz3AmJlk",
"XOjqnnciIF4ST+ltqNYUWIoX3UusaKW/GVst5ayX+Ko5Jh+ctVy6bXffFoCOF7757cnG0gdu3UlAd4xK",
"rr1TagmAjOeYLlqkaSKgCirbIYJkVeDkmuOidldiHHOUMMACFYL/bD+doN9lyEb8bfYltvtNJeUxzemt",
"/dyEKFSAQBlebO0UDG66ClrRwAwJaaklFJOQ4qoFABYgzspYruJ7pxYN5wTuQZwiPUxLT3fo/ZuX5+iP",
"/31dj3clbHB2+Qb96x//ROc4yxY3bMLFPRbZAJe2FFsGE2ASBoQNMijULEGM27wwZ73REpoo1ezo+IaZ",
"Et6nxixIUmTXaZNJbZn7KvO0b4p8oVsTKH2r3/Vl4A0xmTcrajesZAqKG5nWVSp3yQ+ukHymuOCrQW3n",
"tuz6QN/lgPRmfYGV9+SOSzTjOVA8Ru+vjpEpWTchFHwNum+/DZu8YWaX336L+qaSO07VwMiFR6foFTce",
"AxBIqnIsERaAqkYE90TNEMcFGehjbwosuWE271Wivv/8+ds3CZqUWipBP72RRxZeBsw4ByQLSI9v2A07",
"52yu0clZTT55fnR6wwbownqh9Nd9mXd021ZU/vZYv/KWSCVRKQHdfjJ3dFLvjfH51i7eNdQo8JQw6/Dq",
"u4MGmUYB6IeTBOX4AT07OTky8/7EJJ4Aunx/dW2LnxQK3S51UbhFfduPoaB4ge4Jy/i9fftdaQ4FJFzL",
"EIlSLMQC3brb7vYFenVx7To5SHR7cY2ntwm6PLs+f418/Aa69Y0RblHftVTwrRTsZ0LNnQpmz58//z36",
"6frcPL9wQUnmKc4yAVKadY2b0aWo3+ztYRB1PQP07vzSlgOZ4BRQXyoBODczvL6+vkwQn0xISjDVBHT1",
"8k9HNoe4ZCYSXaHbYZ4WtzeMs4oQxoRhsUCYZXowL02FRMtLlq41y7ogoRemWqIpw4PuBS5uWEVPVj9G",
"JqUGYemKLALLCk6YkpYfKUnBuWIck13a7G59XAvqGFOeDodO8z52nuShywKv+RF7lt3OLt/UpJbT3nfH",
"J8cnRs0tgOGC9E57z49Pjp9bJ/DMnHdDc0gMcK01gLs0rRGIcPYm6532/ncJYtHsItDsHPO3eAuKWiH3",
"NQ0zWt5tVH7fYYJ66Mbal2OSc7W5Yei70mGs68jRYaTrtNNhpG0n8vnjUmuOZycnWzWWWAoi9GpDJ/2h",
"ifqIPlJv+7N91TKzhMgdvXLl+CUgYMrEcX1OKsEsvoUAs1oLj1okq+2NgcYww3NiamQYMzueSmPTtmGm",
"Y+LNrg8Dv3BbS7N32rPigJl1GEqntHKSvhbOwqhOTBS0jgqXh6to38Y83oqz8sm9+wT86ngjtLJ5OrYI",
"BLU/P2gCRbhGoZ4XqhJA2zDC8BPJPg9DwxgN6ybFb0BwaASmcVy4AJEmS700PTUCGpItv7DU/srSkomG",
"+QPPFnuQUX3TIa3Vsqnl0kXgzKhqxnhL5G9L8ZnX787Oq7YTVjfoS8KmFAalhAT5vCknUg8kyWBzmaKw",
"jTglNhtjfd6TEXftGfXSLRIJSLnIIDvEzWBxZUJ0gC3qsHTQbQVoV5YJXjfLNGVm4/3XyGBmSDfZq156",
"fhfpy/YveEzBq03s42yXN1cS+P4j9O13sVWdL57watOL8OIe6iuJXl5cnR8dgr3NzNvLe02etZXy14p7",
"53ZIJ6ZdEbs60n6I69iBWX0l9ZVXa5l9vy7Ktm0nno6oHUUcSFizTQ0ymBDm0l8qgnYVHjdIbG2SlY2B",
"stB6QrFqIypdrFYneeS7w346il6bOH4A/NqZEHY47tvTZoDlIMMKJ8ibKX931BnnsePLCOn7yua+PEyT",
"hEzVmB0pyDV3fVTSsVVtvrAk20o5vmHo/pRjZzKyK7GmVUdEuxJKozLKhgtvaWw3K0eoY/ClJU5fGX/V",
"1rFVQ4Bf3SXZ7G/xhLflEjkd4Fh1M4LW7KziqGXLGSAXTchLOfBPkFHLkBKY0KNd7SHOKzW0JYwNaqLJ",
"Lr5OpPR1ipPg7pIIM4SngO5gUWAiEtcF2fy9vfBsYjwateTYRtQXb6Q3HKNzTCkIWy8RUwE4W6AZnkO9",
"1RMz1w6DTJ8ujewIE+hlHRzNU8FWND73dfkf4zRvFpv+wgf6UsXmWJNoMyIHpkJLdOsBtE0MWBYQdgD6",
"th9DGDG498WN//WPfyIiZQmehjz91GgnLKGicke4LSRuWxQ2KPyTpOX08zCtmoREwzA+OA/j/YykM9cL",
"xPT/SKxbzZKtKStt+2349hbItPkwRDwlc2BIeV+j8TIz5B3ApsGH7V7GpAKcIT5BU6JQUVIaI9JXoJoN",
"TlburdgWEGd04RYnw+KIrNZlW4M/f/7cBLvF7z6bhblls/aPjymiNCARO5VdW5AMqMIHoNlXoBwZpPWZ",
"HURxBc5tiTPZSaq9ouW09/ljhLJl1bVkurbI+MAEmBjroHnDOpMzX3jXTvuNXDmx2wnTN0x5dLz7D0Xw",
"ftWp98qBVFsPuXVNXr4sMWS+BcywVgwnKgm/ArXSL+YREbfyrZiV3I9BfvH74+k9g4HgJcsGSpDChNRp",
"wSfEAqU2SggJznMTEoQKPIU9fKz1gjatKogPMNl0hv9IqAJh6iPUmyW6QlwS6dHAMsyUbDu8d7Wwh4zB",
"bV8MVVu3ftOX41374hK7l2P70NXq4cwE5gxdkGzsK7/saXr/t9KS2ouqfSHtyNXko0Qe7NSFinmCslMr",
"V7WzIfHCX29fpSWxUdz/C5sSPRW12xJNrbQMbOWMi2s8bZvSDRuaMW7Cg9ggGVpRDjZQRdOC5AcPg8rY",
"rgafO8223gSkUjtdf6t50JVTzqTWz03cp61KoTXnFBc4NSqwj/g+SpA3ZLnZrbuxqk5rIloiOnNTUdbH",
"oN9dcLnHdIpQheDrpv2VOiBfmP5Xa1S0n3SuLGvSRMgvJZRPyiZhCwgj/VqpAun23/6f8wT95V2CQo2P",
"I2QGmqId+/KTN963SaGB9B7R/NF2fDmcuXpAT4edV6GWwXKI8U6X3AHcJMtpD75HR/3UwQIiXUdqnWOq",
"dgEZTF7cMEIpTDFtTGJjudH3J7/Xcq2ZblA9PzpGlzaKb6o/csPsgai10kX16nPU96dcgMtR9LzT29v1",
"rHtkf0+9ecwXtw+2MYjz+FR361NxiHMYVeUtNIPUOsUsdZE5yKk1NF2uB1WX67Yj7A963Ac7rJM3ySTU",
"NPSQAJ7nphQiycu8d/pDJJXrsbWJ5SDBwiYbtTSJ7VyVqa1Qkv3A1mVttojQmUxsnV2PWmvXngpczFBG",
"XI20Qxi1fc6I/yCZWFOQO9gnmFD5RY/zVYL2AWhy84UcCq10o2gBdOdgvioRLsoRvTE3zBIquZjkPmNi",
"ME8+Ht70vI/Kvb7I+850XJ/2EE7GlwboSNSnNX3Pgyetr6GLAnLk0RMSr7VtB5F6WOVyt1HxcoG1R7w+",
"lz8Vwd5l0w8JhTuG3D4OIN9zSuNF7Iypc1no/1KorJmmTTdeuT5++GLu2rN2OHEePe8qahWtlQ35TwDw",
"E9s2585n8lSmzbnN5D1g0O9rIhUXxtkNnhV2dkTMLbRM7mmrO/DKpgZcAVPIbugYXeB0Zr//jUS3JLv1",
"WdG2B77g94hkqC9AljncMHOQ3b7VorKZYfDm5e1Rgm7N6KV3NVATdJthhcOTP169//MNM68iC+1j9Bqw",
"UGPASp9buYGz5rwF+u4HeYz+AFINYDLhwrhhiXnyr3/884aZutKQoQLEQJZjvdMxCDQuJxMQCcoELwac",
"ZiCVS6K+/O3RC5MG/eriGjmY3TDF0RindxMSd8VfGZi2HVatLpwAAVQImJCHfT02VsmqXmygYO0Mm9lW",
"wYOy4BhUFNQ+4aof9uoCuRcPYfafewKyc6L+1dXF0T7MUUVHrfXTVcN2zYV89Aj5ryQj5d/r6ghZok94",
"fVS0dSjHWJ1at44DTFqcHdczQDPMMgpi2TvRDzF8hgaPEhvDK52fYuiLHyY3DLMMAVEzEAiYsYa7ayFU",
"Y+7bcFaXKHyEuKhFEN6wkDnobG/GB+ILQTRnIgzd+vZytyHq74xKjuDB/NUHudiIHsEpmAA0G45lp3v/",
"57d/Rfd4YcdIvcXYVeA8Ehf1tOOv0n243A7uS7sQK45bwwree4L6uS1R6jLIgxPrEELWh0BAdepzpL1A",
"//q//69KU7WJDfpPjmq3CrGtxR9WYzc7RGq09Hgm3074OIRdLIDYtY37DXIdNHc7o/ayJzSRMLQtIR8l",
"6/vcTP30qDwPXS8P4Go3cyGM/OE6DL04Ub3uwo75xfpsFu0Jxhfm8RVAtrcxZ0l0MIWVuI2Va0Lvr2fv",
"3qJa663VGq1Mccqnu7xq78itX1ySN8ICkto+wuRd5BANUTQu9QV/kMPVJwQgqSfWu5G2plXqWgi8/IPv",
"9P/yAxoiVxLIR+I1UsUWUkHeiXiMPX/dqWqaeG5S1q4UFsET26/7YY9eIJ4TZYxp9zMtMFgPQt+2MGqL",
"vhOc7yTUr3EQPdvgIEpMjVJqCi1aibWzvb57bVKpFqa204SLvLcalrfUKfRnTpj3g4zc37T4VvCipEbC",
"Cy1Wj12/x6TLJtxn4nsINRmXOyd03dVjxq9XLWUjHGkeorl5ujdDXpVjS6macudElpiSv7tabqYHKvoN",
"Mj1QdzDva8arepy2cd6PFEC99mh9NJA227VGwGoHHDC22GzMtcr101qzvunohgjL9Fa42MeMFwptDyVg",
"kbZD+so8Dr05uxksfuktKwH7WQG+At2+0Z30cP6310QdQlH/saR0YPJHLDptkdeA5MpL3fcigEyQ6/jZ",
"YNHwylY09Cl4PzrEZNVp6deGzrem/WwF+ENU7KA0yG1y6HGGbKNbpHg0RrUrGuPcbBrNRj1b3ZnaaEa2",
"KO0G79w7N6jTydLZzdbhng8lcw8iqsTlNN8yMxJ/gEvFa/EHzQaqtsXHbgnXuzjxnvJobfSoPRwv2mmR",
"hAPVTDTEajLyBnZOlAfK3fUarit6bafm+0oXOyCKHCNpKG6b92EaJEVYIgTLdZ+w0Q461pG7Fkey5VpX",
"WjptoqwGRKKfbmyxCwkG1B3mhkeeXIxZuZa7pS/z2nL932qrrZGoX1N36rQRM4tBQEDcpF8PqUVYuhTm",
"UV4qpxgEw7vmHDwI5tD7GTBURdiuWMPriTTXVrv8ipNp9AqfMqHGEvvaAj3PTp51oENrJK9X+tzbaKu0",
"AqNmUFGyUWxszn6NoLvTa9NeE6XY4Sd9HcdK/USkHZfit4Wg0xre/hqLDGVAQZkC8IwrJMui4MJUcZ+Z",
"uvCum65E8ECkrVcQ+oGEFH4bU/DyeYQ1aqHnu3HGFwk/10t7whD0No6oFR56Io6oFSwKWK9CJffhBFeV",
"eH0gwqUftI3svUdxzR0jCrrFP/yaIgl8G/2niyMIpHGgKIKiIjVPzxRcJ7nNkoh/+6A12+KdTpDEE1AL",
"NMd0Du7ovXr1u6NjdBYKfOvjvKhLOyuiztX3bYf1ZehF/+VP6iZJtpZa/qXEAjNlGlVFO/Ks9ryqGv7X",
"ml3VOlsZP1IYE1Nrn7bMcmC4r/GWuK4SkrBnJNR39eYBDVEFWzSsbpKj7ry2dHe4OJuQ/FZS6Fbc/0NJ",
"Qfa+grr0eiGHTJMw+zpwnXkkyqZqVrlXdwioOg/KVaNBQfjaCyN3608i1xzSFnHCEwViNa/bn3zPW/Wx",
"Bqi/Uo2svsZtdLInYfNLG//gtKEGlaB+VmI6iDiz19JMB7Z+7Cqo+1HJI2sn/6bkYSjCsfchCcPFVa4z",
"Rp65MVegFGHTpz3rm2s54HEfdneIgut2kUi6OVH/jlA6kPdEpbMEMZiDGPiaq6aizdEOV0Jcpv2AiTRx",
"jn4RRKI6vVDIUP/ZyTP0myoU8hi95fdgih8RZdMf3NLR7ZTyMabHeroRTtUpuunxyeSmd6s1WJzZmEq7",
"pZEfhO7AZVH4a4fkOWQEK6AL/fWTo1NzNdXAYktxmnnQPXbxMZitLzpiTpsYee52bujt6EeYXjZotM0z",
"9Hhy69fJI2cGmzZhRwliQ7WfUEgOx6OndV+dsnFCvmiQ2fsff9QsEQhyv/NTEHk3MAG/G6TlD0Tenbtx",
"T5lS7JdxSEGZyDvkYXAgeVnU59zyaNToaSQj20OSglV9lwv2Zo00627JOaal5LYBL2uTdfaeqRaGt5Uh",
"e4kUv49VsK25mYAdpufQBcu0VFOfui9BSVsGZqS41V1MJAuR6A4KeyPMTF7j4miXshxtatSFa1lpfWiR",
"QjSrfkHUnxEQWKSzxQDfYwFHL1CKRUYYprZt34SLFLI2RWo9zX0dilR9jU/j3GoWQPgi/ScaFOkilnaq",
"/+J7Dqy7FK7cmM4ZgXCwTPPQU5tNeC/p3TtTUdJLBVEkjfYaf5Q8+C6NgH5NZn6L8ye08nuiO1Tl4kDD",
"27XiqfGITaHB6d2j5M+cpXcO5nGsr9+5ffVw7UrO0ipCEzvg7dippAG9vLTSzcHB965UUIPfIZwQeq2j",
"kilCu1aAb+0RudwRv5p5jyaOX5YiNIADKbgSK9fXbw9BFAIkp/PHoYsPdu4Dk0Y7mleQ+VUgz0Ghwl+O",
"WYkpXeyKPq2qbhAa7JBurTGt+WVkok7/471/zGtdY+Upb3VLFYe61M1sqG9SqlRIrCtA2EdHu7n07bSP",
"7X6wqPiKfe0FYQyykYNqvCbiqrtdY6LV1/71edcdQ3z1vnVDk6bBE9G/eqTs6EavUfjQTdXhNP+LH/n1",
"HWA7H0hhT/vjy03lbT+mtqDF2/bH0J55/yYdT3ZNfro2o7c+iv7dkjnMNg9IOg5sh2B0YBnCDNOFJK58",
"IaU+h8NUJo8kUm2T0PGYyVR6K5CWxnSjpx4DFiDOSjXrnf7to8a47cZuP1wK2jvtDXFBhvPvDD24/ay2",
"bXLJ/S7vPOQVmJKzphRO3Xbe3IZNq1mJQ7Gd1yC0fkuq3lZE2jrLhLPEtzmqFY5yvYxW57zYLtXBzcer",
"7ItPcbuH2aIrLtS3qDY+oEZ3zuiCQg2KqreC69SYBC+lRP0MUpLBEKeqNi3USzR9aom7NEsLopc+z2oz",
"hPNt9f26AyZZCjVKgnOsmsp5UVYnChmSjjRcnnBlq6vlOH6Kpl7JxGYsm+9mRCWu+mCCQja+x1SDy2Lg",
"LrhQq++5Sg6fP37+/wEAAP//VvsxiT/wAAA=",
}
// GetSwagger returns the content of the embedded swagger specification file

View File

@@ -5,6 +5,7 @@ import (
"crypto/rand"
"crypto/sha256"
"encoding/json"
"errors"
"fmt"
"math/big"
"strconv"
@@ -24,7 +25,12 @@ import (
const (
defaultLimit = 50
maxLimit = 200
graphNodeCap = 500
// graphNodeCap bounds the whole-graph view. The cognition transactional
// types (execution, task) are audit records, not topology, and previously
// crowded out every host/lxc/service; the default whole-graph view below
// excludes them so the cap is spent on the actual fleet graph. Operators
// still reach executions/tasks via list_entities.
graphNodeCap = 2000
)
// actorInfo returns the caller's (type, label) from the request context,
@@ -307,14 +313,19 @@ func (s *Server) GetGraph(ctx context.Context, req gen.GetGraphRequestObject) (g
// alphabetically. Without this the cap fills with exec:* rows and
// drops every host/lxc/service/vm — and every edge those entities
// connect — because edges require both endpoints in the node set.
// Exclude the cognition transactional types (execution/task): they
// are audit records rather than topology, and at ~380 rows they
// consumed most of the old 500-node cap.
nodes, err = s.queryEntities(ctx, `
SELECT `+entityCols+`
FROM entities e
LEFT JOIN entity_status st ON st.entity_id = e.id
WHERE e.id IN (
WHERE e.type NOT IN ('execution','task')
AND e.id IN (
SELECT e2.id FROM entities e2
LEFT JOIN relationships r ON r.valid_to IS NULL
AND (r.source_id = e2.id OR r.target_id = e2.id)
WHERE e2.type NOT IN ('execution','task')
GROUP BY e2.id
ORDER BY count(r.type) DESC, e2.slug
LIMIT $1
@@ -807,7 +818,7 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
rows, err := s.pool.Query(ctx, `
SELECT id, ts, actor_type, actor_id::text, action, entity_id::text,
method, path, status_code, detail, source_ip, correlation_id
method, path, status_code, detail, source_ip, correlation_id, session_id::text
FROM audit_log
WHERE ($1::text IS NULL OR actor_type = $1)
AND ($2::text IS NULL OR actor_id::text = $2)
@@ -828,10 +839,10 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
for rows.Next() {
var a gen.AuditEntry
var detailBytes []byte
var actID, entID, method, path, sourceIP, corrID *string
var actID, entID, method, path, sourceIP, corrID, sessionID *string
var statusCode *int
if err := rows.Scan(&a.Id, &a.Ts, &a.ActorType, &actID, &a.Action, &entID,
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID); err != nil {
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID, &sessionID); err != nil {
return nil, err
}
a.ActorId = actID
@@ -989,6 +1000,7 @@ func (s *Server) CreateEntity(ctx context.Context, req gen.CreateEntityRequestOb
entityID := inserted.ID
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
&entityID, "POST", "/api/v1/entities", "",
nil,
map[string]any{"type": req.Body.Type, "slug": slug}); auditErr != nil {
return nil, auditErr
}
@@ -1052,49 +1064,15 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
// Validate lifecycle transition if state is being changed.
if req.Body.State != nil && *req.Body.State != "" {
// Get lifecycle def for the entity's type.
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, current.Type)
if err != nil {
if err == pgx.ErrNoRows {
// No lifecycle defined — any state is allowed.
} else {
return nil, err
}
} else {
var transitions map[string]map[string]json.RawMessage
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
return nil, fmt.Errorf("parse lifecycle transitions: %w", err)
}
fromState := ""
if current.State != nil {
fromState = *current.State
}
toState := *req.Body.State
if toState != fromState {
tos, ok := transitions[fromState]
if !ok {
return nil, fmt.Errorf("%w: no transitions from %q", domain.ErrInvalidTransition, fromState)
}
trans, ok := tos[toState]
if !ok {
return nil, fmt.Errorf("%w: %s → %s", domain.ErrInvalidTransition, fromState, toState)
}
// Parse preconditions: {"requires": ["check-name", ...]}
var gate struct {
Requires []string `json:"requires"`
}
if err := json.Unmarshal(trans, &gate); err == nil && len(gate.Requires) > 0 {
for _, check := range gate.Requires {
if err := checkPrecondition(ctx, tx, id, current.Type, check); err != nil {
return nil, fmt.Errorf("%w: precondition %q not met: %v",
domain.ErrInvalidTransition, check, err)
}
}
}
fromState := ""
if current.State != nil {
fromState = *current.State
}
if err := db.ValidateTransition(ctx, tx, id, current.Type, fromState, *req.Body.State); err != nil {
if errors.Is(err, db.ErrTransitionInvalid) {
return nil, fmt.Errorf("%w: %v", domain.ErrInvalidTransition, err)
}
return nil, err
}
}
@@ -1133,6 +1111,7 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
patchActorType, patchActor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, q, patchActorType, patchActor, "patch",
&id, "PATCH", "/api/v1/entities/"+req.Id, "",
nil,
map[string]any{"version": expectedVersion}); auditErr != nil {
return nil, auditErr
}
@@ -1262,6 +1241,7 @@ func (s *Server) EnrollClient(ctx context.Context, req gen.EnrollClientRequestOb
entityID := id
_ = observability.Audit(ctx, q, "operator", actor, "enroll",
&entityID, "POST", "/api/v1/clients/enroll", "",
nil,
map[string]any{"slug": req.Body.Slug, "mesh_ip": meshIP})
_ = observability.Event(ctx, q, "client.enrolled", &entityID,
"info", "oikos-api", "",
@@ -1456,6 +1436,7 @@ func (s *Server) ProvisionEntity(ctx context.Context, req gen.ProvisionEntityReq
_, actor := actorInfo(ctx)
_ = observability.Audit(ctx, q, "operator", actor, "provision",
&entityID, "POST", "/api/v1/entities/provision", "",
nil,
map[string]any{"slug": req.Body.Slug, "host": hostSlug})
_ = observability.Event(ctx, q, "entity.provisioned", &entityID,
"info", "oikos-api", "",
@@ -1546,97 +1527,5 @@ func generateAgeKeypair() (pubKey, privKey string, err error) {
return pub, priv, nil
}
// checkPrecondition validates a named lifecycle transition precondition.
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
switch check {
case "no-inbound-edges":
var count int
err := tx.QueryRow(ctx,
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count)
if err != nil {
return err
}
if count > 0 {
return fmt.Errorf("%d inbound relationship edges remaining", count)
}
case "backups-verified":
var attrs string
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
if err != nil {
return err
}
if !strings.Contains(attrs, "backups_verified") {
return fmt.Errorf("backup verification not recorded in entity attributes")
}
case "secrets-revoked":
var attrs string
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
if err != nil {
return err
}
if !strings.Contains(attrs, "secrets_revoked") {
return fmt.Errorf("secret revocation not recorded in entity attributes")
}
case "ingress-dns-removed":
var attrs string
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
if err != nil {
return err
}
if !strings.Contains(attrs, "ingress_dns_removed") {
return fmt.Errorf("ingress/DNS removal not recorded in entity attributes")
}
case "age-key-enrolled-if-needed":
if entityType == "workstation" {
var attrs string
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
if err != nil {
return err
}
if !strings.Contains(attrs, "age_pubkey") {
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
}
}
case "mesh-joined-if-needed":
if entityType == "workstation" {
var attrs string
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
if err != nil {
return err
}
if !strings.Contains(attrs, "mesh_ip") {
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
}
}
case "health-check-answering":
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
if err != nil || st.Health == "unknown" || st.Health == "down" {
return fmt.Errorf("health check not answering (status: %s)", st.Health)
}
case "doc-page-complete":
var count int
err := tx.QueryRow(ctx, `
SELECT count(*) FROM relationships r
JOIN entities ke ON ke.id = r.source_id
WHERE r.target_id = $1 AND r.valid_to IS NULL
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
entityID).Scan(&count)
if err != nil {
return err
}
if count == 0 {
return fmt.Errorf("no documentation linked to entity")
}
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
"ingress-live-if-public", "doc-page-stub":
// Soft checks — always pass. These are operator-confirmed via the
// transition request itself, or are not mechanically enforceable.
default:
// Unknown preconditions are skipped (operator intent overrides).
}
return nil
}
// ─── Helpers ───────────────────────────────────────────────────────────

View File

@@ -155,14 +155,14 @@ func (s *Server) GetTrends(ctx context.Context, req gen.GetTrendsRequestObject)
f, _ := slopeNum.Float64Value()
t.Slope = float32Ptr(float32(f.Float64))
if f.Float64 > 0.01 {
t.Direction = gen.Improving
t.Direction = gen.TrendDirectionImproving
} else if f.Float64 < -0.01 {
t.Direction = gen.Degrading
t.Direction = gen.TrendDirectionDegrading
} else {
t.Direction = gen.Stable
t.Direction = gen.TrendDirectionStable
}
} else {
t.Direction = gen.Unknown
t.Direction = gen.TrendDirectionUnknown
}
items = append(items, t)
}

View File

@@ -111,6 +111,7 @@ func (s *Server) PatchPattern(ctx context.Context, req gen.PatchPatternRequestOb
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
&id, "PATCH", "/api/v1/patterns/"+req.Id, "",
nil,
map[string]any{"status": req.Body.Status, "quarantined": req.Body.Quarantined}); auditErr != nil {
return nil, auditErr
}

View File

@@ -65,6 +65,7 @@ func (s *Server) CreateRelationship(ctx context.Context, req gen.CreateRelations
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
nil, "POST", "/api/v1/relationships", "",
nil,
map[string]any{"source": req.Body.Source, "target": req.Body.Target, "type": req.Body.Type}); auditErr != nil {
return nil, auditErr
}
@@ -108,6 +109,7 @@ func (s *Server) EndRelationship(ctx context.Context, req gen.EndRelationshipReq
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "delete",
nil, "DELETE", "/api/v1/relationships", "",
nil,
map[string]any{"source": req.Params.Source, "target": req.Params.Target, "type": req.Params.RelType}); auditErr != nil {
return nil, auditErr
}

View File

@@ -237,6 +237,11 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
// Drift audit: read-only DB-side report of orphan checks, checks on
// retired targets, stuck down/unknown probes, unmonitored declared types,
// and dangling edges. Companion to the knowledge-graph-audit skill.
r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift)
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
// unlike ListExecutions which sorts by target for pagination) and the
// per-session "what did this session do" digest.

View File

@@ -136,6 +136,7 @@ func (s *Server) PatchSkill(ctx context.Context, req gen.PatchSkillRequestObject
actorType, actor := actorInfo(ctx)
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
&id, "PATCH", "/api/v1/skills/"+req.Id, "",
nil,
map[string]any{"status": req.Body.Status, "pinned_version": req.Body.PinnedVersion}); auditErr != nil {
return nil, auditErr
}

View File

@@ -0,0 +1,251 @@
package mcp
// Integration tests for the entity-mutation MCP tools (create_entity,
// update_entity_attributes), focused on the capability gap that stranded
// session 23da10db: entities mutated via MCP must derive/regenerate checks the
// same way the HTTP create/patch paths do. Guarded by OIKOS_TEST_DATABASE_URL
// (see internal/db/integration_test.go); run via `make test-db`.
import (
"context"
"encoding/json"
"fmt"
"math/rand"
"os"
"strings"
"testing"
"github.com/dtoro/oikos/internal/checkdefaults"
"github.com/dtoro/oikos/internal/db"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/modelcontextprotocol/go-sdk/mcp"
)
// newTestPool mirrors internal/httpapi/api_test.go: a throwaway database,
// migrated and seeded with ontology/inventory/policy so create_entity's type
// validation and checkdefaults derivation have a real type tree to work
// against.
func newTestPool(t *testing.T) *db.Pool {
t.Helper()
baseURL := os.Getenv("OIKOS_TEST_DATABASE_URL")
if baseURL == "" {
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
}
ctx := context.Background()
admin, err := pgx.Connect(ctx, baseURL)
if err != nil {
t.Fatalf("connect admin: %v", err)
}
dbName := fmt.Sprintf("oikos_mcp_test_%08x", rand.Int63())
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
admin.Close(ctx)
t.Fatalf("create test db: %v", err)
}
admin.Close(ctx)
qi := strings.Index(baseURL, "?")
base, params := baseURL, ""
if qi >= 0 {
base, params = baseURL[:qi], baseURL[qi:]
}
testURL := base[:strings.LastIndex(base, "/")+1] + dbName + params
pool, err := db.New(ctx, testURL)
if err != nil {
t.Fatalf("connect test db: %v", err)
}
t.Cleanup(func() {
pool.Close()
if admin, e := pgx.Connect(ctx, baseURL); e == nil {
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
admin.Close(ctx)
}
})
if err := pool.Migrate(ctx); err != nil {
t.Fatalf("migrate: %v", err)
}
for _, f := range []string{"ontology.yaml", "inventory.yaml", "policy.yaml"} {
content, err := os.ReadFile("../../seeds/" + f)
if err != nil {
t.Fatalf("read seed %s: %v", f, err)
}
name := f
if err := pool.SeedIngest(ctx, name, content,
func(ctx context.Context, tx pgx.Tx, data map[string]any) error {
var err error
switch name {
case "ontology.yaml":
_, err = db.IngestOntologySeed(ctx, tx, data)
case "inventory.yaml":
_, err = db.IngestInventorySeed(ctx, tx, data)
case "policy.yaml":
_, err = db.IngestPolicySeed(ctx, tx, data)
}
return err
}); err != nil {
t.Fatalf("ingest %s: %v", f, err)
}
}
return pool
}
// callTool invokes a registered tool's handler in-process and returns its
// concatenated text result.
func callTool(t *testing.T, pool *db.Pool, name string, args map[string]any) string {
t.Helper()
var handler toolHandler
for _, r := range allTools(pool, uuid.Nil) {
if r.tool.Name == name {
handler = r.handler
break
}
}
if handler == nil {
t.Fatalf("tool %q not registered", name)
}
argsJSON, _ := json.Marshal(args)
res, err := handler(context.Background(), &mcp.CallToolRequest{Params: &mcp.CallToolParamsRaw{
Name: name,
Arguments: argsJSON,
}})
if err != nil {
t.Fatalf("tool %s returned error: %v", name, err)
}
var sb strings.Builder
for _, c := range res.Content {
if tc, ok := c.(*mcp.TextContent); ok {
sb.WriteString(tc.Text)
}
}
return sb.String()
}
// checkCountFor returns the number of derived check_defs targeting slug.
func checkCountFor(t *testing.T, pool *db.Pool, slug string) int {
t.Helper()
var n int
err := pool.QueryRow(context.Background(),
`SELECT count(*) FROM check_defs cd
JOIN entities e ON e.id = cd.target_id
WHERE e.slug = $1`, slug).Scan(&n)
if err != nil {
t.Fatalf("count check_defs for %s: %v", slug, err)
}
return n
}
// TestCreateEntity_DerivesChecks proves create_entity inserts an entity AND
// derives its default checks in one call (the HTTP create path did this; the
// MCP path previously could not create at all).
func TestCreateEntity_DerivesChecks(t *testing.T) {
pool := newTestPool(t)
slug := "service:mcp-create-test"
out := callTool(t, pool, "create_entity", map[string]any{
"type": "service",
"slug": slug,
"name": "mcp-create-test",
"attributes": `{"url":"https://mcp-create-test.example"}`,
})
if !strings.Contains(out, "Created "+slug) {
t.Fatalf("create_entity result = %q, want Created %s", out, slug)
}
if !strings.Contains(out, "Derived") {
t.Errorf("create_entity result = %q, want a Derived check summary", out)
}
if got := checkCountFor(t, pool, slug); got < 1 {
t.Errorf("check_defs targeting %s = %d, want >=1 (create did not derive checks)", slug, got)
}
}
// TestCreateEntity_DuplicateAndInvalid covers the guard rails: a repeat create
// is reported as "already exists" (not an error), and an unknown type is
// rejected with a clear message.
func TestCreateEntity_DuplicateAndInvalid(t *testing.T) {
pool := newTestPool(t)
if out := callTool(t, pool, "create_entity", map[string]any{
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
}); !strings.Contains(out, "Created service:mcp-dup") {
t.Fatalf("first create = %q", out)
}
if out := callTool(t, pool, "create_entity", map[string]any{
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
}); !strings.Contains(out, "already exists") {
t.Errorf("duplicate create = %q, want 'already exists'", out)
}
if out := callTool(t, pool, "create_entity", map[string]any{
"type": "no-such-type", "slug": "no-such-type:x", "name": "x",
}); !strings.Contains(out, "not found in ontology") {
t.Errorf("unknown type = %q, want 'not found in ontology'", out)
}
}
// TestUpdateEntityAttributes_RegeneratesChecks is the regression guard for the
// haos session: setting an entity's `monitoring` attribute via MCP must
// regenerate checks. Before this fix the MCP update path skipped
// ensureDefaultChecks, so flipping monitoring produced nothing.
func TestUpdateEntityAttributes_RegeneratesChecks(t *testing.T) {
pool := newTestPool(t)
slug := "service:mcp-regen-test"
// Create with monitoring:none — no checks derived.
if out := callTool(t, pool, "create_entity", map[string]any{
"type": "service", "slug": slug, "name": "mcp-regen-test",
"attributes": `{"monitoring":"none","url":"https://mcp-regen.example"}`,
}); !strings.Contains(out, "Created "+slug) {
t.Fatalf("create = %q", out)
}
if got := checkCountFor(t, pool, slug); got != 0 {
t.Fatalf("check_defs with monitoring:none = %d, want 0", got)
}
// Flip monitoring to [http] via update_entity_attributes — checks must
// regenerate. This is exactly what failed for service:haos.
out := callTool(t, pool, "update_entity_attributes", map[string]any{
"slug": slug,
"attributes": `{"monitoring":["http"]}`,
})
if !strings.Contains(out, "Updated "+slug) {
t.Fatalf("update result = %q, want Updated %s", out, slug)
}
if !strings.Contains(out, "Derived") {
t.Errorf("update result = %q, want a Derived check summary (regeneration)", out)
}
if got := checkCountFor(t, pool, slug); got < 1 {
t.Errorf("check_defs after monitoring:[http] = %d, want >=1 (MCP update did not regenerate checks)", got)
}
}
// TestUpdateEntityAttributes_NotFound keeps the existing error contract.
func TestUpdateEntityAttributes_NotFound(t *testing.T) {
pool := newTestPool(t)
out := callTool(t, pool, "update_entity_attributes", map[string]any{
"slug": "service:does-not-exist",
"attributes": `{"x":1}`,
})
if !strings.Contains(out, "not found") {
t.Errorf("update missing entity = %q, want 'not found'", out)
}
}
// TestFormatCheckResult is a pure unit test for the result-message helper, so
// the formatting contract holds even when the DB is unavailable.
func TestFormatCheckResult(t *testing.T) {
if got := formatCheckResult(checkdefaults.Result{Created: 2}); !strings.Contains(got, "Derived 2 check") {
t.Errorf("created-only = %q, want Derived 2", got)
}
got := formatCheckResult(checkdefaults.Result{Created: 1, Skipped: []checkdefaults.Skip{{Kind: "process", Reason: "no host"}}})
if !strings.Contains(got, "Derived 1 check") || !strings.Contains(got, "Skipped process") || !strings.Contains(got, "no host") {
t.Errorf("created+skipped = %q", got)
}
if got := formatCheckResult(checkdefaults.Result{Undeclared: true}); !strings.Contains(got, "no monitoring") {
t.Errorf("undeclared = %q, want no-monitoring hint", got)
}
if formatCreateResult("a", "b", checkdefaults.Result{Created: 0}) != "Created a (b)." {
t.Error("create result with no checks should have no suffix")
}
}

119
internal/mcp/discover.go Normal file
View File

@@ -0,0 +1,119 @@
package mcp
import (
"bufio"
"context"
"strings"
"github.com/dtoro/oikos/internal/db"
"github.com/dtoro/oikos/internal/execlog"
"github.com/dtoro/oikos/internal/remote"
)
// discoverInfraDrift compares the live Proxmox guests (pct/qm list on every
// proxmox host) against the DB graph, surfacing drift the DB-only audit
// cannot see: guests running with no entity (missing), and entities whose
// pve_id is no longer live (ghost). This is the auto-discover/validate half
// of the knowledge-graph audit skill — read-only, reaches hosts over the same
// SSH/pct path the checks use.
func discoverInfraDrift(ctx context.Context, pool *db.Pool) any {
// 1. proxmox hosts to query.
hostRows, err := pool.Query(ctx,
`SELECT slug FROM entities WHERE type='proxmox-host' AND COALESCE(state,'active')='active'`)
if err != nil {
return map[string]any{"error": "query hosts: " + err.Error()}
}
var hosts []string
for hostRows.Next() {
var s string
if hostRows.Scan(&s) == nil {
hosts = append(hosts, s)
}
}
hostRows.Close()
// 2. DB guests keyed by pve_id.
type dbGuest struct {
Slug string `json:"slug"`
Type string `json:"type"`
PveID string `json:"pve_id"`
Host string `json:"host"`
}
dbGuests := map[string]dbGuest{}
gr, err := pool.Query(ctx,
`SELECT slug, type, COALESCE(attributes->>'pve_id',''), COALESCE(attributes->>'host','')
FROM entities WHERE type IN ('lxc','vm')`)
if err != nil {
return map[string]any{"error": "query guests: " + err.Error()}
}
for gr.Next() {
var g dbGuest
if gr.Scan(&g.Slug, &g.Type, &g.PveID, &g.Host) == nil && g.PveID != "" {
dbGuests[g.PveID] = g
}
}
gr.Close()
// 3. enumerate live guests from every host.
live := map[string]string{} // pve_id -> "host:name"
hostErrors := map[string]string{}
for _, hs := range hosts {
et, rerr := remote.ResolveExecTarget(ctx, pool, hs, sshUser)
if rerr != nil {
hostErrors[hs] = "resolve: " + rerr.Error()
continue
}
for _, cmd := range []string{"pct list", "qm list"} {
out, eerr := sshExecStream(ctx, et.Host, et.User, et.Wrap(cmd),
execlog.Sink(func(string, []byte) {}))
if eerr != nil {
hostErrors[hs+" "+cmd] = eerr.Error()
continue
}
scanIDs(out, hs, live)
}
}
// 4. diff.
var missing, ghost []string
for id, hn := range live {
if _, ok := dbGuests[id]; !ok {
missing = append(missing, id+" on "+hn)
}
}
for id, g := range dbGuests {
if _, ok := live[id]; !ok {
ghost = append(ghost, g.Slug+" (pve_id="+id+")")
}
}
return map[string]any{
"hosts_queried": len(hosts),
"live_guests": len(live),
"db_guests": len(dbGuests),
"missing_entities": missing, // in Proxmox, no DB entity
"ghost_entities": ghost, // in DB, not live in Proxmox
"host_errors": hostErrors,
}
}
// scanIDs parses `pct list` / `qm list` output (VMID ... Name) into the live map.
func scanIDs(output, hostSlug string, live map[string]string) {
sc := bufio.NewScanner(strings.NewReader(output))
for sc.Scan() {
line := strings.TrimSpace(sc.Text())
if line == "" || strings.HasPrefix(strings.ToLower(line), "vmid") {
continue
}
fields := strings.Fields(line)
if len(fields) < 2 {
continue
}
id := fields[0]
name := ""
if len(fields) >= 4 {
name = fields[len(fields)-1] // pct: last col is name; qm: name near end
}
live[id] = hostSlug + ":" + name
}
}

View File

@@ -5,7 +5,6 @@ package mcp
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"fmt"
"html"
@@ -25,6 +24,7 @@ import (
"github.com/dtoro/oikos/internal/execlog"
"github.com/dtoro/oikos/internal/observability"
"github.com/dtoro/oikos/internal/policy"
"github.com/dtoro/oikos/internal/remote"
"github.com/google/jsonschema-go/jsonschema"
"github.com/google/uuid"
"github.com/modelcontextprotocol/go-sdk/mcp"
@@ -72,9 +72,121 @@ func newServer(pool *db.Pool, agentID uuid.UUID) *mcp.Server {
for _, t := range allTools(pool, agentID) {
s.AddTool(t.tool, withActivityLogging(pool, agentID, t.tool.Name, t.handler))
}
// Resource templates: let MCP clients browse and attach entities,
// knowledge entries, and executions as conversation resources.
s.AddResourceTemplate(&mcp.ResourceTemplate{
URITemplate: "oikos://entity/{slug}",
Name: "Entity",
Description: "Oikos entity by slug (e.g. host:hubris, lxc:jellyfin)",
MIMEType: "application/json",
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
slug := matches["slug"]
var id uuid.UUID
if u, err := uuid.Parse(slug); err == nil {
id = u
} else {
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", slug).Scan(&id)
}
if id == uuid.Nil {
return "", fmt.Errorf("entity not found: %s", slug)
}
result := queryEntity(ctx, pool, slug)
return result.Content[0].(*mcp.TextContent).Text, nil
}))
s.AddResourceTemplate(&mcp.ResourceTemplate{
URITemplate: "oikos://knowledge/{id}",
Name: "Knowledge",
Description: "Knowledge entry by entity slug or UUID",
MIMEType: "application/json",
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
idOrSlug := matches["id"]
var entityID uuid.UUID
if u, err := uuid.Parse(idOrSlug); err == nil {
entityID = u
} else {
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", idOrSlug).Scan(&entityID)
}
if entityID == uuid.Nil {
return "", fmt.Errorf("knowledge not found: %s", idOrSlug)
}
result := queryRows(ctx, pool, `
SELECT ke.title, ke.content, ke.tags::text, e.slug, e.type AS kind,
ke.updated_at::text
FROM knowledge_entities ke
JOIN entities e ON e.id = ke.entity_id
WHERE ke.entity_id = $1`, entityID)
return result.Content[0].(*mcp.TextContent).Text, nil
}))
s.AddResourceTemplate(&mcp.ResourceTemplate{
URITemplate: "oikos://execution/{id}",
Name: "Execution",
Description: "Execution by UUID (returns status, result, timing)",
MIMEType: "application/json",
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
result := queryRows(ctx, pool, `
SELECT e.entity_id, te.slug AS target, e.action, e.risk_class,
e.status, e.result::text, e.duration_ms,
e.started_at::text, e.completed_at::text
FROM executions e
JOIN entities te ON te.id = e.target_entity_id
WHERE e.entity_id = $1`, matches["id"])
return result.Content[0].(*mcp.TextContent).Text, nil
}))
return s
}
// resourceHandler adapts a simple func(ctx, params) → (string, error) into
// an MCP ResourceHandler, reading the URI matched by a ResourceTemplate.
func resourceHandler(pool *db.Pool, fn func(ctx context.Context, matches map[string]string) (string, error)) mcp.ResourceHandler {
return func(ctx context.Context, req *mcp.ReadResourceRequest) (*mcp.ReadResourceResult, error) {
uri := req.Params.URI
matches := matchURITemplate(uri)
if matches == nil {
return nil, mcp.ResourceNotFoundError(uri)
}
text, err := fn(ctx, matches)
if err != nil {
return nil, mcp.ResourceNotFoundError(uri)
}
result, err := json.MarshalIndent(json.RawMessage(text), "", " ")
if err != nil {
result = []byte(text)
}
return &mcp.ReadResourceResult{
Contents: []*mcp.ResourceContents{{
URI: uri,
MIMEType: "application/json",
Text: string(result),
}},
}, nil
}
}
// matchURITemplate extracts parameters from a URI that matches one of the
// oikos:// resource templates. Returns nil if the URI doesn't match.
func matchURITemplate(uri string) map[string]string {
// oikos://entity/{slug}
if rest, ok := strings.CutPrefix(uri, "oikos://entity/"); ok && rest != "" {
return map[string]string{"slug": rest}
}
// oikos://knowledge/{id}
if rest, ok := strings.CutPrefix(uri, "oikos://knowledge/"); ok && rest != "" {
return map[string]string{"id": rest}
}
// oikos://execution/{id}
if rest, ok := strings.CutPrefix(uri, "oikos://execution/"); ok && rest != "" {
return map[string]string{"id": rest}
}
return nil
}
// withActivityLogging wraps a tool handler to record agent_activity rows.
func withActivityLogging(pool *db.Pool, agentID uuid.UUID, toolName string, next mcp.ToolHandler) mcp.ToolHandler {
if agentID == uuid.Nil {
@@ -461,31 +573,12 @@ func sshExecStream(ctx context.Context, host, user, command string, sink execlog
}
}
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUser string, err error) {
var attrs string
err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", entitySlug).Scan(&attrs)
if err != nil {
return "", "", fmt.Errorf("entity not found: %s", entitySlug)
}
var m map[string]interface{}
if err := json.Unmarshal([]byte(attrs), &m); err != nil {
return "", "", fmt.Errorf("parse attributes: %w", err)
}
if ip, ok := m["lan_ip"].(string); ok && ip != "" {
return ip, sshUser, nil
}
if mesh, ok := m["mesh"].(map[string]interface{}); ok {
for _, proto := range []string{"netbird", "tailscale"} {
if p, ok := mesh[proto].(map[string]interface{}); ok {
if ip, ok := p["ip"].(string); ok && ip != "" {
return ip, sshUser, nil
}
}
}
}
return "", "", fmt.Errorf("no IP found for %s", entitySlug)
// resolveHost resolves a host:<slug> to its reachable IP and SSH user. A thin
// wrapper over the shared resolver (internal/remote), kept so slug-based
// callers keep working; the shared resolver also prefers public_ipv4 over
// mesh and honors a per-entity ssh.user.
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUserOut string, err error) {
return remote.ResolveHost(ctx, pool, entitySlug, sshUser)
}
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
@@ -564,109 +657,28 @@ func isPrivateHost(host string) bool {
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
// `qm guest exec <pve_id> -- ...` for a VM.
//
// The lxc.attributes.host value is stored WITHOUT a "host:" prefix (e.g.
// "strong", not "host:strong") — see pct_create's entity registration. The
// pre-existing pct_exec handler queried resolveHost with that bare value
// directly, which can never match a "host:*" slug and always fails; this
// prefixes it correctly.
//
// vm: support (2026-07-18): VMs in inventory.yaml carry `pve_id` and a `host`
// attribute (or a `hosts` relationship) just like LXCs, but they're reached
// via `qm guest exec` instead of `pct exec`. Previously the agent had to
// SSH-hop via `host:hubris` to reach a VM (e.g. `ssh root@<vm_ip> '...'`),
// which broke on nested shell quoting and forced manual escaping workarounds
// — see plans/2026-07-18-session-review-three-sessions.md P1.6. A VM's
// `host` attribute is optional: if absent, fall back to looking up the
// `hosts` relationship on the VM entity, then to hubris (the documented
// default Proxmox host) — same fallback chain as LXCs.
// Delegates to the shared resolver (internal/remote), the single path used by
// both the MCP `run` tool and the scheduler's checks. The historical notes
// (host attr without prefix, vm host-resolution chain, nested-quoting
// handling via base64) all still hold — they now live in remote.guestWrap.
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
if strings.HasPrefix(targetSlug, "host:") {
host, user, err = resolveHost(ctx, pool, targetSlug)
return host, user, func(cmd string) string { return cmd }, err
et, err := remote.ResolveExecTarget(ctx, pool, targetSlug, sshUser)
if err != nil {
return "", "", nil, err
}
if strings.HasPrefix(targetSlug, "lxc:") {
var pveID, hostAttr string
// COALESCE the host column: many older LXC entities (seeded from
// inventory, not provisioned by pct_create) have pve_id but no host
// attribute at all. Scanning a SQL NULL into a plain string errors
// the whole row, wrongly reporting "missing pve_id" even when it was
// present — COALESCE avoids the NULL, "" is handled below.
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
return "", "", nil, fmt.Errorf("LXC not found or missing pve_id: %s", targetSlug)
}
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
host, user, err = resolveHost(ctx, pool, hostSlug)
id := pveID
return host, user, func(cmd string) string {
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", id, b64)
}, err
}
if strings.HasPrefix(targetSlug, "vm:") {
// VMs: same host-resolution chain as LXCs (attributes.host →
// `hosts` relationship → hubris default), but reached via
// `qm guest exec` instead of `pct exec`. Requires the QEMU
// guest agent running inside the VM (the standard Proxmox
// setup; ZimaOS/HAOS in this fleet already have it).
var pveID, hostAttr string
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
return "", "", nil, fmt.Errorf("VM not found or missing pve_id: %s", targetSlug)
}
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
host, user, err = resolveHost(ctx, pool, hostSlug)
id := pveID
return host, user, func(cmd string) string {
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
// `qm guest exec <id> -- /bin/bash -c '...'` returns JSON by
// default; pipe through `jq -r .out` if available, else cat.
// The base64 round-trip mirrors the LXC path so nested quoting
// (the original VM-target pain point — session 55927f0a) is
// handled identically to LXC dispatch.
return fmt.Sprintf(
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
id, b64, id, b64)
}, err
}
return "", "", nil, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
return et.Host, et.User, et.Wrap, nil
}
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
// LXC/VM target. Resolution order:
// 1. hostAttr if non-empty (the entity's attributes.host — stored without
// "host:" prefix in inventory.yaml and pct_create).
// 2. the `hosts` relationship on the entity (e.g. host:hubris → vm:zimaos),
// looked up in the relationships table — the canonical graph source.
// 3. "hubris" as a documented default Proxmox host fallback.
//
// Returns a slug with the "host:" prefix attached, ready for resolveHost.
// Extracted from the inline LXC path (2026-07-18) so the VM path shares the
// same chain — see plans/2026-07-18-session-review-three-sessions.md P1.6.
// LXC/VM target (see internal/remote.ResolveProxmoxHostSlug for the chain).
// This slug-based wrapper looks up the entity id so slug callers keep working;
// the shared resolver takes an id directly.
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
hostSlug := strings.TrimSpace(hostAttr)
if hostSlug == "" {
// Fall back to the `hosts` relationship — the graph edge from
// the Proxmox host to this LXC/VM. This is the canonical source
// for "who owns this VM" in inventory.yaml; the `host` attribute
// is a denormalized shortcut that not every entity has.
var relHostSlug string
// hosts relationship: source=host, target=lxc/vm. Look up the
// source slug given the target.
if err := pool.QueryRow(ctx, `
SELECT e.slug FROM relationships r
JOIN entities e ON e.id = r.source_id
WHERE r.target_id = (SELECT id FROM entities WHERE slug = $1)
AND r.type = 'hosts' AND r.valid_to IS NULL
LIMIT 1`, entitySlug).Scan(&relHostSlug); err == nil && relHostSlug != "" {
hostSlug = relHostSlug
}
var id uuid.UUID
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", entitySlug).Scan(&id); err != nil {
id = uuid.Nil
}
if hostSlug == "" {
hostSlug = "hubris" // documented default Proxmox host when unset
}
if !strings.HasPrefix(hostSlug, "host:") {
hostSlug = "host:" + hostSlug
}
return hostSlug
return remote.ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
}
// classifyAndGate is the shared classify→execute-or-queue path for every
@@ -728,8 +740,106 @@ func autoRun(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, comma
return out, nil
}
// autoRunAsync starts a command in a goroutine, marking it running and returning
// immediately. The caller gets an execution_id to poll with get_execution_status.
// Used for commands containing sleep/wait/poll loops that would exceed the MCP
// client timeout (120s) — the execution continues server-side.
func autoRunAsync(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, command string) {
startedAt := time.Now()
if _, err := pool.Exec(ctx,
`UPDATE executions SET status='running', started_at=$2 WHERE entity_id=$1`,
id, startedAt); err != nil {
slog.Error("mcp: mark execution running (async)", "error", err, "execution_id", id)
}
host, user, wrap, err := resolveExecTarget(ctx, pool, targetSlug)
if err != nil {
pool.Exec(ctx,
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
id, jsonErr("%s", err.Error()), int(time.Since(startedAt).Milliseconds()))
slog.Error("mcp: async run resolve target", "error", err, "execution_id", id, "target", targetSlug)
return
}
var correlationID string
if qerr := pool.QueryRow(ctx,
`SELECT correlation_id FROM executions WHERE entity_id = $1`, id).Scan(&correlationID); qerr != nil {
correlationID = ""
}
go func() {
defer func() {
if r := recover(); r != nil {
slog.Error("mcp: async run panic", "panic", r, "execution_id", id)
pool.Exec(context.Background(),
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
id, jsonErr("panic: %v", r), int(time.Since(startedAt).Milliseconds()))
}
}()
sink, flush := execlog.New(context.Background(), pool, id, correlationID)
out, execErr := sshExecStream(context.Background(), host, user, wrap(command), sink)
flush()
if execErr != nil {
pool.Exec(context.Background(),
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
id, jsonErr("%s: %s", execErr.Error(), out), int(time.Since(startedAt).Milliseconds()))
slog.Error("mcp: async run failed", "error", execErr, "execution_id", id, "output", out)
} else {
pool.Exec(context.Background(),
`UPDATE executions SET status='completed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
id, jsonOut(out), int(time.Since(startedAt).Milliseconds()))
}
}()
}
// isLongRunningCommand detects shell commands containing sleep, wait, or poll
// loops that indicate the command will exceed the MCP client timeout (120s).
// These commands should use autoRunAsync to avoid the client timing out while
// the command continues server-side.
func isLongRunningCommand(cmd string) bool {
cmd = strings.TrimSpace(cmd)
// sleep with duration — `sleep 30`, `sleep 1m`, etc.
if sleepRe.MatchString(cmd) {
return true
}
// while/shell poll loops with sleep: `while ...; do ... sleep; done`
if pollRe.MatchString(cmd) {
return true
}
// standalone wait command
if waitRe.MatchString(cmd) {
return true
}
return false
}
var (
sleepRe = regexp.MustCompile(`\bsleep\s+\d`)
pollRe = regexp.MustCompile(`\bwhile\b.*\bsleep\b`)
waitRe = regexp.MustCompile(`\bwait\s+\d|[&;]\s*wait\b`)
)
func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.UUID, targetSlug, command, purpose, declaredRisk, sessionID string) *mcp.CallToolResult {
riskClass := policy.ClassifyCommand(command, declaredRisk)
// Transport-aware escalation: read-only commands on LXC targets that
// touch config paths (/opt/, /etc/) escalate to config_mutation.
// The classifier only scores the command text, not the transport layer
// — SSH-ing into a container to read /opt/ is riskier than running
// the same command locally on the Proxmox host via pct exec.
// Caught live: "cat /etc/hostname" on lxc:dns queued as config_mutation
// while "pct exec 107 -- cat /etc/hostname" on host:hubris auto-ran.
if riskClass == policy.RiskReadOnly && strings.HasPrefix(targetSlug, "lxc:") {
if strings.Contains(command, "/opt/") || strings.Contains(command, "/etc/") || strings.Contains(command, "/var/lib/") {
riskClass = policy.RiskConfigMutation
}
}
runParams, _ := json.Marshal(map[string]string{"command": command, "purpose": purpose})
actionCol := "run:" + string(runParams)
@@ -747,6 +857,57 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
return textResult("No plan for this session. Call set_goal then propose_plan before run — even read-only tasks require a one-step plan. A one-step plan (\"Inspect X, report, write back\") is fine for trivial questions; the gate is about ordering, not approval. Read-only commands still auto-execute once a plan exists.")
}
// Target validation: host-only commands (qm, pct, pvesh, iptables) must
// not be dispatched against lxc:/vm: targets — those aren't Proxmox hosts
// and don't have these tools. Caught live 2026-08-04: the agent ran
// `qm stop 100` against lxc:dns, wasting a turn.
// Command syntax validation: catch LLM-generated bash bugs before they
// hit the shell. The model sometimes inserts literal \n between commands
// or puts spaces inside flags — these always fail, so reject early.
if syntaxErr := validateCommandSyntax(command); syntaxErr != "" {
return textResult(syntaxErr)
}
if cmdPrefix, hostOnly := hostOnlyCommand(command); hostOnly && !strings.HasPrefix(targetSlug, "host:") {
hostSuggestion := resolveProxmoxHostSlug(ctx, pool, targetSlug, "")
if hostSuggestion == "" {
hostSuggestion = "host:hubris or host:strong"
}
return textResult(fmt.Sprintf("Cannot run %q on %s — %s is a Proxmox host command. Use target %s instead.",
cmdPrefix, targetSlug, cmdPrefix, hostSuggestion))
}
// systemctl and docker work on hosts and LXCs, but not VMs.
if cmdPrefix, hostLxc := hostLxcCommand(command); hostLxc {
if !strings.HasPrefix(targetSlug, "host:") && !strings.HasPrefix(targetSlug, "lxc:") {
return textResult(fmt.Sprintf("Cannot run %q on %s — %s only works on host:* or lxc:* targets.",
cmdPrefix, targetSlug, cmdPrefix))
}
}
// VM transport pre-flight: qm guest exec requires the QEMU guest agent
// to be running inside the VM. If it's not, the execution would queue
// for approval and never execute — the agent has no way to learn it's
// stuck (spotted live 2026-08-05: vm:zimaos had qemu_guest_agent=not_running,
// the run queued forever, and the agent fell back to unsafe raw SSH).
if strings.HasPrefix(targetSlug, "vm:") {
var rawAttrs []byte
if err := pool.QueryRow(ctx, `SELECT attributes FROM entities WHERE id = $1`, targetID).Scan(&rawAttrs); err == nil {
var attrs map[string]any
if json.Unmarshal(rawAttrs, &attrs) == nil {
if qga, ok := attrs["qemu_guest_agent"]; ok {
qgaStr, _ := qga.(string)
if qgaStr == "not_running" || qgaStr == "" {
return textResult(fmt.Sprintf(
"run on %s blocked: QEMU guest agent is not running (%s). qm guest exec cannot reach this VM. Start the agent inside the guest first (e.g. via SSH/systemctl start qemu-guest-agent), then re-run. If the agent is running but the entity attribute is stale, update it with update_entity_attributes(slug=%s, attributes={\"qemu_guest_agent\":\"running\"}).",
targetSlug, qgaStr, targetSlug))
}
}
}
}
}
// Dedup: an identical pending command (same target, command, and
// purpose) blocks a re-request — stops a tool-calling loop from queuing
// the same approval repeatedly.
@@ -821,8 +982,51 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
SELECT 1 FROM relationships
WHERE source_id = t.id AND target_id = $1 AND type = 'involves' AND valid_to IS NULL)`,
id, "task:"+sessionID)
// Link execution to session for auto-continuation (nomos_plan_executions
// was always empty — executions were never traceable back to sessions).
if sid, serr := uuid.Parse(sessionID); serr == nil {
pool.Exec(ctx, `
INSERT INTO nomos_plan_executions (execution_id, session_id)
VALUES ($1, $2) ON CONFLICT (execution_id) DO NOTHING`, id, sid)
}
}
// Auto-classify: write the classification decision to the classifications
// table (was always empty — 0 rows despite 1,884 executions). The route
// matches the auto-run vs queue-for-approval decision below.
classRoute := "escalate"
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
classRoute = "auto-act"
} else if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
classRoute = "auto-act"
} else if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
classRoute = "auto-act"
}
classReason, _ := json.Marshal(map[string]string{
"command": command, "purpose": purpose, "target": targetSlug, "declared_risk": declaredRisk,
})
classID, _ := uuid.NewV7()
pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'classification', $3, '{}')`,
classID, "classification:"+classID.String(), "classification for "+execSlug)
pool.Exec(ctx, `INSERT INTO classifications (entity_id, action, risk_class, route, reasoning, correlation_id)
VALUES ($1, $2, $3, $4, $5, $6)`,
classID, actionCol, riskClass, classRoute, classReason, correlationID)
// Link classification to execution.
pool.Exec(ctx, `UPDATE executions SET classification_id = $2 WHERE entity_id = $1`, id, classID)
// Audit: record the execution creation with session_id for traceability.
// Every run call, whether auto-run or queued-for-approval, gets an audit
// entry so the agent's activity is traceable back to the originating session.
var auditSessionID *uuid.UUID
if sessionID != "" && sessionID != "ephemeral" {
if sid, serr := uuid.Parse(sessionID); serr == nil {
auditSessionID = &sid
}
}
_ = observability.Audit(ctx, sqlcgen.New(pool), "agent", "nomos", "run",
&id, "POST", "/mcp", correlationID, auditSessionID,
map[string]any{"command": command, "target": targetSlug, "risk_class": riskClass, "purpose": purpose})
// read_only and reversible_low both run unattended, as seeds/policy.yaml
// and .agents/OIKOS.md declare ("reversible_low — restart, cache clear,
// sync pull. Unattended + ledger.").
@@ -841,6 +1045,11 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
// agent still cannot talk a command DOWN: declaring reversible_low on
// something computed as config_mutation keeps config_mutation.
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
if isLongRunningCommand(command) {
autoRunAsync(ctx, pool, id, targetSlug, command)
return textResult(fmt.Sprintf("run on %s (%s, async): started — execution %s. Poll with get_execution_status(%s) for result.",
targetSlug, riskClass, id, id))
}
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
if xerr != nil {
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
@@ -859,6 +1068,12 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
// consent. The assent window, opened only on operator approval, is the
// sole gate for config_mutation auto-run.)
if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
if isLongRunningCommand(command) {
autoRunAsync(ctx, pool, id, targetSlug, command)
slog.Info("mcp: run async via assent window", "target", targetSlug, "execution_id", id)
return textResult(fmt.Sprintf("run on %s (config_mutation, async via assent window): started — execution %s. Poll with get_execution_status(%s) for result.",
targetSlug, id, id))
}
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
if xerr != nil {
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
@@ -874,6 +1089,12 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
// operator isn't asked to re-type "I confirm" for every single command
// against the thing they just confirmed.
if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
if isLongRunningCommand(command) {
autoRunAsync(ctx, pool, id, targetSlug, command)
slog.Info("mcp: run async via destructive window", "target", targetSlug, "execution_id", id)
return textResult(fmt.Sprintf("run on %s (destructive, async via confirmed-target window): started — execution %s. Poll with get_execution_status(%s) for result.",
targetSlug, id, id))
}
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
if xerr != nil {
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
@@ -940,22 +1161,112 @@ func executeApprovedViaAPI(ctx context.Context, execID uuid.UUID, targetSlug, ac
// for config_mutation auto-run now. See sessionHasPlan for the plan-existence
// check used by the P1 plan-first gate.
// hostOnlyCommands maps command prefixes that are only valid on Proxmox host
// targets (not LXCs or VMs). Running these against an lxc: or vm: target
// always fails with "command not found" and wastes a turn.
var hostOnlyCommands = map[string]bool{
"qm": true,
"pct": true,
"pvesh": true,
"iptables": true,
}
// hostLxcCommands maps command prefixes valid on host:* and lxc:* but not vm:*.
var hostLxcCommands = map[string]bool{
"systemctl": true,
"docker": true,
}
// hostOnlyCommand checks whether the leading word of cmd is a host-only
// command. Returns the command word and true if the command can only run on
// a host: target.
func hostOnlyCommand(cmd string) (string, bool) {
trimmed := strings.TrimSpace(cmd)
parts := strings.Fields(trimmed)
if len(parts) == 0 {
return "", false
}
first := parts[0]
// Check for shell wrappers: bash -c 'actual_cmd', sh -c 'actual_cmd'
if (first == "bash" || first == "sh") && len(parts) >= 3 && parts[1] == "-c" {
// The actual command is inside the -c argument; extract the first word.
// This handles `bash -c 'qm stop 100'` but not deeply nested wrappers.
actual := strings.Trim(strings.Join(parts[2:], " "), "'\"")
if inner := strings.Fields(actual); len(inner) > 0 {
first = inner[0]
}
}
// Strip path: /usr/sbin/qm → qm
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
first = first[idx+1:]
}
return first, hostOnlyCommands[first]
}
// hostLxcCommand checks whether the leading word of cmd is a command valid on
// host:* and lxc:* targets but not vm:*. Returns the command word and true if
// the command is restricted to host/lxc.
func hostLxcCommand(cmd string) (string, bool) {
trimmed := strings.TrimSpace(cmd)
parts := strings.Fields(trimmed)
if len(parts) == 0 {
return "", false
}
first := parts[0]
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
first = first[idx+1:]
}
return first, hostLxcCommands[first]
}
// validateCommandSyntax checks for common LLM-generated bash errors that always
// fail at the shell. Returns an error message or "" if the command looks valid.
func validateCommandSyntax(cmd string) string {
// Reject literal \n (the LLM sometimes writes `echo "---" && \n curl ...`
// — the \n is literal in the command string, not an actual newline).
if strings.Contains(cmd, "\\n") {
return fmt.Sprintf("Command contains literal '\\n' — use ';' or '&&' between commands, not a literal backslash-n. Command: %q", cmd)
}
// Reject `&& \n` patterns (the LLM writes `cmd1 && \n cmd2` — the \n is
// a literal newline that bash interprets as a command separator, but the
// leading backslash makes it a syntax error).
if andBackslashRe.MatchString(cmd) {
return fmt.Sprintf("Command contains '&&' followed by a literal backslash-newline — remove the backslash or use ';' instead. Command: %q", cmd)
}
// Reject `\` at end of command with no continuation (last line ends with
// backslash but there's nothing after it).
trimmed := strings.TrimSpace(cmd)
if strings.HasSuffix(trimmed, "\\") {
return fmt.Sprintf("Command ends with a backslash but has nothing after it to continue. Remove the trailing '\\'. Command: %q", cmd)
}
// Warn on common flag typos: `head - n`, `grep - i`, `tail - n`, etc.
// These are space-between-flag-and-value errors the LLM produces.
if flagSpaceRe.MatchString(cmd) {
return fmt.Sprintf("Command has a space between a flag and its value (e.g. 'head - n' instead of 'head -n'). Remove the space. Command: %q", cmd)
}
return ""
}
var andBackslashRe = regexp.MustCompile(`&&\s*\\\s*\n`)
var flagSpaceRe = regexp.MustCompile(`\b(head|tail|grep|sed|awk|sort|uniq|wc)\s+(-\w)\s+\w`)
// sessionHasPlan reports whether this nomos session has any plan step on
// record (any generation, any status). Used by the P1 plan-first gate in
// classifyAndGate to refuse `run` before `propose_plan` has been called.
// A `replaced` step (from a prior plan generation that was superseded by a
// follow-up sub-task — see store.reopenSession) still counts: it proves the
// agent once framed a plan for this session, and the reopen path guarantees a
// fresh `propose_plan` will run before the next `run` anyway. Fails closed
// (returns true) when the query errors so a transient DB issue doesn't block
// an otherwise-valid run.
// record that isn't `replaced`. Replaced steps (from session reopen via
// store.reopenSession) don't count — the agent must propose fresh plan before
// any `run`. Fails closed (returns true) when the query errors so a transient
// DB issue doesn't block an otherwise-valid run.
func sessionHasPlan(ctx context.Context, pool *db.Pool, sessionID string) bool {
if sessionID == "" {
return true // no session → no gate (direct MCP call from a script)
}
var count int
if err := pool.QueryRow(ctx,
`SELECT COUNT(*) FROM session_plan_steps WHERE session_id = $1`,
`SELECT COUNT(*) FROM session_plan_steps
WHERE session_id = $1 AND status <> 'replaced'`,
sessionID).Scan(&count); err != nil {
return true // fail open on DB error — don't block work over a flake
}

File diff suppressed because it is too large Load Diff

View File

@@ -16,7 +16,7 @@ import (
// starts being populated when OIDC identity resolution lands.
func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
action string, entityID *uuid.UUID, method, path, correlationID string,
detail map[string]any) error {
sessionID *uuid.UUID, detail map[string]any) error {
if detail == nil {
detail = map[string]any{}
@@ -36,6 +36,7 @@ func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
Path: &path,
Detail: detailJSON,
CorrelationID: corr,
SessionID: sessionID,
})
}

View File

@@ -105,6 +105,15 @@ var curlLeadRe = regexp.MustCompile(`(?i)^curl\b`)
// When any of these appears, the curl command is no longer read-only.
var curlMutateRe = regexp.MustCompile(`(?i)(?:^|\s)-X\s+(?:post|put|delete|patch|connect|trace)\b|(?:^|\s)-(?:d|F|T|o)\b|(?:^|\s)--(?:data[-a-z]*|request|form|upload-file|output)\b`)
// curlDevNullOutRe matches curl output redirected to /dev/null in any of curl's
// argument forms (space, =, or attached). /dev/null is a no-op sink, so a GET
// that discards its body — the canonical reachability idiom
// `curl -o /dev/null -w '%{http_code}' URL` — is read-only. Output to any real
// path (-o /tmp/x) stays a potential mutation. Stripped before curlMutateRe so
// the remaining flags (-X, -d, ...) still classify correctly: a
// `curl -o /dev/null -X POST` stays config_mutation.
var curlDevNullOutRe = regexp.MustCompile(`(?i)(^|\s)-o\s*/dev/null(\s|$)|(^|\s)--output[=\s]\s*/dev/null(\s|$)`)
// redirectOutRe matches shell output redirection to a file (> or >> followed
// by a path), but excludes the file-descriptor merge form `>&<digit>` (e.g.
// `2>&1`) which only rearranges streams and writes nothing to disk. RE2 has
@@ -308,6 +317,10 @@ func curlIsReadOnly(curlCmd string) bool {
if !curlLeadRe.MatchString(curlCmd) {
return false
}
// -o /dev/null is a no-op sink: strip it before flag detection so the
// canonical GET-and-discard reachability probe stays read-only.
// A `curl -o /dev/null -X POST` still fails curlMutateRe after stripping.
curlCmd = curlDevNullOutRe.ReplaceAllString(curlCmd, " ")
if curlMutateRe.MatchString(curlCmd) {
return false
}

View File

@@ -102,6 +102,33 @@ func TestClassifyCommand_CurlPipeSh_ConfigMutation(t *testing.T) {
}
}
func TestClassifyCommand_CurlDevNull_ReadOnly(t *testing.T) {
// -o /dev/null is a no-op sink — the canonical GET-and-discard
// reachability idiom must stay read_only. Output to real paths stays
// config_mutation. POST/data flags after stripping still gate.
cases := []struct {
cmd string
cls string
}{
// read_only: GET with body discarded to /dev/null
{`curl -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.8.101:8123`, RiskReadOnly},
{`curl -sS -o /dev/null https://home.hubris.network`, RiskReadOnly},
{`curl --output /dev/null https://example.com`, RiskReadOnly},
{`curl -o /dev/null https://example.com`, RiskReadOnly},
{`curl -o/dev/null -w '%{http_code}' https://example.com`, RiskReadOnly},
// config_mutation: POST/data still caught after stripping devnull
{`curl -o /dev/null -X POST https://example.com`, RiskConfigMutation},
{`curl -o /dev/null -d '{"x":1}' https://example.com`, RiskConfigMutation},
// config_mutation: -o to real path stays config_mutation
{`curl -o /etc/caddy/Caddyfile http://example.com`, RiskConfigMutation},
}
for _, c := range cases {
if got := ClassifyCommand(c.cmd, ""); got != c.cls {
t.Errorf("ClassifyCommand(%q) = %q, want %q", c.cmd, got, c.cls)
}
}
}
func TestClassifyCommand_DefaultEscalatesToConfigMutation(t *testing.T) {
cases := []string{
"apt-get install -y nginx",

303
internal/remote/remote.go Normal file
View File

@@ -0,0 +1,303 @@
// Package remote resolves how to execute a command on a target entity and
// turns a plain shell command into whatever must be sent over the SSH
// connection that reaches it.
//
// The canonical access model: a host or workstation is reached by direct SSH
// to its address; an LXC or VM is NEVER SSH'd into directly — it is reached
// through its owning Proxmox host via `pct exec` / `qm guest exec`. One SSH
// credential per host (the host's root key), no per-guest keys, sshd, or
// lan_ip required for execution. Network probes (http/ping) still hit a
// guest's lan_ip directly; only command execution host-hops.
//
// This is the single resolver shared by the scheduler's check execution and
// the MCP `run` tool. Previously they diverged — the scheduler SSHed guests
// directly (broken for headless/keyless/mesh-only guests), while MCP
// host-hopped (working). Keeping one path keeps them in lockstep.
package remote
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"strings"
"github.com/dtoro/oikos/internal/db"
"github.com/google/uuid"
)
// DefaultUser is the SSH user when an entity declares no ssh.user. The
// Proxmox hosts and their guests are all administered as root.
const DefaultUser = "root"
// ExecTarget is a resolved execution endpoint: the SSH address and user to
// connect to, plus Wrap, which rewrites a plain command for transport.
type ExecTarget struct {
Host string
User string
// Wrap turns a plain shell command into the form that must be sent over
// the SSH connection to this target: the identity function for a host,
// `pct exec <id> -- bash -c 'echo <b64> | base64 -d | bash'` for an LXC,
// the `qm guest exec` equivalent for a VM. The base64 round-trip keeps
// nested quoting identical across both guest kinds.
Wrap func(cmd string) string
}
// IsGuest reports whether an entity type is reached via pct/qm exec through a
// Proxmox host rather than by direct SSH. docker-container is reached via its
// host's docker socket, not pct, so it is not a guest here.
func IsGuest(entityType string) bool {
return entityType == "lxc" || entityType == "vm"
}
// ResolveHost resolves a `host:<slug>` to its reachable network address and
// SSH user. Address preference: lan_ip, then public_ipv4, then mesh IP, then
// mesh fqdn. Preferring public_ipv4 over mesh matters because the scheduler
// container has no mesh interface — a standalone-server with only a mesh IP
// (netbird-vps) was unreachable, and a public_ipv4 was sitting unused.
//
// fallbackUser is used when the entity declares no ssh.user; callers pass
// their configured default (the scheduler uses "root", the MCP run tool uses
// its configured OIKOS_SSH_USER).
func ResolveHost(ctx context.Context, pool *db.Pool, hostSlug, fallbackUser string) (addr, user string, err error) {
var raw string
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", hostSlug).Scan(&raw); err != nil {
return "", "", fmt.Errorf("entity not found: %s", hostSlug)
}
var m map[string]any
if err = json.Unmarshal([]byte(raw), &m); err != nil {
return "", "", fmt.Errorf("parse attributes for %s: %w", hostSlug, err)
}
if v, ok := m["lan_ip"].(string); ok && v != "" {
addr = v
} else if v, ok := m["public_ipv4"].(string); ok && v != "" {
addr = v
} else if mesh, ok := m["mesh"].(map[string]any); ok {
if nb, ok := mesh["netbird"].(map[string]any); ok {
if v, ok := nb["ip"].(string); ok && v != "" {
addr = v
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
addr = v
}
}
}
if addr == "" {
return "", "", fmt.Errorf("no IP found for %s", hostSlug)
}
user = fallbackUser
if ssh, ok := m["ssh"].(map[string]any); ok {
if u, ok := ssh["user"].(string); ok && u != "" {
user = u
}
}
return addr, user, nil
}
// ResolveProxmoxHostSlug resolves the Proxmox host slug that owns a guest.
// Resolution order: the hostAttr if non-empty (the entity's attributes.host,
// stored without the "host:" prefix), the `hosts` relationship on the guest
// (the canonical graph edge), then "hubris" as the documented default.
//
// entityID is the guest's entity id; the relationship lookup uses it
// directly rather than a slug subquery.
func ResolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entityID uuid.UUID, hostAttr string) string {
hostSlug := strings.TrimSpace(hostAttr)
// Only trust a clean token as a host name. The attribute is operator/
// agent-writable and has been polluted with prose before ("hubris
// (confirmed via pct config…)") — using that verbatim produces a slug that
// never resolves. Treat anything with whitespace or parens as invalid and
// fall back to the canonical `hosts` edge below.
if strings.ContainsAny(hostSlug, " \t()") {
hostSlug = ""
}
if hostSlug == "" {
var relHostSlug string
if err := pool.QueryRow(ctx, `
SELECT e.slug FROM relationships r
JOIN entities e ON e.id = r.source_id
WHERE r.target_id = $1
AND r.type = 'hosts' AND r.valid_to IS NULL
LIMIT 1`, entityID).Scan(&relHostSlug); err == nil && relHostSlug != "" {
hostSlug = relHostSlug
}
}
if hostSlug == "" {
hostSlug = "hubris" // documented default Proxmox host when unset
}
if !strings.HasPrefix(hostSlug, "host:") {
hostSlug = "host:" + hostSlug
}
return hostSlug
}
// ResolveExecTarget resolves a target slug (host:, lxc:, or vm:) to its
// execution endpoint. This is the slug-based entry used by the MCP `run` tool.
func ResolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug, fallbackUser string) (ExecTarget, error) {
switch {
case strings.HasPrefix(targetSlug, "host:"):
addr, user, err := ResolveHost(ctx, pool, targetSlug, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
case strings.HasPrefix(targetSlug, "lxc:"), strings.HasPrefix(targetSlug, "vm:"):
var (
id uuid.UUID
pveID string
typ string
hostAttr string
)
if err := pool.QueryRow(ctx,
"SELECT id, type, attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE slug = $1",
targetSlug).Scan(&id, &typ, &pveID, &hostAttr); err != nil || pveID == "" {
return ExecTarget{}, fmt.Errorf("guest not found or missing pve_id: %s", targetSlug)
}
hostSlug := ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(typ, pveID)}, nil
}
return ExecTarget{}, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
}
// ResolveExecTargetForCheck resolves an execution endpoint keyed by the
// target's id and type — the data the scheduler has at check-execution time
// (check_defs carry target_id + target_type, not a slug). Guests route via
// pct/qm exec; everything else (hosts, workstations, services resolved to
// their hosting machine) is reached by direct SSH to the entity's own address.
func ResolveExecTargetForCheck(ctx context.Context, pool *db.Pool, targetID uuid.UUID, targetType, fallbackUser string) (ExecTarget, error) {
if IsGuest(targetType) {
return resolveGuest(ctx, pool, targetID, targetType, fallbackUser)
}
// A service (or other non-compute target) has no address of its own — it
// runs on whatever compute entity provides/hosts it. Resolve that host and
// route through it: pct if the host is a guest, direct SSH (with the
// host's correct user) if it's a machine. Previously a service check baked
// its hosting LXC's lan_ip and SSHed it directly as root, which fails
// because the scheduler key isn't in each LXC — only on the Proxmox hosts.
if hostID, hostType, ok := hostingCompute(ctx, pool, targetID); ok {
if IsGuest(hostType) {
return resolveGuest(ctx, pool, hostID, hostType, fallbackUser)
}
addr, user, err := resolveHostByID(ctx, pool, hostID, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
}
// No hosting entity found: reach the target directly at its own address
// (a host/workstation, or a service whose host wasn't resolvable).
addr, user, err := resolveHostByID(ctx, pool, targetID, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
}
// hostingCompute walks the provides/runs-on/hosts edges backward from a target
// to the compute entity that runs it (a service's LXC, an LXC's Proxmox host).
// Returns the host's id, type, and whether one was found. Most-specific edge
// first: provides names the runtime container directly.
func hostingCompute(ctx context.Context, pool *db.Pool, targetID uuid.UUID) (uuid.UUID, string, bool) {
var hid uuid.UUID
var htype string
err := pool.QueryRow(ctx, `
SELECT e.id, e.type FROM relationships r
JOIN entities e ON e.id = r.source_id
WHERE r.target_id = $1 AND r.valid_to IS NULL
AND r.type IN ('provides','runs-on','hosts')
ORDER BY CASE r.type WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1 ELSE 2 END
LIMIT 1`, targetID).Scan(&hid, &htype)
if err != nil {
return uuid.Nil, "", false
}
return hid, htype, true
}
// resolveGuest resolves a guest's execution endpoint: the owning Proxmox host
// (SSH'd directly) with a pct/qm exec wrapper around the command.
func resolveGuest(ctx context.Context, pool *db.Pool, guestID uuid.UUID, guestType, fallbackUser string) (ExecTarget, error) {
var pveID, hostAttr string
if err := pool.QueryRow(ctx,
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
guestID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
return ExecTarget{}, fmt.Errorf("guest %s missing pve_id", guestID)
}
hostSlug := ResolveProxmoxHostSlug(ctx, pool, guestID, hostAttr)
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(guestType, pveID)}, nil
}
// resolveHostByID is ResolveHost keyed by entity id.
func resolveHostByID(ctx context.Context, pool *db.Pool, id uuid.UUID, fallbackUser string) (addr, user string, err error) {
var raw string
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE id = $1", id).Scan(&raw); err != nil {
return "", "", fmt.Errorf("entity %s not found", id)
}
var m map[string]any
if err = json.Unmarshal([]byte(raw), &m); err != nil {
return "", "", fmt.Errorf("parse attributes: %w", err)
}
for _, key := range []string{"lan_ip", "public_ipv4", "mesh_ip"} {
if v, ok := m[key].(string); ok && v != "" {
addr = v
break
}
}
if addr == "" {
if mesh, ok := m["mesh"].(map[string]any); ok {
if nb, ok := mesh["netbird"].(map[string]any); ok {
if v, ok := nb["ip"].(string); ok && v != "" {
addr = v
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
addr = v
}
}
}
}
if addr == "" {
return "", "", fmt.Errorf("no IP found for entity %s", id)
}
user = fallbackUser
if ssh, ok := m["ssh"].(map[string]any); ok {
if u, ok := ssh["user"].(string); ok && u != "" {
user = u
}
}
if u, ok := m["user"].(string); ok && u != "" && user == fallbackUser {
// Workstations carry their login as a top-level `user` attribute
// (mac-mini: user: dtoro), not under ssh.user. Take it only when no
// explicit ssh.user was set, so a host that genuinely wants root still
// gets root.
user = u
}
return addr, user, nil
}
// guestWrap builds the pct/qm exec wrapper for a guest of the given type.
func guestWrap(entityType, pveID string) func(cmd string) string {
if entityType == "vm" {
return func(cmd string) string {
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
// `qm guest exec` returns JSON; pipe through jq for a clean stdout,
// falling back to the raw form. Mirrors the LXC base64 round-trip.
return fmt.Sprintf(
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
pveID, b64, pveID, b64)
}
}
return func(cmd string) string {
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", pveID, b64)
}
}

View File

@@ -0,0 +1,203 @@
package remote
import (
"context"
"encoding/base64"
"strings"
"testing"
"github.com/dtoro/oikos/internal/db"
"github.com/google/uuid"
)
// guestWrap and IsGuest are pure logic — always tested. The DB-backed
// resolvers are integration tests guarded by OIKOS_TEST_DATABASE_URL, the
// same convention as internal/scheduler/coverage_test.go.
func TestIsGuest(t *testing.T) {
cases := map[string]bool{
"lxc": true, "vm": true,
"proxmox-host": false, "workstation": false,
"service": false, "docker-container": false,
}
for typ, want := range cases {
if got := IsGuest(typ); got != want {
t.Errorf("IsGuest(%q) = %v, want %v", typ, got, want)
}
}
}
func TestGuestWrapLXC(t *testing.T) {
w := guestWrap("lxc", "132")
out := w("/opt/oikos/checks/cpu_check.sh 'svc'")
if !strings.Contains(out, "pct exec 132 -- bash -c ") {
t.Fatalf("lxc wrap must use pct exec: %q", out)
}
if strings.Contains(out, "qm guest exec") {
t.Fatalf("lxc wrap must not use qm: %q", out)
}
// The base64 payload must round-trip to the original command.
i := strings.Index(out, "echo ")
j := strings.LastIndex(out, " | base64 -d | bash")
if i < 0 || j < 0 || j <= i {
t.Fatalf("cannot locate base64 payload in %q", out)
}
dec, err := base64.StdEncoding.DecodeString(out[i+len("echo ") : j])
if err != nil {
t.Fatalf("decode payload: %v", err)
}
if string(dec) != "/opt/oikos/checks/cpu_check.sh 'svc'" {
t.Fatalf("round-trip mismatch: %q", string(dec))
}
}
func TestGuestWrapVM(t *testing.T) {
w := guestWrap("vm", "108")
out := w("uname -a")
if !strings.Contains(out, "qm guest exec 108") {
t.Fatalf("vm wrap must use qm guest exec: %q", out)
}
}
func TestResolveExecTargetUnsupported(t *testing.T) {
// No DB needed: an unsupported slug prefix errors before any query.
if _, err := ResolveExecTarget(context.Background(), nil, "service:gitea", DefaultUser); err == nil {
t.Fatal("expected error for unsupported target prefix")
}
}
// --- integration tests (require a real Postgres) ---
func newRemotePool(t *testing.T) *db.Pool {
t.Helper()
base := testDatabaseURL(t)
return createTestDB(t, base)
}
func testDatabaseURL(t *testing.T) string {
t.Helper()
u := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
if u == "" {
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
}
return u
}
func TestResolveHostPrefersLAN(t *testing.T) {
pool := newRemotePool(t)
ctx := context.Background()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'host:x','proxmox-host','x','active','{"lan_ip":"10.0.0.1","public_ipv4":"1.2.3.4","mesh":{"netbird":{"ip":"100.64.0.1"}}}'::jsonb,1,now(),now())`,
uuid.New())
addr, user, err := ResolveHost(ctx, pool, "host:x", DefaultUser)
if err != nil {
t.Fatalf("ResolveHost: %v", err)
}
if addr != "10.0.0.1" {
t.Errorf("addr = %q, want lan_ip 10.0.0.1", addr)
}
if user != "root" {
t.Errorf("user = %q, want root", user)
}
}
func TestResolveHostFallsBackToPublicIPv4(t *testing.T) {
// netbird-vps: no lan_ip, has public_ipv4 + mesh ip. Must prefer
// public_ipv4 — the scheduler container has no mesh interface.
pool := newRemotePool(t)
ctx := context.Background()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'host:vps','standalone-server','vps','active','{"public_ipv4":"82.165.190.79","mesh":{"netbird":{"ip":"100.122.165.149"}}}'::jsonb,1,now(),now())`,
uuid.New())
addr, _, err := ResolveHost(ctx, pool, "host:vps", DefaultUser)
if err != nil {
t.Fatalf("ResolveHost: %v", err)
}
if addr != "82.165.190.79" {
t.Errorf("addr = %q, want public_ipv4 (mesh unreachable from container)", addr)
}
}
func TestResolveExecTargetForCheckLXCRoutesViaHost(t *testing.T) {
// An LXC guest with a `hosts` edge to a proxmox host must resolve to the
// HOST's address (the host-hop target), wrapped as `pct exec`.
pool := newRemotePool(t)
ctx := context.Background()
hostID := uuid.New()
guestID := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'lxc:rclone','lxc','rclone','active','{"pve_id":"132"}'::jsonb,1,now(),now())`, guestID)
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at)
VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
et, err := ResolveExecTargetForCheck(ctx, pool, guestID, "lxc", DefaultUser)
if err != nil {
t.Fatalf("ResolveExecTargetForCheck: %v", err)
}
if et.Host != "192.168.8.77" {
t.Errorf("Host = %q, want proxmox host lan_ip 192.168.8.77 (host-hop)", et.Host)
}
out := et.Wrap("/opt/oikos/checks/cpu_check.sh")
if !strings.Contains(out, "pct exec 132") {
t.Errorf("guest wrap must use pct exec 132, got %q", out)
}
}
func TestResolveExecTargetForCheckHostIsDirect(t *testing.T) {
// A host-like target resolves to its own address with identity wrap.
pool := newRemotePool(t)
ctx := context.Background()
hid := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'ws:mini','workstation','mini','active','{"lan_ip":"192.168.178.182","user":"dtoro"}'::jsonb,1,now(),now())`, hid)
et, err := ResolveExecTargetForCheck(ctx, pool, hid, "workstation", DefaultUser)
if err != nil {
t.Fatalf("ResolveExecTargetForCheck: %v", err)
}
if et.Host != "192.168.178.182" {
t.Errorf("Host = %q, want 192.168.178.182", et.Host)
}
// Workstation's top-level `user` must be honored (the mac-mini fix).
if et.User != "dtoro" {
t.Errorf("User = %q, want dtoro (top-level user attr)", et.User)
}
if cmd := et.Wrap("uptime"); cmd != "uptime" {
t.Errorf("host wrap must be identity, got %q", cmd)
}
}
func TestResolveExecTargetForCheckServiceRoutesViaHostingGuest(t *testing.T) {
// A service has no address of its own; it must route through its hosting
// LXC via the provides edge, host-hopping through the LXC's proxmox host.
pool := newRemotePool(t)
ctx := context.Background()
hostID := uuid.New()
guestID := uuid.New()
svcID := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'lxc:gitea','lxc','gitea','active','{"pve_id":"104","lan_ip":"192.168.8.121"}'::jsonb,1,now(),now())`, guestID)
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'service:gitea','service','gitea','active','{}'::jsonb,1,now(),now())`, svcID)
// provides: lxc -> service; hosts: proxmox-host -> lxc
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'provides',now(),now())`, guestID, svcID)
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
et, err := ResolveExecTargetForCheck(ctx, pool, svcID, "service", DefaultUser)
if err != nil {
t.Fatalf("ResolveExecTargetForCheck for service: %v", err)
}
// Reaches the proxmox host (host-hop), wrapped as pct exec into the guest.
if et.Host != "192.168.8.77" {
t.Errorf("Host = %q, want proxmox host 192.168.8.77 (via provides->hosts)", et.Host)
}
if out := et.Wrap("p"); !strings.Contains(out, "pct exec 104") {
t.Errorf("service check must wrap as pct exec 104, got %q", out)
}
}

View File

@@ -0,0 +1,68 @@
package remote
import (
"context"
"fmt"
"math/rand"
"os"
"strings"
"testing"
"github.com/dtoro/oikos/internal/db"
"github.com/jackc/pgx/v5"
)
// createTestDB provisions a throwaway migrated database off baseURL, the same
// convention as internal/scheduler/coverage_test.go. The base URL must point
// at a Postgres superuser-capable connection.
func createTestDB(t *testing.T, baseURL string) *db.Pool {
t.Helper()
ctx := context.Background()
admin, err := pgx.Connect(ctx, baseURL)
if err != nil {
t.Fatalf("connect admin: %v", err)
}
dbName := fmt.Sprintf("oikos_rem_%08x", rand.Int63())
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
admin.Close(ctx)
t.Fatalf("create test db: %v", err)
}
admin.Close(ctx)
at := strings.LastIndex(baseURL, "/")
testURL := baseURL[:at+1] + dbName
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
testURL += baseURL[at+q:]
}
pool, err := db.New(ctx, testURL)
if err != nil {
t.Fatalf("connect test db: %v", err)
}
if err := pool.Migrate(ctx); err != nil {
t.Fatalf("migrate: %v", err)
}
t.Cleanup(func() {
pool.Close()
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
admin.Close(ctx)
}
})
return pool
}
func getenvOrDefault(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
t.Helper()
if _, err := pool.Exec(ctx, q, args...); err != nil {
t.Fatalf("exec %s: %v", q, err)
}
}

View File

@@ -85,7 +85,16 @@ func coverageSweep(ctx context.Context, pool *db.Pool) {
case !mon.Declared:
undeclared++
case mon.None():
// Explicitly unmonitorable. Nothing to say.
// Explicitly unmonitorable. Nothing to raise — but a type that
// USED to declare monitoring (e.g. dns-zone, [dns]→none) may have
// open `unmonitored` signals from before the change. They are no
// longer a gap, so close them; otherwise they linger forever,
// because resolveCoverageSignal only runs from the hasCheck path
// and a None() entity never gains a check.
if resolveCoverageSignal(ctx, pool, e.id) {
resolved++
slog.Info("scheduler: type now unmonitorable, resolving stale signal", "entity", e.slug)
}
case e.hasCheck:
if resolveCoverageSignal(ctx, pool, e.id) {
resolved++

View File

@@ -0,0 +1,25 @@
package scheduler
import (
"context"
"testing"
"time"
)
// The scheduler runs in Docker on macOS, whose VM does not route ICMP to the
// LAN — every ping check reported "down" for hosts that were demonstrably up.
// tcpReachable is the fallback that keeps "is it reachable" answerable.
func TestTCPReachableAnswersWhenICMPCannot(t *testing.T) {
ctx := context.Background()
// localhost:22 is open on this machine (sshd), and port 1 is not.
if !tcpReachable(ctx, "127.0.0.1", 22, 3*time.Second) {
t.Skip("no sshd on localhost — cannot exercise the positive case")
}
if tcpReachable(ctx, "127.0.0.1", 1, 1*time.Second) {
t.Error("port 1 should not be reachable")
}
// Defaults to 22 when unset, which is what checkdefaults' ping configs use.
if !tcpReachable(ctx, "127.0.0.1", 0, 3*time.Second) {
t.Error("port 0 should default to 22")
}
}

View File

@@ -23,6 +23,7 @@ import (
"github.com/dtoro/oikos/internal/db"
"github.com/dtoro/oikos/internal/db/sqlcgen"
"github.com/dtoro/oikos/internal/observability"
"github.com/dtoro/oikos/internal/remote"
"github.com/google/uuid"
"golang.org/x/sync/errgroup"
"golang.org/x/sys/unix"
@@ -111,12 +112,20 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
q := sqlcgen.New(pool)
start := time.Now()
result := executeCheck(ctx, cd)
result := executeCheck(ctx, pool, cd)
// Stamp the run before processing the result: due-ness must advance even
// when a check fails, or a permanently failing check would be re-run on
// every pass instead of at its declared interval.
if err := q.MarkCheckRun(ctx, cd.EntityID); err != nil {
// every pass instead of at its declared interval. last_health records THIS
// check's own verdict, which is what makes the aggregation below possible.
checkHealth := result.health
if checkHealth == "" {
checkHealth = "healthy"
}
if err := q.MarkCheckRun(ctx, sqlcgen.MarkCheckRunParams{
EntityID: cd.EntityID,
LastHealth: &checkHealth,
}); err != nil {
slog.Error("scheduler: mark check run", "entity", cd.EntitySlug, "error", err)
}
@@ -150,16 +159,12 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
if result.signalKind == "" || result.health == "healthy" {
resolveSignal(ctx, pool, cd.EntityID, targetID, cd.EntitySlug)
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
EntityID: targetID,
Health: "healthy",
LastCheckAt: &[]time.Time{time.Now()}[0],
Details: []byte(`{}`),
})
if prevHealth != "" && prevHealth != "healthy" {
emitSchedulerEvent(ctx, pool, "health.changed", targetID, "info",
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": "healthy"})
}
// NOT unconditionally "healthy": this check passing says nothing about
// the entity's other checks. Writing healthy here is what let one
// passing probe erase a genuine failure reported by another — and,
// alternating with a failing probe, produced 226 health flips an hour
// on a host that was fine throughout.
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
return
}
@@ -180,22 +185,52 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
return
}
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
EntityID: targetID,
Health: result.health,
LastCheckAt: &[]time.Time{time.Now()}[0],
Details: []byte(`{}`),
})
_ = sig
if prevHealth == "" || prevHealth == "healthy" {
emitSchedulerEvent(ctx, pool, "signal.raised", targetID, severity,
map[string]any{"slug": cd.EntitySlug, "kind": result.signalKind, "evidence": result.evidence})
}
if prevHealth != result.health {
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": result.health})
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
}
// applyAggregateHealth sets the target's health to the worst verdict across
// all of its enabled checks, and emits health.changed only when that aggregate
// actually moves.
//
// Health is a property of the entity, but each check only ever observes one
// facet of it — reachability, disk, a systemd unit. Letting whichever check
// finished last overwrite the entity's health meant a host with six checks
// reported whichever facet was sampled most recently, so one failing probe and
// five passing ones oscillated forever instead of settling on "degraded".
func applyAggregateHealth(ctx context.Context, pool *db.Pool, q *sqlcgen.Queries,
targetID uuid.UUID, checkSlug, prevHealth string) {
health, err := q.WorstHealthForTarget(ctx, &targetID)
if err != nil {
slog.Error("scheduler: aggregate health", "entity", checkSlug, "error", err)
return
}
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
EntityID: targetID,
Health: health,
LastCheckAt: &[]time.Time{time.Now()}[0],
Details: []byte(`{}`),
})
if prevHealth == health {
return
}
severity := "info"
switch health {
case "down":
severity = "critical"
case "degraded", "stale":
severity = "warning"
}
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
map[string]any{"slug": checkSlug, "from": prevHealth, "to": health})
}
// currentHealth reads the last recorded health for an entity, or "" if none.
@@ -217,7 +252,6 @@ func emitSchedulerEvent(ctx context.Context, pool *db.Pool, eventType string, en
// checkID matches how signals are keyed (UpsertSignal uses the check's own
// entity id); targetID is the observed entity whose status this affects.
func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UUID, slug string) {
q := sqlcgen.New(pool)
// Check if there's an open signal on this entity
tag, err := pool.Exec(ctx, `UPDATE signals SET state = 'resolved', updated_at = now()
WHERE entity_id = $1 AND state = 'raised'`, checkID)
@@ -227,14 +261,12 @@ func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UU
if tag.RowsAffected() > 0 {
emitSchedulerEvent(ctx, pool, "signal.resolved", targetID, "info",
map[string]any{"slug": slug})
slog.Info("scheduler: signal resolved", "entity", slug)
}
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
EntityID: targetID,
Health: "healthy",
LastCheckAt: &[]time.Time{time.Now()}[0],
Details: []byte(`{}`),
})
slog.Info("scheduler: signal resolved", "entity", slug)
// Deliberately does NOT write health. Resolving THIS check's signal says
// nothing about the target's other checks; the caller re-derives health
// from all of them. Forcing "healthy" here was a second path by which one
// passing probe erased another probe's genuine failure.
}
// checkResult bundles the outcome of a single check execution.
@@ -246,8 +278,10 @@ type checkResult struct {
err error
}
// executeCheck dispatches to the appropriate checker by kind.
func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
// executeCheck dispatches to the appropriate checker by kind. pool is needed
// by the ssh-script path, which resolves the target's execution endpoint
// (guests route through their Proxmox host; see internal/remote).
func executeCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
switch cd.Kind {
case "http":
return checkHTTP(ctx, cd)
@@ -257,12 +291,16 @@ func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) check
return checkDisk(ctx, cd)
case "cert-expiry":
return checkCertExpiry(ctx, cd)
case "vm-status":
return checkVMStatus(ctx, pool, cd)
case "ping":
return checkPing(ctx, cd)
case "ssh-script":
return checkSSHScript(ctx, cd)
return checkSSHScript(ctx, pool, cd)
case "backup-freshness":
return checkBackupFreshness(ctx, cd)
case "dns":
return checkDNS(ctx, cd)
default:
return checkResult{health: "unknown"}
}
@@ -448,6 +486,53 @@ func checkTCP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResu
return checkResult{health: "healthy"}
}
// checkDNS verifies a DNS name resolves, catching a stale or unreachable
// zone. It looks up NS records first (a zone always has NS), falling back to
// an A/AAAA lookup for hostnames. Uses the system resolver; for split-horizon
// correctness reserve an explicit `server` in the config.
func checkDNS(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
cfg := struct {
Name string `json:"name"`
Server string `json:"server"`
}{}
if len(cd.Config) > 0 {
_ = json.Unmarshal(cd.Config, &cfg)
}
if cfg.Name == "" {
return checkResult{health: "healthy"}
}
// Resolve via an explicit server when supplied (split-horizon), else the
// system default resolver.
lookup := func(q string) (int, error) {
r := &net.Resolver{}
if cfg.Server != "" {
r = &net.Resolver{PreferGo: true, Dial: func(ctx context.Context, network, _ string) (net.Conn, error) {
d := net.Dialer{Timeout: 5 * time.Second}
return d.DialContext(ctx, network, net.JoinHostPort(cfg.Server, "53"))
}}
}
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
ns, err := r.LookupNS(ctx, q)
if err == nil && len(ns) > 0 {
return len(ns), nil
}
addrs, err2 := r.LookupHost(ctx, q)
return len(addrs), err2
}
n, err := lookup(cfg.Name)
if err != nil || n == 0 {
return checkResult{
health: "down", signalKind: "dns",
evidence: fmt.Sprintf("DNS resolution failed for %q: %v", cfg.Name, err),
err: err,
}
}
return checkResult{health: "healthy"}
}
// checkDisk performs a disk usage check.
func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
cfg := struct {
@@ -502,6 +587,12 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
cfg := struct {
Host string `json:"host"`
// Dial is an optional explicit dial address (the TLS terminator's IP)
// for when the hostname doesn't resolve/reach from the scheduler — the
// container has no mesh interface and the host resolver doesn't know
// the split-horizon zone, so *.hubris.network dials Caddy's lab IP
// directly while SNI/cert-read still uses Host.
Dial string `json:"dial"`
Port int `json:"port"`
WarnDays int `json:"warn_days"`
CritDays int `json:"crit_days"`
@@ -522,9 +613,13 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
timeout = 30 * time.Second
}
addr := net.JoinHostPort(cfg.Host, fmt.Sprintf("%d", cfg.Port))
dialHost := cfg.Host
if cfg.Dial != "" {
dialHost = cfg.Dial
}
addr := net.JoinHostPort(dialHost, fmt.Sprintf("%d", cfg.Port))
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true}}
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true, ServerName: cfg.Host}}
conn, err := d.DialContext(ctx, "tcp", addr)
if err != nil {
return checkResult{
@@ -569,11 +664,61 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
return checkResult{health: "healthy", metrics: metrics}
}
// checkVMStatus reports whether a VM is powered on, via `qm status <pve_id>`
// run on its Proxmox host. This is the right reachability probe for a VM that
// blocks ICMP (haos) and has no guest agent: it doesn't need the VM's network
// at all — "status: running" means the VM is up. The command runs on the host
// (not inside the VM), so it uses the host's address with identity wrap.
func checkVMStatus(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
if cd.TargetID == nil {
return checkResult{health: "unknown", evidence: "vm-status needs a target VM"}
}
var pveID, hostAttr string
if err := pool.QueryRow(ctx,
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
*cd.TargetID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
return checkResult{health: "unknown", signalKind: "vm-status",
evidence: fmt.Sprintf("vm %s has no pve_id", cd.EntitySlug)}
}
hostSlug := remote.ResolveProxmoxHostSlug(ctx, pool, *cd.TargetID, hostAttr)
addr, user, err := remote.ResolveHost(ctx, pool, hostSlug, sshUser)
if err != nil {
return checkResult{health: "down", signalKind: "vm-status",
evidence: fmt.Sprintf("resolve proxmox host for %s: %v", cd.EntitySlug, err), err: err}
}
timeout := time.Duration(cd.TimeoutS) * time.Second
if timeout <= 0 {
timeout = 15 * time.Second
}
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
out, err := sshExec(ctx, addr, "22", user, "qm status "+pveID, timeout)
if err != nil {
return checkResult{health: "down", signalKind: "vm-status",
evidence: fmt.Sprintf("qm status %s on %s: %v", pveID, addr, err), err: err}
}
// `qm status <id>` prints "status: running" (or stopped/paused).
if strings.Contains(string(out), "status: running") {
return checkResult{health: "healthy"}
}
trimmed := strings.TrimSpace(string(out))
if trimmed == "" {
trimmed = "(no output)"
}
return checkResult{health: "down", signalKind: "vm-status",
evidence: fmt.Sprintf("%s not running: %s", cd.EntitySlug, trimmed)}
}
// checkPing performs an ICMP ping check using the system ping command.
func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
cfg := struct {
Host string `json:"host"`
Count int `json:"count"`
// Port for the TCP fallback below. Defaults to 22; set it for hosts
// that answer on something else (a Home Assistant VM has no sshd).
Port int `json:"port"`
}{}
if len(cd.Config) > 0 {
_ = json.Unmarshal(cd.Config, &cfg)
@@ -607,9 +752,26 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
output, err := cmd.Output()
if err != nil {
// ICMP failing does not mean the host is down — it may mean ICMP is
// simply unavailable from here. On this deployment the scheduler runs
// in Docker on macOS, whose VM network stack does not route ICMP to
// the LAN: loopback pings succeed, every LAN ping fails, and all seven
// ping checks reported "down" for hosts that were demonstrably up
// (including the Docker host itself). Under health aggregation that one
// broken probe was enough to drag each entity to down.
//
// The question this check exists to answer is "is it reachable", and
// ICMP is only one way to ask. Fall back to a TCP connect before
// concluding anything.
if tcpReachable(ctx, cfg.Host, cfg.Port, timeout) {
return checkResult{
health: "healthy",
metrics: map[string]float64{},
}
}
return checkResult{
health: "down", signalKind: "ping",
evidence: fmt.Sprintf("ping %s: %v", cfg.Host, err),
evidence: fmt.Sprintf("no ICMP or TCP response from %s: %v", cfg.Host, err),
err: err,
}
}
@@ -623,6 +785,24 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
return checkResult{health: "healthy", metrics: metrics}
}
// tcpReachable reports whether a TCP handshake completes, used as the
// reachability answer when ICMP is unavailable rather than unanswered.
func tcpReachable(ctx context.Context, host string, port int, timeout time.Duration) bool {
if port == 0 {
port = 22
}
if timeout <= 0 {
timeout = 5 * time.Second
}
d := net.Dialer{Timeout: timeout}
conn, err := d.DialContext(ctx, "tcp", net.JoinHostPort(host, strconv.Itoa(port)))
if err != nil {
return false
}
conn.Close()
return true
}
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
func parsePingLatency(output []byte) float64 {
@@ -637,8 +817,16 @@ func parsePingLatency(output []byte) float64 {
return val
}
// checkSSHScript executes an allowlisted script on a remote host via SSH.
func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
// checkSSHScript executes an allowlisted script on a remote target via SSH.
//
// Routing follows the canonical access model (internal/remote): an LXC or VM
// is NEVER SSH'd into directly — it is reached through its Proxmox host via
// `pct exec`/`qm guest exec`, so a guest needs no lan_ip, sshd, or authorized
// key of its own. Hosts and workstations are reached by direct SSH, resolved
// live so a workstation's login (mac-mini: `user: dtoro`) is honored without
// a re-seed. Services and other entities fall back to the host address baked
// into check config at seed time (their hosting container's address).
func checkSSHScript(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
cfg := struct {
Host string `json:"host"`
Port int `json:"port"`
@@ -656,12 +844,6 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
if cfg.Host == "" || cfg.Script == "" {
return checkResult{health: "healthy"}
}
if cfg.Port == 0 {
cfg.Port = 22
}
if cfg.User == "" {
cfg.User = sshUser
}
if !allowlistedScript(cfg.Script) {
return checkResult{
@@ -684,13 +866,51 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
// the remote command. The script name itself is allowlisted above.
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
}
port := strconv.Itoa(cfg.Port)
output, err := sshExec(ctx, cfg.Host, port, cfg.User, scriptPath, timeout)
// Resolve the execution endpoint. The resolver handles every target kind:
// LXC/VM host-hop via pct/qm exec; hosts/workstations direct at their own
// address; services route through their hosting compute entity (via the
// provides edge) so a service check reaches the right machine with the
// right user instead of baking an LXC lan_ip and SSHing it as root.
host, port, user := cfg.Host, strconv.Itoa(oru(cfg.Port, 22)), orStr(cfg.User, sshUser)
wrap := func(cmd string) string { return cmd }
targetType := ""
if cd.TargetType != nil {
targetType = *cd.TargetType
}
// target_type was omitted by older writeCheck inserts, so resolve it from
// the target entity when the column is blank — otherwise the guest routing
// below (IsGuest) never triggers and a guest check falls back to its baked
// (often mesh-only) address.
if targetType == "" && cd.TargetID != nil {
if err := pool.QueryRow(ctx, "SELECT type FROM entities WHERE id = $1", *cd.TargetID).Scan(&targetType); err != nil {
targetType = ""
}
}
if cd.TargetID != nil && targetType != "" {
et, err := remote.ResolveExecTargetForCheck(ctx, pool, *cd.TargetID, targetType, sshUser)
if err != nil {
if remote.IsGuest(targetType) {
return checkResult{
health: "down", signalKind: "ssh-script",
evidence: fmt.Sprintf("route guest %s: %v", cd.EntitySlug, err), err: err,
}
}
// Non-guest: log the resolution failure so an opaque ssh "down"
// doesn't hide that the real cause was host/user resolution, then
// fall back to the baked config below.
slog.Warn("scheduler: target resolution failed, using baked config",
"entity", cd.EntitySlug, "target_type", targetType, "error", err)
} else {
host, port, user, wrap = et.Host, "22", et.User, et.Wrap
}
}
output, err := sshExec(ctx, host, port, user, wrap(scriptPath), timeout)
if err != nil {
return checkResult{
health: "down", signalKind: "ssh-script",
evidence: fmt.Sprintf("ssh %s:%s %s: %v", cfg.Host, strconv.Itoa(cfg.Port), cfg.Script, err),
evidence: fmt.Sprintf("ssh %s:%s %s: %v", host, port, cfg.Script, err),
err: err,
}
}
@@ -728,6 +948,20 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
}
}
// oru returns v when nonzero, else def. orStr returns v when non-empty, else def.
func oru(v, def int) int {
if v != 0 {
return v
}
return def
}
func orStr(v, def string) string {
if v != "" {
return v
}
return def
}
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
func allowlistedScript(name string) bool {

View File

@@ -0,0 +1,28 @@
-- 027_check_last_health.up.sql
-- Aggregate an entity's health across its checks instead of last-writer-wins.
--
-- runCheck wrote entity_status.health on every check completion, so an
-- entity's health was simply whichever of its checks finished most recently.
-- host:hubris has 6 checks, host:strong 6 — one failing probe alternating with
-- five passing ones produced a permanent flap: 226 health.changed events for
-- host:strong in a single hour, oscillating down/healthy, while the host was
-- fine the whole time.
--
-- On this fleet the trigger is a known false positive: the scheduler's network
-- vantage point cannot ICMP host:strong, so its ping check fails while every
-- ssh-script check succeeds. Under last-writer-wins that one probe was enough
-- to declare the whole host down, twice a minute.
--
-- Storing each check's own verdict lets entity health be derived as the worst
-- current result across that entity's enabled checks — so a single failing
-- probe degrades the entity honestly without erasing what the other five say,
-- and a passing probe cannot mask a genuine failure elsewhere.
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_health TEXT;
COMMENT ON COLUMN check_defs.last_health IS 'This check''s own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target''s enabled checks.';
-- The aggregation reads every enabled check for one target on each completion.
CREATE INDEX IF NOT EXISTS idx_check_defs_target_health
ON check_defs (target_id)
WHERE enabled AND target_id IS NOT NULL;

View File

@@ -0,0 +1,75 @@
-- 028_relationship_blast_direction.up.sql
-- Make blast_radius answer the question it is named after.
--
-- blast_radius walked source_id -> target_id for every relationship type. But
-- which end of an edge is the DEPENDENT differs per type:
--
-- machine --hosts--> container if the machine dies, the container dies
-- -> dependent is the TARGET (forward)
-- service --depends-on--> service if the target dies, the SOURCE breaks
-- -> dependent is the SOURCE (backward)
-- ingress --routes-to--> service if the service dies, the route 502s
-- -> dependent is the SOURCE (backward)
-- document --documents--> entity neither breaks the other
-- -> no runtime dependency at all
--
-- Walking everything forwards meant the answer was right for `hosts` and
-- `provides` and wrong for every backward edge, while `documents`, `involves`
-- and `targets` (2,800+ edges of pure bookkeeping) polluted the result with
-- tasks and executions that cannot "break".
--
-- Direction is therefore a property of the relationship type, declared in
-- seeds/ontology.yaml — the same shape as the `monitoring:` declaration on
-- entity types.
--
-- forward : if the SOURCE fails, the TARGET is affected
-- backward : if the TARGET fails, the SOURCE is affected
-- none : no runtime dependency (default — bookkeeping and documentation)
--
-- Defaulting to 'none' is deliberate: an undeclared edge contributes nothing
-- rather than silently producing a wrong answer, which is how the old
-- everything-is-forward behaviour went unnoticed.
ALTER TABLE relationship_types
ADD COLUMN IF NOT EXISTS blast_direction TEXT NOT NULL DEFAULT 'none'
CHECK (blast_direction IN ('forward', 'backward', 'none'));
COMMENT ON COLUMN relationship_types.blast_direction IS
'Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().';
-- Walk the dependency graph in the direction each edge type declares.
--
-- Returns everything that is affected when start_id fails, with the number of
-- hops. Cycles are guarded by the path array, as before.
CREATE OR REPLACE FUNCTION blast_radius(start_id UUID, max_depth INT DEFAULT 3,
rel_types TEXT[] DEFAULT NULL)
RETURNS TABLE(entity_id UUID, depth INT) AS $$
WITH RECURSIVE walk AS (
SELECT start_id AS entity_id, 0 AS depth, ARRAY[start_id] AS path
UNION ALL
SELECT next_id, w.depth + 1, w.path || next_id
FROM walk w
JOIN LATERAL (
-- forward: this entity is the source, so the target depends on it
SELECT r.target_id AS next_id
FROM relationships r
JOIN relationship_types rt ON rt.name = r.type
WHERE r.source_id = w.entity_id
AND r.valid_to IS NULL
AND rt.blast_direction = 'forward'
AND (rel_types IS NULL OR r.type = ANY(rel_types))
UNION ALL
-- backward: this entity is the target, so the source depends on it
SELECT r.source_id AS next_id
FROM relationships r
JOIN relationship_types rt ON rt.name = r.type
WHERE r.target_id = w.entity_id
AND r.valid_to IS NULL
AND rt.blast_direction = 'backward'
AND (rel_types IS NULL OR r.type = ANY(rel_types))
) nxt ON TRUE
WHERE w.depth < LEAST(max_depth, 5)
AND NOT nxt.next_id = ANY(w.path)
)
SELECT entity_id, MIN(depth) FROM walk GROUP BY entity_id;
$$ LANGUAGE sql STABLE;

View File

@@ -0,0 +1,33 @@
-- 029_plan_generation_relative_seq.up.sql
-- Make plan-step seq generation-relative: 1..N within each
-- (session_id, generation). Before this, seq was globally increasing across
-- generations (gen1: 1..6, gen2: 7..12), so the model's 1-based
-- update_plan_step calls — which the prompt and schema explicitly tell it to
-- use — landed on superseded gen-1 rows after a re-plan while the live gen-2
-- work went unrecorded (or, worse, resurrected a `replaced` row as `done`).
-- The addressing key is now (session_id, generation, seq); updatePlanStep
-- resolves against MAX(generation), so a 1-based seq always maps to the
-- CURRENT plan. See plans/2026-07-30-session-review-plan-drift-and-dead-
-- activity-panel.md P0.1.
-- Renumber existing rows so seq resets to 1..N per (session, generation),
-- preserving each generation's step order.
WITH ranked AS (
SELECT id,
ROW_NUMBER() OVER (
PARTITION BY session_id, generation
ORDER BY seq, created_at
) AS new_seq
FROM session_plan_steps
)
UPDATE session_plan_steps s
SET seq = ranked.new_seq
FROM ranked
WHERE s.id = ranked.id AND s.seq <> ranked.new_seq;
-- (session_id, seq) is no longer unique once seq resets per generation; the
-- store resolves via (session_id, generation, seq). Drop the old composite
-- index (it now collides on seq) and add the generation-scoped unique index.
DROP INDEX IF EXISTS idx_plan_steps_session;
CREATE UNIQUE INDEX IF NOT EXISTS idx_plan_steps_session_gen_seq
ON session_plan_steps (session_id, generation, seq);

View File

@@ -0,0 +1,8 @@
-- 030_plan_step_replaced_reason.up.sql
-- Add replaced_reason to session_plan_steps so the agent must explain why
-- a step was replaced (wrong_diagnosis, scope_change, blocked, superseded,
-- operator_override) rather than silently replacing entire plans. The column
-- is also set by bulk-replace operations (proposePlan, setGoal, reopenSession)
-- for auditability.
ALTER TABLE session_plan_steps ADD COLUMN IF NOT EXISTS replaced_reason TEXT;

View File

@@ -47,6 +47,24 @@ Read-only commands auto-run (no approval). Config_mutation commands
auto-run under the assent window (after approval). Destructive commands
always need explicit typed confirmation.
**Never mark a step `done` if its tool calls errored.** If `run` timed out,
`update_entity_attributes` returned "not found", `create_relationship` returned
"source entity not found", or any tool returned an error — the step is NOT done.
Diagnose the error, try an alternative (e.g. use `create_entity` when
`update_entity_attributes` reports the entity doesn't exist), and only advance
to `done` when the step's intended work actually completed. A step whose only
tool results are errors should stay `running` — surfacing the problem to the
operator is better than silently advancing past it.
**Complete or skip steps — don't replace silently.** Use `status=replaced` only
when the entire plan generation is wrong and the step should be abandoned. When
you replace a step, provide `replaced_reason` with the cause
(`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`).
Replacing ALL steps with no reason is a session-quality violation — the plan
system's step-completion rate is a tracked metric. Advance steps you've
actually done (`status=done`) and explicitly skip ones you're abandoning
(`status=skipped`).
### 6. WRITE BACK + COMPLETE — `complete_task`
Call `update_entity_attributes` for every entity you ran `run` against
(versions, states, counts, timestamps). Call `create_relationship` for any
@@ -59,6 +77,15 @@ is a pure-DB Q&A that called *no* `run` at all (only get_entity/list_lxcs/
search_knowledge): answer directly, `complete_task` with a one-line summary,
no writeback needed.
**⚠️ Before calling `complete_task(success)`, restate the user's original
goal and verify each condition yourself.** "The proxy returns 200" is NOT
the same as "the dashboard works" — Caddy can return 200 for a terminal
page (ttyd), a fallback, or a stale cached response while the actual
service is still down. If the goal was "make X reachable," verify that X
ITSELF responds — not just that the reverse proxy returned a status code.
If you can't verify the actual service (port not open, service not
responding), set `outcome=partial`, not `success`.
`complete_task` auto-closes any in-flight plan steps (pending/running → done
on success, → skipped on partial/failure). You do NOT need to call
`update_plan_step` for every step right before completing — once your work
@@ -85,6 +112,13 @@ generation — the panel will show it as a new list), execute, write back,
the panel in your reply. Never re-run `run` just to fix a display mismatch.
- Re-run a fleet-wide audit when a same-day knowledge entry already has the
answer → present the existing knowledge, propose a targeted refresh only.
- Pivot to a subsystem unrelated to the user's expressed goal without asking →
when investigation leads to a different subsystem or root cause (e.g.
debugging DHCP reservations when the goal was "make the dashboard reachable"),
call `session_questions` with the discovery and options BEFORE taking action.
Example: "The dashboard hasn't started since July 19 — this predates my work.
Do you want me to debug the dashboard service [A], skip it and stabilize the
current state [B], or stop here [C]?"
## Source of truth
@@ -205,6 +239,21 @@ disappear.
`list_lxcs` answers the same question in one call. Use it.
- When a bulk tool's summary isn't enough for a specific entity, call the
per-entity tool for that one entity — not for every entity in the fleet.
- **Cap pre-plan exploration:** prefer `list_entities(limit)` +
`get_entity_knowledge` (context for one entity, one call) over N+1
`get_entity`/`get_relations` chains. If you've already called
`get_entity_knowledge(slug)` and need more, call `get_entity(slug)` +
`get_relations(slug)` — not `list_entities` without a limit scanning the
whole entity table.
- **Group parallel reads:** `get_entity_knowledge`, `search_knowledge`,
`get_entity`, and `get_relations` are all read-only DB calls that can
be batched in a single tool-call block. Do not sequentialize them one
per turn when they are independent.
- **Source-reading on prod (`run cat/grep/find /opt/…`) is NOT the way to
learn how the platform works.** The MCP tools ARE the interface. If you
need to understand a check lifecycle or a scheduler behavior, search
`search_knowledge("oikos check lifecycle")` or ask the operator — do
not treat the prod host as a code repository you grep.
## Policy awareness
@@ -355,6 +404,26 @@ port is busy, find a free one. Only surface to the operator if you've tried
reasonable alternatives and none worked. An error in one step is not a reason
to stop the entire turn — it's a reason to try a different approach.
**When you hit a genuine missing capability — STOP and ask, don't bypass:**
If a tool returns `entity … not found` when you're trying to create something
(a check, an ingress, a cert, a new service), the entity doesn't exist yet —
use `create_entity`. If you need to retire/delete an entity, use
`set_entity_state`. If you need to remove a relationship, use
`end_relationship`. If NONE of these fit and you truly lack a tool, **tell the
operator directly: "I need to X, but no MCP tool does that — can you create it
via the API?"** Do NOT pivot to `run find/grep/cat` on `/opt/homelab-context`
to reverse-engineer how the platform works — MCP tools are the interface, not
the prod source tree.
**Self-grounding — use the DB, don't invent:**
- `run` targets must be `host:<slug>`, `lxc:<slug>`, or `vm:<slug>` — never
`ws:`, raw container names, or Docker Compose service aliases.
- Never invent an IP address or subnet. Query `get_entity("service:oikos")` for
the real API address, `get_entity("host:<name>")` for a host's real LAN IP,
`list_lxcs` for container addresses. The DB is authoritative; your guess is
wrong (the homelab has multiple subnets — `192.168.8.0/24`, `192.168.178.0/24`,
etc. — and guessing the wrong one wastes turns).
**A hung command is not a failed command — investigate before retrying.**
If a `run` call times out or returns "ERROR" (e.g. SSH killed, signal,
gateway timeout), DO NOT immediately retry the same command with different
@@ -391,6 +460,21 @@ before producing the plan. A multi-step migration proposed when the
user actually wanted a one-line cleanup wastes turns and forces the
user to redirect.
**Scope gate — ask before chasing unrelated subsystems.** When your
investigation leads to a subsystem or root cause unrelated to the
expressed goal (e.g. the user asked "why is X unreachable?" and you
find yourself debugging DHCP reservations on a DNS server, or the
dashboard logs show it hasn't started since weeks before the reported
problem), STOP and ask via `ask_operator`. Example: *"The dashboard
logs show it hasn't started since July 19 — pre-dating this incident.
Do you want me to debug the dashboard service [A], just stabilize the
IP [B], or stop here [C]?"* Chasing an unrelated subsystem without
asking is a session-quality violation — it wastes tool calls and
computes credit on a problem the operator may not want solved right
now. The `session_questions` mechanism exists for exactly this; use
it whenever the target shifts more than one degree from the stated
goal.
**Multi-goal sessions: summarize the arc, not just the last goal.**
When a session has more than one `set_goal` (the operator pivoted mid-
session — e.g. "actually, just keep ludo-library"), the final

View File

@@ -0,0 +1,224 @@
# 2026-08-03 — Session review: `service:haos` monitoring + agent capability gaps
**Status:** Plan (audit complete; ready to implement).
**Reviewed session:** `23da10db-46a9-444c-bbde-ca9457bd9087` — *"Work out what
monitoring checks service:haos should have and configure them."*
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
`agent_activity`/`session_plan_steps` on the prod mac-mini (oikos prod runs here
in docker compose project `oikos`; gateway `:8092`), cross-referenced with the
code paths in `internal/mcp`, `internal/httpapi`, `internal/policy`,
`internal/checkdefaults`, `internal/db/seed.go`.
---
## 1. Session audit (objective vs outcome)
| Dimension | Finding |
|---|---|
| Objective | Determine + configure monitoring checks for `service:haos` (HAOS VM 108, `home.hubris.network`, `192.168.8.101:8123`). |
| Outcome | ❌ **Failed/stuck.** `status=executing`, `outcome=null` ~4 min after last activity (UTC); never reached a terminal state. Only the *existing* `check:vm-status:vm:haos:0` stub got populated; the three **new** checks (`http:service`, `http:ingress`, `cert-expiry`) and their `ingress:`/`cert:` entities were never created. |
| Tool calls | **116** (vs the >30 N+1 failure signature). ~45 redundant `list_entities`/`get_entity`/`get_relations`, then a ~15-min storm of `run` doing `find`/`grep`/`cat` on prod source. |
| Plan | 8 steps proposed; steps 14 genuinely done; **step 5 falsely marked "done"** after both its tool calls errored `entity not found`; steps 68 never started. |
| Operator friction | 3 manual interventions: `status`, `proceed`, *"why dony you use the mcp?"*; plus a **44-minute approval stall** (19:52→20:36) on two trivial reachability curls. |
| Severity | **blocker** (capability gap) + **friction** (classifier, plan-state, reaping). |
### Timeline (UTC)
- **19:4519:50** — read-only exploration; `run` correctly blocked ("No plan… call set_goal then propose_plan"). Good guard.
- **19:50** — `propose_plan` (8 steps).
- **19:52** — two `curl … -o /dev/null -w '%{http_code}'` reachability probes → both classified `config_mutation` → one queued for approval (`019fc92e…`), second blocked ("approval already pending").
- **19:52 → 20:36 (44 min)** — idle, waiting on operator approval.
- **20:36** — approval granted ("auto via assent window"); both curls → 200/200.
- **20:37** — step 4 ✅: populated `check:vm-status:vm:haos:0` + `checks` edge.
- **20:37:58** — step 5 ❌: `update_entity_attributes("check:http:service:haos:0")`**`entity not found`**; `create_relationship`**`source entity not found`**. *(There is no create tool.)*
- **20:3820:47** — spiral: `search_knowledge` (empty), then `run find/grep/cat` across `/opt/homelab-context/**/*.go` to reverse-engineer check creation. Reads `checkdefaults.go`, `monitoring.go`, `default_checks.go`, `checks.go`, `coverage.go`.
- **20:42** — sets `service:haos` `monitoring: ["http"]` via `update_entity_attributes`, hoping `checkdefaults.Ensure()` auto-generates. **It does not** (see A2).
- **20:4220:50** — tries to reach the REST API directly: `psql` on hubris (cmd 127), `docker exec` on hubris (docker absent), `curl http://192.168.178.25:8090` (wrong subnet; real net is `192.168.8.x`; exit 7), `curl http://oikos-api:8090` (MCP routes to hubris which can't resolve the mac-mini docker alias; 30s timeouts ×2), `ssh root@192.168.178.25` (no route). Final `update_entity_attributes` on `ingress:`/`cert:``not found`.
- **20:50:38** — last activity: a failed 30s `run`. Session goes silent, never terminates.
---
## 2. Root-cause findings (with code evidence)
### A1 — No entity-creation capability in the MCP toolset *(the blocker)*
`internal/mcp/tools.go` registers **37 tools**; the only entity-mutation surface is
`update_entity_attributes` (merge into an **existing** entity) and
`create_relationship` (needs **existing** source+target). Neither can create a new
entity. The capability **does** exist at the HTTP layer — `CreateEntity`
(`internal/httpapi/impl.go:865`, `POST /api/v1/entities`) — it is simply not exposed
to the agent. Every "set up / onboard / configure entity X" task that needs a new
check/ingress/cert/service hits this wall.
### A2 — MCP `update_entity_attributes` bypasses `ensureDefaultChecks`
`ensureDefaultChecks` (`internal/httpapi/default_checks.go:20`) is invoked **only**
from the HTTP handlers: `CreateEntity` (`impl.go:1012`) and `PatchEntity`
(`impl.go:1280`). `grep ensureDefaultChecks internal/mcp/`**no matches**: the MCP
tool writes attributes straight to the store, so flipping `service:haos`
`monitoring:["http"]` never regenerated its checks. The agent's fallback strategy
was structurally doomed via MCP.
### A3 — `-o /dev/null` curl idiom misclassified as `config_mutation`
`internal/policy/command.go:106` `curlMutateRe` matches `(?:^|\s)-(?:d|F|T|o)\b` — so
`-o` (output-file) is treated as mutation. The canonical read-only reachability probe
`curl -sS -o /dev/null -w '%{http_code}' …` therefore escalates to approval. This is
the entire 44-minute stall. (`curlIsReadOnly` at `command.go:307` only passes for GET
with no `-o`/`-d`/`-X`/`>`.) A pure GET that discards the body is the single most
common health probe and shouldn't need approval.
### A4 — No platform self-knowledge doc for the check lifecycle
`search_knowledge("create check entity how to add new check monitoring")` → empty.
The agent re-derived the whole mechanism from source on prod (~15 min, dozens of
`run`). There is no agent/operator runbook explaining: check slugs are
`check:<kind>:<target>:<n>`; `check_defs` are derived from the type's `monitoring`
spec by `checkdefaults.Ensure`; Ensure runs at **seed/deploy** and on **HTTP
create/patch**, not via MCP.
### A5 — False plan progress (step marked done on failure)
At 20:37:58 both tool calls for step 5 returned `error: entity not found`, yet the
agent advanced step 5→`done`. Plan-state integrity hole: a step whose actions error
should not transition to `done`. (`session_plan_steps` confirms seq 5 = `done`.)
### A6 — No "missing-capability" escalation; self-grounding failures
On detecting the dead-end (no create tool) the agent never told the operator *"I lack
a tool to create entities — please create them"*; instead it tried to bypass its own
platform. Grounding errors: invented IP `192.168.178.25` (real LAN is `192.168.8.x`),
ran `run` against `ws:mac-mini` ("unsupported target — must be host:/lxc:/vm:"),
assumed `docker` exists on hubris, assumed the docker-alias `oikos-api` resolves from
hubris. The agent didn't query `get_entity("service:oikos")` for the real address.
### A7 — Sessions never reap from `executing`
Last activity 20:50; status still `executing` with no turn running. There is no
idle-timeout / abandoned transition when a turn ends without resolution. (Fleet-wide:
176 done / 9 failed / 1 executing; the 9 prior failures are pre-v0.15.0, mostly
approval-stalls and entity-not-found — same families.)
### A8 — N+1 tool fan-out (116 calls)
Dozens of redundant `list_entities`/`get_entity`/`get_relations` before proposing a
plan, plus the source-reading `run` storm. Above the >30-per-turn signature; indicates
weak bulk-tool use and under-constrained exploration before planning.
---
## 3. Improvement plan (ordered)
**Scope decision (confirmed with operator):** general `create_entity` MCP tool **+
wire regen** — solves this case and the 67-entity blast radius (§4).
### Task 1 — `create_entity` MCP tool *(fixes A1; the centerpiece)*
- Register a new tool `create_entity(slug, type, name, attributes?)` in
`internal/mcp/tools.go` that **reuses** `httpapi.CreateEntity`
(`impl.go:865`) / the same store path — do not hand-roll. It must run
`ensureDefaultChecks` (free, since it goes through the create path).
- **Approval policy:** no approval required for the entity itself — it mutates the
knowledge graph, matching the existing no-approval stance of
`update_entity_attributes`/`create_relationship`/`upsert_knowledge`. (Derived checks
are safe/read-side; if a check kind is ever deemed mutating, gate *that* in the
scheduler, not here.)
- Validate `type` against `entity_types`; reject unknown slugs/types with a clear
error. Idempotent on existing slug (return the existing entity, mirroring the HTTP
`ETag`/conflict behavior).
- Expose to the agent via the tool-list build path used by `cmd/nomos/agent.go`.
### Task 2 — MCP `update_entity_attributes` triggers `ensureDefaultChecks` *(fixes A2)*
- After the attribute merge in the MCP handler, call `ensureDefaultChecks` with the
post-merge entity (same args as `impl.go:1280`). This makes "set monitoring → checks
regenerate" work via MCP, matching HTTP semantics.
- Mind the `default_checks.go:14-19` caveat: a service whose address comes from its
host edge may still produce no checks until the hosting edge exists — log/return
that as an explicit result so the agent knows to create the edge next.
### Task 3 — Classifier: read-only `curl` with `-o /dev/null` *(fixes A3)*
- In `internal/policy/command.go` `curlIsReadOnly`, treat `-o /dev/null` (and
`--output /dev/null`) as read-only — it's a no-op sink. Keep `-o <realpath>` as
mutation. Add `TestClassifyCommand_CurlDevNull_ReadOnly` next to the existing
`TestClassifyCommand_CurlPipeSh_ConfigMutation`.
- Coach complement: in `nomos/SOUL.md`, note that reachability probes should use
`curl -I` or `-o /dev/null` GETs (now read-only) rather than POSTs.
### Task 4 — Plan-state integrity: don't mark `done` on errored actions *(fixes A5)*
- In `cmd/nomos` (`agent.go`/`tasks.go` where `update_plan_step` is emitted), a step
whose turn ended with only error/`not-found` tool results must **not** auto-advance
to `done`; leave it `running`/`blocked` and surface the failure to the operator.
Minimal: if every tool call in the step returned an `error:*` result, hold the step.
### Task 5 — Stuck-session reaping *(fixes A7)*
- Add an idle sweep (extend the existing continuation/idle worker in `cmd/nomos`) that
transitions a session from `executing``failed` (or a new `stuck`) when no turn has
run for N minutes and no approval is pending. Emit an event so the UI (F3 terminal
handling) clears the spinner. Pick N (recommend 30 min) — confirm in review.
### Task 6 — Missing-capability escalation + grounding *(fixes A6)*
- `nomos/SOUL.md`: when a mutation tool returns `entity … not found` on a create
intent, the agent must **stop and ask the operator** (or now use `create_entity`)
rather than pivot to `run`/SSH/API-bypass. Forbidden: inventing IPs/subnets; instead
`get_entity("service:oikos")` for the real API address. `run` targets must be
`host:/lxc:/vm:` slugs (state the contract explicitly).
### Task 7 — Runbook: "how checks work / how to add monitoring" *(fixes A4)*
- Upsert a knowledge doc (via `upsert_knowledge`, linked to the `agent:nomos` and
`document:infrastructure/monitoring` entities) covering: check slug grammar,
`checkdefaults.Ensure` triggers (seed + HTTP create/patch, now also MCP), the
`monitoring` per-entity override, the host-edge caveat, and the canonical way to add
monitoring to an entity (create/patch entity → checks derive).
### Task 8 — (Lower priority) exploration budget / bulk-tool use *(A8)*
- `nomos/SOUL.md`: prefer `list_entities(limit)` + `get_entity_knowledge` bulk calls
over N+1 `get_entity`/`get_relations` fans; cap pre-plan exploration. Optional
guardrail in `agent.go` (warn at >N same-tool calls per turn).
### Recommended sequence
1 → 2 → 3 → 4 → 7 → 5 → 6 → 8. (1+2 unblock the whole task class; 3 kills the
approval stall; 4+5 fix state integrity; 7 is cheap leverage; 6+8 are persona
hardening.)
---
## 4. Uncovered cases — the capability-gap blast radius
The existing F1F8 plans (`2026-08-03-nomos-chat-reliability-and-ux-audit.md`,
shipped v0.15.0) and the turn-scheduler review cover **only** UI / streaming / turn
serialization / connection UX. **None** addresses agent *capability* or
MCP↔HTTP integration. This session exposes the uncovered class:
- **67 entities currently have no `check:` relationship** (DB query): 40 `lxc`, 26
`service`, 1 `vm`. Any "add monitoring to X" task fails identically until Tasks 1+2.
- **Whole task families blocked by the no-create gap:** onboarding a new host/LXC/VM,
declaring a new service/ingress/cert/dns, adding any check that doesn't already
exist, registering a relationship target that doesn't exist yet. All currently
require an operator to hand-edit `seeds/inventory.yaml` and re-seed.
- **MCP↔HTTP semantic drift (generalize A2):** audit other MCP mutation tools for
side-effects that the HTTP handlers perform but the MCP path skips (check regen,
drift-flagging, audit fields, idempotency). Each is a latent "agent did the right
thing but nothing happened" bug.
- **Classifier read-only false-positives (generalize A3):** beyond `-o /dev/null`,
review other common read-only idioms that escalate (`curl` with benign flags,
compound read-only commands) — friction compounds into approval stalls and stuck
sessions.
- **No terminal/`stuck` reaping (generalize A7):** any turn that ends unresolved
leaves the session `executing` forever; the UI never shows "done/failed".
---
## 5. Validation
- **Task 1/2:** `go test ./internal/mcp/... ./internal/httpapi/...` — new test creates
`check:http:service:haos:0` via `create_entity`, asserts the entity exists **and**
that a `check_def` row was derived; then `update_entity_attributes(service:haos,
monitoring:["http"])` via MCP and assert checks regenerate (currently absent).
- **Task 3:** `go test ./internal/policy/` — `curl -sS -o /dev/null -w '%{http_code}'
URL` ⇒ `read_only`; `curl -o /tmp/x URL` ⇒ `config_mutation`.
- **Task 4:** `cmd/nomos` test — a step whose only tool result is `error:*` stays
non-`done`.
- **Task 5:** idle-sweep test — session with no turn for N min and no pending approval
⇒ `failed` (+ event emitted).
- **End-to-end re-run:** replay the haos goal against a local nomos; expect the three
checks + `ingress:`/`cert:` entities created in <15 tool calls with **zero**
approvals and a `done` outcome.
## 6. Out of scope / open questions
- Whether `create_entity` for sensitive types (e.g. `secret`, `key`) should require
approval even though it's graph-only — recommend: same no-approval stance now, add
type-specific gating later if abused.
- The exact stuck-reap window N (recommend 30 min) and whether to introduce a distinct
`stuck` status vs reuse `failed`.
- Whether to also expose a `delete_entity`/`retire_entity` MCP tool (not needed for
this case; lifecycle retirement is a separate flow).

View File

@@ -1,5 +1,9 @@
# 2026-07-21 Chat window full polish
**Status:** Implemented. Streaming affordance, inline tool rendering, message
timestamps, code-copy buttons, and per-session store isolation all landed in
`web/src/lib/components/ChatThread.svelte` + the chat stores (v0.8.x0.10.x).
## Context
After fixing the streaming reactivity bug and merging the double thinking

View File

@@ -0,0 +1,413 @@
# Plan: Make health reflect reality + complete the knowledge graph
Status: Implemented (v0.14.x0.16.x). Shipped across `c9a00a9` (per-entity
monitoring override), `a3914eb`/`8eb1ca2` (process check opt-in + probe_unit),
`0929c17` (discover_infra_drift), and the vm-status/layered-probe/route-via-
proxmox-host decisions now in project memory. Created 2026-07-29.
## Context
`ws:mac-mini` reports health `down` despite being the healthy control-plane host.
Investigation showed the problem is systemic, not local: **49 enabled checks report
`down`**, almost all `ssh-script`, because the resource/updates probes assume
**scripts are deployed at `/opt/oikos/checks/` AND root SSH works on every target**
both false for macOS, non-enrolled LXCs, and mesh-only entities. The knowledge graph
also has real gaps (unmodeled TLS certs, empty `skills` table, seed drift, a capped
topology view).
The DB is the source of truth; live state was verified via the REST API
(`Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN`, token in `oikos-api-1` container env)
and `docker exec oikos-postgres-1 psql`. Direct psql access is available for cleanup.
## Decisions (confirmed with operator)
1. **Monitoring philosophy: make checks work everywhere** — via the proven `pct exec`/
`qm guest exec` host-routing the MCP `run` tool already uses (no per-guest SSH keys),
plus deploy the check scripts INTO each guest and make them macOS-aware. Hosts/workstations
use direct SSH with the correct per-target user.
2. **Canonical host-hop access**`pct exec`/`qm guest exec` through the proxmox host is
the ONLY execution path for any LXC/VM command (scheduler + MCP `run` + agent). Direct
guest SSH is retired for execution; `lan_ip` stays for network probes only. (A1.)
3. **Auto-provision monitoring for new entities** — wire script-deploy + the
`health-check-answering` lifecycle gate into entity creation so any entity Nomos creates
becomes monitorable with zero manual steps (Track E).
4. **Lifecycle gate: skip monitoring for `deprecated`/`destroyed` targets** — no
permanent false alarms from retired things.
5. **Knowledge graph: address ALL gaps** — model TLS certificates, fix dns-zone gap,
re-export seeds, seed skills, raise graph cap.
6. **Read-only audit skill** — a `read_only` operator skill discovers live infra and diffs
it against the DB graph, producing a ranked drift report; the operator acts on findings
via existing lifecycle runbooks. No auto-fix. (Track F.)
## Findings (evidence)
### A. Health-check reality gaps (49 checks `down`)
**Root cause is a routing mismatch, verified live (tests use the scheduler's own key
`-i /etc/oikos/ssh_key`, not a default-key test):**
The MCP `run` tool already reaches every guest correctly via
`resolveExecTarget` (`internal/mcp/server.go:582`): resolve the proxmox host
(`attributes.host``hosts` edge → hubris default), SSH there, run
`pct exec <pve_id> -- bash -c 'echo <b64> | base64 -d | bash'` (VMs: `qm guest exec`).
That path needs **no per-guest lan_ip, no per-guest authorized_keys, no per-guest sshd**.
The **scheduler's `checkSSHScript` does not use it** — it SSHes directly to each
entity's own resolved address (`internal/scheduler/scheduler.go:758`,
`internal/checkdefaults/defaults.go:376 resolveHost`) and runs
`/opt/oikos/checks/<script>`. That is the bug. Decomposed by class:
| Class | Targets (verified) | Root cause |
|---|---|---|
| **Guests reached wrong** | `lxc:rclone` (mesh-only, no lan_ip), `lxc:nfs-export` (192.168.8.200: **ssh port 22 timeout** — no sshd), `lxc:teddycloud` (**key not authorized** — "not a homelab client"), `lxc:grimmory/romm/seanime` (strong: pct-exec reachable, **scripts not inside**) | scheduler SSHes the guest directly; should route via proxmox host `pct exec` like `resolveExecTarget`. rclone is correctly parented on hubris (`hosts` edge verified) and IS reachable via `pct exec 132` — the mesh fqdn is a red herring. |
| macOS host | `ws:mac-mini` (5 resource/updates checks `down`) | root SSH disabled (macOS); `user: dtoro` never read by resolver (`defaults.go:406` reads `attrs["ssh"]["user"]` only); scripts not deployed; scripts Linux-only |
| External / mesh-only | `host:netbird-vps` (no lan_ip; mesh unreachable from container) | `resolveHost` picks mesh IP over `public_ipv4` (`defaults.go:376`); sshd also "locked to hubris pubkey" |
| Dead route | `ingress:secrets.hubris.network` http `down` | `service:secrets-issuance` is `deprecated` but its ingress check still enabled — no lifecycle gate |
| ICMP-blocked | `vm:haos` ping `down` while up | HAOS blocks ICMP |
**Working** (prove the host-SSH model is sound): `host:hubris`, `host:strong` SSH with
the scheduler key → **SCRIPTS_PRESENT**; `lxc:gitea` direct-SSH → **SCRIPTS_PRESENT**
(it's a homelab client with root key + scripts). So the host hop is the reliable path.
**Parentage verified correct** (all `hosts` edges checked in DB): strong guests on
strong, hubris guests on hubris. No misplaced parents — the gap is routing + in-guest
script deployment, not topology.
Health aggregation itself is correct: `WorstHealthForTarget`
(`internal/db/sqlcgen/operations.sql.go:1472`) = worst enabled check. One failing
ssh-script drags an otherwise-healthy entity to `down`.
### B. Dead/stale data
- **24 orphan check_defs** + check entities, slugs `^check:(ping|ssh-script|disk):[0-9a-f]{8}$`
(e.g. `check:ssh-script:0d31fdd1`), `enabled=false`, `last_health=NULL`, `state=NULL`.
Leftover from the old `shortSlug()` collision bug (fixed in `defaults.go:263`).
- `service:secrets-issuance` = `deprecated`; `ingress:secrets.hubris.network` still
routes to it and alarms permanently.
### C. Knowledge-graph gaps
- **TLS certificates unmodeled**: `certificate` type + `uses-certificate` edge + `cert-expiry`
checker all exist, but **0** certificate entities. Cert expiry is invisible.
- **`dns-zone` declares `monitoring: [dns]`** (`seeds/ontology.yaml:382`) but no `dns`
checker exists → every zone is an `unmonitored` signal.
- **Seed drift**: 23 `dns-record` entities in DB, 0 in `seeds/inventory.yaml`.
- **`skills` table = 0** despite `.agents/skills/*/SKILL.md` on disk (runbooks = 15).
- **Graph capped at 500 nodes** (`internal/httpapi/impl.go:27 graphNodeCap = 500`);
299 `execution` + 87 `task` rows dominate, so `/graph` is not a faithful topology view.
---
## Work breakdown
### Track A — Make ssh-script checks work everywhere (route through the proxmox host)
Core idea: stop having the scheduler SSH each guest directly. Reuse the MCP `run`
tool's proven `resolveExecTarget` pattern — reach every LXC/VM **through its proxmox
host** via `pct exec`/`qm guest exec`. This fixes rclone (no lan_ip), nfs-export
(no sshd), teddycloud (no key), and every strong guest in one stroke, because the host
hop already has working root SSH. Hosts/workstations keep direct SSH.
**A1. Canonicalize host-hop as the ONLY execution path for LXC/VM (the real fix + simplification).**
Principle: **never SSH directly into a guest to run a command.** Every LXC/VM command
execution — scheduler checks, the MCP `run` tool, and the agent — routes through the
owning proxmox host via `pct exec <pve_id> -- ...` (VMs: `qm guest exec`). One SSH
credential per host (root key, already authorized on hubris/strong), no per-guest keys,
sshd, or lan_ip needed for execution. Verified this works: `pct exec 132` reaches rclone;
the MCP `run` tool already does it for every guest (`internal/mcp/server.go:582`).
- Network probes (http/ping) keep hitting the guest's `lan_ip`/URL directly — they don't
execute inside the guest, so they're unaffected. For LXCs all checks are ssh-script, so
they all route via the host; `lan_ip` becomes optional metadata, not a monitoring prereq.
- Extract `resolveExecTarget`/`resolveProxmoxHostSlug` out of `internal/mcp` into a shared
package (e.g. `internal/remote`) so the scheduler's `checkSSHScript`
(`internal/scheduler/scheduler.go:710`) and `checkBackupFreshness` (`backup.go:79`, the
other direct-SSH path) and the MCP `run` tool share ONE resolver. Today they diverge —
the scheduler SSHes guests directly (broken), MCP host-hops (works).
- `checkSSHScript`/`checkBackupFreshness`: when the target is `lxc:`/`vm:`, resolve the
proxmox host and wrap the invocation as `pct exec <pve_id> -- bash -c 'echo <b64> |
base64 -d | bash'` (VMs: the `qm guest exec` form at `server.go:625`). For `host:`/`ws:`
keep direct SSH (they ARE the host).
- **Risk class:** `config_mutation` (changes how probes reach every guest) → operator
approval. Verify one LXC end-to-end (rclone) before fanning out.
**A2. Deploy check scripts INTO guests (via `pct push`), not just to the host.**
- Verified: scripts exist on hubris/strong (the hosts) but `NO_SCRIPTS` inside grimmory,
romm, seanime, rclone. A `pct exec`-routed check still runs inside the guest, so the
scripts must live in the guest.
- Add a fleet-deploy tool (`tools/deploy-checks.sh`): for each LXC, from its proxmox
host, `pct push <id> checks/<script> /opt/oikos/checks/<script>` + chmod 755 (loop the
`checks/*.sh` set). For VMs, scp/agent; for hosts/workstations, run `checks/install.sh`.
- Backfill once now (all guests + mac-mini). See Track E for the automated version.
**A3. Fix per-target SSH user + resolver (hosts/workstations only).**
- `internal/checkdefaults/defaults.go:406 resolveSSHUser`: also read top-level
`attrs["user"]` (workstations carry `user: dtoro`, not `ssh.user`). Returns `dtoro`
for mac-mini. Re-derive mac-mini's check_defs so config carries the user.
- **Do NOT enable root SSH on mac-mini** — use `dtoro` (keeps macOS hardening).
**A4. macOS-aware check scripts.**
- `checks/cpu_check.sh:5` `top -bn1` (Linux) → branch on `uname -s == Darwin`
(`top -l 1`/`sysctl`). Same for `memory_check.sh`, `load_check.sh`, `disk_usage_check.sh`
(`df` differs), `updates_check.sh` (already apt-guarded; on Darwin report `healthy`
with `security_updates=0` or read `softwareupdate --list`).
- Each must still emit `{"health":..,"metrics":{..}}` JSON
(`internal/scheduler/scheduler.go:767`).
**A5. Reachability for external/mesh-only hosts.**
- `internal/checkdefaults/defaults.go:376 resolveHost`: prefer `public_ipv4` over mesh IP
for `standalone-server`/external so `host:netbird-vps` (82.165.190.79) is probeable.
Note sshd is "locked to hubris pubkey" (`inventory.yaml:89`) — either add the scheduler
key or proxy via hubris. Confirm before assuming direct SSH works.
- `ws:republic-laptop`: roving laptop on mesh only. ping-`down` when asleep is real;
keep ping-only and accept transient `down`, or set `monitoring: none`. (Decision in
Open Questions.)
- `lxc:rclone` no longer a special case — handled by A1's pct routing.
**A6. ICMP-blocked VMs.**
- `vm:haos` ping `down` while up: optional `tcp`-ping fallback in `checkPing`
(`internal/scheduler/scheduler.go:604`) for VMs that block ICMP, gated by an attribute.
Lower priority — confirm haos blocks ICMP before building.
### Track B — Lifecycle monitoring gate
**B1. Skip monitoring for deprecated/destroyed targets.**
- Disable (set `enabled=false`) and skip-scheduling `check_defs` whose `target` entity
`state` ∈ {`deprecated`,`destroyed`}.
- Implement by joining target state in `ListEnabledCheckDefs`
(`internal/db/sqlcgen/operations.sql.go`, the `ListEnabledCheckDefs` query) — exclude rows
whose target is retired — **or** in a `housekeeping` sweep
(`internal/scheduler/scheduler.go:302`) that disables them. Prefer the query filter
(no write needed at runtime).
- Matches `policy.yaml` lifecycle philosophy (`destroyed.refuse: all`); extend the comment.
- Effect: dead `ingress:secrets.hubris.network` alarm goes silent automatically.
### Track C — Dead-data cleanup
**C1. Delete 24 orphan check_defs + check entities.**
- Direct SQL (have psql access): delete `check_defs` then `entities` matching
`slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`. Confirm `state IS NULL` /
`enabled=false` first (already verified).
- Wrap as a one-shot migration or `scripts/cleanup-orphan-checks.sh`. **Risk class:** read
the rows first; this is `config_mutation` → operator approval.
**C2. Retire the secrets route.**
- With B1 in place the alarm silences. Optionally set `ingress:secrets.hubris.network`
`deprecated`/`destroyed` and remove its `routes-to` edge to service:secrets-issuance
(or keep as archaeology). Decide with operator.
**C3. Destroy 7 stray test LXCs (active cruft in the graph).**
- DB shows these with live `hosts` edges on strong, never cleaned up:
`lxc:preflight-test`, `lxc:preflight-test2`, `lxc:test-autocontinue`,
`lxc:test-decompose3`, `lxc:test-livewatch`, `lxc:test-livewatch2`, `lxc:typetype`.
- First confirm they're really gone in Proxmox (`pct list` on strong); if so, set their
entity state → `destroyed` (move to archaeology) and drop the `hosts` edges. If any
container still exists, destroy via `pct destroy` first (destructive → approval).
- They currently generate checks and pollute the graph/health view.
### Track D — Knowledge graph
**D1. Model TLS certificates.**
- Seed `certificate` entities (one per `*.hubris.network` route, or per Caddy-managed
cert) + `uses-certificate` edges from each `ingress-route`.
- Source real data: read Caddy's cert store (LXC 121) expiry via the existing `cert-expiry`
checker's discovery, or seed from Caddyfile and backfill `expires` live.
- Wires the `cert-expiry` checker (`internal/scheduler/scheduler.go`, `cert-expiry` kind)
against real entities instead of nothing.
**D2. dns-zone monitoring gap.**
- `seeds/ontology.yaml:382`: change `dns-zone` `monitoring: [dns]``monitoring: none`
with a comment "no dns checker yet; revisit when implemented". Stops the per-zone
`unmonitored` noise. Re-seed.
**D3. Re-export seeds to fix drift.**
- Run `oikos export` (or the export endpoint) so the 23 runtime `dns-record` entities +
other runtime-created topology land in `seeds/inventory.yaml`. Diff, review, commit.
**D4. Seed skills from disk.**
- Ingest `.agents/skills/*/SKILL.md` as `skill` entities (mirror how runbooks seed → 15
exist). Add to the knowledge seed ingest path (`internal/db/seed.go`) or a one-shot
ingest. `get_skills()` then returns data.
**D5. Raise graph node cap.**
- `internal/httpapi/impl.go:27 graphNodeCap = 500` → raise (e.g. 5000) **and/or**
paginate `/api/v1/graph`. Ensure the query stays performant (it already limits by default;
confirm no full-table risk). Optionally exclude cognition rows (`execution`/`task`) from
the default topology view via a `?layer=infrastructure` filter so infra isn't crowded out.
### Track E — Auto-provision monitoring when a new entity is created
Goal: the operator's request — "make sure this is handled automatically in the future
when the agent creates new entities." Today `ensureDefaultChecks`
(`internal/httpapi/default_checks.go:9`) writes check_defs on entity creation but does
NOT make the target probe-ready (no script deploy, no host-routing). Its own comment
admits the gap. A new entity should become monitorable with zero manual steps.
**E1. Hook script-deploy into entity creation / provisioning.**
- Extend `ensureDefaultChecks` (called on entity create, `default_checks.go`) so that,
after writing check_defs, it also ensures the target can answer:
- **LXC/VM**: `pct push` the `checks/*.sh` set into the guest from its proxmox host
(reuse the host resolution from A1). Idempotent (skip if present + unchanged).
- **host/workstation**: ensure scripts at `/opt/oikos/checks/` (run `checks/install.sh`
over SSH; locally on mac-mini).
- Because the check itself is routed via `pct exec` (Track A), no per-guest SSH key or
sshd is needed — host hop + in-guest scripts are the only prerequisites, both now
automated. mac-mini still needs its `dtoro` key (A3) once.
**E2. Tie into the lifecycle `provisioning → active` gate.**
- The ontology already requires `health-check-answering` for `provisioning → active`
(`seeds/ontology.yaml:39`, checked by `internal/ontology/validate.go:167`).
- Make that gate actually run one check against the new entity and require a non-`down`
verdict before the transition is allowed. This closes the loop: an entity isn't "active"
(and isn't trusted for blast-radius/auto decisions) until monitoring proves it answers.
**E3. Re-run on re-seed / attribute change.**
- `checkdefaults.Ensure` already re-derives check config from the seed on re-ingest
(`internal/checkdefaults/defaults.go:301`, seed wins, `enabled` preserved). Mirror that
for script deploy: when `pve_id`/`host`/address attributes change, re-target the check
and re-deploy scripts to the new guest.
**Net effect:** a new LXC provisioned by Nomos (via `pct_create`, which registers the
entity + `hosts` edge, `internal/httpapi/actuator.go:615`) automatically gets
script-pushed + check_defs + a passing `health-check-answering` gate before going active.
### Track F — Read-only knowledge-graph audit skill
Goal: the operator's request — a skill that auto-discovers live infra and validates the
knowledge graph (entities, parentage, checks, scripts, seeds, certs) against reality,
producing a ranked drift report. **Read-only; no auto-fix** — the operator routes each
finding to the relevant lifecycle runbook.
**Precedent (reuse, don't duplicate):** existing drift/quality machinery is fragmented and
knowledge-content focused. The audit orchestrates these + fills the topology/script gaps:
- `internal/httpapi/knowledge_drift.go` — duplicate notes, orphan notes, tag splits (already endpoints).
- `internal/scheduler/coverage.go coverageSweep` — unmonitored declared types (re-use its logic/SQL).
- MCP discovery: `list_lxcs` (`internal/mcp/tools.go:478`), `get_lxc_state`, `list_entities`,
`get_relations`, `http_get`. These already enumerate live LXC/VM state from the proxmox host.
**F1. Add an on-demand audit primitive (MCP tool + endpoint).**
- New MCP tool `audit_knowledge_graph` (+ `GET /api/v1/audit/drift`) — read-only, runs the
discovery+diff in one pass and returns a ranked report. Each finding = `{category, severity,
entities, evidence, suggested_runbook}`.
- Discovery sources (all via the canonical host-hop / existing tools): `pct list` + `pct
config` on hubris & strong (guests, `net0` IP, onboot state); `qm list` (VMs); Caddy admin
API / Caddyfile (routes → certs); docker `ps` on compose hosts; the `checks/*.sh` set vs
what's deployed at `/opt/oikos/checks/` per target.
- Report categories (the gaps this investigation found):
1. **Ghost entities** — in DB but not in Proxmox (e.g. stray `lxc:test-*`).
2. **Missing entities** — in Proxmox/Caddy/docker but no DB entity.
3. **Misplaced parent** — `hosts` edge disagrees with where the guest actually runs (the
rclone class — though rclone's parent is correct; this catches real migrations).
4. **Orphan/dead checks** — `check_defs` whose target is deprecated/destroyed, or random-slug
orphans (`^check:(ping|ssh-script|disk):[0-9a-f]{8}$`).
5. **Undeployed scripts** — checks expect `/opt/oikos/checks/<script>` but it's absent in
the guest (the strong-guest/rclone class).
6. **Unmonitored declared types** — reuse `coverageSweep` SQL (dns-zone today, agents).
7. **Seed drift** — entities/edges in DB but not in `seeds/inventory.yaml` (23 dns-records),
via `oikos export` diff.
8. **Unmodeled certs** — Caddy serves a cert with no `certificate` entity + `uses-certificate` edge.
9. **Knowledge rot** — delegate to the existing `knowledge_drift` endpoints (duplicates/orphans/tags).
**F2. Author the skill.**
- `.agents/skills/knowledge-graph-audit/SKILL.md` — front-matter
`risk_class: read_only`, `inputs: [scope?]`, `verification: "drift report returns ok"`.
Body: run `audit_knowledge_graph`, read the ranked report, and for each category point at
the remediation runbook (`lifecycle-deprecate-node`, `lifecycle-destroy-node`,
`config-change-deploy` for scripts, `lifecycle-migrate-node` for parents, this plan's
tracks for cert/seed/graph-cap work). No mutating steps.
- Seed a matching `runbook:knowledge-graph-audit` entity in `seeds/knowledge.yaml`
(bound by `applies_to_type`) so `search_knowledge`/`get_skills` surface it (also fixes the
empty-skills-table gap, Track D4).
**F3. Optional: periodic sweep (later).** Wrap categories 4/6 as a scheduler housekeeping
sweep that raises `drift` signals, mirroring `coverageSweep`. Out of scope for this plan
unless the operator wants continuous drift signals; the on-demand skill is the deliverable.
**Risk class:** `read_only`. The audit only reads (pct list/config, docker ps, Caddy API,
DB selects, an `oikos export` to a temp file). No writes. Safe to run unattended.
---
## Validation
After each track, verify via API (read-only, no approval):
- `GET /api/v1/entities/ws:mac-mini` → `health` ∈ {healthy,degraded} (not `down`).
- `GET /api/v1/entities/lxc:rclone` → `health` healthy (proves pct-routing through hubris;
rclone currently unreachable because it resolves to a mesh fqdn). Verify its checks now
route via `pct exec 132` on hubris.
- Strong guests (`lxc:grimmory`, `lxc:romm`, `lxc:seanime`) → ssh-script checks healthy
after scripts pushed inside + routed via strong's `pct exec`.
- `GET /api/v1/checks?include_disabled=false` → `down` count drops from 49 to the
genuinely-down set (republic-laptop asleep, real outages only). Re-run the per-class table.
- `GET /api/v1/entities/service:secrets-issuance` + its ingress → no enabled check.
- Orphan cleanup: `SELECT count(*) FROM check_defs cd JOIN entities e ON e.id=cd.entity_id
WHERE e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$';` → 0.
- Test LXCs (C3): `SELECT count(*) FROM entities WHERE slug IN
('lxc:preflight-test','lxc:test-livewatch',...) AND state<>'destroyed';` → 0.
- Provision a throwaway LXC via Nomos → it auto-gets scripts + check_defs + passes
`health-check-answering` before reaching `active` (E1/E2).
- `GET /api/v1/entities?type=certificate&limit=1` → >0; cert-expiry checks created.
- `GET /api/v1/entities?type=skill&limit=50` → >0.
- `GET /api/v1/graph` node count > 500 (or infra fully represented with a layer filter).
- `oikos export` diff shows dns-record entities present; `git diff seeds/inventory.yaml`.
- Scheduler logs: `checkdefaults: declared check not created` warnings gone for dns-zone.
- **Canonical access (A1):** no scheduler code path SSHes a guest directly —
`grep -rn "sshExec" internal/scheduler` shows it only for `host:`/`ws:` targets; LXC/VM
go through the shared `pct exec`/`qm guest exec` resolver.
- **Audit skill (F1/F2):** `audit_knowledge_graph` MCP tool returns a ranked report with
the 9 categories; running it against current state reproduces this plan's findings
(orphan checks, stray test LXCs, undeployed scripts, seed drift, 0 certs). The skill
is read-only — confirm it performs no DB writes (audit-log shows only reads).
Unit/integration tests to add/update:
- `internal/checkdefaults` / shared `internal/remote` resolver: LXC/VM check routes via
`pct exec`/`qm guest exec` to the resolved proxmox host; resolver reads top-level `user`;
`public_ipv4` preferred for standalone-server (`defaults_test.go`).
- `internal/scheduler`: `ListEnabledCheckDefs` excludes deprecated/destroyed targets
(new test); `coverage_test.go` still green; `sshExec` no longer called for guest slugs.
- macOS script branches: assert JSON shape unchanged on `Darwin` (shunit2 or a smoke run).
- E1: new-entity creation triggers script push (mock pct/SSH in test).
- F1: `audit_knowledge_graph` against a fixture DB+mock discovery returns the expected
category counts (ghost, missing, orphan, undeployed, drift).
## Risks
- **Canonical host-hop (A1)** makes each proxmox host the single SSH dependency for all its
guests. This is already true (pct exec requires the host up) and is a net improvement
(one credential vs many), but a host outage now fails all its guest checks together —
which is the *correct* blast radius (guests are unreachable when their host is down).
- **Routing change (A1)** alters how probes reach every guest — `config_mutation`. Verify
one LXC end-to-end (rclone via `pct exec 132`) before fanning out. Extracting
`resolveExecTarget` into a shared package keeps scheduler + MCP in lockstep.
- **Script push into guests (A2/E1)** writes to guest filesystems — `config_mutation`.
Idempotent + content-checked; never clobber a same-named operator script without diffing.
- **mac-mini root SSH**: do NOT enable root login; use `dtoro` (A3) — keeps macOS hardening.
- **`netbird-vps` sshd locked to hubris pubkey**: may need the scheduler key added or
proxying via hubris; confirm before assuming direct SSH works (A5).
- **`health-check-answering` gate (E2)** could block a legitimately-active entity whose
only working check is ICMP-blocked (haos). Allow the gate to pass on any non-`down`
reachable probe, or grant an operator override.
- **Audit skill (F1)** discovers infra via `pct`/Caddy/docker reads — keep it strictly
read_only; ensure discovery commands are in the read-only allowlist (no state change).
- **Seed re-export** can surface large diffs (cognition entities) — scope export to
topology entities, or review carefully before commit. Bump `VERSION` per repo rule.
- **Graph cap raise**: large node sets may slow the graph render; pair with a layer filter.
## Open questions (none blocking; confirm during implementation)
- republic-laptop: mesh-only roving laptop — keep ping-only (accept transient `down`) or
`monitoring: none`? (A5)
- secrets ingress: keep as archaeology or destroy the route? (C2)
- certificates: seed statically from Caddyfile, or auto-discover live from Caddy store? (D1)
- netbird-vps: add scheduler key to its sshd, or always proxy through hubris? (A5)
- Audit discovery for docker hosts/stacks: enumerate via `docker ps`, or model compose
stacks only? (F1)
## Suggested order
A1 (canonical host-hop routing — unblocks rclone + all guests) → A2 (push scripts into
guests) → A3 → A4 (mac-mini) → A5 → E1/E2 (automate for new entities) → B1 → C1 → C3 → C2
→ D2 (quick, silences dns noise) → F1/F2 (audit skill — also validates the above worked)
→ D1 → D4 → D3 → D5. Validate after each track.

View File

@@ -0,0 +1,293 @@
# 2026-07-30 — Session review: plan drift & a dead activity panel
**Status:** Done — 2026-08-03. Shipped in `467589d` (v0.14.1), deployed to
production. The two operator-reported complaints are resolved and verified on
the bug-report session itself (`398f5eda`); see [Resolution](#resolution-2026-08-03)
at the end. Items P0.1, P0.2 (fix 1+2), P1.1, P1.2 are complete; P0.2 fix 3,
P2.1, P2.2 are deferred (the reported symptoms no longer reproduce).
**Scope:** The five most-recently-active `agent:nomos` sessions by
`last_active_at`, pulled from the live Postgres on 2026-07-30, plus the
code paths they exercise (`cmd/nomos/store.go`, `cmd/nomos/tasks.go`,
`web/src/lib/stores/{activity,workspace,chat}.ts`,
`web/src/lib/components/UnifiedTimeline.svelte`).
**Trigger:** Operator report — "the plan was off, the activity sidepanel
was not kept up to date and feels off, not live."
Both complaints are real, both reproduce deterministically, and both have
a single-line root cause. They are *not* the same bug, but they compound:
the plan bug produces the exact event stream that the activity panel
silently discards.
---
## Sessions reviewed
| # | sid | goal (short) | outcome | activity rows | plan gens | re-planned? |
|---|---|---|---|---|---|---|
| 1 | `398f5eda` | hubris recurring network outage → EEE mitigation | success | 48 | 2 | **yes** |
| 2 | `0a49ba3d` | triage active signals on host:strong | success | 30 | 1 | no |
| 3 | `9368633d` | sensor temperatures on host:strong | success | 12 | 1 | no |
| 4 | `2065a29a` | temps → pivot to "fun fact about chickens" | success | 18 | 2 | **yes** |
| 5 | `bad26076` | greeting / responsiveness test | success | 6 | 1 | no |
**Score: 5 success / 0 partial / 0 failed.** The agent's *reasoning* was
fine in all five. Every defect below is in the bookkeeping and the
rendering — the parts the operator actually looks at.
**The correlation that matters: both sessions that re-planned (`398f5eda`,
`2065a29a`) recorded a corrupt plan. Neither of the three that didn't
re-plan did.** Re-planning was a 100% failure path (pre-fix).
---
## P0.1 — `update_plan_step` addresses the wrong plan generation
This is "the plan was off," and it was fully deterministic.
`proposePlan` numbered a new generation's steps *continuing* from the old
one (`store.go:937`):
```go
seq := startSeq + i + 1 // startSeq = MAX(seq) of all prior steps
```
So on generation 2 of `398f5eda`, the six new steps landed at **seq 712**.
But the tool result the model got back never mentioned those numbers
(`tasks.go:282`):
```
"Plan set (6 steps). If all steps are read-only, execute now — …"
```
…while `update_plan_step`'s schema told it (`tasks.go:81`):
```go
"seq": "1-based step number from propose_plan."
```
The model had no way to learn the real seq numbers and was explicitly told
to use 1-based ones. It did exactly that.
**What the DB recorded for `398f5eda`:**
```
20:07:40 propose_plan → gen 2 created at seq 7..12
gen 1 (seq 1..6) marked `replaced`
20:08:10 update_plan_step seq=1 running ← hits gen-1 step 1
20:08:18 update_plan_step seq=1 done ← resurrects a `replaced` row
20:08:18 update_plan_step seq=2 running
20:09:57 complete_task
```
Result — the persisted plan was a lie in three separate ways:
- **Steps 15 (the abandoned "force 1Gbps" plan) show `done`** with real
start/finish timestamps. Work that was never performed was recorded as
performed. `updatePlanStep` wrote status by seq with no guard, so it
happily flipped `replaced``running``done`.
- **Steps 712 (the actual EEE work that ran) had `started_at = NULL`**
and were bulk-closed to `done` by `completeTask`'s auto-close sweep
(`store.go:1096`) at 20:09:57 — all six sharing one timestamp.
- **The panel shows 12 steps**, because `getPlanSteps` returned every
generation unfiltered (`store.go:1356`) and the frontend never reads the
`generation` field at all (`grep generation web/src` → zero hits outside
the API type).
`2065a29a` had the identical signature: gen 2 at seq 45, gen-1 steps 1
and 2 flipped to `done`/`skipped` four seconds later.
### Fix (implemented)
1. **Make seq generation-relative.** `proposePlan` resets seq to `1..N` per
generation; `(session_id, generation, seq)` is the addressing key.
`updatePlanStep` resolves against `MAX(generation)`. This matches what
the model naturally does and what every prompt already says.
2. **Return the seq numbers to the model.** The `propose_plan` result now
enumerates them (`1=…; 2=…`).
3. **Refuse writes to superseded rows.** `updatePlanStep` addresses only
the current generation; a stale/out-of-range seq returns
`errPlanStepNotFound` (never resurrects a `replaced` row).
4. **Filter by generation on read.** `getPlanSteps` returns only
`MAX(generation)` by default; `?all=true` for the audit/eval view.
5. **Stamp `started_at` in the auto-close sweep.** `completeTask` closing
a step sets `started_at = COALESCE(started_at, now())`.
A migration (`029`) renumbers existing rows to per-generation `1..N` and
replaces the `(session_id, seq)` index with a unique
`(session_id, generation, seq)`.
---
## P0.2 — The activity panel invents its own timestamps
This is "not live / feels off," and it was worse than a staleness bug: the
times on screen were **fabricated at render time**.
`activity.ts:118` — every tool entry:
```ts
timestamp: now - ($msgs.length - mi) * 1000
```
`now` was `Date.now()` captured at the top of `computeActivityLog`. So a
tool call's displayed time was *"the moment this function last ran, minus
one second per message from the end."* Not when the call happened.
Three consequences, all of which read as "not live":
- **The clock was wrong.** `UnifiedTimeline` rendered these through
`hhmm()` / `hhmmss()`, so opening yesterday's session showed every step
timestamped *right now*, one second apart.
- **It churned every 3 seconds.** The message poller re-set `messages`
unconditionally on every tick, which re-derived `activityLog`, which
re-captured `now`. Every entry's timestamp marched forward 3s at a time,
forever. Motion with no information.
- **Real and fake timestamps sorted together.** Plan steps used the
genuine `started_at`; tool calls used the synthetic value; the final
sort mixed them. Steps with no `started_at` fell back to `now` — **97 of
339 non-pending steps in the DB (29%) had `started_at = NULL`** — so they
landed at the bottom of the timeline regardless of when they ran.
The real data already existed and was already served: `agent_activity`
holds true `ts`, `duration_ms`, `success`, and
`correlation_id = session_id`, exposed at `GET /agent-activity`. The panel
ignored it and reconstructed a worse version from the message blob.
### Fix (implemented — fix 1 + 2)
1. **Carry real timestamps on tool calls.** `computeActivityLog` uses each
tool call's message `created_at` (a true persisted time). The
`now - (len - mi) * 1000` expression is gone entirely.
2. **Only fall back to wall-clock for genuinely-live entries, and freeze
it once assigned** — a `Map<id, timestamp>` outside the derivation, so
re-deriving never moves an existing entry. This is what kills the churn.
Deferred to a later pass: backing the panel with `agent_activity` for
historical sessions (fix 3, unlocks `duration_ms`) — the two reported
symptoms (wrong clock, churn) no longer reproduce without it.
---
## P1.1 — Plan-step events for a superseded generation were silently dropped
The frontend half of P0.1, and the reason the panel *froze* rather than
merely showing wrong steps.
On `plan.proposed` with `appended: false`, the store replaced its step
list wholesale — so after the re-plan it held seq 712. Every subsequent
`plan.step.started` / `plan.step.finished` carried seq 15 and a gen-1
`step_id`, and `applyPlanStepEventTo` bailed on no match:
```ts
if (i === -1) return steps
```
So for the entire second half of `398f5eda` — the half where all the real
work happened — **the panel showed six pending steps and nothing ever
moved.** Then `completeTask` closed them in the DB while emitting only
`task.status`, no per-step events, so they stayed pending on screen even
after the session finished.
### Fix (implemented)
- Fixing P0.1 removed the cause (the events now carry the correct
generation-relative seq + the panel's current steps match). The `i === -1`
branch now `console.warn`s and increments an exported
`droppedPlanStepEvents` counter instead of returning silently, so the
next divergence is visible instead of looking like a dead UI.
- **`completeTask`'s auto-close sweep now emits `plan.step.finished` per
closed step** (scoped to the current generation). General rule enforced:
no plan-step status change without a corresponding event.
---
## P1.2 — Every plan carried a duplicate writeback step
In `398f5eda` gen 2, step 11 was the model's own writeback step and step 12
was the auto-appended one. The detector substring-matched the literal tool
names `update_entity_attributes` / `create_relationship` in the title or
detail; the model wrote a natural-language equivalent, so the match failed
and a redundant step was appended. Same pattern in `0a49ba3d` and
`9368633d`.
### Fix (implemented)
Broadened the detector to a case-insensitive check for `write back` /
`writeback` / `upsert_knowledge` in the title or detail, on top of the
existing tool-name match.
---
## P2.1 — Long unexplained stalls, invisible in the UI *(deferred)*
- `bad26076`: a greeting took **16 minutes** wall-clock with 6 activity rows.
- `2065a29a`: step 1 showed `started_at``finished_at` spanning **16 minutes**
for a `sensors` call that returned in milliseconds.
The work took under a second; the step was *open* for 16 minutes. The panel
has no way to distinguish "working" from "waiting for a nudge." Surfaces a
step's idle time: mark a `running` step *stalled* when it has had no
`agent_activity` row for >60s. Deferred — needs the `agent_activity`-backed
panel (P0.2 fix 3).
## P2.2 — `agent_activity` is a single-type table *(deferred — decision)*
All rows are `activity_type = 'tool_call'`. Either start emitting the other
types the schema anticipates (`reasoning`, `plan`, `error`) or drop the
dimension. Worth a decision, not urgent.
---
## Recommended sequence (executed)
| Order | Item | Status |
|---|---|---|
| 1 | P0.1 fix 3 + 4 (refuse superseded writes, filter on read) | done |
| 2 | P0.2 fix 1 + 2 (real timestamps, frozen fallback) | done |
| 3 | P1.1 (emit events from the auto-close sweep) | done |
| 4 | P0.1 fix 2 (generation-relative seq) + migration | done |
| 5 | P1.2, P2.1 | P1.2 done; P2.1 deferred |
| 6 | P0.2 fix 3 (back the panel with `agent_activity`) | deferred |
| 7 | P2.2 | deferred |
## Regression coverage (added)
- `store_test.go`: `TestUpdatePlanStep_GenerationRelative` — re-plan →
`update_plan_step(seq=1)` must address gen-2 and never resurrect a
superseded gen-1 `replaced` row; out-of-range seq → `errPlanStepNotFound`.
- `store_test.go`: `TestCompleteTask_AutoCloseEmitsEvents` — auto-close
emits one `plan.step.finished` per closed step and stamps `started_at`.
- `store_test.go`: `TestProposePlan_RefuseInFlight` — updated for
generation-relative seq + `?all=true`.
- `web/src/lib/stores/activity.test.ts`: `computeActivityLog` is pure w.r.t.
wall-clock (two calls 50ms apart → identical output), persisted tool calls
use real `created_at`, live entries freeze instead of churning.
---
## Resolution (2026-08-03)
Shipped in commit `467589d` (VERSION `0.14.0``0.14.1`), pushed to
`origin/main`, deployed via the Gitea webhook (`scripts/deploy.sh`):
pg_dump → pull → `docker compose build``up -d` → health check (healthy).
Verification on the bug-report session `398f5eda` post-migration:
```
gen 1: seq 1..6 (the abandoned "force 1Gbps" plan — superseded)
gen 2: seq 1..6 (the real EEE work — was seq 7..12, now normalized to 1..6)
```
- `schema_migrations` v29 applied; old `idx_plan_steps_session` dropped,
unique `idx_plan_steps_session_gen_seq` in place.
- Containers recreated; `healthz` and `/agent/sessions/:id/plan` HTTP 200.
- Full `cmd/nomos` suite (23 tests) + web suite (70 tests) green; `go vet`
clean; ESLint/Prettier clean.
Note: historical `started_at = NULL` on already-completed steps (visible on
`398f5eda` gen 2) is left as-is — backfilling would fabricate times. Going
forward `completeTask` stamps `started_at`, and the frontend freezes
NULL-started steps stably so they no longer churn.

View File

@@ -0,0 +1,503 @@
# 2026-08-03 — Adopt cyberspace.online terminal aesthetic + dithered images
**Status:** Implemented in v0.16.0 (`757ef2f`). Shipped as a **full theme
replacement** (Terracotta/Carbon → cyberspace BBS/terminal style), not the
opt-in addition originally drafted below — the operator chose full replacement
during execution (see decision `theme.replace_with_cyberspace`). The `<RasterImage>`
Atkinson-dithering component and the warm-cream/JetBrains-Mono look landed as
drafted; only the "opt-in vs replace" scope changed.
Adopt the look of https://cyberspace.online/ (a BBS / "social media
de-imagined" terminal aesthetic) as a **new, opt-in theme family** in oikos,
with **both light and dark variants**, plus a reusable **`<RasterImage>`**
component that renders images to a `<canvas>` with Atkinson dithering (the
"kinda dithered" image style). The existing Terracotta/Carbon themes stay the
default; this adds, it does not replace.
---
## TL;DR
1. Add a third theme family — **"Cyberspace Dark"** and **"Cyberspace Light"** —
wired through the same `--background` / `--foreground` / … token layer every
component already uses, so nothing in the UI tree changes; only the tokens
get new values. Square corners (`--radius: 0`), warm cream-on-black, mono
everything.
2. Extend `web/src/lib/stores/theme.svelte.ts` from a 2-state `'light'|'dark'`
toggle to a named-theme model, keeping `.dark` class behavior for
compatibility.
3. Self-host JetBrains Mono (body) + a pixel/terminal face (VT323 or Departure
Mono) for the logo/headings accents, replacing the Google Fonts `<link>`.
4. Build `web/src/lib/components/RasterImage.svelte`: draws any image to a
`<canvas>` reduced to a 2-color (theme `fg`/`bg`) palette via **Atkinson
dithering**, with an `<img>` fallback and a skeleton placeholder — exactly
the cyberspace pattern. Re-renders when the theme changes (palette flips).
5. Optional cosmetic idioms (terminal-box focus ring, braille spinner, `<s>`
strike lists) as small additive utilities, not a redesign.
The whole thing is **non-breaking and incremental**: each step ships behind the
existing theme picker, so Terracotta/Carbon users see nothing until they opt in.
---
## 1. Extracted style spec (source of truth from cyberspace.online)
Captured from the live site's SSR HTML + inline boot script. This is the
reference the tokens below are derived from.
### 1.1 Color model
Cyberspace defines **exactly three colors per theme**`fg`, `bg`, `fgDim`
applied to CSS custom properties. Everything else (borders, primary, cards) is
*derived* from those three. There are 11 named themes total; the two we care
about:
| Theme | `fg` (text) | `bg` (canvas) | `fgDim` (muted) |
|---------|--------------|---------------|-----------------|
| Dark | `#efe5c0` | `#000000` | `#a89984` |
| Light | `#000000` | `#efe5c0` | `#3a3a3a` |
Note the elegance: **light and dark are exact inverses** — they share the same
warm cream (`#efe5c0`, a Gruvbox-ish paper tone) and just swap which side of it
is ink vs. paper. The muted tone `#a89984` is straight out of the Gruvbox
palette. This is why both themes read as "the same site" despite opposite
polarity.
Boot-time fallback (the site's original/GRiD theme) is amber `#FF9810` on
`#120900` — useful as a *third* optional accent if we ever want a true-phosphor
variant.
### 1.2 Type
- **Body / mono:** JetBrains Mono (self-hosted `.woff2`, Regular).
- **Boot + logo accents:** Departure Mono (self-hosted `.woff2`). A quirky
monospace; VT323 (Google, free) is a close, easy substitute.
- **Stylized wordmark** (`ᑕ¥βєяรקค¢є`, class `.font-vt`): a terminal/pixel face.
Rule lives in their external `entry.*.css` (not in the SSR dump); VT323 is the
safe assumption.
cyberspace sets `font-mono` on the root wrapper — the **entire UI is
monospace**. There is no proportional body face. Headings use the same mono
family at larger size / normal weight.
### 1.3 Layout & component idioms
- **Left rail nav:** fixed, icon-only when minimized (~80px), expands on click.
Square buttons, Phosphor icons, uppercase `text-xs` labels.
- **`.terminal-box`:** the universal card. Bordered (`border border-border`),
**square corners** (`rounded-none` everywhere — `--radius` is effectively 0),
and on focus/emphasis gets `ring-2 ring-fg` (a 2px ring in the foreground
color).
- **Emphasis by inversion:** active/primary state is `bg-fg text-bg` — fill with
foreground ink, text becomes the canvas color. No separate "accent" hue; the
accent *is* fg.
- **Strikethrough as a feature list:** `<s>Ads</s> <s>Videos</s> …` — crossed-out
`<s>` elements spell out what the product removes. Cheap, on-brand.
- **Braille spinner:** `BrailleSpinner` component animates braille block chars
(`⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏`) for loading states instead of a circle.
- **Max content width** `max-w-4xl`, centered; generous vertical rhythm; thin
2px scrollbars colored `--color-border`.
- Borders are `1px solid` in a border color derived from `fg`/`fgDim` at low
alpha (their `--color-border` is not literally in the dump, but every
bordered surface uses it, and it tracks `fg`).
### 1.4 The dithered image (`RasterImage`) — what we actually know
From the SSR HTML the component is unambiguous about its *shape*, silent on its
*algorithm* (the dither JS is in an external `/_nuxt/*.js` bundle not present in
the page dump):
- Renders a **`<canvas>`** as primary output, with an **`<img>` fallback** as a
sibling. Parent selectors `[&>canvas]:max-w-full [&>canvas]:h-auto` and
`[&>img]:…` size both responsively.
- Emits a **`.raster-image-skeleton`** placeholder (empty div, `background:
var(--color-bg)`) during SSR/before hydration — no flash of the raw photo.
- Scoped styles (`data-v-4d61df89`): `.raster-image { display:block }`,
`.raster-image-skeleton { display:block; background:var(--color-bg) }`.
**Inferred technique** (standard for this look): Canvas 2D → `drawImage` →
`getImageData` → per-pixel luminance reduction to a 2-color palette (`fg`/`bg`)
with an **error-diffusion** pass (Atkinson or FloydSteinberg) → `putImageData`.
This produces the characteristic speckled 1-bit halftone. Target is almost
certainly the theme's own fg/bg, which is *why* the dithered art recolors
correctly when you flip themes.
We will implement Atkinson (see §4) — it's the classic Mac/BBS dither, slightly
softer than FloydSteinberg, and matches "kinda dithered" precisely.
---
## 2. Recommended approach: opt-in theme family (not a rebrand)
oikos today = Art-Nouveau / terracotta / rounded / serif-heading (Inknut
Antiqua), floating-window desktop shell. cyberspace = BBS / mono / square /
cream-on-black. These are **opposite poles**; a flat rebrand would discard the
existing art direction and rework every component's rounding/spacing.
**Decision: add cyberspace as a new theme family, selectable in the existing
theme picker.** This is low-risk, reversible, and lets the dithered images +
terminal idioms land incrementally. The full-rebrand alternative is documented
in §7 for if you later decide to make it the default.
Because every oikos component consumes colors through the Tailwind v4 token
layer (`--background`, `--foreground`, `--card`, `--border`, `--primary`, …)
defined in `web/src/app.css` `@theme inline`, a new theme is **just a new set
of values for those same custom properties** — zero component edits required
for the recolor. That indirection is the whole reason this is cheap.
---
## 3. Theme token additions (`web/src/app.css`)
Add two new blocks alongside the existing `:root` (Terracotta) and `.dark`
(Carbon). They set the *same* token names to cyberspace's values, plus pin
`--radius: 0` for square corners and remap fonts (see §5).
Driven by a `data-theme` attribute on `<html>` (set by the store, §6), so all
four states — Terracotta, Carbon, Cyberspace Dark, Cyberspace Light — coexist:
```css
/* ── Cyberspace Dark (cream on black) ── */
:root[data-theme='cyber-dark'] {
--radius: 0px;
--background: #000000;
--foreground: #efe5c0;
--card: #000000; /* cyberspace has no card tint; cards are just bordered bg */
--card-foreground: #efe5c0;
--popover: #000000;
--popover-foreground: #efe5c0;
--primary: #efe5c0; /* emphasis = fg ink */
--primary-foreground: #000000; /* inverted */
--secondary: #1a1a1a;
--secondary-foreground: #efe5c0;
--muted: #141414;
--muted-foreground: #a89984; /* fgDim */
--accent: #efe5c0;
--accent-foreground: #000000;
--destructive: #cc241d; /* Gruvbox red, sits in the same palette */
--destructive-foreground: #efe5c0;
--border: color-mix(in oklab, #efe5c0 22%, transparent); /* fg-derived hairline */
--input: color-mix(in oklab, #efe5c0 28%, transparent);
--ring: #efe5c0; /* the ring-2 ring-fg look */
--sidebar: #000000;
--sidebar-foreground: #efe5c0;
--sidebar-primary: #efe5c0;
--sidebar-primary-foreground: #000000;
--sidebar-accent: #1a1a1a;
--sidebar-accent-foreground: #efe5c0;
--sidebar-border: color-mix(in oklab, #efe5c0 22%, transparent);
--sidebar-ring: #efe5c0;
--chart-1: #efe5c0; --chart-2: #a89984; --chart-3: #fabd2f;
--chart-4: #b8bb26; --chart-5: #83a598; /* Gruvbox for charts */
--success: #b8bb26; --warning: #fabd2f;
/* oikos semantic aliases (app.css :root block) */
--bg: var(--background); --bg-surface: var(--card); --bg-deeper: #050505;
--bg-hover: var(--secondary); --bg-active: var(--accent);
--text: var(--foreground); --text-muted: var(--muted-foreground);
--accent-blue: #83a598; --accent-green: var(--success);
--accent-red: var(--destructive); --accent-orange: var(--warning);
/* terminal face for this theme only (see §5) */
--font-sans: 'JetBrains Mono', ui-monospace, Menlo, monospace;
--font-mono: 'JetBrains Mono', ui-monospace, Menlo, monospace;
--font-heading: 'VT323', 'JetBrains Mono', monospace; /* pixel wordmark feel */
}
/* ── Cyberspace Light (black on cream paper) — exact inverse ── */
:root[data-theme='cyber-light'] {
--radius: 0px;
--background: #efe5c0;
--foreground: #000000;
--card: #efe5c0;
--card-foreground: #000000;
--popover: #efe5c0;
--popover-foreground: #000000;
--primary: #000000;
--primary-foreground: #efe5c0;
--secondary: #e0d6b0;
--secondary-foreground: #000000;
--muted: #e6dcc0;
--muted-foreground: #3a3a3a; /* fgDim */
--accent: #000000;
--accent-foreground: #efe5c0;
--destructive: #9d0006;
--destructive-foreground: #efe5c0;
--border: color-mix(in oklab, #000000 22%, transparent);
--input: color-mix(in oklab, #000000 28%, transparent);
--ring: #000000;
--sidebar: #efe5c0;
--sidebar-foreground: #000000;
--sidebar-primary: #000000;
--sidebar-primary-foreground: #efe5c0;
--sidebar-accent: #e0d6b0;
--sidebar-accent-foreground: #000000;
--sidebar-border: color-mix(in oklab, #000000 22%, transparent);
--sidebar-ring: #000000;
--chart-1: #000000; --chart-2: #3a3a3a; --chart-3: #b57614;
--chart-4: #79740e; --chart-5: #076678;
--success: #79740e; --warning: #b57614;
--bg: var(--background); --bg-surface: var(--card); --bg-deeper: #e6dcc0;
--bg-hover: var(--secondary); --bg-active: var(--accent);
--text: var(--foreground); --text-muted: var(--muted-foreground);
--accent-blue: #076678; --accent-green: var(--success);
--accent-red: var(--destructive); --accent-orange: var(--warning);
--font-sans: 'JetBrains Mono', ui-monospace, Menlo, monospace;
--font-mono: 'JetBrains Mono', ui-monospace, Menlo, monospace;
--font-heading: 'VT323', 'JetBrains Mono', monospace;
}
```
Two notes:
- **`.dark` vs `data-theme`.** The current store flips `.dark` on `<html>`. To
keep Carbon working unchanged, leave `.dark` logic alone and layer
`data-theme` on top: when a cyberspace theme is active the store sets
`data-theme` and **removes** `.dark` (cyberspace themes are self-contained —
they set both polarities explicitly). See §6.
- **Borders from `fg`.** cyberspace's hairline tracks the foreground, not a
fixed gray. `color-mix(in oklab, <fg> 22%, transparent)` reproduces that and
auto-flips between the two themes. Tune the % after visual review.
---
## 4. The dithered image component (`RasterImage.svelte`)
**File:** `web/src/lib/components/RasterImage.svelte` (sibling of the existing
`Spinner.svelte`).
### 4.1 API
```svelte
<RasterImage src={entity.iconUrl} alt="host icon" width={320} />
<!-- optional: scale (downsample factor), threshold bias, mono palette override -->
```
- `src`, `alt` — as `<img>`.
- `width` — render width in CSS px; canvas is sized to this × natural aspect.
Downscaling before dithering is what sells the "lo-fi" look (defaults ~256
320). Expose `scale` (01) to control.
- Reads the active theme's `--foreground` / `--background` via
`getComputedStyle(document.documentElement)` so the dither palette **follows
the theme** (cream/black in cyber-dark, black/cream in cyber-light, and
perfectly sensible in Terracotta/Carbon too).
### 4.2 Behavior
1. Show `.raster-image-skeleton` (empty, `background: var(--background)`) until
the source image loads — matches cyberspace's no-flash placeholder.
2. On load: create an offscreen canvas at `width × (h/w*width)`, `drawImage`
(with `imageSmoothingEnabled = true` for the downscale), pull
`getImageData`.
3. Run **Atkinson dithering** to 2 colors:
- For each pixel: luminance `Y = 0.299R + 0.587G + 0.114B`.
- Threshold at 128 (+ optional `bias`), snap to either `fg` or `bg`.
- Push **1/8 of the quantization error** to each of 6 neighbors (Atkinson's
kernel): right, below-left, below, below-right, and two pixels down on the
next-next row. (Atkinson diffuses less than FloydSteinberg → softer, more
"screen-printed" — exactly the cyberspace feel.)
- Write `fg`/`bg` (read from CSS vars at render time) into the buffer.
4. `putImageData`. Canvas is the visible output; the loaded `<img>` is kept as
`aria-hidden` fallback for no-JS / copy-image / accessibility.
5. **Re-dither on theme change**: subscribe to the theme store; when it flips,
re-read `--foreground`/`--background` and re-run steps 34 (cheap — the
decoded `ImageBitmap` is cached, only the palette pass reruns). This is the
detail that makes the art flip polarity with the theme toggle.
6. **Respect `prefers-reduced-data` / reduced motion?** Dithering is not motion,
but offer a `plain` prop to skip the canvas and render the raw `<img>` for
users who want crisp photos (e.g. entity detail screens where legibility
beats aesthetic).
### 4.3 Reference dither kernel (Atkinson)
```
* → 1/8 1/8
1/8 1/8 1/8 (current pixel = *)
1/8 1/8 (* is at top-left of this 4×? — see standard Atkinson spread)
```
Spread pattern (error e from pixel at (x,y) distributed):
```
px x+1 (1/8) x+2 (1/8)
x-1 (1/8) x (1/8) x+1 (1/8)
x+1 (1/8) x+2 (1/8) [next row offsets]
```
Concretely, 6 neighbors each get `e/8`: `(x+1,y)`, `(x+2,y)`, `(x-1,y+1)`,
`(x,y+1)`, `(x+1,y+1)`, `(x,y+2)`. (Clamp at edges — drop, don't wrap.)
### 4.4 Where to use it
- Entity icons / host thumbnails in the KB and entity desktop (the obvious win).
- Mascot or login/Config background art (`ConfigBackground.svelte` already
exists — a dithered backdrop there would be striking).
- Any user-uploaded image in chat/knowledge where we want the "de-imagined"
tone. Keep it **opt-in per call site** via the `plain` prop — don't dither
diagrams/screenshots that need to stay readable.
### 4.5 Cross-origin caveat
`getImageData` throws on tainted canvases. If `src` is cross-origin and the
server doesn't send CORS headers, fall back to the plain `<img>` (log once).
For self-hosted assets (the common case here) it's a non-issue.
---
## 5. Fonts: self-host JetBrains Mono + VT323
cyberspace self-hosts both faces as `.woff2`. oikos currently pulls DM Sans /
DM Mono / Inknut Antiqua from Google Fonts via a `<link>` in
`web/index.html:10`.
- Drop `JetBrainsMono-Regular.woff2` and `VT323-Regular.woff2` under
`web/static/fonts/` (or `web/public/fonts/` — match where static assets are
served from; check `vite.config`).
- Add `@font-face` blocks at the top of `app.css` with `font-display: swap`.
- For the cyberspace themes only, the `--font-sans`/`--font-mono`/`--font-heading`
overrides in §3 remap the families — Terracotta/Carbon keep DM Sans/Inknut
untouched. This is the key trick: **font choice is part of the theme**, not a
global swap, so the two art directions don't fight.
- Leave the Google Fonts `<link>` in place for now (Terracotta/Carbon still need
it); add a follow-up to self-host those too if we want to kill the external
request entirely. Out of scope for this plan.
VT323 vs Departure Mono: VT323 is free on Google Fonts and trivial to self-host;
Departure Mono is the authentic cyberspace face but needs a license check.
**Recommend VT323** to start; swap to Departure Mono later if you want exact
fidelity.
---
## 6. Theme store changes (`web/src/lib/stores/theme.svelte.ts`)
Current: `Theme = 'light' | 'dark'`, flips `.dark` class. Extend to a named set
while preserving the existing API (callers of `toggleTheme`/`getTheme` keep
working):
```ts
export type ThemeName = 'terracotta' | 'carbon' | 'cyber-dark' | 'cyber-light'
// Back-compat aliases used by existing callers:
// 'light' -> 'terracotta', 'dark' -> 'carbon'
```
- Store key stays `oikos-theme`; migrate old `'light'`/`'dark'` values on read.
- `applyClass` becomes `applyTheme`: sets `data-theme` on `<html>` and toggles
`.dark` **only** for `carbon` (so Terracotta and both cyberspace themes run
with no `.dark`). This is important: the `.dark` block in `app.css` must not
layer on top of the cyberspace token blocks — cyberspace sets its own
polarities.
- Update `THEME_LABELS` to the four names; update whatever UI surfaces the
picker (search for `THEME_LABELS` / `toggleTheme` usages — likely
`Settings.svelte` or the desktop shell's chrome) to a 4-option control instead
of a binary toggle.
**Watch out:** any code that assumes `document.documentElement.classList.contains('dark')`
≡ "dark colors" will be wrong for `cyber-dark`. Audit `grep -rn "classList.*dark\|\.dark" web/src` and prefer reading `getTheme()`/`data-theme` instead.
---
## 7. Optional cosmetic idioms (additive utilities)
Small, theme-aware utilities in `app.css` — usable in any theme but idiomatic
for cyberspace:
- `.terminal-box` — `{ border:1px solid var(--border); border-radius:0 }` plus a
`.terminal-box:focus-within { box-shadow: 0 0 0 2px var(--ring) }` to mirror
the `ring-2 ring-fg` focus. Lets cards opt into the terminal look without a
component rewrite.
- `.braille-spinner` — keyframe cycling `⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏` as `::after` content, colored
`var(--muted-foreground)`. Alternative to `Spinner.svelte` for loading states
under cyberspace themes.
- `.font-vt` — `{ font-family: var(--font-heading) }` so the wordmark class
cyberspace uses maps to our heading var (VT323 under cyber themes, Inknut
under Terracotta). Drop-in for any stylized title.
- `.strike-list` — `li > s { color: var(--muted-foreground) }` convenience for
the crossed-out feature-list pattern in marketing/empty states.
None of these are required for the theme to work; they're palette for the
"de-imagined" voice where we want it.
---
## 8. Implementation order (incremental, each step shippable)
1. **Fonts** (§5) — self-host JetBrains Mono + VT323, `@font-face` in app.css.
No visual change yet (only cyberspace themes reference them).
2. **Tokens** (§3) — add the two `:root[data-theme='cyber-*']` blocks.
3. **Store** (§6) — extend `theme.svelte.ts` to named themes + `data-theme`;
update the picker UI. **At this point both cyberspace themes are live and
fully recolor the whole app** — the cheapest milestone, biggest visible win.
4. **`RasterImage.svelte`** (§4) — build + wire into entity icons and
`ConfigBackground`. This is the "dithered image" deliverable.
5. **Idioms** (§7) — terminal-box, braille spinner, etc., applied opportunistically.
Each step is independently mergeable. Step 3 alone satisfies "light + dark
cyberspace themes"; step 4 satisfies "dithered images."
---
## 9. Verification
- `cd web && npm run build` (or the repo's build command — confirm in
`web/package.json`) — Tailwind v4 must accept the new `data-theme` selectors
and `color-mix()` (both standard; no config change expected).
- `npm run check` / `svelte-check` for the store + component TS.
- Manual: cycle all four themes in the picker; confirm no `.dark` bleed on
`cyber-light`; confirm `RasterImage` re-dithers on theme flip; confirm
`prefers-reduced-data`/`plain` prop shows crisp image; confirm cross-origin
`src` degrades to `<img>` without console errors.
- Lighthouse / a11y: 1-bit dithered images still need a real `alt` (kept on the
fallback `<img>`); contrast on `#a89984`-on-black passes WCAG AA for body text
(ratio ≈ 7.4:1) — fine.
---
## 10. Alternatives considered
- **Full rebrand (replace Terracotta/Carbon).** Highest visual payoff, highest
cost: every component's rounding/serif/spacing was authored for the Art
ouveau
direction; square + mono would need a component-level sweep, not just tokens.
Defer unless you want cyberspace as *the* oikos look — then do it as a
follow-up that deletes Terracotta/Carbon and makes `cyber-dark` the sole
default.
- **CSS-only image dither (filters / SVG turbulence).** Cheaper, but can't do
true 1-bit error diffusion or recolor to theme fg/bg. Rejected — the canvas
pass is the whole point and is ~60 lines.
- **Ordered (Bayer) dither instead of Atkinson.** More regular/grid-like
("newspaper halftone"). Atkinson is softer and more terminal-like; keep
Bayer as a `algorithm='bayer'` prop option later if wanted.
- **Server-side dithering.** Could pre-dither icons at ingest. Rejected for
now — client canvas keeps one source of truth (the original image) and lets
the palette follow the live theme, which a baked asset can't.
---
## 11. Non-goals / out of scope
- Replicating cyberspace's sidebar-rail *layout* (oikos uses a floating-window
desktop shell; the rail is a different app model). We take the *visual*
language, not the IA.
- Porting the 9 other novelty themes (C64, Matrix, VT320, …). Two (light/dark)
satisfy the request; the token model makes adding more trivial later.
- Removing the Google Fonts dependency for Terracotta/Carbon (follow-up).
- Licensing/redistributing Departure Mono (use VT323 unless cleared).
---
## 12. Risks
- **`.dark` coupling.** Existing code may equate `.dark` with "dark UI".
Mitigation: audit in step 3; the grep is small.
- **Dither perf on large images.** Atkinson is O(n) and runs on a downscaled
canvas (≤~320px wide), so per-image cost is negligible; but batch-rendering
many entity icons on first paint could jank. Mitigation: dither lazily (on
intersection) and cache the result on the element.
- **Tainted canvas** on cross-origin images → silent fallback to `<img>`
(already handled in the design).
- **Token drift.** If a component hardcodes a color instead of using a token,
it won't recolor under cyberspace. This is the same risk Carbon already has;
no new exposure, just more visible under a stronger theme.

View File

@@ -0,0 +1,184 @@
# 2026-08-03 — Review: nomos chat reliability/UX changes (F1F7)
**Status:** Implemented (P0, P1, P2 all done). See
[Resolution](#resolution) at the end.
A critical self-review of the uncommitted F1F7 changeset
(`plans/2026-08-03-nomos-chat-reliability-and-ux-audit.md` Resolution). The
change set is mostly sound and builds/tests green, but **F1 introduced one
real lost-work regression** by changing the contract of `resumeSession` (it can
now skip) without updating two callers that mutate state *before* calling it.
That must be fixed before this ships.
## What was changed (for orientation)
- F1 `cmd/nomos/turngate.go` (+test): per-session single-flight; `resumeSession`
acquires non-blocking and **skips** if a turn is active; `handleChat` live path
acquires with a 5s wait.
- F2/F3 `web/src/lib/stores/chat.ts`: humanized errors, `clearTurnState` on
terminal `task.status`, turn-free reconnect.
- F4 streaming in global `activityLog` + inline `ToolCallCard`.
- F5 artifact/knowledge deep links; F6 step-first headline; F7 stable layout.
---
## P0 — F1 loses finished-execution continuations (must fix before shipping)
**Bug.** `processContinuations` (`cmd/nomos/continue.go:166-167`) calls
`a.store.markContinued(ctx, p.ExecID)` **before** dispatching
`continueSession → resumeSession`. `markContinued` sets `continued_at`, and
`pendingContinuations` (`store.go:1763`) filters `WHERE continued_at IS NULL`
so a marked execution is **never re-queued**.
Before F1, `resumeSession` always ran, so marking-first was safe. F1 made
`resumeSession` skip when a turn is already active for the session. Now:
- **Two executions for one session finish near-simultaneously** (the common
multi-step case): the loop marks BOTH, spawns two goroutines; goroutine 1
acquires and runs, goroutine 2's `resumeSession` **skips** → execution 2 is
marked continued but its result is **never fed back to the agent. Lost.**
- **A live turn is streaming when an async execution finishes**: continuation
marks + dispatches; `resumeSession` skips (live turn holds the permit) →
result lost.
This silently drops auto-continuation — worse than the interleaving F1 set out
to fix.
**Fix.** Make `resumeSession` report whether it actually ran, and mark-continued
only after a successful run; on a busy-skip, leave the execution pending for the
next worker tick.
1. `cmd/nomos/continue.go` — change `resumeSession` to return `bool`:
```go
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
if !a.gate.acquire(sessionID, 0) {
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
return false
}
defer a.gate.release(sessionID)
…existing body…
return true
}
```
2. `continueSession` — mark only after a real run; on skip, leave pending:
```go
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
return
}
a.store.markContinued(ctx, p.ExecID)
}
```
3. `processContinuations` — **delete** the `a.store.markContinued(ctx, p.ExecID)`
line at `continue.go:166` (the dispatch `safego.Go(... continueSession ...)`
stays). The `markContinued` at `:162` (the no-assent-window branch, which
saves a note and does **not** call resumeSession) stays as-is — that path
intentionally consumes the item.
4. Update every other `resumeSession` caller to ignore the new return value
(`/resume`, `handleAnswerQuestion`, the empty-message reconnect in
`handleChat`) — they don't need the bool; a bare call discards it. No behavior
change for them (their skip semantics are already correct/desired).
**Why this preserves the original "no re-continue loop" guarantee:** a
`resumeSession` that *runs* always returns `true` (even on its internal LLM
failure path — it has already persisted a failure note), so it gets marked and
won't loop. Only a *busy-skip* returns `false` and stays pending, which is
correct (retry once the turn frees). Crash-safety also improves: a crash between
acquire and mark leaves the item un-marked → re-queued on restart.
**Validation:**
- New test: two `pendingContinuation`s for one session dispatched concurrently;
assert both are eventually processed (both `continued_at` set) and at no point
do two `resumeSession` bodies overlap (reuse the `turnGate` single-flight
pattern, or assert via a shared counter in a stubbed `chatWith`).
- Existing `cmd/nomos` suite stays green; `go vet` clean.
---
## P1 — F1 can false-auto-close a merely-busy session (low risk, fix for robustness)
**Bug.** `processIdleSweep` (`continue.go:78-89`) bumps `completion_nudges`
**before** calling `resumeSession`. If `resumeSession` skips (busy), the nudge is
counted as unanswered; the next sweep sees `CompletionNudges >= 1` and
**auto-closes** a session that was just busy.
**Likelihood is low** because `staleGoalSessions` (`store.go:1336`) filters
`last_active_at < now() - threshold` and an active turn keeps updating
`last_active_at` — so a busy session shouldn't appear stale. But the coupling is
the same shape as P0 and worth closing.
**Fix.** Gate the bump on the run, mirroring P0:
```go
safego.Go("nomos:idle-nudge:"+s.ID, func() {
note := …
if a.resumeSession(ctx, s.ID, note) {
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil { … }
}
})
```
(If skipped, leave `completion_nudges` at 0 so a genuinely-stale sweep nudges
again later.)
---
## P2 — Minor / hygiene (optional, can ship without)
- **Redundant catch-up turn on reconnect.** When the live turn *already ended*
before a dropped-SSE reconnect fires, the empty-message path still runs a
"report your state" `resumeSession` turn the operator didn't ask for. F1 makes
it non-concurrent (good) but it's still a spare turn. Consider: in
`handleChat`'s empty-message branch, skip the `resumeSession` if the session
is already terminal (`done`/`failed`/`abandoned`) or had activity within the
last few seconds — just return 202 and let the poller catch up.
- **Top-level side-effect on import.** `chat.ts` now calls `subscribeEvents()` +
`liveEvents.subscribe(...)` at module top level. It works (and `vitest` stays
green because tests mock `./chat`), but a hidden SSE-connect-on-import is
fragile for future tests. Prefer a lazy `ensureChatEventSync()` called from
the window mount path, matching how `workspace.ts` subscribes inside
`startWorkspace` rather than at import.
- **F7 follow-up (already documented):** the `NewTaskChat → SessionChatWindow`
window-swap on first send still flashes; an in-place handoff would remove it.
- **Pre-existing, not introduced:** `a.chat` retries the LLM stream on
`ctx`-cancellation (client disconnect) up to 3×, holding the turn permit a few
extra seconds. Out of scope here.
---
## Out of scope
- F8 (ordering toggle + live background tool-delta streaming) — deferred in the
original plan; its main symptom is removed by F1.
- `run` execution deep-links (need an execution-view opener).
## Recommended order
1. **P0** (lost continuations) — blocks shipping F1.
2. **P1** (idle-sweep nudge gate) — small, same pattern.
3. P2 items as time allows.
4. Re-run `go test ./cmd/nomos/`, `go vet`, web `vitest`, `vite build`; keep
`VERSION` at `0.15.0` (these are correctness fixes to the same changeset, not
a new bump) — or bump patch to `0.15.1` if shipped as a follow-up commit.
---
## Resolution
All review items implemented. The whole batch (F1F7 + these review fixes)
remains one uncommitted changeset at `VERSION 0.15.0`.
| Item | Fix | Where |
|---|---|---|
| **P0** | `resumeSession` returns `bool` (false on busy-skip). `continueSession` marks an execution `continued` **only after** the turn ran; on a skip it defers and the next worker tick retries (item stays pending). Removed the pre-dispatch `markContinued` in `processContinuations`. Other callers (`/resume`, answer-question, reconnect) ignore the return. | `cmd/nomos/continue.go` |
| **P0 test** | `TestResumeSession_SkipsWhenBusy`, `TestContinueSession_DefersWhenBusy` — DB-free contract tests proving the skip path returns false without running the body (nil provider would panic otherwise). | `cmd/nomos/continue_test.go` |
| **P1** | Idle sweep bumps `completion_nudges` only after `resumeSession` actually runs, so a busy-skip can't be counted as an unanswered nudge → no false auto-close. | `cmd/nomos/continue.go` (`processIdleSweep`) |
| **P2.1** | Empty-message reconnect (now defensive — the frontend no longer POSTs empty messages post-F2) skips a terminal session instead of spawning a spare "report state" turn. | `cmd/nomos/main.go` (`handleChat`) |
| **P2.2** | Event subscription armed lazily from `chatFor()` (`ensureChatEventSync`) instead of at module import — no SSE-connect-on-import side-effect. | `web/src/lib/stores/chat.ts` |
**Verification:** `go test -count=1 ./cmd/nomos/` green (incl. the two new
contract tests); `go vet` clean. Web `vitest` 70/70; `vite build` succeeds; no
new `tsc`/eslint errors in any touched file.
**Note on the P0 end-to-end test:** the full "two continuations both processed,
no overlap" scenario needs a live LLM provider (chatWith isn't stubbable without
a refactor) and was therefore covered at the contract level (the skip returns
false without running the body) plus the existing `turnGate` single-flight test
for serialization, rather than as a DB integration test.

View File

@@ -0,0 +1,356 @@
# 2026-08-03 — Nomos chat: reliability & predictability audit
**Status:** Implemented (F1F7) in v0.15.0; F8 deferred. See
[Resolution](#resolution-2026-08-03) at the end.
**Scope:** The live chat/task UX across one production session, audited through
the code paths behind each operator-reported symptom —
`cmd/nomos/{main.go,agent.go,continue.go,store.go}`,
`web/src/lib/stores/{chat,activity,execstream,events,workspace}.ts`,
`web/src/lib/components/{ChatThread,AgentTrace,ToolCallCard,UnifiedTimeline,TaskContextPanel,SessionChatWindow}.svelte`.
**Trigger:** Operator report — streaming invisible in the tool card; the
activity/plan panel wrong about parallel/nested runs and timestamps with no clear
sequence; no links to artifacts/knowledge referenced in chat; agent "thinking"
flickers/overwrites itself; layout jumps when a chat goes from empty to content;
"Agent connection lost / Error in input stream" messages that aren't actionable
and don't self-resolve; overall flaky/disconnected feel where the task never
cleanly ended.
The prior round (`2026-07-30-session-review-plan-drift-and-dead-activity-panel.md`,
shipped in `467589d`) fixed the plan-seq and fabricated-timestamp rendering bugs.
This round's symptoms are a different layer: **turn orchestration, streaming
wiring, and connection-state UX**. One architectural gap (F1) is the common
cause behind several of them.
---
## The one root cause that compounds everything: F1
### F1 — No per-session turn serialization (concurrent turns corrupt the view)
`handleChat` runs `a.chat(ctx, ...)` directly in the HTTP request goroutine, and
every "resume" path (`resumeSession`, the reconnect empty-message path, the
auto-continuation worker, the idle sweep, answer-question) launches **another
goroutine** (`safego.Go`) running a full turn. There is **no mutex keyed on
`sessionID`** anywhere. The codebase already knows this is a hazard —
`agent.go:316-323` marks approved executions `continued` specifically because
"two concurrent LLM calls for the same session cause empty responses and race
conditions" — but the fix is per-path patching, not a general lock.
What this produces, deterministically:
- A network blip on the browser↔nomos stream fires `handleDisconnect`
(`chat.ts:383`), which POSTs an **empty-message reconnect**
`main.go:194-206` spawns `resumeSession` as a **new goroutine**. If the
original turn is still alive (or finishes its current tool call), **two turns
now run for one session**: interleaved `tool_use`/`text_delta` events, a
re-proposed plan, and "the agent is repeating itself."
- The activity timeline (`activity.ts:119-185`) groups tools under a plan step
by *inferring* `currentStepSeq` from `update_plan_step` calls in the message
stream. Two interleaved turns make that inference wrong → tools land under the
wrong step, steps appear to nest/parallelize that never did, the sequence
reads as garbage. This is the "parallel runs / nesting / no clear sequence"
report.
- Two turns appending to the same session's messages is also the source of the
duplicate-tool-call/empty-response class of bugs the prior plan docs keep
patching individually.
**This is why the experience "felt flaky and disconnected" and "the task didn't
end":** the panel is faithfully rendering a corrupted, interleaved event stream.
### Fix (proposed)
1. **One in-flight turn per session, server-side.** Add a per-`sessionID`
turn mutex (a `sync.Map[string]*singleflight` or a keyed `sync.Mutex`) in
`handleChat`/`resumeSession`/`continue.go`. A second attempt to start a turn
for a session that already has one running must **queue** (preferred — the
operator's message waits its turn) or **return 409 "turn in progress"** (the
frontend then just re-polls; no new goroutine). This single change removes
the interleaving that drives F2/F3/F8.
2. **Make the empty-message reconnect a no-op when a turn is already running.**
Today it *always* spawns `resumeSession`. Gate it on "is any turn active for
this session?" — if yes, return 202 and let the existing turn + the poller do
the work. A blip should never *create* work.
---
## F2 — Reconnect spawns a new turn and surfaces raw, non-actionable errors
`chat.ts:383-426` `handleDisconnect`: on a dropped SSE it sets
`connectionState='disconnected'`, starts the 3s poller, shows
`"Agent connection lost. The task is still running — retrying…"`, then calls
`streamChat('', sid, …)` up to 3× — each of which is the empty-message POST that
triggers F1's new `resumeSession` goroutine. Separately, the LLM stream errors
surface verbatim: `agent.go:388` does `emitError("llm: %v", err)`, so an
OpenRouter transport break reaches the operator as `llm: error in input stream:
…` (the openai-go SDK's SSE-reader text), shown raw in `ChatThread`'s error bar.
Combined with F1, this is the exact "messages not actionable and not
self-resolving" + "task didn't end" experience: a blip both invents a duplicate
turn and paints a scary, unfixable error that lingers.
Secondary defects in the same path:
- `streaming` stays `true` for the entire reconnect window, so the composer is
disabled and the poller's `if (streaming && connected) return` guard
(`chat.ts:177`) suppresses updates except while disconnected — fragile.
- The **per-window** error path (`sendSessionMessage`, `startTask`) does **not**
auto-reconnect at all — it only polls. Its `onReconnect` in
`SessionChatWindow.svelte:110` is `() => loadSessionChat(sessionId)`, which
just *re-fetches the transcript* and never re-attaches to a live stream. And
the global `reconnect()` (`chat.ts:428`) keys off the **global**
`currentSession`, so a floating window's Reconnect button can target the wrong
session. Two different, both-broken reconnect behaviors.
### Fix (proposed)
1. **Stop the empty-message-reconnect from creating turns** (depends on F1.2).
Reconnect should mean "catch up," not "run more."
2. **Humanize + bucket error strings.** Map known transport errors to
operator-readable, actionable copy with a single primary action:
- `llm: …input stream…` / 502/503/timeout → "The model connection dropped.
The task is still running in the background — it'll catch up
automatically." (auto-dismiss when the next event/poll lands)
- `HTTP 401/403` → "Session expired — reconnect." (action: re-auth)
- unknown → show the raw text but behind a "Details" toggle, not as the
headline.
3. **Make errors self-resolving.** Clear the error + connection-lost banner the
moment the poller sees a newer message or any live event for the session
arrives (wire `eventsConnected` / a session-scoped event into the banner's
visibility). Today the banner stays until manual dismiss even after recovery.
4. **Unify reconnect.** One `reconnect(sessionId)` that (a) re-fetches the
transcript, (b) if no turn is active, is a pure no-op refresh; used by both
the main view and windows. Drop the global-`currentSession` coupling.
---
## F3 — The UI can't tell when a turn truly ended (so it never looks "done")
When the SSE stream ends without a `done` event, `streamChat`'s `onDone`
(`chat.ts:355-368`) calls `handleDisconnect`. Even if the backend turn then
finishes and persists its final message, the frontend only learns via the 3s
poller re-setting `messages` — but nothing transitions `streaming``false` or
`connectionState``connected` from that path, so the spinner/indicator and the
"connection lost" banner can persist indefinitely. That is "the task didn't
end / backend connection was lost."
The backend does emit a terminal signal — `task.status` events on
`complete_task`/auto-complete (`workspace.ts:82-88` `STATUS_AFFECTING`) — but
nothing in the chat store reacts to a terminal `task.status` to force
`streaming=false` + clear the banner. The signal exists; the chat ignores it.
### Fix (proposed)
1. **Treat a terminal `task.status` (done/failed) for the viewed session as
authoritative end-of-turn** in `chat.ts`: set `streaming=false`,
`connectionState='connected'`, dismiss any connection-lost error. The poller
already refreshes messages; this just closes the loop on the *state* flags.
2. **Add a `task.completed` / `turn.ended` SSE event** from the backend on every
terminal path (today `done` is a chat-stream-only event; background turns
have no equivalent). The always-on events stream already reaches the panel —
route the same signal to the chat store so background-completed turns clear
the UI without waiting on a poll.
---
## F4 — Command streaming isn't shown where the operator looks
Streaming **exists** (`execstream.ts` `liveExecutionOutputFor`, fed by
`fetchExecutionLogs` via the always-on events stream) and the
`UnifiedTimeline` **does** render `tool.liveOutput` with tail-pinned scroll
(`UnifiedTimeline.svelte:451-457`). But:
- The **global** `activityLog` (`activity.ts:236`) — used by the main Chat page's
panel — never calls `withLiveOutput`. Only the **per-window**
`activityLogFor(sessionId)` (`activity.ts:271`) attaches live output. So the
main chat view's timeline shows no streaming at all.
- The **inline chat tool cards**`ToolCallCard.svelte` (rendered inside
`AgentTrace.svelte`) — show only args/result/error. They never read
`liveOutput`. Expanding a running `run` call in the transcript (the natural
place to "check the tool") shows nothing live; output appears all at once when
the `tool_result` lands.
This is the report: "I expected checking on the tool to let me see the
streaming."
### Fix (proposed)
1. **Wire live output into the global `activityLog`** so the main chat panel
streams too (call `withLiveOutput` in the `activityLog` derivation, same as
`activityLogFor`).
2. **Show streaming in the inline tool card.** Pass the session's live-output
store into `AgentTrace`/`ToolCallCard` (or attach `liveOutput` to the running
`run` tool entry the way the timeline does) and render a tail-pinned `<pre>`
while the call is `tool_use`/running. Reuse the UnifiedTimeline's scroll-pin
pattern. Gated runs (queued-for-approval) should instead show a "queued —
watch in entity detail" affordance (per `execstream.ts` header comment).
---
## F5 — Artifacts and knowledge referenced in chat aren't navigable
When the agent records knowledge, the activity panel shows `Recorded: <title>`
(`activity.ts:188-203`) but it's plain text — no link. The backend already
emits `knowledge.recorded` and links the note to the task
(`store.go:1572 linkKnowledgeToTask`, `agent.go:594`), and the Wiki reader
exists (`web/src/lib/components/knowledge/WikiReader.svelte`). Nothing connects
them. Same for `get_entity`/`run` results: slugs and execution ids appear in
tool output but aren't clickable to open the entity window or execution view.
### Fix (proposed)
1. **Make activity/tool entries link-bearing.** Add an optional
`link?: { kind: 'knowledge'|'entity'|'execution', id: string }` to
`ActivityEntry`. Populate it from `upsert_knowledge` (title→knowledge id from
the result), `get_entity` (slug), and `run` (execution id). Render a
clickable chip that opens the right surface: knowledge → Wiki reader (new tab
/ window), entity → entity detail window, execution → execution log pane
(already fetched by `EntityDetailContent.svelte`).
2. **Render entity/knowledge mentions in assistant markdown as links** when they
resolve to known slugs (lightweight: a post-process pass on rendered text, or
let the model emit explicit `[slug](entity:…)` markers it already has tools to
discover).
---
## F6 — "Thinking" is an unstable single-line headline, not a predictable trace
`ChatThread`'s `indicatorLabel` (`ChatThread.svelte:83-89`) returns the **first**
running activity entry's description; `AgentTrace`'s `headline` mirrors it. As
tools fire sequentially the running entry changes, so the one line rewrites
itself every call — "the thinking overwrites itself." There is no persistent,
additive reasoning surface, and no predictable turn structure (plan → steps →
answer) the operator can learn to read. Claude-Code-style predictability is
absent.
### Fix (proposed)
1. **A stable, additive per-turn reasoning block.** Keep the collapsed trace as
a *summary* ("Step 2 of 4 · running `run`"), but when expanded show an
**append-only** log of (a) the model's intermediate `text` (reasoning before
each tool call — already emitted at `agent.go:458-460` and persisted) and
(b) each tool call as a fixed row, instead of a single mutating headline.
2. **Predictable turn shape.** Enforce/cue a consistent sequence in the UI —
Goal → Plan → Steps (each with its tools nested) → Final answer — and render
each phase as a stable section that fills in rather than a line that
overwrites. The UnifiedTimeline already models most of this; surface the same
model in the inline trace so chat and panel tell one story.
---
## F7 — Layout jumps when a chat goes from empty to content
`SessionChatWindow.svelte:58-63` gates the right rail on `hasContext`: empty
task → `ChatThread` full-width; first activity/touched entity → switches to
`Splitpanes` with the `TaskContextPanel` rail. The swap is instant and
**reflows the chat column width** the moment the first event lands — "switching
from empty to chat with something, the layout was off." Compounded by the
`NewTaskChat` → real `SessionChatWindow` window-swap on first send
(`NewTaskChat.svelte:17-22`).
### Fix (proposed)
1. **Reserve the rail's space from the start** (collapse to a thin sliver / icon
rail when empty) instead of mounting it on demand, so adding content doesn't
change the chat column width. Or animate the rail in.
2. **Avoid the window swap on first send** — let the new-task window *become* the
session window in place once the id is assigned (same component, swap the
store source) rather than close+open.
---
## F8 — Activity/plan ordering & parallelism *(largely a symptom of F1)*
With F1 fixed (no interleaved turns) the heuristic step-grouping in
`activity.ts` becomes reliable again. Remaining standalone items:
- The timeline is **newest-first** with ts-0 goal/pending parked at the bottom
(`UnifiedTimeline.svelte:119-127`); for a long task this can read as
"sequence is off." Consider an explicit **oldest-first / seq-ordered** mode
toggle, and always show the step number prominently so order is unambiguous
regardless of sort.
- Background/auto-continued turns still rely on the 3s poller for their result
to appear; until F3's terminal event lands, the panel can lag. The
always-on events stream already carries `plan.*` and `entity.touched` live —
extend it to carry per-tool `tool.*` deltas for background turns so the panel
is live, not polled, during autonomous work.
---
## Recommended sequence
| Order | Item | Why first |
|---|---|---|
| 1 | **F1** per-session turn mutex + no-op reconnect-when-busy | Removes the interleaving that is the root cause of F2/F3/F8 symptoms; everything else is cosmetics on top of a corrupted stream. |
| 2 | **F3** terminal-event → clear chat state | Once turns can't double, make "the task ended" unambiguous so the UI stops lingering. |
| 3 | **F2** humanized/self-resolving errors + unified reconnect | Turns the scary, sticky "connection lost / input stream" into recoverable, auto-clearing UX. |
| 4 | **F4** streaming in the global log + inline tool card | Highest-visibility "I can't see what it's doing" fix; small, isolated change. |
| 5 | **F6** stable additive reasoning trace | Predictability of the interaction model (the Claude-Code feel). |
| 6 | **F5** artifact/knowledge deep links | Navigation completeness. |
| 7 | **F7** layout stability | Polish. |
| 8 | **F8** ordering mode + live background deltas | Polish, partly free after F1. |
## Verification hooks (when implementing)
- `cmd/nomos`: a test that starts two turns for the same session and asserts the
second queues/is-rejected (no interleaved `tool_use` order in persisted
messages).
- `web/src/lib/stores`: extend `activity.test.ts`/`execstream.test.ts` — global
`activityLog` now carries `liveOutput`; tool-card live output renders while
`tool_use` and clears on `tool_result`.
- A reconnect/integration test: drop the SSE mid-turn, assert (a) no duplicate
`resumeSession` goroutine, (b) banner auto-clears on next event, (c)
`streaming` returns to false on terminal `task.status`.
---
## Note on method
This audit was done against the **code paths** behind the reported symptoms, not
a single session transcript (no MCP/DB access from this session). To tie a
specific finding to a specific past session, pull the session via
`docker exec oikos-postgres-1 psql -U oikos oikos -c "select id,goal,outcome
from agent_sessions order by last_active_at desc limit 5"` and cross-reference
its `agent_activity` rows / persisted messages against the F1 interleaving
signature (two assistant turns' tool ids interleaved in one message shell).
---
## Resolution (2026-08-03)
Implemented F1F7 in v0.15.0 (`VERSION 0.14.2 → 0.15.0`). F8 deferred (its
primary symptom — interleaved/out-of-order entries — is removed by F1; the
ordering toggle and live background tool-delta streaming remain as nice-to-
haves).
| Item | What shipped | Where |
|---|---|---|
| **F1** | Per-session single-flight turn gate (`turnGate`): at most one in-flight turn per session. Background resume paths (`resumeSession` — covers the continuation worker, idle sweep, answer-question, /resume, and the empty-message reconnect) skip non-blocking when busy; the live chat path waits briefly then bails with an actionable error instead of stacking a second turn. | `cmd/nomos/turngate.go` (+`turngate_test.go`), wired in `agent.go` (struct/init), `continue.go` (`resumeSession`), `main.go` (`handleChat`). |
| **F3** | Terminal `task.status` events (done/failed/abandoned/awaiting_input) now clear a stuck chat view's `streaming`/`connectionState` and dismiss the connection-lost toasts — the authoritative "turn ended" signal the UI was ignoring. Poller safety net catches the edge where the event fired during the disconnect window. | `web/src/lib/stores/chat.ts` (`clearTurnState`, liveEvents subscription, `startSessionPolling`). |
| **F2** | Raw errors humanized ("The model connection dropped. The task keeps running…") and bucketed; one connection surface per drop (not banner+toast+raw error); errors self-clear via F3. The turn-spawning reconnect attempt loop is gone (dead global path simplified to a turn-free refresh); window "Reconnect" re-fetches + resets state. | `web/src/lib/stores/chat.ts` (`humanizeChatError`, error handlers, `loadSessionChat`, `handleDisconnect`/`reconnect`), `web/src/lib/components/ChatThread.svelte` (banner copy). |
| **F4** | Command streaming now shows (a) in the **global** activity timeline (live output wired into `activityLog`, was only per-window) and (b) in the **inline chat tool card** — expanding a running `run` shows live output auto-opened and tail-pinned. | `web/src/lib/types.ts` (`liveOutput`), `web/src/lib/stores/activity.ts` (`currentLiveOutput`), `web/src/lib/components/ChatThread.svelte` (`toolsWithLive`), `web/src/lib/components/ToolCallCard.svelte`. |
| **F6** | The "thinking" headline is now step-first (stable across a step's many tool calls) instead of rewriting per command; falls back to the current tool / "thinking…" only when no step is active. | `web/src/lib/components/ChatThread.svelte` (`indicatorLabel`). |
| **F5** | Activity entries now carry a deep link: recorded knowledge docs and `get_entity` lookups get an "open artifact" chip that opens the entity/knowledge window directly. | `web/src/lib/stores/activity.ts` (`link`, `knowledgeLinkFromResult`, `entityLinkFromArgs`), `web/src/lib/components/UnifiedTimeline.svelte`. |
| **F7** | The empty→content layout reflow is gone: `SessionChatWindow` now has one stable `Splitpanes`+`ChatThread` from open (no more destroy/remount of the thread or column reflow when the rail appears). | `web/src/lib/components/SessionChatWindow.svelte`. |
**Verification:**
- `go test ./cmd/nomos/` green (incl. new `turngate_test.go`: non-blocking skip,
blocking-waits-for-release, timeout, and a 50-goroutine single-flight
concurrency test asserting max in-flight = 1). `go vet` clean.
- Web `vitest` 70/70 green (incl. `activity.test.ts`/`execstream.test.ts`); the
`activity.test.ts` chat mock gained `currentSession` for the new
`currentLiveOutput` derivation.
- `vite build` succeeds (all Svelte components compile). Pre-existing `tsc`
strictness errors in unrelated files (`ui/*`, `oidc.ts`, `windows.ts`,
`workspace.ts`) are unchanged; no new errors in any touched file.
**Follow-ups (not in this pass):**
- F8: oldest-first ordering toggle; emit per-tool `tool.*` events on the
always-on stream during background `resumeSession` turns so the panel is live
(not 3s-polled) during autonomous work.
- F5: `run` execution deep-links (open the entity detail's execution pane) —
needs an execution-view opener; knowledge/entity links shipped first as the
explicit complaint.
- F7: the `NewTaskChat → SessionChatWindow` window-swap on first send (a
windows.ts open/close) still causes a brief flash; an in-place handoff
(same window, swap store source) would remove it.

View File

@@ -0,0 +1,184 @@
# 2026-08-03 — Nomos chat: working-visibility, message queue, generation-aware timeline
**Status:** Implemented (F1F4) in v0.17.0. See
[Resolution](#resolution-2026-08-03) at the end.
## Context (grounded in last-session logs + DB, not just code)
Operator report: *"On the chat window I can't tell the agent is working; it's
making tool calls but no feedback. Typing returns 'Nomos is still finishing a
previous step…'. Activity not up to date. Several plans at once, some don't
execute."*
Verified against runtime state:
- **Last session `23da10db`** ran ONE live turn for **6m33s** (21 iterations,
19:45:36→19:52:03, correlation `679566cb`). At 19:48:00 the operator typed
`status`; at **19:48:05 the turn gate deferred it** (`turn already active,
deferring operator message`). The operator could type at all only because the
client had already lost the stream (`streaming=false`) while the server kept
running — i.e. the client showed an *idle* window over a *working* turn. It
ended `awaiting_input`.
- **Turn runtimes are long**: sessions in the DB run 15-27 min
(e.g. `44df8802` 24:24, `4319b9f8` 27:05). `handleChat` (main.go:170) has **no
SSE keepalive**; inter-iteration gaps reach 20-40s, so a proxy/browser idle
close mid-turn resets `streaming` while the turn continues on
`context.Background()` (pctx).
- **Re-proposing is real**: `44df8802` has **generation 1 (5 steps, all
`replaced`) → generation 2 (25 steps, done)**, with **2 `propose_plan` + 52
`update_plan_step`** calls persisted. The activity timeline renders every one
of those across both generations.
## Root causes
- **G1 — "working" == `streaming`.** Every working-indication in the chat window
(AgentTrace running status, indicator headline, stream cursor, panel spinner,
`disabled={streaming}` input) is gated on the live SSE flag. A background turn
(`resumeSession`/continuation worker) has no stream; a desynced long live turn
has a dead stream. In both cases `streaming=false` while the server is actively
working. The **session `status`** (`planning`/`executing`/`awaiting_input`) is
the reliable "server is running a turn" signal and is already live-refreshed
(`workspace.ts` `taskFor`, `STATUS_AFFECTING`), but the chat UI never uses it.
- **G2 — busy-turn message is rejected, not queued.** main.go:292-302: the turn
gate waits 5s then emits the "still finishing a previous step" error and
returns. The user message *is* persisted (main.go:270) but is **inert** — the
user must manually re-send.
- **G3 — activity is poll/event laggy.** Tool-level activity derives from
`messages`, refreshed only by the 3s poller; plan steps are **events-only**
(`workspace.ts` `hydrateSession`) with no poll, so a missed `plan.proposed`
event leaves the panel stuck on a stale generation.
- **G4 — timeline is generation-unaware.** `activity.ts` `computeActivityLog`
walks **all** messages' tool calls, so a re-proposed task renders N
"Proposed plan" entries and attributes tools to steps via `currentStepSeq`
inferred from `update_plan_step` calls across **every** generation — tools land
under the wrong (current-gen) step or under steps that were `replaced`. This is
the "several plans / some steps never run" view.
## Fixes (ordered)
### F1 — Status-driven `working` signal (fixes G1)
Add a derived store `taskWorking(sessionId)` = `$streaming OR status ∈
{planning, executing}` (explicitly **not** `awaiting_input` — that is paused for
input), plus a global `currentWorking` for the main view backed by `currentTask`.
Use it wherever `streaming` currently drives "is it working":
- `ChatThread.svelte`: `traceStatus` last-message = `working ? 'running' : …`;
`indicatorLabel` and the AgentTrace `status`/`label` props.
- `TaskContextPanel.svelte:137` spinner and `UnifiedTimeline` `streaming` prop →
`working`.
- Keep a separate `streaming` for the literal "live text deltas are arriving"
cursor; `working` is the superset for indicators/input.
- Input stays **enabled** while `working` (the user must be able to interject);
the send path queues when busy (F2). Show a muted "Nomos is working…" hint in
the composer when `working && !streaming`.
### F2 — Queue operator messages; auto-run when free (fixes G2)
- Server: in-memory per-session FIFO on the `agent` struct (mirrors `turnGate`),
`{message, reply}` entries. `handleChat`: when the gate is busy, **enqueue**
instead of rejecting, and emit a `queued` SSE event (replaces today's error at
main.go:294-302). Persist the user message as today (already done pre-acquire).
- Drain: arm a per-session drainer that, on gate release, acquires again and runs
the next queued message as a normal turn (same persist/emit path as
`handleChat`). Strictly one-at-a-time under the gate — this cannot stack turns
(the hazard v0.15.0 F1 removed); background `resumeSession` keeps its
non-blocking skip and never touches the queue.
- If the session is terminal (`done`/`failed`) or `awaiting_input` when a queued
message runs, `reopenSession`/answer handling applies as for any follow-up.
- Frontend: on the `queued` event show an inline "Queued — will run when the
current step finishes" chip on that user bubble; clear it when the turn's real
events begin. Drop the humanized "still finishing" error for the busy case.
### F3 — SSE keepalive on `handleChat` (prevents the G1 desync at the source)
Wrap `a.chat(...)` in a goroutine + `select` with a **10-15s ticker** that writes
an SSE comment (`:keepalive\n\n`) and flushes, so 20-40s inter-iteration gaps no
longer trip proxy/browser idle timeouts. Stop the ticker when `a.chat` returns.
(EventSource ignores comment lines by spec — safe.)
### F4 — Generation-aware timeline + self-healing plan panel (fixes G3/G4)
- `activity.ts` `computeActivityLog`: find the **last** `propose_plan` in the
message stream; ignore `propose_plan`/`update_plan_step` calls **before** it
for both rendering and `currentStepSeq` inference. Render at most one
"Proposed plan" entry (the current generation). Steps continue to come from
`$steps` (already current-gen via `fetchPlan` MAX(generation)). Optionally emit
a single "Plan revised" entry when >1 generation exists.
- Plan-panel resilience: on any `STATUS_AFFECTING` event (and on reconnect),
re-fetch the plan (`fetchPlan`) in addition to the live `plan.proposed` handler,
so a missed event self-heals instead of leaving a stale generation.
## Validation
- `go test ./cmd/nomos/`: extend `turngate_test.go`/new `messagequeue_test.go`
queued message runs strictly after release; FIFO order preserved across 3
queued sends; a background `resumeSession` busy-skip does **not** consume or
starve the queue; queued message runs even if session went `awaiting_input`.
- Web `vitest`: `activity.test.ts` — add a 2-generation fixture (2× propose_plan,
interleaved update_plan_step) asserting exactly one "Proposed plan" and correct
step attribution to gen-2 steps; `chat`/store test — `working` is true from
`status==='executing'` even with `streaming=false`; `queued` event renders the
queued chip and clears on first tool_use.
- Manual: (a) start a long task, **reload the window mid-turn** → the working
indicator stays on (status-driven); (b) send a message mid-turn → "Queued" →
runs after the turn; (c) open `44df8802`-style 2-gen session → timeline shows
one plan, no ghost proposals.
## Risks
- **F2 must not reintroduce concurrent turns.** The queue drains one-at-a-time
under the gate; background resume remains non-blocking and queue-agnostic.
Existing `turngate_test.go` concurrency assertion (max in-flight = 1) must stay
green.
- **Status-driven `working` could stick on** if a terminal event is missed.
Mitigated by the existing terminal `task.status``clearTurnState` recovery
plus a `loadSessions` refresh on reconnect (F4).
- **Keepalive comments** must stay SSE comments (`:` prefix) so they aren't
parsed as events.
## Out of scope / follow-ups
- Model efficiency: the 8+ pure-exploration iterations (repeated
`list_entities`/`get_relations`) that inflate turn length to 15-27 min —
prompt/iteration-budget tuning, separate effort.
- F8 from the prior plan (oldest-first timeline toggle; per-tool `tool.*` events
for background turns). F1's status-driven `working` makes background work
visible without live per-tool deltas, so this remains lower priority.
## Open implementation note
Host the per-session message queue on the `agent` struct (in-memory `map[string]
[]queuedMsg` + per-session drainer goroutine), mirroring `turnGate`. No DB table
needed — messages are already persisted by `handleChat` before enqueue; the queue
only schedules *when* a turn runs, not *whether* the message is stored.
---
## Resolution (2026-08-03)
Implemented F1F4 in v0.15.1 → v0.17.0 (the intermediate 0.16.0 was the
cyberspace-aesthetic commit, landed via auto-pull during this work).
| Item | What shipped | Where |
|---|---|---|
| **F1** | Status-driven `working` signal (`taskWorking(sessionId)` / `currentWorking`) = live stream OR session status ∈ {planning, executing}. Drives the chat trace running state, the "thinking" headline, the activity spinner, and the timeline `streaming` prop — so a background/long/desynced turn still looks alive (the "can't tell it's working" symptom). The composer stays enabled during background work so the operator can interject. | `web/src/lib/stores/workspace.ts` (`isWorking`, `taskWorking`, `currentWorking`), `ChatThread.svelte` (`working` prop, `traceStatus`, indicator), `TaskContextPanel.svelte`, `SessionChatWindow.svelte`, `NewTaskChat.svelte`. |
| **F2** | Operator messages sent during an in-flight turn are now QUEUED and auto-run when the gate frees, replacing the "still finishing a previous step… send it again" rejection. Per-session in-memory FIFO drained strictly one-at-a-time under the turn gate (no concurrent-turn reintroduction). A `queued` SSE event tells the client, which drops the optimistic bubble and shows a "Queued — will run when it finishes the current step" hint (derived from `working` + last-message shape, so it survives the poller). | `cmd/nomos/messagequeue.go` (+`messagequeue_test.go`), `agent.go` (queue field), `main.go` (`runChatTurn`, `drainQueued`, handleChat queue path), `continue.go` (resumeSession drains on release), `web/src/lib/types.ts` (`ChatQueuedEvent`), `chat.ts` (`queued` handling in sendSessionMessage/startTask). |
| **F3** | SSE keepalive: a 12s `:keepalive` comment ticker during `handleChat` so 20-40s inter-iteration gaps no longer trip a proxy/browser idle timeout (the desync root cause). All SSE writes (events + keepalive) serialized through one mutex — `http.ResponseWriter` is not concurrency-safe. | `cmd/nomos/main.go` (`writeMu`/`writeEvent`, keepalive goroutine). |
| **F4** | Generation-aware activity timeline: only the LAST `propose_plan` renders as "Proposed plan"; superseded ones collapse to a single "Earlier plan revised" marker, and step-attribution only follows the current generation's `update_plan_step` calls. Plus plan-panel self-heal: the plan is refetched (debounced) on any task-lifecycle event so a missed `plan.proposed` no longer freezes the panel on a stale generation. | `web/src/lib/stores/activity.ts` (`computeActivityLog`), `workspace.ts` (`schedulePlanRefetch`). |
**Verification:**
- `go vet ./cmd/nomos/` clean; `go test ./cmd/nomos/` green, incl. new
`messagequeue_test.go` (FIFO, requeueFront, per-session isolation, concurrency,
drainQueued no-op-on-empty, drainQueued requeues-when-busy). Existing
`turngate_test.go`/`continue_test.go` still green (single-flight guarantee
intact).
- Web `vitest` 72/72 green (added 2 F4 generation-awareness tests to
`activity.test.ts`: one "Proposed plan" + revised marker + current-gen-only
step attribution; plan-less Q&A attributes nothing).
- `vite build` succeeds. `tsc --noEmit` shows only the pre-existing baseline
errors (`ui/*`, `oidc.ts`, `windows.ts`, `workspace.ts:123/201/221`) noted in
v0.15.0 — no new errors from this change. ESLint: no new errors (the one new
`svelte/valid-compile` on `chatWorking` got the same disable its siblings have).
**Follow-ups (not in this pass):**
- Model efficiency: the long (15-27 min) exploration-heavy turns that made the
desync so painful — prompt / iteration-budget tuning, separate effort.
- F8 from the prior plan (oldest-first timeline toggle; per-tool `tool.*` events
for background turns). F1's status-driven `working` makes background work
visible without live per-tool deltas, so this stays lower priority.

View File

@@ -0,0 +1,191 @@
# 2026-08-04 — Chat interaction overhaul: inline progressive stream (Claude Code style)
**Status:** Planned — not started. (Refocused from the earlier feature-heavy
draft; backend features deferred — see "Deferred".)
## Goal
Streamline agent interactions — thinking, plan, tool usage, responses — into
**one linear progressive inline stream per turn** (the Claude Code / Cline /
Roo pattern), instead of the current split where the transcript shows a
collapsed trace and the real live activity lives in a separate rail timeline.
The right rail becomes **graph-only** (and auto-zooms to fit all entities).
## Locked decisions (operator interview)
| Decision | Choice |
|---|---|
| Live activity layout | **Inline stream (Claude Code)** — one progressive column per turn; rail keeps ONLY the Scope graph; Activity timeline tab removed |
| Tool-call detail | **Per-tool progressive lines** — each tool its own compact live line (spinner → one-line result summary), expandable to raw |
| Feature phases | **Defer** — edit/resubmit, @mentions, attachments are later phases; this plan is interaction-focused + graph auto-zoom |
## Diagnosis (grounded in current code)
- The transcript (`ChatThread``AgentTrace`) collapses a whole turn's tool
calls into one line ("Proposed plan" / "N tool calls"), raw-JSON detail on
expand. Not progressive; you can't see what's happening without expanding.
- The actual live plan + tool timeline lives in the **right rail**
(`TaskContextPanel``UnifiedTimeline`): newest-first backbone + tool stubs.
So "what is the agent doing" is in a **second place** — a cognitive split.
- `UnifiedTimeline` is imported **only** by `TaskContextPanel` (grep confirms),
so removing the Activity pane is self-contained.
- The `activityLog` **store** stays required: it feeds inline labels
(`toolActivityLabel`), live `run` output (`toolsWithLive`), and the mascot
(`mascot/stimuli.ts`). Only the timeline *view* is removed.
- Tool events already arrive separately (`tool_use` then `tool_result` in
`chat.ts`), and the activity log already carries humanized labels + per-tool
`stepSeq` attribution. So progressive per-tool lines + step grouping are a
**presentation** change, not a data/model change.
- `run` results are free-form text (e.g. `"run on lxc:caddy: ERROR exit status
1"`) → one-line result summaries are best-effort text parsing, no backend.
## Design
### D1 — One progressive inline stream per turn
Replace `AgentTrace` (one collapsed blob per turn) with a new
**`TurnTrace.svelte`** rendered inline for each assistant turn, top-to-bottom:
1. **Live plan checklist** (only on the most-recent/running turn — see D3).
2. **Tool lines grouped by plan step** (D2), then orphan tools (no step).
3. **Streamed text answer** (existing `markdown-body prose-chat`), with the
blinking cursor while streaming (existing).
4. A compact **"Thinking" line** while `working` and before any output: reuses
the existing `indicatorLabel` (running step → tool → "Agent is thinking…").
Fades once text/tools arrive; reappears between steps.
### D2 — Per-tool progressive lines (the Claude-Code signature)
One `ToolLine.svelte` per tool call (replaces `ToolCallCard`'s row style):
- Left: state icon — spinner while `tool_use`-only, ✓ on result, ✗ on error.
- Label: existing `toolActivityLabel(tool)` (humanized action).
- **One-line result summary** on completion — new `toolResultSummary(tool)`
in `activity.ts` (see plumbing). E.g.:
- `run` → `exit 0 · <first line>` (parse "exit status N" / "ERROR")
- `get_entity` → `host:hubris (healthy)`; `get_health_summary` → `healthy X · degraded Y · down Z`
- `list_entities`/`list_lxcs` → `N entities`; `get_relations` → `N relations`
- `search_knowledge` → `N results`; `upsert_knowledge` → `recorded document:…`
- `update_plan_step` → `step <seq> → <status>`; `propose_plan` → `N steps`
- default → first non-empty line of stringified result (≤80ch); `done` if empty
- Live `run` output: while streaming, the line auto-expands a pinned-tail mini
pane (reuse the `liveOutput` path from `toolsWithLive`).
- Click → expand raw args/result (border-driven `<pre>`, cyberspace-square).
- Border-driven, no rounded/shadow (per `border_driven_language`).
### D3 — Live plan checklist (TodoWrite-style)
On the **running/last** turn, render the current-generation `planSteps`
(already generation-aware via `workspace.ts`) as a checklist: pending = hollow,
running = spinner + highlight, done = ✓, failed = ✗, blocked = pause. Steps
check off live as `plan.step.*` events land. This is the unified timeline's
plan view, moved inline and scoped to the active turn. Past turns render only
their tool lines + text (the plan is session-level; the running turn carries
its current state, mirroring how TodoWrite re-displays state each turn). On a
terminal task state (`done`/`failed`), the checklist collapses to one line:
`Plan complete — N steps` / `Plan failed — step K`.
### D4 — Rail → graph only
`TaskContextPanel`: remove the Activity pane and the `UnifiedTimeline` import;
the panel becomes the Scope graph full-height (keep the collapsible "Scope"
header + the `nowTouching` strip). The graph is now the rail's entire job, so
auto-fit (D6) matters more. `activityLog*` stores remain imported only where
the inline stream/mascot need them.
### D5 — Cyberspace cohesion of the stream
Apply alongside the rewrite so the new inline view is on-system from day one:
- Transcript → **terminal log rows** (square, full-width, `YOU`/`NOMOS`
role-tags, hairline `divide-y` separators; no bubbles, no soft shadow).
Delete `.user-msg { box-shadow }`.
- Tool lines + expanded `<pre>`: border-driven, square, opaque.
- Composer: opaque `bg-background`, square (remove `rounded-2xl`/`bg-card/50`).
- Rewrite the stale "Art Nouveau" `<style>` comments → "cyberspace/terminal".
- Per `central_css_override`: drive surface styling centrally in `app.css`
where it's a primitive concern; no ad-hoc `rounded-*`/`shadow-*`/`backdrop-blur`.
### D6 — Graph auto-fit + drag-pan (`SessionGraph.svelte`) (carried over)
- Wrap nodes+links in `<g transform="translate(tx,ty) scale(s)">`; fit the bbox
of all nodes (radius + label + padding) into `cw`/`ch`; cap `s ∈ [0.2, 2.5]`.
- Re-fit on: mount, node-set change, container resize, sim-settle
(`alpha > 0.05`), background double-click. **Not** every tick (fights pan).
A `userPanned` flag pauses auto-follow after a manual pan until next
membership/resize/double-click.
- Background drag = pan (`tx`/`ty`); node drag converts screen→graph via the
inverse transform before setting `fx`/`fy`. Dot-grid stays in screen space.
- Keep: open-on-click, `touched` pulse, health-diff label, selection ring.
Respect `scrollIntoView` pitfall (transform, not scroll).
## Phased task list (each independently shippable; all frontend)
- **P1 — Inline progressive stream.** `TurnTrace.svelte` + `ToolLine.svelte`;
wire into `ChatThread` per turn; "Thinking" line; tool→step grouping via
activity-log `stepSeq` matched by tool id; keep `toolsWithLive` for `run`.
- **P2 — Live plan checklist.** Inline current-gen `planSteps` on the running
turn; collapse-to-summary at terminal state.
- **P3 — Rail → graph only.** Strip Activity pane + `UnifiedTimeline` from
`TaskContextPanel`; verify no other importers (grep: only TaskContextPanel).
- **P4 — Cyberspace cohesion.** Terminal log rows; remove rounded/shadow/
translucency; square composer; centralize in `app.css`; fix stale comments.
- **P5 — Graph auto-fit + drag-pan.** D6.
- **Polish (small, frontend-only):** per-message/tool **copy**; **scroll-to-
bottom** button (uses `container.scrollTo`, never `scrollIntoView`).
## Plumbing specifics (grounded, no backend)
- New `toolResultSummary(t: ToolCallResult): string` in `activity.ts`, beside
`toolActivityLabel`. Per-name switch (D2 list), graceful fallback.
- Tool→step grouping: build `id → stepSeq` from the activity log once per turn;
tools with no step render as orphans.
- Reuse: `planSteps` (generation-aware), `indicatorLabel`, `toolsWithLive`,
`toolActivityLabel`, `liveOutput` streaming path.
## Constraints honored (saved decisions)
- `design_system.central_css_override`, `border_driven_language`: square,
hairline, opaque, focus-by-color, no soft shadows/glows.
- `chat_thread.pane_layout`: dynamic status (Thinking line, live checklist)
lives in the **message Pane**, never the input Pane.
- `wmkit.scrollintoview_reflow_pitfall`: `container.scrollTo` for scroll-to-
bottom; transform (not scroll) for graph pan.
## Risks
- **Removing the rail timeline loses the "overview" view.** Mitigation: the
inline checklist + per-turn tool lines carry the same info progressively; the
graph still shows fleet scope. If operators miss the overview, a collapsed
"full timeline" can return as a toggle (follow-up).
- **Auto-fit vs manual pan** — handled by `userPanned` + settle-alpha gate.
- **Inline stream length on long turns** (1527 min, many tools) — progressive
lines can get long; mitigate by auto-collapsing finished steps (keep the
running step + its tools expanded, prior steps as one-line summaries).
- **Best-effort result summaries** may misformat unusual payloads — fallback is
always a truncated raw line + expandable raw detail, never a blank.
## Validation
- `npm run lint`, `tsc --noEmit` (no NEW errors beyond the known baseline in
`ui/*`, `oidc.ts`, `windows.ts`, `workspace.ts`), `vite build`, `vitest`
(add a `toolResultSummary` unit test per tool name + fallback).
- Manual matrix: (a) start a long task → Thinking line → plan checklist
appears and checks off live → each tool streams as its own line with a
one-line summary → text streams; (b) reload mid-turn → working still shows;
(c) `run` tool → live output pins to tail then collapses to summary; (d)
graph auto-fits at settle + on new entity + drag-pan + double-click reset;
(e) no rounded/soft-shadow remains on chat surfaces; (f) rail shows graph only.
## Deferred (later phases, after this lands + validates)
- **Edit-and-resubmit** — `truncateFrom` store method + `POST /sessions/{id}/edit`
(extract `streamTurn` from `handleChat`); reuse `reopenSession` (already
exists, store.go:838 — marks prior `session_plan_steps` `replaced`, clears
outcome) for the reset. Reject edit while the gate is busy (HTTP 409); edit
cannot queue (truncation must be atomic). Regenerate = no-op-edit case.
- **@entity mentions** — small `GET /api/v1/entities/search?q=` + composer
autocomplete inserting `type:name` slugs the agent/graph already parse.
- **Attachments** — multipart upload + `agent_attachments` table + configured
`OIKOS_ATTACHMENTS_DIR` (explicit volume, not relative) + capped text inlining.
- **Continue button** — needs `/resume` to `reopenSession` first for terminal
sessions (today `/resume` does not reopen `done`/`failed`; `handleChat`'s
follow-up path does). Small backend tweak.
- **Image vision** pending provider confirmation.
## Out of scope / follow-ups
- A collapsible "full timeline" overview toggle if the rail removal is missed.
- `read_attachment` MCP tool (lazy full-content fetch, lower context than inlining).
- Oldest-first timeline toggle / per-tool `tool.*` events for background turns.

View File

@@ -0,0 +1,145 @@
# 2026-08-04 — Hermes MCP client integration: native tool surface for oikos
**Status:** Plan.
**Context:** Hermes Agent (mac-mini workstation) now connects to oikos's MCP server
as a native MCP client (`mcp_servers.oikos` in `~/.hermes/config.yaml`). All 37+ MCP
tools are available as `mcp__oikos__*` first-class Hermes tool calls — no more raw
curl with batch-initialize SSE parsing. The integration works; this plan tightens the
remaining seams.
**Trigger:** First-use retrospective identified three areas that make the integration
harder to use than it should be.
---
## 1. Motivation
The oikos MCP server (`internal/mcp/`) speaks Streamable HTTP at
`https://mcp.hubris.network/mcp`. Hermes Agent's native MCP client connects to it on
startup, discovers tools, and registers them as callable functions. This replaces the
previous pattern where agents fired raw curl requests with batch `initialize` +
`tools/call` envelopes.
Three friction points observed:
- **No lightweight connectivity check.** The `/healthz` HTTP endpoint exists but isn't
exposed at the MCP protocol layer. An agent that wants to verify the MCP server is
reachable must call a real tool (e.g. `list_entities` with a limit of 1) — every call
carries the Streamable HTTP session-initialization overhead.
- **Bearer token in plaintext.** `~/.hermes/config.yaml` stores the token directly in the
`mcp_servers.oikos.headers.Authorization` value. Hermes does not support env-var
interpolation in MCP server configs, so the token can't live only in `.env`.
- **Zero-visibility streaming overhead.** Streamable HTTP batches `initialize` +
`tools/call` per request. This adds ~2KB of transport per tool call that the agent
never sees. For a single `get_health_summary` call this is negligible; for a 10-tool
exploration pass it's 20KB of invisible overhead.
---
## 2. Changes
### I — MCP health/ping tool (`mcp__oikos__ping`)
**Why:** Agents need a zero-cost connectivity check before calling production tools.
Currently every check incurs the full Streamable HTTP initialize + tools/call round-trip.
**What:**
Add a `ping` tool that returns `{"ok": true, "server": "oikos", "version": "dev"}`.
No arguments. No DB hit. No auth check (already protected by the MCP transport's auth
layer — the request won't arrive if the bearer token is missing).
```go
// internal/mcp/tools.go
{
Name: "ping",
Description: "Lightweight connectivity check. Returns immediately with server identity, no DB hit.",
InputSchema: jsonschema.Must(nil), // no params
Handler: func(ctx context.Context, args json.RawMessage, caller CallerInfo) (json.RawMessage, error) {
return json.RawMessage(`{"ok":true,"server":"oikos","version":"` + version.Version + `"}`), nil
},
}
```
**Risk class:** read-only. No auth, no DB, no state. Auto-approves.
**Test:** `hermes mcp test oikos` (from the Hermes CLI) verifies MCP server reachability
independently; the `ping` tool gives agent code the same signal programmatically.
### II — Tool name documentation in server metadata
**Why:** Hermes prefixes MCP tools as `mcp_{server}_{tool}`, so `get_health_summary`
becomes `mcp__oikos__get_health_summary`. Agents discover tool names at runtime via
`tools/list`, but there's no short summary of what each tool group does that survives
into the MCP tool description.
**What:**
Audit and tighten every tool's `Description` field in `internal/mcp/tools.go` so the
first 812 words are a searchable one-liner an agent can pattern-match against.
Current descriptions that are vague or redundant get a prefix rewrite:
| Tool | Current description | Revised |
|------|-------------------|---------|
| `get_entity` | "Get entity metadata" | "Look up one entity by slug or UUID — type, state, attributes, health" |
| `list_entities` | "List entities" | "Browse entities by type, state, or name substring — paginated" |
| `upsert_knowledge` | "Record what you learned" | "Write a document/investigation/runbook to the knowledge graph — idempotent" |
| `run` | "Run ANY shell command" | "Execute a shell command on any host/LXC/VM — auto-classified by risk" |
Existing tools pass through unchanged if their description is already crisp. ~15 tools
get description rewrites.
**Risk class:** read-only (config change). No runtime effect.
### III — Env-var interpolation docs for Hermes config (oikos-side documentation)
**Why:** The bearer token lives in `~/.hermes/config.yaml` in plaintext because Hermes
does not support `${VAR}` interpolation in MCP server configs. This is a Hermes
upstream feature request, not an oikos change — but oikos should document the
workaround and track the upstream ask.
**What:**
Add a `### Hermes MCP client` subsection to `docs/infrastructure/mcp-server.md` (or
create it if it doesn't exist) that covers:
1. The config block to add to `~/.hermes/config.yaml` (already done — record it
for the next person).
2. The token exposure caveat: Hermes doesn't support env-var interpolation in
`mcp_servers` `headers` yet (upstream issue nousresearch/hermes-agent#TODO — file
once).
3. Workaround: `hermes config set security.redact_secrets true` (already default) so
the token value is stripped from tool output and logs even if it appears in
diagnostic text.
4. How to verify the connection: `hermes mcp list``hermes mcp test oikos`.
**Risk class:** docs-only.
---
## 3. Open questions
| Question | Decision |
|----------|----------|
| Should `ping` bypass auth entirely or still require a valid bearer token? | **Still requires auth.** The MCP transport layer validates the token before routing to `ping` — no special treatment needed. If the token is missing, the request never reaches the handler. |
| Who files the Hermes upstream feature request for `${VAR}` interpolation? | **Oikos operator** (dtoro). The need is specific to this deployment. File at https://github.com/NousResearch/hermes-agent/issues. |
---
## 4. Not doing (yet)
- **Persistent MCP sessions** — Streamable HTTP stateless mode is fine for the
current tool-call volume (~15 calls per agent turn). Persistent sessions would
save ~2KB per call but add connection lifecycle complexity. Revisit if per-turn
tool calls exceed 20.
- **`tools/list` caching** — Hermes already caches tool discovery at session start.
The 37-tool list is ~4KB; caching adds complexity for negligible savings.
---
## 5. Verification
1. `curl -X POST https://mcp.hubris.network/mcp ... -d '...ping...'` returns `{"ok":true,"server":"oikos","version":"dev"}`
2. `hermes mcp list` shows `ping` among oikos tools
3. `hermes doctor` passes
4. Tool descriptions are crisp: `hermes mcp list` output for oikos shows prefixed summaries

View File

@@ -0,0 +1,369 @@
# 2026-08-04 — Session audit: agent reliability, plan system, and learning loop gaps
**Status:** Done — all 12 tasks implemented, tested, deployed in v0.21.0. Verified in production
with live session tests (classifications, feedback, token tracking, execution linkage all confirmed).
**Reviewed sessions:** Past 5 completed plus ZimaOS continuation (268895a5)
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
`agent_activity`/`session_plan_steps`/`executions`/`classifications`/`feedback`/
`patterns`/`skills`/`approvals`/`nomos_plan_executions`/`audit_log` on the prod
mac-mini. Cross-referenced with `internal/audit/`, `internal/mcp/`,
`internal/httpapi/`, `internal/policy/`.
---
## 1. Sessions audited
| # | Session | Turns | Run calls | Failures | Plan steps (done/total) | Duration | Outcome |
|---|---------|-------|-----------|----------|-------------------------|----------|---------|
| S1 | Pocket-pascal deploy (a433b386) | 8 | 111 | 1 | 10/11 | 1.5h | Success (truncated by turn limit) |
| S2 | SSH re-investigation (6f0ade08) | 4 | 42 | 0 | 0/4 | 11m | Success (all steps replaced) |
| S3 | Webhook HMAC (0f509508) | 2 | 40 | 0 | 5/5 | 2m | Success (clean; best session) |
| S4 | Check scripts (d458a5f8) | 23 | 96 | 4 | 0/14 | 2h | Success (3 plan gens; 0 steps done) |
| S5 | ZimaOS outage (268895a5) | 18+10 | 209 | 8 | 0/5 | 2h30m | **Marked success; dashboard still broken** |
**Headline:** 85% session success rate, but plan adherence is ~38% (15/39 steps
ever reached `done`). The ZimaOS session was the worst: marked `success` while
the dashboard was still down, then burned 81 more calls and 30 minutes chasing
irrelevant DHCP reservations.
---
## 2. Cross-cutting findings
### F1 — MCP 30s client timeout kills every long-running command (P0)
All **9** `run` failures across the audit are `Post "http://api:8090/mcp":
context deadline exceeded` at exactly 30s. Commands with `sleep 30`, `qm
shutdown`+wait, or async poll loops always hit this. The agent retries with
longer sleeps and hits the same wall.
**Root cause:** `cmd/nomos` MCP client uses a 30s timeout; the `run` tool
blocks synchronously waiting for command completion. No async path exists for
long-running commands.
### F2 — Premature success: `complete_task` fires before the goal is actually met (P0)
Session 268895a5 called `complete_task(success)` at 18:29 with summary "zimaos
returns 200." The endpoint was serving ttyd (terminal), not the ZimaOS
dashboard. The agent conflated "HTTPS 200" with "dashboard works." The user had
to resume the session.
**Root cause:** No pre-completion validation. The agent can mark success with a
summary that doesn't match reality. `complete_task` is a write-and-forget
operation with no state check.
### F3 — Wrong target: `run` doesn't validate the target can execute the command (P0)
In the ZimaOS continuation, `qm stop 100 --skiplock` was executed on
`lxc:dns`. The command failed (`qm: command not found`) because the agent
copied the command from a previous run but forgot to change the target.
Similarly, `cmd/nomos` tried `docker exec` on `host:hubris` (no docker).
**Root cause:** `run(target, command)` in `internal/mcp/server.go` doesn't
validate that the target type can execute the given command. A simple
allowlist would catch `qm`/`pct`/`pvesh` on non-host targets.
### F4 — Plan system is decorative: 62% of steps never reach `done` (P1)
Across 5 sessions: 39 plan steps. 24 (62%) were `replaced`, 15 (38%) reached
`done`. Session d458a5f8 had 3 complete plan regenerations with **zero**
completed steps. The agent replaces plans instead of completing or explicitly
skipping steps.
**Root cause:** Plan steps carry status (`pending`/`running`/`done`/`failed`/
`skipped`/`blocked`/`replaced`) but `replaced` has no `replaced_reason`
field. The model can silently replace every step and the system doesn't flag
it. No constraint ties `propose_plan` to existing plan state.
### F5 — No plan on session resume: continuation sessions run ad-hoc (P1)
The ZimaOS continuation (18:29→19:01) had 81 calls with **zero**
`propose_plan` calls. The agent ran ad-hoc tool calls with no structure.
**Root cause:** When a session resumes, the `must_have_plan` guard is already
satisfied by the old (completed) plan. The agent doesn't re-plan on resume.
### F6 — Scope expansion / rabbit holes: agent chases irrelevant sub-goals (P2)
In the ZimaOS continuation, the agent spent ~40 calls trying to fix Technitium
DHCP reservations — a completely different subsystem from the goal ("make the
dashboard reachable"). The ZimaOS dashboard hadn't started since July 19
(3-week-old issue), making the DHCP reservation effort moot. The agent never
surfaced a question like "This is pre-existing — should I still fix DHCP?"
**Root cause:** No scope gate. When the agent pivots to a subsystem unrelated
to the stated goal, nothing stops it. The `session_questions` mechanism exists
(2 calls in 193 sessions) but the model never uses it.
### F7 — Command generation errors: malformed bash from the LLM (P2)
In the ZimaOS continuation, the agent generated:
- `head - n` instead of `head -n` → bash syntax error
- `echo "---" && \n curl ...` → literal `\n` in command → ambiguous redirect
**Root cause:** The LLM generates bash commands inline in content blocks. No
syntax validation, no escape-character handling. The `run` tool should reject
malformed commands before execution.
### F8 — Learning pipeline completely empty (P4)
| Table | Rows |
|-------|------|
| `classifications` | **0** |
| `feedback` | **0** |
| `patterns` | **0** |
| `skills` | **0** |
Despite 1,884 executions and 263 approvals, the system learns nothing from
outcomes. The ZimaOS session discovered that the dashboard hadn't started since
July 19 — this was never persisted. The HMAC trailing-newline discovery was
persisted manually; if the agent forgot `upsert_knowledge`, it would be lost.
### F9 — Observability gaps (P3)
| Gap | Detail |
|-----|--------|
| Token tracking | `agent_activity.token_count` is NULL for every row |
| Execution linkage | `nomos_plan_executions` is empty; `audit_log.session_id` is null |
| Plan quality | No metric for step completion rate (currently 38%) |
| MCP timeout rate | No counter for `run` calls that hit the client timeout |
### F10 — Execution success rate is 74% (P2)
1,884 executions: 1,400 completed (74%), 295 failed (15.6%), 152 cancelled
(8%), 37 denied (2%). One in four execution attempts doesn't complete.
---
## 3. Improvement plan (ordered by impact/effort)
### Task 1 — Prevent premature `complete_task(success)` *(fixes F2)*
**`internal/mcp/tools.go`** — In the `complete_task` handler, when
`outcome=success`: require the summary field to contain a verifiable state
assertion. Minimum: if the goal mentions a URL, check that the summary doesn't
contradict known state. Lightweight: log a warning if the summary says "returns
200" but the last `ping_service` or `run` result says otherwise.
**Coach:** `nomos/SOUL.md` — explicit rule: *"Before calling
complete_task(success), restate the user's original goal in your own words and
verify each condition. If any condition is 'probably works' rather than
'verified,' ask the operator or set outcome=partial."*
### Task 2 — Target validation in `run` *(fixes F3)*
**`internal/mcp/server.go`** — In the `run` handler, before dispatching:
validate that the command prefix matches the target type.
```
pct/qm/pvesh/iptables → only host:* targets
systemctl/docker → host:* or lxc:* targets
curl/nmap/ss → any target
```
If mismatched, return a clear error: *"Cannot run `qm` on lxc:dns — `qm` is a
Proxmox host command. Use target host:hubris or host:strong."* Do not classify
or execute.
**Test:** `TestClassifyCommand_WrongTarget` → commands with host-only prefixes
on LXC targets return error without execution.
### Task 3 — Raise MCP client timeout; add async path for long-running commands *(fixes F1)*
**`cmd/nomos`** — Raise the MCP client timeout from 30s to 120s.
**`internal/mcp/server.go`** — For `run` commands that the classifier
determines will exceed the client timeout (presence of `sleep`, `wait`,
`timeout` in the command), return immediately with an `execution_id` and status
`running`. The agent already has `get_execution_status` — use it:
1. Classify the command; if it contains `sleep`, `wait`, or shell constructs
that imply polling, flag it as `async_potential`.
2. Start the command, return the `execution_id` immediately.
3. Agent polls with `get_execution_status(execution_id)`.
4. If the client timeout is hit mid-poll, the execution continues on the server
— it's not lost.
**Test:** `run` with `sleep 60; echo done` on host:hubris → returns
immediately (not 30s timeout), `get_execution_status` eventually returns
`completed`.
### Task 4 — Plan step integrity: require `replaced_reason` on replacement *(fixes F4)*
**`session_plan_steps` migration** — Add `replaced_reason TEXT` column.
**`cmd/nomos`** — When the agent emits `update_plan_step` with
`status=replaced`, require a non-empty `replaced_reason`. Valid reasons:
`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`.
**Coach:** `nomos/SOUL.md` — explicit rule: *"Complete (status=done) or
explicitly skip (status=skipped) steps. Use status=replaced only when the
entire plan generation is wrong; include the reason. Replacing all steps with
no reason is a session-quality violation."*
### Task 5 — Force `propose_plan` on session resume *(fixes F5)*
**`cmd/nomos`** — When a session with status `done` or `failed` receives a new
user message, reset the plan state: clear step status, require a new
`propose_plan` call before any `run` calls. The "must have plan" guard should
consider the resumed session as plan-less until a fresh `propose_plan` is
called.
**Guard:** `set_goal` + `propose_plan` must be called before any `run` in a
resumed session. Reuse the existing "No plan — call set_goal then propose_plan"
error from `internal/mcp/server.go`.
### Task 6 — Scope gate: surface `session_questions` on context switch *(fixes F6)*
**Coach:** `nomos/SOUL.md` — explicit rule: *"Before pivoting to a subsystem
not mentioned in the user's goal, ask via session_questions. Example: 'The
dashboard logs show it hasn't started since July 19. Do you want me to debug
the dashboard service itself [A], skip it and just stabilize the IP [B], or
stop here [C]?'"*
**`cmd/nomos` prompt** — Add to the system prompt: *"When the investigation
leads to a subsystem or root cause unrelated to the expressed goal, surface a
session_question before taking action."*
### Task 7 — Auto-upsert knowledge on session close *(fixes F8)*
**`cmd/nomos`** — On `complete_task` (any outcome: success, partial, failure),
auto-generate a knowledge entry:
```
title: "<date>: <goal summary>"
content: "## Outcome\n<outcome>\n## Root cause\n<extracted>\n## What was done\n<summary>\n## What was left\n<unresolved>"
tags: [session:<id>]
about: [entities involved]
```
This ensures every session leaves a trace regardless of whether the agent
remembered to call `upsert_knowledge`.
### Task 8 — Token tracking *(fixes F9)*
**`cmd/nomos`** — After each LLM call, extract `usage.prompt_tokens`,
`usage.completion_tokens`, `usage.total_tokens` from the response and write to
`agent_activity.token_count`. Currently the field exists but is never populated
(NULL for all rows).
### Task 9 — Execution linkage *(fixes F9)*
**`internal/mcp/server.go`** — When `run` creates an execution, write a row
into `nomos_plan_executions` linking `session_id`, `plan_step_seq`, and
`execution_id`.
**`internal/mcp/server.go`** — Pass `session_id` (from MCP request headers)
into `audit_log` writes. Currently `audit_log.session_id` is NULL — the
`createAuditLog` function in `internal/httpapi/impl.go` receives the
correlation_id but not the session_id from the MCP path.
### Task 10 — Plan quality metric *(fixes F9)*
**`cmd/nomos`** — At session close, compute: `completed_steps /
total_steps_per_plan` (currently ~38%). Log as a metric or write as a session
attribute. Track over time to measure plan-adherence improvements from Tasks
4+5.
### Task 11 — Auto-classify every `run` call *(fixes F8)*
**`internal/mcp/server.go`** — The `run` handler already calls the classifier
(`classifyCommand` in `internal/policy/command.go`) to determine risk_class and
approval route. Write the result to the `classifications` table. Currently the
table is empty (0 rows) despite 1,884 executions being classified.
### Task 12 — Auto-feedback on session close *(fixes F8)*
**`cmd/nomos`** — On `complete_task`, generate a `feedback` entry:
```
session_id: <id>
outcome: <outcome>
observation: <summary>
lesson: <extracted from complete_task.summary>
side_effects: <entities created/modified during session>
```
**`cmd/oikos`** — Add a daily cron or scheduler job that reads recent
`feedback` entries and extracts `patterns` (recurring root causes, same-fix
applied multiple times, known-broken services). Seed the pattern table.
### Task 13 — Command syntax validation in `run` *(fixes F7)*
**`internal/mcp/server.go`** — Before executing a `run` command, do
lightweight bash syntax validation:
```
- Reject literal \n in commands (should be ; or &&)
- Reject commands where the last line ends with \ (backslash-continuation)
but no next line
- Warn on common typos: "head - n", "grep - i", spaces before flags
- Reject `&& \n` patterns (the LLM sometimes inserts literal \n between && chains)
```
### Task 14 — Stuck-session reaping (from prior plan; re-confirmed)
This session exhibited the same idle zombie pattern (23da10db — no closed_at,
status `failed` but outcome `failure`). Task 5 from the 2026-08-03 plan is
still open. Copying here for completeness.
---
## 4. Recommended sequence
```
P0 (blocks operational waste):
1 → 2 → 3
P1 (fixes plan architecture):
4 → 5
P2 (cognitive guardrails):
6 → 7 → 13
P3 (observability):
8 → 9 → 10
P4 (learning loop):
11 → 12
```
Sequence rationale: Tasks 1-3 stop the worst outcomes (premature success,
wrong-target execution, MCP timeouts). Tasks 4-5 make the plan system actually
useful instead of decorative. Tasks 6-7 add guardrails that prevent the ZimaOS
rabbit-hole class of failure. Tasks 8-10 give us visibility into whether any of
the previous tasks are working. Tasks 11-12 close the learning loop.
---
## 5. Validation
| Task | Test |
|------|------|
| 1 | Session with goal "make X reachable" where last ping shows 502 → `complete_task(success)` is rejected or warns |
| 2 | `run("lxc:dns", "qm stop 100")` → error: "qm is a Proxmox host command" |
| 3 | `run` with `sleep 45; echo done` → returns execution_id immediately, `get_execution_status` shows final result |
| 4 | `update_plan_step(status=replaced)` with no reason → rejected; with reason → accepted |
| 5 | Resumed session calls `run` before `propose_plan` → blocked: "No plan — call propose_plan" |
| 6 | Agent pivots to unrelated subsystem → `session_questions` is called before action |
| 7 | `complete_task` → knowledge entry created automatically with session link |
| 8 | `agent_activity.token_count` is non-NULL after any LLM call |
| 9 | `nomos_plan_executions` has rows linking session + step + execution |
| 10 | Session close writes `plan_adherence` attribute (step-completion %) |
| 11 | `classifications` table has 1 row per `run` call with risk_class + route |
| 12 | `complete_task` → auto `feedback` entry; daily pattern job finds recurring issues |
| 13 | `run` with `head - n /etc/hosts` → rejected with clear error about malformed command |
---
## 6. Out of scope / open questions
- Whether to raise `auto_act` from `off` for `reversible_low` actions (separate
policy decision; would reduce approval pileup without code changes).
- Whether to add a `delete_entity` MCP tool for lifecycle management (separate
from this reliability plan).
- The exact TTL for stuck-session reaping (30 min recommended, confirmed in
2026-08-03 plan).
- Whether `run` async mode should be opt-in (command contains sleep/wait) or
universal (every run returns immediately, agent always polls). Recommend
opt-in for now — most commands complete in <5s.

View File

@@ -0,0 +1,304 @@
# 2026-08-04 — Unified plan: external MCP agents + Nomos reliability
**Status:** Complete. All 5 stages implemented (2026-08-04).
---
## 1. Context
Two audiences, two gaps:
| Audience | Current state | Goal |
|---|---|---|
| **Nomos** (internal agent) | 85% session success, but 38% plan adherence, 9 timeout failures, premature success, empty learning table | Reliable, self-correcting, leaves a trace |
| **External agents** (Claude, Goose, etc.) | 40 MCP tools — heavy on observe, light on act. Can't manage signals, checks, executions, or knowledge beyond `upsert`. | Full Oikos surface: observe + act + curate |
The two plans share infrastructure (`internal/mcp/server.go`) and have cross-task
dependencies. This document merges them into one sequenced plan.
---
## 2. Cross-plan dependencies
Three audit tasks are **prerequisites** for external-agent mutation tools:
| Audit task | Enables | Why |
|---|---|---|
| **Task 3** — async `run` + timeout | `cancel_execution`, `list_executions` with live status | Without async `run`, every command >30s hits a client timeout. The agent loses track of the execution and can't cancel it. |
| **Task 9** — execution linkage | `list_executions` filtered by session/entity | `nomos_plan_executions` table is empty; `audit_log.session_id` is NULL. Without linkage, execution queries are blind. |
| **Task 2** — target validation in `run` | All Phase 2 mutation tools | Can't let external agents mutate state on targets that can't execute the command (audit found `qm` run on `lxc:dns`). |
Two items were **dropped** from the original MCP expansion:
- **`approve_execution` / `deny_execution`** — separation-of-duties violation. An
MCP agent approving its own queued commands breaks the approval model. The
correct fix is raising `auto_act` for `reversible_low` (policy change, zero
code — listed in the audit plan's out-of-scope). The existing assent-window
path in `run()` already auto-approves reversible-low commands.
- **Execution log streaming** — MCP has no push model. `get_execution_status`
returns the latest output; full streaming stays WebUI-only.
---
## 3. Unified sequence (5 stages, 31 tasks)
### Stage 1: Foundation — shared infrastructure (3 tasks)
These unlock everything downstream. Do first.
**T1 — Target validation in `run`** *(audit Task 2)*
`internal/mcp/server.go`: before dispatching a `run` command, validate prefix
against target type. `qm`/`pct`/`pvesh``host:*` only. `systemctl`/`docker`
`host:*` or `lxc:*`. Mismatch returns error without execution.
**T2 — Async `run` + 120s timeout** *(audit Task 3)*
`cmd/nomos`: raise MCP client timeout 30s → 120s.
`internal/mcp/server.go`: when classifier detects `sleep`/`wait`/poll loops in
the command, start execution and return `execution_id` immediately. Agent polls
with `get_execution_status`. Execution continues server-side even if client
timeout.
**T3 — Execution linkage** *(audit Task 9)*
`internal/mcp/server.go`: when `run` creates an execution, write into
`nomos_plan_executions` (session_id + plan_step_seq + execution_id). Pass
`session_id` from MCP request headers into `audit_log` writes.
---
### Stage 2: External agent observe (8 tasks)
All read-only MCP tools. Zero risk, ship fast. Unblocks external agents from
understanding system state.
**T4 — `get_dashboard_summary`**
Fleet health counts, signals by severity, pending approvals, event rate. One
call instead of 4. Wraps existing `GetDashboardSummary` DB query.
**T5 — `get_ontology`**
Entity types, relationship types, lifecycle states, monitoring specs. Agents
need this to reason about the schema.
**T6 — `list_checks`**
Per-entity health checks with verdict, last run, probe output. Filter by entity
slug or check state.
**T7 — `list_executions`**
Cursor-paginated execution history. Filter by entity slug, status, risk class.
Depends on T3 (execution linkage) for session/entity filtering.
**T8 — `get_knowledge_revisions`**
Version history for a knowledge entity. Agent can see what changed and when.
**T9 — `get_knowledge_duplicates`**
Near-duplicate knowledge entries via trigram clustering. Wraps existing
`knowledgeDuplicates` query.
**T10 — `get_knowledge_orphans`**
Knowledge entries not linked to any entity. Agent can suggest cleanup.
**T11 — `list_knowledge_tags`**
All tags with counts. Agent can see the taxonomy.
**T12 — `list_entity_sessions`** *(Phase 3 polish)*
Active Nomos sessions linked to an entity. Depends on T3 (execution linkage).
**T13 — `find_entities_by`** *(Phase 3 polish)*
Search entities by attribute (IP, port, version, tag). More flexible than
`list_entities` (type/state only).
**T14 — MCP resources**
Expose entities, knowledge entries, and executions as MCP resource templates:
`oikos://entity/{slug}`, `oikos://knowledge/{id}`, `oikos://execution/{id}`.
MCP clients that support resources (Claude Desktop, Goose) can browse and attach
them to conversations.
---
### Stage 3: Nomos reliability (6 tasks)
Fixes the worst failure modes found in the 5-session audit.
**T15 — Prevent premature `complete_task(success)`** *(audit Task 1)*
`internal/mcp/tools.go`: when `complete_task` with `outcome=success`, verify the
summary doesn't contradict known state. If goal mentions a URL but the last
probe shows non-200, log a warning.
`nomos/SOUL.md`: explicit rule — restate goal, verify every condition before
calling success. If any condition is "probably works," use `outcome=partial`.
**T16 — Plan step integrity: require `replaced_reason`** *(audit Task 4)*
Schema: add `replaced_reason TEXT` to `session_plan_steps`.
`cmd/nomos`: when agent emits `update_plan_step(status=replaced)`, require
non-empty reason (enum: `wrong_diagnosis`, `scope_change`, `blocked`,
`superseded`, `operator_override`).
`nomos/SOUL.md`: explicit rule — complete or skip steps. Replacing all steps
with no reason is a session-quality violation.
**T17 — Force `propose_plan` on session resume** *(audit Task 5)*
`cmd/nomos`: when a session with status `done`/`failed` receives a new user
message, reset plan state. Require fresh `propose_plan` before any `run`.
The "must have plan" guard treats resumed sessions as plan-less.
**T18 — Scope gate: surface `session_questions` on context switch** *(audit Task 6)*
`cmd/nomos` system prompt: "When investigation leads to a subsystem unrelated to
the expressed goal, call `session_questions` before taking action."
`nomos/SOUL.md`: explicit rule — ask before pivoting.
**T19 — Command syntax validation in `run`** *(audit Task 13)*
`internal/mcp/server.go`: before executing, reject literal `\n` in commands,
backslash-continuation on last line, `head - n`/`grep - i` space-before-flag
typos, and `&& \n` patterns from LLM formatting errors.
**T20 — Stuck-session reaping** *(audit Task 14, from 2026-08-03 plan)*
Reap sessions with `closed_at IS NULL` and no message in 30 minutes. Set
status=failed, outcome=failure.
---
### Stage 4: External agent act (9 tasks)
Mutation MCP tools. Each writes audit log + emits event. Requires Stage 1
infrastructure (T1 target validation, T2 async run, T3 execution linkage).
Follows existing direct-DB patterns — no HTTP API calls.
**T21 — `ack_signal(signal_id)`**
Acknowledge an open signal. Agent investigating an alert marks it acknowledged.
**T22 — `resolve_signal(signal_id, resolution?)`**
Resolve a signal with optional resolution note.
**T23 — `mute_signal(signal_id, duration?)`**
Temporarily mute a signal. Optional duration (default 1h).
**T24 — `cancel_execution(execution_id, reason)`**
Cancel a queued/running execution. Depends on T2 (async `run` returns
`execution_id`) and T3 (execution linkage for audit context).
**T25 — `update_check(check_id, enabled)`**
Enable/disable a health check. Agent suppresses a noisy probe.
**T26 — `delete_knowledge(knowledge_id)`**
Soft-delete a knowledge entry (move to trash, restorable).
**T27 — `restore_knowledge(knowledge_id)`**
Restore a trashed knowledge entry.
**T28 — `merge_knowledge(source_id, target_id)`**
Fold one knowledge entry into another. Source gets soft-deleted, content
appended to target.
**T29 — `rename_knowledge_tag(old_name, new_name)`**
Bulk-rename a tag across all knowledge entries.
---
### Stage 5: Close the learning loop (5 tasks)
Turn execution data into persistent knowledge. Currently all learning tables are
empty (0 classifications, 0 feedback, 0 patterns, 0 skills).
**T30 — Auto-classify every `run` → `classifications` table** *(audit Task 11)*
`internal/mcp/server.go`: `run` already calls `classifyCommand`. Write the
result to the `classifications` table (risk_class + route + patterns matched).
Currently 0 rows despite 1,884 executions.
**T31 — Auto-upsert knowledge on session close** *(audit Task 7)*
`cmd/nomos`: on `complete_task` (any outcome), auto-generate a knowledge entry:
title=`<date>: <goal>`, content with Outcome/Root cause/What was done/Unresolved
sections, tags=`[session:<id>]`, linked to involved entities.
**T32 — Auto-feedback on session close** *(audit Task 12)*
`cmd/nomos`: on `complete_task`, generate a `feedback` entry: session_id,
outcome, observation, lesson, side_effects. Daily cron job reads recent feedback
and extracts patterns (recurring root causes, same-fix-applied-multiple-times).
**T33 — Token tracking** *(audit Task 8)*
`cmd/nomos`: after each LLM call, extract `usage.total_tokens` from the response
and write to `agent_activity.token_count`. Currently NULL for all rows.
**T34 — Plan quality metric** *(audit Task 10)*
`cmd/nomos`: at session close, compute `completed_steps / total_steps` (currently
~38%). Write as session attribute. Track over time to measure impact of T16+T17.
---
## 4. Validation
| Task | Test |
|---|---|
| T1 | `run("lxc:dns", "qm stop 100")` → error: "qm is a Proxmox host command" |
| T2 | `run` with `sleep 45; echo done` → returns `execution_id` immediately; `get_execution_status` eventually shows completed |
| T3 | After `run`, `nomos_plan_executions` has row linking session + step + execution |
| T4 | `get_dashboard_summary()` returns health counts, signal counts, approval count in one call |
| T5 | `get_ontology()` returns entity_types, relationship_types, lifecycle_states |
| T6 | `list_checks(entity_slug="lxc:jellyfin")` returns all checks with verdict + last run |
| T7 | `list_executions(entity_slug="host:hubris", limit=10)` returns cursor-paginated list |
| T8 | `get_knowledge_revisions(id)` returns ordered revision list with timestamps |
| T9 | `get_knowledge_duplicates()` returns clusters with similarity scores |
| T10 | `get_knowledge_orphans()` returns knowledge entries with zero entity links |
| T11 | `list_knowledge_tags()` returns {name, count} for all tags |
| T12 | `list_entity_sessions("lxc:jellyfin")` returns active sessions with goal + status |
| T13 | `find_entities_by(ip="10.0.0.5")` returns matching entities |
| T14 | MCP client can browse `oikos://entity/*` resources |
| T15 | Session with goal "make X reachable" where last ping shows 502 → `complete_task(success)` warns or rejects |
| T16 | `update_plan_step(status=replaced)` with no reason → rejected |
| T17 | Resumed session calls `run` before `propose_plan` → blocked |
| T18 | Agent pivots to unrelated subsystem → `session_questions` is called |
| T19 | `run` with `head - n /etc/hosts` → rejected with syntax error |
| T20 | Session idle for 30+ min with no `closed_at` → reaped (status=failed) |
| T21 | `ack_signal(id)` → signal status transitions to acknowledged, audit logged |
| T22 | `resolve_signal(id, "fixed DNS")` → resolved with note |
| T23 | `mute_signal(id, 3600)` → muted for 1 hour, auto-unmutes |
| T24 | `cancel_execution(id, "wrong target")` → execution cancelled, audit logged |
| T25 | `update_check(id, false)` → check disabled, scheduler stops probing |
| T26 | `delete_knowledge(id)` → soft-deleted (trashed), restorable |
| T27 | `restore_knowledge(id)` → restored from trash, reappears in list |
| T28 | `merge_knowledge(src, dst)` → src deleted, content appended to dst |
| T29 | `rename_knowledge_tag("old", "new")` → all entries updated |
| T30 | After any `run`, `classifications` has row with risk_class + route |
| T31 | `complete_task` → knowledge entry created automatically with session link |
| T32 | `complete_task` → feedback entry created; daily job extracts pattern if same root cause appears ≥3 times |
| T33 | `agent_activity.token_count` is non-NULL after LLM call |
| T34 | Session close writes `plan_adherence` attribute (% steps completed) |
---
## 5. Files touched
| File | Tasks |
|---|---|
| `internal/mcp/server.go` | T1, T2, T3, T19, T24, T30 |
| `internal/mcp/tools.go` | T4T14, T21T29 |
| `internal/mcp/discover.go` | (no changes — references for T10/T11 patterns) |
| `cmd/nomos/main.go` (or config) | T2 (timeout), T16, T17, T18, T31, T32, T33, T34 |
| `nomos/SOUL.md` | T15, T16, T18 |
| `internal/httpapi/` | T3 (audit_log.session_id plumbing) |
| DB migrations | T16 (replaced_reason column) |
---
## 6. What stays WebUI-only
| Feature | Reason |
|---|---|
| FleetMap visual graph | Canvas rendering — not an MCP concern |
| uPlot metric charts | Raw data available via `query_metrics`/`get_trend` |
| Desktop shell, Cluck, App Store | Pure UI layer |
| SSE event streaming | MCP has no push model; polling covers it |
| Execution log streaming | `get_execution_status` returns latest output |
| Nomos session chat | MCP is a tool interface, not a chat agent |
| Knowledge wiki editor (revision browse, cleanup UI) | MCP tools expose the data + mutations; UI provides the editing experience |
| Approval queue with Approve/Deny buttons | Approvals stay operator-gated via WebUI/Matrix |
| Client enrollment flow | Enrollment is IP-gated, not an MCP tool |
---
## 7. Out of scope
- **`approve_execution` / `deny_execution` MCP tools** — dropped. The fix is raising
`auto_act` for `reversible_low` (policy change, no code).
- **`delete_entity` MCP tool** — separate lifecycle management concern.
- **Per-client bearer tokens** — open item tracked in CLIENTS.md. Until they
exist, external agents share the same `OIKOS_MCP_BEARER_TOKEN`.
- **Exactly-once pattern extraction from feedback** — T32 seeds the pipeline;
the full pattern-mining algorithm (TF-IDF clustering, causal inference from
event timelines) is future work.

View File

@@ -0,0 +1,158 @@
# 2026-08-05 — Agent execution safety: QEMU guest agent guardrails + host-mutation gate
**Status:** Plan.
**Context:** ZimaOS NFS recovery session (2026-08-04/05) surfaced three systemic
failures in how agents drive oikos mutations. The `run` tool queued an execution
against a VM whose QEMU guest agent was down — it sat `pending_approval` forever,
never executed, and the agent silently fell back to raw SSH. That same raw-SSH
fallback was then used to `apt-get install nfs-kernel-server` directly on the
hubris **PVE host**, crashing it and taking the whole homelab subnet down for
~15 minutes.
**Trigger:** Incident `investigation:nomos/incident-hubris-crash-from-nfs-kernel-server-on-pve-host-2026-08-05`
(2026-08-05) + session audit. The crash was caused by an agent bypassing the
`run` approval gate, which exists precisely to catch that kind of mistake.
---
## 1. Motivation
The OODA loop's Act phase is the security boundary (ADR 0012: "Hermes has no
direct SSH access... all mutations go through the execution queue"). This session
proved the boundary has three leaks:
1. **`run` on a VM with a dead QEMU guest agent queues silently.** The execution
is classified `config_mutation`, queued for approval, and *never fails* — it
just sits in `pending_approval` while the agent assumes progress. There is no
feedback that the underlying transport (`qm guest exec`) cannot work.
2. **No guardrail against host-level package/kernel mutations.** `apt-get install`
targeting a `proxmox-host` entity is classified `config_mutation` and gated —
*if* the agent routes it through `run`. When the first `run` call stalls
(leak #1), the agent falls back to raw SSH, which has no classification at all.
The crash was the direct result of that fallback.
3. **Agents are trusted to self-report the `health` attribute.** `update_entity_attributes`
let the agent set `health:"healthy"` on `lxc:nfs-export`, which derived 4
spurious health checks. Health is scheduler-owned; agents shouldn't write it.
---
## 2. Changes
### I — `run` pre-flights the execution transport before queueing
**Why:** A queued execution that can never run is worse than a failed one — it
looks like progress, stalls the agent, and (this session) pushed the agent into
the unsafe raw-SSH fallback.
**What:**
In the `run` tool handler (`internal/mcp/`), before inserting the execution row:
- If target type is `vm`, read the target entity's attributes. If
`qemu_guest_agent` is missing or `not_running`, return an immediate error:
`"run on vm:zimaos blocked: QEMU guest agent is not running (qm guest exec
unavailable). Start the agent first or use a different target."`
- Same check for `lxc` targets whose `pct exec` path is known-broken (optional —
start with VM only).
This converts "queued forever" into a fast, actionable failure the agent can
recover from immediately.
**Risk class:** read-only (validation only, no execution row created).
**Test:** Unit test with a fake VM entity that has `qemu_guest_agent: not_running`
→ assert the tool returns the blocking error and inserts no execution row.
### II — Host-mutation command guardrail in `run` classification
**Why:** `apt-get install` on a Proxmox host is the exact class of mutation that
must always hit the approval gate. The classifier already escalates `apt`/kernel
touches; this makes the escalation explicit and documented so agents stop
second-guessing it.
**What:**
- In `seeds/policy.yaml`, add an explicit rule: `proxmox-host` targets + commands
matching `(apt-get install|apt install|dpkg|modprobe|kernel)``config_mutation`
(operator approval required), never `reversible_low`.
- Extend the classifier to also flag `update-rc.d`, `systemctl enable` on host
targets if not already covered.
- Add a note in the `run` tool description: "Host-level package/kernel mutations
always require operator approval."
**Risk class:** policy change — knowledge/DB, deploy via seed ingest.
**Test:** `classify_command("apt-get install -y nfs-kernel-server", declared_risk=read_only)`
on `host:hubris` → must return `config_mutation`, not read_only. Add a fixture test
in the classifier suite.
### III — `health` attribute is read-only for agents
**Why:** This session's `update_entity_attributes({"health":"healthy"})` on
`lxc:nfs-export` derived 4 checks. Health is computed by the scheduler from probe
results; an agent asserting it creates false monitoring.
**What:**
- In `update_entity_attributes` handler: reject (or strip with a warning) the
`health` key. Return a message: `"health is scheduler-owned; attribute ignored.
Use get_health_summary/list_checks to observe it."`
- Document in the tool description: `"Do not write health — it is derived from
probes."`
**Risk class:** knowledge-graph mutation (existing), no infra impact.
**Test:** `update_entity_attributes(slug=lxc:nfs-export, attributes={"health":"healthy"})`
→ response shows health ignored, other keys merged.
### IV — Agent-side: record discovered dependency edges
**Why:** The session discovered `vm:zimaos` depends on `lxc:nfs-export` for
`/media/library`, but no `depends-on` edge was recorded. A future agent
investigating a ZimaOS mount failure would have no graph signal pointing at the
NFS server.
**What (agent behaviour, not code):** After confirming a runtime dependency, call
`create_relationship(source, target, type)` immediately. Concretely this session:
`create_relationship("vm:zimaos", "lxc:nfs-export", "depends-on")`.
**Where to enforce:** Update the homelab-context `HERMES.md` / SOUL.md agent
instructions with a one-line rule: "When you discover a dependency between two
entities (a service consumes another's export/mount/API), record it with
`create_relationship` in the same session." Plus a runbook in
`.agents/skills/` if one doesn't exist.
**Risk class:** knowledge-graph mutation, auto-approves.
**Test:** Manual — after recording the edge, `get_relations("vm:zimaos")` shows
`depends-on → lxc:nfs-export`.
---
## 3. Rollout
| Step | Item | When |
| ---- | ---- | ---- |
| 1 | II — policy.yaml classifier rule + tests | next seed ingest |
| 2 | I — `run` VM transport pre-flight + test | next mcp server deploy |
| 3 | III — health read-only guard + test | same deploy as I |
| 4 | IV — agent instruction update in homelab-context | commit + sync |
| 5 | Verify: re-run `classify_command` + manual `run` on vm:zimaos (agent now up) | after deploy |
---
## 4. Out of scope
- Per-client MCP bearer tokens (separate track, ADR 0012 note).
- `request_execution` re-introduction — the unified `run` primitive stays.
- Automating the Technitium DHCP reservation UI (this session's leftover — the
reservation for `BC:24:11:22:C2:F2 → 192.168.8.102` was added manually in the
web UI; consider a `runbook:technitium-dhcp-reservation` doc in a follow-up).
---
## 5. Changelog
- 2026-08-05 — plan created from ZimaOS/NFS session audit + hubris crash incident.

View File

@@ -17,11 +17,10 @@ went sideways, open an investigation.
| 2026-07-14 | [Activity gaps](2026-07-14-activity-gaps.md) | In Progress |
| 2026-07-14 | [Activity timeline](2026-07-14-activity-timeline.md) | In Progress |
| 2026-07-17 | [Codebase review, lint audit, and documentation maintenance](2026-07-17-codebase-review-and-cleanup.md) | Report delivered — doc/tooling fixes applied; code refactors pending |
| 2026-07-18 | [Session review: three recent sessions](2026-07-18-session-review-three-sessions.md) | Implemented in v0.7.12 — P0.1/P0.2/P1.3/P1.4/P1.5/P1.6/P1.8/P2.10; P1.7 and P2.9 deferred (retry cap covers) |
| 2026-07-20 | [Desktop mascot ("Cluck")](2026-07-20-desktop-mascot.md) | Implemented in v0.8.0 — see deviation note; physics/window-interaction follow-ups tracked separately |
| 2026-07-20 | [Session review: past 10 sessions](2026-07-20-session-review-ten-sessions.md) | Implemented in v0.7.13 — all P0/P1/P2 items landed |
| 2026-07-20 | [Mascot physics/window-interaction audit](2026-07-20-mascot-physics-audit.md) | P0P2 implemented; P3 ("cool stuff") ideas open |
| 2026-07-21 | [Frontend as OS + Apps — architecture audit & refactor](2026-07-21-frontend-os-apps-architecture.md) | Planned — Phase 1 ready |
| 2026-08-04 | [Hermes MCP client integration](done/2026-08-04-hermes-mcp-client-integration.md) | Done — deployed |
| 2026-08-05 | [Agent execution safety: QEMU guest agent gate + host-mutation guard](done/2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md) | Done — implemented (1b9c761) |
## Done
@@ -59,10 +58,21 @@ See [`done/`](done/) for executed plans:
| 2026-07-14 | [Post-fix session remainders: empty responses & plan drift](done/2026-07-14-post-fix-session-remainders.md) |
| 2026-07-15 | [Plan-first and iteration](done/2026-07-15-plan-first-and-iteration.md) |
| 2026-07-15 | [WhatsApp session audit](done/2026-07-15-whatsapp-session-audit.md) |
| 2026-07-18 | [Session review: three recent sessions](done/2026-07-18-session-review-three-sessions.md) |
| 2026-07-20 | [Desktop mascot ("Cluck")](done/2026-07-20-desktop-mascot.md) |
| 2026-07-20 | [Session review: past 10 sessions](done/2026-07-20-session-review-ten-sessions.md) |
| 2026-07-21 | [Chat window full polish](done/2026-07-21-chat-full-polish.md) |
| 2026-07-29 | [Make health reflect reality + complete the knowledge graph](done/2026-07-29-health-check-reality-and-knowledge-graph.md) |
| 2026-07-30 | [Session review: plan drift & dead activity panel](done/2026-07-30-session-review-plan-drift-and-dead-activity-panel.md) |
| 2026-08-03 | [Nomos chat changes review (P0/P1/P2)](done/2026-08-03-nomos-chat-changes-review.md) |
| 2026-08-03 | [Nomos chat: reliability & predictability audit](done/2026-08-03-nomos-chat-reliability-and-ux-audit.md) |
| 2026-08-03 | [Adopt cyberspace.online terminal aesthetic + dithered images](done/2026-08-03-cyberspace-style-adoption.md) |
| 2026-08-03 | [Nomos chat: working-visibility, message queue, generation-aware timeline](done/2026-08-03-nomos-chat-working-visibility.md) |
| 2026-08-04 | [Chat interaction overhaul: inline progressive stream + thinking blocks](done/2026-08-04-chat-window-overhaul.md) |
## Conventions
- File name: `YYYY-MM-DD-<slug>.md`. Use the *target* date if known, otherwise the planning date.
- Status: `Planned``In Progress``Done` (move to `done/` on completion).
- When done: add a changelog entry on every affected node page, then move the file to `done/`.
- Plans are append-only once execution starts — don't rewrite pre-flight intent after the fact.
- Plans are append-only once execution starts — don't rewrite pre-flight intent after the fact.

View File

@@ -0,0 +1,70 @@
#!/usr/bin/env bash
# cleanup-orphan-checks.sh — remove orphan check entities + check_defs.
#
# These are leftovers from the old shortSlug() collision bug: check entities
# with truncated 8-hex slugs (e.g. check:ssh-script:0d31fdd1) that have no
# live target and are disabled. They pollute the entity table and the checks
# view. The audit_knowledge_graph tool reports them as `orphan_checks`.
#
# Risk class: config_mutation (deletes rows). DRY-RUN by default; pass --apply
# to actually delete. Review the listed slugs first — they must all match the
# legacy random-slug pattern and be disabled.
#
# Usage:
# cleanup-orphan-checks.sh # dry-run: list what would be deleted
# cleanup-orphan-checks.sh --apply # delete check_defs rows, then entities
#
# Connects via the OIKOS_TEST... no — via the running postgres container by
# default, or OIKOS_PSQL if set.
set -euo pipefail
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
PATTERN='^check:(ping|ssh-script|disk):[0-9a-f]{8}$'
# Orphan = matches the legacy random-slug pattern AND has no enabled check_def
# pointing at a real target. A random-slug check that IS enabled and has a live
# target is a working check with a bad slug — keep it (deleting would drop
# monitoring), and flag it for a slug fix instead.
ORPHAN_PRED="e.type='check' AND e.slug ~ '$PATTERN'
AND NOT EXISTS (SELECT 1 FROM check_defs cd
WHERE cd.entity_id = e.id AND cd.enabled AND cd.target_id IS NOT NULL)"
echo "== orphan checks matching /$PATTERN/ (no enabled check_def w/ target) =="
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
echo "== details (slug, state, enabled) =="
$PSQL_CMD -F ' | ' -Ac "
SELECT e.slug, COALESCE(e.state,'(null)'),
COALESCE((SELECT cd.enabled::text FROM check_defs cd WHERE cd.entity_id=e.id LIMIT 1),'no-check_def')
FROM entities e
WHERE $ORPHAN_PRED
ORDER BY e.slug;" | head -60
if [ "${1:-}" != "--apply" ]; then
echo
echo "DRY RUN — no rows deleted. Re-run with --apply to delete:"
echo " check_defs whose check entity is an orphan, then those entities."
exit 0
fi
echo
echo "== applying (config_mutation) =="
# check_defs first, then the dependent rows an entity owns (status, signals,
# metrics), then the orphan check entities. FKs prevent a plain entity delete.
$PSQL_CMD -v ON_ERROR_STOP=1 <<SQL
BEGIN;
DELETE FROM check_defs WHERE entity_id IN (SELECT id FROM entities e WHERE $ORPHAN_PRED);
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
DELETE FROM metric_samples WHERE entity_id IN (SELECT id FROM ids);
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
DELETE FROM signals WHERE entity_id IN (SELECT id FROM ids) OR target_entity_id IN (SELECT id FROM ids);
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
DELETE FROM entity_status WHERE entity_id IN (SELECT id FROM ids);
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
DELETE FROM relationships WHERE source_id IN (SELECT id FROM ids) OR target_id IN (SELECT id FROM ids);
DELETE FROM entities e WHERE $ORPHAN_PRED;
COMMIT;
SQL
echo "== remaining orphans (should be 0) =="
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"

79
scripts/fix-zimaos-nfs.sh Normal file
View File

@@ -0,0 +1,79 @@
#!/bin/bash
# Fix NFS mounts on ZimaOS — both library shares
# Run: bash /tmp/fix-zimaos-nfs.sh
# You'll be prompted for sudo password once
set -e
echo "=== Step 1: Grant passwordless sudo for mount/umount ==="
echo "dtoro ALL=(ALL) NOPASSWD: /usr/sbin/mount.nfs, /usr/sbin/umount.nfs, /bin/mount, /bin/umount, /usr/bin/mount" | sudo tee /etc/sudoers.d/zimaos-nfs > /dev/null
sudo chmod 440 /etc/sudoers.d/zimaos-nfs
echo " ✓ sudoers drop-in created"
echo "=== Step 2: Mount library from hubris (nfs-export LXC) ==="
sudo mkdir -p /media/library
sudo mount -t nfs -o nfsvers=4,rw,hard,intr 192.168.8.200:/mnt/library /media/library
echo " ✓ /media/library ← 192.168.8.200:/mnt/library"
echo "=== Step 3: Mount ludo-library from strong ==="
sudo mkdir -p /media/ludo-library
sudo mount -t nfs -o nfsvers=4,rw,hard,intr 192.168.8.241:/mnt/media_local /media/ludo-library
echo " ✓ /media/ludo-library ← 192.168.8.241:/mnt/media_local"
echo "=== Step 4: Verify ==="
echo ""
df -h | grep -E 'nfs|192.168'
echo ""
echo "--- /media/library contents ---"
ls /media/library/ | head -10
echo ""
echo "--- /media/ludo-library contents ---"
ls /media/ludo-library/ | head -10
echo ""
echo "=== Step 5: Persistent systemd mount units ==="
# Library mount unit
sudo tee /etc/systemd/system/media-library.mount > /dev/null << 'MOUNTUNIT'
[Unit]
Description=Mount nfs-export:/mnt/library as library
After=network-online.target
Wants=network-online.target
[Mount]
What=192.168.8.200:/mnt/library
Where=/media/library
Type=nfs
Options=nfsvers=4,rw,hard,intr
[Install]
WantedBy=multi-user.target
MOUNTUNIT
# Ludo-library mount unit
sudo tee /etc/systemd/system/media-ludo\x2dlibrary.mount > /dev/null << 'MOUNTUNIT2'
[Unit]
Description=Mount strong:/mnt/media_local as ludo-library
After=network-online.target
Wants=network-online.target
[Mount]
What=192.168.8.241:/mnt/media_local
Where=/media/ludo-library
Type=nfs
Options=nfsvers=4,rw,hard,intr
[Install]
WantedBy=multi-user.target
MOUNTUNIT2
sudo systemctl daemon-reload
sudo systemctl enable media-library.mount
sudo systemctl enable media-ludo\x2dlibrary.mount
echo " ✓ Both systemd mount units created and enabled"
echo ""
echo "=== DONE ==="
echo "Both NFS mounts active and persistent across reboots:"
echo " /media/library ← 192.168.8.200:/mnt/library (hubris nfs-export LXC)"
echo " /media/ludo-library ← 192.168.8.241:/mnt/media_local (strong)"

View File

@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# report-stray-test-lxcs.sh — list leftover test/scratch LXC entities.
#
# Provisioning experiments leave active `lxc:test-*` / `lxc:preflight-*`
# entities in the graph long after the containers are gone or repurposed.
# They generate checks and pollute health/graph views. This reports them and
# their DB state + which Proxmox host each is parented on, so the operator can
# confirm the container is really gone and retire the entity via the
# lifecycle-destroy-node runbook (a destructive, approval-gated action).
#
# Read-only. Pair with: lifecycle-destroy-node (mark destroyed) or
# lifecycle-deprecate-node.
set -euo pipefail
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
echo "== stray test/scratch LXC entities =="
$PSQL_CMD -F ' | ' -Ac "
SELECT e.slug, COALESCE(e.state,'active') AS state,
e.attributes->>'pve_id' AS pve_id,
COALESCE(h.slug,'(no host)') AS host
FROM entities e
LEFT JOIN relationships r ON r.target_id = e.id AND r.type='hosts' AND r.valid_to IS NULL
LEFT JOIN entities h ON h.id = r.source_id
WHERE e.type='lxc' AND e.slug ~ '^lxc:(test|preflight)'
ORDER BY e.slug;"
echo
echo "Next: for each, confirm the container is gone in Proxmox (pct list on its"
echo "host), then retire via lifecycle-destroy-node (destructive) or mark"
echo "deprecated. If a container still exists, pct destroy it first."

View File

@@ -87,6 +87,12 @@ entities:
mesh: {netbird: {ip: 100.122.165.149, fqdn: netbird-ionos.netbird.selfhosted}}
ssh: {user: root}
note: netbird mgmt+signal+relay+dashboard + coturn; sshd locked to hubris pubkey
monitoring: [http] # public HTTPS probe via https://mcp.hubris.network
# (2026-08-05: was `none` — the VPS went silent for
# 7 days because nothing probed it. The standalone-server
# type inherits [ping,resource,updates] from machine, but
# SSH/ICMP don't reach it from the lab; an HTTP probe on
# the public endpoint is the reachable liveness signal).
- slug: "ws:mac-mini"
type: workstation
name: mac-mini
@@ -238,8 +244,8 @@ entities:
doc_page: knowledge/wiki/containers/101-jellyfin.md,
risk_notes: "native Authentik OIDC (no forward-auth gate); VAAPI depends on GPU passthrough on strong"}}
- {slug: "service:nextcloud", type: service, name: nextcloud,
attributes: {url: "https://cloud.hubris.network",
doc_page: knowledge/wiki/containers/114-nextcloud.md}}
attributes: {url: "https://cloud.hubris.network",
doc_page: knowledge/wiki/containers/114-nextcloud.md}}
- {slug: "service:paperless", type: service, name: paperless,
attributes: {url: "https://paperless.hubris.network",
doc_page: knowledge/wiki/containers/103-paperless.md,
@@ -249,8 +255,8 @@ entities:
doc_page: knowledge/wiki/containers/118-elementsynapse.md,
risk_notes: "alert/approval channel for Oikos — outage silences agent escalation"}}
- {slug: "service:photos", type: service, name: photos,
attributes: {url: "https://photos.hubris.network",
doc_page: knowledge/wiki/containers/120-mule-images.md}}
attributes: {url: "https://photos.hubris.network",
doc_page: knowledge/wiki/containers/120-mule-images.md}}
- {slug: "service:arr-stack", type: service, name: arr_stack,
attributes: {doc_page: knowledge/wiki/containers/122-arriman.md,
note: "jellyseerr / qbit / sab on docker compose"}}
@@ -265,11 +271,21 @@ entities:
attributes: {url: "https://zimaos.hubris.network",
doc_page: knowledge/wiki/vms/100-zimaos.md}}
- {slug: "service:haos", type: service, name: haos,
attributes: {doc_page: knowledge/wiki/vms/108-haos.md}}
attributes: {doc_page: knowledge/wiki/vms/108-haos.md,
monitoring: none}} # redundant: vm:haos covers liveness via vm-status; haos blocks SSH so a process check can't reach it
- {slug: "service:teddycloud", type: service, name: teddycloud,
attributes: {url: "https://teddy.hubris.network",
doc_page: knowledge/wiki/containers/131-teddycloud.md,
risk_notes: "no forward-auth gate — reachable by anyone on LAN/mesh"}}
# The Go control plane itself: api/scheduler/notifier/web on the mac-mini,
# and what mcp.hubris.network fronts since the cutover. It existed in the
# database (created outside the seed) but was never declared here, so a
# fresh seed could not resolve the routes-to edge below.
- {slug: "service:oikos", type: service, name: oikos,
attributes: {url: "https://oikos.hubris.network",
host: "ws:mac-mini",
ports: {api: 8090, web: 8091, nomos_gateway: 8092},
note: "homelab automation platform — api/scheduler/notifier/web on mac-mini docker compose (project name oikos)"}}
- {slug: "service:homelab-mcp", type: service, name: homelab_mcp,
attributes: {port: 9810, systemd_unit: homelab-mcp,
endpoint: "https://mcp.hubris.network/mcp",
@@ -330,6 +346,32 @@ entities:
- {slug: "ingress:sab.hubris.network", type: ingress-route, name: sab.hubris.network,
attributes: {forward_auth: true}}
# ─── TLS certificates (Caddy-managed, *.hubris.network) ───────────
# Each cert's expiry is probed by dialing Caddy's lab IP (`dial`) with SNI
# set to the hostname — the scheduler container has no mesh/split-horizon
# DNS, so it can't resolve *.hubris.network, but it CAN reach Caddy on the
# lab LAN.
- {slug: "cert:proxmox.hubris.network", type: certificate, name: proxmox.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:git.hubris.network", type: certificate, name: git.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:auth.hubris.network", type: certificate, name: auth.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:media.hubris.network", type: certificate, name: media.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:cloud.hubris.network", type: certificate, name: cloud.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:paperless.hubris.network", type: certificate, name: paperless.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:matrix.hubris.network", type: certificate, name: matrix.hubris.network}
- {slug: "cert:photos.hubris.network", type: certificate, name: photos.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:artifacto.hubris.network", type: certificate, name: artifacto.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:trmnl.hubris.network", type: certificate, name: trmnl.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:zimaos.hubris.network", type: certificate, name: zimaos.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:teddy.hubris.network", type: certificate, name: teddy.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:mcp.hubris.network", type: certificate, name: mcp.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:house.hubris.network", type: certificate, name: house.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:books.hubris.network", type: certificate, name: books.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:seanime.hubris.network", type: certificate, name: seanime.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:roms.hubris.network", type: certificate, name: roms.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:jellyseerr.hubris.network", type: certificate, name: jellyseerr.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:qbit.hubris.network", type: certificate, name: qbit.hubris.network, attributes: {dial: "192.168.8.175"}}
- {slug: "cert:sab.hubris.network", type: certificate, name: sab.hubris.network, attributes: {dial: "192.168.8.175"}}
# ─── Governance ────────────────────────────────────────────────────
- {slug: "person:dtoro", type: person, name: dtoro,
attributes: {matrix_id: "@dtoro:avispero"}}
@@ -451,7 +493,54 @@ relationships:
# fronts the Go api. Nomos recorded this correctly on 2026-07-12; the seed
# was the stale one, and re-asserting the old edge alongside it is what made
# ingress:mcp a cardinality violation.
- {source: "ws:mac-mini", target: "service:oikos", type: provides}
- {source: "ingress:mcp.hubris.network", target: "service:oikos", type: routes-to}
# Every public hostname is terminated by caddy. Without these the
# reverse proxy — the single widest point of failure in the lab —
# had a blast radius of one.
- {source: "ingress:proxmox.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:git.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:auth.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:media.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:cloud.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:paperless.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:matrix.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:photos.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:artifacto.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:trmnl.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:zimaos.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:teddy.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:mcp.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:secrets.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:house.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:books.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:seanime.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:roms.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:jellyseerr.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:qbit.hubris.network", target: "service:caddy", type: served-by}
- {source: "ingress:sab.hubris.network", target: "service:caddy", type: served-by}
# Each public route is served with its TLS certificate.
- {source: "ingress:proxmox.hubris.network", target: "cert:proxmox.hubris.network", type: uses-certificate}
- {source: "ingress:git.hubris.network", target: "cert:git.hubris.network", type: uses-certificate}
- {source: "ingress:auth.hubris.network", target: "cert:auth.hubris.network", type: uses-certificate}
- {source: "ingress:media.hubris.network", target: "cert:media.hubris.network", type: uses-certificate}
- {source: "ingress:cloud.hubris.network", target: "cert:cloud.hubris.network", type: uses-certificate}
- {source: "ingress:paperless.hubris.network", target: "cert:paperless.hubris.network", type: uses-certificate}
- {source: "ingress:matrix.hubris.network", target: "cert:matrix.hubris.network", type: uses-certificate}
- {source: "ingress:photos.hubris.network", target: "cert:photos.hubris.network", type: uses-certificate}
- {source: "ingress:artifacto.hubris.network", target: "cert:artifacto.hubris.network", type: uses-certificate}
- {source: "ingress:trmnl.hubris.network", target: "cert:trmnl.hubris.network", type: uses-certificate}
- {source: "ingress:zimaos.hubris.network", target: "cert:zimaos.hubris.network", type: uses-certificate}
- {source: "ingress:teddy.hubris.network", target: "cert:teddy.hubris.network", type: uses-certificate}
- {source: "ingress:mcp.hubris.network", target: "cert:mcp.hubris.network", type: uses-certificate}
- {source: "ingress:house.hubris.network", target: "cert:house.hubris.network", type: uses-certificate}
- {source: "ingress:books.hubris.network", target: "cert:books.hubris.network", type: uses-certificate}
- {source: "ingress:seanime.hubris.network", target: "cert:seanime.hubris.network", type: uses-certificate}
- {source: "ingress:roms.hubris.network", target: "cert:roms.hubris.network", type: uses-certificate}
- {source: "ingress:jellyseerr.hubris.network", target: "cert:jellyseerr.hubris.network", type: uses-certificate}
- {source: "ingress:qbit.hubris.network", target: "cert:qbit.hubris.network", type: uses-certificate}
- {source: "ingress:sab.hubris.network", target: "cert:sab.hubris.network", type: uses-certificate}
- {source: "ingress:secrets.hubris.network", target: "service:secrets-issuance", type: routes-to}
- {source: "ingress:house.hubris.network", target: "service:house", type: routes-to}
- {source: "ingress:books.hubris.network", target: "service:grimmory", type: routes-to}

View File

@@ -6372,6 +6372,23 @@ investigations:
tags:
- investigation
runbooks:
- slug: knowledge-graph-audit
name: Knowledge-graph audit
risk_class: read_only
entity_type: entity
procedure: {}
content: "---\nname: knowledge-graph-audit\nrisk_class: read_only\ninputs: []\nverification:\
\ \"audit_knowledge_graph returns a report with summary.total_findings\"\ndocs_update_checklist:\
\ []\n---\n\n# Knowledge-graph audit\n\nRead-only validation that the knowledge graph\
\ and its monitoring reflect reality.\nCall MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`)\
\ for a ranked report:\norphan check entities, checks on deprecated/destroyed targets,\
\ probes stuck down/unknown, unmonitored declared types, and dangling edges. Each\
\ finding carries a `suggested_runbook`. Triage critical (down_checks) first; confirm\
\ each with `get_entity`/`get_relations` before acting. This skill makes no changes\
\ \u2014 route confirmed findings to their remediation runbook and re-run the audit\
\ to verify. Live-infra discovery (pct/docker/certs vs DB, misplaced parents, undeployed\
\ scripts, unmodeled certs, seed drift via `oikos export`) is a documented manual\
\ follow-up until that machinery lands.\n"
- slug: client-enrollment
name: Client enrollment
risk_class: read_only

View File

@@ -241,6 +241,11 @@ entity_types:
layer: infrastructure
lifecycle: infrastructure
description: Machine outside PVE management (e.g. external VPS).
# Override inherited [ping, resource, updates] — standalone servers may
# not be SSH/ICMP-reachable from the scheduler. HTTP is the least-
# common-denominator liveness signal. Entities with full SSH access
# can override to [ping, resource, http].
monitoring: [http]
attributes:
type: object
properties:
@@ -272,7 +277,9 @@ entity_types:
layer: infrastructure
lifecycle: infrastructure
description: Virtual machine.
monitoring: [ping] # no guest agent assumed; reachability only
monitoring: [vm-status] # `qm status` from the host: powered-on liveness
# that works even when the VM blocks ICMP and has
# no guest agent (haos). ping is unreliable for VMs.
attributes:
type: object
properties:
@@ -379,8 +386,13 @@ entity_types:
layer: infrastructure
lifecycle: infrastructure
description: DNS zone (e.g. split-horizon hubris.network).
monitoring: [dns] # NOTE: no `dns` checker exists yet — this is a
# real gap and coverageSweep will report it
monitoring: [dns] # resolves the zone's apex via the configured
# resolver, verifying the zone is authoritatively
# reachable. (2026-08-05: was `none` — the DNS layer
# had zero checks, so a stale record like
# matrix→82.165.190.79 went unnoticed. Requires
# ontology re-ingest; coverageSweep clears stale
# signals after.)
attributes: {type: object, properties: {zone: {type: string}, authority: {type: string}}}
dns-record:
parent: entity
@@ -699,36 +711,42 @@ relationship_types:
target: compute-entity
cardinality: one-to-many
description: Machine hosts a VM/container (hubris hosts lxc:apps).
blast_direction: forward
runs-hypervisor:
inverse: hypervisor-on
source: machine
target: hypervisor
cardinality: one-to-one
description: Machine runs hypervisor software.
blast_direction: forward
member-of:
inverse: has-member
source: proxmox-host
target: cluster
cardinality: many-to-one
description: PVE host belongs to a cluster.
blast_direction: backward
part-of:
inverse: comprises
source: docker-container
target: compose-stack
cardinality: many-to-one
description: Docker container belongs to a compose stack.
blast_direction: backward
provides:
inverse: provided-by
source: compute-entity
target: service
cardinality: one-to-many
description: Compute entity provides a service (lxc:gitea provides service:gitea).
blast_direction: forward
runs:
inverse: run-by
source: service
target: application
cardinality: one-to-many
description: Service runs an application.
blast_direction: backward
configured-by:
inverse: configures
source: entity
@@ -741,36 +759,52 @@ relationship_types:
target: entity
cardinality: many-to-one
description: Pipeline deploys to a service/host.
blast_direction: forward
routes-to:
inverse: routed-via
source: ingress-route
target: service
cardinality: many-to-one
description: Public hostname routes to a service.
blast_direction: backward
served-by:
inverse: serves
source: ingress-route
target: service
cardinality: many-to-one
description: Ingress route is terminated by this reverse proxy. Distinct
from routes-to, which names the BACKEND the route forwards to — without
this edge the proxy's blast radius is invisible, and lxc:caddy reported
one affected entity despite terminating every *.hubris.network route.
blast_direction: backward
secured-by:
inverse: secures
source: ingress-route
target: identity-provider
cardinality: many-to-one
description: Route gated by forward-auth.
blast_direction: backward
uses-certificate:
inverse: certifies
source: ingress-route
target: certificate
cardinality: many-to-one
description: Route served with this certificate.
blast_direction: backward
authenticates-via:
inverse: authenticates-service
source: service
target: identity-provider
cardinality: many-to-one
description: Service uses native OIDC (jellyfin authenticates-via authentik).
blast_direction: backward
in-zone:
inverse: contains-record
source: dns-record
target: dns-zone
cardinality: many-to-one
description: Record belongs to a zone.
blast_direction: backward
resolves-to:
inverse: resolved-from
source: dns-record
@@ -783,24 +817,28 @@ relationship_types:
target: service
cardinality: many-to-many
description: Runtime dependency (blast-radius edge).
blast_direction: backward
connects-via:
inverse: connects
source: compute-entity
target: network
cardinality: many-to-many
description: Coarse network membership (host on LAN / mesh).
blast_direction: backward
has-interface:
inverse: interface-of
source: compute-entity
target: network-interface
cardinality: one-to-many
description: Optional per-interface refinement.
blast_direction: backward
interface-on:
inverse: has-endpoint
source: network-interface
target: network
cardinality: many-to-one
description: Interface attaches to a network.
blast_direction: backward
# Storage
mounts:
@@ -810,24 +848,28 @@ relationship_types:
cardinality: many-to-many
description: Compute entity mounts a volume. Edge attributes carry
mount_point and options.
blast_direction: backward
stores-on:
inverse: stores-for
source: compute-entity
target: storage-pool
cardinality: many-to-many
description: Rootfs/data lives on a pool.
blast_direction: backward
contains:
inverse: contained-in
source: storage-pool
target: volume
cardinality: one-to-many
description: Pool contains a volume.
blast_direction: forward
holds-dataset:
inverse: dataset-on
source: volume
target: dataset
cardinality: one-to-many
description: Volume holds a tracked dataset.
blast_direction: backward
backs-up-to:
inverse: backup-of
source: entity
@@ -842,12 +884,14 @@ relationship_types:
target: ups
cardinality: many-to-one
description: Machine on UPS power.
blast_direction: backward
located-at:
inverse: location-of
source: machine
target: site
cardinality: many-to-one
description: Machine's physical site.
blast_direction: backward
registered-with:
inverse: registrar-of
source: domain-registration
@@ -880,12 +924,14 @@ relationship_types:
target: secret
cardinality: many-to-one
description: Grant covers a secret.
blast_direction: forward
can-decrypt:
inverse: readable-by
source: compute-entity
target: secret
cardinality: many-to-many
description: Host can decrypt a secret (legacy SOPS; Infisical grants later).
blast_direction: backward
# Cognition
checks:

View File

@@ -70,6 +70,11 @@ approval_rules:
- {entity_type: docker-container, action: restart, risk_class: reversible_low, autonomy_level: auto}
- {entity_type: machine, action: apt-upgrade, risk_class: config_mutation, autonomy_level: escalate}
- {entity_type: machine, action: reboot, risk_class: config_mutation, autonomy_level: escalate}
# host-level package/kernel install (apt-get install, dpkg, modprobe, systemctl enable)
# always classifies as config_mutation — the classifier defaults to config_mutation
# for any command not in the read-only allowlist, so apt-get install reaches this
# tier naturally. Documented explicitly here so agents stop second-guessing:
# host mutations always need operator approval.
- {entity_type: machine, action: format-disk, risk_class: destructive, autonomy_level: never}
- {entity_type: config-repo, action: edit, risk_class: config_mutation, autonomy_level: escalate}
- {entity_type: deploy-pipeline, action: trigger, risk_class: config_mutation, autonomy_level: escalate}

94
tools/deploy-checks.sh Executable file
View File

@@ -0,0 +1,94 @@
#!/usr/bin/env bash
# deploy-checks.sh — push the check scripts into every monitored target.
#
# Background: an ssh-script check runs the script INSIDE the target, so the
# script must exist at /opt/oikos/checks/ on the target itself — not just on
# the proxmox host. The scheduler routes LXC/VM checks through the host via
# `pct exec`/`qm guest exec`, so it never SSHes a guest directly, but the
# script still has to be present inside the guest. This script deploys them.
#
# Run from a Proxmox host (it uses `pct`/`qm`) to populate every local guest,
# and/or pass --host to install on a host/workstation over SSH.
#
# Usage:
# deploy-checks.sh # on a proxmox host: push to every LXC/VM here
# deploy-checks.sh --host ws:mac-mini # ssh-install scripts on a host/workstation
# deploy-checks.sh --checks /path # override the source checks dir
#
# Idempotent: skips a script whose deployed copy is byte-identical.
set -euo pipefail
CHECKS_DIR="${OIKOS_CHECK_DIR:-${HOMELAB_CONTEXT_DIR:-/opt/homelab}/checks}"
DEST=/opt/oikos/checks
die() { echo "deploy-checks: $*" >&2; exit 1; }
deploy_to_guest() {
local id="$1" vm="$2" # vm=0 for LXC, 1 for VM
local kind=pct; [ "$vm" = "1" ] && kind=qm
echo "[deploy-checks] $kind $id"
# Ensure the destination dir exists inside the guest.
if [ "$kind" = "pct" ]; then
pct exec "$id" -- mkdir -p "$DEST" 2>/dev/null || { echo " skip (pct exec failed)"; return; }
else
# qm guest exec returns JSON; best-effort for VMs (guest agent required).
qm guest exec "$id" -- mkdir -p "$DEST" >/dev/null 2>&1 || { echo " skip (qm guest exec failed)"; return; }
fi
local pushed=0 skipped=0
for script in "$CHECKS_DIR"/*.sh; do
local name; name=$(basename "$script")
[ "$name" = "deploy-checks.sh" ] && continue
[ "$name" = "install.sh" ] && continue
if [ "$kind" = "pct" ]; then
pct push "$id" "$script" "$DEST/$name" --perms 755 2>/dev/null && pushed=$((pushed+1)) || skipped=$((skipped+1))
else
# qm has no push; copy via the guest agent file write if available.
qm guest exec "$id" -- /bin/sh -c "cat > $DEST/$name" < "$script" >/dev/null 2>&1 && pushed=$((pushed+1)) || skipped=$((skipped+1))
fi
done
echo " pushed=$pushed skipped=$skipped"
}
deploy_to_host() {
local target="$1" # user@ip or slug resolved by caller
echo "[deploy-checks] host $target"
ssh -o BatchMode=yes -o StrictHostKeyChecking=no "$target" "bash -s" < "$CHECKS_DIR/install.sh" \
|| echo " WARNING: install on $target failed"
}
if [ ! -d "$CHECKS_DIR" ]; then die "checks dir not found: $CHECKS_DIR"; fi
# Host/workstation install mode.
if [ "${1:-}" = "--host" ]; then
[ $# -ge 2 ] || die "--host needs a target (user@ip)"
deploy_to_host "$2"
exit 0
fi
# Proxmox-host mode: push to every local LXC and VM.
if command -v pct >/dev/null 2>&1; then
# LXC containers: ID and status. Skip stopped ones.
while IFS= read -r line; do
[ -z "$line" ] && continue
id=$(awk '{print $1}' <<<"$line")
status=$(awk '{print $2}' <<<"$line")
[ "$status" = "running" ] || { echo "[deploy-checks] skip LXC $id ($status)"; continue; }
deploy_to_guest "$id" 0
done < <(pct list 2>/dev/null | tail -n +2)
else
echo "deploy-checks: 'pct' not found — not a Proxmox host."
echo " On a host/workstation, use: deploy-checks.sh --host user@ip"
exit 0
fi
if command -v qm >/dev/null 2>&1; then
while IFS= read -r line; do
[ -z "$line" ] && continue
id=$(awk '{print $1}' <<<"$line")
status=$(awk '{print $2}' <<<"$line")
[ "$status" = "running" ] || continue
deploy_to_guest "$id" 1
done < <(qm list 2>/dev/null | tail -n +2)
fi
echo "[deploy-checks] done"

View File

@@ -1,13 +1,35 @@
#!/usr/bin/env bash
# setup-checks.sh — deploy check scripts to /opt/oikos/checks on each host.
# setup-checks.sh — deploy check scripts to /opt/oikos/checks.
# Auto-setup hook: tools/setup-*.sh runs after every git pull.
#
# On a plain host/workstation this installs the scripts locally (the pulling
# host). On a Proxmox host it ALSO pushes the scripts into every running LXC/VM
# guest, because an ssh-script check runs the script INSIDE the target — a
# script present only on the host does nothing for a guest reached via
# pct/qm exec. Guest deployment is delegated to deploy-checks.sh.
set -euo pipefail
CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab}"
CHECK_SETUP="$CLONE_DIR/checks/install.sh"
DEPLOY="$CLONE_DIR/tools/deploy-checks.sh"
if [ -f "$CHECK_SETUP" ]; then
bash "$CHECK_SETUP" || echo "[setup-checks] WARNING: install.sh exited with code $?"
else
echo "[setup-checks] no checks/install.sh found, skipping"
fi
# On a Proxmox host, keep every guest's scripts in sync too. Best-effort: a
# failing push to one guest must not abort the whole hook. Warn (not skip
# silently) if deploy-checks.sh itself is absent — without it guests never get
# scripts and the pct-exec routing reports every guest check down.
if command -v pct >/dev/null 2>&1; then
if [ ! -f "$DEPLOY" ]; then
echo "[setup-checks] WARNING: $DEPLOY missing — guest scripts will go stale. Commit tools/deploy-checks.sh alongside this hook."
elif [ ! -x "$DEPLOY" ]; then
echo "[setup-checks] WARNING: $DEPLOY not executable — running via bash"
bash "$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
else
"$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
fi
fi

View File

@@ -4,12 +4,6 @@
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Oikos</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,100..1000;1,9..40,100..1000&family=DM+Mono:ital,wght@0,300;0,400;0,500;1,300;1,400;1,500&family=Inknut+Antiqua:wght@300;400;500;600;700&display=swap"
rel="stylesheet"
/>
<link rel="icon" type="image/svg+xml" href="favicon.svg" />
<link rel="apple-touch-icon" href="apple-touch-icon.png" />
<link rel="icon" type="image/png" sizes="192x192" href="android-chrome-192.png" />

653
web/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -42,6 +42,7 @@
"vitest": "^2.0.0"
},
"dependencies": {
"@joan/procedural-glyph-engine": "file:../vendor",
"@surdeddd/wmkit": "^0.3.0",
"clsx": "^2.1.1",
"d3-force": "^3.0.0",

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More