From 9daf8220f247bf9b8a8e67b14bf0f3efa5bcbba8 Mon Sep 17 00:00:00 2001 From: dtoro Date: Fri, 10 Jul 2026 09:12:09 +0200 Subject: [PATCH] plan: add observability layer and chat-assent approval to gated-execution plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator directives: (1) the UI must show what's executing, its risk classification, live status, and what knowledge the session created — the system's growth should be visible, not just trusted. (2) approval should be granted by chat assent ("go ahead"), not a separate button; destructive actions still require a typed confirmation phrase. Co-Authored-By: Claude Opus 4.8 --- plans/2026-07-10-general-gated-execution.md | 111 +++++++++++++++++--- 1 file changed, 99 insertions(+), 12 deletions(-) diff --git a/plans/2026-07-10-general-gated-execution.md b/plans/2026-07-10-general-gated-execution.md index 4b307a5..08d894e 100644 --- a/plans/2026-07-10-general-gated-execution.md +++ b/plans/2026-07-10-general-gated-execution.md @@ -19,6 +19,19 @@ Chosen autonomy posture for v1: **approve-most (cautious)** — only genuinely read-only commands auto-run; anything that changes state requires operator approval. We can relax later once the classifier and ledger have earned trust. +Operator directive #2 (2026-07-10): **"I want to see the system come alive and +learn and get better."** Observability is a first-class deliverable, not a +side-effect. As a user I must be able to see, in real time: what is being +executed, on what, and why; how it was classified and routed; what the outcome +was; and — crucially — **what knowledge the session created** (new runbooks, +patterns, resolved signals, ledger entries) so the system's growth is visible. + +Operator directive #3 (2026-07-10): **approval is granted by chat assent, not a +button.** When Nomos proposes a plan/action and the operator replies "go ahead" +/ "yes" / "do it" in the chat, that assent *is* the approval. No separate +Approve button for the normal case. (Destructive actions still require an +explicit typed confirmation phrase — see Safety.) + ## This is a realignment, not a new idea [.agents/OIKOS.md](../.agents/OIKOS.md) already specifies this exact model: @@ -85,13 +98,36 @@ Every call flows through: (mirrors "can only lower autonomy, never raise"). 2. **Route** (approve-most posture): - `read_only` → auto-run + ledger, no approval. - - `reversible_low` / `config_mutation` → **operator approval** (v1 gates all - state changes; a later posture can auto-run `reversible_low`). + - `reversible_low` / `config_mutation` → **operator approval via chat assent** + (v1 gates all state changes; a later posture can auto-run `reversible_low`). - `destructive` → approval **+ typed confirmation phrase**. 3. **Execute** (existing SSH/`pct exec`), **verify** (optional check command), **ledger** (`executions` + `audit_log`), **stream feedback to chat** (reuse the `GET /executions/{id}` polling + `InlineApproval` phases already built). +### Approval by chat assent (replaces the Approve button) + +The operator is already authenticated in the chat session, so their words are +the authorization — a separate button is redundant friction. Flow: + +- Nomos proposes an action/plan; the gated `run` calls sit in `pending_approval` + (created in the same turn, tied to that turn's `correlation_id`). +- The operator's next message is checked for **assent** ("go ahead", "yes", + "do it", "proceed", "ship it") scoped to *that* proposal. On assent, the + pending approvals from that turn are granted and execute. +- Mechanism: Nomos detects assent and calls an `approve_pending(correlation_id)` + action; the backend flips the linked approvals → the existing + `executeApprovedAction` path runs. The **grant is recorded with the exact + operator message** that constituted assent (audit). +- Guards: assent only applies to approvals from the immediately-preceding turn + (no stale "yes" approving something old); ambiguous replies ("maybe", + "later", a follow-up question) do **not** grant — Nomos re-confirms; + **destructive** actions ignore loose assent and still require the typed + confirmation phrase. +- The inline UI still *shows* the pending action and its classification (so the + operator sees what they're assenting to) and reflects the grant — but the + primary path is "say yes," with the button demoted to an optional affordance. + Layer 0 alone delivers "the agent can attempt anything; state changes are gated." ### Layer 1 — runbooks as executable data (reliability without rigidity) @@ -119,6 +155,41 @@ Successful ad-hoc `run` sequences get promoted into runbooks/patterns (the `learning` engine + `skills` table already exist for this); the failure ledger informs retries. The system grows more capable **as data**, not as code. +## Layer 3 — Observability: watch the system come alive + +The user must *see* the OODA loop working, not just trust it. Four surfaces, +built on data the loop already produces (`executions`, `audit_log`, `signals`, +`skills`, `knowledge_entities`) — the job is to make it visible, live, and +legible, not to invent new telemetry. + +**1. Live action feed (in the chat turn).** Every `run` renders a card as it +happens: `target` · `purpose` · **risk badge** (green read-only / amber +config / red destructive) · status (queued → running → ok/failed) · collapsible +output. Streams in real time (SSE, extend the existing execution-status feed). +The operator watches Nomos *work*, step by step, with the reasoning (`purpose`) +and the classifier's verdict on every step. + +**2. "What this session did" digest.** At the end of a task/turn, a summary +card: N commands (X auto / Y assented / Z denied), entities changed (linked), +signals resolved, and **knowledge created** — new/updated runbooks, patterns +promoted, notes written — each linked to its record. This is the "what did the +agent actually change and learn" answer in one glance. + +**3. The learning view — "the system is getting better."** A dedicated page: +runbooks and their **success-rate trend**, newly promoted skills, pattern +confidence (Wilson bounds already computed by the learning engine), recent +auto-acts that succeeded unattended, and a **capability timeline** ("2026-07-11: +learned to deploy Compose stacks; success 4/4"). Growth made tangible. + +**4. Global activity/ledger stream.** A live feed of every action across the +fleet — command, target, classification, decision (auto / assented-by-whom), +outcome — the audit log rendered as a heartbeat. Filterable by entity, risk, +outcome. + +These reuse existing tables; the work is API endpoints + SSE fan-out + Svelte +views, plus writing knowledge-creation events into the ledger so the digest has +something to show. + ## Safety model (the whole point of the gate) - **Default-escalate.** Nothing is *forbidden*; risky things need the operator's @@ -154,15 +225,25 @@ informs retries. The system grows more capable **as data**, not as code. mutating / catastrophic commands. 2. **`run` tool** — new MCP tool routing classify → gate → execute → the existing feedback path. Ship alongside the current tools (no removal yet). -3. **Approval context** — surface risk class + blast radius + purpose on the - approval (chat `InlineApproval` + Ops page); typed-confirmation for - destructive. -4. **Runbook execution** — a "provision LXC" runbook (ports the current +3. **Live action feed (UI)** — render each `run` as a streaming card in chat: + purpose, target, risk badge, status, output. This is the first "come alive" + win and validates the SSE fan-out. +4. **Chat-assent approval** — assent detection scoped to the last turn's + `correlation_id` → `approve_pending`; grant records the operator's message; + destructive still needs the typed phrase. Demote the Approve button. +5. **Approval context** — surface risk class + blast radius + purpose inline so + the operator sees what they're assenting to. +6. **Session digest + activity stream (UI)** — "what this session did / created" + card and the global ledger feed; write knowledge-creation events to the + ledger so there's something to show. +7. **Runbook execution** — a "provision LXC" runbook (ports the current `pct_create` logic) executed via `run`; validate parity with today's handler. -5. **Retire the enum** — convert remaining hard-coded actions to runbooks; make +8. **Learning view (UI)** — runbook success-rate trends, promoted skills, + capability timeline. +9. **Retire the enum** — convert remaining hard-coded actions to runbooks; make `request_execution` a thin deprecated alias or remove it. -6. **Revive auto-act** — replace the actuator stub, reusing the *same* classifier - for the Observe→Act direction (signals), still approve-most. +10. **Revive auto-act** — replace the actuator stub, reusing the *same* + classifier for the Observe→Act direction (signals), still approve-most. ## Verification @@ -171,8 +252,11 @@ informs retries. The system grows more capable **as data**, not as code. commands escalate. No command auto-runs that mutates state. - End-to-end: operator asks Nomos a novel task **not** in the old enum (e.g. "tail caddy's error log and restart it if it's flapping"); Nomos composes - `run` calls; read-only steps auto-run, the restart gates for approval; chat - shows live status; ledger records each command + classification. + `run` calls; read-only steps auto-run and **stream as live cards**; the restart + gates; the operator types "go ahead" and the restart executes (no button); + ledger records each command + classification + the assent message. +- Observability: the session ends with a digest listing what ran, what changed, + and any knowledge created; the learning view shows the run's contribution. - Parity: "provision an LXC with a service" via the runbook path matches the reliability proven for the `pct_create` handler (free VMID, DNS, install, verify), then destroy. @@ -182,7 +266,10 @@ informs retries. The system grows more capable **as data**, not as code. - **Reversible-low posture:** keep gating restarts/syncs in v1 (chosen), or auto-run them once the classifier is trusted? - **Confirmation phrase:** per-action typed phrase for destructive, or a global - one? + one? (Assent covers non-destructive; destructive keeps the typed phrase.) +- **Assent detection:** rule/keyword match, or let the model judge assent (with + a re-confirm on ambiguity)? How strict — does "yeah do the restart but not the + upgrade" partially grant? - **Runbook authorship:** operator-authored only, or may Nomos propose new runbooks (subject to approval) from successful ad-hoc sequences? - **Blast-radius threshold:** should a large blast radius force approval even for