plan: add observability layer and chat-assent approval to gated-execution plan
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled

Operator directives: (1) the UI must show what's executing, its risk
classification, live status, and what knowledge the session created — the
system's growth should be visible, not just trusted. (2) approval should be
granted by chat assent ("go ahead"), not a separate button; destructive
actions still require a typed confirmation phrase.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-10 09:12:09 +02:00
parent 4a96f46e76
commit 9daf8220f2

View File

@@ -19,6 +19,19 @@ Chosen autonomy posture for v1: **approve-most (cautious)** — only genuinely
read-only commands auto-run; anything that changes state requires operator read-only commands auto-run; anything that changes state requires operator
approval. We can relax later once the classifier and ledger have earned trust. approval. We can relax later once the classifier and ledger have earned trust.
Operator directive #2 (2026-07-10): **"I want to see the system come alive and
learn and get better."** Observability is a first-class deliverable, not a
side-effect. As a user I must be able to see, in real time: what is being
executed, on what, and why; how it was classified and routed; what the outcome
was; and — crucially — **what knowledge the session created** (new runbooks,
patterns, resolved signals, ledger entries) so the system's growth is visible.
Operator directive #3 (2026-07-10): **approval is granted by chat assent, not a
button.** When Nomos proposes a plan/action and the operator replies "go ahead"
/ "yes" / "do it" in the chat, that assent *is* the approval. No separate
Approve button for the normal case. (Destructive actions still require an
explicit typed confirmation phrase — see Safety.)
## This is a realignment, not a new idea ## This is a realignment, not a new idea
[.agents/OIKOS.md](../.agents/OIKOS.md) already specifies this exact model: [.agents/OIKOS.md](../.agents/OIKOS.md) already specifies this exact model:
@@ -85,13 +98,36 @@ Every call flows through:
(mirrors "can only lower autonomy, never raise"). (mirrors "can only lower autonomy, never raise").
2. **Route** (approve-most posture): 2. **Route** (approve-most posture):
- `read_only` → auto-run + ledger, no approval. - `read_only` → auto-run + ledger, no approval.
- `reversible_low` / `config_mutation` → **operator approval** (v1 gates all - `reversible_low` / `config_mutation` → **operator approval via chat assent**
state changes; a later posture can auto-run `reversible_low`). (v1 gates all state changes; a later posture can auto-run `reversible_low`).
- `destructive` → approval **+ typed confirmation phrase**. - `destructive` → approval **+ typed confirmation phrase**.
3. **Execute** (existing SSH/`pct exec`), **verify** (optional check command), 3. **Execute** (existing SSH/`pct exec`), **verify** (optional check command),
**ledger** (`executions` + `audit_log`), **stream feedback to chat** (reuse **ledger** (`executions` + `audit_log`), **stream feedback to chat** (reuse
the `GET /executions/{id}` polling + `InlineApproval` phases already built). the `GET /executions/{id}` polling + `InlineApproval` phases already built).
### Approval by chat assent (replaces the Approve button)
The operator is already authenticated in the chat session, so their words are
the authorization — a separate button is redundant friction. Flow:
- Nomos proposes an action/plan; the gated `run` calls sit in `pending_approval`
(created in the same turn, tied to that turn's `correlation_id`).
- The operator's next message is checked for **assent** ("go ahead", "yes",
"do it", "proceed", "ship it") scoped to *that* proposal. On assent, the
pending approvals from that turn are granted and execute.
- Mechanism: Nomos detects assent and calls an `approve_pending(correlation_id)`
action; the backend flips the linked approvals → the existing
`executeApprovedAction` path runs. The **grant is recorded with the exact
operator message** that constituted assent (audit).
- Guards: assent only applies to approvals from the immediately-preceding turn
(no stale "yes" approving something old); ambiguous replies ("maybe",
"later", a follow-up question) do **not** grant — Nomos re-confirms;
**destructive** actions ignore loose assent and still require the typed
confirmation phrase.
- The inline UI still *shows* the pending action and its classification (so the
operator sees what they're assenting to) and reflects the grant — but the
primary path is "say yes," with the button demoted to an optional affordance.
Layer 0 alone delivers "the agent can attempt anything; state changes are gated." Layer 0 alone delivers "the agent can attempt anything; state changes are gated."
### Layer 1 — runbooks as executable data (reliability without rigidity) ### Layer 1 — runbooks as executable data (reliability without rigidity)
@@ -119,6 +155,41 @@ Successful ad-hoc `run` sequences get promoted into runbooks/patterns (the
`learning` engine + `skills` table already exist for this); the failure ledger `learning` engine + `skills` table already exist for this); the failure ledger
informs retries. The system grows more capable **as data**, not as code. informs retries. The system grows more capable **as data**, not as code.
## Layer 3 — Observability: watch the system come alive
The user must *see* the OODA loop working, not just trust it. Four surfaces,
built on data the loop already produces (`executions`, `audit_log`, `signals`,
`skills`, `knowledge_entities`) — the job is to make it visible, live, and
legible, not to invent new telemetry.
**1. Live action feed (in the chat turn).** Every `run` renders a card as it
happens: `target` · `purpose` · **risk badge** (green read-only / amber
config / red destructive) · status (queued → running → ok/failed) · collapsible
output. Streams in real time (SSE, extend the existing execution-status feed).
The operator watches Nomos *work*, step by step, with the reasoning (`purpose`)
and the classifier's verdict on every step.
**2. "What this session did" digest.** At the end of a task/turn, a summary
card: N commands (X auto / Y assented / Z denied), entities changed (linked),
signals resolved, and **knowledge created** — new/updated runbooks, patterns
promoted, notes written — each linked to its record. This is the "what did the
agent actually change and learn" answer in one glance.
**3. The learning view — "the system is getting better."** A dedicated page:
runbooks and their **success-rate trend**, newly promoted skills, pattern
confidence (Wilson bounds already computed by the learning engine), recent
auto-acts that succeeded unattended, and a **capability timeline** ("2026-07-11:
learned to deploy Compose stacks; success 4/4"). Growth made tangible.
**4. Global activity/ledger stream.** A live feed of every action across the
fleet — command, target, classification, decision (auto / assented-by-whom),
outcome — the audit log rendered as a heartbeat. Filterable by entity, risk,
outcome.
These reuse existing tables; the work is API endpoints + SSE fan-out + Svelte
views, plus writing knowledge-creation events into the ledger so the digest has
something to show.
## Safety model (the whole point of the gate) ## Safety model (the whole point of the gate)
- **Default-escalate.** Nothing is *forbidden*; risky things need the operator's - **Default-escalate.** Nothing is *forbidden*; risky things need the operator's
@@ -154,15 +225,25 @@ informs retries. The system grows more capable **as data**, not as code.
mutating / catastrophic commands. mutating / catastrophic commands.
2. **`run` tool** — new MCP tool routing classify → gate → execute → the 2. **`run` tool** — new MCP tool routing classify → gate → execute → the
existing feedback path. Ship alongside the current tools (no removal yet). existing feedback path. Ship alongside the current tools (no removal yet).
3. **Approval context** — surface risk class + blast radius + purpose on the 3. **Live action feed (UI)** — render each `run` as a streaming card in chat:
approval (chat `InlineApproval` + Ops page); typed-confirmation for purpose, target, risk badge, status, output. This is the first "come alive"
destructive. win and validates the SSE fan-out.
4. **Runbook execution** — a "provision LXC" runbook (ports the current 4. **Chat-assent approval** — assent detection scoped to the last turn's
`correlation_id` → `approve_pending`; grant records the operator's message;
destructive still needs the typed phrase. Demote the Approve button.
5. **Approval context** — surface risk class + blast radius + purpose inline so
the operator sees what they're assenting to.
6. **Session digest + activity stream (UI)** — "what this session did / created"
card and the global ledger feed; write knowledge-creation events to the
ledger so there's something to show.
7. **Runbook execution** — a "provision LXC" runbook (ports the current
`pct_create` logic) executed via `run`; validate parity with today's handler. `pct_create` logic) executed via `run`; validate parity with today's handler.
5. **Retire the enum** — convert remaining hard-coded actions to runbooks; make 8. **Learning view (UI)** — runbook success-rate trends, promoted skills,
capability timeline.
9. **Retire the enum** — convert remaining hard-coded actions to runbooks; make
`request_execution` a thin deprecated alias or remove it. `request_execution` a thin deprecated alias or remove it.
6. **Revive auto-act** — replace the actuator stub, reusing the *same* classifier 10. **Revive auto-act** — replace the actuator stub, reusing the *same*
for the Observe→Act direction (signals), still approve-most. classifier for the Observe→Act direction (signals), still approve-most.
## Verification ## Verification
@@ -171,8 +252,11 @@ informs retries. The system grows more capable **as data**, not as code.
commands escalate. No command auto-runs that mutates state. commands escalate. No command auto-runs that mutates state.
- End-to-end: operator asks Nomos a novel task **not** in the old enum (e.g. - End-to-end: operator asks Nomos a novel task **not** in the old enum (e.g.
"tail caddy's error log and restart it if it's flapping"); Nomos composes "tail caddy's error log and restart it if it's flapping"); Nomos composes
`run` calls; read-only steps auto-run, the restart gates for approval; chat `run` calls; read-only steps auto-run and **stream as live cards**; the restart
shows live status; ledger records each command + classification. gates; the operator types "go ahead" and the restart executes (no button);
ledger records each command + classification + the assent message.
- Observability: the session ends with a digest listing what ran, what changed,
and any knowledge created; the learning view shows the run's contribution.
- Parity: "provision an LXC with a service" via the runbook path matches the - Parity: "provision an LXC with a service" via the runbook path matches the
reliability proven for the `pct_create` handler (free VMID, DNS, install, reliability proven for the `pct_create` handler (free VMID, DNS, install,
verify), then destroy. verify), then destroy.
@@ -182,7 +266,10 @@ informs retries. The system grows more capable **as data**, not as code.
- **Reversible-low posture:** keep gating restarts/syncs in v1 (chosen), or - **Reversible-low posture:** keep gating restarts/syncs in v1 (chosen), or
auto-run them once the classifier is trusted? auto-run them once the classifier is trusted?
- **Confirmation phrase:** per-action typed phrase for destructive, or a global - **Confirmation phrase:** per-action typed phrase for destructive, or a global
one? one? (Assent covers non-destructive; destructive keeps the typed phrase.)
- **Assent detection:** rule/keyword match, or let the model judge assent (with
a re-confirm on ambiguity)? How strict — does "yeah do the restart but not the
upgrade" partially grant?
- **Runbook authorship:** operator-authored only, or may Nomos propose new - **Runbook authorship:** operator-authored only, or may Nomos propose new
runbooks (subject to approval) from successful ad-hoc sequences? runbooks (subject to approval) from successful ad-hoc sequences?
- **Blast-radius threshold:** should a large blast radius force approval even for - **Blast-radius threshold:** should a large blast radius force approval even for