plan: add observability layer and chat-assent approval to gated-execution plan
Operator directives: (1) the UI must show what's executing, its risk
classification, live status, and what knowledge the session created — the
system's growth should be visible, not just trusted. (2) approval should be
granted by chat assent ("go ahead"), not a separate button; destructive
actions still require a typed confirmation phrase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -19,6 +19,19 @@ Chosen autonomy posture for v1: **approve-most (cautious)** — only genuinely
|
||||
read-only commands auto-run; anything that changes state requires operator
|
||||
approval. We can relax later once the classifier and ledger have earned trust.
|
||||
|
||||
Operator directive #2 (2026-07-10): **"I want to see the system come alive and
|
||||
learn and get better."** Observability is a first-class deliverable, not a
|
||||
side-effect. As a user I must be able to see, in real time: what is being
|
||||
executed, on what, and why; how it was classified and routed; what the outcome
|
||||
was; and — crucially — **what knowledge the session created** (new runbooks,
|
||||
patterns, resolved signals, ledger entries) so the system's growth is visible.
|
||||
|
||||
Operator directive #3 (2026-07-10): **approval is granted by chat assent, not a
|
||||
button.** When Nomos proposes a plan/action and the operator replies "go ahead"
|
||||
/ "yes" / "do it" in the chat, that assent *is* the approval. No separate
|
||||
Approve button for the normal case. (Destructive actions still require an
|
||||
explicit typed confirmation phrase — see Safety.)
|
||||
|
||||
## This is a realignment, not a new idea
|
||||
|
||||
[.agents/OIKOS.md](../.agents/OIKOS.md) already specifies this exact model:
|
||||
@@ -85,13 +98,36 @@ Every call flows through:
|
||||
(mirrors "can only lower autonomy, never raise").
|
||||
2. **Route** (approve-most posture):
|
||||
- `read_only` → auto-run + ledger, no approval.
|
||||
- `reversible_low` / `config_mutation` → **operator approval** (v1 gates all
|
||||
state changes; a later posture can auto-run `reversible_low`).
|
||||
- `reversible_low` / `config_mutation` → **operator approval via chat assent**
|
||||
(v1 gates all state changes; a later posture can auto-run `reversible_low`).
|
||||
- `destructive` → approval **+ typed confirmation phrase**.
|
||||
3. **Execute** (existing SSH/`pct exec`), **verify** (optional check command),
|
||||
**ledger** (`executions` + `audit_log`), **stream feedback to chat** (reuse
|
||||
the `GET /executions/{id}` polling + `InlineApproval` phases already built).
|
||||
|
||||
### Approval by chat assent (replaces the Approve button)
|
||||
|
||||
The operator is already authenticated in the chat session, so their words are
|
||||
the authorization — a separate button is redundant friction. Flow:
|
||||
|
||||
- Nomos proposes an action/plan; the gated `run` calls sit in `pending_approval`
|
||||
(created in the same turn, tied to that turn's `correlation_id`).
|
||||
- The operator's next message is checked for **assent** ("go ahead", "yes",
|
||||
"do it", "proceed", "ship it") scoped to *that* proposal. On assent, the
|
||||
pending approvals from that turn are granted and execute.
|
||||
- Mechanism: Nomos detects assent and calls an `approve_pending(correlation_id)`
|
||||
action; the backend flips the linked approvals → the existing
|
||||
`executeApprovedAction` path runs. The **grant is recorded with the exact
|
||||
operator message** that constituted assent (audit).
|
||||
- Guards: assent only applies to approvals from the immediately-preceding turn
|
||||
(no stale "yes" approving something old); ambiguous replies ("maybe",
|
||||
"later", a follow-up question) do **not** grant — Nomos re-confirms;
|
||||
**destructive** actions ignore loose assent and still require the typed
|
||||
confirmation phrase.
|
||||
- The inline UI still *shows* the pending action and its classification (so the
|
||||
operator sees what they're assenting to) and reflects the grant — but the
|
||||
primary path is "say yes," with the button demoted to an optional affordance.
|
||||
|
||||
Layer 0 alone delivers "the agent can attempt anything; state changes are gated."
|
||||
|
||||
### Layer 1 — runbooks as executable data (reliability without rigidity)
|
||||
@@ -119,6 +155,41 @@ Successful ad-hoc `run` sequences get promoted into runbooks/patterns (the
|
||||
`learning` engine + `skills` table already exist for this); the failure ledger
|
||||
informs retries. The system grows more capable **as data**, not as code.
|
||||
|
||||
## Layer 3 — Observability: watch the system come alive
|
||||
|
||||
The user must *see* the OODA loop working, not just trust it. Four surfaces,
|
||||
built on data the loop already produces (`executions`, `audit_log`, `signals`,
|
||||
`skills`, `knowledge_entities`) — the job is to make it visible, live, and
|
||||
legible, not to invent new telemetry.
|
||||
|
||||
**1. Live action feed (in the chat turn).** Every `run` renders a card as it
|
||||
happens: `target` · `purpose` · **risk badge** (green read-only / amber
|
||||
config / red destructive) · status (queued → running → ok/failed) · collapsible
|
||||
output. Streams in real time (SSE, extend the existing execution-status feed).
|
||||
The operator watches Nomos *work*, step by step, with the reasoning (`purpose`)
|
||||
and the classifier's verdict on every step.
|
||||
|
||||
**2. "What this session did" digest.** At the end of a task/turn, a summary
|
||||
card: N commands (X auto / Y assented / Z denied), entities changed (linked),
|
||||
signals resolved, and **knowledge created** — new/updated runbooks, patterns
|
||||
promoted, notes written — each linked to its record. This is the "what did the
|
||||
agent actually change and learn" answer in one glance.
|
||||
|
||||
**3. The learning view — "the system is getting better."** A dedicated page:
|
||||
runbooks and their **success-rate trend**, newly promoted skills, pattern
|
||||
confidence (Wilson bounds already computed by the learning engine), recent
|
||||
auto-acts that succeeded unattended, and a **capability timeline** ("2026-07-11:
|
||||
learned to deploy Compose stacks; success 4/4"). Growth made tangible.
|
||||
|
||||
**4. Global activity/ledger stream.** A live feed of every action across the
|
||||
fleet — command, target, classification, decision (auto / assented-by-whom),
|
||||
outcome — the audit log rendered as a heartbeat. Filterable by entity, risk,
|
||||
outcome.
|
||||
|
||||
These reuse existing tables; the work is API endpoints + SSE fan-out + Svelte
|
||||
views, plus writing knowledge-creation events into the ledger so the digest has
|
||||
something to show.
|
||||
|
||||
## Safety model (the whole point of the gate)
|
||||
|
||||
- **Default-escalate.** Nothing is *forbidden*; risky things need the operator's
|
||||
@@ -154,15 +225,25 @@ informs retries. The system grows more capable **as data**, not as code.
|
||||
mutating / catastrophic commands.
|
||||
2. **`run` tool** — new MCP tool routing classify → gate → execute → the
|
||||
existing feedback path. Ship alongside the current tools (no removal yet).
|
||||
3. **Approval context** — surface risk class + blast radius + purpose on the
|
||||
approval (chat `InlineApproval` + Ops page); typed-confirmation for
|
||||
destructive.
|
||||
4. **Runbook execution** — a "provision LXC" runbook (ports the current
|
||||
3. **Live action feed (UI)** — render each `run` as a streaming card in chat:
|
||||
purpose, target, risk badge, status, output. This is the first "come alive"
|
||||
win and validates the SSE fan-out.
|
||||
4. **Chat-assent approval** — assent detection scoped to the last turn's
|
||||
`correlation_id` → `approve_pending`; grant records the operator's message;
|
||||
destructive still needs the typed phrase. Demote the Approve button.
|
||||
5. **Approval context** — surface risk class + blast radius + purpose inline so
|
||||
the operator sees what they're assenting to.
|
||||
6. **Session digest + activity stream (UI)** — "what this session did / created"
|
||||
card and the global ledger feed; write knowledge-creation events to the
|
||||
ledger so there's something to show.
|
||||
7. **Runbook execution** — a "provision LXC" runbook (ports the current
|
||||
`pct_create` logic) executed via `run`; validate parity with today's handler.
|
||||
5. **Retire the enum** — convert remaining hard-coded actions to runbooks; make
|
||||
8. **Learning view (UI)** — runbook success-rate trends, promoted skills,
|
||||
capability timeline.
|
||||
9. **Retire the enum** — convert remaining hard-coded actions to runbooks; make
|
||||
`request_execution` a thin deprecated alias or remove it.
|
||||
6. **Revive auto-act** — replace the actuator stub, reusing the *same* classifier
|
||||
for the Observe→Act direction (signals), still approve-most.
|
||||
10. **Revive auto-act** — replace the actuator stub, reusing the *same*
|
||||
classifier for the Observe→Act direction (signals), still approve-most.
|
||||
|
||||
## Verification
|
||||
|
||||
@@ -171,8 +252,11 @@ informs retries. The system grows more capable **as data**, not as code.
|
||||
commands escalate. No command auto-runs that mutates state.
|
||||
- End-to-end: operator asks Nomos a novel task **not** in the old enum (e.g.
|
||||
"tail caddy's error log and restart it if it's flapping"); Nomos composes
|
||||
`run` calls; read-only steps auto-run, the restart gates for approval; chat
|
||||
shows live status; ledger records each command + classification.
|
||||
`run` calls; read-only steps auto-run and **stream as live cards**; the restart
|
||||
gates; the operator types "go ahead" and the restart executes (no button);
|
||||
ledger records each command + classification + the assent message.
|
||||
- Observability: the session ends with a digest listing what ran, what changed,
|
||||
and any knowledge created; the learning view shows the run's contribution.
|
||||
- Parity: "provision an LXC with a service" via the runbook path matches the
|
||||
reliability proven for the `pct_create` handler (free VMID, DNS, install,
|
||||
verify), then destroy.
|
||||
@@ -182,7 +266,10 @@ informs retries. The system grows more capable **as data**, not as code.
|
||||
- **Reversible-low posture:** keep gating restarts/syncs in v1 (chosen), or
|
||||
auto-run them once the classifier is trusted?
|
||||
- **Confirmation phrase:** per-action typed phrase for destructive, or a global
|
||||
one?
|
||||
one? (Assent covers non-destructive; destructive keeps the typed phrase.)
|
||||
- **Assent detection:** rule/keyword match, or let the model judge assent (with
|
||||
a re-confirm on ambiguity)? How strict — does "yeah do the restart but not the
|
||||
upgrade" partially grant?
|
||||
- **Runbook authorship:** operator-authored only, or may Nomos propose new
|
||||
runbooks (subject to approval) from successful ad-hoc sequences?
|
||||
- **Blast-radius threshold:** should a large blast radius force approval even for
|
||||
|
||||
Reference in New Issue
Block a user