Operator-reported bug: on 'proceed with the rest' the agent re-proposed the
plan, duplicating it in the sidebar. Root cause was a three-bug chain, not
one bug:
1. Trigger — model empty-response on 'proceed' (approval vocabulary didn't
list 'proceed', so the agent wasn't sure it was approved and no-op'd).
2. Amplifier — chatWith emitted 'error' without 'done' on empty response
(agent.go:370). The frontend's onComplete saw !receivedDone and
misclassified the model failure as a network disconnect, calling
handleDisconnect -> resumeSession.
3. Divergence — the reconnect note was generic ('report your state'), so
the agent re-proposed + re-executed instead of advancing the plan.
Fixes (shipped, e2e-validated against the live agent on oikos-nomos-1):
- A.2: proposePlan refuses re-proposal once a step has started (returns
errPlanInFlight). Drops the append-mode safety net (commit 5384499) that
was the direct source of the sidebar duplication. The agent must advance
with update_plan_step + run; the tool result directs it.
- A.1: proposePlan sets the 'generation' column on INSERT (migration 020
added the column + frontend grouping, but the INSERT never wired it).
- A.3: propose_plan tool description restated as a crisp contract (ONCE,
STOP and wait, REFUSES once a step started, advance with update_plan_step).
- F.3: approval vocabulary expanded to approved/yes/go/proceed/continue/ok/
go ahead; propose_plan result string tightened to an imperative.
- B.1: chatWith emits 'done' after 'error' on every terminal path via a new
emitError helper. The frontend now treats model errors as ended (not
disconnected), so no auto-reconnect -> resumeSession fires.
- B.2: reconnect/resume note carries the operator's last message + an
explicit 'advance the plan, do NOT call propose_plan again' directive when
a plan is in flight. Wired into all 4 resume entry points (reconnect,
/resume, idle-sweep, question-answer) via enrichResumeNote.
- B.3: resumeSession escalates the recovery note across its 3 attempts (final
retry: 'pick the lowest-pending step, mark it running, call run — do that
now') instead of 3 identical notes -> 3 identical empties.
Verification: TestProposePlan_RefuseInFlight replaces TestProposePlan_
AppendVsReplace. e2e conversations against the rebuilt container:
conv2 ('proceed with the rest') -> 0 propose_plan calls, plan stayed at
3 steps (was 6+ before), update_plan_step x5 + run x2 + complete_task.
conv3 (full plan, 'go ahead') -> apt-get update on lxc:dns auto-ran under
the plan window, update_entity_attributes writeback, clean complete_task.
nomos logs show zero reconnect/resume entries for the plan-proposing
sessions (the three-bug chain is closed).
Remaining (not in this commit): D.1 refuse complete_task without writeback
(next blocker), C.1/C.2, F.1/F.2 SOUL.md consolidation, B.4-B.6, E.1/E.2.
See plans/2026-07-14-post-fix-session-remainders.md.
Also: re-audit 2026-07-10-general-gated-execution.md — request_execution enum
retirement (60effcb) closes item 9; only auto-act revival (item 10) remains.
Version 0.4.1 -> 0.5.0 (minor: new structural behavior, not a bugfix).
288 lines
16 KiB
Markdown
288 lines
16 KiB
Markdown
# 2026-07-10 — General gated execution: from fixed actions to unlimited-but-gated
|
||
|
||
**Status:** In Progress — re-audited 2026-07-14. Done: `ClassifyCommand` risk
|
||
classifier, general `run` MCP tool, chat-assent approval (no button
|
||
required), blast radius on approval cards, session digest, global activity
|
||
feed (`Ops.svelte` "Executions" tab, risk-badged), Learning view
|
||
(success-rate trend), **and now the `request_execution` enum retirement**
|
||
(commit `60effcb`, 2026-07-14 — `run` is the only mutation tool; the legacy
|
||
handler functions are kept as reference only, with a "DO NOT re-register"
|
||
guard in `internal/mcp/server.go:360`). Still open: **revive auto-act** —
|
||
`internal/actuator/actuator.go:~125` is still a literal
|
||
`{"success": true, "message": "stub execution"}` stub (item 10). The
|
||
`run`-gated path covers operator-initiated work end-to-end; auto-act is the
|
||
observe→Act direction (signals triggering actions), still unimplemented.
|
||
|
||
## Goal
|
||
|
||
Make Nomos able to do **anything** needed to maintain the homelab — provision
|
||
LXCs, deploy services, debug, restart, fix configs, investigate — without that
|
||
capability being a fixed enum of hand-coded actions. The action space is
|
||
unlimited; the *gate* on it is a risk classifier + operator approval, not a
|
||
whitelist of tricks. Knowledge (the graph + runbooks) supplies the *how*; the
|
||
agent's reasoning supplies the *what*; the classifier supplies the *may I*.
|
||
|
||
Operator directive (2026-07-10): **"The number of actions the agent should be
|
||
able to do is unlimited. We need logic to gate destructive actions, but we
|
||
should not limit what the agent can do."**
|
||
|
||
Chosen autonomy posture for v1: **approve-most (cautious)** — only genuinely
|
||
read-only commands auto-run; anything that changes state requires operator
|
||
approval. We can relax later once the classifier and ledger have earned trust.
|
||
|
||
Operator directive #2 (2026-07-10): **"I want to see the system come alive and
|
||
learn and get better."** Observability is a first-class deliverable, not a
|
||
side-effect. As a user I must be able to see, in real time: what is being
|
||
executed, on what, and why; how it was classified and routed; what the outcome
|
||
was; and — crucially — **what knowledge the session created** (new runbooks,
|
||
patterns, resolved signals, ledger entries) so the system's growth is visible.
|
||
|
||
Operator directive #3 (2026-07-10): **approval is granted by chat assent, not a
|
||
button.** When Nomos proposes a plan/action and the operator replies "go ahead"
|
||
/ "yes" / "do it" in the chat, that assent *is* the approval. No separate
|
||
Approve button for the normal case. (Destructive actions still require an
|
||
explicit typed confirmation phrase — see Safety.)
|
||
|
||
## This is a realignment, not a new idea
|
||
|
||
[.agents/OIKOS.md](../.agents/OIKOS.md) already specifies this exact model:
|
||
|
||
> The classifier scores **risk class × blast radius × confidence** and routes:
|
||
> auto-act / escalate / queue. The classifier can only *lower* autonomy relative
|
||
> to policy, never raise it. When in doubt, escalate.
|
||
> Act — execute through `homelab` commands or **runbooks** (never ad-hoc SSH).
|
||
|
||
So the target architecture is the *documented* architecture. The problem is the
|
||
implementation diverged from it on the agent's action path.
|
||
|
||
## Gap analysis (grounded in code)
|
||
|
||
| Designed (OIKOS.md) | Actually implemented today |
|
||
|---|---|
|
||
| Classifier routes every action by risk × blast × confidence | [`internal/policy/classify.go`](../internal/policy/classify.go) `ClassifySignal()` only classifies **Signals** (the Observe pipeline), by entity+action-type. It is **not** called by the agent's mutation path. |
|
||
| Agent acts through unlimited **runbooks** | Agent acts through `request_execution` with a **hard-coded enum**: `restart, systemctl, pct_exec, apt_upgrade, pct_create` ([`internal/mcp/server.go`](../internal/mcp/server.go)), each bespoke Go, gated by per-action `if`s, not the classifier. New capability = new Go + redeploy. |
|
||
| Runbooks/skills are executable data | `skills` table + `.agents/skills/*` + `knowledge_entities` exist and are *readable* (`get_skills`), but **nothing executes a runbook**. The knowledge is inert w.r.t. action. |
|
||
| Auto-act loop consumes classified signals and acts | [`internal/actuator/actuator.go:124`](../internal/actuator/actuator.go) is a literal `"stub execution"` — it marks work done without doing it. |
|
||
| "Never ad-hoc SSH" | `pct_exec` **is** ad-hoc SSH (arbitrary shell in a container) and **auto-runs with no approval or classification**. |
|
||
|
||
Net: the elegant model exists as scaffolding (classifier, policy schema, risk
|
||
classes, blast-radius graph walks, skills-as-data, ledger, and the
|
||
approval+feedback plumbing hardened in the 2026-07-09/10 sessions), but the live
|
||
agent→action path is a bag of tricks that bypasses all of it. Everything added
|
||
in the recent LXC-deploy work (`pct_create` + DNS/VMID/template logic) made the
|
||
bag *bigger* — reliable, but on the wrong axis.
|
||
|
||
**Bones that already exist and get reused:** `internal/policy` (classifier +
|
||
`computeBlastRadius`/`blast_radius()` SQL), `risk_classes`/`action_risk` tables,
|
||
`seeds/policy.yaml`, `executions`/`approvals`/`audit_log`, the MCP SSH machinery,
|
||
and the inline approval + execution-status feedback loop (chat polls
|
||
`GET /executions/{id}`).
|
||
|
||
## Target architecture — three layers
|
||
|
||
### Layer 0 — one general gated primitive (the foundation)
|
||
|
||
Collapse the fixed enum into essentially one tool:
|
||
|
||
```
|
||
run(target, command, purpose, [declared_risk])
|
||
```
|
||
|
||
- `target` — any host or LXC slug; resolves to SSH (host) or `pct exec` (LXC).
|
||
- `command` — arbitrary shell.
|
||
- `purpose` — the agent's stated intent (shown to the operator, feeds classify).
|
||
- `declared_risk` — optional agent self-assessment.
|
||
|
||
Every call flows through:
|
||
|
||
1. **Classify** the command → `read_only | reversible_low | config_mutation |
|
||
destructive`. Rule-based:
|
||
- read-only **allowlist** (e.g. leading verb in `cat, ls, stat, journalctl,
|
||
systemctl status|is-active, pct config|status, df, free, uptime, ip, ss,
|
||
docker ps|logs, git status|log`) → `read_only`;
|
||
- destructive **denylist** (`rm -rf`, `dd`, `mkfs`, `wipefs`, `pct destroy`,
|
||
`qm destroy`, `shutdown`, `reboot`, `> /dev/`, `:(){ :|:& };:`, secret
|
||
exfiltration, piping remote scripts to a root shell) → `destructive`;
|
||
- anything writing state / installing / editing configs → `config_mutation`;
|
||
- **default → escalate** (`config_mutation`) when unsure.
|
||
The classifier may only make `declared_risk` **stricter**, never looser
|
||
(mirrors "can only lower autonomy, never raise").
|
||
2. **Route** (approve-most posture):
|
||
- `read_only` → auto-run + ledger, no approval.
|
||
- `reversible_low` / `config_mutation` → **operator approval via chat assent**
|
||
(v1 gates all state changes; a later posture can auto-run `reversible_low`).
|
||
- `destructive` → approval **+ typed confirmation phrase**.
|
||
3. **Execute** (existing SSH/`pct exec`), **verify** (optional check command),
|
||
**ledger** (`executions` + `audit_log`), **stream feedback to chat** (reuse
|
||
the `GET /executions/{id}` polling + `InlineApproval` phases already built).
|
||
|
||
### Approval by chat assent (replaces the Approve button)
|
||
|
||
The operator is already authenticated in the chat session, so their words are
|
||
the authorization — a separate button is redundant friction. Flow:
|
||
|
||
- Nomos proposes an action/plan; the gated `run` calls sit in `pending_approval`
|
||
(created in the same turn, tied to that turn's `correlation_id`).
|
||
- The operator's next message is checked for **assent** ("go ahead", "yes",
|
||
"do it", "proceed", "ship it") scoped to *that* proposal. On assent, the
|
||
pending approvals from that turn are granted and execute.
|
||
- Mechanism: Nomos detects assent and calls an `approve_pending(correlation_id)`
|
||
action; the backend flips the linked approvals → the existing
|
||
`executeApprovedAction` path runs. The **grant is recorded with the exact
|
||
operator message** that constituted assent (audit).
|
||
- Guards: assent only applies to approvals from the immediately-preceding turn
|
||
(no stale "yes" approving something old); ambiguous replies ("maybe",
|
||
"later", a follow-up question) do **not** grant — Nomos re-confirms;
|
||
**destructive** actions ignore loose assent and still require the typed
|
||
confirmation phrase.
|
||
- The inline UI still *shows* the pending action and its classification (so the
|
||
operator sees what they're assenting to) and reflects the grant — but the
|
||
primary path is "say yes," with the button demoted to an optional affordance.
|
||
|
||
Layer 0 alone delivers "the agent can attempt anything; state changes are gated."
|
||
|
||
### Layer 1 — runbooks as executable data (reliability without rigidity)
|
||
|
||
The hard-won procedures become **runbooks in the knowledge DB**, retrieved and
|
||
executed step-by-step via Layer 0 — not frozen Go:
|
||
|
||
- `pct_create` + its DNS-self-heal / VMID-collision / template-resolution /
|
||
locale logic becomes the canonical **"provision LXC" runbook** (parametric
|
||
steps the agent fills in and runs through `run`). The reliability survives as
|
||
documented, reusable steps rather than a compiled handler.
|
||
- New capability = **new runbook (data)**, no redeploy.
|
||
- Keep a *small* set of mechanical helpers where a shell step is genuinely
|
||
fiddly (e.g. "pick a free cluster VMID"), exposed as callable sub-tools — but
|
||
the flow is agent-driven, not enum-driven.
|
||
|
||
This is the crucial **both/and**: the general primitive is the unlimited escape
|
||
hatch; curated runbooks are the reliable fast-path so the agent doesn't
|
||
re-derive DNS/VMID/docker every time (the exact thing that failed repeatedly in
|
||
the 2026-07-09 sessions).
|
||
|
||
### Layer 2 — learning closes the loop
|
||
|
||
Successful ad-hoc `run` sequences get promoted into runbooks/patterns (the
|
||
`learning` engine + `skills` table already exist for this); the failure ledger
|
||
informs retries. The system grows more capable **as data**, not as code.
|
||
|
||
## Layer 3 — Observability: watch the system come alive
|
||
|
||
The user must *see* the OODA loop working, not just trust it. Four surfaces,
|
||
built on data the loop already produces (`executions`, `audit_log`, `signals`,
|
||
`skills`, `knowledge_entities`) — the job is to make it visible, live, and
|
||
legible, not to invent new telemetry.
|
||
|
||
**1. Live action feed (in the chat turn).** Every `run` renders a card as it
|
||
happens: `target` · `purpose` · **risk badge** (green read-only / amber
|
||
config / red destructive) · status (queued → running → ok/failed) · collapsible
|
||
output. Streams in real time (SSE, extend the existing execution-status feed).
|
||
The operator watches Nomos *work*, step by step, with the reasoning (`purpose`)
|
||
and the classifier's verdict on every step.
|
||
|
||
**2. "What this session did" digest.** At the end of a task/turn, a summary
|
||
card: N commands (X auto / Y assented / Z denied), entities changed (linked),
|
||
signals resolved, and **knowledge created** — new/updated runbooks, patterns
|
||
promoted, notes written — each linked to its record. This is the "what did the
|
||
agent actually change and learn" answer in one glance.
|
||
|
||
**3. The learning view — "the system is getting better."** A dedicated page:
|
||
runbooks and their **success-rate trend**, newly promoted skills, pattern
|
||
confidence (Wilson bounds already computed by the learning engine), recent
|
||
auto-acts that succeeded unattended, and a **capability timeline** ("2026-07-11:
|
||
learned to deploy Compose stacks; success 4/4"). Growth made tangible.
|
||
|
||
**4. Global activity/ledger stream.** A live feed of every action across the
|
||
fleet — command, target, classification, decision (auto / assented-by-whom),
|
||
outcome — the audit log rendered as a heartbeat. Filterable by entity, risk,
|
||
outcome.
|
||
|
||
These reuse existing tables; the work is API endpoints + SSE fan-out + Svelte
|
||
views, plus writing knowledge-creation events into the ledger so the digest has
|
||
something to show.
|
||
|
||
## Safety model (the whole point of the gate)
|
||
|
||
- **Default-escalate.** Nothing is *forbidden*; risky things need the operator's
|
||
"yes." Unknown/unparseable risk → approval.
|
||
- **Hard denylist** for catastrophic patterns → always typed confirmation, even
|
||
if the agent declared them safe.
|
||
- **Blast radius at approval time** — graph walk (`blast_radius()` exists):
|
||
"this restarts caddy → 8 downstream services."
|
||
- **Preview / dry-run** where the command supports it.
|
||
- **Kill-switch** (`global.auto_act`, per-target `never_auto_act.*`) already
|
||
exists; extend to a global "require approval for everything" flip.
|
||
- **Full audit ledger** — every command, its classification, decision, actor,
|
||
output. Non-negotiable.
|
||
- **Scope guards** — resolve `target` to a real entity first; refuse commands
|
||
against `destroyed`/unknown targets; cap output size (already done).
|
||
|
||
## Honest risks / tradeoffs
|
||
|
||
- Trades a small vetted surface (5 actions) for arbitrary root across the fleet,
|
||
LLM-driven, gated only by classifier + approval. Classifying arbitrary shell
|
||
perfectly is impossible; **default-escalate + hard denylist + always-on audit**
|
||
is the mitigation, not perfect classification.
|
||
- Approve-most means more operator clicks initially. Acceptable while trust is
|
||
built; the posture is a config knob, not a rewrite.
|
||
- Runbook-as-data can drift from reality like any doc; the ledger + verify step
|
||
+ learning loop are the correction mechanism.
|
||
|
||
## Migration path (incremental, each step shippable)
|
||
|
||
1. **Command classifier** — extend `internal/policy` with
|
||
`ClassifyCommand(cmd, declaredRisk) → riskClass` (allowlist/denylist/default-
|
||
escalate + can-only-escalate rule). Unit-tested against a corpus of safe /
|
||
mutating / catastrophic commands.
|
||
2. **`run` tool** — new MCP tool routing classify → gate → execute → the
|
||
existing feedback path. Ship alongside the current tools (no removal yet).
|
||
3. **Live action feed (UI)** — render each `run` as a streaming card in chat:
|
||
purpose, target, risk badge, status, output. This is the first "come alive"
|
||
win and validates the SSE fan-out.
|
||
4. **Chat-assent approval** — assent detection scoped to the last turn's
|
||
`correlation_id` → `approve_pending`; grant records the operator's message;
|
||
destructive still needs the typed phrase. Demote the Approve button.
|
||
5. **Approval context** — surface risk class + blast radius + purpose inline so
|
||
the operator sees what they're assenting to.
|
||
6. **Session digest + activity stream (UI)** — "what this session did / created"
|
||
card and the global ledger feed; write knowledge-creation events to the
|
||
ledger so there's something to show.
|
||
7. **Runbook execution** — a "provision LXC" runbook (ports the current
|
||
`pct_create` logic) executed via `run`; validate parity with today's handler.
|
||
8. **Learning view (UI)** — runbook success-rate trends, promoted skills,
|
||
capability timeline.
|
||
9. **Retire the enum** — convert remaining hard-coded actions to runbooks; make
|
||
`request_execution` a thin deprecated alias or remove it.
|
||
10. **Revive auto-act** — replace the actuator stub, reusing the *same*
|
||
classifier for the Observe→Act direction (signals), still approve-most.
|
||
|
||
## Verification
|
||
|
||
- Classifier corpus test: read-only commands auto-pass; a set of known
|
||
catastrophic commands always route to destructive+confirmation; ambiguous
|
||
commands escalate. No command auto-runs that mutates state.
|
||
- End-to-end: operator asks Nomos a novel task **not** in the old enum (e.g.
|
||
"tail caddy's error log and restart it if it's flapping"); Nomos composes
|
||
`run` calls; read-only steps auto-run and **stream as live cards**; the restart
|
||
gates; the operator types "go ahead" and the restart executes (no button);
|
||
ledger records each command + classification + the assent message.
|
||
- Observability: the session ends with a digest listing what ran, what changed,
|
||
and any knowledge created; the learning view shows the run's contribution.
|
||
- Parity: "provision an LXC with a service" via the runbook path matches the
|
||
reliability proven for the `pct_create` handler (free VMID, DNS, install,
|
||
verify), then destroy.
|
||
|
||
## Open questions for the operator
|
||
|
||
- **Reversible-low posture:** keep gating restarts/syncs in v1 (chosen), or
|
||
auto-run them once the classifier is trusted?
|
||
- **Confirmation phrase:** per-action typed phrase for destructive, or a global
|
||
one? (Assent covers non-destructive; destructive keeps the typed phrase.)
|
||
- **Assent detection:** rule/keyword match, or let the model judge assent (with
|
||
a re-confirm on ambiguity)? How strict — does "yeah do the restart but not the
|
||
upgrade" partially grant?
|
||
- **Runbook authorship:** operator-authored only, or may Nomos propose new
|
||
runbooks (subject to approval) from successful ad-hoc sequences?
|
||
- **Blast-radius threshold:** should a large blast radius force approval even for
|
||
otherwise-reversible actions?
|