feat(agent): plan-first gate, iterative follow-ups, reasoning persistence
P1 plan-first: run handler refuses without propose_plan (structural gate,
not SOUL.md prose). Plan window decoupled from set_goal — config_mutation
auto-run only on operator approval (assent window). Closes the approval-free
config_mutation hole confirmed in session d0d562e0.
P2 iteration: reopenSession flips terminal→executing, marks prior plan steps
replaced, clears outcome. proposePlan excludes replaced from in-flight check,
bumps generation. A follow-up on a completed session starts a new sub-task
with a fresh plan — no more errPlanInFlight dead end.
P3 reasoning: accumulate per-iteration text into the persisted row instead
of overwriting with the last text event. Reload shows intermediate thinking,
not just the final summary.
P4 read-only allowlist: add find, tree, locate, systemctl list-timers/
list-unit-files/show, timedatectl, hostnamectl, systemd-analyze, rclone
ls/lsl/md5sum/check/cryptcheck. Fixes the find misclassification from
d0d562e0.
P5 eval harness: new assertion kinds (proposes_plan, plan_before_run,
plan_generations), multi-turn followups, fetch /sessions/{id}/plan. Four
manifests under evals/.
P6 SOUL.md: strip degenerate-case carve-out, add ITERATE step, update
set_goal guidance.
VERSION 0.6.0 → 0.7.0
This commit is contained in:
16
evals/iteration-followup.yaml
Normal file
16
evals/iteration-followup.yaml
Normal file
@@ -0,0 +1,16 @@
|
||||
# P5 eval: iteration. A read-only task completes; the follow-up asks the
|
||||
# agent to act on what it found (a config_mutation). Asserts the session
|
||||
# reopens, a second plan generation is created, and the agent completes
|
||||
# both sub-tasks without duplicate-complete.
|
||||
- name: iteration-followup
|
||||
prompt: "When was the last backup to Proton Drive done, and when is the next one?"
|
||||
followups:
|
||||
- "The repos folder failed last time. Reset the failed service and re-run the backup."
|
||||
assertions:
|
||||
- kind: completes
|
||||
- kind: plan_generations
|
||||
value: 2
|
||||
- kind: proposes_plan
|
||||
- kind: writes_back
|
||||
- kind: max_run_calls
|
||||
value: 6
|
||||
13
evals/iteration-readonly.yaml
Normal file
13
evals/iteration-readonly.yaml
Normal file
@@ -0,0 +1,13 @@
|
||||
# P5 eval: two read-only sub-tasks back-to-back. Asserts the session
|
||||
# reopens and a second plan generation is created for the follow-up.
|
||||
- name: iteration-readonly
|
||||
prompt: "When was the last backup to Proton Drive done, and when is the next one?"
|
||||
followups:
|
||||
- "Now check when the last snapshot of the prometheus LXC was taken."
|
||||
assertions:
|
||||
- kind: completes
|
||||
- kind: plan_generations
|
||||
value: 2
|
||||
- kind: proposes_plan
|
||||
- kind: max_run_calls
|
||||
value: 6
|
||||
10
evals/no-plan-no-run.yaml
Normal file
10
evals/no-plan-no-run.yaml
Normal file
@@ -0,0 +1,10 @@
|
||||
# P5 eval: a pure-DB Q&A that calls NO run. This is the ONLY remaining
|
||||
# carve-out from plan-first: a task that never touches a live target via
|
||||
# `run` doesn't need propose_plan (the gate only fires on run). Asserts
|
||||
# the agent answers directly and completes without ceremony.
|
||||
- name: no-plan-no-run
|
||||
prompt: "List all LXC containers and their current health."
|
||||
assertions:
|
||||
- kind: completes
|
||||
- kind: no_run
|
||||
- kind: no_propose_plan
|
||||
13
evals/plan-always-readonly.yaml
Normal file
13
evals/plan-always-readonly.yaml
Normal file
@@ -0,0 +1,13 @@
|
||||
# P5 eval: a read-only question that would have been a "degenerate case"
|
||||
# under the old SOUL.md carve-out. Asserts the plan-first gate works: the
|
||||
# agent must propose_plan before run, even for a trivial read-only task.
|
||||
# Run against a live nomos with a real rclone LXC.
|
||||
- name: plan-always-readonly
|
||||
prompt: "When was the last backup to Proton Drive done, and when is the next one?"
|
||||
assertions:
|
||||
- kind: completes
|
||||
- kind: proposes_plan
|
||||
- kind: plan_before_run
|
||||
- kind: writes_back
|
||||
- kind: max_run_calls
|
||||
value: 3
|
||||
Reference in New Issue
Block a user