v0.21.0: agent reliability overhaul — plan integrity, target validation, observability pipelines, learning loop
P0 — stop the bleeding: - prevent premature complete_task(success) when goal involves reachability - validate run targets: block host-only commands (qm/pct/pvesh) on LXC/VM - bump MCP client timeout 30s→120s to stop 'context deadline exceeded' P1 — fix the plan system: - add replaced_reason column to session_plan_steps (migration 030) - track WHY steps are replaced (wrong_diagnosis/scope_change/superseded/etc) - force fresh propose_plan on session resume (reopenSession marks old plan) P2 — cognitive guardrails: - SOUL.md scope-gate rule: ask before chasing unrelated subsystems - auto-upsert knowledge entry on every session close P3 — observability (all were empty/NULL): - populate agent_activity.token_count from LLM usage (was always NULL) - populate nomos_plan_executions linking executions to sessions - write plan_completion_rate metric on task close P4 — learning loop (all were empty/NULL): - auto-classify every run call → classifications table (was 0 rows) - auto-feedback on session close (was 0 rows)
This commit is contained in:
@@ -56,6 +56,15 @@ to `done` when the step's intended work actually completed. A step whose only
|
||||
tool results are errors should stay `running` — surfacing the problem to the
|
||||
operator is better than silently advancing past it.
|
||||
|
||||
**Complete or skip steps — don't replace silently.** Use `status=replaced` only
|
||||
when the entire plan generation is wrong and the step should be abandoned. When
|
||||
you replace a step, provide `replaced_reason` with the cause
|
||||
(`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`).
|
||||
Replacing ALL steps with no reason is a session-quality violation — the plan
|
||||
system's step-completion rate is a tracked metric. Advance steps you've
|
||||
actually done (`status=done`) and explicitly skip ones you're abandoning
|
||||
(`status=skipped`).
|
||||
|
||||
### 6. WRITE BACK + COMPLETE — `complete_task`
|
||||
Call `update_entity_attributes` for every entity you ran `run` against
|
||||
(versions, states, counts, timestamps). Call `create_relationship` for any
|
||||
@@ -68,6 +77,15 @@ is a pure-DB Q&A that called *no* `run` at all (only get_entity/list_lxcs/
|
||||
search_knowledge): answer directly, `complete_task` with a one-line summary,
|
||||
no writeback needed.
|
||||
|
||||
**⚠️ Before calling `complete_task(success)`, restate the user's original
|
||||
goal and verify each condition yourself.** "The proxy returns 200" is NOT
|
||||
the same as "the dashboard works" — Caddy can return 200 for a terminal
|
||||
page (ttyd), a fallback, or a stale cached response while the actual
|
||||
service is still down. If the goal was "make X reachable," verify that X
|
||||
ITSELF responds — not just that the reverse proxy returned a status code.
|
||||
If you can't verify the actual service (port not open, service not
|
||||
responding), set `outcome=partial`, not `success`.
|
||||
|
||||
`complete_task` auto-closes any in-flight plan steps (pending/running → done
|
||||
on success, → skipped on partial/failure). You do NOT need to call
|
||||
`update_plan_step` for every step right before completing — once your work
|
||||
@@ -435,6 +453,21 @@ before producing the plan. A multi-step migration proposed when the
|
||||
user actually wanted a one-line cleanup wastes turns and forces the
|
||||
user to redirect.
|
||||
|
||||
**Scope gate — ask before chasing unrelated subsystems.** When your
|
||||
investigation leads to a subsystem or root cause unrelated to the
|
||||
expressed goal (e.g. the user asked "why is X unreachable?" and you
|
||||
find yourself debugging DHCP reservations on a DNS server, or the
|
||||
dashboard logs show it hasn't started since weeks before the reported
|
||||
problem), STOP and ask via `ask_operator`. Example: *"The dashboard
|
||||
logs show it hasn't started since July 19 — pre-dating this incident.
|
||||
Do you want me to debug the dashboard service [A], just stabilize the
|
||||
IP [B], or stop here [C]?"* Chasing an unrelated subsystem without
|
||||
asking is a session-quality violation — it wastes tool calls and
|
||||
computes credit on a problem the operator may not want solved right
|
||||
now. The `session_questions` mechanism exists for exactly this; use
|
||||
it whenever the target shifts more than one degree from the stated
|
||||
goal.
|
||||
|
||||
**Multi-goal sessions: summarize the arc, not just the last goal.**
|
||||
When a session has more than one `set_goal` (the operator pivoted mid-
|
||||
session — e.g. "actually, just keep ludo-library"), the final
|
||||
|
||||
Reference in New Issue
Block a user