The previous commit fixed a context-cancellation bug in the auto-approve path
and appeared to fix things, but re-testing end-to-end after deploy showed the
execution STILL never completed — just via a different symptom
("no pending execution found for approval" in the logs). Dug further and
found the real, deeper bug underneath: this whole mechanism has never
actually worked.
autoApprove() directly flipped BOTH approvals.status and executions.status to
'approved' via raw SQL, then called executeApprovedViaAPI to POST to the
decision endpoint. But DecideApproval's own logic specifically looks for the
execution still at status='pending_approval' to find and dispatch the real
SSH work (executeApprovedAction) — autoApprove's premature flip meant that
lookup always found zero rows. DecideApproval's UpdateApprovalStatus call
also silently no-ops the same way (sqlc :exec doesn't surface "0 rows
affected" as an error). Every assent-window auto-approved pct_create/
apt_upgrade has been sitting at 'approved' forever with the real work never
triggered — indistinguishable from "still running" until you check.
Fix: remove autoApprove() entirely. Call executeApprovedViaAPI directly
against the untouched pending_approval row from createApproval — identical
to the manual Approve-button path, just without the human click. DecideApproval
is now the single place that transitions status and dispatches, for both the
manual and auto-approved paths, closing the class of bug where two code paths
raced to do the same state transition.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live testing the new atomic pct_create: an assent-window
auto-approved pct_create appeared to "run" (logged "auto-approved... running
now") but the execution stayed stuck at 'approved' forever. Root cause:
`go executeApprovedViaAPI(ctx, ...)` passed the MCP tool-call's own context —
which is cancelled the instant the triggering /chat request's HTTP response
completes, i.e. on every normal turn. The spawned goroutine's POST to the
approval-decision endpoint died with "context canceled" before it could even
start the real work, and nothing surfaced this to the operator or the agent —
the execution just sat at 'approved' with no error, indistinguishable from
"still running."
This is exactly the context-lifetime bug class httpapi's own approval
goroutine (executeApprovedAction) already avoided by using
context.Background() — it had just been missed in these two call sites
(apt_upgrade and pct_create auto-approve). Fixed both to use
context.Background(), matching the correct pattern already in place
elsewhere. Audited for other goroutines spawned with a request-scoped ctx —
none found; the sshExec internal goroutines are synchronous/waited-on via
select and correctly scoped to the call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the two remaining open points from the auto-continuation work.
1. Atomic pct_create (observability, the bigger of the two):
pct_create used to bundle create + apt install + post_install script into
one black-box multi-minute SSH call — the agent got back a single opaque
success/fail with no way to see (or fix) which step actually broke.
Removed the whole post-create provisioning block (and the now-dead
provisionScript/sanitizePkgs helpers + their tests). pct_create is now
create + start + register ONLY — fast, and its result is fed back to the
agent via auto-continuation almost immediately. The agent installs
packages and runs setup as its OWN sequence of `run` calls against the new
lxc:<hostname>, observing each command's real output and able to diagnose
and retry exactly the step that failed — the same recovery loop already
proven for the general case, now applied to installs too, instead of
requiring a separate black-box mechanism.
- services/post_install removed from the pct_create params struct and
from the MCP tool schema/SOUL.md docs.
- SOUL.md: explains the new flow, moves the Docker CLI gotcha and DNS
troubleshooting guidance to be steps the agent runs itself.
2. Scoped destructive window (targeted autonomy for recovery):
Verified live in the previous session that a destructive recovery (a
failed destroy needing stop-then-destroy on the same container) required
TWO separate typed confirmations for what was clearly one recovery
action. Added a narrow, TARGET-scoped 15-minute grant
(destructive_window.agent:<id>.target:<slug> in autonomy_settings,
shared key format across cmd/nomos and internal/mcp) that opens only
after an EXPLICIT typed confirmation (never loose assent) or an explicit
button-approval of a destructive step, and only ever covers further
destructive commands against that SAME target. A different target always
needs its own fresh confirmation — this narrows risk instead of loosening
it globally, unlike broadening the general assent window to cover
destructive actions would have.
- cmd/nomos/store.go: openDestructiveWindow/destructiveWindowActive/
executionTarget.
- cmd/nomos/agent.go: opens the window when a typed confirmation grants a
destructive chat-assent execution.
- internal/mcp/server.go: `run` tool checks the window before gating a
destructive command; auto-runs if active.
- internal/httpapi/phase3.go: DecideApproval opens the same window when a
destructive execution is approved via the button/API, for parity with
the chat-assent path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live: the new auto-continuation worker (previous commit) worked end
to end for the happy path (provision -> auto-verify -> report success, zero
operator ticks). But testing a failure-recovery case (a destroy that failed
because the container was still running) surfaced a real bug: continueSession's
emit closure only captured "text" events, so when chatWith ended the turn on
an "error" event (LLM returned an empty/refusal response, internal retry also
empty), the worker persisted a completely blank, uninformative "auto" message
— no sign anything had gone wrong, undermining observability of the very
mechanism just built.
- Capture "error" events and, if the turn produced no text/tool_calls at all,
persist an explanatory placeholder instead of blank.
- Add one outer retry of the whole chatWith call when the first attempt
produces nothing — the principle behind this whole feature ("don't give up
on the first error") should apply to the continuation mechanism itself, not
just the homelab commands it's continuing.
Also verified live: recovery-from-failure works via the normal chat path once
prompted, and cleaned up the test container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three fixes for the session where the agent proposed a plan, waited
for 'proceed', then re-queued instead of being auto-approved:
1. Chat-assent fallback: when the operator says 'proceed' but the
preceding turn had NO pending approvals (agent proposed plan in text
without calling request_execution), inject a system note telling the
agent to execute the plan now. Opens the assent window so subsequent
config_mutation commands auto-run.
2. SOUL.md: instruct agent to ALWAYS call request_execution/run when
proposing a plan, not wait for 'proceed' first. This ensures a
pending approval exists for chat-assent to grant.
3. SOUL.md: stronger Docker instructions — Debian 13's docker.io package
installs the daemon but NOT the docker CLI binary. Must use
get.docker.com in post_install. Added 'handling errors' section:
diagnose, try alternatives, continue — don't stop after one failure.
When the operator has approved a plan via chat assent (assent window
active), pct_create and apt_upgrade now auto-approve and execute
instead of queuing for a separate approval round. The auto-approve
path updates the approval+execution status in the DB, then calls the
HTTP API's decision endpoint to trigger executeApprovedAction — same
code path as a manual Approve button, consistent audit trail.
Agent stopped after every approval step, forcing operator to type
'continue' 7× per deploy session. Root causes and fixes:
1. Compound read-only commands (e.g. 'systemctl status; journalctl')
defaulted to config_mutation — now splits on ;/&&/||/| and classifies
as read_only if all segments are inspection verbs. Added grep, wc,
sort, uniq, cut, tr, dpkg -l, apt list, docker stats to allowlist.
2. curl|sh was classified destructive, forcing typed confirmation for
legitimate installs (get.docker.com). Demoted to config_mutation —
loose assent grants it, no typed phrase needed.
3. SOUL.md said 'STOP after queuing' — replaced with 'continue working
on non-blocked steps'. Added assent window section instructing agent
to carry out the full plan after approval.
4. Assent window: when operator approves a plan via chat assent, a
30-minute window opens where config_mutation commands auto-run
without re-approval. Agent writes expiry to autonomy_settings; MCP
run tool checks it before gating. Destructive never auto-runs.
5. System note after approval now says 'CONTINUE executing the full
plan — do not stop and wait for continue.'
Verified live that after deploying the "fixed" bridge-bound pre-flight, it
still let a known-bad vmbr0+192.168.8.2 config straight through to a full
pct_create with no error. Root cause: the check used
`strings.Contains(pingOut, "REACHABLE")` against markers "REACHABLE" /
"UNREACHABLE" — but "UNREACHABLE" contains "REACHABLE" as a substring, so the
containment check was true for BOTH outcomes. The pre-flight was structurally
incapable of ever failing, regardless of the actual ping result.
Fixed with distinct, non-overlapping markers (PREFLIGHT_OK/PREFLIGHT_FAIL)
and exact-match comparison, pulled into a small gatewayPreflightPassed()
helper with a unit test asserting the exact historical bug case
("UNREACHABLE" must be false) so this bug class can't silently recur.
Re-verified live end-to-end: manually re-tested the exact ping command
(confirmed UNREACHABLE via vmbr0), and this was caught only by actually
running the check against production, not by reading the code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live that the pre-flight check added in the previous commit had a
real gap: a plain `ping <gateway>` from the Proxmox host succeeds via the
HOST's own routing table (which can have routes to a subnet through paths
the host alone knows about), even when the CONTAINER — attached via a plain
bridge with only a naive on-link default route — can never actually ARP that
gateway. Confirmed by creating a real test container on vmbr0 with
gw=192.168.8.2: the host-wide ping had said "reachable," but pinging from
inside the container showed 100% packet loss. Fixed by binding the pre-flight
ping to the specific requested bridge (`ping -I <bridge>`), which correctly
rejects vmbr0 for that gateway instead of false-positiving via the host's
broader routing table.
Also confirmed live: vmbr1 does exist and is up on strong (contrary to the
possibly-stale host doc), matching what romm/seanime's docs already said.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Investigated why the operator couldn't get past "no DNS/connectivity" across
multiple retries even after Nomos correctly diagnosed and fixed the gateway
(192.168.8.1 -> 192.168.8.2). It still failed. Root cause, confirmed from
strong's own documented network topology: on `strong`, vmbr0 physically
bridges only to 192.168.178.0/24 — the 192.168.8.0/24 service network is
reached via a Fritz!Box static route, not a local bridge. A container
attached to vmbr0 can never reach a 192.168.8.x gateway no matter which
address in that range is picked; ARP for it just gets silently dropped
(matching the earlier hang symptom). The gateway was never the problem — the
bridge was. 192.168.8.0/24 is also segmented into /28 blocks each with their
own gateway (192.168.8.2 is only the .0-.15 block's gateway), so even a
correct bridge with a copy-pasted gateway from a different block would still
fail.
No amount of retrying with a different gateway guess could have fixed this —
the missing fact (which bridge reaches which subnet, and the per-/28 gateway)
isn't inferable from the subnet alone.
- pct_create gets a `bridge` param (was hardcoded to vmbr0) so a correct
bridge can actually be requested once known.
- Fast pre-flight: for any static IP, ping the gateway from the target HOST
before creating anything. Was: a bad config took a multi-minute hang (or,
after last commit's timeout fix, ~2min) before failing. Now: ~2 seconds,
with a message that explicitly says not to guess a different gateway in
the same subnet — find a real neighbor's config or use DHCP.
- SOUL.md: DHCP is now framed as the default, not a fallback; static IP
requires finding an existing LXC on the same host in the same /28 and
copying its bridge+gateway verbatim — inventing one is explicitly called
out as the failure mode that caused this exact incident.
- MCP tool schema: pct_create's params description now documents `bridge`
and the neighbor-copy rule directly in what the model reads at call time,
not just in SOUL.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).
Two compounding bugs made this unrecoverable without manual intervention:
1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
NO execution timeout — `session.CombinedOutput()` blocks until the remote
command exits, with no deadline. A hung remote process blocks the Go
goroutine forever; the execution can never leave 'running', and the
operator has no way to make it stop. Fixed: both now race the SSH call
against a 10-minute hard timeout, closing the session/client and
returning a clear "timed out after 10m0s" error if exceeded. (The
mcp/server.go copy also still had the original "swallowed non-zero exit"
bug from before that fix was applied to httpapi's copy only — fixed here
too.)
2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
connectivity — it doesn't; a black-holed network can make each call hang
far past the resolver's nominal timeout, so the documented "~90s" budget
was never real. Wrapped every attempt in `timeout 3` so the wall-clock
budget is now actually enforced (~2min worst case), and the failure
message now suggests checking the net0 gateway.
Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):
- InlineApproval's track() polling loop had its own ~6min ceiling and simply
STOPPED polling after that — silently going stale before the backend (now
correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
with margin, and added a distinct 'stalled' state if that's ever exceeded
(explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
execution's created_at), the actual command being run, and the execution
ID — previously just a static "this can take a minute" with zero
information. Also added command display to the destructive pending-
approval card for full transparency before confirming.
Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "chat gave me no further feedback — had to go to Ops": two
compounding bugs, found by reading the actual production session transcript.
1. InlineApproval.svelte — all of last session's live-status/self-heal work —
was never imported or rendered anywhere. Chat.svelte had its own separate,
much dumber approval bar (no status tracking, no destructive handling, just
silently disappears after clicking) that WAS the one users actually saw.
Deleted the dead bar and its state; InlineApproval now renders per-message.
2. chat.ts's extractApprovals hardcoded `tool.name === 'request_execution'`,
so any approval raised by the newer `run` tool was invisible — no card, no
feedback, nothing to self-heal, forcing the operator to the Ops page with
zero acknowledgement in the conversation. This was the actual proximate
cause of last night's destroy-135 session. Fixed to match on response
shape, not tool name, so it doesn't silently break again for the next new
gated tool.
3. Nomos was telling operators "type something like 'I confirm destroy 135'"
for destructive actions (SOUL.md) but no backend path ever consumed that
phrase — chat-assent explicitly (and correctly) excludes destructive from
loose assent, but I never built the alternative. Added
isTypedConfirmation() (cmd/nomos/assent.go): stricter than loose assent,
requires an explicit "confirm" statement, only applies to destructive-
flagged pending approvals.
4. InlineApproval's completed-state hardcoded "Provisioned successfully" —
wrong/confusing for a destroy or arbitrary `run` command. Now says
"Completed on <target>" and shows the actual command output, verified live
against the real destroy-135 execution.
Verified live in a real browser against the production API/DB (dev server
proxying to :8090): the historical stuck session now retroactively renders
both executions as resolved with correct wording and real output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live testing hit `entities_slug_key` violations: exec slugs used an 8-char
prefix of a UUIDv7, whose leading bytes encode a millisecond timestamp — two
executions created seconds apart can share a prefix. Use the full UUID
(guaranteed unique) for the exec entity's slug/name in request_execution, the
new `run` tool, and the REST RequestExecution handler — all three had the
same pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found live: `run` against lxc:caddy failed with "missing pve_id" even though
pve_id=121 was present — caddy is an inventory-seeded LXC with no `host`
attribute at all (only pct_create-provisioned LXCs set one). The combined
query scanned attributes->>'host' (SQL NULL) into a plain Go string, which
errors the whole Scan — including the pve_id column that scanned fine.
COALESCE the host column to '' so a missing host attribute degrades to the
documented default instead of failing the whole resolution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the first slice of plans/2026-07-10-general-gated-execution.md:
Nomos gets one general execution tool instead of only a fixed action enum,
gated by an automatic risk classifier, and approval can be granted by the
operator just replying in chat instead of clicking a button.
- internal/policy/command.go: ClassifyCommand(cmd, declaredRisk) — rule-based
read-only allowlist + destructive denylist, default-escalate to
config_mutation for anything else. Classification can only ESCALATE the
caller's declared risk, never de-escalate it (destructive always wins even
if declared read_only). Compound commands (&&, ;, |, $()) never qualify for
the read-only fast path. Full test corpus.
- internal/mcp/server.go: new `run` MCP tool — target (host:/lxc:), command,
purpose, optional declared_risk. Read-only commands execute immediately;
everything else queues an approval exactly like pct_create today, executed
via httpapi's existing executeApprovedAction. Also fixes a real latent bug:
pct_exec resolved an LXC's host attribute without the "host:" prefix, so it
could never find the Proxmox host — new resolveExecTarget/resolveRunTarget
helpers (mcp + httpapi) fix this for both the new `run` action and existing
actions that route through the same execution path.
- internal/httpapi/phase3.go: "run" case in executeApprovedAction; fixes two
bugs found while wiring this up — (1) DecideApproval hardcoded risk_class to
'config_mutation' on every approve, silently corrupting the audit ledger for
every other risk class; (2) denying/revoking an approval never updated the
linked execution's status, so it stayed 'pending_approval' forever instead
of reflecting the decision.
- cmd/nomos/assent.go: deterministic (not LLM-judged) chat-assent detection.
Scoped to the immediately-preceding assistant turn's pending approvals only
— an old "yes" can't retroactively approve something new. Destructive-risk
actions are excluded from loose assent. Approves via the same HTTP decision
endpoint the UI button calls, so both paths share one audit trail.
- web/.../InlineApproval.svelte: self-healing poll — a pending approval card
now picks up being decided via ANY path (chat assent, Ops page, Matrix), not
just its own button. Previously the banner stayed stuck showing
Approve/Deny even after the action had already run elsewhere.
- nomos/SOUL.md: `run` is now the general capability ("no fixed menu, only a
risk gate"); documents chat-assent behavior and the destructive exception.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator directives: (1) the UI must show what's executing, its risk
classification, live status, and what knowledge the session created — the
system's growth should be visible, not just trusted. (2) approval should be
granted by chat assent ("go ahead"), not a separate button; destructive
actions still require a typed confirmation phrase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Evaluate the agent's action path against the OIKOS.md design. Finding: the
intended model (unlimited runbook-driven actions gated by a risk classifier)
already exists on paper and in scaffolding, but the live agent path regressed
to a hard-coded 5-action enum that bypasses the classifier. Plan a layered
realignment: (0) one general gated `run` primitive, (1) runbooks-as-data as the
reliable fast-path, (2) learning. Chosen v1 posture: approve-most (read-only
auto-runs, all state changes gate). Incremental, each step shippable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fresh Debian LXCs have no locale configured, spamming "apt-listchanges:
Can't set locale" / perl warnings across every install and breaking some
packages' post-install scripts. Pin LANG/LC_ALL=C.UTF-8 (and hoist
DEBIAN_FRONTEND) at the top of the in-container bootstrap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Production session provisioned the container but the service never installed:
apt failed with "Temporary failure resolving deb.debian.org" — a static-IP LXC
whose assigned nameserver couldn't resolve. The operator also got zero feedback:
the approval banner just sat there with no running/complete/failed status.
Backend robustness (provisionScript):
- Wait for real DNS/connectivity inside the container before apt, and self-heal
/etc/resolv.conf to a public resolver (1.1.1.1/8.8.8.8) if the assigned one
is dead. `set -e` after the gate so apt/post_install failures surface.
- apt-get update/install with Acquire::Retries=3.
Frontend feedback (InlineApproval):
- After approve, poll GET /executions/{id} and show live phase: submitting →
provisioning… → provisioned successfully / execution failed (with the error).
- add getExecution() to api.ts.
Agent guidance (SOUL.md):
- omit vmid (auto-assigned), prefer dhcp, docker-compose-plugin is not in Debian
(use docker.io + get.docker.com), end post_install with a health check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The final "UPDATE executions SET result=$::jsonb" built its payload with
fmt.Sprintf and only escaped newlines. apt/pct output contains quotes,
backslashes and control chars, so the payload was invalid JSON, the jsonb
cast failed, and the (unchecked) UPDATE was silently discarded — the
execution stayed 'approved' with a NULL result even though the LXC was fully
provisioned (verified live: vmid auto-assigned, container running, service
installed, post_install ran).
- executeApprovedAction: marshal result via json.Marshal; log UPDATE errors
- add jsonErr() helper; route all pct_create failure-path results through it
- mcp/server.go: add jsonOut() for restart/systemctl/pct_exec inline results
- regression test for JSON validity on quote/backslash/control-char output
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups found while verifying the approve→provision path end to end:
- vmid is now optional: the early required-field check rejected vmid:0
before the cluster VMID guard could auto-assign a free id. Only hostname
is required now; 0 (or a collision) resolves to `pvesh get /cluster/nextid`.
- net0: use ip=dhcp with no gateway when no static IP is given (Proxmox
rejects gw alongside dhcp); only attach gw for a static CIDR.
- bump post-create settle to 10s so a DHCP lease is up before apt runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real production failure when the operator clicked Approve in chat: nothing
provisioned, banner never cleared, execution marked completed.
Three root causes:
- sshExec swallowed non-zero exits when the command produced output, so a
`pct create` that printed "CT 132 already exists" and failed was reported
as success and a bogus lxc entity was registered. Now any non-zero exit
returns an error (with output) so the execution is correctly marked failed.
- The LLM reused VMID 132 (belongs to lxc:rclone; VMIDs are cluster-wide).
pct_create now checks in-use VMIDs via `pvesh get /cluster/resources` and
falls back to `pvesh get /cluster/nextid` when the requested id is taken.
- InlineApproval.svelte reset its state on every prop change (done was also
compared against the wrong string), so the banner never cleared and each
click re-POSTed /decision. Rewritten to track outcome per executionId,
clear on success, and block resubmits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "asks permission but never acts": the approved pct_create
execution failed to parse because the LLM emitted `"privileged":0` /
`"nesting":1` (numbers) into strict `bool` fields, so the container was
never created. Compounded by a hardcoded template name (debian-13.0-1)
that no longer exists on the host, and no way for the agent to read the web.
- flexBool: accept 0/1, "true", bool for privileged/nesting (the exact prod failure)
- pct_create template pre-flight: list host cache, validate/auto-pick newest debian
- pct_create services[] + post_install: one approval provisions a working service
- new http_get MCP tool (sanitized, size-capped, SSRF-guarded) — agent can read repos/sites
- request_execution description: target=host, full JSON schema + example
- SOUL.md: agent CAN fetch the web; prefer one-step provisioning
- default model deepseek-v4-flash -> v4-pro; maxIterations 15 -> 25
- unit tests for flexBool, template resolve, pkg sanitize, HTML sanitize + SSRF block
Verified live on host:strong with a throwaway VMID 999: template auto-resolved,
container created + booted, services installed, post_install ran, then destroyed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Execution entity name now includes UUID suffix: 'pct_create on host:strong (abc12345)'
so the (type,name) UNIQUE constraint doesn't block subsequent executions for the
same target+action. Dedup now uses JOIN + LIKE prefix match to find only
pending_approval executions.
- Move persistent approval bar from top of messages area to just above the
chat input box (bottom-fixed position, above the textarea form).
- Add dedup in request_execution: check entities(type,name) uniqueness before
creating duplicate executions. Returns 'already queued' message to the LLM,
preventing tool-calling loops.
- Fix createApproval JSON payload: use json.Marshal instead of fmt.Sprintf
to escape params (could contain unescaped double quotes from JSON config).
- Add ON CONFLICT DO NOTHING to entity/execution inserts for dedup race safety.
- Persistent approval bar at top of Chat.svelte: aggregates pendingApprovals
from all messages, fixed position (won't scroll away). Approve/deny/approve-all.
- Update SOUL.md: agent must STOP after queuing a gated action.
- Fix ToolCallGroup reactivity: wasActive = (active).
- Replace text-based regex parsing in InlineApproval with structured
pendingApprovals extracted from request_execution tool results. The tool
result text is deterministic (not LLM-generated), making UUID extraction
reliable regardless of how the LLM rephrases the response.
- Fix ToolCallGroup reactivity: wasActive = active captured initial
value. Now uses (active) so re-runs on prop changes.
- Extract approvals in both live streaming (done event) and history loading
for consistent behavior on resumed sessions.
- Empty/refusal responses retried once, then surfaced as errors instead of silent blanks
- Chinese refusal boilerplate detected via denylist + non-ASCII heuristic
- Bulk-tool preference added to SOUL.md (list_lxcs over per-entity get_lxc_state)
- Tool results truncated to 4KB on persist; get_state_snapshot filters null-state entities
- Session delete (DELETE /sessions/{id} + confirm-on-second-click UI)
- Session titles auto-generated from assistant answer instead of raw user message
Inspected the live agent_sessions/agent_messages tables on mac-mini and
found silent empty responses, a canned non-English refusal after 22 tool
calls, 70-call fan-out for simple fleet questions, 100KB+ persisted
messages, and no session delete/title hygiene.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two chat UX changes (share Chat.svelte, committed together).
Tool-call grouping (ToolCallGroup.svelte):
- Replaced the one-<details>-per-tool-call list with a single
collapsible group per assistant turn, headed by tool count + a
name preview and a status icon (spinning wrench in progress, check
done, X on error).
- The group auto-collapses the instant its turn finishes streaming, so
a completed round shows as one compact pill; historical/loaded turns
start collapsed. The auto-collapse fires once at the streaming→done
transition, leaving manual toggles alone afterward.
Session graph rail (SessionGraph.svelte) — replaces the old
ContextRail (fleet health / pending approvals / live events), which is
deleted:
- A force-directed graph that starts empty (animated constellation
empty state) and grows as the conversation references entities.
Slugs are extracted from message text and tool *arguments* only —
never bulk result rows, so a single get_health_summary doesn't dump
all 168 entities — then validated against the backend via fetchGraph
(cached) with check/execution probe entities excluded. Nodes are
colored by health; edges appear once both endpoints are present.
- Clicking a node highlights it and its neighbors and opens an inline
detail panel below: slug/type/state, health + freshness, top
attributes, in-graph relations (clickable to hop), and a Full detail
button opening the entity sheet.
- The rail is resizable via a drag handle (260–620px, persisted to
localStorage). The header's global fleet-health dots are unchanged;
only the right-rail content was replaced.
Risk: reversible_low (UI-only). The slug extractor is scoped to
focused mentions by design; edges may be slightly incomplete since
only root-fetched entities contribute edges, which is acceptable for a
session overview.
Verification: verified in the browser preview — loading a real session
built a 4-node graph (hubris/caddy/netbird-vps/strong) with the
hubris→caddy relationship edge; clicking hubris showed
"proxmox-host · active · healthy · checked 40s ago" with attributes and
relations; dragging the handle resized 320→440px and persisted; a
34-tool historical turn renders as one collapsed pill that expands on
click. tsc clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three UI issues reported after the neutral-gray redesign:
- favicon.svg was still filled #58a6ff (the pre-redesign accent blue);
changed to white to match the sidebar logo mark.
- Sidebar nav items all showed a filled background even when inactive.
Root cause: sidebar-menu-button.svelte (and -sub-button) rendered
`data-active="false"` as a literal attribute, but Tailwind's bare
`data-active:` variant matches attribute *presence*, not value — so
data-active:bg-sidebar-accent applied to every item regardless of
state. Fixed by emitting the attribute only when active
(`isActive || undefined`), a latent bug in the vendored shadcn
component that read as intentional until flagged.
- Tailwind's preflight resets <button> to cursor: default, so no button
in the app showed a pointer. Added one base rule restoring
cursor: pointer for buttons, [role=button], links, summary, and
select (respecting :disabled / aria-disabled) rather than annotating
each call site — covers new interactive elements automatically.
Risk: reversible_low (UI-only).
Verification: verified in the browser preview that inactive sidebar
items are transparent (only the current page shows a background),
nav buttons report cursor: pointer via computed styles, and the
favicon renders white in the tab.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: two issues surfaced after the dashboard-01 shell change
(cbfd09c). (1) Sidebar.Inset previously had an explicit h-svh that
hard-capped the app's height at the viewport; adding variant="inset"
put a margin on that same fixed-height box, pushing it taller than the
viewport with nothing left in the chain to cap it (Sidebar.Provider's
own wrapper only sets min-h-svh — a floor, not a ceiling). Result: the
whole page scrolled as one long document instead of each page's own
content scrolling internally with the header pinned — confirmed via
computed styles, e.g. Entities.svelte's table wrapper measured
scrollHeight 6531px against a 900px viewport, all of it spilling past
body instead of scrolling in its own rounded-border container.
(2) The color palette was GitHub-dark-inspired (blue-tinted grays:
#0d1117 bg, #58a6ff primary/accent) rather than the neutral grays the
shadcn-svelte dashboard-01 reference actually uses.
Change:
- App.svelte: moved the height cap up to Sidebar.Provider itself
(class="h-svh") instead of Sidebar.Inset, since the cap needs to sit
above wherever the inset variant's margin gets applied, not on the
same box as the margin.
- app.css: replaced the core tokens (background/foreground/card/
popover/primary/secondary/muted/accent/border/input/ring/sidebar-*)
with shadcn's canonical dark-theme OKLCH values (0-chroma neutral
grays), pulled directly from huntabyte/shadcn-svelte's own
docs/src/app.css rather than approximated. --success/--warning
deliberately kept as real, distinguishable colors — they signal
actual health state, and desaturating them to match the neutral
chrome would reintroduce the "can't tell what's actually happening"
problem this whole project started from (see 279549c). --accent-blue
now aliases --sidebar-primary (still a real blue) instead of
--primary, so the couple of spots wanting an interactive "pop" still
have one while buttons/links/focus rings ride the neutral --primary.
Risk: reversible_low (UI-only).
Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). go build/vet clean (backend
untouched, sanity check only). Manually verified in the browser
preview at 1400px: document.body.scrollHeight now exactly matches
window.innerHeight on both Overview and the 193-row Entities table
(previously 6531px vs 900px); scrolled the Entities table wrapper to
row ~60 and confirmed the header/filter bar/column headers stay
pinned while only the table body scrolls; confirmed neutral gray
rendering across Overview's stat cards, the event-rate chart, and
Chat's tool-call list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problem: requested visual alignment with the shadcn-svelte dashboard-01
reference block (shadcn-svelte.com/blocks#dashboard-01) — the app's
sidebar/header shell and Overview stat cards looked plain by comparison.
Change: pulled the actual reference source (app-sidebar.svelte,
nav-main.svelte, site-header.svelte, section-cards.svelte from
huntabyte/shadcn-svelte) rather than approximating from screenshots.
- App.svelte: Sidebar.Root now uses variant="inset" (the floating,
rounded, shadowed content panel — already fully built into the
existing Sidebar.Inset component via peer-data selectors, just never
enabled). Brand mark is now a proper Sidebar.MenuButton matching the
reference's padding/hover treatment; "New chat" uses the reference's
primary-colored button styling. Header matches the reference exactly:
h-(--header-height) (48px, was 44px), vertical separator after the
sidebar trigger, right-aligned actions group.
- Overview.svelte: stat cards rebuilt to match section-cards.svelte —
gradient background, Card.Action badge, Card.Footer with a bold line
+ muted context line, tabular-nums, responsive @container grid
(1/2/4 columns). Deliberately did NOT copy the reference's fake
trend-percentage badges (Oikos doesn't track historical trends, and
this project's whole thrust has been eliminating dishonest UI state —
see 279549c). Badges instead reflect real current-state signals
(healthy/degraded/down, clear/needs-review) computed from the actual
dashboard summary.
- EntityDetailContent.svelte + Entities/Signals/Ops/Events/Agent/
Audit/Knowledge pages: normalized root padding to p-4 md:p-6 (was a
flat p-6) to match the reference's responsive py-4 md:py-6 convention.
Risk: reversible_low (UI-only, no data or behavior changes).
Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings, same as prior commits). go build/vet
clean (backend untouched, sanity check only). Manually verified in the
browser preview at 1400px: inset sidebar's margin/rounded-corner/shadow
classes confirmed applied via computed styles; Overview cards render
with real live numbers from the now-fixed dashboard summary endpoint;
Signals/Ops pages confirmed visually consistent with the new spacing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problem: the web UI felt dead and hard to navigate — the Entities table
had no health/freshness signal (just a meaningless row-mutation
timestamp), no way to see what was actually monitoring an entity,
sessions couldn't be reopened, and every drill-down was a full page
navigation that lost the list.
Change:
- Entities table: Updated column replaced with a health dot + relative
"checked Xm ago", sourced from the backend's new health/last_check_at
fields.
- New EntityDetailContent.svelte extracted from EntityDetail.svelte and
shared between the full #/entity/:slug page and a new EntitySheet.svelte
opened from the Entities table (master-detail, row click opens a panel
instead of navigating away). Adds a Monitoring card listing the
entity's check_defs (kind, interval, enabled/disabled with
click-to-toggle via the existing PatchCheck endpoint) and renders
attributes as key/value pairs instead of raw JSON.
- Sessions: fixed a bug where clicking a session loaded it into the
chat store but never navigated to the chat page, so nothing appeared
to happen. Added a SessionRail inside Chat so switching sessions
never leaves the chat surface.
- Fixed the local dev proxy (vite.config.ts): production Caddy strips
the /agent prefix before forwarding to nomos; the dev proxy didn't,
so every session/chat fetch 404'd locally while working in prod.
- Found and fixed a real latent bug while testing the session fix:
chat.ts's loadSessionMessages passed the persisted tool_calls array
straight through, but nomos stores the tool_use and tool_result as
two entries sharing one id. Chat.svelte's keyed {#each tool (tool.id)}
throws on the duplicate key, which silently blanked the entire
message list — invisible until sessions were actually clickable.
Fixed by merging tool_calls by id before rendering, matching the
shape the live-streaming path already produces.
- UI polish: sidebar logo is now just the omicron mark in white (was
icon+text in the accent color); removed the sheet overlay's
backdrop-blur (distracting per feedback); the Attributes/Relations/
Signals grids used viewport-based lg:/3xl: breakpoints, which forced
multi-column layouts based on browser width regardless of the sheet's
actual rendered width — switched to Tailwind v4 container queries
(@lg:/@2xl:/@3xl:) so layout responds to the real available width in
both the full page and the narrower sheet.
Risk: reversible_low (UI-only; no destructive operations; the tool_calls
merge and dev-proxy fix are corrections to broken paths, not behavior
changes to working ones).
Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). Manually verified in the browser
preview against the live dev API: Entities table health column renders
correctly; clicking a row opens the EntitySheet with a populated
Monitoring card (16 checks for host:hubris, verified via psql that
check_defs.target_id links them correctly); clicking a session now
loads its full transcript inline (was blank before the tool_calls fix);
sheet has no blur and lays out single/multi-column correctly at the
sheet's actual width.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.
Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
cd.TargetID (falling back to the check's own id if unset) and write
status/metrics/events there. Signals stay keyed by the check entity,
unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
observation is older than 3x its fastest enabled check's interval
(floor 5m) is marked 'stale' and emits health.stale, so a stalled
scheduler or disabled check_def can no longer look like current data
forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
rows so dashboard/fleet-health rollups stop double-counting probes as
monitored entities. Historical metric_samples on check entities are
left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
the conversation instead of dropping them (previously only final text
was replayed, forcing the agent to re-derive fleet state every turn),
and inject a compact live fleet-health snapshot into the system prompt
each turn so it starts oriented instead of spending an iteration on
discovery.
Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.
Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- disk_usage_check.sh: sed 's/-/0/' for filesystems without inodes
- checkdefaults.Ensure: includes target_id in check_defs INSERT so
signals get proper target slug instead of null
Add docs/signal-triggers.md covering:
- End-to-end sequence diagram (Nomos → API → Scheduler → target host)
- Two paths: autonomous collection (scheduler) + query (MCP)
- All 6 check kinds and 17 available scripts
- Script deployment flow via sync timer
- Signal lifecycle, threshold evaluation, data flow through DB tables
- Prerequisites for SSH checks in Docker
- InsertMetricSample now includes ts=now() (TimescaleDB hypertable requires it)
- ssh-script: pass host and port separately (ssh uses -p flag, not host:port)
- ssh-script: use OIKOS_SSH_USER from config/env, default root
- Add -o LogLevel=ERROR to suppress SSH warnings polluting JSON output
- Use Output() (stdout only) instead of CombinedOutput()
- Set OIKOS_SSH_USER=root in scheduler docker-compose service
- Switch Dockerfile from distroless/static to alpine:3.21
- Install openssh-client-default in runtime image
- Mount SSH key in scheduler service (docker-compose)
- Add NET_RAW capability for ping checks
- Wire OIKOS_SSH_KEY_PATH and OIKOS_SSH_USER env vars in scheduler
- sshExec uses configured key path with StrictHostKeyChecking=no
