ListEnabledCheckDefs selected interval_s but never filtered on it, so every
enabled check ran on every 30s pass and the declared per-check intervals were
decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126
SSH connections every 30s (~363k/day) and `apt update` on every machine every
30 seconds — 14,400 mirror hits a day to answer a question that changes daily.
- check_defs.last_run_at (migration 026) + a due-ness predicate in the query.
A column rather than scheduler memory because this control plane restarts on
every deploy, and an in-memory map would re-fire every check on each restart.
- runCheck stamps last_run_at before processing the result, so a permanently
failing check backs off to its interval instead of re-running every pass.
- updates and backup-freshness drop to daily. Both answer questions whose
answers change about once a day; 60s was just the shared ssh-script default.
- last_run_at is seeded to a random offset within the interval so checks
created by the same seed do not stay in lockstep — otherwise ~165 probes
land in the same instant each minute instead of spread across it.
Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them.
Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day
(down from 14,400), with each 60s check landing at its own point in the minute.
Also renumbers 022→023, 023→024, 024→025: origin/main added its own
022_knowledge_revisions, and prod has already applied version 22. Left
colliding, prod would have skipped the monitoring_spec migration entirely and
then failed the seed on a missing column.
Co-Authored-By: Claude <noreply@anthropic.com>
Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by
discarded errors in checkdefaults:
- writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT
(slug) DO NOTHING, then wrote a check_defs row referencing it. On any
re-seed the slug already existed, the entity insert no-oped, and the FK
violated — aborting the ingest transaction and surfacing as an unrelated
failure several entities later. Re-seeding has been broken since; prod's
coverage was frozen at its first successful seed. This is what
TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting.
- shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed
to ".network" and overwrote each other; service:jellyfin collided with
lxc:jellyfin.
- The ssh-script checker never read the `args` config checkdefaults wrote, so
process_check.sh always ran without its unit name and returned "unknown".
Coverage is now 75/89. Monitoring is declared per entity type in
seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say
it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap.
coverageSweep raises an `unmonitored` signal only where a type declares
monitoring it lacks — 8 real gaps, no false positives.
Also:
- entity_types.attribute_schema was never ingested: the seed loader read
"attribute_schema" but the YAML says "attributes", so all 60 types stored
JSON null.
- ListExecutions ignored its declared target/action/correlation_id filters and
paginated on a non-unique target slug, dropping and repeating rows.
- started_at was captured but only written at terminal state, so a running
execution reported NULL for its whole life. The three MCP auto-run copies
wrote no timing at all; they are now one autoRun helper.
- SSH output was buffered to completion and discarded entirely on timeout.
Both sshExec copies now stream through a shared execlog sink into
execution_logs, and keep partial output when a command is cancelled.
- executions.correlation_id was a random per-execution uuid that correlated
nothing; it is now the chat session id, which is what lets the chat tail
live output.
- reversible_low had no auto-run branch despite policy declaring it
unattended. Since computeCommandRisk never returns it, the class only arises
when an agent declares it over a read_only command — so gating it penalised
candor without adding safety.
- backup-target gains a backup-freshness checker (portable find -mmin, since
the first target is on macOS), resolving its host by walking backs-up-to
backwards. The pre-deploy pg_dump is now a tracked backup target.
UI: an Executions section on entity detail with live output tailing, and
streamed output under a running `run` call in the chat timeline.
Migrations 022-024. Ops.svelte and context.ts exclude execution.output from
their refetch triggers, which would otherwise fire once a second per command.
Co-Authored-By: Claude <noreply@anthropic.com>
Classifier now unwraps pct exec / qm guest exec / bash -c / sh -c / sudo
and env-var assignments before classification, so read-only inspection
wrapped in pct exec no longer escalates to config_mutation. curl GET
(default method, no -d/-F/-T/-o/>) is read-only. Eliminates the three
duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) that bounced
off the classifier for the same goal.
New classify_command MCP tool: command-scoped preflight that returns the
exact risk class run would assign. Documented in SOUL.md with guidance
to pre-classify before run when the verdict is uncertain.
set_goal surfaces prior partial/failed sessions from the last 24h so the
agent picks up the thread instead of rediscovering it.
completeTask auto-closes in-flight plan steps (pending/running -> done
on success, skipped on partial/failure), so one-step plans no longer
need the per-step running->done dance right before completion.
Migration 021 adds blocker + closed_at to agent_sessions. completeTask
sets closed_at once and derives a structured blocker reason
(approval_timeout, user_abandoned, classifier_overreach, model_refusal,
tool_error, ...) from the last assistant message.
/sessions list now carries message_count, tool_call_count,
duration_seconds (server-side aggregates — no more N+1 transcript
fetches to audit a fleet). GET /sessions/{id} returns both metadata
and messages. New query params filter + paginate: outcome, status,
entity_id, blocker, since (RFC3339 or Go duration), cursor, limit.
Titles now prefer the goal when set; sessions without a goal fall back
to the first assistant text.
New GET /sessions/{id}/tool_calls flat view for audit scripts.
Plan: plans/2026-07-20-session-review-ten-sessions.md. VERSION 0.7.12 -> 0.7.13.
Implements fixes 1-3 of plans/2026-07-11-task-completion-safety-net.md.
Confirmed live that 50/50 production sessions never reached a terminal
status because the model almost never calls complete_task, even for
trivial single-tool Q&A turns SOUL.md explicitly calls out as needing it.
- Inline safety net (agent.go): a session that never called set_goal never
framed itself as a structured task, so its first plain-text turn-end IS
the task ending — auto-complete it there instead of leaving status stuck
at its creation default forever.
- Idle sweep (continue.go, new completion_nudges column): goal-bearing
sessions that stall get one nudge, then auto-close with outcome=partial
if the nudge goes unanswered, mirroring the pattern resumeSession already
uses for a different stuck-session failure mode.
Fix 4 (backfill of the 50 already-stuck live sessions) is deliberately
separate — deferred until this is deployed and verified live, per the
plan's implementation order.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Migration 018 adds goal/status/outcome/summary/entity_id to agent_sessions
and creates session_plan_steps + session_questions. Registers a 'task'
entity type and an 'involves' (task→entity) relationship in the ontology
so each session anchors its knowledge and involved-entity edges on the
existing relationships graph.
nomos createSession now mints a task:<session-id> entity (type task) and
links it via agent_sessions.entity_id — best-effort so chat never blocks on
it. listSessions/GET /sessions surface the new task fields.
No behaviour change yet; this is the data foundation for the task board and
live context panel. Verified end-to-end against the local stack: migration
applied, ontology ingested (60 types/47 rels), a new session mints a linked
task entity and the API returns status/goal/entity_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.
Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
cd.TargetID (falling back to the check's own id if unset) and write
status/metrics/events there. Signals stay keyed by the check entity,
unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
observation is older than 3x its fastest enabled check's interval
(floor 5m) is marked 'stale' and emits health.stale, so a stalled
scheduler or disabled check_def can no longer look like current data
forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
rows so dashboard/fleet-health rollups stop double-counting probes as
monitored entities. Historical metric_samples on check entities are
left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
the conversation instead of dropping them (previously only final text
was replayed, forcing the agent to re-derive fleet state every turn),
and inject a compact live fleet-health snapshot into the system prompt
each turn so it starts oriented instead of spending an iteration on
discovery.
Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.
Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 1 implementation from the client-lifecycle plan.
- Migration 012: provisioning_steps table, context_version, context_files,
enrolled_at column, slug+type index for machine entities
- API endpoints (openapi.yaml + generated code):
POST /clients/enroll — age key issuance, Infisical identity, state transition
GET /clients/{slug}/context — agent file delta polling (replaces git pull)
GET /clients/{slug}/secrets — scoped secret listing
POST /entities/provision — compute entity creation with constraint validation
GET /entities/{slug}/provision/status — step-by-step provisioning progress
- Handlers in impl.go: enrollment with state validation and age key generation,
provisioning with execution tracking and relationship creation,
context endpoint with since-based delta queries
- Server struct extended with secretsBackend interface for key storage
- All tests pass, build clean
- sqlc.yaml + internal/db/queries/*.sql: typed queries for entities,
relationships, ontology, operations (signals, events, audit,
idempotency, entity_status)
- internal/db/sqlcgen/: generated Go from sqlc (pgx/v5)
- internal/observability/record.go: Audit() and Event() helpers that
write in the caller's transaction (SG10). actorLabel is interim text
identity in detail JSON until OIDC resolution lands; actor_id column
exists but is not yet populated
- migrations/008: post-commit pg_notify trigger on events table for
SSE fan-out (SG8/SG10)
Review of aa2ca0a found and fixed:
- re-ingest duplicated ALL edges (upsert conflicted on valid_from=now(),
never fired) — migration 007 dedupes + partial unique index on current
edges; upsert now targets it. Regression-tested.
- export was a stub that overwrote seeds/*.yaml with 11-byte "version: 1"
files — implemented real deterministic export (ontology/inventory/policy,
cognition-layer excluded); round-trip is byte-stable (tested)
- DB password leaked in startup logs (slog JSON bypasses String()) —
Config now implements slog.LogValuer; regression-tested
- docker-compose had literal '***' as DB password — env-interpolated
- uuid.New() (v4) → uuid.NewV7() per ADR-0005
- no ontology validation on ingest — internal/ontology TypeTree: abstract
instantiation rejected, relationship endpoints hierarchy-validated,
cardinality enforced in-transaction, lifecycle states checked, default
state applied (Phase 1 gate items, R3-1)
- getOrCreateEntityID swallowed non-ErrNoRows errors
- migration runner now holds a session advisory lock on one connection
- Makefile: hardcoded /opt/homebrew/bin/go → go; test-db target
Tests: 4 unit suites + 7 integration tests (env-guarded, throwaway DB per
run): migrate idempotent, seed idempotent + no dup edges, abstract/edge/
cardinality rejection, blast_radius cycle termination, export round-trip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
