Adds pvecm_quorum_check.sh probe script and wires it into the
checkdefaults system as a new 'quorum' monitoring kind on proxmox-host
entities. Runs every 60s via ssh-script, surfaces unhealthy signal when
cluster loses quorum.
Closes the monitoring blind spot that let the 2026-08-12 3.5h corosync
flapping outage go undetected (ping passed, cluster was non-quorate).
Changes:
- seeds/ontology.yaml: proxmox-host declares monitoring: [quorum]
- internal/checkdefaults/defaults.go: KindQuorum builder
- internal/checkdefaults/build_test.go: 2 new test cases
- checks/pvecm_quorum_check.sh: new probe (deployed to hubris + strong)
- VERSION: 0.30.2 -> 0.31.0
Two fixes from the deploy pipeline audit:
1. Infisical tag v0.99.1 no longer exists on Docker Hub — bumped to
v0.162.19 (latest available). This was silently breaking the full
deploy pipeline (docker compose up failed on image pull).
2. Deploy failures now notify via two channels:
- Oikos API event (deploy.failed, severity=critical) — picked up by
the scheduler's notifier for Matrix alert
- Matrix webhook URL if MATRIX_WEBHOOK_URL is configured
Uses a trap with _ok flag to catch any non-zero exit path,
including CI gate rejections and health check timeouts.
Webhook now resolves and passes OIKOS_API_TOKEN to deploy.sh.
Registers a handler that serves the embedded OpenAPI 3.0 spec
(compiled into the binary via oapi-codegen) at a browseable
endpoint. Uses gen.GetSwagger() to deserialize the embedded
base64+gzip spec and returns it as JSON.
46 paths, 42 schemas — agents and humans can now introspect the
full API surface without reading Go source.
Read-only diagnostic commands ethtool, lsmod, lspci, modinfo, and dkms
were missing from the readOnlyLeadPattern in the command classifier,
causing compound diagnostic commands (e.g. 'uname -r && ethtool -i eno1
&& lsmod | grep r8169') to be misclassified as config_mutation instead
of read_only. This forced operator approval for simple hardware/driver
inspection during the 2026-08-12 hubris NIC cutover session.
Added regression test with the exact compound command from that session.
New EntityGraph.svelte app (sigma.js + graphology):
- Theme-aware Gruvbox palettes (light/dark) with reactive color switching
- Lucide icons rendered synchronously via Path2D canvas per entity type
- Color mode toggle (Health / Type) with per-type distinct colors
- Health distribution bar with clickable filters
- Entity type filters grouped by ontological layer (collapsible)
- Relationship type edge filters with color-coded swatches
- Quick presets: All / Problems / Infra
- Node selection with live blast radius from API
- Hover neighborhood highlighting with muted fade
- Isolated node hiding, edge alpha tuning, dot-grid background
- Search with camera focus on highest-degree match
createTaskEntity creates the task entity but never adds any graph
edges. The involves edges are only added by the run handler, so
sessions that call set_goal but never make a run call leave orphan
task entities with zero relationships.
Adds an idempotent involves edge from the new task to agent:nomos
at creation time, matching the same pattern used for run's involves
edges in server.go.
Every run call creates a classification entity, but it was never
connected to the execution via a graph edge — only via a DB column
(executions.classification_id). The ontology requires:
classification —precedes→ execution
Without this edge, all 53 classification entities had zero
relationships, making them invisible to get_relations and
blast-radius analysis.
Adds an idempotent INSERT into relationships after the existing
classification_id update, matching the same pattern used for
targets edges.
The healthcheck subcommand was reachable only after main()'s Infisical secrets
resolution (4x retries/key, ~30s when Infisical is down), which blew the 5s
Docker healthcheck timeout — so nomos stayed docker-unhealthy despite serving
/healthz fine. Short-circuit 'nomos healthcheck' at the top of main() before
any secrets init; measured 0.58s, no Infisical retries.
The nomos runtime image is gcr.io/distroless/static (no shell/wget), so the
wget-based healthcheck (D5) could never run — nomos showed docker-unhealthy
despite serving /healthz fine. Add a 'nomos healthcheck' subcommand that
self-probes NOMOS_LISTEN/healthz (exit 0 on 200), and point the compose
healthcheck at ["/nomos", "healthcheck"].
Measured startup is ~93s: NewHandler stalls on Infisical auth retries (~40s)
and OIDC discovery timeouts to auth.hubris.network (~35s) before binding
:8090. The api IS healthy once bound (serves /healthz); the window just needs
to clear both external-timeout phases so nomos (depends_on: api-healthy) can
start and the deploy completes.
The api retries Infisical at startup (4x with backoff) before binding :8090.
When Infisical is unreachable that adds ~60s, and the old start_period (10s)
+ 10 retries (~60s grace) ran out just before the bind — marking the api
unhealthy and failing the deploy (nomos depends_on api-healthy). 90s covers
the slow-startup window; the api genuinely serves /healthz once bound.
The procedural-glyph-engine dep pointed at a non-portable file:/private/tmp/orby-pkg
path, breaking npm ci in Docker and every main deploy since v0.20.0 (the build
cache masked it until it busted ~Aug 5). Vendor Orby v5.0.0 into web/vendor/,
switch the dep to file:../vendor, and use npm install in the web Dockerfile
(file: deps need install, not ci). Cherry-picked from 3cd4cf9.
- Wire secretsManager in NewHandler() — instantiate InfisicalBackend
when OIKOS_INFISICAL_SITE_URL is set (previously always nil)
- Add get_secret, list_secrets, set_secret MCP tools with nil-backend
graceful degradation
- Add oikos secret get|set|list CLI subcommands for Infisical
- Fix Set() bug: create-before-update so new keys are created;
add Type: "shared" to Update so it finds the right secret;
disable SDK cache so Get returns fresh data after Set
- Clean enrollment response: remove fake infisical_client_id/
infisical_client_secret stubs, store age key in Infisical for real
- classifyAndGate: escalate read-only commands on lxc: targets that
touch /opt/, /etc/, /var/lib/ to config_mutation. The classifier
scores command text only, not the SSH transport layer — SSH-ing into
a container to read config is riskier than pct exec from the host.
- ontology: standalone-server monitoring override from inherited
[ping, resource, updates] to [http]. VPS-like machines may not be
SSH/ICMP-reachable from the scheduler; HTTP is the LCD liveness
signal. Entities with full SSH can override per-entity.
- get_health_summary: filter out state=destroyed entities (was noise
from 20+ destroyed test LXCs, deprecated services, etc.)
- create_entity: document the monitoring footgun in the tool description
(creating a type=check entity does NOT wire a check_def; the correct
path is update_entity_attributes with monitoring + url attributes)
Adds a new 'dns' semantic monitoring kind that probes whether a DNS name
resolves. Uses net.LookupNS (NS records) with fallback to net.LookupHost
(A/AAAA). Supports an explicit server config for split-horizon resolution.
Changes:
- seeds/ontology.yaml: dns-zone monitoring: none → [dns] (was deferred
since 2026-06 with a comment 'no dns checker exists yet')
- seeds/inventory.yaml: host:netbird-vps monitoring: [http] (was none;
VPS was invisible for 7 days during the 2026-07-29 outage)
- internal/checkdefaults/defaults.go: add KindDNS, buildKind case for 'dns'
that creates a check_def at 5-minute intervals
- internal/scheduler/scheduler.go: add checkDNS probe + wire in executeCheck
The DNS checker catches stale/unreachable zones (e.g. matrix.hubris.network
pointing to a dead VPS IP). The VPS HTTP check probes the public endpoint
every 60s, closing the 7-day monitoring gap.
I — run pre-flights QEMU guest agent before queueing VM execution
classifyAndGate now checks vm: targets for qemu_guest_agent attribute.
If not_running/missing, returns immediate error instead of queuing forever.
II — policy.yaml: documented host-mutation classifier rule
Added comment clarifying that host-level package/kernel mutations
(apt-get install, dpkg, systemctl enable) always classify as
config_mutation and thus need operator approval.
III — health attribute read-only in update_entity_attributes
Strips scheduler-owned keys (health, last_check_at, last_check) from
attribute updates with a clear message directing agents to
get_health_summary / list_checks instead.
IV — Recorded discovered dependency edges
vm:zimaos → depends-on → lxc:nfs-export (NFS /media/library mount)
vm:zimaos → depends-on → host:strong (NFS /media/ludo-library mount)
Also updated the run tool description to mention both guardrails.
get_relations now accepts an optional 'types' (comma-separated) parameter
to filter relationship types — filters out the noisy exec/targets edges
that previously drowned useful host/provides edges.
get_health_summary now accepts an optional 'health' (comma-separated)
parameter to return only entities in specific health states (e.g.
'health=down,stale') instead of the full 100+ entity list.
ping_service now:
- Falls back to e.attributes->>'public_host' when 'url' is not set
(covers LXCs that only have public_host in the graph)
- Performs a live HTTP HEAD probe against the resolved URL, returning
the actual status code instead of just the scheduler's stale health
state
Also: fixed matrix.hubris.network DNS record (was pointing to dead VPS),
pruned 6 dead graph edges, wired url attributes on 7 LXCs, added VPS
HTTP monitoring check, and resolved the 18k-occurrence unmonitored signal.
This session's audit is documented as
document:nomos/2026-08-05-dns-monitoring-improvements-for-strong-hosted-services.
- Add MCP tool — lightweight connectivity check returning server
identity, no DB hit (resolves agent connection-test friction)
- Tighten 6 tool descriptions (get_relations, get_health_summary,
query_metrics, get_trend, get_event_timeline, ping) to be searchable
in the first 8-12 words
- Document Hermes MCP client setup in ADR-0012 with token security caveat
- Move completed plan to plans/done/
Backend:
- Add isThinking flag to agentEvent for text before tool calls
- Separate thinking from response text in runChatTurn and continue.go
- Persist thinking in a dedicated field in message content
Frontend:
- Add thinking field to MessageContent, ChatMessage, ChatTextEvent types
- Create ThinkingBlock.svelte — collapsible block with brain icon
- SSE handler moves text_delta content to thinking on isThinking flag
- Render thinking block between tools and response in ChatThread
- Fix chat window scroll reset on focus change (stable windowKeys order)
- Remove redundant #key id wrapper in WindowLayer
- Enlarge sidebar rail (24→32 default, 40→60 max)
- Remove glyph from sidebar, square graph at top
- Replace AgentTrace/ToolCallCard/UnifiedTimeline with TurnTrace/ToolLine
The "Working — message will queue" strip lived inside the sized input Pane, so
appearing/disappearing ate the textarea's fixed height and clipped it, forcing
a resize. Move it into the message Pane alongside the connection/error banners
— those correctly consume transcript space (flex-1) rather than the input's
fixed height. The input Pane is now stable whether or not a background turn is
running. Styling/idiom unchanged.
VERSION: 0.17.2 -> 0.17.3
The background-working hint was a plain muted text line bolted above the
textarea — misaligned with the input column and off-idiom. Restyle it as a
terminal status strip: spinner + uppercase fg "Working" label + muted detail,
hairline primary-tinted border (matching .trace.running), square, aligned to
the textarea's max-w-3xl column. Reads as part of the working state now.
VERSION: 0.17.1 -> 0.17.2
git mv staged the pre-edit index content; the status edits to these two
files landed in the working tree but not the archive commit. Amending the
status now so the archived copies reflect Implemented.
Move ten completed plans from plans/ to plans/done/ and update the index:
- 2026-07-18 session-review-three-sessions, 2026-07-20 desktop-mascot,
2026-07-20 session-review-ten-sessions, 2026-07-21 chat-full-polish,
2026-07-29 health-check-reality-and-knowledge-graph,
2026-07-30 session-review-plan-drift, and the four 2026-08-03 chat plans
(changes-review, reliability-and-ux-audit, cyberspace-style-adoption,
working-visibility).
- Refresh two stale statuses: cyberspace-style-adoption ("Draft" -> shipped as
full replacement in v0.16.0/757ef2f) and health-check-reality ("ready for
implementation" -> shipped across the v0.14.x-0.16.x check commits).
- .gitignore: ignore local tooling artifacts (.playwright-mcp/, config-screen.png).
No code change. index.md Active/Done tables now match the filesystem (no orphans).
VERSION: 0.17.0 -> 0.17.1
Make background/long/desynced turns visible and queueable, fixing the four
symptoms that survived the v0.15.0 chat reliability pass.
F1 - status-driven working signal (workspace.ts taskWorking/currentWorking =
streaming OR status in {planning,executing}). Drives the chat trace, indicator,
and activity spinner so a turn with no live stream (background resume, a dropped
SSE, an idle-close mid long turn) still looks alive.
F2 - operator messages sent during an in-flight turn are now QUEUED and
auto-run when the gate frees, replacing the "still finishing a previous step...
send it again" rejection. Per-session in-memory FIFO (messagequeue.go, capped at
20) drained one-at-a-time under the turn gate; a `queued` SSE event drives a
"Queued" hint. drainQueued releases via a per-iteration deferred closure so a
runChatTurn panic can't deadlock the session's gate.
F3 - SSE keepalive (12s `:keepalive` comment) in handleChat so 20-40s
inter-iteration gaps no longer trip a proxy/browser idle close (the desync root
cause). All SSE writes serialized through one mutex.
F4 - generation-aware activity timeline (only the last propose_plan renders;
superseded ones collapse to one "Earlier plan revised" marker; step-attribution
follows only the current generation) + debounced plan refetch on lifecycle
events so a missed plan.proposed self-heals.
Verified against the last session (23da10db: 6m33s turn, operator "status"
deferred at 19:48:05). go test ./cmd/nomos/ green (new messagequeue tests);
web vitest 72/72 (new F4 generation tests); vite build clean.
VERSION: 0.16.0 -> 0.17.0
Rebrand light/dark themes to the cyberspace.online look: warm cream-on-black
palette (light/dark are exact inverses), self-hosted JetBrains Mono + VT323,
square corners, border-driven surfaces with no soft shadows. Adds a terminal
design-system CSS layer (DOS double-border modals with hatched corner, fg focus,
inversion-on-hover), a theme-aware <RasterImage> (Atkinson-dithered canvas with
img fallback), and unifies desktop icons, taskbar, window controls, pills and
links under one idiom. Pins window titlebars to a fixed height and switches chat
auto-scroll off scrollIntoView to avoid titlebar reflow.
VERSION 0.15.1 -> 0.16.0
The Send button's disabled={!input.trim()} threw "trim is not a function" when
input was initialized from a non-string initialDraft (a Svelte 5 prop-init edge
where a null/undefined draft reached $state). Coerce at init so the composer
state is always a string.
VERSION: 0.15.0 -> 0.15.1
The agent could run two turns for one session at once (a reconnect resumed
while the live turn was still going), and their interleaved tool calls
corrupted the activity panel, fabricated a confusing "parallel/nested"
sequence, and made tasks feel stuck/never-ending. Several UX gaps compounded it.
Turn serialization (F1):
- turnGate: at most one in-flight turn per session. Background resume paths
(continuation worker, idle sweep, answer-question, /resume, reconnect)
skip non-blocking when busy; the live chat path waits briefly then bails
cleanly instead of stacking a second turn.
- resumeSession returns whether it ran; continueSession marks an execution
"continued" only after a real run (review P0) so a busy-skip can't lose a
finished-execution result. Idle nudge bumps only after delivery (P1).
Connection state (F2/F3, web):
- humanize/bucket raw errors ("model connection dropped..."); one surface
per drop; a terminal task.status event clears stuck streaming/disconnected
state and dismisses the connection toast. Reconnect no longer spawns turns.
Streaming where you look (F4, web):
- live command output in the global activity timeline and in the inline
tool card (auto-opened, tail-pinned) -- not just the per-window rail.
Other (web): artifact/knowledge deep links (F5); step-first stable
"thinking" headline (F6); stable chat layout, no empty->content reflow (F7);
lazy event sync (P2.2); reconnect skips a terminal session (P2.1).
VERSION: 0.14.2 -> 0.15.0
The plan recorded false history after a re-plan and the activity panel
showed fabricated, churning timestamps. Two bugs compounding on one event
stream.
Plan drift (P0.1):
- proposePlan seq is now 1..N per generation; (session,generation,seq) is
the addressing key. The model's 1-based update_plan_step calls always map
to the CURRENT plan after a re-plan, instead of resurrecting a superseded
`replaced` row as done while the live work went unrecorded.
- updatePlanStep resolves against MAX(generation); a stale/out-of-range seq
returns errPlanStepNotFound (never touches a superseded generation).
- getPlanSteps returns only the current generation by default; ?all=true
keeps the audit/eval view (plan_generations assertion).
- completeTask auto-close scopes to the current gen, stamps started_at, and
emits one plan.step.finished per closed step so the panel converges
instead of freezing on "running" after completion (P1.1).
- propose_plan result enumerates step seqs; writeback detector matches
"write back"/"writeback"/"upsert_knowledge" so a natural-language final
step isn't doubled (P1.2).
- migration 029 renumbers existing seq per generation + unique index.
Activity panel (P0.2 / P1.1, web):
- computeActivityLog uses the real message created_at for tool calls; live
entries fall back to wall-clock frozen on first sight, killing the 3s
poll churn. Steps use real started_at.
- dropped plan-step events warn + count instead of a silent no-op.
Tests: TestProposePlan updated; + generation-relative-seq and auto-close
event-emission regression tests; + web activity purity/timestamp tests.
VERSION: 0.14.0 -> 0.14.1
Every docker build sent the whole repo root as context, including every git
worktree under .claude/worktrees/ (200-300MB each) — that crossed 390MB of
cruft and starved mac-mini's disk mid-build on 2026-07-27 (873b00a). None of
it belongs in an image.
A `monitoring` attribute on an entity now overrides its type's declaration:
"none" opts out, a list overrides the kinds. service:haos uses it to opt out —
haos blocks SSH (no process probe can reach it) and the VM is already covered
by vm:haos's vm-status check, so the redundant process check only ever reported
false-down. vm:haos -> service:haos via provides confirms the coverage.
VMs declared monitoring [ping], but many block ICMP and lack a guest agent
(haos), so ping was the wrong probe — a powered-on VM reported "down". Add a
vm-status check: `qm status <pve_id>` on the VM's Proxmox host, which tests
"powered on" without needing the VM's network at all. vm type monitoring is
now [vm-status].
matrix.hubris.network is a public hostname (federation) resolving to
netbird-vps, not served by the lab Caddy — so its cert-expiry check's
dial=caddy IP failed. Drop the dial for matrix; it dials by name (DNS ->
public) like wget already proved works.
The ontology's stated intent was "http when it has a url, else a process
check", but the implementation emitted BOTH for every url-service — so ~17
fronted services carried a redundant process check that, under worst-of
aggregation, let a fragile supplementary probe (wrong unit name, unreachable
host, no guest agent) veto two healthy http checks and report the service
"down" while it was up (authentik, zimaos, house, matrix, ...).
buildKind now emits a process check only for services WITHOUT a url, or when
an explicit probe_unit opts into binary-level depth. http is the canonical
service-liveness probe (tests the real endpoint through the TLS terminator);
the redundant process checks were removed.
process_check.sh ran `systemctl is-active <entity-name>`, but a service's name
is a logical label, not its unit/container name — matrix is matrix-synapse.service
+ element-web/mautrix-* containers, authentik is authentik-server/-worker
containers. So every multi-component or docker service reported "inactive"
while up (authentik, matrix, photos, house, arr-stack, …).
Resolve in order: exact systemd unit, a unit with the name as prefix
(matrix -> matrix-synapse.service), or a running docker container whose name
contains it. checkdefaults passes a declared probe_unit/systemd_unit/container
attribute when set, for precision.
The DB-only audit_knowledge_graph can't see guests running in Proxmox that
have no entity, or entities whose pve_id is no longer live — the drift that
the stray test LXCs were a symptom of. discover_infra_drift enumerates running
guests via pct/qm list on every proxmox host (over the same SSH/pct path the
checks use) and diffs against the DB: returns missing (live, no entity) and
ghost (DB, not live). Read-only.
Companion to audit_knowledge_graph; the skill now runs both and treats the
remaining checks (misplaced parent, undeployed scripts, seed drift) as manual.
resolveProxmoxHostSlug trusted attributes.host verbatim, so a value polluted
with prose — lxc:teddycloud carried host="hubris (confirmed via pct config…)" —
became a slug that never resolved, leaving its checks 'down' despite a correct
`hosts` edge. Treat an attribute containing whitespace/parens as invalid and
fall back to the canonical hosts edge.
The audit now reports `polluted_attrs` — entities whose routing-critical
attributes carry prose — so this class is visible instead of a silent
resolution failure.
The B1 target-state filter `tgt.state NOT IN ('deprecated','destroyed')`
evaluates to NULL (unknown) when a target's state is NULL, which the WHERE
clause treats as false — so freshly-seeded entities without an explicit state
(the 20 TLS certificates) were silently dropped from ListEnabledCheckDefs and
never monitored. Treat NULL state as active (only explicit deprecated/
destroyed is excluded): `tgt.state IS NULL OR tgt.state NOT IN (...)`.
checkCertExpiry now accepts a `dial` address and sets ServerName to the
hostname — it connects to the terminator's IP while SNI/cert-read use the
hostname. The scheduler container has no mesh interface and the host resolver
doesn't know the split-horizon zone, so *.hubris.network can't be dialed by
name from there; dialing Caddy's lab IP (reachable on the LAN) makes the probe
work. The builder passes through a cert entity's `dial` attribute.
Re-seed the 20 *.hubris.network certificate entities with dial=192.168.8.175
(Caddy) and uses-certificate edges; cert-expiry monitoring now has real data.
disk_usage_check.sh built its mount list with `df`, which blocks on a wedged
filesystem (stale NFS export, a stuck ZFS pool) — and that stalled the whole
check past the scheduler's 30s budget, leaving host:hubris:4 perpetually down.
Build the mount list from /proc/mounts (a read that never stats anything), and
bound every per-mount `df` with `timeout 8` so a single stuck mount is skipped
instead of hanging the probe. Degrades to plain `df` on hosts without
`timeout`//proc/mounts (macOS), whose local mounts don't hang.
The cert-expiry builder and checkCertExpiry probe are correct, but the
scheduler container can't reach *.hubris.network:443 — its DNS forwards to the
host resolver, which doesn't know the split-horizon zone, and overriding the
container DNS would break docker service-name resolution. Seeding the 20 cert
entities now produced 20 false-down certificates.
Keep the builder (committed), drop the entities + edges until the scheduler can
reach Caddy (extra_hosts mapping, or a SNI-dial enhancement) — then re-add them.
The ontology declared monitoring [cert-expiry] on the certificate type and a
working checkCertExpiry probe existed, but checkdefaults had no cert-expiry
builder and no certificate entities were seeded — so certificate expiry, a
real failure mode, was invisible.
Add a KindCertExpiry builder (dials the cert's hostname on :443 hourly, warns
at 30d / crit at 7d) and seed certificate entities for the 20 public
*.hubris.network routes plus uses-certificate edges from each ingress route.
A service check used to bake its hosting LXC's lan_ip and SSH it directly as
root, which failed because the scheduler key is authorized on the Proxmox hosts
but not inside every guest — leaving all 8 service process checks 'down' even
after the guest routing and scripts were fixed.
ResolveExecTargetForCheck now, for a non-guest target, walks the
provides/runs-on/hosts edges to the compute entity that runs it and routes
through that: pct/qm exec if the host is a guest, direct SSH with the host's
correct user (workstation `user` attr) if it's a machine. The guest-resolution
path is shared via resolveGuest, and the scheduler no longer needs an
isMachine special case — one resolver handles guest, machine, and service.
Older writeCheck inserts omitted target_type, so every seed-created check_def
had a NULL/empty target_type. checkSSHScript's IsGuest check then never matched,
and guest checks silently fell back to their baked (often mesh-only) address —
keeping them 'down' even after the pct-exec routing and deployed scripts were
in place. rclone stayed down for exactly this reason after the host-hop fix.
writeCheck now writes target_type, and checkSSHScript resolves the type from
the target_id when the column is blank (a runtime safety net for existing rows;
the seed rows were also backfilled in the live DB).
Graph view: raise the node cap 500 -> 2000 and exclude execution/task audit
rows from the default whole-graph view so the cap is spent on actual topology
rather than ~380 cognition records that crowded out every host/lxc/service.
dns-zone monitoring [dns] -> none: no dns checker exists, so the declaration
only produced unresolvable `unmonitored` noise (requires ontology re-ingest;
coverageSweep now auto-clears the stale signals). Flip back to [dns] when a
checker lands.
Operator tooling: tools/deploy-checks.sh pushes check scripts into guests via
pct push (a pct-exec-routed check runs the script INSIDE the guest), wired
into the post-pull setup-checks hook so guests stay in sync on Proxmox hosts;
scripts/cleanup-orphan-checks.sh (dry-run by default) and
report-stray-test-lxcs.sh retire legacy cruft. VERSION 0.13.0 -> 0.14.0.
Plan: plans/2026-07-29-health-check-reality-and-knowledge-graph.md.
Adds audit_knowledge_graph (MCP tool) and GET /api/v1/audit/drift (endpoint)
backed by a shared internal/audit package. One pass surfaces the structural
gaps an operator otherwise finds by accident: orphan check entities, checks
targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored
declared types, and live edges pointing at destroyed targets. Each finding
carries a suggested remediation runbook. Read-only and safe to run unattended.
Ships the knowledge-graph-audit skill (SKILL.md + seeded runbook) that
interprets the report and routes findings to the lifecycle runbooks.
ListEnabledCheckDefs now LEFT JOINs the target entity and excludes rows whose
target is deprecated or destroyed, so retired things (secrets-issuance,
homelab-mcp, the dead secrets ingress route) stop generating permanent false
alarms instead of waiting for an operator to disable the check_def by hand.
coverageSweep's None() branch previously did nothing, so a type changed from
declared monitoring to `monitoring: none` (dns-zone) left its open
`unmonitored` signals lingering forever — a None() entity never gains a check,
so the hasCheck resolution path never fired. It now resolves those signals.
The scheduler SSHed each guest directly and assumed a deployed probe script
plus working root SSH at the guest's address — false for headless (nfs-export),
keyless (teddycloud), mesh-only (rclone), and macOS (mac-mini) targets, which
left 49 enabled checks stuck "down" on a healthy fleet.
Extract the MCP run tool's resolveExecTarget into a shared internal/remote
package and make it the single execution path for both the scheduler and MCP.
LXC/VM checks now host-hop via pct exec / qm guest exec through the owning
Proxmox host (no per-guest lan_ip, sshd, or authorized key needed); hosts and
workstations resolve their address and user live, so mac-mini's `user: dtoro`
is honored without a re-seed. Address preference now prefers public_ipv4 over
mesh, so netbird-vps is probeable from the scheduler container.
cpu_check.sh gains a real Darwin branch (it reported cpu_pct 0 before).
checkdefaults.resolveSSHUser reads the top-level `user` attribute too.
A machine-target resolution failure is now logged before falling back to baked
config, so a broken probe-config is distinguishable from a real outage.
Two things the entity window redesign surfaced but deliberately left alone.
**blast_radius answered the wrong question.** It walked source→target for every
relationship type, but which end of an edge is the dependent differs per type:
"machine hosts container" means the target breaks, while "service depends-on
service" and "ingress routes-to service" mean the SOURCE breaks. Walking
everything forwards was right for hosts/provides and backwards for everything
else — and swept in 2,800+ documents/involves/targets edges of pure bookkeeping,
so the result contained tasks and executions that cannot break.
Direction is now declared per relationship type in seeds/ontology.yaml
(blast_direction: forward | backward | none), the same shape as the entity
types' monitoring: declaration, and defaults to none so an undeclared edge
contributes nothing rather than a confidently wrong answer.
It also needed a modelling fix: `routes-to` names an ingress's BACKEND, so
nothing recorded that all 21 public hostnames are terminated by caddy. A
`served-by` edge type now says so.
pool:ludo-lvm 2 -> 23 (every container storing on it, then their services)
lxc:caddy 4 -> 22 (service:caddy, then all 21 ingress routes)
service:authentik 7 (what authenticates via it)
**Every ping check was reporting down.** Not a host:strong false positive: all
seven, including ws:mac-mini — the Docker host itself. The scheduler runs in
Docker on macOS, whose VM does not route ICMP to the LAN; loopback pings succeed
and every LAN ping fails. Under health aggregation each broken probe dragged its
entity to down.
The question the check exists to answer is "is it reachable", and ICMP is only
one way to ask it. checkPing now falls back to a TCP connect before concluding
anything, which restores an honest verdict for the four hosts that are genuinely
up while leaving the genuinely unreachable ones down.
TestBlastRadiusTerminatesOnCycles asserted the old direction (caddy=1,
authentik=2 — the cycle walked the wrong way); it now asserts the corrected
depths, and its exact-node-count check is relaxed because walking the right way
also surfaces the seed's own real dependents, which are correct answers.
Co-Authored-By: Claude <noreply@anthropic.com>
The window rendered the same 13 collapsible sections for every entity, sorted
only by "does it have content". Audit trail carried the same visual weight as
Health, and the window answered "what data do we hold about X?" rather than
"what do I need to know, and what should I do?".
Measured against prod: host:hubris has 223 relations, 1,601 events, 2.7M metric
samples and 148 executions; an ingress route has three facts. Both got 13
identical headers. Expanding a host put ~540 interactive elements on screen.
- **A verdict header that never collapses.** Not just "down" but *why*:
"ping failing · 5 of 6 checks passing". That line did not previously exist
and could not have — checks rendered as configuration, never as results.
- **Sections composed per type.** A document has no checks, metrics or blast
radius; a signal or execution is a record, not a thing. Infrastructure gets
Status/Impact/Activity/Metrics/Reference, knowledge types lead with Content,
records get a minimal view. Unknown types fall back to infrastructure so a
new entity type is never a blank window.
- **Status replaces Monitoring**, showing each check's own verdict and when it
last ran — the section that answers the header's "why".
- **Impact** finally calls /entities/{id}/blast-radius. The endpoint has existed
since the first API and had no frontend caller anywhere, despite
.agents/OIKOS.md naming blast radius as the reason the ontology exists. Its
outgoing-edges-only limitation is stated in the UI rather than hidden.
- **Activity merges four lists** (executions, signals, events, agent activity)
that were telling one story in four places.
- **Relations cap at 8 with a drill-in** — 540 interactive elements down to 126.
- **Ask Nomos** opens a task pre-scoped to what you are looking at, seeded with
the verdict just computed, via an optional draft threaded through
openNewTaskWindow -> NewTaskChat -> ChatThread.
Requires exposing check_defs.last_health/last_run_at through the API (the
columns landed with the health-aggregation work but were never surfaced).
Adding a fourth enum containing "unknown" made oapi-codegen disambiguate all
enum constants by type prefix, so metrics.go moves to gen.TrendDirection*.
Verdict derivation and type->section composition live in $lib/entityView.ts as
pure functions with 15 unit tests, including the host:strong case that
motivated this.
Co-Authored-By: Claude <noreply@anthropic.com>
host:strong logged 226 health.changed events in one hour, oscillating
down/healthy while the host was fine throughout. host:hubris did it 126 times.
runCheck wrote entity_status.health on every check completion, so an entity's
health was simply whichever of its checks finished most recently. A host with
six checks reported whichever facet happened to be sampled last, and one
failing probe alternating with five passing ones flapped forever. resolveSignal
forced "healthy" too, a second path by which one passing probe erased another
probe's genuine failure.
On this fleet the trigger is a known false positive: the scheduler's network
vantage point cannot ICMP host:strong, so its ping check fails while every
ssh-script check succeeds. Under last-writer-wins that single probe declared
the whole host down, twice a minute.
Each check now records its own verdict (check_defs.last_health, migration 027)
and the entity's health is the worst across its enabled checks. A failing probe
now degrades the entity honestly and *stably*, without erasing what the other
five report, and health.changed fires only when that aggregate actually moves.
Checks that have never run are ignored rather than counted as unknown, so
adding a check cannot drag a known-good entity down before it has a verdict.
Also declares service:oikos in the seed. The previous commit re-pointed the mcp
ingress at it, but the entity only ever existed in the production database — so
a fresh seed (a new install, or a DR restore) failed on an unresolvable edge.
Caught by seeding an empty database rather than a copy of prod, which is the
only way that class of bug shows up.
Co-Authored-By: Claude <noreply@anthropic.com>
Two unrelated console errors.
effect_update_depth_exceeded — mine, from the previous commit. The live-update
effects both read and wrote the same state: FleetMap's health patch builds a
new `graph` object every run, and EntityDetailContent's refreshExecutions()
assigns a fresh `executions` array. Svelte tracked those reads, so each write
re-triggered the effect, which wrote again, until it gave up. The effects now
depend on liveEvents alone and do their work inside untrack(). Applied to all
four live effects, including the two that happened to settle on their own —
relying on "applyHealthEvent returns the same reference when nothing changed"
to break a feedback loop is far too subtle to leave implicit.
SyntaxError: expected expression, got '<' — pre-existing, and unrelated to the
live-update work. index.html loads /wails/runtime.js unconditionally; that file
only exists inside the Wails desktop wrapper, which serves the same dist/ from
its own asset handler. In a browser it is missing, and the SPA fallback
answered it with index.html — so the browser parsed "<!doctype html>" as
JavaScript on every single page load. The web Caddyfile now returns a real 404
for /wails/*, and more generally serves asset extensions without the SPA
fallback: a missing .js or .css answered with HTML is always a confusing parse
error rather than an honest 404.
Co-Authored-By: Claude <noreply@anthropic.com>
nomos declared `depends_on: api: condition: service_started`, which only waits
for the container to exist. It came up while the API was still binding :8090,
failed its MCP initialize with "connection refused", exited 1, and crash-looped
for ~25 seconds on every single deploy. It always recovered on its own, which
is precisely why it went unnoticed.
service_healthy waits for the API to answer, so this needs api to declare a
healthcheck — wget is BusyBox's, already present in the alpine runtime image,
so nothing new is installed. /healthz pings the database, so "healthy" means
genuinely able to serve rather than merely listening.
Co-Authored-By: Claude <noreply@anthropic.com>
Refetching everything on a health event was wasteful and churned the UI: one
container going degraded pulled down the entire fleet entity list (plus its
parent-grouping pass), or the whole fleet graph, to learn something the event
had already delivered.
health.changed / health.stale carry the new value in their payload, so the
views that hold the entity just patch it:
- Fleet table: patch the row. Only entity.* changes which entities exist, so
only that still refetches.
- Fleet map: patch the node AND graph.health[id] — healthOf() reads the side
map in preference to the node's own field, so patching only the nodes would
have left the rendered colour unchanged.
- Entity detail: patch the open entity. Signals still need a read (the event
says one was raised, not what the list now contains) but only the signals,
not the entity and checks alongside them.
Shared in $lib/health.ts, which returns the original array when an event does
not apply so unrelated rows keep their identity and do not re-render. Note it
matches on entity_id, never data.slug: the scheduler emits health.changed with
entity_id = the observed entity but slug = the *check's* slug.
Separately, events.ts had no reconnect. onerror was empty on the assumption
the browser retries, but EventSource only does that for a transient failure --
once it reaches CLOSED (an HTTP error on connect, e.g. the API restarting
during a deploy) it stays closed forever. A single blip silently froze every
live surface in the app with nothing on screen to say so. Now reconnects with
capped exponential backoff, and exports eventsConnected so a future indicator
can show when the stream is down.
Verified against live prod: flipping lxc:apps health recoloured the map node
and moved its counts (30 healthy -> 29, 9 down -> 10) with ZERO network
requests.
Co-Authored-By: Claude <noreply@anthropic.com>
The SSE stream already carried health.changed, health.stale, signal.raised,
signal.resolved and coverage.unmonitored, but two of the places that render
health never listened for them.
- The Fleet table refreshed only on entity.*, so its Health column sat at
whatever it was when the page mounted while the map view beside it — which
did listen — updated live. Health arrives on its own events, not entity.*.
Coalesced on a 400ms timer because health.stale fires once per entity during
a sweep, and refetching the whole fleet per event would mean a burst of
identical requests.
- The entity detail window loaded health, signals and monitoring once on open
and never again, so a window left on screen kept showing the health it had
at mount. That is the same staleness this whole change set has been about,
reproduced one window at a time. Now scoped by entity_id, and re-reads only
what a health or signal event can actually change rather than re-running the
full 11-request load().
Verified against live prod: flipping lxc:apps healthy -> degraded -> healthy
updated the Fleet table and an open detail window together, without a reload.
Co-Authored-By: Claude <noreply@anthropic.com>
`systemctl is-active` prints the state AND exits non-zero when a unit is not
active, so `... || echo unknown` appended a second line: STATE became
"inactive\nunknown" and the script emitted a raw newline inside a JSON string.
The scheduler rejected all 14 process checks with "invalid character '\n' in
string literal".
Latent since the script was written — process checks never actually ran,
because checkdefaults wrote an `args` config the ssh-script checker ignored.
Passing args through finally executed them and exposed it.
- head -1 keeps the state, and the fallback only fires on empty output.
- Quotes are stripped from both the unit name and the state; either would
break the hand-built JSON just as thoroughly.
- signalKind is now the constant "process" rather than "$SERVICE". Emitting
the service name minted a distinct signal kind per service (kind=paperless,
kind=qbit, …) — nothing an approval_rule can match, and it makes "how many
process checks are failing?" unanswerable.
Co-Authored-By: Claude <noreply@anthropic.com>
Rehearsing the deploy against a full copy of prod surfaced 40+ cardinality
violations that would have failed the seed. Since api/scheduler/notifier all
depend on `seed: service_completed_successfully`, and this change alters the
seed files (so the content hash changes and a full re-ingest runs for the
first time in months), that failure would have stopped those services from
starting at all.
None of them are new. The foreign-key bug in checkdefaults was aborting the
ingest earlier, during entity ingest, so ValidateCardinality at the end never
got the chance to run. Fixing the first failure revealed the next.
- `documents` was declared many-to-one, meaning a document may document at
most one entity. Nomos has been writing docs that cover several (a
fleet-wide apt audit documents every host it touched) for months, which is
reasonable — the ontology was the strict one. Now many-to-many.
- The mcp ingress still routed to service:homelab-mcp, which prod marks
deprecated: the Python MCP server on apps/105 was stopped at the Go cutover.
Nomos re-pointed it at service:oikos on 2026-07-12 and was right; the seed
was stale, and re-asserting the old edge alongside the new one is what made
it a violation.
Remaining after this: one genuine drift, `hosts target=lxc:caddy (2 edges)`,
which needs a prod data fix rather than a code change — see the follow-up.
Co-Authored-By: Claude <noreply@anthropic.com>
ListEnabledCheckDefs selected interval_s but never filtered on it, so every
enabled check ran on every 30s pass and the declared per-check intervals were
decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126
SSH connections every 30s (~363k/day) and `apt update` on every machine every
30 seconds — 14,400 mirror hits a day to answer a question that changes daily.
- check_defs.last_run_at (migration 026) + a due-ness predicate in the query.
A column rather than scheduler memory because this control plane restarts on
every deploy, and an in-memory map would re-fire every check on each restart.
- runCheck stamps last_run_at before processing the result, so a permanently
failing check backs off to its interval instead of re-running every pass.
- updates and backup-freshness drop to daily. Both answer questions whose
answers change about once a day; 60s was just the shared ssh-script default.
- last_run_at is seeded to a random offset within the interval so checks
created by the same seed do not stay in lockstep — otherwise ~165 probes
land in the same instant each minute instead of spread across it.
Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them.
Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day
(down from 14,400), with each 60s check landing at its own point in the minute.
Also renumbers 022→023, 023→024, 024→025: origin/main added its own
022_knowledge_revisions, and prod has already applied version 22. Left
colliding, prod would have skipped the monitoring_spec migration entirely and
then failed the seed on a missing column.
Co-Authored-By: Claude <noreply@anthropic.com>
Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by
discarded errors in checkdefaults:
- writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT
(slug) DO NOTHING, then wrote a check_defs row referencing it. On any
re-seed the slug already existed, the entity insert no-oped, and the FK
violated — aborting the ingest transaction and surfacing as an unrelated
failure several entities later. Re-seeding has been broken since; prod's
coverage was frozen at its first successful seed. This is what
TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting.
- shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed
to ".network" and overwrote each other; service:jellyfin collided with
lxc:jellyfin.
- The ssh-script checker never read the `args` config checkdefaults wrote, so
process_check.sh always ran without its unit name and returned "unknown".
Coverage is now 75/89. Monitoring is declared per entity type in
seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say
it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap.
coverageSweep raises an `unmonitored` signal only where a type declares
monitoring it lacks — 8 real gaps, no false positives.
Also:
- entity_types.attribute_schema was never ingested: the seed loader read
"attribute_schema" but the YAML says "attributes", so all 60 types stored
JSON null.
- ListExecutions ignored its declared target/action/correlation_id filters and
paginated on a non-unique target slug, dropping and repeating rows.
- started_at was captured but only written at terminal state, so a running
execution reported NULL for its whole life. The three MCP auto-run copies
wrote no timing at all; they are now one autoRun helper.
- SSH output was buffered to completion and discarded entirely on timeout.
Both sshExec copies now stream through a shared execlog sink into
execution_logs, and keep partial output when a command is cancelled.
- executions.correlation_id was a random per-execution uuid that correlated
nothing; it is now the chat session id, which is what lets the chat tail
live output.
- reversible_low had no auto-run branch despite policy declaring it
unattended. Since computeCommandRisk never returns it, the class only arises
when an agent declares it over a read_only command — so gating it penalised
candor without adding safety.
- backup-target gains a backup-freshness checker (portable find -mmin, since
the first target is on macOS), resolving its host by walking backs-up-to
backwards. The pre-deploy pg_dump is now a tracked backup target.
UI: an Executions section on entity detail with live output tailing, and
streamed output under a running `run` call in the chat timeline.
Migrations 022-024. Ops.svelte and context.ts exclude execution.output from
their refetch triggers, which would otherwise fire once a second per command.
Co-Authored-By: Claude <noreply@anthropic.com>
The six loading states across the Knowledge wiki (initial app load, the
reader's note/history fetches, and the four Cleanup tabs) all showed a
centered spinner with no relation to what was about to render — costing
a full reflow the instant real content landed. Replaces each with a
skeleton shaped like its actual content (tree rows, reader header +
prose, revision list + diff, cluster cards, table rows, flat lists)
using the existing shadcn Skeleton primitive already used elsewhere.
Verified each of the six by temporarily injecting a delay into
fetchWithAuth and screenshotting the transient state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the read-only stats dashboard with a three-pane wiki: a
navigator tree (group by folder/type/tag/entity), a reader/editor with
bare-slug auto-linking and revision history + diff, and a context rail
for backlinks and related notes. Adds a Cleanup mode for the drift
tools (duplicates, tag manager, orphans, trash) and a Cmd+K quick-open.
Also:
- Adds a real landing view (hero count, KPI row, Nomos-share meter,
recently-updated, busiest tags) in place of the old "Select a note"
empty state, and extends the design pass across the tree/reader/rail
(kind icons instead of repeated text badges, accent-bar selection,
constrained prose measure).
- Guards every note-selection path behind a confirm when there's an
unsaved edit in progress, so switching notes can no longer silently
discard a draft.
- Extracts the markdown-rendering CSS duplicated across ChatThread,
EntityDetailContent, and the new WikiReader into a shared
.markdown-body class in app.css, with ChatThread keeping only its
decorative deltas.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Knowledge page was read-only from the HTTP API — the only writer was
the agent's MCP upsert_knowledge tool. Adds create/update/soft-delete/
restore/trash endpoints, a DB-trigger-backed revision history (catches
both the web UI and the MCP tool), and maintenance endpoints: duplicate
detection (pg_trgm + complete-linkage clustering), tag rename/normalize,
orphan detection, and merge.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.prettierrc.json was missing "semi": false, so prettier wanted to add
semicolons to a codebase written without them (763 semicolon-free
statements vs. 150 with, in hand-written .ts; zero hand-written .svelte
files use them at all). That's why prettier --check failed on 249 files
— not because the code was unformatted, but because the config didn't
match the actual house style. Added "semi": false; left printWidth/etc
as configured (printWidth barely moves the failure count: 218/213/212
files at 100/120/140).
Ran `prettier --write .` with the corrected config. Verified
semantics-preserving before and after:
- eslint: 142 problems both before and after, byte-identical
- build passes, 38/38 tests pass
- token-stream diff (whitespace/semicolons/quotes normalized) on all
218 changed files: only 52 had any remaining token change, all either
trailing-comma removal (matching trailingComma: "none") or import/
ternary reflow — no semantic changes
- live smoke test: Knowledge, Tasks, Fleet map, and a chat window
(AgentTrace, markdown, Scope graph, activity rail) all render
correctly, no console errors
Most of the diff is shadcn/ui vendor files (lib/components/ui/) moving
from the CLI's own style (double quotes, tabs, semicolons) to house
style; re-running `shadcn-svelte add` on a component will need a
follow-up format pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- drop the unused KnowledgeItem type import
- the svelte/no-at-html-tags disable comment sat on the wrong line: the
multi-line Card.Description opening tag meant "next line" wasn't the
line with {@html}, so it never suppressed. Reformatted so the {@html}
is on its own line, directly after the disable comment. Sanitization
(DOMPurify with ALLOWED_TAGS: ['b']) is unchanged — this was a false
positive, verified live (search results still render <b> highlights,
no script/attribute injection).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Taskbar: the theme toggle is icon-only now, matching the Settings
button beside it. The theme name moves into the title/aria-label so
an icon-only control still has an accessible name and the current
theme stays discoverable on hover.
- Pages: Fleet, Knowledge, Learning, Ops and Signals used p-4 (or
p-4 md:p-6) while Tasks used p-2, so windows didn't line up. All now
p-2. App Store and Settings are deliberately untouched — they have no
root padding, using a full-bleed header whose border spans the window;
insetting them would break that.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The chat rendered one card per tool call, so a 20-call turn buried the
answer under 20 stacked cards. Merge them with the "thinking" indicator
into a single collapsible strip above the answer:
- collapsed: the live activity while running, a count once finished
- expanded: the turn's work in humanized language (reuses the activity
log's toolActivityLabel, so ten identical "run · target: host:strong"
rows now read as what they actually did)
- per row: the raw args/result, one more click in
Also flip the Activity rail to newest-first with the current step on top:
- follow-mode/auto-scroll re-anchored to the top to match, or it would
jump to the oldest entry on every new event
- pending plan steps park at the tail rather than sorting above the
running step and pushing it off the top; the goal anchors the bottom
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- FleetMap: service-centric host -> container -> service graph replacing
the WebGL 3D force graph, with health coloring, hover-to-trace blast
radius, and click-to-open
- Desktop background: configurable CSS pattern picker in Settings ->
Appearance (8 patterns, color/fill/opacity/fade/size/rotation),
replacing the hardcoded ambient graph background
- Fix missing data-orientation/data-disabled Tailwind custom variants so
the shadcn Slider's track actually renders
- Rename "Knowledge Base" app to "Fleet"; scope its table to the same
fleet entities as the graph (compute-entity descendants + service)
instead of all entities
- Remove dead code: EntityGraph, GraphBackground, categories.ts,
MultiSelectFilter (all superseded by the above)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Windows: make floating windows fully opaque (drop backdrop-blur/color-mix
transparency), add margin around windows, reduce Overview padding to p-2.
- DataTable: fix Toolbar always rendering an empty padded bar (children
slot was always truthy regardless of actual content); split header/body
into separate tables so the scrollbar no longer overlaps the sticky
header; make sort work for derived/synthetic columns by sorting on the
column's accessor instead of a nonexistent row key.
- Overview: enable sorting on Status and Task columns via accessors.
- TaskContextPanel: give the Activity pane more height by default (Scope
30% / Activity 70%), fixing that the saved split sizes were never
actually applied to the bound Pane sizes.
- windows.ts: clamp new/resized windows to the desktop viewport so
content-heavy entity windows can't grow taller than the visible screen;
fixes a bad defaultSize.height ('30vh', an invalid non-numeric value)
that had silently left window height unconstrained.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problem: the mascot's right-click menu was non-interactive — RadialMenu's
root div rendered inside MascotLayer's pointer-events-none root (and the
new DockedLayer wrapper compounded it) without re-enabling pointer-events,
so clicks passed straight through. The desktop right-click menu was a
hand-rolled positioned div, inconsistent with the rest of the UI.
Change: both menus now use the shadcn-svelte context-menu primitive
(bits-ui, portaled to <body>).
- Mascot: MascotMenu.svelte renders the action tree recursively —
children become ContextMenu.Sub (native hover sub-menu navigation,
replacing the manual breadcrumb stack), leaves become ContextMenu.Item
with onSelect. MascotLayer wraps <Mascot> in a ContextMenu.Trigger;
visibility predicates read reactively off ctx.model so items
appear/disappear live. Removed the manual menuPos/openMenu/closeMenu
machinery. RadialMenu.svelte deleted.
- Desktop: the surface's bare-desktop hit area is now a
ContextMenu.Trigger layer (absolute inset-0, pointer-events-auto)
placed before the icons/windows in the DOM. The DOM-structure gate
(icons/windows are pointer-events-auto siblings that paint on top and
intercept their own right-clicks; bare desktop falls through to the
trigger) replaces the old fragile e.currentTarget === e.target check.
Left-click blur moved onto the trigger; Undo/Redo disabled state
snapshotted via onOpenChange (canUndo/canRedo are wmkit methods).
Risk: the blocker that made the mascot menu non-interactive in the first
place — Mascot.svelte's handleContextMenu called e.stopPropagation(),
which would have prevented a ContextMenu.Trigger wrapper from ever
seeing the right-click. Removed that handler; bits-ui now owns
right-click on the mascot, left-click drag/pet passes through. The
context-menu content portals to <body>, escaping the pointer-events-none
mascot and docked layers entirely — the structural fix, not just a
component swap.
Verification: vitest 38/38; svelte-check + tsc clean for changed files;
eslint clean (the shadcn-generated ui/context-menu/* files carry the
same baseline custom_element_props_identifier warnings as the rest of
the ui/ folder, not from this change); vite build green; runtime
confirmed — right-click mascot opens the action tree with hover
sub-menus, right-click bare desktop opens Cascade/Tile/Show/Reset/
Undo/Redo, right-click on an icon or window does not.
Problem: the frontend had an implicit OS+Apps metaphor (desktop, floating
windows, an app registry) but the contract was informal — the mascot was
hardcoded into the shell, all apps were statically imported into one
800KB bundle, and there was no install/uninstall path.
Change: three phases landed.
- Phase 1 (contract + docked kind): AppDef extended with docked/noIcon
and optional geometry; the mascot registered as a docked app via a
generic DockedLayer that replaces the hardcoded <MascotLayer />;
openAppWindow branches on docked → toggleDocked; persisted docked
visibility store (absent key = visible, no APPS import to avoid a
static cycle).
- Phase 2 (lazy loading): AppDef.component is now a dynamic-import
loader; LazyApp renders with a loading skeleton; Vite code-splits
each app (main bundle 800KB→485KB); the LazyMascot wrapper is gone
since the lazy loader breaks the import cycle directly.
- Phase 3 (installable apps, local bundles): AppManifest + catalog +
installApp/uninstallApp + localStorage persistence; reactive apps
store (built-in + installed) and derived appById; App Store page;
Notes demo app; icons.ts and WindowLayer's orphan-close react to
registration so installs appear without a reload.
- Structure: data-table casing unified to PascalCase; the mislabeled
DataTable.svelte.ts (pure types, not runes) renamed to types.ts;
LazyApp colocated with its desktop-shell consumers; app-store moved
under lib/ so the dependency direction is consistent.
Risk: the app registry is now a reactive store, not a static array, so
every consumer (Desktop, DockedLayer, Taskbar, icons, windows) reads
from derived stores. Two static-cycle traps are documented in
docs/mbse/components.md §9: docked.ts must not import APPS (it would
fire a TDZ at init via the apps.ts→pages→windows.ts→here path), and
apps.ts must not statically import the mascot (the lazy loader defers
its module graph). Remote bundle loading, the /api/v1/apps endpoint,
and permission enforcement are deliberately NOT in this commit — they
are security-critical and deferred to Phase 4 with an ADR.
Verification: vitest 38/38; svelte-check + tsc clean for changed files;
eslint clean; vite build green; runtime smoke confirmed (install
Notes → icon appears → open → uninstall → icon + window gone; survives
reload). docs/mbse/components.md Component 9 and the plan updated.
Plan: plans/2026-07-21-frontend-os-apps-architecture.md
A config_mutation/destructive run() queued for approval never touched
agent_sessions.status — only ask_operator did that, setting
awaiting_input. So a task blocked on an execution approval was
indistinguishable from one still genuinely working: the frontend's
"Needs input" bucket only checks status===awaiting_input (never lit
up for these), and the idle-sweep safety net only excludes
awaiting_input from its stale-task query, so after ~30 minutes idle
it would nudge the agent and then auto-close the task with
outcome=partial while the approval was still sitting there undecided.
classifyAndGate now flips the session into awaiting_input the moment
an execution is queued (internal/mcp/server.go), and DecideApproval
flips it back to executing once the approval is approved, denied, or
revoked (internal/httpapi/approvals.go) — mirroring askOperator /
answerQuestion's existing pattern for session_questions. Both emit
task.status so the board updates live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sticky top-0 on each <th> (not the <thead> itself — more consistent
sticky support across browsers for table headers) plus a background so
scrolled rows don't show through underneath it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
text-primary instead of text-success — keeps the "done" state on-brand
with the rest of the UI (buttons, focus rings) rather than introducing
a separate green that only really worked well on the dark theme.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
text-success/50 and /60 washed out to almost nothing against the light
theme's cream card background — full-opacity text-success still reads
as a calm, muted green (not alarming) but is actually visible on both
themes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The launcher's textarea inherited the base Textarea component's default
field-sizing:content (auto-grow to fit typed content) — ChatThread's
input already overrides this with field-sizing-fixed, but the desktop
launcher never did, so the box would jump taller the moment you started
typing. Also bumps the floating-window frosted-glass opacity from 70%
to 85%: backdrop-filter's blur strength isn't consistent across
engines, and Firefox blurs noticeably less than Chromium at the same
radius, making the Chromium-tuned opacity look far too see-through
there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pending operator-question card now renders inline in ChatThread (the
newest thing in the conversation) instead of in the context rail — it's
part of the chat, not a separate side panel, and the panel's hasContext
gate no longer needs to special-case it.
The desktop mascot's reactions are now entirely about whichever task
window has focus, not fleet-wide events: thinking/talking is a new
continuous `busy` behavior that tracks the focused session's own
streaming state (thinking before any text arrives, talking once it
does — using the previously-unwired peep/talk sprite), eureka fires with
the actual knowledge title that was recorded, happy fires with the
task's own completion summary, and alarmed now means "this task needs
your OK" (an operator question was raised) rather than a fleet-wide
critical/signal event.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New Task now opens directly as an empty ChatThread (NewTaskChat) instead of
a separate compose screen, sized like a real task window. The Scope/Activity
context rail in a task window no longer renders until there's actually
something to show (touched entities, activity, or an open question),
avoiding an empty-placeholder sidebar on every new task. Also fixes the
chat input defaulting to several lines tall on window open, centers the
empty-chat greeting vertically, and gives floating windows the same
frosted-glass look as the desktop's task launcher card.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Audits and fixes ground-teleport/flat-fall/toss-momentum physics bugs,
fixes drag getting stuck via missing pointercancel handling, replaces
sprite-based speech bubbles with real HTML text/emoji bubbles, adds
drag-onto-icon "investigate" reactions and idle chatter, merges the
name badge and reaction bubble into one floating element, and caps the
bubble to one line with a teleprompter-style auto-scroll instead of
ellipsizing overflow text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements plans/2026-07-20-desktop-mascot.md. New code under
web/src/lib/mascot/ (types/sprites/render/state/behavior/actions/
stimuli + Mascot/MascotLayer/RadialMenu/NameDialog components) plus
CC0 sprite sheets at web/public/mascot/ (chicken + Onocentaur egg pack
+ reaction bubbles). MascotLayer is inserted into Desktop.svelte after
WindowLayer; <2-line integration.
Tamagotchi: egg -> chick -> adult lifecycle persisted to
localStorage['oikos-mascot'] (debounced 300ms). Egg hatches on first
naming (no timed incubation per implementation deviation). Chick/adult
wander, peck, sleep, blink autonomously via a weighted-random FSM; the
chicken walks above windows (ground line = highest window top edge
beneath its x, recomputed each tick from wmState; rides the ground when
the window beneath is dragged).
Interaction: draggable with flutter-fall physics on release mid-air;
plain click = pet (heart bubble + happy anim); right-click opens a
rounded-button radial menu (Interact/Care/Identity/Debug nested groups)
mirroring the desktop's own right-click menu styling; auto-flips above/
left near screen edges.
Awareness: stimulus bus subscribes to chat.ts streaming, activity.ts
activityLog (knowledge-entry diff), events.ts liveEvents (critical/
signal -> alarmed, execution -> happy), with priority+cooldown gating.
Egg-stage reactions are suppressed. Reaction bubbles are anti-aliased.
Sprite loop runs at ~60fps via setTimeout (not rAF) per GraphBackground
convention, dt clamped to 100ms; position via transform: translate3d
+ will-change: transform for compositor-friendly motion. Z-index
ordering: WindowLayer z-40 < MascotLayer z-[45] < desktop context menu
z-50 < RadialMenu/NameDialog z-[60].
Docs: plan + docs/mascot/README.md (MBSE subsystem model) updated to
Implemented with a deviations note covering hatch-on-naming, PNG-sheet
art, button-column radial menu, 60fps loop, egg-reaction suppression,
and window-walking ground model. VERSION bumped 0.7.13 -> 0.8.0.
Classifier now unwraps pct exec / qm guest exec / bash -c / sh -c / sudo
and env-var assignments before classification, so read-only inspection
wrapped in pct exec no longer escalates to config_mutation. curl GET
(default method, no -d/-F/-T/-o/>) is read-only. Eliminates the three
duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) that bounced
off the classifier for the same goal.
New classify_command MCP tool: command-scoped preflight that returns the
exact risk class run would assign. Documented in SOUL.md with guidance
to pre-classify before run when the verdict is uncertain.
set_goal surfaces prior partial/failed sessions from the last 24h so the
agent picks up the thread instead of rediscovering it.
completeTask auto-closes in-flight plan steps (pending/running -> done
on success, skipped on partial/failure), so one-step plans no longer
need the per-step running->done dance right before completion.
Migration 021 adds blocker + closed_at to agent_sessions. completeTask
sets closed_at once and derives a structured blocker reason
(approval_timeout, user_abandoned, classifier_overreach, model_refusal,
tool_error, ...) from the last assistant message.
/sessions list now carries message_count, tool_call_count,
duration_seconds (server-side aggregates — no more N+1 transcript
fetches to audit a fleet). GET /sessions/{id} returns both metadata
and messages. New query params filter + paginate: outcome, status,
entity_id, blocker, since (RFC3339 or Go duration), cursor, limit.
Titles now prefer the goal when set; sessions without a goal fall back
to the first assistant text.
New GET /sessions/{id}/tool_calls flat view for audit scripts.
Plan: plans/2026-07-20-session-review-ten-sessions.md. VERSION 0.7.12 -> 0.7.13.
Design-only (no code yet): an MBSE subsystem model for a chicken mascot
that roams the desktop shell, is draggable, opens a Sims-style nested
radial menu, and has a tamagotchi lifecycle (egg -> chick -> adult) that
reacts to real app activity (chat streaming, knowledge-graph writes,
signals). Everything (animations, autonomous behaviors, menu actions,
environment reactions) is scoped as a data-driven registry for easy
extension.
- docs/mascot/README.md: subsystem Model conforming to docs/mbse's
Holt-based Framework — mission/boundary, requirements, structural view
(module registry map), behavioral view (behavior FSM + lifecycle state
machines + a stimulus sequence diagram), interfaces view (which web
stores it observes, read-only), extension guide, verification view.
- plans/2026-07-20-desktop-mascot.md: the concrete file-by-file
implementation plan for web/src/lib/mascot/ derived from the model,
with an ordered build sequence and a manual browser verification
checklist.
- Indexed both in docs/index.md and plans/index.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The taskbar's gear icon reopened the full-page "Connect to Oikos" screen
even once already connected. Split that: Config.svelte stays as the
first-run/unconfigured screen; a new Settings app (windowed, like Tasks or
Operations) now handles in-session changes, with a section list (Connection,
Appearance) built to grow — future settings are one more entry, not a new
screen.
- pages/Settings.svelte: Connection (server URL/token/Authentik, reusing
config.ts + oidc.ts) and Appearance (Terracotta/Carbon picker) sections.
- apps.ts: registered as a normal desktop app.
- Taskbar's gear button now opens the Settings window; removed the
onOpenConnection prop threaded through App -> Desktop -> Taskbar, since
Settings' "Forget saved connection" (clear config + reload) replaces it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace hand-rolled pointer-resize logic (TaskContextPanel's 3-way vertical
split, SessionChatWindow's rail, ChatThread's message/input split) with
svelte-splitpanes, themed onto the app's existing border/primary tokens.
- TaskContextPanel: Scope/Plan/Event-log sections collapse to a fixed header
height and restore their last size on reopen.
- ChatThread: input area is now a separate resizable pane, clamped to a
measured one-line minimum and a 45% max, instead of a fixed max-h textarea.
- Send button restyled to sit inside the input's corner (Claude-style),
swapping the up-arrow for a corner-down-left return icon.
- Adds a $app/environment shim + optimizeDeps exclude, since
svelte-splitpanes assumes SvelteKit and this is a plain Vite app.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the sidebar + hash-routed page shell with a desktop metaphor:
draggable app icons, apps opening as floating wmkit windows, a centered
"What should Nomos do?" task launcher, and a bottom taskbar showing all
open windows plus a system tray.
- New app registry ($lib/apps.ts) — adding an app is one entry, nothing
else to touch.
- New desktop shell components (Desktop, WindowLayer, DesktopIcon,
Taskbar, TaskLauncher) under $lib/components/desktop-shell/.
- Icon positions are a persisted, collision-avoiding grid ($lib/stores/icons.ts).
- Window layout persists across reloads (wmkit persist), with
drag-to-maximize, F6 window cycling, and now a right-click desktop menu
(cascade/tile/show desktop/reset icons) plus Cmd/Ctrl+Z undo/redo for
window moves, resizes, and closes.
- Taskbar buttons get a hover-close and self-correct their title once a
new task's real goal is known.
- New task windows (desktop launcher and the Tasks app's "New task"
button) open as a window, not a dialog, and hand off to the real
session window once the backend assigns an id.
- Fixed a real gap along the way: GET /sessions/{id} couldn't tell
"session deleted" from "session has no messages yet" (both returned
200 with an empty list) — cmd/nomos/main.go now checks existence and
404s, so a stale/persisted task window shows "Task not found" instead
of a misleadingly empty, live-looking chat.
- Test coverage for the new pure logic (icon placement/collision
avoidance, app registry id helpers) plus a vitest matchMedia polyfill
needed to import anything touching the theme store.
Deletes the now-superseded sidebar shell, MinimizedWindowsBar, and the
standalone Chat/EntityDetail pages (folded into the window layer).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Clicking a task now opens it as a wmkit floating window (like entity
windows already do) instead of navigating away from wherever you were.
Several task windows can be open and actively streaming at once, each
fully independent — no "which one's on screen" guard needed, since
each window owns its own store bundle:
- chat.ts: chatFor(sessionId)/loadSessionChat/sendSessionMessage give
each window its own messages/streaming/connectionState, alongside
the existing singleton path the main Chat page still uses unchanged.
- workspace.ts: same split for plan/questions/touched/health-diffs
(workspaceFor/startSessionWorkspace), each with its own live-event
watermark since several windows can watch the same event stream.
- activity.ts: activityLogFor(sessionId) mirrors the global derivation.
SessionGraph.svelte, OperatorQuestion.svelte, and ActivityTimeline.svelte
were converted from store-importing to prop-driven (matching the new
ChatThread.svelte, extracted from Chat.svelte's transcript/input so both
the main page and task windows share one implementation instead of
duplicating markup/styling) so each can render either the global
"current session" or a specific window's session.
Also: minimized-window taskbar chips now cap at a max width with
middle-ellipsis truncation instead of growing unbounded, and the
window header's title/action-button row is fixed to genuinely match
heights (not just share a center point) for more robust alignment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the Fleet/Network/Identity/Knowledge category tabs (which
scoped entity fetches server-side) with a single "Types" multiselect
shared by both the table and graph views — both now fetch the whole
entity set (paginated via the new fetchAllEntities) and filter
client-side, defaulting to fleet's types. Table and graph also share
one search/highlight field instead of two separately-labeled ones.
Along the way, fixed a real bug the wider entity set exposed: the
treegrid's parent/child grouping fired one fetchGraph call per
candidate root entity, fine for the old ~50-entity fleet scope but an
ERR_INSUFFICIENT_RESOURCES flood once scoped to the full ~1700-entity
set. Replaced with a single whole-graph fetch, deriving parent/child
pairs from its edges client-side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Quadratic-bezier edges instead of straight lines, and drop the
auto-refit-on-load that caused a jarring zoom/pan snap once the force
simulation settled. Also namespace each graph's dot-grid pattern id
with a per-instance uuid — multiple SessionGraph instances can now be
mounted at once (one per open task window), and duplicate SVG ids
silently blanked out every graph's background but the first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Session-review implementation for the three sessions audited in
plans/2026-07-18-session-review-three-sessions.md. v0.7.11 → v0.7.12.
P0.1 — retry cap + investigate-before-retry (cmd/nomos/retrycap.go,
agent.go): after 3 identical failing run calls in a single turn, refuse
to dispatch the call again and return a directive to investigate *why*
(ps/strace/lsof) or surface the blocker. Per-turn scope so a fresh turn
after the operator responds can retry once more. Session 1e9c7691's 20+
identical chown retries (knfsd held a kernel lock on the exported NFS
dir) is the direct motivation.
P0.2 + P1.8 + P2.10 — SOUL.md guidance: hung command is not a failed
command (investigate before retry); ask before proposing a multi-step
migration; multi-goal sessions summarize the arc not just the last goal.
P1.3 — two new runbook entities in seeds/knowledge.yaml:
- nfs-exported-dir-mutation-hang (the knfsd fchownat lock procedure:
killall → exportfs -u → mutate → exportfs -a → verify)
- netbird-mgmt-oidc-race-after-upgrade (docker restart netbird-mgmt
after ~30s for the traefik/authentik OIDC race)
P1.4 — setGoal emits task.superseded event when prior goal is overwritten
by a different goal (store.go, TestSetGoal_SupersededEvent). Session
55927f0a had two set_goal calls with the first silently abandoned.
P1.5 — inspect_path MCP tool: runs mount/df/ls/stat for one path across
up to 8 targets in one parallel call, replacing the 15+ run-call
fact-gathering fan-out sessions 1 and 2 each spent on cross-target path
tracing (tools.go, server.go: inspectPathAcrossTargets, inspectOneTarget).
P1.6 — vm: target support in run via qm guest exec (no more SSH-hop
with nested quoting). Extracted shared resolveProxmoxHostSlug for
LXC + VM, with hosts-relationship fallback when attributes.host is
absent (server.go, tools.go). Session 55927f0a's SSH-hop workarounds
for vm:zimaos are the direct motivation.
Deferred (documented in plan): P1.7 (approval window auto-extend on
timeout) and P2.9 (long-running command PENDING detection) — both
addressed at lower cost by the retry cap. Session 3's poll-after-timeout
pattern already works; the cap protects against the failure mode.
Every place that showed entity detail (Knowledge Base's right sidebar,
the EntitySheet drawer used by Knowledge and the chat session graph,
the standalone /entity/:slug page) now opens the entity in its own
floating, draggable, resizable window instead — several can be open
side by side, and clicking a relation inside one opens another,
building up a stack. Windows are managed by one global wmkit instance
(new $lib/stores/windows.ts + $lib/components/EntityDesktop.svelte,
mounted once in App.svelte), themed with the app's own card/border/ring
tokens rather than wmkit's bundled themes (app.css).
- Delete EntitySheet.svelte (redundant) and the KnowledgeBase resizable
detail pane; row/graph-node click handlers now call
openEntityWindow(slug) instead of setting local sidebar state.
- SessionGraph (chat's "Scope" mini-graph): clicking a node opens its
window directly instead of a click-through mini-detail panel with
its own resize handle and "Full detail" button — that whole
subsystem is now dead and removed. Node highlight ring is kept
(still useful to see what you last opened) and now clears itself via
an effect watching the shared window-manager store, so closing a
window drops the highlight instead of leaving it pointing at nothing
— same fix applied to Knowledge Base's row highlight.
- Compact the entity-detail panel's padding (container + each
DetailSection) now that it's typically viewed in a small window
rather than a full-height sidebar.
- Fix KnowledgeBase's browse pane losing its flex-1/min-w-0 (and thus
full width) when the wrapping single-child div around it was removed
along with the old detail-pane split.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
import.meta.env (used by main.ts's dev-token auto-config) was untyped
since that landed — vite-env.d.ts never referenced Vite's client types.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The clickable-button variant of relationRow was missing the truncate
class that the read-only span fallback already had — long slugs (task
UUIDs, exec IDs) rendered at their full pre-truncated length inside a
shrink-only flex item, overflowing the narrow detail sidebar and
wrapping to extra lines. Give both sides flex-1 + truncate so they
share the row's width evenly and always stay on one line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fetchGraph({root, depth:1}) is backed by blast_radius, which only
walks outgoing edges — so it could never surface an edge some other
entity points at this one (e.g. host:hubris —hosts→ lxc:sophia) unless
that other entity happened to also be reachable going forward from
here. The Outgoing/Incoming split was filtering correctly, but
"Incoming" was starved of data by construction.
Switch to GET /entities/{id}/relations?direction=both — a dedicated
endpoint that matches on source_id OR target_id directly — via a new
fetchEntityRelations(). Simplifies the incoming/outgoing derivation
too, since every relation returned is now actually incident to the
entity (no more sibling-edge filtering needed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Knowledge Base / Fleet browsing:
- EntityTable renders as a treegrid (arbitrary depth, expand/collapse,
ARIA row/level/expanded), grouped by parent-child relationships
derived entirely from the live ontology graph (cardinality ->
direction; typeDepth specificity for ties) rather than a hardcoded
relationship list — see loadFleetGrouping in KnowledgeBase.svelte.
- Fold Services and Storage categories into Fleet (services/pools/
volumes/datasets now nest under the compute entity or pool that
provides/contains them instead of having their own browsing tabs).
- Drop `cluster` entities from Fleet browsing so a host's `located-at`
(site) relationship wins the tree-parent slot without needing a
hardcoded priority override — member-of simply has no valid target
left to point at.
- Add a "show destroyed/inactive" Switch (default off) filtering on
entity.state, replacing an always-on checkbox.
Entity detail panel:
- Split the Relations section into Outgoing/Incoming groups (relative
to the viewed entity), and scope the section's count to edges
actually incident to it rather than the whole depth-1 neighborhood.
Dev experience:
- Auto-fill the SPA's token from the dev server's own OIKOS_API_TOKEN
(vite.config.ts define + main.ts, dev-only, only when unconfigured)
so the "Connect to Oikos" prompt doesn't reappear on every reload.
- .claude/launch.json: autoPort, since port 5173 is often already
claimed by another worktree's dev server.
Adds ui/checkbox and ui/switch (bits-ui primitives, following the
existing shadcn-svelte wrapper pattern) and fetchOntology()/
RelationshipTypeDef to api.ts. Also fixes a missing types.ts import
in api.ts (ChatEvent/MessageContent) that predates this branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
req.Params.RelType is *[]string; passing the nil pointer straight
through as a pgx query arg (both in the blast_radius() call and in
ListGraphEdges) panics because pgx can't infer the array element type
from a nil *[]string, only from a concrete (possibly nil) []string.
Dereference once up front instead. Also affected the sqlc-based
ListGraphEdges path added by the R3 refactor, which had the same bug.
Add a regression test for GET /api/v1/graph?root=X&depth=N with no
rel_type.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
R13 — on-client path reconciliation:
- AGENTS.md: 6 occurrences of /opt/homelab-context/ → /opt/homelab/
(sections 1, 2, 5, 7)
- .agents/NOMOS.md: 2 occurrences of /opt/homelab-context/ → /opt/homelab/
- CLIENTS.md already used /opt/homelab/ — now consistent across all docs.
The repo is still named 'homelab-context' (git remote), it just clones
to /opt/homelab/ on enrolled clients per CLIENTS.md.
R14 — golangci-lint/staticcheck/govulncheck tooling:
- .golangci.yml (new): config enabling govet, staticcheck, ineffassign,
unused, errcheck, gosimple, typecheck, misspell, revive. Excludes
generated code (internal/httpapi/gen/, internal/db/sqlcgen/) and
relaxes errcheck in test files.
- Makefile: split 'lint' target into vet, golangci, govulncheck subtargets.
Each checks if the tool is installed and prints install instructions
if not. 'make lint' runs all three.
- CI already had golangci-lint-action + govulncheck (both advisory);
the action auto-discovers .golangci.yml.
10 routes are registered manually on the chi router in server.go rather
than generated from openapi.yaml. Added a 'Non-OpenAPI routes' comment
block at the top of NewHandler listing each route with its structural
reason for the carve-out:
- Auth/infra: /healthz, /api/v1/auth/oidc-*, /oidc-callback — bypass
auth middleware or aren't JSON API
- SSE override: /api/v1/events/stream — re-registered for Flush()
- Ad-hoc aggregations: /knowledge/recent, /knowledge/content/{id},
/activity/recent, /activity/session/{id}, /learning/timeline,
/learning/trend — derived shapes with no schema type yet
Updated .agents/dev/CONTRIBUTING.md §OpenAPI codegen with the carve-out
policy: if an ad-hoc route stabilizes, promote it to openapi.yaml with a
proper schema and migrate the serve* function to a strict handler.
5 warnings → 0:
1. ActivityTimeline.svelte:103 — replaced deprecated <svelte:component
this={icon}> with direct dynamic component rendering ({@const IconComp
= icon}<IconComp />). In Svelte 5 runes mode, components are dynamic by
default; <svelte:component> is unnecessary.
2. DetailSection.svelte:18 — 'let open = (defaultOpen)' captured only
the initial value. Changed to (false) + to sync with
defaultOpen prop changes.
3. EntitySheet.svelte:10 — 'let currentSlug = (slug)' had the same
issue. Changed to <string|null>(null) + (the was
already there, now the initial value doesn't reference the prop).
4. theme.svelte.ts:23 — 'applyClass(current)' at module level referenced a
variable, capturing only the initial value. Changed to apply the
plain storedTheme() result for initialization; setTheme() already calls
applyClass() on changes.
5. Chat.svelte:326 — unused CSS selector '.prose-chat
:global(:first-child):is(h1,h2,h3)' replaced with explicit
:global(> h1:first-child) etc. (the :first-child pseudo wasn't matching
because the scoped wrapper div is the actual first child).
Build is now warning-free.
Rewrote .agents/domains/knowledge/schema.md and .agents/shared/llm-wiki.md
which described the deleted Python substrate (bin/homelab, oikos/cards/,
oikos/ledger.py, root inventory.yaml, knowledge/sources/, get_page/
search_docs MCP tools). Now reflect ADR 0003: Postgres DB is the single
source of truth for structured data and narrative knowledge; seeds/*.yaml
are bootstrap+DR manifests (content-hashed via seed_versions); archive/
knowledge/ is the frozen legacy wiki; MCP search_knowledge/get_entity_
knowledge replace get_page/search_docs.
Swept substrate refs in .agents/shared/{writing-style,page-templates}.md
and .agents/domains/operations/schema.md: bare inventory.yaml ->
seeds/inventory.yaml; knowledge/sources/ -> archive/knowledge/sources/
(historical); get_changelog/oikos/ledger.py -> DB audit trail / structured
document changelog field; HERMES -> Nomos.
Root inventory.yaml (618-line Python-era file superseded 2026-07-07 by
seeds/inventory.yaml) replaced with a deprecation stub pointing to the seed
and DB. Kept as a stub rather than deleted because AGENTS.md §1/§2 still
point clients at /opt/homelab-context/inventory.yaml; full on-client path
reconciliation deferred to R13.
Flagged export gap: oikos export regenerates seeds/{ontology,inventory,
policy}.yaml but NOT seeds/knowledge.yaml — API-added knowledge lives only
in the DB until hand-edited into the seed.
VERSION 0.7.7 -> 0.7.8. Plan R5 marked done.
Deleted 8 genuinely unused sqlc queries (no inline equivalent):
- UpsertCurrentRelationship, ListEntitiesCapped, ListEntityStatus,
UpdateSignalState, InsertClassification, InsertFeedback, InsertSkill,
UpsertCurrentRelationship — all had zero call sites.
Migrated 9 inline raw SQL sites to use sqlc queries:
- GetOntology (impl.go): ListEntityTypes, ListRelationshipTypes,
ListLifecycleDefs — replaces 3 raw pool.Query blocks with typed sqlcgen
calls, eliminating manual row scanning.
- EndRelationship (phase3.go): EndCurrentRelationship — replaces tx.Exec
with sqlcgen.New(tx).EndCurrentRelationship.
- checkPrecondition (impl.go): GetEntityStatus — replaces tx.QueryRow +
manual Scan with sqlcgen.New(tx).GetEntityStatus.
- GetEntityRelations (impl.go): ListEntityRelations — replaces raw pool.Query
+ scanRelationships helper (now deleted).
- GetGraph (impl.go): ListGraphEdges — replaces raw pool.Query +
scanRelationships.
- resolveEntityID (impl.go): GetEntityBySlug/GetEntityByID — replaces
raw pool.QueryRow + Scan.
- createApproval (mcp/server.go): InsertApproval — replaces raw pool.Exec
with sqlcgen.InsertApproval.
Deleted scanRelationships helper (was only used by the two migrated
graph queries above).
Regenerated sqlcgen — also picks up stale model updates (AgentSession,
SessionPlanStep, SessionQuestion, etc. from recent migrations).
Documented the carve-out in .agents/dev/CONTRIBUTING.md §SQL conventions:
sqlc is the default; raw pool.Query/Exec is reserved for LISTEN/NOTIFY,
dynamic WHERE builders, blast_radius(), and COPY.
go vet, build, httpapi/mcp/db tests all pass. -383/+170 lines.
Tool-renderer registry (21 files, ~1.5k lines):
- src/lib/tool-renderers.ts — registry + getToolRenderer (exported, never
imported anywhere)
- src/lib/renderers/index.ts + 10 .ts registrars + 10 .svelte components
- main.ts: removed the requestAnimationFrame(() => import('./lib/renderers'))
that was the only thing keeping the dead subsystem alive
Dead components (never imported):
- ToolCallGroup, PlanProgress, GoalHeader, InlineApproval, SessionDigest
Dead store exports (written, never read):
- context.ts: pendingApprovals writable (+ Approval type import)
- events.ts: connectionState writable (+ its .set() calls)
Dead API surface:
- api.ts: SessionDigest interface + fetchSessionDigest (only caller was the
dead SessionDigest.svelte)
Dead npm deps:
- mode-watcher (0 imports; superseded by stores/theme.svelte.ts)
- @internationalized/date (0 imports)
Also: fix stale comments referencing deleted symbols, update plan R1/R2
status. Build clean (4683 modules, down from 4706; one Svelte 5 warning
gone — the dead HealthSummary.svelte was emitting state_referenced_locally).
- internal/httpapi/stubs.go: delete — 5-line comment-only orphan file with
no declarations; its own comment said the stubs live in phase3.go.
- internal/notifier/notifier.go: delete VerifyApprovalToken — zero call
sites; phase3.go:DecideApproval reimplements the check inline (noted as
dead in docs/mbse). hashToken stays (used by generateApprovalToken).
- internal/checkdefaults/defaults.go: unexport ResolveHost, ForEntityType,
ShortSlug, DefaultInterval — only called within the package. Ensure stays
exported (called by internal/db/seed.go).
go vet, go build, and affected tests pass.
Four cross-linked documents under docs/mbse/, structured after Jon Holt's
Systems Engineering Demystified (2nd ed.): Framework = Ontology + Viewpoints,
producing a Model made of Views.
- framework.md — the Ontology (SE meta-concepts + Oikos's domain ontology)
and an 11-entry Viewpoint catalog (two repeating: Component, Ontology).
- README.md — the Model's 9 concern-based Views (mission, requirements,
functional/physical architecture, interfaces, behavior, V&V, risk, roadmap).
- components.md — 8 per-component Views going one layer deeper into each
running part of the system's own internal structure.
- ontology.md — 4 Views on the domain ontology itself: entity type
hierarchy (split into 9 digestible per-domain diagrams), full relationship
catalog, lifecycle state machines with their requires: gates, and concrete
population.
Grounded in direct verification against source (grep/read), not just
existing docs — every finding is graded verified vs. per-research-pass.
Surfaced several real, previously undocumented findings along the way:
the policy kill-switch (global.auto_act/never_auto_act) is checked only by
dead code and an unstarted actuator package, so it doesn't gate the live
run path; internal/actuator and internal/learning are compiled but never
started by any process; the relationship catalog grew from 34 to 47 types
since ADR-0014; and task has no registered lifecycle_defs entry despite
having a documented, code-enforced state machine.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The approval entry type was removed from ActivityEntry upstream; this
case/import were unreachable leftovers after merging that change in.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Plan/Activity panels: humanize step titles, richer icons, empty states
matching Scope's illustration style, pretty-printed expandable detail
- Activity log renamed to Event log; every step now expandable
- Chat: middle-truncate header title, remove redundant task-list rail and
header stat cluster (duplicated in the sidebar), simplify markdown styling
- Fix --font-mono actually being a monospace font (was aliased to DM Sans)
- Replace rotating loader-circle spinner with a smoother fading-blade Spinner
- SessionGraph entity detail panel: resizable and self-clamping against its
live container size (was overflowing into sibling sections), close button
- Dev launch config: fetch bearer token from the running api container so
`npm run dev` works against the local compose stack without a hardcoded
secret in a tracked file
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The auto-complete fired when the agent hit the P5 approval gate — it
queued a config_mutation run for approval, the P5 gate blocked further
runs, the turn ended, and auto-complete closed the session as 'partial'.
The operator's approval would then land on a dead task.
Fix: hasPendingApprovals check — if the session has any executions in
pending_approval state, skip auto-complete. The session stays in
'executing' until the operator approves (or denies).
VERSION 0.7.5 → 0.7.6
The len(pending)>0 path still injected a brief note saying 'execution(s)
are now running' — the model saw this, thought work was being done for
it, and no-op'd (finish_reason=stop, content_len=0). Same confusion as
the len(pending)==0 case, just from the other branch.
Fix: both assent paths are now fully silent. No system note at all. The
model sees 'go ahead' in the replayed history and responds naturally.
Also removed chat_assent tool_use/tool_result emit events. These were
persisted in the transcript and confused the model on replay — it saw
its own 'tool calls' (chat_assent) and thought it had already acted.
VERSION 0.7.4 → 0.7.5
The auto-complete safety net required hadEntityWriteback to be true,
which meant sessions where the agent did the work but forgot to call
update_entity_attributes stayed stuck in 'executing' forever.
Relax: auto-complete fires if the agent did discovery (ran run),
regardless of writeback. If writeback happened → success; if not →
partial (honest: work was done but knowledge graph not updated).
VERSION 0.7.3 → 0.7.4
When the chat handler approves a pending execution via chat-assent, the
execution completes in ~2s. The continuation worker detects the completed
execution and calls resumeSession — while the chat handler is still
processing 'go ahead'. Two concurrent LLM calls for the same session cause
empty responses (finish_reason=stop) and race conditions.
Fix: mark the execution as continued immediately after chat-assent grants
it, so the continuation worker skips it. The chat handler will drive the
continuation itself (the model sees 'go ahead' and executes the plan).
VERSION 0.7.2 → 0.7.3
The assent pre-processing injected verbose system notes ('the operator
approved... they are now running... you MUST continue...') on top of the
replayed user message ('go ahead'). The model saw both, latched onto
'now running', concluded the work was being done for it, and no-op'd
(finish_reason=stop, content_len=0) — leaving the session stuck in
'executing'.
Root cause: the model already sees 'go ahead' in the replayed history
(the user message is saved to the DB before chat() is called, and
getRecentMessages replays it). The system note was redundant AND
confusing — it told the model work was 'running' when it wasn't.
Fix:
- len(pending)==0 (plan-proposal approval): open assent window silently.
No system note. The model sees 'go ahead' and responds naturally.
- len(pending)>0 (actual pending executions): brief note naming the
specific execution IDs that were approved ('don't re-request those').
No 'continue the plan' directive — the model knows to continue.
VERSION 0.7.1 → 0.7.2
P1: add docker compose (logs|ps|top|config|images|port|cp) to read-only
allowlist. docker compose logs was classified as config_mutation, causing
individual approval cards for read-only inspection commands.
P2: remove approval entries from activityLog. They were always status=running
and never transitioned to done (the derived store builds from tool-call
text, not execution status), causing AgentIndicator to latch onto a stale
'Approval: ...' entry and never clear — even after the session completed.
P3: remove InlineApproval from Chat.svelte. The green 'Completed in 1s on
lxc:...' boxes were noise in the chat stream. Approval UX belongs in the
Operations page (already has it via Ops.svelte), not inline in the chat.
P4: stale execution cleanup. Startup sweep (mark >1hr non-terminal as
cancelled) + 5-min periodic sweep (mark >10min non-terminal as cancelled).
98 orphaned executions accumulated from eval testing (39 running from
apt_upgrade:audit timeouts, 19 pending_approval, 3 approved).
P5: refuse second config_mutation run when an approval is already pending
for the session. Without this, the agent queues N individual approvals
before the operator can respond — confirmed in session 20757eb9 (two
approval cards for what should have been one plan-level approval).
VERSION 0.7.0 → 0.7.1
The agent legitimately runs 20+ diagnostic commands for a config_mutation
task (reset service, re-run backup, verify, check logs). Cap of 8 was
too strict.
The agent often skips update_plan_step bookkeeping (leaving steps
pending/running) but still does the work + writeback. The strict
allPlanStepsTerminal check missed these cases.
Add path (b): if the agent did discovery (ran `run`) AND wrote back
(update_entity_attributes/create_relationship), auto-complete. D.1 already
enforces writeback before completion — if writeback happened, the work
is done.
The #1 remaining model reliability gap: the agent does the work (proposes
plan, executes all steps, writes back) but forgets to call complete_task,
leaving the session stuck in 'executing'. The eval showed 3/8 failures
with this pattern.
Fix: autoCompleteIfPlanDone — a structural safety net that fires at both
chat exit paths (normal completion + maxIterations). If the session has a
goal, the agent didn't call complete_task, and ALL plan steps are in a
terminal state (done/failed/replaced/skipped/blocked), auto-complete with
the agent's final text as the summary. Mirrors autoCompleteTrivialTask
but for structured tasks where the work is provably done.
Also: bump maxLLMRetries from 2 to 3 (complex multi-turn flows benefit
from one more retry on empty responses).
The assent system note said 'Do not re-request or call run again for
these' — the LLM interpreted this as 'don't call run at all' and produced
empty responses (finish_reason=stop, content_len=0) until retries were
exhausted, leaving the session stuck in 'executing'.
Fix: rewrite both assent notes (pending-approval path and pure-plan-approval
path) to be directive about WHAT TO DO NEXT: call update_plan_step(running)
then run for each remaining step. The 'don't re-request' guidance is now
scoped to 'THOSE SPECIFIC' executions, not all run calls.
Also bump maxLLMRetries from 2 to 3 — the empty-response flake on complex
multi-turn flows benefits from one more retry.
reopenSession was replacing plan steps on every follow-up message —
including approvals ('go ahead') — which destroyed the plan the operator
just approved, leaving the agent unable to track step progress and looping
run calls until maxIterations.
Fix: setGoal is the explicit signal for 'new sub-task' (the agent calls
it at the start of each follow-up direction). Step replacement now happens
there, not in reopenSession. An approval ('go ahead') does NOT call
set_goal, so the plan stays intact and the agent can execute + complete
it.
The check len(lastAssistantCalls) == 0 was too restrictive — it only
fired when the assistant had ZERO tool calls. But propose_plan + pre-plan
research are tool calls, so the assent window never opened when the
operator said 'go ahead' after a plan proposal. The agent then tried to
execute config_mutation run calls without the assent window, they queued
for approval, and the turn deadlocked.
Fix: check len(pending) == 0 (no pending APPROVALS) instead of
len(lastAssistantCalls) == 0 (no tool calls at all).
A follow-up on an executing session (first turn didn't complete_task) is
still a new direction — the old plan's steps must not block the new one.
Previously reopenSession was a no-op for executing sessions, leaving done
steps that caused errPlanInFlight on the next propose_plan call.
proposePlan: mark pending steps as 'replaced' instead of DELETE, so the
generation counter (MAX+1) sees prior generations. Without this, a first
plan that was proposed but never executed would be wiped, resetting the
counter — a follow-up's plan would look like generation 1 instead of 2.
plan-always-readonly: raise max_run_calls from 3 to 6 (agent inspects
thoroughly).
SOUL.md step 4: all-read-only plans skip the approval wait and execute
immediately. Only config_mutation/destructive steps need operator approval.
set_goal + propose_plan return text updated to match.
Fixes 3/4 eval failures where the agent proposed a plan then waited
for approval on a read-only task.
plan-always-readonly: prompt now demands live systemd timer inspection,
not just DB lookup. Added calls_tool: run assertion.
iteration-followup: added 'go ahead' as second followup so the
config_mutation plan gets approved and can execute.
iteration-readonly: replaced nonexistent lxc:prometheus with lxc:dns,
keep it read-only so no approval needed.
P1 plan-first: run handler refuses without propose_plan (structural gate,
not SOUL.md prose). Plan window decoupled from set_goal — config_mutation
auto-run only on operator approval (assent window). Closes the approval-free
config_mutation hole confirmed in session d0d562e0.
P2 iteration: reopenSession flips terminal→executing, marks prior plan steps
replaced, clears outcome. proposePlan excludes replaced from in-flight check,
bumps generation. A follow-up on a completed session starts a new sub-task
with a fresh plan — no more errPlanInFlight dead end.
P3 reasoning: accumulate per-iteration text into the persisted row instead
of overwriting with the last text event. Reload shows intermediate thinking,
not just the final summary.
P4 read-only allowlist: add find, tree, locate, systemctl list-timers/
list-unit-files/show, timedatectl, hostnamectl, systemd-analyze, rclone
ls/lsl/md5sum/check/cryptcheck. Fixes the find misclassification from
d0d562e0.
P5 eval harness: new assertion kinds (proposes_plan, plan_before_run,
plan_generations), multi-turn followups, fetch /sessions/{id}/plan. Four
manifests under evals/.
P6 SOUL.md: strip degenerate-case carve-out, add ITERATE step, update
set_goal guidance.
VERSION 0.6.0 → 0.7.0
- Terracotta (light) and Carbon (dark) themes with toggle
- Inknut Antiqua headings, DM Sans body
- Dot grid background on EntityGraph and GraphBackground
- Theme-adaptive graph colors on EntityGraph
- Art Nouveau chat styling (borders, underlines, blockquote quotes)
- Bullet point styles in chat prose
- Task goal in header, rename Overview→Tasks, New Task labels
- Logo uses var(--primary) for theme awareness
The unrooted graph endpoint caps at 500 entities with ORDER BY e.slug, which fills the cap with exec:* rows and excludes every host/lxc/service/vm entity. Since edges require both endpoints in the node set (ANY/ANY), 99.9% of edges were dropped — 500 nodes but only 1 edge survived.
Fix: select the 500 most-connected entities (by relationship count descending) so the topology is preserved. Result: 500 nodes, 900 edges across all relationship types.
The OIDC fix was committed in 3b98097 by a concurrent session. The plan's
status block was stale ('not yet committed/deployed') — updated to reflect
it's done. No remaining open items in this plan.
Status: In Progress → Done. All 18 fixes (A.1-A.3, B.1-B.6, C.1-C.2, D.1-D.2,
E.1-E.2, F.1-F.3) shipped in commits 337d577 + 3de359b + dd3076a, deployed
to oikos-nomos-1 (v0.5.3). The golden eval harness (cmd/nomos/eval/) passes
4/4 conversations, validating the structural gates + the SOUL.md
consolidation. Also fixed a pre-existing tool-call doubling bug found by
the eval harness.
Only remaining open item: the OIDC token-refresh fix (PM addition, web/src/
lib/{config,oidc,events}.ts) — implemented, not yet committed/deployed.
Ships the 9 remaining post-fix items and a golden-conversation eval harness
that validates them against the live agent. All 4 evals pass.
SOUL.md (F.1, C.2, E.1):
- Consolidated three overlapping task-flow sections (MANDATORY TASK FLOW,
'Every chat is a task', 'AFTER EVERY TASK: WRITE BACK') into one. ~50
lines shorter. The operator's 'be more crisp' feedback.
- Added anti-patterns: don't re-execute on UI/sidebar complaints (C.2);
don't re-run fleet-wide audits when same-day knowledge exists (E.1).
- Updated approval vocabulary in step 4 to match tasks.go (approved/yes/
go/proceed/continue/ok/go ahead).
Tool-result strings (F.2):
- set_goal: tightened to 'Goal set. NEXT: pre-plan (read-only tools only).
Then propose_plan. Do not call run.'
- update_plan_step: added '(Advance with update_plan_step + run; do not
re-propose.)'
C.1 — completeTask rejects re-completion of a terminal session:
- Returns errTaskAlreadyComplete when status is already done/failed.
- The tool result directs: 'Task is already complete. Do not call
complete_task again. If the operator pointed out a UI/sidebar
inconsistency, fix it with update_plan_step...'
B.4 — Surface real model error text:
- chatWith's error event now includes finish_reason + refusal text:
'Nomos returned an empty or unusable response (finish_reason=length).
Retry or rephrase.' instead of generic 'empty response'.
- The resume-failed note already carried errText (B.3), which now has
the real context.
B.5 — Back off between resume retries (4s, 8s):
- resumeSession now sleeps before attempts 1 and 2 (exponential backoff).
A transient provider issue gets time to clear instead of 3 identical
calls in 3 seconds.
B.6 — Don't persist the empty placeholder as a visible bubble:
- If a chat turn ends with no text and no tool calls (model empty-response'd
and all retries failed), delete the placeholder row instead of persisting
an empty bubble. The error was already streamed via done+error=true.
E.2 — list_lxcs last-audited hint:
- The list_lxcs result now includes last_audited_at — the most recent
knowledge entry (tagged audit/update, or titled audit/update) linked
via an 'about' edge. The agent can see 'nextcloud — last audited today'
and skip re-running it.
Tool-call doubling bug fix (found by the eval harness):
- main.go + continue.go: the tool_use and tool_result events were both
appending separate entries to the persisted tool_calls array, doubling
every tool call in the transcript. Confirmed pre-existing (d9cdcee1,
v0.3.x era). Fixed: tool_use creates the entry, tool_result merges the
result into the same entry (matched by id). One entry per tool call.
Golden eval harness (cmd/nomos/eval/):
- A standalone Go program that loads YAML manifests of golden conversations
+ assertions, sends prompts to the chat endpoint, drains the SSE stream
(keeping the agent's context alive), and scores structural assertions
against the persisted transcript.
- 4 golden conversations covering: trivial read-only (degenerate case),
plan + proceed (the original duplication bug), UI complaint (no re-exec),
fleet audit (knowledge preferred over re-execution).
- Structural assertions only (tool-call sequences, plan steps, writeback,
completion) — text quality is model-dependent and not scored.
- Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest
cmd/nomos/eval/evals/*.yaml (~$0.10/run in OpenRouter credits).
Eval results (4/4 passed):
trivial_readonly: 2 tool calls, no plan, no run
plan_advances_on_proceed: 13 tool calls, propose_plan x1, writes back
ui_complaint_no_rerun: 12 tool calls, propose_plan x1, writes back
knowledge_preferred_over_rerun: 7 tool calls, search_knowledge x1, 0 run
Version 0.5.2 -> 0.5.3 (minor: eval harness + structural hardening).
D.1 (complete_task refused without writeback) and D.2 (propose_plan
auto-appends writeback step) shipped in 3de359b (v0.5.1), e2e-validated
against the live agent. The knowledge loop is now structurally closed —
no blockers remain. Remaining items (F.1, F.2, C.1, C.2, B.4-B.6, E.1,
E.2) are all friction/cosmetic.
The overview background graph and the Knowledge Base graph both rendered
empty because the SPA's OIDC access token expired (~5 min TTL) and was
never refreshed. fetchWithAuth called getToken() synchronously (no refresh);
ensureToken returned the stale token without refreshing; storeTokens
discarded expires_in; the resulting 401 made fetchGraph return null and
both graphs drew nothing, with no error surfaced.
- oidc.ts: track expiresAt from expires_in; getToken() returns null within
30s of expiry; ensureToken/initOIDC refresh instead of returning stale
tokens; isOIDCConfigured no longer claims configured on expired-only state
- config.ts: fetchWithAuth awaits ensureToken (refresh on demand), falls
back to static token if OIDC can't yield one, flushes OIDC session on 401;
sseUrl is async + refreshes before constructing the EventSource
- stores/events.ts: connect() awaits the now-async sseUrl
D.1 — complete_task structural gate:
- hadDiscovery(ctx, session) reports whether the session ran `run` successfully
against a live target (NOT get_entity/list_lxcs — those are DB lookups, not
new facts). A trivial Q&A that only calls get_entity is a degenerate case
and must NOT be blocked.
- complete_task with outcome=success is REFUSED when hadDiscovery && !
hadEntityWriteback. The refusal fires BEFORE completeTask runs, so the
session stays in 'executing' state and the agent must call
update_entity_attributes/create_relationship then retry complete_task.
An explicit failure/partial is allowed through (the agent is acknowledging
it didn't finish — no reason to force writeback).
- Replaces the prior advisory warning (5.5) which the agent consistently
ignored. The agent saw the warning and ended the task anyway; this gate
makes the writeback a hard prerequisite for success.
D.2 — propose_plan auto-append writeback step:
- When the agent proposes a plan whose steps don't mention
update_entity_attributes or create_relationship, D.2 appends a final
'Write back: update_entity_attributes + create_relationship +
upsert_knowledge' step before persisting. The result string tells the
agent it was appended.
- With the seq-order enforcement (5.6) and D.1's complete_task gate, the
agent must complete the writeback step (and actually call the tools) to
finish. Neither relies on the agent reading SOUL.md.
- Removed the old advisory writeback nudge from propose_plan's result
string — D.2 makes it structural.
- Updated the propose_plan tool description to state both gates crisply.
Verification:
- TestHadDiscoveryAndWriteback: hadDiscovery true only after a successful
`run`; false after failed run, get_entity, or no calls. hadEntityWriteback
true only after update_entity_attributes/create_relationship.
- e2e against the live agent (oikos-nomos-1, v0.5.1):
- D.2: agent proposed 3 steps (no writeback); D.2 auto-appended step 4
'Write back: update_entity_attributes + ...'. Result string said
'(appended a writeback step — your plan didn't include one; step 4)'.
- D.1: agent ran `run` (uptime on lxc:gitea), called complete_task, was
REFUSED ('Refused: this session ran run against live targets (discovery)
but did not call update_entity_attributes...'). Agent self-corrected:
called update_entity_attributes, retried complete_task, succeeded.
Knowledge loop closed end-to-end.
Version 0.5.0 -> 0.5.1 (patch: structural enforcement of existing intent).
Status was 'shipped & e2e-validated'; now reflects the commit (337d577),
push to main, and deploy to oikos-nomos-1 (v0.5.0) that followed the
e2e validation. D.1 (refuse complete_task without writeback) is the next
blocker.
Operator-reported bug: on 'proceed with the rest' the agent re-proposed the
plan, duplicating it in the sidebar. Root cause was a three-bug chain, not
one bug:
1. Trigger — model empty-response on 'proceed' (approval vocabulary didn't
list 'proceed', so the agent wasn't sure it was approved and no-op'd).
2. Amplifier — chatWith emitted 'error' without 'done' on empty response
(agent.go:370). The frontend's onComplete saw !receivedDone and
misclassified the model failure as a network disconnect, calling
handleDisconnect -> resumeSession.
3. Divergence — the reconnect note was generic ('report your state'), so
the agent re-proposed + re-executed instead of advancing the plan.
Fixes (shipped, e2e-validated against the live agent on oikos-nomos-1):
- A.2: proposePlan refuses re-proposal once a step has started (returns
errPlanInFlight). Drops the append-mode safety net (commit 5384499) that
was the direct source of the sidebar duplication. The agent must advance
with update_plan_step + run; the tool result directs it.
- A.1: proposePlan sets the 'generation' column on INSERT (migration 020
added the column + frontend grouping, but the INSERT never wired it).
- A.3: propose_plan tool description restated as a crisp contract (ONCE,
STOP and wait, REFUSES once a step started, advance with update_plan_step).
- F.3: approval vocabulary expanded to approved/yes/go/proceed/continue/ok/
go ahead; propose_plan result string tightened to an imperative.
- B.1: chatWith emits 'done' after 'error' on every terminal path via a new
emitError helper. The frontend now treats model errors as ended (not
disconnected), so no auto-reconnect -> resumeSession fires.
- B.2: reconnect/resume note carries the operator's last message + an
explicit 'advance the plan, do NOT call propose_plan again' directive when
a plan is in flight. Wired into all 4 resume entry points (reconnect,
/resume, idle-sweep, question-answer) via enrichResumeNote.
- B.3: resumeSession escalates the recovery note across its 3 attempts (final
retry: 'pick the lowest-pending step, mark it running, call run — do that
now') instead of 3 identical notes -> 3 identical empties.
Verification: TestProposePlan_RefuseInFlight replaces TestProposePlan_
AppendVsReplace. e2e conversations against the rebuilt container:
conv2 ('proceed with the rest') -> 0 propose_plan calls, plan stayed at
3 steps (was 6+ before), update_plan_step x5 + run x2 + complete_task.
conv3 (full plan, 'go ahead') -> apt-get update on lxc:dns auto-ran under
the plan window, update_entity_attributes writeback, clean complete_task.
nomos logs show zero reconnect/resume entries for the plan-proposing
sessions (the three-bug chain is closed).
Remaining (not in this commit): D.1 refuse complete_task without writeback
(next blocker), C.1/C.2, F.1/F.2 SOUL.md consolidation, B.4-B.6, E.1/E.2.
See plans/2026-07-14-post-fix-session-remainders.md.
Also: re-audit 2026-07-10-general-gated-execution.md — request_execution enum
retirement (60effcb) closes item 9; only auto-act revival (item 10) remains.
Version 0.4.1 -> 0.5.0 (minor: new structural behavior, not a bugfix).
SOUL.md: mandatory 6-step task flow at TOP of file, unmissable.
Agent MUST: set_goal → pre-plan (research only) → propose_plan → STOP
and wait for approval → execute (auto-run under plan window).
Backend:
- set_goal now opens plan window immediately (config_mutation auto-runs)
- set_goal result tells agent to do pre-plan + propose_plan, not run
- propose_plan result tells agent to STOP and wait for approval
- plan window value unified to 'active' (set_goal + propose_plan)
This prevents 23 individual approval popups — one plan approval instead.
- activityLog now detects 'requires approval' in tool results
- Adds approval entries with shield icon + description + execution ID
- Works for both run and remaining approval paths
Backend:
- proposePlan sets plan window in autonomy_settings (nomos:plan:<session>)
- run handler checks plan window — auto-executes config_mutation commands
within plan without per-action approval
- planWindowActive function in server.go
- Plan window cleaned up on completeTask (already covered by LIKE '%:' || )
Frontend:
- Removed 'Session graph' header bar
- Cooler empty states: Plan shows animated dots + 'Awaiting plan…',
Activity shows pulsing dots + 'Waiting for activity…'
- AgentIndicator now shows only during stream or when running tools exist
(not on session status=executing which never cleared)
- Activity timeline: oldest-first ordering (reads top-to-bottom naturally)
- Removed unused liveStatus derivation and currentTask import from Chat
- Plan: Phase A+B+C for activity gaps + plan-approve-once policy
- Tool calls in Activity timeline are now tagged with current plan step
- Indented entries show which step they belong to
- Step tracking via update_plan_step(status=running) tool calls
- Removed inline tool renderers from chat (health summary, fleet snapshot, etc.)
— all tool output now visible only in sidebar Activity timeline
- TaskContextPanel restructured into 3 collapsible sections:
Scope (graph), Plan (goal + steps + progress), Activity (timeline)
- Collapsed headers show compact live status: 'Graph', 'Step X/N', 'N actions'
- Sections are vertically resizable via drag handles
- Plan section shows goal inline + step list + progress bar
- GoalHeader and PlanProgress no longer rendered separately
- ActivityTimeline header moved to TaskContextPanel
- New ActivityTimeline: unified timeline in sidebar showing all agent actions
(goal, plan steps, tool calls, knowledge, completion) in reverse chron order
- activityLog derived store merges messages + planSteps + currentTask
- AgentIndicator stays in chat (thinking/working indicator), simplified props
- ToolCallGroup removed from chat — tools visible only in sidebar timeline
- SessionDigest replaced by ActivityTimeline
- PlanProgress restored in sidebar (conceptual steps, separate from timeline)
- New AgentIndicator component: replaces 3 separate indicators
(loading dots, ToolCallGroup summary, activity bar) with one
- Positioned as last item in message list — scrolls naturally
- Shows current tool action: 'Researching lxc:nfs-export…' etc
- Spinner during work, check on completion, X on error
- Fades out 3s after turn completes
- Activity bar, loading dots, statusLabel removed from Chat
- Continue button moved to sidebar session panel
- SessionDigest now includes live tool timeline, plan steps, knowledge
- ToolCallGroup compact: single-line with collapsible names only (no JSON)
- Activity bar moved to bottom of messages, smart scroll respects user position
- setGoal now sets status=executing (removed stuck planning state)
- PlanProgress merged into SessionDigest, removed from TaskContextPanel
- New toolTimeline derived store in chat.ts
- Activity bar moved to bottom of message list (before messagesEnd)
- Smart scroll: auto-scroll only during streaming or when near bottom
- Scrolling up pauses auto-scroll until next send
- Removed duplicate $effect block
- Plan: tool timeline in sidebar (plans/2026-07-14-tool-timeline-sidebar.md)
- VERSION file at repo root (0.3.0)
- vite.config.ts reads VERSION at build time, injects __OIKOS_VERSION__
- App.svelte shows version in sidebar tooltip + subtle text below logo
- AGENTS.md §9: every commit to main MUST bump VERSION
(patch=bugfix, minor=new features, major=breaking changes)
RegisterHook + e.Cancel() prevents Wails from destroying the WebView
when the window is closed. The app now hides to the system tray. Left-
click on the tray icon correctly restores the window.
- Tray 'Check for Updates' now checks immediately and shows dialog
- Dialog has 'Install' and 'Later' buttons
- /update/check endpoint on local server for SPA to query
The apiUrl configured on the Config page was lost when the webview
navigated away to localhost and back. Now it's included in the return
URL as ?desktop=1&apiUrl=...&token=...
SPA navigates to 127.0.0.1:18901/oidc/start, passing ret URL.
Go opens browser, waits for callback, saves token, returns HTML with
<meta refresh> back to Wails app with ?desktop=1&token=TOKEN.
main.ts extracts token from URL on reload.
SPA fetches /oidc/open (returns session ID immediately), then polls
/oidc/result every 500ms. Go server opens browser in a goroutine.
Webview never leaves the Wails origin. Token is saved to keychain and
returned through the poll response.
The webview navigates to http://127.0.0.1:18901/oidc/open?apiUrl=...
The Go server opens the system browser to Authentik, waits for callback,
exchanges code for token, saves to keychain, then redirects the webview
back with ?desktop=1&token=TOKEN. main.ts extracts the token from URL.
The local HTTP server approach (fetch to 127.0.0.1) doesn't work in the
Wails webview. Simplify: use window.open() to launch OIDC in the real
browser. After authentication, the callback page at the server shows the
token. User copies and pastes into the Token tab.
Also fix: SetSize before app.Run() crashes with nil pointer — use
WebviewWindowOptions width/height directly from restored state.
The Wails runtime isn't reliably loading for IPC calls. Replace the
binding-based StartOIDCLogin with a local HTTP server on 127.0.0.1:18901:
- /oidc/login?apiUrl=... — opens system browser, waits for token
- /oidc/callback — Authentik redirect target, exchanges code
- /oidc/config?apiUrl=... — fetches OIDC provider config
- SPA detects desktop via ?desktop=1 URL param
- SPA calls localhost directly via fetch() instead of Wails IPC
- Remove custom asset handler — it broke Wails IPC routing
- Use application.AssetFileServerFS(distFS) so Wails serves its own runtime
- Add GetStoredConfig binding: SPA calls it on startup to retrieve keychain config
- main.ts: loadDesktopConfig() fetches stored creds before mounting
- Remove runtime.js embed (Wails serves it internally)
The SPA needs /wails/runtime.js for window.wails to be available.
Since we use a custom AssetOptions.Handler, Wails' internal routing
doesn't serve it. Embed the runtime and serve it explicitly.
ConfigService.StartOIDCLogin():
- Fetches OIDC config from the API
- Generates PKCE params
- Starts local HTTP server on 127.0.0.1:18901
- Opens system browser to Authentik
- Captures callback directly (no copy-paste)
- Exchanges code for token, saves to keychain
- Returns token to SPA → auto-connects
Config.svelte detects Wails environment and calls the binding.
The old icon.icns was copied from favicon.png which was actually
a dark-background .icns file. Regenerated from favicon.svg via
qlmanage → sips → iconutil to get white logo on transparent bg.
- Server: /oidc-callback HTML page exchanges Authentik code for token,
displays it for user to copy into the desktop app's Token tab
- oidc.ts: desktop mode uses apiUrl+/oidc-callback as redirect URI,
encodes PKCE verifier in state parameter
- Config.svelte: add Server URL field to OIDC tab for desktop UX
- Caddy: add /oidc-callback to enroll bypass (no Authentik gate)
- App: favicon.png as system tray icon, window title 'Oikos'
- web/index.html: title 'Oikos'
wails3 build v3 alpha delegates to Taskfile; the go build produces a raw
binary, not a .app. Package step now creates the bundle structure
(Contents/MacOS, Info.plist) and zips it.
12 of 33 MCP tools now render as rich inline cards instead of raw JSON:
EntityCard, HealthSummary, LXCList, EntityTable, KnowledgeResults,
BlastRadius, ChangeLog, FleetSnapshot, MetricChart.
Architecture:
- Server: annotateJSONResult() wraps queryRows with __renderer hints
- Registry: match/dispatch system maps tool names to Svelte components
- Chat: inline dispatch with 5-card limit, overflow to collapsed group
- ToolCallGroup: unmatched prop, hides when all matched, ARIA labels
Tests: 3 new Go tests for annotateJSONResult (wrap, no-op, multi-row).
Problem: the Oikos control room was browser-only — no native desktop
experience (system tray, notifications, keychain-persisted auth).
Change: add a Wails v3 thin-shell desktop app at cmd/desktop/ that embeds
the existing SPA in a webview. The Go side is ~380 lines — no bundled
server, no Postgres connection. It reads auth from the OS keychain,
injects it into the SPA on load, and the SPA talks HTTPS to the homelab
same as a browser.
Phase 1.0 — Scaffold + window:
- Embed web/dist/ into the Wails binary
- Inject window.__OIKOS_CONFIG__ with keychain-stored apiUrl + token
- 1400×900 window, min 1024×700
- System tray: Open/Quit, click toggles window
Phase 1.1 — Native shell:
- Poll /api/v1/dashboard/summary every 30s; osascript notification
when approvals or critical signals increase
- Save/restore window position to ~/.config/oikos/window.json
- EnableAutoStart/DisableAutoStart — macOS LaunchAgent plist
Phase 1.2 — Token management:
- Config.svelte calls window.wails.Call.ByName('SaveConfig') after
successful connection — persists to OS keychain
- ConfigService binds SaveConfig, ClearConfig, EnableAutoStart,
DisableAutoStart to the Wails runtime
Phase 1.3 — Auto-update:
- Poll Gitea releases API every 6h, compare semver, show dialog
- 'Check for Updates' tray menu item triggers immediate poll
Phase 1.4 — Distribution:
- macOS entitlements.plist: network client + keychain access
- .gitea/workflows/desktop.yml: CI builds macOS arm64 + Linux amd64
on 'desktop-*' / 'v*' tags, attaches artifacts to release
- Makefile: desktop (build), desktop-package (build + zip/tar.gz)
- CONTRIBUTING.md: documented desktop app + commands
Risk: low. Wails v3 alpha API may shift; the Go glue is ~380 lines and
trivially portable. The desktop app is additive — zero changes to the
existing server or SPA logic. No config mutation, no infrastructure
impact.
Verification: go build, go vet, go mod tidy all pass.
Fresh nodes (no prior x/y) get placed by d3-force's default init, which
spirals out from the ORIGIN — not (width/2, height/2) — while the
centering forces here are deliberately weak (0.04, so they don't fight
the link/collide layout) and alphaDecay stops the sim before a weak force
can always pull a far-off cluster back to center. Net effect: graphs could
settle visibly off-center on load, cramped in a corner of the pane.
Fixed by computing the actual node bounding box once the simulation's
'end' event fires and setting the view transform to fit it, instead of
relying on the force balance to land on center by itself. Gated behind a
`fit` flag so passive background reloads (live entity/relationship
events) don't yank the view out from under someone actively panning or
zoomed in on a specific area — only fresh loads (mount, root/depth
change, reset, re-root) reframe.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
search_knowledge and get_entity_knowledge only ever returned a ts_headline
snippet/short headline — enough to find a note, not enough to act on it.
Add get_knowledge_content(slug), mirroring the web UI's
/api/v1/knowledge/content/{id}, so the agent can read a document/
investigation/runbook's full markdown body once it knows which one it
needs. upsert_knowledge already covered the write side. Cross-referenced
all three tool descriptions so the agent discovers the full-read path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Executions were being created with no outgoing edges to what they acted
on or which task/session drove them, silently starving the graph of new
data going forward — found during this session's DB audit, which had to
backfill 245+25 missing targets/involves edges for existing executions.
This closes the gap at the source: every execution now gets a
target-->targets-->execution edge, and (when the caller supplies a
session/task) a task-->involves-->execution edge, both idempotent
(NOT EXISTS guards) so retries/backfills don't duplicate.
Two call sites: the deduped systemctl/apt_upgrade/pct_create fast path
and the general classifyAndGate path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two bugs found while verifying against real production data:
- Excluded activity types (execution/check/task etc., see categories.ts)
were falling through inCategory's "unknown type -> always visible"
fallback, since typeCategory only stored entries whose category was
defined. That fallback exists for types the ontology never returned at
all; it wrongly re-admitted types the ontology returned but categories.ts
deliberately excludes. Fixed by storing every type (including undefined
categories) and checking key presence, not value truthiness.
- Once that was fixed, the previous commit's 1-hop neighbor expansion
(dimmed cross-category context) turned out fine for a rooted view but
flooded an unrooted "browse the whole category" view: Fleet's ~49 focus
entities are hub-like enough that 1-hop pulled in 325+ of the system's
479 total entities. Neighbor expansion now only applies when a root is
set; the unscoped view goes back to same-category-only edges, which
measured at a clean 49 nodes for Fleet.
Verified against live production data (real bearer token, real DB) rather
than mocks: Fleet unrooted = 49 nodes matching the DB's compute+physical
count exactly; rooting on host:strong shows 33 nodes with both bright
same-category and dimmed cross-category neighbors, no isolated dots.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
chi.URLParam returns the raw, still-encoded path segment — unlike the
OpenAPI-generated routes, which decode via
runtime.BindStyledParameterWithOptions before the handler sees them. Slugs
like "document:containers/101-jellyfin" (encoded by the frontend's
encodeURIComponent) were arriving undecoded and matching no row. Found via
a standalone chi repro, not by patching the live deploy checkout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two fixes to the new category taxonomy:
- Knowledge Base couldn't show a document/investigation/runbook's own
markdown body — knowledge_entities.content was never exposed by any
endpoint (GetEntityKnowledge answers "what knowledge references this
entity", not "what is this entity's content"). Add GET
/api/v1/knowledge/content/{id} and render it with the existing
marked+DOMPurify pipeline in a new Content section.
- The graph hid any edge whose other endpoint wasn't in the active
category, so nodes with only cross-category neighbors rendered as
disconnected dots. Queried the real relationship table: ~70% of infra
edges cross Fleet/Network/Services/Storage lines (compute+network+
software+storage+physical used to be one "infrastructure" layer).
EntityGraph now keeps 1-hop neighbors visible but dimmed instead of
hiding them, so the edges — and what they connect to — stay visible.
- categories.ts: `cognition` domain conflated true knowledge (document/
investigation/runbook, 58 entities) with operational telemetry
(execution/check/task/signal/approval/pattern/skill/classification/
feedback, 300+ entities with their own Operations/Signals/Learning
pages). Mapping the whole domain to Knowledge pulled in 245 execution
entities fanning out from ~17 compute nodes via `targets` edges — the
single biggest source of graph clutter. Knowledge now maps by type
(document/investigation/runbook only); the rest of cognition is
excluded from Knowledge Base browsing entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the layer-based (Infrastructure/Governance/Cognition) browsing tabs
with a synthesized category taxonomy built from the ontology's finer-grained
`domain` field, since layer lumped unrelated entity types (an LXC and a DNS
record and a storage volume) into one bucket. Network and Fleet each span
two domains, so the table view now fans out per-domain fetches and merges,
while the graph view maps domain->category client-side. Also carries over
several detail-panel polish items (Tasks-not-raw-executions, slug URL
encoding, MultiSelectFilter) from earlier in this session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The SPA-from-binary split (0c0f35a) left `make deploy-ui` pointing at a
deploy path that was never actually wired up: scp to a "mac-mini" SSH
host that doesn't resolve from itself, a /var/www/oikos-ui/ that doesn't
exist, and `systemctl reload caddy` on a box with no Caddy installed at
all (not brew, not a container, nothing on 80/443).
Add a `web` service (compose/web/Dockerfile: node build -> caddy:2-alpine
static + SPA-fallback serving) to docker-compose.yml so the UI deploys
through the same push-to-main -> webhook -> docker compose build/up
pipeline the rest of the stack already uses, instead of a manual
scp/ssh step. Drop the broken `deploy-ui` Makefile target; `make ui`
stays as a local build sanity-check.
Update the reference Caddy config (compose/caddy/Caddyfile.oikos) to
reverse_proxy the new :8091 service instead of reading static files off
local disk, and fill in the <mac-mini-mesh-ip> placeholders with the
actual LAN IP (192.168.178.182 — the LXC and mac-mini subnets are
routed). This file is a reference only; the real caddy-conf repo change
is applied separately after review.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the separate Entities/Graph nav items with one Knowledge Base
page that browses all entities as either a table or a force-graph,
scoped by ontology layer (Infrastructure/Governance/Cognition), with a
resizable browse/detail split instead of a slide-over sheet.
- New KnowledgeBase.svelte: layer tabs, view toggle, resizable
browse/detail split (pattern from Chat.svelte's rail).
- EntityTable/EntityGraph extracted as presentational sub-components;
their search/filter/root/depth toolbars live in the shared page
toolbar (not the resizable pane) so they don't truncate when the
divider is dragged narrow, and both views start flush with the
detail pane for consistent height.
- EntityTable columns are sortable (slug/type/name/state/health).
- EntityDetailContent redesigned as a single-column list of
collapsible sections (DetailSection.svelte), collapsed by default
when empty; relation entries are clickable and select the entity in
the browse pane + detail pane (and drill in-place in EntitySheet
wherever it's used elsewhere in the app).
- api.ts: add layer filter to fetchEntities, add fetchEntityTypes for
client-side graph layer scoping (the graph endpoint has no layer
param).
Old hash routes (#/entities, #/graph) redirect to #/kb.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Documentation and repo-hygiene pass following the client/server split:
Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.
Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).
Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).
Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.
Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.
Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
GetClientContext handler) since the mechanism's introduction on
2026-06-02 — never matched any real filename, so no client has ever
picked up an auto-setup script via git-pull or the context-poller sync.
Fixed all three; the Go server-side fix is the one that actually matters
since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
(parsed, never consumed) left over from an earlier clone-based model.
Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 0 of plans/2026-07-12-wails-desktop-app.md. The control-room SPA
is no longer embedded (web/embed.go deleted); it's a standalone static
build served separately (make ui / make deploy-ui). The api process
adds CORS and drops the dev-open auth bypass — every route now needs a
real bearer token, including SSE (?token= query param, EventSource
can't set headers) and api's own /agent proxy to nomos (previously
unauthenticated by omission).
nomos was an unauthenticated client of api's /mcp and approval-decision
endpoints; closing dev-open would have broken it, so it now sends
Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN on every call back to api.
SPA gets a runtime config module (config.ts) and a Config.svelte
first-launch/reconfigure page, reachable afterwards via a "Connection"
entry in the sidebar footer. Every fetch() in api.ts routes through
fetchWithAuth so the same build works same-origin (browser prod, Vite
dev proxy) or cross-origin (future Wails webview, remote access).
Six gaps found against the plan and the live Caddy topology while
implementing — documented in the plan's "Plan review" section, most
notably: api's own /agent mount was never behind combinedAuth (fixed),
and production's Authentik forward-auth needs a bearer-token bypass for
API routes that this repo's Caddyfile.oikos reference copy now has, but
the real dtoro/caddy-conf deploy does not yet.
Verified live: cross-origin static SPA + API, CORS, bearer auth, SSE
query-token auth, and localStorage persistence all confirmed working
in-browser. Full Go test suite and npm run build pass with no
regressions against the pre-change baseline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gitea (LXC 104, 192.168.8.x) can't reach mac-mini (192.168.178.182) due to
ALLOWED_HOST_LIST. As a fallback, a 2-minute launchd poller checks if
origin/main has new commits and runs deploy.sh if so.
- cmd/webhook/main.go: HMAC-validated webhook receiver on :9797
- launchd plist: keeps webhook running, PATH includes docker
- Makefile: 'make webhook' target
- Registered as Gitea webhook id 15 on dtoro/oikos
Fixes: auto-deploy was not wired on mac-mini after the consolidation
Overview replaces Tasks as the default route: a centered new-task entry
with live fleet metrics, a scrollable/filterable task table, and an
ambient canvas rendering of the real entity graph (autonomous camera
drift + mouse parallax) behind it. Tasks sidebar entry is removed;
its status-bucketing logic moves to lib/tasks.ts for reuse.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Search hits and entity-knowledge hits never selected an id column, so
every KnowledgeHit.Id defaulted to the zero UUID. The frontend's keyed
{#each results as hit (hit.id)} then had all-duplicate keys, which
silently broke Svelte 5's if-block branch swap for the results panel —
search would set searched=true (Clear button appeared) but the view
never switched away from "Recently learned". Select e.id in both
queries and key the each block on hit.slug (guaranteed unique) instead.
Events, Agent, and Audit were standalone read-only pages that never
cross-referenced the entity they related to. Fold them into EntityDetail
as entity-scoped cards (Agent activity, Audit trail) alongside the
existing Signals/Executions/Knowledge cards, and give the Signals card
real Ack/Mute/Resolve actions. Signals stays a standalone page since
it's the only one with cross-entity triage value (badge count, actions).
Also fixes the underlying reason those new cards would've stayed empty:
agent_activity rows were never tagged with entity_id at insert time
(cmd/nomos/store.go, internal/mcp/server.go), even though the column
and the API filter both support it. Added a best-effort resolver that
checks common tool-arg keys (target, entity_slug, slug, ...) against
the entities table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes 1-3 deployed and verified live: fresh trivial Q&A sessions now reach
done immediately, and a goal-bearing session that stalled was correctly
nudged by the idle sweep. Fix 4 (backfill) was replaced with deletion after
the operator's call — verified against the DB first that zero knowledge
notes were linked to or written by any of the 53 removed sessions, so
nothing was lost. Documents the pagination gap in listSessions (hardcoded
LIMIT 50, no total count) that hid 6 of those sessions from the original
audit.
Also fixes relative links in this plan and in the UI-review plan that broke
when both moved from plans/ to plans/done/ (one directory level deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements fixes 1-3 of plans/2026-07-11-task-completion-safety-net.md.
Confirmed live that 50/50 production sessions never reached a terminal
status because the model almost never calls complete_task, even for
trivial single-tool Q&A turns SOUL.md explicitly calls out as needing it.
- Inline safety net (agent.go): a session that never called set_goal never
framed itself as a structured task, so its first plain-text turn-end IS
the task ending — auto-complete it there instead of leaving status stuck
at its creation default forever.
- Idle sweep (continue.go, new completion_nudges column): goal-bearing
sessions that stall get one nudge, then auto-close with outcome=partial
if the nudge goes unanswered, mirroring the pattern resumeSession already
uses for a different stuck-session failure mode.
Fix 4 (backfill of the 50 already-stuck live sessions) is deliberately
separate — deferred until this is deployed and verified live, per the
plan's implementation order.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Traced during UI-review verification: 50/50 live sessions are stuck
active/planning, never done/failed. Root cause confirmed against the
running DB — set_goal called once, propose_plan and complete_task
called zero times across all 50 sessions. The model consistently
skips the terminal complete_task call despite SOUL.md explicitly
instructing it to, especially for trivial single-tool Q&A turns.
Plan proposes an inline safety net for the common case plus an idle
sweep for structured goal/plan sessions that stall.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes the reviewed gaps: keyboard-inaccessible delete controls (SessionRail,
Entities row), case-sensitive entity filter, two competing entity-detail
navigation patterns (standardize on EntitySheet), non-clickable Overview KPI
cards, a bare button bypassing the shared Button component, inconsistent
blur-only vs live filtering, and an unenforced sanitization assumption on
search snippet HTML (now using the already-present dompurify dependency).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every finding from the review is now implemented and verified live:
A1 (3919ec3), B1+B2 (c5ffaec), A3 (926969a), D1-D3 (76f7630), A2 (c390164),
B3 (6d4f6de), F1 (11c18e8). C1 (nomos gateway has no authentication) remains
explicitly deferred per operator instruction. Kept in plans/ (not moved to
done/) since C1 is still open, matching how other partially-complete plans
in this index are tracked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix F1 of plans/2026-07-11-nomos-agent-code-review.md, the last item.
buildTools called listToolsFull (a tools/list MCP round-trip) at the start of
EVERY chat turn, including every auto-continuation resume — the tool list is
static for the lifetime of one MCP connection, changing only when the api
process re-registers tools (a restart, which this client already detects and
reacts to via reconnectLocked). Re-fetching it every single turn was
avoidable network+parsing work on the hot path.
mcpClient now caches the parsed tool list after its first fetch, guarded by
its own mutex (kept separate from the request-serializing mu so a cache
check never contends with an in-flight doRequest call). reconnectLocked
clears the cache — an api restart may have changed what's registered, so a
stale cache would be wrong, not just slow. fleetSnapshot's get_health_summary
call is deliberately left uncached — it's meant to be "as of now."
Since each session gets its own client (the per-session pool from the
concurrency work), this caches per-task-conversation rather than globally: a
task's FIRST turn still pays the round-trip, every turn after reuses the
cached list — which is exactly the case that mattered (long-running,
heavily-autonomous tasks with many auto-continuation resumes).
Verified live via the api's request log: a brand-new session's first turn
made 3 MCP calls (initialize, tools/list, get_health_summary); a second turn
on the SAME session made exactly 1 (only get_health_summary) — tools/list
correctly skipped.
This completes the implementation order in
plans/2026-07-11-nomos-agent-code-review.md — every A/B/D/E/F finding from
the review (excluding C1, explicitly deferred per operator instruction) is
now fixed, tested, and verified live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix B3 of plans/2026-07-11-nomos-agent-code-review.md. resumeSession's retry
loop (used by both auto-continuation and panel-answered questions) already
retried once on a transient LLM failure, but if BOTH attempts came back
empty/erroring, the code just logged and returned — the task was left at
whatever status it already had (typically 'executing' or 'awaiting_input')
with no outcome, no operator-visible signal beyond an inert error line
buried in the transcript, and no way to tell a genuinely stuck task apart
from one quietly still working.
On permanent failure, now calls store.completeTask(outcome='failure', a
summary built from the error) so the task board reflects reality instead of
showing a task that looks perpetually in-progress. Uses context.Background()
for that write, matching resumeSession's own persistence pattern, since the
context that led to the failure may itself be in a bad state. This doesn't
prevent the operator from continuing to work the task via a fresh chat
message afterward — it only replaces silent hanging with a real status.
A full live induction of a permanent LLM outage would require breaking the
model/API-key config for the whole nomos container — too invasive for this
fix's priority. Verified instead that the new branch stays correctly dormant
on the happy path: ran a real ask_operator → panel-answer → resume cycle
end-to-end and confirmed the task landed at status='executing' with no
outcome set, proving the failure-handling code doesn't false-positive on a
normal successful resume.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix A2 of plans/2026-07-11-nomos-agent-code-review.md. chatWith replayed a
session's ENTIRE message history into the LLM's context on EVERY turn, no
windowing, no token budget — confirmed against a documented production case
(a single turn with 70 tool calls, messages up to 106KB). Every subsequent
turn of a long-running or heavily-autonomous task re-sent that ever-growing
history in full — a real cost/latency/eventual-context-limit risk for
exactly the tasks this system runs longest (many auto-continuation cycles).
Design call (flagged in the review as needing one before implementation):
a fixed-size window for LLM replay specifically, not the UI's own transcript
view. Simplest option that still keeps roughly the current task's working
context; a token-aware trim or LLM-summarize-on-drop are documented as
stretch options if 30 proves insufficient in practice.
- store.go: new getRecentMessages(ctx, sessionID, limit) — last `limit`
messages in chronological order, plus whether older ones were omitted.
getMessages (used by the UI's GET /sessions/{id}) is untouched and stays
unbounded — the operator should still see a task's full history regardless
of length; only what gets sent to the model is bounded.
- agent.go: chatWith uses getRecentMessages(sessionID, historyWindowSize=30)
instead of the unbounded getMessages. When truncated, injects a system
note telling the model explicitly that older turns exist but aren't shown,
so it checks upsert_knowledge/search_knowledge rather than assuming
something wasn't done just because it isn't visible.
New cmd/nomos/store_test.go: real Postgres integration tests (mirroring
internal/db/integration_test.go's throwaway-database pattern, guarded by
OIKOS_TEST_DATABASE_URL). TestGetRecentMessages_Truncation is the direct
proof for this fix (35 messages → 30 returned, correctly ordered,
truncated=true; 5 messages → all 5, truncated=false) — both cases run
against a fully-migrated database, not mocked. Also added
TestProposePlan_AppendVsReplace, closing part of the review's test-coverage
finding (E) by permanently regression-testing the earlier append-vs-replace
plan fix (commit 5384499), which had only been verified manually until now.
Verified live: inflated a real session to 42 persisted messages via direct
SQL, then continued it with a real chat call — the turn proceeded normally
(multiple real tool-call iterations, no crash, no context-length error);
nomos stayed healthy throughout. A3's incremental persistence separately
confirmed to have caught the 7 real tool calls made before the client
connection was cut, cleanly closing out both fixes' interaction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes D1-D3 of plans/2026-07-11-nomos-agent-code-review.md:
- D1: deleted isTaskTool — defined, never called (dispatch already checks
handleTaskTool's own `handled` return value).
- D2: recordTouched issued one SELECT per entity slug found in a tool call's
args; batched into one `WHERE slug = ANY($1)` query. Verified live: a turn
naming three separate entities recorded involves edges for all three via
the single batched lookup.
- D3: complete_task's outcome had a declared enum (success|failure|partial)
in its tool schema but nothing validated it — an out-of-enum value (model
typo or a weaker model not respecting the schema) silently persisted as-is,
with only "failure" special-cased (anything else became status='done'
regardless of what the value actually said). Now validated in
handleTaskTool: empty defaults to "success" (unchanged), a recognized value
passes through, anything else defaults to "partial" (safer than silently
treating an unrecognized value as success) with a warning logged. Verified
live: instructed the agent to call complete_task with outcome="unclear" —
persisted as outcome='partial', not the literal invalid string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix A3 of plans/2026-07-11-nomos-agent-code-review.md. handleChat only ever
saved the assistant message ONCE, after a.chat(...) returned, using
ctx := r.Context() for that write — the same context that cancels the instant
the client disconnects (Stop button, tab close, network blip). A disconnect
mid-turn meant the final save ran with an already-cancelled context and its
error was never checked: the entire turn's tool-call history was silently
lost from the persisted transcript, even though real work (executions
launched, knowledge written) had already happened server-side.
Brought handleChat in line with resumeSession's existing pattern
(continue.go): insert a placeholder assistant row immediately, update the
SAME row after every tool call. The key fix is WHICH context the writes use —
a new pctx := context.Background() for every DB write in this handler
(session creation/touch, the user message, question auto-close, the
placeholder + incremental updates, the title update), while ctx/r.Context()
still gates the agent's own work (a.chat) and the SSE writes exactly as
before — a disconnect still correctly stops the agent from doing further
work, it just no longer also erases what it already did.
Verified live: sent a message requiring 6 tool calls (get_entity/
get_relations/get_blast_radius on two targets) and force-killed the client
connection mid-stream with curl -m 12 (confirmed via exit code 28). Before
this fix the persisted transcript would show 0 tool-call entries; after,
all 12 raw tool_use/tool_result entries (6 calls × 2) were present and
correctly attributed by tool name — proving both that progress survives an
abort and that the incremental writes aren't corrupting the data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together,
since the right granularity for B1 in the auto-continuation worker turned
out to require B2's restructuring anyway (see below).
B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned
nothing before this — every explicitly-spawned goroutine (continuation
worker, resumed chat turns, async execution dispatch, the SSE listener, two
duplicate sshExec implementations' output-collector goroutines) crashed the
whole process on an unhandled panic, not just that one goroutine. More
consequential post-concurrency: more simultaneous unattended background work
means more surface area for one bad input to end every running task.
New internal/safego package: Go(label, fn) launches fn in a goroutine with a
recover-and-log wrapper. Applied at every bare `go` spawn site across the
three packages. Two sites needed bespoke handling instead of the generic
helper because their callers block on a channel and a silent recover would
just make them hang until timeout: sshExec's output-collector goroutine (two
near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go)
and httpapi's ListenAndServe goroutine — both now recover AND send a
synthetic error result so the waiting select unblocks immediately instead of
waiting out the full timeout.
httpapi's sseListener got extra treatment: its per-notification handling was
extracted into handleNotification with its own recover, so a panic decoding
ONE malformed pg_notify payload can't kill the listener goroutine for every
connected SSE client — the outer goroutine spawn only needs to guard the
connection setup/reconnect code around it.
B2: cmd/nomos/continue.go's processContinuations used to run every pending
continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the
ticker — meaning (a) task B's continuation waited for task A's full (up to
10-minute) resumed turn to finish first, undercutting this session's earlier
concurrency work on exactly the path autonomous tasks depend on most, and
(b) an unrecovered panic anywhere in that call chain didn't just crash the
process (B1) — even WITH B1's recovery wrapped only at the top-level worker
spawn, the panic would still unwind the ENTIRE ticker-loop goroutine,
silently ending auto-continuation for every task until nomos restarted.
Fixed by spawning each pending item via safego.Go individually: real
parallelism, and a bad item can now only ever take down its own goroutine.
Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete
proof — a deliberate panic inside Go() that would otherwise crash the whole
test binary; reaching the assertion after it IS the evidence recovery works.
Verified live against the rebuilt containers: full chat turn round-tripped
correctly (hostname lookup, 2 iterations, normal completion) — no regression
from threading safego.Go through the tool-dispatch/continuation paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix A1 of plans/2026-07-11-nomos-agent-code-review.md. isAssent and
isTypedConfirmation used a space-padded word-boundary check for negation
words but a bare strings.Contains for assent/confirm words — confirmed live
via test probes: isAssent("...maybe yesterday's logs...") returned true
("yes" matched inside "yesterday"), and isTypedConfirmation("I haven't
confirmed anything yet") returned true ("confirm" matched inside "confirmed",
and "haven't" wasn't in negationWords — only "don't"/"do not" were).
isTypedConfirmation is the sole gate for DESTRUCTIVE actions, so the second
case meant a message merely stating something hadn't been confirmed could
read as an explicit confirmation.
- Replaced the ad-hoc space-padding/prefix-check negation logic with proper
tokenization (wordTokenRe) + containsPhrase, matching WHOLE tokens/phrases
only — never a mid-word substring. Handles curly apostrophes too (a
pre-existing gap: the old straight-quote-only check would have missed
"don't" typed with a smart quote).
- Added contracted negatives (haven't, hasn't, isn't, wasn't, aren't, can't,
cannot, won't, wouldn't, shouldn't, didn't, doesn't) to negationWords.
Deliberately did NOT add a bare "not" — too broad, would false-negative
ordinary assent like "go ahead, this is not risky".
- Added regression tests for both confirmed cases plus a couple of adjacent
ones (eyesight/isn't, can't confirm) so a future change can't silently
reintroduce either bug.
All existing assent/confirmation tests pass unchanged — this is a pure
robustness fix, not a behavior change for any previously-correct case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full read-through of cmd/nomos/ (agent.go, store.go, main.go, continue.go,
assent.go, tasks.go). Findings, ranked:
- A1 (confirmed via runnable probe): isAssent/isTypedConfirmation use
unpadded substring matching for assent/confirm words while negation uses
word-boundary checks — "yes" matches inside "yesterday", "confirm" matches
inside "confirmed" with no negation word covering contracted negatives
("haven't"). isTypedConfirmation gates DESTRUCTIVE actions specifically.
- A2: chatWith replays a session's ENTIRE message history every turn, no
windowing/token budget — confirmed unbounded against a documented
production case (70 tool calls, 106KB messages).
- A3: a live turn's tool-call history is lost entirely if the client
disconnects mid-stream (single end-of-turn save using the same
connection-tied, possibly-cancelled context) — resumeSession already has
the fix pattern (incremental placeholder+update), handleChat doesn't use it.
- B1: zero recover() anywhere in cmd/nomos/internal/mcp/internal/httpapi —
every explicitly-spawned goroutine (continuation worker, resumeSession,
executeApprovedViaAPI, sse listeners) crashes the whole process on panic.
- B2: auto-continuation processes its batch sequentially, one full LLM turn
at a time, undercutting this session's own concurrency work on exactly the
path autonomous tasks depend on most.
- B3: no terminal state for a permanently-failed auto-continuation.
- C1: nomos's own gateway (port 8092, directly published + mesh-reachable)
has ZERO authentication on any endpoint — chat, session read/delete,
chat-assent approval of gated executions, all open to anyone on the LAN.
- D1-D3: dead code (isTaskTool unused), N+1 query in recordTouched, no
validation on complete_task's outcome enum.
- E: zero automated tests for agent.go/store.go/main.go/tasks.go — including
today's new safety-critical logic (session-scoped windows, mcpClientPool,
proposePlan's append-vs-replace), verified only by live manual testing.
- F1: tool list + fleet snapshot re-fetched every turn (minor).
Prioritized implementation order in the doc: A1 → C1 → B1 → B2 → A3 → D1-3 →
A2 → B3/F1, tests landing alongside each fix rather than as a deferred pass.
Also archives the now-fully-shipped concurrent-task-execution plan to done/
(all 3 required fixes deployed this session; fix 4 explicitly deferred per
its own recommendation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 3 of plans/2026-07-11-concurrent-task-execution.md, the throughput one.
nomos held exactly one *mcpClient for the whole process, shared by every
/chat goroutine. Its mutex was held for the full duration of each tool
round-trip, and `run` executes its SSH command SYNCHRONOUSLY inside that
round-trip (capped at up to 10 minutes) — so while Task A was mid-`run`,
every other task's tool calls, even a trivial get_entity, queued behind that
single lock. Tasks could think (LLM calls) in parallel but never act in
parallel.
The MCP server has no per-connection state to protect (newServer returns one
shared *mcp.Server instance whose handlers close only over the DB
connection pool, already safe for concurrent use) — the mutex existed purely
because the client reused one stateful transport session. So the fix doesn't
touch the server at all:
- New mcpClientPool (cmd/nomos/main.go): one *mcpClient per session id,
created lazily (a real MCP initialize handshake) on first use and cached;
session-less traffic (the ephemeral no-DB-store path, the structured
/query endpoint) gets its own fixed, reused key instead of a fresh
connection per request. Idle clients (20 min past last use — long enough
to outlive a single slow `run`) are evicted on a 5-minute sweep ticker.
- agent.go: `client *mcpClient` → `clients *mcpClientPool`; every call site
(buildTools, fleetSnapshot, the tool-dispatch loop) now resolves its own
session's client via clients.get(sessionID) instead of reaching for one
shared field. A task's own tool calls stay sequential (already true — the
agent loop calls tools one at a time within a turn) but no longer block
anyone else's.
- main.go: handleQuery takes the pool instead of a client (keyed "query", a
fixed non-session slot); shutdown calls pool.closeAll().
Verified live against the deployed stack: fired a slow-but-ungated command
(`ping -c 15 127.0.0.1`, read-only per policy's allowlist, no approval
needed) as Task A, then — 2s into A's run — a trivial hostname lookup as
Task B, both through the real /chat endpoint. Task A's ping genuinely ran
~14.3s (confirmed via its own execution record and the agent's reported
output). Task B returned in 6s total, well before A finished — proving it
was never queued behind A's connection. Before this fix, B would have been
forced to wait out A's entire ~14.3s hold on the single shared client.
This completes plans/2026-07-11-concurrent-task-execution.md's required
scope — only the explicitly optional/deferred Fix 4 (a concurrency/cost cap,
pending real usage data) remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the known gap flagged in the previous commit (9131559). A single
module-level `activeController` meant cancelStream()/newChat() always
aborted whichever stream was MOST RECENTLY STARTED, regardless of what the
operator was currently viewing: start Task A, switch to an already-loaded
Task B, click "New task" — the click's cancelStream() would silently abort
Task A's still-running turn, even though the operator was never looking at
it and never asked to cancel it.
- Replaced the single controller with activeControllers (Map<sessionID,
AbortController>) plus pendingController for the brief pre-'session'-event
window of a brand-new task. Registered immediately in sendMessage (keyed by
the continuing session id right away, or held pending until the 'session'
event assigns a new one) and cleaned up on completion.
- cancelStream() now looks up by $currentSession (falling back to
pendingController when no session is assigned yet) — it can only ever
touch the stream belonging to the view being left, never an unrelated
background task's.
- newChat() unchanged in behavior (still calls cancelStream()), now correctly
scoped through the above.
Verified live, reproducing the exact bug: started Task A (slow, 5 tool
calls), switched to an existing Task B, clicked "New task" while viewing
B — Task A was NOT aborted, ran to completion server-side with a full,
correct final summary (previously this exact sequence would have killed it).
Confirmed the positive path is unaffected: started a task, clicked Stop
while actively viewing it — input re-enabled, stream genuinely aborted
("BodyStreamBuffer was aborted"), turn stopped mid-flight as expected.
This closes out Fix 2's scope from
plans/2026-07-11-concurrent-task-execution.md; only Fix 3 (per-session MCP
client pool, throughput) and the optional Fix 4 (concurrency cap) remain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 2 of plans/2026-07-11-concurrent-task-execution.md. sendMessage's SSE
callback mutated the global messages/currentSession stores unconditionally,
assuming only one task's turn is ever in flight. It isn't — the backend runs
every /chat request as its own goroutine with no serialization. Switching to
a different task while a previous one was still streaming let that
background stream's later events (tool_use, text_delta, ..., and worst of
all 'done''s currentSession.set) get applied to whatever the operator is now
looking at: corrupting another task's transcript, or yanking the view back
to the one they left.
- Captures the session a stream belongs to (openedFor at call time, updated
to the real id once the 'session' event assigns one) and checks
$currentSession still matches before every messages/error/streaming
mutation. The task keeps running server-side regardless — dropped events
just mean the live view isn't watching it; navigating back re-hydrates via
REST, same as already happens for auto-continuation.
- The 'session' event itself only claims currentSession if the operator
hasn't already navigated elsewhere since the call started (comparing
against openedFor, which is null for a brand-new task).
- loadSessionMessages/newChat now reset `streaming` to false unconditionally
on navigation — needed so the new guard can't leave a DIFFERENT task's view
stuck showing streaming=true (which would also silently stop startPolling's
loop from ever applying updates, since it bails while $streaming is true).
Known residual gap, not fixed here (matches the plan's "contained fix, not a
rearchitecture" scope): activeController is still a single global slot, so
starting a new task while another is mid-stream, then clicking "New task"
again, aborts whichever stream that slot last pointed at rather than only
the one being left. A genuine multi-session controller/store is the
plan's deferred "stretch" fix, not required for correctness here.
Verified live: started Task A with a deliberately slow 4-tool-call turn,
switched to an existing Task B mid-stream — Task B's transcript stayed
correct with zero A-originated entries and the input was NOT stuck disabled.
Task A kept running and completed normally server-side (status=done, full
6-tool transcript, 5-entity graph); navigating back loaded its complete,
uncorrupted result via REST.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 1 of plans/2026-07-11-concurrent-task-execution.md — the safety-critical
one. The assent window (and destructive window) were keyed purely by agent
id ("assent_window.agent:<uuid>"). With one agent:nomos entity serving every
concurrent task, this meant approving Task A's plan opened a window that ANY
concurrently-running task's config-mutation/destructive actions could also
ride, auto-executing without their own approval.
- store.go / agent.go: assentWindowActive/openAssentWindow and
destructiveWindowActive/openDestructiveWindow/destructiveWindowKey all gain
a sessionID parameter; keys become
"assent_window.agent:<id>.session:<sessionID>" and
"destructive_window.agent:<id>.target:<slug>.session:<sessionID>". Missing
session id fails closed (no window) rather than falling back to the old
agent-wide key.
- continue.go: the auto-continuation worker's window check moved from once-
per-batch to once-per-pending-item, scoped to that item's own session —
it was previously checking ONE agent-wide window for a batch that can span
multiple tasks.
- agent.go tool-dispatch: injects `_session_id` into a COPY of the wire args
sent to the MCP server (never into the args used for the emitted/logged/
persisted tool call, and never part of any tool's declared InputSchema —
invisible to the model) so the gating checks on the OTHER side of the
process boundary know which task is asking.
- internal/mcp/server.go: assentWindowActive/destructiveWindowActive/
classifyAndGate gain the same sessionID parameter, read from
args["_session_id"] at the three call sites (request_execution's
apt_upgrade/pct_create branches, and the shared classifyAndGate used by
restart/pct_exec/systemctl/run).
Verified against the live stack with the exact scenario from the plan: opened
an assent window for session A only, then called `run` with an identical
config-mutation command for session A (window open) and session B (same
agent, no window). A auto-ran (execution status completed); B correctly
queued for approval (pending_approval) instead of bleeding through — proven
at both the MCP response text and the executions table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New plan grounds three concurrency issues found by tracing the actual code
(not assumed): the assent/destructive windows are keyed by agent id only
(no session dimension), so an approved plan in one task can auto-run
unapproved actions in a concurrently-running task; nomos shares one
mutex-guarded MCP client across all sessions, so a single slow `run` call
serializes every other task's tool calls behind it; and chat.ts's SSE
callback has no session guard, so switching tasks mid-stream lets the
backgrounded task's events corrupt whatever's now displayed. Proposes
session-scoping the windows (critical/first), a frontend stream guard
(contained/second), a per-session MCP client pool (throughput/third), and
an optional concurrency cap (deferred pending real usage data).
Also archives the goal-oriented-chat-control-panel plan to done/ — all 7
phases shipped and are live in production (SHA e30813a) — fixing its
internal relative links for the new depth and pointing forward to the new
concurrency plan as follow-up hardening.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the gap that made the knowledge loop optional/implicit: every
non-trivial task now has an EXPLICIT first plan step (research) and last
plan step (write back), not just background behavior the model might skip.
New MCP tools (the agent had no way to do these before — only REST endpoints
existed, unexposed to it):
- update_entity_attributes(slug, attributes): shallow-merge new/changed facts
into an entity (an IP, a version, a discovered port) so a future task
doesn't have to rediscover them from scratch. No approval required — this
updates the knowledge graph, not live infra.
- create_relationship(source, target, type): record a discovered edge
(depends-on, hosts, provides, ...). Idempotent, FK-validated against the
ontology's relationship_types, no approval required.
SOUL.md: restructured the task loop so step 1 is explicitly "gather
knowledge, not just status" (get_entity_knowledge, search_knowledge,
get_relations, get_blast_radius, http_get) and the last step before
complete_task is explicitly "write back" (update_entity_attributes,
create_relationship, upsert_knowledge) — both called out as real plan
entries the operator should see in propose_plan, not silent side-work. This
is what prevents the graph drifting from reality and is the concrete
mechanism behind "tasks compound."
propose_plan's tool description reinforces the same first-step/last-step
convention at the call site.
Verified against the live stack: both tools registered and callable via MCP;
update_entity_attributes merged an attribute correctly; create_relationship
rejected an invalid type (FK violation, clear error) and succeeded with a
valid type+direction, confirmed idempotent (2 calls, 1 row).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause: proposePlan unconditionally deleted and replaced the whole
session_plan_steps list on every call. The model isn't strictly held to
"call propose_plan once with the full list" — nothing stopped it (and
production evidence + live testing showed it happening) from calling
propose_plan once per step as it worked. Each such call wiped every
already-completed step, so the operator only ever saw the model's latest
single step ("1/1") instead of the real, growing plan.
Fix, two layers:
- store.go: proposePlan now only does a destructive replace when no step
has left 'pending' yet (a genuine pre-execution revision). Once any step
has started, a new call APPENDS after the current max seq instead of
wiping — so the panel accumulates the full history regardless of how the
model chooses to call the tool. plan.proposed now carries `appended` so
the frontend knows whether to replace or append.
- workspace.ts: plan.proposed handler respects `appended` (update vs set).
- tasks.go / SOUL.md: strengthened the propose_plan description and task-
loop guidance to call it ONCE with the complete step list end-to-end,
using update_plan_step (not re-calling propose_plan) to advance — fixing
the root behavioral cause, with the store-side append as a safety net
that holds even if the model still calls it incrementally.
Verified: forced the exact incremental-call pattern (propose_plan with 1
step, mark it running, propose_plan again with 1 more step) — the second
call appended at seq 2 instead of erasing seq 1, and its plan.proposed
event carried appended=true.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the chat right rail's ad-hoc Digest+Graph stack with a single
TaskContextPanel that renders the task's live working state, driven by the
always-on events stream (not the per-turn chat SSE) so it keeps updating
during server-side auto-continuation/resume:
- GoalHeader: goal + status pill (planning/executing/awaiting_input/done/
failed), sourced from the sessions list.
- PlanProgress: ordered steps with live status icons + progress bar, hydrated
via new GET /sessions/{id}/plan; clicking a step with a target opens its
EntitySheet (no fake "jump to transcript" — bits-ui Collapsible content
isn't force-mounted, so a DOM-scroll jump would silently no-op for
collapsed tool groups).
- OperatorQuestion: the pinned structured question card (prompt/why/entity
chips/option buttons/free-text), hydrated via new GET /sessions/{id}/
questions; answering POSTs to the existing answer endpoint.
- SessionGraph upgraded to a live entity panel: entity.touched pulses the
node (animated ring) and shows "Now touching <slug>"; health.changed shows
a transient diff badge for touched entities.
- SessionDigest gains a success/failure/partial outcome banner and now also
refetches when the task's status changes, not just on session switch.
Two bugs found and fixed while wiring this up:
- workspace.ts's status-refresh trigger only covered goal.set/task.status;
question.raised/answered didn't refresh the sessions list, so GoalHeader's
pill went stale after answering via the panel (resumeSession runs entirely
server-side — no client 'done' event to piggyback a refresh on). Now every
status-affecting event triggers the (debounced) refetch.
- Forgot to rebuild the nomos container after adding the /plan and
/questions endpoints, so they silently fell through to the old default GET
handler — caught via a live curl diff against the running container,
not a code read.
Verified end-to-end against the live stack: goal/plan/question all update
without a reload as the agent works; answering a question via the panel
resumes the agent and the header pill correctly flips to Executing;
entity.touched pulses the live graph.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframes the chat surface as tasks:
- New Tasks.svelte: a card grid of tasks, each showing status (Running /
Needs input / Done / Failed), the goal as title, the outcome summary, and
relative time; filterable by status with live counts; delete on hover;
"New task" and per-card click open the conversation.
- Board updates LIVE off the events stream (goal.set / task.status /
question.*) via an explicit liveEvents.subscribe with a debounced refetch —
scanning all events newer than the last seen, since entity.touched bursts
bury task events below index 0.
- App shell: primary nav "Chat" → "Tasks" (board is now the home route),
"New chat" → "New task", conversation header gets a Tasks / Conversation
breadcrumb. Removed the superseded Sessions page.
- api.ts Session type carries the task fields (goal/status/outcome/summary).
Also fixes a pre-existing SSE bug that blocked ALL live updates app-wide:
writeSSE emitted `event: <type>`, which EventSource only delivers to
addEventListener(type) handlers — but stores/events.ts (and every page reading
liveEvents) consumes via onmessage, which never fires for named events. So the
live stream delivered nothing to the UI. Dropped the event-name line; the type
is already in the JSON payload, and new event types now need zero client
changes. SSE test still green (it parses data: lines).
Verified in the browser against the live stack: the board renders 50 tasks
with correct status buckets; a goal-driven task appears and flips to a Done
card with its summary in real time without a reload; Events page confirms the
stream now delivers to onmessage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The last backend piece: when the agent hits a decision only the operator can
make, it surfaces a structured question instead of guessing or stalling.
- ask_operator(prompt, why?, options?, context_entities?): nomos-local tool
that records a session_questions row, moves the task to awaiting_input, emits
question.raised, and ENDS the turn (the agent loop returns after it, so the
agent can't barrel past its own question). The prompt becomes the assistant's
visible message so the question also shows inline in the transcript.
- Two resume paths, both close the question + emit question.answered + return
the task to executing:
- Panel: POST /sessions/{id}/questions/{qid}/answer → resumes the agent in the
background with the answer injected (reusing the continuation machinery,
refactored continueSession → resumeSession). Returns 202; the reply lands via
message polling.
- Chat reply: the next chat message on a task with an open question IS the
answer — auto-closed in handleChat; the turn itself is the resume.
Verified end-to-end: forcing a decision paused the task at awaiting_input with
the structured question (prompt/why/options/entities); a panel answer resumed
the agent (it acknowledged host:strong and continued); a plain chat reply
auto-closed a second question. Cleanup + tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gives a task a legible, live-advancing plan via three more nomos-local tools:
- set_goal(goal): records the task goal, status → planning, emits goal.set.
- propose_plan(steps[]): persists ordered steps (clean replace for v1 — a
revision starts a new list), status → executing, emits plan.proposed with
the persisted steps (id+seq) so the panel can address them.
- update_plan_step(seq, status, execution_id?): advances a step, stamping
started_at/finished_at, emits plan.step.started/finished. Anchors the event
to the step's target entity when it has one.
Belt-and-suspenders: when an execution linked to a step reaches a terminal
state, the api auto-closes the step (closePlanStepForExecution in
emitExecutionEvent) and emits plan.step.finished — so the board stays honest
even if the agent forgets to close a step it started.
Verified end-to-end: a goal-driven task fired goal.set → plan.proposed →
2× step.started/finished → task.status on the SSE stream; both steps persisted
done with start/finish timestamps; status progressed planning→executing→done.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the compounding knowledge loop the task model is built around:
- complete_task(outcome, summary): a nomos-LOCAL, session-scoped tool (the
shared MCP server has no session id). Introduces the local-tool mechanism —
buildTools appends task tools, the agent loop routes them to handleTaskTool
instead of the MCP client. Sets the task's terminal status/outcome/summary,
mirrors it onto the task entity, and emits task.status.
- Knowledge → task linkage: after a successful upsert_knowledge in a task,
nomos links the note to the task entity (documents) and emits
knowledge.recorded, so the task's outcome view shows what it learned. The
note's about-link to the involved entity (written by upsert_knowledge) is the
retrieval path future tasks use.
- SOUL: every chat is a task loop — retrieve prior knowledge FIRST
(get_entity_knowledge on the target), plan, execute, record learnings, then
complete_task. Scales down for trivial read-only tasks.
- deleteSession now cleans up the task entity, its relationships, and its
task-scoped events (was orphaning them); the knowledge doc itself and its
about-links survive, as knowledge should outlive the task.
Verified end-to-end: a task recorded a note and completed; task.status +
knowledge.recorded hit the SSE stream; status=done/outcome=success persisted;
the note linked to both lxc:caddy (retrieval) and the task; a future
get_entity_knowledge(lxc:caddy) surfaces it; delete cleaned edges+events (0/0/0)
while the knowledge survived.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
As the agent runs a task, record which entities each tool call references:
write an idempotent task —involves→ entity relationship and publish one
entity.touched event per entity (correlation_id = session, data {slug,tool}).
Extracted from tool ARGS only — never results — so a bulk fleet query can't
drag every entity into the task graph; bulk/no-slug tools stay silent.
Emitted from the nomos agent loop rather than the shared MCP wrapper, which
has no session id. The involves edges make a task's graph neighborhood its
involved-entity set (queryable via get_relations) — the substrate for the
knowledge loop; the events are the live pulse the context panel consumes in
phase 6.
Verified end-to-end on the local stack: a chat referencing lxc:caddy/lxc:gitea
produced entity.touched on the browser SSE stream with slug+tool+correlation,
and exactly one involves edge per entity despite repeated touches.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Migration 018 adds goal/status/outcome/summary/entity_id to agent_sessions
and creates session_plan_steps + session_questions. Registers a 'task'
entity type and an 'involves' (task→entity) relationship in the ontology
so each session anchors its knowledge and involved-entity edges on the
existing relationships graph.
nomos createSession now mints a task:<session-id> entity (type task) and
links it via agent_sessions.entity_id — best-effort so chat never blocks on
it. listSessions/GET /sessions surface the new task fields.
No behaviour change yet; this is the data foundation for the task board and
live context panel. Verified end-to-end against the local stack: migration
applied, ontology ingested (60 types/47 rels), a new session mints a linked
task entity and the API returns status/goal/entity_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframes the chat surface as a board of tasks: each task carries a goal,
a plan approved once, a lifecycle status, an outcome, and a knowledge
loop that links learnings to the involved entities (and the task entity
itself) via relationships so future tasks compound. Supersedes the
sidebar-only framing and the free-form chat portion of the control-room
web UI plan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audited all 10 active plan docs against the codebase (not just commit
titles). 5 were fully shipped and stale-tagged "Planned"/"In Progress" —
moved to done/ with verification notes. The other 4 got corrected
Planned→In Progress status plus concrete remaining-gap notes so the next
pass doesn't re-derive what's already done.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The plan's "learning view" (runbook success-rate trends, promoted
skills, capability timeline) assumes the patterns/skills/feedback
pipeline is populated. It isn't: all three tables are empty in
production and nothing in the codebase ever writes to feedback, so
building the UI against them today would ship a permanently-empty
page. Scoped instead around data that's real and growing —
executions — while still wiring up /patterns and /skills so the page
needs no rework once that pipeline exists.
New /api/v1/learning/timeline (per-verb first-success date + success
rate, parsed via the existing splitAction helper) and
/api/v1/learning/trend (30-day daily success/fail counts), both
read-only queries against executions. Patterns and skills sections
call the existing (untouched) ListPatterns/ListSkills endpoints and
render an explanatory empty state instead of nothing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live: a chat request to restart caddy (the reverse proxy for the
whole fleet) executed instantly over SSH with zero approval. Root
cause was in request_execution's legacy handler — restart, pct_exec,
and systemctl (outside enable/disable) executed immediately with a
hardcoded risk_class='reversible_low' that was never actually checked
against anything, bypassing the classifier entirely. Only the `run`
tool's commands were ever gated.
Extracted the run tool's classify -> execute-or-queue logic into a
shared classifyAndGate() and route restart/pct_exec/systemctl through
it too, so every mutating path — regardless of which tool the model
reaches for — gets the same read-only/config-mutation/destructive
classification and approval gate. systemctl restart is already covered
by an existing classifier test (config_mutation), so no new test
needed; the gap was that request_execution never called the
classifier at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pending-approval cards showed target and risk but not what else the
action would affect — the operator approved config_mutation/destructive
commands blind to downstream impact, even though the graph-walk
(blast_radius() SQL, GetBlastRadius endpoint) already existed and was
just never wired into the approval path.
Fetch it once per pending approval and render "Affects N downstream: …"
on both the normal and destructive approval cards, reusing the existing
fetchBlastRadius() API client function which was already written but
unused anywhere.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ops "Executions" tab showed raw target UUIDs, alphabetical (not
recency) order, and a stale status vocabulary from an earlier schema
iteration — never actually usable as a live "what's happening" view.
Replaced with a new recency-ordered /api/v1/activity/recent endpoint
and matching table (human-readable action summaries, risk/status
badges, duration, inline error preview).
Also added /api/v1/activity/session/{id} + a collapsible SessionDigest
panel in the chat rail, answering "what did this session actually do"
(executions by status, entities touched, knowledge written) — the
missing piece for proactive outcome reporting to be visible in the UI,
not just in the chat transcript.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified live immediately after deploying: the endpoint returned 200 with
correct-looking stats (total=56, agent_authored=2) but items=[] always,
regardless of limit/source. Root cause: pgx v5 can't scan a timestamptz
column directly into a Go string — Scan() errored on every single row, and
that error was silently swallowed by a bare `continue`, so every row was
dropped with no trace in the logs. Fixed by casting updated_at::text in the
SQL (matching how every other handler in this codebase already returns
timestamps) and logging scan failures instead of swallowing them, so this
class of bug can't hide silently again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First slice of the observability/learning UI (the "see the system come alive
and learn" ask). The Knowledge page was search-only — blank until you typed —
so the knowledge Nomos now writes via upsert_knowledge was invisible unless
you knew to search for it. Now the page LEADS with what the system knows and
is learning:
- internal/httpapi/knowledge.go: GET /api/v1/knowledge/recent — recency-ordered
knowledge + a stats header (total, agent-authored, learned-this-week,
by-kind). Custom route (not OpenAPI-generated), same auth as the rest.
- web Knowledge page rewrite: stat cards up top (Total / Written by Nomos /
Learned this week / runbooks-investigations), then a "Recently learned" feed
with agent-authored notes highlighted and badged "learned by Nomos", tags,
and relative timestamps. A toggle filters to Nomos-only. Search still works,
now as a mode you enter/clear rather than the whole page.
This turns "the system is getting smarter" from a claim into something you
watch fill up: every gotcha the agent records shows here within seconds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
From the last (successful) TypeType deploy session, two gaps the operator hit:
1. Knowledge write-back — the missing half of the loop.
The agent could read the knowledge base (search_knowledge/get_entity_knowledge)
but had no way to WRITE it, so everything it learned (the Dragonfly memlock
rlimit gotcha, the NAT-hairpin DNS issue, etc.) lived only in an ephemeral
chat message and was lost — the system could never actually "get better."
This is the `upsert_knowledge` MCP tool the 2026-07-08 gaps plan called for.
- internal/mcp/server.go: upsert_knowledge(title, content, about?, tags?,
kind?) writes a document/investigation/runbook entity + knowledge_entities
row (search column is generated), upserts by slug so re-titling updates in
place, and optionally links it to the entity it's about so
get_entity_knowledge surfaces it there.
- SOUL.md: capture non-obvious findings/deploys/gotchas as part of finishing
work, not only when asked "what did we learn".
2. "I had to ask for status multiple times."
The clearest cause: a long working turn (64 tool calls) that exhausted the
iteration cap ended with a bare "max iterations reached without final
answer" — a dead end that forced the operator to ask what happened.
- cmd/nomos/agent.go: on exhaustion, make one final no-tools LLM call
(finalSummary) asking for a status report — what was accomplished, current
state, what remains — so the turn always ends with a real outcome.
- maxIterations 25 -> 40 (the decomposed per-step pct_create flow legitimately
needs more steps).
- SOUL.md: always end a turn with a clear outcome; never end silently or on a
bare tool call — the operator can't see the tools working and reads silence
as "nothing happened".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: "I'd like to be able to see in the chat what the agent is actually
running, right now I just wait while nothing happens." Two compounding gaps:
1. The auto-continuation worker (cmd/nomos/continue.go) had zero live push —
its result only appeared on a manual page reload, so approving a plan and
watching the chat looked completely dead even while the agent was actively
working.
2. Even with polling, continueSession only persisted ONE message at the very
end of a continuation — a continuation that runs several tool calls before
concluding would still show total silence for however long that took.
Fixed both:
- web/src/lib/stores/chat.ts: polls the current session's messages every 3s
between turns (never while a live stream owns the message list) and merges
in anything new. Started after a live turn ends and when a session loads;
stopped on new-chat/session-switch.
- cmd/nomos/store.go: insertMessageReturningID/updateMessage — lets a message
be created as a placeholder and updated in place.
- cmd/nomos/continue.go: continueSession now inserts a placeholder the
instant it starts (renders as the existing "thinking" dots — immediate
feedback that something is happening) and updates that SAME row after
EVERY tool call, not just at the end. A poll within ~3s of any tool call
landing shows it — individual `run` commands appear as the agent issues
them, not just the final rolled-up summary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit fixed a context-cancellation bug in the auto-approve path
and appeared to fix things, but re-testing end-to-end after deploy showed the
execution STILL never completed — just via a different symptom
("no pending execution found for approval" in the logs). Dug further and
found the real, deeper bug underneath: this whole mechanism has never
actually worked.
autoApprove() directly flipped BOTH approvals.status and executions.status to
'approved' via raw SQL, then called executeApprovedViaAPI to POST to the
decision endpoint. But DecideApproval's own logic specifically looks for the
execution still at status='pending_approval' to find and dispatch the real
SSH work (executeApprovedAction) — autoApprove's premature flip meant that
lookup always found zero rows. DecideApproval's UpdateApprovalStatus call
also silently no-ops the same way (sqlc :exec doesn't surface "0 rows
affected" as an error). Every assent-window auto-approved pct_create/
apt_upgrade has been sitting at 'approved' forever with the real work never
triggered — indistinguishable from "still running" until you check.
Fix: remove autoApprove() entirely. Call executeApprovedViaAPI directly
against the untouched pending_approval row from createApproval — identical
to the manual Approve-button path, just without the human click. DecideApproval
is now the single place that transitions status and dispatches, for both the
manual and auto-approved paths, closing the class of bug where two code paths
raced to do the same state transition.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live testing the new atomic pct_create: an assent-window
auto-approved pct_create appeared to "run" (logged "auto-approved... running
now") but the execution stayed stuck at 'approved' forever. Root cause:
`go executeApprovedViaAPI(ctx, ...)` passed the MCP tool-call's own context —
which is cancelled the instant the triggering /chat request's HTTP response
completes, i.e. on every normal turn. The spawned goroutine's POST to the
approval-decision endpoint died with "context canceled" before it could even
start the real work, and nothing surfaced this to the operator or the agent —
the execution just sat at 'approved' with no error, indistinguishable from
"still running."
This is exactly the context-lifetime bug class httpapi's own approval
goroutine (executeApprovedAction) already avoided by using
context.Background() — it had just been missed in these two call sites
(apt_upgrade and pct_create auto-approve). Fixed both to use
context.Background(), matching the correct pattern already in place
elsewhere. Audited for other goroutines spawned with a request-scoped ctx —
none found; the sshExec internal goroutines are synchronous/waited-on via
select and correctly scoped to the call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the two remaining open points from the auto-continuation work.
1. Atomic pct_create (observability, the bigger of the two):
pct_create used to bundle create + apt install + post_install script into
one black-box multi-minute SSH call — the agent got back a single opaque
success/fail with no way to see (or fix) which step actually broke.
Removed the whole post-create provisioning block (and the now-dead
provisionScript/sanitizePkgs helpers + their tests). pct_create is now
create + start + register ONLY — fast, and its result is fed back to the
agent via auto-continuation almost immediately. The agent installs
packages and runs setup as its OWN sequence of `run` calls against the new
lxc:<hostname>, observing each command's real output and able to diagnose
and retry exactly the step that failed — the same recovery loop already
proven for the general case, now applied to installs too, instead of
requiring a separate black-box mechanism.
- services/post_install removed from the pct_create params struct and
from the MCP tool schema/SOUL.md docs.
- SOUL.md: explains the new flow, moves the Docker CLI gotcha and DNS
troubleshooting guidance to be steps the agent runs itself.
2. Scoped destructive window (targeted autonomy for recovery):
Verified live in the previous session that a destructive recovery (a
failed destroy needing stop-then-destroy on the same container) required
TWO separate typed confirmations for what was clearly one recovery
action. Added a narrow, TARGET-scoped 15-minute grant
(destructive_window.agent:<id>.target:<slug> in autonomy_settings,
shared key format across cmd/nomos and internal/mcp) that opens only
after an EXPLICIT typed confirmation (never loose assent) or an explicit
button-approval of a destructive step, and only ever covers further
destructive commands against that SAME target. A different target always
needs its own fresh confirmation — this narrows risk instead of loosening
it globally, unlike broadening the general assent window to cover
destructive actions would have.
- cmd/nomos/store.go: openDestructiveWindow/destructiveWindowActive/
executionTarget.
- cmd/nomos/agent.go: opens the window when a typed confirmation grants a
destructive chat-assent execution.
- internal/mcp/server.go: `run` tool checks the window before gating a
destructive command; auto-runs if active.
- internal/httpapi/phase3.go: DecideApproval opens the same window when a
destructive execution is approved via the button/API, for parity with
the chat-assent path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live: the new auto-continuation worker (previous commit) worked end
to end for the happy path (provision -> auto-verify -> report success, zero
operator ticks). But testing a failure-recovery case (a destroy that failed
because the container was still running) surfaced a real bug: continueSession's
emit closure only captured "text" events, so when chatWith ended the turn on
an "error" event (LLM returned an empty/refusal response, internal retry also
empty), the worker persisted a completely blank, uninformative "auto" message
— no sign anything had gone wrong, undermining observability of the very
mechanism just built.
- Capture "error" events and, if the turn produced no text/tool_calls at all,
persist an explanatory placeholder instead of blank.
- Add one outer retry of the whole chatWith call when the first attempt
produces nothing — the principle behind this whole feature ("don't give up
on the first error") should apply to the continuation mechanism itself, not
just the homelab commands it's continuing.
Also verified live: recovery-from-failure works via the normal chat path once
prompted, and cleaned up the test container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three fixes for the session where the agent proposed a plan, waited
for 'proceed', then re-queued instead of being auto-approved:
1. Chat-assent fallback: when the operator says 'proceed' but the
preceding turn had NO pending approvals (agent proposed plan in text
without calling request_execution), inject a system note telling the
agent to execute the plan now. Opens the assent window so subsequent
config_mutation commands auto-run.
2. SOUL.md: instruct agent to ALWAYS call request_execution/run when
proposing a plan, not wait for 'proceed' first. This ensures a
pending approval exists for chat-assent to grant.
3. SOUL.md: stronger Docker instructions — Debian 13's docker.io package
installs the daemon but NOT the docker CLI binary. Must use
get.docker.com in post_install. Added 'handling errors' section:
diagnose, try alternatives, continue — don't stop after one failure.
When the operator has approved a plan via chat assent (assent window
active), pct_create and apt_upgrade now auto-approve and execute
instead of queuing for a separate approval round. The auto-approve
path updates the approval+execution status in the DB, then calls the
HTTP API's decision endpoint to trigger executeApprovedAction — same
code path as a manual Approve button, consistent audit trail.
Agent stopped after every approval step, forcing operator to type
'continue' 7× per deploy session. Root causes and fixes:
1. Compound read-only commands (e.g. 'systemctl status; journalctl')
defaulted to config_mutation — now splits on ;/&&/||/| and classifies
as read_only if all segments are inspection verbs. Added grep, wc,
sort, uniq, cut, tr, dpkg -l, apt list, docker stats to allowlist.
2. curl|sh was classified destructive, forcing typed confirmation for
legitimate installs (get.docker.com). Demoted to config_mutation —
loose assent grants it, no typed phrase needed.
3. SOUL.md said 'STOP after queuing' — replaced with 'continue working
on non-blocked steps'. Added assent window section instructing agent
to carry out the full plan after approval.
4. Assent window: when operator approves a plan via chat assent, a
30-minute window opens where config_mutation commands auto-run
without re-approval. Agent writes expiry to autonomy_settings; MCP
run tool checks it before gating. Destructive never auto-runs.
5. System note after approval now says 'CONTINUE executing the full
plan — do not stop and wait for continue.'
Verified live that after deploying the "fixed" bridge-bound pre-flight, it
still let a known-bad vmbr0+192.168.8.2 config straight through to a full
pct_create with no error. Root cause: the check used
`strings.Contains(pingOut, "REACHABLE")` against markers "REACHABLE" /
"UNREACHABLE" — but "UNREACHABLE" contains "REACHABLE" as a substring, so the
containment check was true for BOTH outcomes. The pre-flight was structurally
incapable of ever failing, regardless of the actual ping result.
Fixed with distinct, non-overlapping markers (PREFLIGHT_OK/PREFLIGHT_FAIL)
and exact-match comparison, pulled into a small gatewayPreflightPassed()
helper with a unit test asserting the exact historical bug case
("UNREACHABLE" must be false) so this bug class can't silently recur.
Re-verified live end-to-end: manually re-tested the exact ping command
(confirmed UNREACHABLE via vmbr0), and this was caught only by actually
running the check against production, not by reading the code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live that the pre-flight check added in the previous commit had a
real gap: a plain `ping <gateway>` from the Proxmox host succeeds via the
HOST's own routing table (which can have routes to a subnet through paths
the host alone knows about), even when the CONTAINER — attached via a plain
bridge with only a naive on-link default route — can never actually ARP that
gateway. Confirmed by creating a real test container on vmbr0 with
gw=192.168.8.2: the host-wide ping had said "reachable," but pinging from
inside the container showed 100% packet loss. Fixed by binding the pre-flight
ping to the specific requested bridge (`ping -I <bridge>`), which correctly
rejects vmbr0 for that gateway instead of false-positiving via the host's
broader routing table.
Also confirmed live: vmbr1 does exist and is up on strong (contrary to the
possibly-stale host doc), matching what romm/seanime's docs already said.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Investigated why the operator couldn't get past "no DNS/connectivity" across
multiple retries even after Nomos correctly diagnosed and fixed the gateway
(192.168.8.1 -> 192.168.8.2). It still failed. Root cause, confirmed from
strong's own documented network topology: on `strong`, vmbr0 physically
bridges only to 192.168.178.0/24 — the 192.168.8.0/24 service network is
reached via a Fritz!Box static route, not a local bridge. A container
attached to vmbr0 can never reach a 192.168.8.x gateway no matter which
address in that range is picked; ARP for it just gets silently dropped
(matching the earlier hang symptom). The gateway was never the problem — the
bridge was. 192.168.8.0/24 is also segmented into /28 blocks each with their
own gateway (192.168.8.2 is only the .0-.15 block's gateway), so even a
correct bridge with a copy-pasted gateway from a different block would still
fail.
No amount of retrying with a different gateway guess could have fixed this —
the missing fact (which bridge reaches which subnet, and the per-/28 gateway)
isn't inferable from the subnet alone.
- pct_create gets a `bridge` param (was hardcoded to vmbr0) so a correct
bridge can actually be requested once known.
- Fast pre-flight: for any static IP, ping the gateway from the target HOST
before creating anything. Was: a bad config took a multi-minute hang (or,
after last commit's timeout fix, ~2min) before failing. Now: ~2 seconds,
with a message that explicitly says not to guess a different gateway in
the same subnet — find a real neighbor's config or use DHCP.
- SOUL.md: DHCP is now framed as the default, not a fallback; static IP
requires finding an existing LXC on the same host in the same /28 and
copying its bridge+gateway verbatim — inventing one is explicitly called
out as the failure mode that caused this exact incident.
- MCP tool schema: pct_create's params description now documents `bridge`
and the neighbor-copy rule directly in what the model reads at call time,
not just in SOUL.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).
Two compounding bugs made this unrecoverable without manual intervention:
1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
NO execution timeout — `session.CombinedOutput()` blocks until the remote
command exits, with no deadline. A hung remote process blocks the Go
goroutine forever; the execution can never leave 'running', and the
operator has no way to make it stop. Fixed: both now race the SSH call
against a 10-minute hard timeout, closing the session/client and
returning a clear "timed out after 10m0s" error if exceeded. (The
mcp/server.go copy also still had the original "swallowed non-zero exit"
bug from before that fix was applied to httpapi's copy only — fixed here
too.)
2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
connectivity — it doesn't; a black-holed network can make each call hang
far past the resolver's nominal timeout, so the documented "~90s" budget
was never real. Wrapped every attempt in `timeout 3` so the wall-clock
budget is now actually enforced (~2min worst case), and the failure
message now suggests checking the net0 gateway.
Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):
- InlineApproval's track() polling loop had its own ~6min ceiling and simply
STOPPED polling after that — silently going stale before the backend (now
correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
with margin, and added a distinct 'stalled' state if that's ever exceeded
(explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
execution's created_at), the actual command being run, and the execution
ID — previously just a static "this can take a minute" with zero
information. Also added command display to the destructive pending-
approval card for full transparency before confirming.
Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "chat gave me no further feedback — had to go to Ops": two
compounding bugs, found by reading the actual production session transcript.
1. InlineApproval.svelte — all of last session's live-status/self-heal work —
was never imported or rendered anywhere. Chat.svelte had its own separate,
much dumber approval bar (no status tracking, no destructive handling, just
silently disappears after clicking) that WAS the one users actually saw.
Deleted the dead bar and its state; InlineApproval now renders per-message.
2. chat.ts's extractApprovals hardcoded `tool.name === 'request_execution'`,
so any approval raised by the newer `run` tool was invisible — no card, no
feedback, nothing to self-heal, forcing the operator to the Ops page with
zero acknowledgement in the conversation. This was the actual proximate
cause of last night's destroy-135 session. Fixed to match on response
shape, not tool name, so it doesn't silently break again for the next new
gated tool.
3. Nomos was telling operators "type something like 'I confirm destroy 135'"
for destructive actions (SOUL.md) but no backend path ever consumed that
phrase — chat-assent explicitly (and correctly) excludes destructive from
loose assent, but I never built the alternative. Added
isTypedConfirmation() (cmd/nomos/assent.go): stricter than loose assent,
requires an explicit "confirm" statement, only applies to destructive-
flagged pending approvals.
4. InlineApproval's completed-state hardcoded "Provisioned successfully" —
wrong/confusing for a destroy or arbitrary `run` command. Now says
"Completed on <target>" and shows the actual command output, verified live
against the real destroy-135 execution.
Verified live in a real browser against the production API/DB (dev server
proxying to :8090): the historical stuck session now retroactively renders
both executions as resolved with correct wording and real output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live testing hit `entities_slug_key` violations: exec slugs used an 8-char
prefix of a UUIDv7, whose leading bytes encode a millisecond timestamp — two
executions created seconds apart can share a prefix. Use the full UUID
(guaranteed unique) for the exec entity's slug/name in request_execution, the
new `run` tool, and the REST RequestExecution handler — all three had the
same pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found live: `run` against lxc:caddy failed with "missing pve_id" even though
pve_id=121 was present — caddy is an inventory-seeded LXC with no `host`
attribute at all (only pct_create-provisioned LXCs set one). The combined
query scanned attributes->>'host' (SQL NULL) into a plain Go string, which
errors the whole Scan — including the pve_id column that scanned fine.
COALESCE the host column to '' so a missing host attribute degrades to the
documented default instead of failing the whole resolution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the first slice of plans/2026-07-10-general-gated-execution.md:
Nomos gets one general execution tool instead of only a fixed action enum,
gated by an automatic risk classifier, and approval can be granted by the
operator just replying in chat instead of clicking a button.
- internal/policy/command.go: ClassifyCommand(cmd, declaredRisk) — rule-based
read-only allowlist + destructive denylist, default-escalate to
config_mutation for anything else. Classification can only ESCALATE the
caller's declared risk, never de-escalate it (destructive always wins even
if declared read_only). Compound commands (&&, ;, |, $()) never qualify for
the read-only fast path. Full test corpus.
- internal/mcp/server.go: new `run` MCP tool — target (host:/lxc:), command,
purpose, optional declared_risk. Read-only commands execute immediately;
everything else queues an approval exactly like pct_create today, executed
via httpapi's existing executeApprovedAction. Also fixes a real latent bug:
pct_exec resolved an LXC's host attribute without the "host:" prefix, so it
could never find the Proxmox host — new resolveExecTarget/resolveRunTarget
helpers (mcp + httpapi) fix this for both the new `run` action and existing
actions that route through the same execution path.
- internal/httpapi/phase3.go: "run" case in executeApprovedAction; fixes two
bugs found while wiring this up — (1) DecideApproval hardcoded risk_class to
'config_mutation' on every approve, silently corrupting the audit ledger for
every other risk class; (2) denying/revoking an approval never updated the
linked execution's status, so it stayed 'pending_approval' forever instead
of reflecting the decision.
- cmd/nomos/assent.go: deterministic (not LLM-judged) chat-assent detection.
Scoped to the immediately-preceding assistant turn's pending approvals only
— an old "yes" can't retroactively approve something new. Destructive-risk
actions are excluded from loose assent. Approves via the same HTTP decision
endpoint the UI button calls, so both paths share one audit trail.
- web/.../InlineApproval.svelte: self-healing poll — a pending approval card
now picks up being decided via ANY path (chat assent, Ops page, Matrix), not
just its own button. Previously the banner stayed stuck showing
Approve/Deny even after the action had already run elsewhere.
- nomos/SOUL.md: `run` is now the general capability ("no fixed menu, only a
risk gate"); documents chat-assent behavior and the destructive exception.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator directives: (1) the UI must show what's executing, its risk
classification, live status, and what knowledge the session created — the
system's growth should be visible, not just trusted. (2) approval should be
granted by chat assent ("go ahead"), not a separate button; destructive
actions still require a typed confirmation phrase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Evaluate the agent's action path against the OIKOS.md design. Finding: the
intended model (unlimited runbook-driven actions gated by a risk classifier)
already exists on paper and in scaffolding, but the live agent path regressed
to a hard-coded 5-action enum that bypasses the classifier. Plan a layered
realignment: (0) one general gated `run` primitive, (1) runbooks-as-data as the
reliable fast-path, (2) learning. Chosen v1 posture: approve-most (read-only
auto-runs, all state changes gate). Incremental, each step shippable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fresh Debian LXCs have no locale configured, spamming "apt-listchanges:
Can't set locale" / perl warnings across every install and breaking some
packages' post-install scripts. Pin LANG/LC_ALL=C.UTF-8 (and hoist
DEBIAN_FRONTEND) at the top of the in-container bootstrap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Production session provisioned the container but the service never installed:
apt failed with "Temporary failure resolving deb.debian.org" — a static-IP LXC
whose assigned nameserver couldn't resolve. The operator also got zero feedback:
the approval banner just sat there with no running/complete/failed status.
Backend robustness (provisionScript):
- Wait for real DNS/connectivity inside the container before apt, and self-heal
/etc/resolv.conf to a public resolver (1.1.1.1/8.8.8.8) if the assigned one
is dead. `set -e` after the gate so apt/post_install failures surface.
- apt-get update/install with Acquire::Retries=3.
Frontend feedback (InlineApproval):
- After approve, poll GET /executions/{id} and show live phase: submitting →
provisioning… → provisioned successfully / execution failed (with the error).
- add getExecution() to api.ts.
Agent guidance (SOUL.md):
- omit vmid (auto-assigned), prefer dhcp, docker-compose-plugin is not in Debian
(use docker.io + get.docker.com), end post_install with a health check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The final "UPDATE executions SET result=$::jsonb" built its payload with
fmt.Sprintf and only escaped newlines. apt/pct output contains quotes,
backslashes and control chars, so the payload was invalid JSON, the jsonb
cast failed, and the (unchecked) UPDATE was silently discarded — the
execution stayed 'approved' with a NULL result even though the LXC was fully
provisioned (verified live: vmid auto-assigned, container running, service
installed, post_install ran).
- executeApprovedAction: marshal result via json.Marshal; log UPDATE errors
- add jsonErr() helper; route all pct_create failure-path results through it
- mcp/server.go: add jsonOut() for restart/systemctl/pct_exec inline results
- regression test for JSON validity on quote/backslash/control-char output
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups found while verifying the approve→provision path end to end:
- vmid is now optional: the early required-field check rejected vmid:0
before the cluster VMID guard could auto-assign a free id. Only hostname
is required now; 0 (or a collision) resolves to `pvesh get /cluster/nextid`.
- net0: use ip=dhcp with no gateway when no static IP is given (Proxmox
rejects gw alongside dhcp); only attach gw for a static CIDR.
- bump post-create settle to 10s so a DHCP lease is up before apt runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real production failure when the operator clicked Approve in chat: nothing
provisioned, banner never cleared, execution marked completed.
Three root causes:
- sshExec swallowed non-zero exits when the command produced output, so a
`pct create` that printed "CT 132 already exists" and failed was reported
as success and a bogus lxc entity was registered. Now any non-zero exit
returns an error (with output) so the execution is correctly marked failed.
- The LLM reused VMID 132 (belongs to lxc:rclone; VMIDs are cluster-wide).
pct_create now checks in-use VMIDs via `pvesh get /cluster/resources` and
falls back to `pvesh get /cluster/nextid` when the requested id is taken.
- InlineApproval.svelte reset its state on every prop change (done was also
compared against the wrong string), so the banner never cleared and each
click re-POSTed /decision. Rewritten to track outcome per executionId,
clear on success, and block resubmits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "asks permission but never acts": the approved pct_create
execution failed to parse because the LLM emitted `"privileged":0` /
`"nesting":1` (numbers) into strict `bool` fields, so the container was
never created. Compounded by a hardcoded template name (debian-13.0-1)
that no longer exists on the host, and no way for the agent to read the web.
- flexBool: accept 0/1, "true", bool for privileged/nesting (the exact prod failure)
- pct_create template pre-flight: list host cache, validate/auto-pick newest debian
- pct_create services[] + post_install: one approval provisions a working service
- new http_get MCP tool (sanitized, size-capped, SSRF-guarded) — agent can read repos/sites
- request_execution description: target=host, full JSON schema + example
- SOUL.md: agent CAN fetch the web; prefer one-step provisioning
- default model deepseek-v4-flash -> v4-pro; maxIterations 15 -> 25
- unit tests for flexBool, template resolve, pkg sanitize, HTML sanitize + SSRF block
Verified live on host:strong with a throwaway VMID 999: template auto-resolved,
container created + booted, services installed, post_install ran, then destroyed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Execution entity name now includes UUID suffix: 'pct_create on host:strong (abc12345)'
so the (type,name) UNIQUE constraint doesn't block subsequent executions for the
same target+action. Dedup now uses JOIN + LIKE prefix match to find only
pending_approval executions.
- Move persistent approval bar from top of messages area to just above the
chat input box (bottom-fixed position, above the textarea form).
- Add dedup in request_execution: check entities(type,name) uniqueness before
creating duplicate executions. Returns 'already queued' message to the LLM,
preventing tool-calling loops.
- Fix createApproval JSON payload: use json.Marshal instead of fmt.Sprintf
to escape params (could contain unescaped double quotes from JSON config).
- Add ON CONFLICT DO NOTHING to entity/execution inserts for dedup race safety.
- Persistent approval bar at top of Chat.svelte: aggregates pendingApprovals
from all messages, fixed position (won't scroll away). Approve/deny/approve-all.
- Update SOUL.md: agent must STOP after queuing a gated action.
- Fix ToolCallGroup reactivity: wasActive = (active).
- Replace text-based regex parsing in InlineApproval with structured
pendingApprovals extracted from request_execution tool results. The tool
result text is deterministic (not LLM-generated), making UUID extraction
reliable regardless of how the LLM rephrases the response.
- Fix ToolCallGroup reactivity: wasActive = active captured initial
value. Now uses (active) so re-runs on prop changes.
- Extract approvals in both live streaming (done event) and history loading
for consistent behavior on resumed sessions.
- Empty/refusal responses retried once, then surfaced as errors instead of silent blanks
- Chinese refusal boilerplate detected via denylist + non-ASCII heuristic
- Bulk-tool preference added to SOUL.md (list_lxcs over per-entity get_lxc_state)
- Tool results truncated to 4KB on persist; get_state_snapshot filters null-state entities
- Session delete (DELETE /sessions/{id} + confirm-on-second-click UI)
- Session titles auto-generated from assistant answer instead of raw user message
Inspected the live agent_sessions/agent_messages tables on mac-mini and
found silent empty responses, a canned non-English refusal after 22 tool
calls, 70-call fan-out for simple fleet questions, 100KB+ persisted
messages, and no session delete/title hygiene.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two chat UX changes (share Chat.svelte, committed together).
Tool-call grouping (ToolCallGroup.svelte):
- Replaced the one-<details>-per-tool-call list with a single
collapsible group per assistant turn, headed by tool count + a
name preview and a status icon (spinning wrench in progress, check
done, X on error).
- The group auto-collapses the instant its turn finishes streaming, so
a completed round shows as one compact pill; historical/loaded turns
start collapsed. The auto-collapse fires once at the streaming→done
transition, leaving manual toggles alone afterward.
Session graph rail (SessionGraph.svelte) — replaces the old
ContextRail (fleet health / pending approvals / live events), which is
deleted:
- A force-directed graph that starts empty (animated constellation
empty state) and grows as the conversation references entities.
Slugs are extracted from message text and tool *arguments* only —
never bulk result rows, so a single get_health_summary doesn't dump
all 168 entities — then validated against the backend via fetchGraph
(cached) with check/execution probe entities excluded. Nodes are
colored by health; edges appear once both endpoints are present.
- Clicking a node highlights it and its neighbors and opens an inline
detail panel below: slug/type/state, health + freshness, top
attributes, in-graph relations (clickable to hop), and a Full detail
button opening the entity sheet.
- The rail is resizable via a drag handle (260–620px, persisted to
localStorage). The header's global fleet-health dots are unchanged;
only the right-rail content was replaced.
Risk: reversible_low (UI-only). The slug extractor is scoped to
focused mentions by design; edges may be slightly incomplete since
only root-fetched entities contribute edges, which is acceptable for a
session overview.
Verification: verified in the browser preview — loading a real session
built a 4-node graph (hubris/caddy/netbird-vps/strong) with the
hubris→caddy relationship edge; clicking hubris showed
"proxmox-host · active · healthy · checked 40s ago" with attributes and
relations; dragging the handle resized 320→440px and persisted; a
34-tool historical turn renders as one collapsed pill that expands on
click. tsc clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three UI issues reported after the neutral-gray redesign:
- favicon.svg was still filled #58a6ff (the pre-redesign accent blue);
changed to white to match the sidebar logo mark.
- Sidebar nav items all showed a filled background even when inactive.
Root cause: sidebar-menu-button.svelte (and -sub-button) rendered
`data-active="false"` as a literal attribute, but Tailwind's bare
`data-active:` variant matches attribute *presence*, not value — so
data-active:bg-sidebar-accent applied to every item regardless of
state. Fixed by emitting the attribute only when active
(`isActive || undefined`), a latent bug in the vendored shadcn
component that read as intentional until flagged.
- Tailwind's preflight resets <button> to cursor: default, so no button
in the app showed a pointer. Added one base rule restoring
cursor: pointer for buttons, [role=button], links, summary, and
select (respecting :disabled / aria-disabled) rather than annotating
each call site — covers new interactive elements automatically.
Risk: reversible_low (UI-only).
Verification: verified in the browser preview that inactive sidebar
items are transparent (only the current page shows a background),
nav buttons report cursor: pointer via computed styles, and the
favicon renders white in the tab.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: two issues surfaced after the dashboard-01 shell change
(cbfd09c). (1) Sidebar.Inset previously had an explicit h-svh that
hard-capped the app's height at the viewport; adding variant="inset"
put a margin on that same fixed-height box, pushing it taller than the
viewport with nothing left in the chain to cap it (Sidebar.Provider's
own wrapper only sets min-h-svh — a floor, not a ceiling). Result: the
whole page scrolled as one long document instead of each page's own
content scrolling internally with the header pinned — confirmed via
computed styles, e.g. Entities.svelte's table wrapper measured
scrollHeight 6531px against a 900px viewport, all of it spilling past
body instead of scrolling in its own rounded-border container.
(2) The color palette was GitHub-dark-inspired (blue-tinted grays:
#0d1117 bg, #58a6ff primary/accent) rather than the neutral grays the
shadcn-svelte dashboard-01 reference actually uses.
Change:
- App.svelte: moved the height cap up to Sidebar.Provider itself
(class="h-svh") instead of Sidebar.Inset, since the cap needs to sit
above wherever the inset variant's margin gets applied, not on the
same box as the margin.
- app.css: replaced the core tokens (background/foreground/card/
popover/primary/secondary/muted/accent/border/input/ring/sidebar-*)
with shadcn's canonical dark-theme OKLCH values (0-chroma neutral
grays), pulled directly from huntabyte/shadcn-svelte's own
docs/src/app.css rather than approximated. --success/--warning
deliberately kept as real, distinguishable colors — they signal
actual health state, and desaturating them to match the neutral
chrome would reintroduce the "can't tell what's actually happening"
problem this whole project started from (see 279549c). --accent-blue
now aliases --sidebar-primary (still a real blue) instead of
--primary, so the couple of spots wanting an interactive "pop" still
have one while buttons/links/focus rings ride the neutral --primary.
Risk: reversible_low (UI-only).
Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). go build/vet clean (backend
untouched, sanity check only). Manually verified in the browser
preview at 1400px: document.body.scrollHeight now exactly matches
window.innerHeight on both Overview and the 193-row Entities table
(previously 6531px vs 900px); scrolled the Entities table wrapper to
row ~60 and confirmed the header/filter bar/column headers stay
pinned while only the table body scrolls; confirmed neutral gray
rendering across Overview's stat cards, the event-rate chart, and
Chat's tool-call list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problem: requested visual alignment with the shadcn-svelte dashboard-01
reference block (shadcn-svelte.com/blocks#dashboard-01) — the app's
sidebar/header shell and Overview stat cards looked plain by comparison.
Change: pulled the actual reference source (app-sidebar.svelte,
nav-main.svelte, site-header.svelte, section-cards.svelte from
huntabyte/shadcn-svelte) rather than approximating from screenshots.
- App.svelte: Sidebar.Root now uses variant="inset" (the floating,
rounded, shadowed content panel — already fully built into the
existing Sidebar.Inset component via peer-data selectors, just never
enabled). Brand mark is now a proper Sidebar.MenuButton matching the
reference's padding/hover treatment; "New chat" uses the reference's
primary-colored button styling. Header matches the reference exactly:
h-(--header-height) (48px, was 44px), vertical separator after the
sidebar trigger, right-aligned actions group.
- Overview.svelte: stat cards rebuilt to match section-cards.svelte —
gradient background, Card.Action badge, Card.Footer with a bold line
+ muted context line, tabular-nums, responsive @container grid
(1/2/4 columns). Deliberately did NOT copy the reference's fake
trend-percentage badges (Oikos doesn't track historical trends, and
this project's whole thrust has been eliminating dishonest UI state —
see 279549c). Badges instead reflect real current-state signals
(healthy/degraded/down, clear/needs-review) computed from the actual
dashboard summary.
- EntityDetailContent.svelte + Entities/Signals/Ops/Events/Agent/
Audit/Knowledge pages: normalized root padding to p-4 md:p-6 (was a
flat p-6) to match the reference's responsive py-4 md:py-6 convention.
Risk: reversible_low (UI-only, no data or behavior changes).
Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings, same as prior commits). go build/vet
clean (backend untouched, sanity check only). Manually verified in the
browser preview at 1400px: inset sidebar's margin/rounded-corner/shadow
classes confirmed applied via computed styles; Overview cards render
with real live numbers from the now-fixed dashboard summary endpoint;
Signals/Ops pages confirmed visually consistent with the new spacing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problem: the web UI felt dead and hard to navigate — the Entities table
had no health/freshness signal (just a meaningless row-mutation
timestamp), no way to see what was actually monitoring an entity,
sessions couldn't be reopened, and every drill-down was a full page
navigation that lost the list.
Change:
- Entities table: Updated column replaced with a health dot + relative
"checked Xm ago", sourced from the backend's new health/last_check_at
fields.
- New EntityDetailContent.svelte extracted from EntityDetail.svelte and
shared between the full #/entity/:slug page and a new EntitySheet.svelte
opened from the Entities table (master-detail, row click opens a panel
instead of navigating away). Adds a Monitoring card listing the
entity's check_defs (kind, interval, enabled/disabled with
click-to-toggle via the existing PatchCheck endpoint) and renders
attributes as key/value pairs instead of raw JSON.
- Sessions: fixed a bug where clicking a session loaded it into the
chat store but never navigated to the chat page, so nothing appeared
to happen. Added a SessionRail inside Chat so switching sessions
never leaves the chat surface.
- Fixed the local dev proxy (vite.config.ts): production Caddy strips
the /agent prefix before forwarding to nomos; the dev proxy didn't,
so every session/chat fetch 404'd locally while working in prod.
- Found and fixed a real latent bug while testing the session fix:
chat.ts's loadSessionMessages passed the persisted tool_calls array
straight through, but nomos stores the tool_use and tool_result as
two entries sharing one id. Chat.svelte's keyed {#each tool (tool.id)}
throws on the duplicate key, which silently blanked the entire
message list — invisible until sessions were actually clickable.
Fixed by merging tool_calls by id before rendering, matching the
shape the live-streaming path already produces.
- UI polish: sidebar logo is now just the omicron mark in white (was
icon+text in the accent color); removed the sheet overlay's
backdrop-blur (distracting per feedback); the Attributes/Relations/
Signals grids used viewport-based lg:/3xl: breakpoints, which forced
multi-column layouts based on browser width regardless of the sheet's
actual rendered width — switched to Tailwind v4 container queries
(@lg:/@2xl:/@3xl:) so layout responds to the real available width in
both the full page and the narrower sheet.
Risk: reversible_low (UI-only; no destructive operations; the tool_calls
merge and dev-proxy fix are corrections to broken paths, not behavior
changes to working ones).
Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). Manually verified in the browser
preview against the live dev API: Entities table health column renders
correctly; clicking a row opens the EntitySheet with a populated
Monitoring card (16 checks for host:hubris, verified via psql that
check_defs.target_id links them correctly); clicking a session now
loads its full transcript inline (was blank before the tool_calls fix);
sheet has no blur and lays out single/multi-column correctly at the
sheet's actual width.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.
Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
cd.TargetID (falling back to the check's own id if unset) and write
status/metrics/events there. Signals stay keyed by the check entity,
unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
observation is older than 3x its fastest enabled check's interval
(floor 5m) is marked 'stale' and emits health.stale, so a stalled
scheduler or disabled check_def can no longer look like current data
forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
rows so dashboard/fleet-health rollups stop double-counting probes as
monitored entities. Historical metric_samples on check entities are
left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
the conversation instead of dropping them (previously only final text
was replayed, forcing the agent to re-derive fleet state every turn),
and inject a compact live fleet-health snapshot into the system prompt
each turn so it starts oriented instead of spending an iteration on
discovery.
Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.
Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- disk_usage_check.sh: sed 's/-/0/' for filesystems without inodes
- checkdefaults.Ensure: includes target_id in check_defs INSERT so
signals get proper target slug instead of null
Add docs/signal-triggers.md covering:
- End-to-end sequence diagram (Nomos → API → Scheduler → target host)
- Two paths: autonomous collection (scheduler) + query (MCP)
- All 6 check kinds and 17 available scripts
- Script deployment flow via sync timer
- Signal lifecycle, threshold evaluation, data flow through DB tables
- Prerequisites for SSH checks in Docker
- InsertMetricSample now includes ts=now() (TimescaleDB hypertable requires it)
- ssh-script: pass host and port separately (ssh uses -p flag, not host:port)
- ssh-script: use OIKOS_SSH_USER from config/env, default root
- Add -o LogLevel=ERROR to suppress SSH warnings polluting JSON output
- Use Output() (stdout only) instead of CombinedOutput()
- Set OIKOS_SSH_USER=root in scheduler docker-compose service
- Switch Dockerfile from distroless/static to alpine:3.21
- Install openssh-client-default in runtime image
- Mount SSH key in scheduler service (docker-compose)
- Add NET_RAW capability for ping checks
- Wire OIKOS_SSH_KEY_PATH and OIKOS_SSH_USER env vars in scheduler
- sshExec uses configured key path with StrictHostKeyChecking=no
Add three new pages completing the control-room web UI:
- Agent activity: polls /agent-activity every 5s, filterable by type/agent
- Knowledge search: FTS over /knowledge/search with snippet + entity links
- Audit trail: browseable audit log with actor/action/entity filters
Enhanced live events page with correlation-id clustering (Groups toggle).
Added fetchAgentActivity/searchKnowledge/fetchAudit to the API client.
11 nav items now cover all planned control-room views.
After an api (MCP server) restart, nomos held a dead session id and every
tool call failed with "unexpected end of JSON input" until nomos was manually
restarted — which happens on every deploy. The MCP client now detects a
rejected session (4xx or empty body) and transparently re-initializes and
retries once. Also raise the SSE scanner buffer to 4MB so large tool results
don't exceed the 64KB default token limit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
http.FileServer canonicalizes /index.html -> "./", which for /ui/ produced a
301 redirect loop and made the control room unreachable. Serve embedded files
directly with http.ServeContent instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nomos (from oikonomos, steward of the oikos) avoids the name collision
with Nous Research's Hermes Agent. N0 enumerates the full rename scope:
cmd/, hermes/ dir, env vars, config fields, compose service, Caddy vhost,
identity-preserving DB slug migration + seed update, persona docs.
History, the Matrix bot user, and legacy bin/hermes stay untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Swap anthropic-sdk-go for openai-go against the OpenRouter API; default
model deepseek/deepseek-v4-flash with Exacto routing and ZDR provider
pinning. Record why Nous Hermes Agent (and hosted MCP connectors) were
rejected for the resident role.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hermes becomes an LLM-backed agent loop (hand-rolled tool loop over the
existing mcpClient, not the public MCP connector), with Postgres-backed
sessions, SSE chat streaming, and Authentik-gated /agent routing. The
control-room plan is amended to make the chat the main entry point.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documents a full-project review (confirmed bugs, security gaps, user- and
agent-perspective gaps) and a realtime control-room web UI plan, per prior
codebase exploration on this branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Consolidate Oikos on mac-mini (2026-07-06)
- Client lifecycle in Go (2026-07-07)
- Comprehensive audit & next steps (2026-07-07)
- DB as source of truth (2026-07-07)
- MCP tool completion (2026-07-07)
Paths fixed in index.md to reflect planes/done/ locations.
Active plans remaining: Prometheus LXC (Planned), implementation audit (Active).
Removed via Gitea API using Infisical-stored token:
- Hook 10: 192.168.8.205:9811 (homelab-mcp-deploy)
- Hook 11: 192.168.8.205:9821 (secrets-issuance-deploy)
- Hook 14: 192.168.8.205:9831 (oikos-console-deploy) — found during cleanup
All three pointed to apps/105 (192.168.8.205). No remaining webhooks
fire to the old Python stack. Zero hooks remain on dtoro/oikos.
Cutover checklist updated. Only item remaining: archive apps/105 LXC.
scripts/cleanup-apps105-webhooks.sh: lists webhooks on dtoro/oikos,
filters for apps/105 deployment endpoints, prompts for confirmation,
deletes matching hooks via Gitea API.
Usage: GITEA_TOKEN='...' ./scripts/cleanup-apps105-webhooks.sh
Also updated cutover-checklist.md with the script reference and LXC
archive command.
Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items:
1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy
proxy). Only pages when BOTH paths fail. Partial failure logged but not
paged (distinguishes stack problem from mesh/Caddy issue).
2. deploy.sh — pre-deploy pg_dump before each deploy saves to
/opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a
guaranteed recovery point.
3. docs/operations/rollback.md — runbook documenting automated rollback,
manual recovery, decision tree, backup schedule, and rehearsal log.
Two operational items remain (require operator on Proxmox/Gitea):
- Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs
- Archive apps/105 LXC (pct stop 105 + archive)
All active config (seeds, compose, scripts) is already clean of apps/105 refs.
Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).
Added 12-point verification table with file:line evidence for every item.
Noted two minor deviations: inline SQL (not sqlc) and synthetic Infisical
IDs (real Infisical pending consolidation plan). Precondition checks
enumerated with hard/soft classification.
Plan #4 done. Audit inventory verified against codebase:
- 9 superseded oikos/*.py files deleted (only gen-topology.py remains)
- bin/homelab deleted, bin/oikos deleted, oikos/cards/ deleted
- .hermes/plans/ already archived to archive/hermes-plans/ (all 7 files)
- TRMNL plan already in Done table
- seanime + romm documented in seeds/knowledge.yaml (DB-native, no wiki needed)
- Traefik references valid (VPS still runs traefik for public termination)
- ADR-0011 exists (client lifecycle); consolidation plan is Go rewrite record
- Prometheus plan updated: Python refs replaced with Go scheduler, check_defs,
MCP request_execution; LXC 131 identified as teddycloud
Remaining items (cutover, Infisical, watchdog, rollback, apps/105) belong to
consolidation plan (#1). 4 of 6 plans now Done.
Plan #5 done. Wiki already archived to archive/knowledge/. seeds/knowledge.yaml
has 24 docs + 6 investigations + 3 runbooks.
- MCP search_knowledge: upgraded from ILIKE to PostgreSQL ts_rank/ts_headline
- MCP get_entity_knowledge: new tool, walks relationship edges to return
all docs/investigations/runbooks linked to an entity
- HTTP endpoints (SearchKnowledge, GetEntityKnowledge) already used full FTS
- Plan index + audit cross-reference updated
Thin client model (rev 2): no git clone, no sync timer.
Changes:
- Fetches only CLIENTS.md, AGENTS.md, OIKOS.md, tools/ from raw Gitea URL
- Enrolls via POST /api/v1/clients/enroll (replaces dead Python
secrets-issuance service)
- Receives age keypair + Infisical identity from Oikos API
- Context poller replaces 5-minute git pull (launchd/systemd timer hits
GET /api/v1/clients/{slug}/context?since=)
- Removed --no-secrets, --no-mesh flags (degraded modes TBD)
- Removed dead bin/homelab symlink
- Removed Gitea credential configuration (no git clone = no git auth)
- Kept --with-mcp and --with-hermes flags for optional tooling
- auto-setup scripts run from fetched tools/ directory
Phase 1 implementation from the client-lifecycle plan.
- Migration 012: provisioning_steps table, context_version, context_files,
enrolled_at column, slug+type index for machine entities
- API endpoints (openapi.yaml + generated code):
POST /clients/enroll — age key issuance, Infisical identity, state transition
GET /clients/{slug}/context — agent file delta polling (replaces git pull)
GET /clients/{slug}/secrets — scoped secret listing
POST /entities/provision — compute entity creation with constraint validation
GET /entities/{slug}/provision/status — step-by-step provisioning progress
- Handlers in impl.go: enrollment with state validation and age key generation,
provisioning with execution tracking and relationship creation,
context endpoint with since-based delta queries
- Server struct extended with secretsBackend interface for key storage
- All tests pass, build clean
CLIENTS.md is the entry point for a machine joining the homelab.
AGENTS.md is the AI agent persona layer on top. Both fetched, but
CLIENTS.md comes first.
Problem: Repo had no developer guide, no client onboarding doc, no agent
dev instructions. Stale files (675KB SQL dump, one-off convert script,
legacy MCP builder) cluttered the tree. Client enrollment was a documented
intention with no Go implementation.
Changes:
- New docs: CONTRIBUTING.md (dev setup), CLIENTS.md (client onboarding),
.agents/dev/CONTRIBUTING.md (agent codebase map)
- New plan: plans/2026-07-07-client-lifecycle-in-go.md — full client
lifecycle (planned→provisioning→active→deprecated→destroyed) in Go,
replacing archived Python secrets-issuance, adding client API endpoints
and 6 missing MCP tools
- Cleanup: deleted archive/convert-wiki.py (one-off), archive/mcp/
build_host_files.py (legacy), backups/pre-deploy-7f7d039.sql (local)
- Fixes: plans/index.md duplicate row removed, README.md repo layout
updated for current state, AGENTS.md header points to new guides
Risk: low. Docs only + stale file deletion. No code changes. New plan is
proposal, not implementation.
Verification: git diff reviewed, all changes are prose/docs/plans.
- knowledge.go: scan tags as []string from pgx (not JSON)
- seed.go: convert tags to PG array format, fix runbook applies_to_type
- convert-wiki.py: fix entity slug prefixes to match inventory.yaml
(host: not proxmox-host:, ws: not workstation:, service:homelab-mcp with hyphen)
- convert-wiki.py: read from archive/knowledge/ since wiki was archived
- Rewrote README from 'living documentation' to 'Oikos — agentic homelab
operating system written in Go.'
- Added quick start, architecture diagram, component ports table.
- Added Phase 1-6 status table with checkmarks.
- Added API usage examples, Hermes query examples, CLI reference.
- Added repo layout table.
- Note: Gitea repo path (dtoro/Homelab-Docs → dtoro/oikos) requires
Gitea UI rename — references in oikos/cards/, bootstrap.sh, and
deploy scripts will need updating after the rename.
- .agents/OIKOS.md: rewrote entire Build Status section from Python 30-day
roadmap to Go Phases 1-6 status. Added Python-era backlog preservation.
- knowledge/wiki/containers/105-apps.md: added DEPRECATED notices for
homelab-mcp and secrets-issuance services, pointing to Go equivalents
and cutover checklist.
- knowledge/wiki/infrastructure/auto-deploy.md: marked webhook ids 10+11
as deprecated, replaced Go Docker stack.
- knowledge/wiki/infrastructure/index.md: noted topology gen as Python
with Go DB-native replacement planned.
- .agents/operations/hermes-agent.md: updated MCP references from
FastMCP SSE Python to Streamable HTTP Go SDK.
- .agents/shared/writing-style.md: updated MCP reference, topology note.
- .agents/domains/knowledge/schema.md: updated MCP server reference.
Removed:
- bin/hermes (9.3MB compiled binary accidentally committed to git)
- oikos/console/ (Flask web console — replaced by Go REST API + SSE)
- mcp/server.py (Python MCP server — replaced by internal/mcp/)
- oikos/policy.yaml, oikos/ontology.yaml (duplicates of seeds/)
Kept:
- oikos/*.py kernel files (12 files) — still imported by bin/homelab
for operational CLI commands (ssh, pct, logs, restart, status,
open, secret, client, sync, mcp). Will be removed when bin/homelab
is ported to Go.
- mcp/build_host_files.py — generates hosts/*.yaml from inventory,
still operational. Will be ported to Go.
- bin/homelab — active Python CLI, still operational.
Updated:
- .gitignore: added bin/hermes, cleaned up legacy comments
- plans/index.md: listed all 4 active plans with accurate statuses
- phase3.go: RequestExecution now calls InsertEntity before InsertExecution
(executions.entity_id references entities.id via FK constraint).
- mcp/server.go: request_execution MCP tool same fix — inserts entities row
with slug 'exec:<target>:<id8>' before executions insert.
- docker-compose.yml: fix seed OIKOS_SEEDS_DIR from /app/seeds to /seeds
(distroless image COPY destination).
- actuator/ssh.go: custom errorsAs chain broken — all SSH errors classified as SSHErrorOther.
Replaced with standard errors.As + errors.Is.
- scheduler/scheduler.go: all four check functions were stubs returning healthy.
Implemented real HTTP GET, TCP dial, unix.Statfs disk, and TLS cert expiry checks.
- learning/learning.go: uuid.NewV7() called unconditionally before ON CONFLICT upsert.
Now looks up existing pattern first, reuses entity_id.
- notifier/notifier.go: removed dead var_, fixed token regeneration every 15s.
Now skips if token_hash already set.
- phase3.go: removed dead GetPattern+dummy args call in PatchPattern.
- classify.go: removed unused var_ guard.
2026-07-07 15:27:31 +02:00
4283 changed files with 1071746 additions and 13914 deletions
| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
| Concern | Where it lives | How it gets there |
|---------|----------------|-------------------|
| Knowledge content — documents, investigations, runbooks | `knowledge_entities` table (rows linked to `entities` via `documents` / `about` edges) | Seeded from `seeds/knowledge.yaml` at deploy; mutated at runtime via the API |
| Seed manifest (bootstrap + DR) | `seeds/knowledge.yaml` | Hand-edited or regenerated; ingested idempotently (content-hashed via `seed_versions`) |
# Agent enrollment — bootstrap a client into the homelab context system
# Agent enrollment — operational notes
This walks through enrolling a new machine (workstation, LXC, or VM) so it
joins the cross-client context system: a `/opt/homelab-context/` clone of
this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
**For the actual enrollment flow, see [CLIENTS.md](../../CLIENTS.md#enrollment)
— it's the current, authoritative version.** This page used to duplicate
that flow in more detail, describing a `homelab` CLI-based two-step
ceremony (`homelab client add` reserves an inventory slot → client
bootstraps → operator finalizes the pubkey). That CLI and that flow don't
exist anymore — enrollment today is one shot: `bootstrap.sh` calls
`POST /api/v1/clients/enroll` directly and gets back an age keypair +
Infisical identity in the same response. What's left here is the handful
of things that are still true and weren't already covered elsewhere.
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
> See [hermes-agent.md](hermes-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-hermes` flag.
Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.
## Prerequisites the client must satisfy
## Prerequisites
| Requirement | Why | How to check |
| --- | --- | --- |
| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| Hostname matches an entry in `inventory.yaml` | `EnrollClient` looks up the entity by slug derived from hostname; it must exist in `planned`/`provisioning` state. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit `<bootstrap-tier1>`). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`.
**Path B — interactive OIDC (default; recommended):**
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
**Path A — setup-key (headless/scripted onboarding):**
Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.
1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`.
3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver):
```bash
sudo netbird up --setup-key <KEY> \
--management-url https://netbird.hubris.network \
--ssh-jwt-cache-ttl 86400
```
**Why we can't OIDC-login from the public internet (still open as a follow-up):**
`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124.
### DNS prerequisite
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:
- **LAN client**: set DNS to 192.168.8.180 (per-interface or
`/etc/resolv.conf`).
- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script).
Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.
revocation). Likely maps to an entity lifecycle transition
(`.agents/skills/lifecycle-deprecate-node/` or `lifecycle-destroy-node/`)
but those skills reference the same dead CLI and need their own check.
- **Granting a secret to an already-enrolled client.** The old flow
hand-edited `.sops.yaml``creation_rules` + `sops updatekeys`. Given
Infisical is now the primary secrets backend (SOPS is the DR fallback),
the current mechanism is probably Infisical-side, not a `.sops.yaml` edit
— not confirmed.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Enrollment 404s / entity not found | Hostname doesn't match a `planned`/`provisioning` inventory entry | See "Hostname mismatch" above |
| `gnutls_handshake() failed` / TLS errors reaching `*.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS instead of the LAN/mesh path | See the networking runbook (split-horizon DNS section) |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). |
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
Added a new "Post-bootstrap: SSH reachability" section covering SSH key
generation, pubkey publication, deployment to hosts, SSH config generation,
and LAN IP registration. New workstations enrolled via this doc will
automatically join the universal SSH mesh.
### 2026-07-12 — trimmed to current architecture
Removed everything describing the retired `homelab` CLI-based two-step
enrollment ceremony (now: `CLIENTS.md`'s one-shot flow), the Nous-Hermes/
Goose cross-link (that whole flow was removed the same day), and CLI-syntax
troubleshooting rows with no current equivalent. Migrated the still-true
Netbird/DNS/SSH-distribution content to a knowledge-base runbook rather
than duplicating it here. What's left is genuinely current or explicitly
flagged as unverified. Original ~365-line version is in git history
(`git log -- .agents/operations/agent-enrollment.md`) if any of the removed
detail turns out to still be needed.
### 2026-05-31 — cross-link to hermes-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
Added three rows to the troubleshooting table covering issues surfaced
during the netbird vanilla migration. (Migrated 2026-07-12 to the
networking runbook.)
### 2026-05-20 — initial page
Captures the enrollment flow validated during Phase 2 of the homelab
Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
## Proxmox CLI
@@ -8,13 +8,13 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
| `pct enter <id>` | Shell into a container |
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
| `pvesm status` | Storage pools status |
| `pvesh get /nodes --output-format json` | Node summary as JSON |
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog`Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog`Nomos cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
@@ -22,22 +22,22 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
## Reverse proxy
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
- See [DNS](../../knowledge/wiki/infrastructure/dns.md).
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
## Web access
@@ -49,43 +49,45 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
-`ras-mc-ctl --errors` — full event log
-`cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
-`cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
-`ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md))
-`ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/2026-04-21-hubris-crash-loop.md))
## Fleet apt operations
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
| Command | What it does |
| --- | --- |
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
**No current CLI equivalent.**`homelab apt-audit`/`apt-upgrade` (dpkg-state
audit, fanned-out apt upgrade with pre-upgrade snapshots) were part of the
retired Python `homelab` CLI and don't have a ported replacement — apt
patching today is ad hoc `run` MCP tool calls per host, without the
audit/snapshot/status wrapping this used to provide. If that wrapping is
still wanted, it needs to be rebuilt (e.g. as a runbook driving `run`, or a
new MCP tool) — see
[runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) for
the dpkg-interrupted recovery procedure specifically.
## Oikos (agent OS layer)
See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
See [OIKOS.md](../OIKOS.md) for the operating model. The `homelab` CLI this
section used to document is retired; the actual current interface is the
MCP tool catalog in [AGENTS.md §3](../../AGENTS.md#3-the-mcp-server) plus
the REST API. Closest current equivalents for what used to live here:
| Command | What it does |
| Old `homelab` command | Current equivalent |
| --- | --- |
| `homelab service <name> explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history |
| `homelab node <name> relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius |
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
| `homelab signal list\|ack\|resolve\|mute` | MCP `get_signal_history`, or REST `POST /api/v1/signals/{id}/ack\|resolve\|mute` (the control-room UI's Signals page wraps these) |
| `homelab approval request\|list\|reply\|check` | REST `GET/POST /api/v1/approvals*` (Matrix-delivered via the notifier, or the control-room UI's Operations page) |
| `homelab restart <service> --approval-id <id>` | MCP `run` (policy-gated — auto-executes if read-only/reversible_low, otherwise queues for the same Matrix/UI approval) |
| `homelab decide <action> <entity>` | No direct equivalent — classification now happens inline inside `run`, not as a separate dry-run call |
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md).
There is no separately-deployed "Oikos Console" anymore — the control-room
SPA (`web/`) is the operator dashboard, served standalone (see
| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py` — `mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
## Cross-references
- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the
Hermes flow assumes is done.
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
**Content / narrative pages:** lowercase-with-dashes, date-prefixed as needed
- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from `inventory.yaml`.
- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from the entity's attributes in the DB (seeded via `seeds/inventory.yaml`).
- **Infrastructure / cross-cutting pages:** `<topic>.md` (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node.
@@ -36,7 +36,7 @@ Every doc-level page follows the same shape so a reader scans it in one pass.
1.**One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`.
2.**Opening definition.** First paragraph, 1–3 sentences, says what the thing is. No motivation, no marketing, no setup.
3.**Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md).
4.**`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (`get_changelog` in `mcp/server.py`); keep the `### YYYY-MM-DD — title` shape.
4.**`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is stored as a structured field on the `document` entity in the DB; keep the `### YYYY-MM-DD — title` shape so it parses cleanly.
5.**Related links** only at the bottom, only when a reference cannot be woven inline.
## Section indexes (folder READMEs)
@@ -53,12 +53,12 @@ duplicated prose, no narrative between the intro and the table.
- Prefer **tables** for enumerable items with internal structure (service/port maps, field lists, status grids). Reserve bullets for short non-structured lists.
- Use the **bold-leading-phrase pattern** for structured points: `**Read-only by construction.** The MCP server never mutates state.` — a bold noun phrase, a period, then the explanation.
- When enumerating across services or nodes, give each its own `###` sub-section or a table row, not one run-on paragraph.
- Use backticks for code, paths, hostnames, and file names (`inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology.
- Use backticks for code, paths, hostnames, and file names (`seeds/inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology.
- Use `>` blockquotes for caveats and gaps that interrupt the main flow: `> **Outstanding gap.** DNS-vs-inventory drift check not yet wired.` One thought per blockquote.
## Diagrams
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it.
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` in the archived wiki was generated by the retired `oikos/gen-topology.py`; the DB-native equivalent is a future task — do not hand-edit the archived file expecting it to regenerate.
- ASCII box diagrams are fine for small shape diagrams; keep them to one screen.
description: "Examine a Nomos chat session, compare the user's objective with the actual outcome, identify causes of failure (missing tools, excessive tool calls, blocked actions, model behavior), and propose concrete fixes."
risk_class: reversible_low
inputs: [session_id]
---
# Session review
Analyze Nomos chat sessions from the live database, diff objectives
4. Token is persisted to the macOS keychain — subsequent launches skip setup
The app stores credentials via `github.com/zalando/go-keyring` (service: `com.hubris.oikos-desktop`).
### Desktop app auto-update
- Checks Gitea releases every 6 hours
- System tray → **Check for Updates** triggers an immediate check
- Download, extract, replace the app in `/Applications`, and relaunch
- Versions are compared against the `version` var in `main.go`, injected from the repo `VERSION` file at link time (`make desktop` passes `-ldflags "-X main.version=$(cat VERSION)"`)
## Project structure
```
cmd/desktop/ Wails v3 desktop app (macOS + Linux)
main.go Thin shell: webview, system tray, notifications, auto-update
@command -v golangci-lint >/dev/null 2>&1&& golangci-lint run ||echo"golangci-lint not installed, skipping"
golangci:
@command -v golangci-lint >/dev/null 2>&1&& golangci-lint run --config .golangci.yml ||echo"golangci-lint not installed — see https://golangci-lint.run/usage/install/"
govulncheck:
@command -v govulncheck >/dev/null 2>&1&& govulncheck ./... ||echo"govulncheck not installed — run: go install golang.org/x/vuln/cmd/govulncheck@latest"
.PHONY:lintvetgolangcigovulncheck
generate:
$(GO) run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.4.1 \
@@ -42,8 +55,45 @@ export:
dev:
docker compose --profile dev up -d
# Local sanity-check build of the SPA. Not embedded in the oikos binary
# (plans/2026-07-12-wails-desktop-app.md 0.1) — deploys as its own
# container (compose/web/Dockerfile) via `docker compose --profile full
# up -d web`, same push-to-main pipeline as everything else.
ui:
cd web && npm run build
desktop:ui## Build the Wails desktop app for the current platform
rm -rf cmd/desktop/frontend/dist
mkdir -p cmd/desktop/frontend/dist
cp -r web/dist/* cmd/desktop/frontend/dist/
cd cmd/desktop &&CGO_ENABLED=1 go build -ldflags "-X main.version=$$(cat ../VERSION)" -o build/bin/Oikos .
desktop-package:desktop## Build + package the desktop app (zip on macOS, tar.gz on Linux)
Living documentation for the **hubris** Proxmox homelab + Oikos operating system.
Agentic homelab operating system written in Go. Single binary (`cmd/oikos`),
Docker-deployed on mac-mini, with a standalone Nomos MCP agent gateway
(`cmd/nomos`). Manages the **hubris** Proxmox homelab autonomously — observes
state, classifies actions against policy, executes approved procedures over SSH,
learns from outcomes, and escalates when uncertain.
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md), then [OIKOS.md](.agents/OIKOS.md).
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md).
**For client machines:** see [CLIENTS.md](CLIENTS.md).
**For developers:** see [CONTRIBUTING.md](CONTRIBUTING.md).
---
## Quick start
## For Agents — Navigation & Entry Points
```bash
# Dev stack (postgres + api + scheduler + notifier). The api/nomos
# services need a shared token — every route requires a real bearer
# credential, there's no dev-open bypass.
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile dev up -d
### You are running on a client enrolled in the hubris homelab
# Full stack (adds Nomos agent gateway)
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile full up -d
1.**First:** Read [AGENTS.md](AGENTS.md) once. It explains who you are, the topology, available tools, conventions, and how to act.
2.**Before any mutation:** Read [OIKOS.md](.agents/OIKOS.md). It defines the operating model, risk classes, approval flow, and the ontology you'll consult.
3.**For specific workflows:** Load the matching skill from `.agents/skills/<name>/SKILL.md` (e.g., [service-health-check](.agents/skills/service-health-check/SKILL.md)).
4.**When in doubt:** Use MCP tools (`search_docs`, `get_page`, `explain`, `get_changelog`) — they're cheaper and more reliable than grepping.
# Build standalone binary
go build -o bin/oikos -tags timetzdata ./cmd/oikos
1.**Decide:** Use `homelab decide <action> <entity>` to classify (risk class × blast radius × confidence).
2.**Escalate if needed:**`homelab approval request` (Matrix-delivered to operator; see [operations/commands.md](.agents/operations/commands.md)).
3.**Execute:** Use `homelab` CLI (not ad-hoc SSH) — it enforces policy, logs mutations, and verifies outcomes.
4.**Document:** Update wiki in the same session (per [AGENTS.md §5](AGENTS.md#5-acting-on-the-homelab) and the [same-session rule](.agents/shared/page-templates.md#same-session-update-rule)).
Full plan: [plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md).
Everything that can break, be changed, or hold data has an entity in `inventory.yaml` + `oikos/ontology.yaml`. Blast-radius questions ("what breaks if strong goes down?") are graph walks via `homelab node <name> relations`, not doc archaeology.
| **Sources** | Raw evidence: incidents, external refs, live state | `knowledge/sources/investigations/` | Yes | Read to understand root causes; do not rewrite |
| **Wiki** | Synthesized current-state: one page per node & per system | `knowledge/wiki/{containers,hosts,vms,infrastructure}/` | No | This is the reference layer — if wiki disagrees with live state, update it *in the same session* |
| **Index** | Pure listings — every page in scope with one-line summary | `index.md` / folder `README.md` | No | Navigation aid; keep it current when wiki restructures |
| **Log** | Append-only doc-maintenance record (restructures, ingests, lints) | `knowledge/log.md` | Yes (append-only) | Read to understand past doc changes; never edit directly |
# Natural language
curl -X POST localhost:8092/query -H "Content-Type: application/json"\
-d '{"query":"what depends on authentik?"}'
```
**Changelog ≠ Log:** Each wiki page ends with a `## Changelog` (infrastructure changes to that node, machine-parsed). That's not the Log; the Log records *doc operations* only.
- **File naming.** Foundational docs (entry-points, agent instruction, references) are ALL-CAPS (`AGENTS.md`, `OIKOS.md`, `GLOSSARY.md`); containers use `<id>-<name>.md`; infrastructure pages use lowercase-with-dashes; plans and incidents use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`. See [page-templates.md](.agents/shared/page-templates.md#file-naming) for the full rules.
- **Voice & vocabulary.** Concise, technical, sysadmin-to-sysadmin. No marketing prose, no puffers (seamless, robust, leverage, etc.). Full rules in [writing-style.md](.agents/shared/writing-style.md).
- **Cross-linking is mandatory.** If a page references a node or system, link to it. Treat orphans as a bug.
- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running state, fix the wiki *and* add a changelog entry *in the same session*.
- **Tracked configs.** Pages for configs living in git repos (Caddy, Gitea, Artifacto, mule-image) must note the repo. Edits go through commit+push, never local changes. See [auto-deploy](knowledge/wiki/infrastructure/auto-deploy.md).
- **No secrets.** This is a private repo, but still: reference secret *paths*, never secret *values*.
**For agents:** Read [caveman.md](.agents/shared/caveman.md) (terse communication standard). Use templates at [page-templates.md](.agents/shared/page-templates.md) when creating pages.
---
## Updating the Wiki
### When You Change Infrastructure
1. Update the relevant page (config snapshot, ports, mounts, IP address).
2. Add a `### YYYY-MM-DD — title` entry to the page's `## Changelog` section (reverse chronological order).
3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link from the changelog.
4. If it's an incident, add a record to [`knowledge/sources/investigations/`](knowledge/sources/investigations/index.md).
### When You Restructure the Wiki
1. Update the relevant `index.md` / `README.md` in that section.
2. Add a single-line entry to [`knowledge/log.md`](knowledge/log.md): `## [YYYY-MM-DD] <operation> | <summary>` (e.g., `## [2026-07-06] restructure | split infrastructure/dns into dns.md + dns-advanced.md`).
### The Same-Session Update Rule
**Any meaningful state change made in this session requires a wiki update before the session closes.** A change that touches a container page must also update:
- The `containers/index.md` table (IPs, host, mounts, status)
- The root `README.md` table (if affected)
- The Caddy page site list (if affects `*.hubris.network` routing)
- The DNS / ingress infrastructure pages (if affects routing)
- The `hosts/hubris.md` or `hosts/strong.md` page (if container count changes)
- The `inventory.yaml` host entry (source of truth for `hosts/*.yaml` generation)
- The `knowledge/wiki/infrastructure/topology.md` (regenerate if needed)
Not updating all linked places is a bug. See [page-templates.md — same-session update rule](.agents/shared/page-templates.md#same-session-update-rule).
- **Machine-readable substrate** → `inventory.yaml`, `oikos/policy.yaml`, `oikos/ontology.yaml` (not part of the wiki; see [llm-wiki.md](.agents/shared/llm-wiki.md#rules))
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
