The api retries Infisical at startup (4x with backoff) before binding :8090.
When Infisical is unreachable that adds ~60s, and the old start_period (10s)
+ 10 retries (~60s grace) ran out just before the bind — marking the api
unhealthy and failing the deploy (nomos depends_on api-healthy). 90s covers
the slow-startup window; the api genuinely serves /healthz once bound.
The procedural-glyph-engine dep pointed at a non-portable file:/private/tmp/orby-pkg
path, breaking npm ci in Docker and every main deploy since v0.20.0 (the build
cache masked it until it busted ~Aug 5). Vendor Orby v5.0.0 into web/vendor/,
switch the dep to file:../vendor, and use npm install in the web Dockerfile
(file: deps need install, not ci). Cherry-picked from 3cd4cf9.
- Wire secretsManager in NewHandler() — instantiate InfisicalBackend
when OIKOS_INFISICAL_SITE_URL is set (previously always nil)
- Add get_secret, list_secrets, set_secret MCP tools with nil-backend
graceful degradation
- Add oikos secret get|set|list CLI subcommands for Infisical
- Fix Set() bug: create-before-update so new keys are created;
add Type: "shared" to Update so it finds the right secret;
disable SDK cache so Get returns fresh data after Set
- Clean enrollment response: remove fake infisical_client_id/
infisical_client_secret stubs, store age key in Infisical for real
- classifyAndGate: escalate read-only commands on lxc: targets that
touch /opt/, /etc/, /var/lib/ to config_mutation. The classifier
scores command text only, not the SSH transport layer — SSH-ing into
a container to read config is riskier than pct exec from the host.
- ontology: standalone-server monitoring override from inherited
[ping, resource, updates] to [http]. VPS-like machines may not be
SSH/ICMP-reachable from the scheduler; HTTP is the LCD liveness
signal. Entities with full SSH can override per-entity.
- get_health_summary: filter out state=destroyed entities (was noise
from 20+ destroyed test LXCs, deprecated services, etc.)
- create_entity: document the monitoring footgun in the tool description
(creating a type=check entity does NOT wire a check_def; the correct
path is update_entity_attributes with monitoring + url attributes)
Adds a new 'dns' semantic monitoring kind that probes whether a DNS name
resolves. Uses net.LookupNS (NS records) with fallback to net.LookupHost
(A/AAAA). Supports an explicit server config for split-horizon resolution.
Changes:
- seeds/ontology.yaml: dns-zone monitoring: none → [dns] (was deferred
since 2026-06 with a comment 'no dns checker exists yet')
- seeds/inventory.yaml: host:netbird-vps monitoring: [http] (was none;
VPS was invisible for 7 days during the 2026-07-29 outage)
- internal/checkdefaults/defaults.go: add KindDNS, buildKind case for 'dns'
that creates a check_def at 5-minute intervals
- internal/scheduler/scheduler.go: add checkDNS probe + wire in executeCheck
The DNS checker catches stale/unreachable zones (e.g. matrix.hubris.network
pointing to a dead VPS IP). The VPS HTTP check probes the public endpoint
every 60s, closing the 7-day monitoring gap.
I — run pre-flights QEMU guest agent before queueing VM execution
classifyAndGate now checks vm: targets for qemu_guest_agent attribute.
If not_running/missing, returns immediate error instead of queuing forever.
II — policy.yaml: documented host-mutation classifier rule
Added comment clarifying that host-level package/kernel mutations
(apt-get install, dpkg, systemctl enable) always classify as
config_mutation and thus need operator approval.
III — health attribute read-only in update_entity_attributes
Strips scheduler-owned keys (health, last_check_at, last_check) from
attribute updates with a clear message directing agents to
get_health_summary / list_checks instead.
IV — Recorded discovered dependency edges
vm:zimaos → depends-on → lxc:nfs-export (NFS /media/library mount)
vm:zimaos → depends-on → host:strong (NFS /media/ludo-library mount)
Also updated the run tool description to mention both guardrails.
get_relations now accepts an optional 'types' (comma-separated) parameter
to filter relationship types — filters out the noisy exec/targets edges
that previously drowned useful host/provides edges.
get_health_summary now accepts an optional 'health' (comma-separated)
parameter to return only entities in specific health states (e.g.
'health=down,stale') instead of the full 100+ entity list.
ping_service now:
- Falls back to e.attributes->>'public_host' when 'url' is not set
(covers LXCs that only have public_host in the graph)
- Performs a live HTTP HEAD probe against the resolved URL, returning
the actual status code instead of just the scheduler's stale health
state
Also: fixed matrix.hubris.network DNS record (was pointing to dead VPS),
pruned 6 dead graph edges, wired url attributes on 7 LXCs, added VPS
HTTP monitoring check, and resolved the 18k-occurrence unmonitored signal.
This session's audit is documented as
document:nomos/2026-08-05-dns-monitoring-improvements-for-strong-hosted-services.
- Add MCP tool — lightweight connectivity check returning server
identity, no DB hit (resolves agent connection-test friction)
- Tighten 6 tool descriptions (get_relations, get_health_summary,
query_metrics, get_trend, get_event_timeline, ping) to be searchable
in the first 8-12 words
- Document Hermes MCP client setup in ADR-0012 with token security caveat
- Move completed plan to plans/done/
Backend:
- Add isThinking flag to agentEvent for text before tool calls
- Separate thinking from response text in runChatTurn and continue.go
- Persist thinking in a dedicated field in message content
Frontend:
- Add thinking field to MessageContent, ChatMessage, ChatTextEvent types
- Create ThinkingBlock.svelte — collapsible block with brain icon
- SSE handler moves text_delta content to thinking on isThinking flag
- Render thinking block between tools and response in ChatThread
- Fix chat window scroll reset on focus change (stable windowKeys order)
- Remove redundant #key id wrapper in WindowLayer
- Enlarge sidebar rail (24→32 default, 40→60 max)
- Remove glyph from sidebar, square graph at top
- Replace AgentTrace/ToolCallCard/UnifiedTimeline with TurnTrace/ToolLine
The "Working — message will queue" strip lived inside the sized input Pane, so
appearing/disappearing ate the textarea's fixed height and clipped it, forcing
a resize. Move it into the message Pane alongside the connection/error banners
— those correctly consume transcript space (flex-1) rather than the input's
fixed height. The input Pane is now stable whether or not a background turn is
running. Styling/idiom unchanged.
VERSION: 0.17.2 -> 0.17.3
The background-working hint was a plain muted text line bolted above the
textarea — misaligned with the input column and off-idiom. Restyle it as a
terminal status strip: spinner + uppercase fg "Working" label + muted detail,
hairline primary-tinted border (matching .trace.running), square, aligned to
the textarea's max-w-3xl column. Reads as part of the working state now.
VERSION: 0.17.1 -> 0.17.2
git mv staged the pre-edit index content; the status edits to these two
files landed in the working tree but not the archive commit. Amending the
status now so the archived copies reflect Implemented.
Move ten completed plans from plans/ to plans/done/ and update the index:
- 2026-07-18 session-review-three-sessions, 2026-07-20 desktop-mascot,
2026-07-20 session-review-ten-sessions, 2026-07-21 chat-full-polish,
2026-07-29 health-check-reality-and-knowledge-graph,
2026-07-30 session-review-plan-drift, and the four 2026-08-03 chat plans
(changes-review, reliability-and-ux-audit, cyberspace-style-adoption,
working-visibility).
- Refresh two stale statuses: cyberspace-style-adoption ("Draft" -> shipped as
full replacement in v0.16.0/757ef2f) and health-check-reality ("ready for
implementation" -> shipped across the v0.14.x-0.16.x check commits).
- .gitignore: ignore local tooling artifacts (.playwright-mcp/, config-screen.png).
No code change. index.md Active/Done tables now match the filesystem (no orphans).
VERSION: 0.17.0 -> 0.17.1
Make background/long/desynced turns visible and queueable, fixing the four
symptoms that survived the v0.15.0 chat reliability pass.
F1 - status-driven working signal (workspace.ts taskWorking/currentWorking =
streaming OR status in {planning,executing}). Drives the chat trace, indicator,
and activity spinner so a turn with no live stream (background resume, a dropped
SSE, an idle-close mid long turn) still looks alive.
F2 - operator messages sent during an in-flight turn are now QUEUED and
auto-run when the gate frees, replacing the "still finishing a previous step...
send it again" rejection. Per-session in-memory FIFO (messagequeue.go, capped at
20) drained one-at-a-time under the turn gate; a `queued` SSE event drives a
"Queued" hint. drainQueued releases via a per-iteration deferred closure so a
runChatTurn panic can't deadlock the session's gate.
F3 - SSE keepalive (12s `:keepalive` comment) in handleChat so 20-40s
inter-iteration gaps no longer trip a proxy/browser idle close (the desync root
cause). All SSE writes serialized through one mutex.
F4 - generation-aware activity timeline (only the last propose_plan renders;
superseded ones collapse to one "Earlier plan revised" marker; step-attribution
follows only the current generation) + debounced plan refetch on lifecycle
events so a missed plan.proposed self-heals.
Verified against the last session (23da10db: 6m33s turn, operator "status"
deferred at 19:48:05). go test ./cmd/nomos/ green (new messagequeue tests);
web vitest 72/72 (new F4 generation tests); vite build clean.
VERSION: 0.16.0 -> 0.17.0
Rebrand light/dark themes to the cyberspace.online look: warm cream-on-black
palette (light/dark are exact inverses), self-hosted JetBrains Mono + VT323,
square corners, border-driven surfaces with no soft shadows. Adds a terminal
design-system CSS layer (DOS double-border modals with hatched corner, fg focus,
inversion-on-hover), a theme-aware <RasterImage> (Atkinson-dithered canvas with
img fallback), and unifies desktop icons, taskbar, window controls, pills and
links under one idiom. Pins window titlebars to a fixed height and switches chat
auto-scroll off scrollIntoView to avoid titlebar reflow.
VERSION 0.15.1 -> 0.16.0
The Send button's disabled={!input.trim()} threw "trim is not a function" when
input was initialized from a non-string initialDraft (a Svelte 5 prop-init edge
where a null/undefined draft reached $state). Coerce at init so the composer
state is always a string.
VERSION: 0.15.0 -> 0.15.1
The agent could run two turns for one session at once (a reconnect resumed
while the live turn was still going), and their interleaved tool calls
corrupted the activity panel, fabricated a confusing "parallel/nested"
sequence, and made tasks feel stuck/never-ending. Several UX gaps compounded it.
Turn serialization (F1):
- turnGate: at most one in-flight turn per session. Background resume paths
(continuation worker, idle sweep, answer-question, /resume, reconnect)
skip non-blocking when busy; the live chat path waits briefly then bails
cleanly instead of stacking a second turn.
- resumeSession returns whether it ran; continueSession marks an execution
"continued" only after a real run (review P0) so a busy-skip can't lose a
finished-execution result. Idle nudge bumps only after delivery (P1).
Connection state (F2/F3, web):
- humanize/bucket raw errors ("model connection dropped..."); one surface
per drop; a terminal task.status event clears stuck streaming/disconnected
state and dismisses the connection toast. Reconnect no longer spawns turns.
Streaming where you look (F4, web):
- live command output in the global activity timeline and in the inline
tool card (auto-opened, tail-pinned) -- not just the per-window rail.
Other (web): artifact/knowledge deep links (F5); step-first stable
"thinking" headline (F6); stable chat layout, no empty->content reflow (F7);
lazy event sync (P2.2); reconnect skips a terminal session (P2.1).
VERSION: 0.14.2 -> 0.15.0
The plan recorded false history after a re-plan and the activity panel
showed fabricated, churning timestamps. Two bugs compounding on one event
stream.
Plan drift (P0.1):
- proposePlan seq is now 1..N per generation; (session,generation,seq) is
the addressing key. The model's 1-based update_plan_step calls always map
to the CURRENT plan after a re-plan, instead of resurrecting a superseded
`replaced` row as done while the live work went unrecorded.
- updatePlanStep resolves against MAX(generation); a stale/out-of-range seq
returns errPlanStepNotFound (never touches a superseded generation).
- getPlanSteps returns only the current generation by default; ?all=true
keeps the audit/eval view (plan_generations assertion).
- completeTask auto-close scopes to the current gen, stamps started_at, and
emits one plan.step.finished per closed step so the panel converges
instead of freezing on "running" after completion (P1.1).
- propose_plan result enumerates step seqs; writeback detector matches
"write back"/"writeback"/"upsert_knowledge" so a natural-language final
step isn't doubled (P1.2).
- migration 029 renumbers existing seq per generation + unique index.
Activity panel (P0.2 / P1.1, web):
- computeActivityLog uses the real message created_at for tool calls; live
entries fall back to wall-clock frozen on first sight, killing the 3s
poll churn. Steps use real started_at.
- dropped plan-step events warn + count instead of a silent no-op.
Tests: TestProposePlan updated; + generation-relative-seq and auto-close
event-emission regression tests; + web activity purity/timestamp tests.
VERSION: 0.14.0 -> 0.14.1
Every docker build sent the whole repo root as context, including every git
worktree under .claude/worktrees/ (200-300MB each) — that crossed 390MB of
cruft and starved mac-mini's disk mid-build on 2026-07-27 (873b00a). None of
it belongs in an image.
A `monitoring` attribute on an entity now overrides its type's declaration:
"none" opts out, a list overrides the kinds. service:haos uses it to opt out —
haos blocks SSH (no process probe can reach it) and the VM is already covered
by vm:haos's vm-status check, so the redundant process check only ever reported
false-down. vm:haos -> service:haos via provides confirms the coverage.
VMs declared monitoring [ping], but many block ICMP and lack a guest agent
(haos), so ping was the wrong probe — a powered-on VM reported "down". Add a
vm-status check: `qm status <pve_id>` on the VM's Proxmox host, which tests
"powered on" without needing the VM's network at all. vm type monitoring is
now [vm-status].
matrix.hubris.network is a public hostname (federation) resolving to
netbird-vps, not served by the lab Caddy — so its cert-expiry check's
dial=caddy IP failed. Drop the dial for matrix; it dials by name (DNS ->
public) like wget already proved works.
The ontology's stated intent was "http when it has a url, else a process
check", but the implementation emitted BOTH for every url-service — so ~17
fronted services carried a redundant process check that, under worst-of
aggregation, let a fragile supplementary probe (wrong unit name, unreachable
host, no guest agent) veto two healthy http checks and report the service
"down" while it was up (authentik, zimaos, house, matrix, ...).
buildKind now emits a process check only for services WITHOUT a url, or when
an explicit probe_unit opts into binary-level depth. http is the canonical
service-liveness probe (tests the real endpoint through the TLS terminator);
the redundant process checks were removed.
process_check.sh ran `systemctl is-active <entity-name>`, but a service's name
is a logical label, not its unit/container name — matrix is matrix-synapse.service
+ element-web/mautrix-* containers, authentik is authentik-server/-worker
containers. So every multi-component or docker service reported "inactive"
while up (authentik, matrix, photos, house, arr-stack, …).
Resolve in order: exact systemd unit, a unit with the name as prefix
(matrix -> matrix-synapse.service), or a running docker container whose name
contains it. checkdefaults passes a declared probe_unit/systemd_unit/container
attribute when set, for precision.
The DB-only audit_knowledge_graph can't see guests running in Proxmox that
have no entity, or entities whose pve_id is no longer live — the drift that
the stray test LXCs were a symptom of. discover_infra_drift enumerates running
guests via pct/qm list on every proxmox host (over the same SSH/pct path the
checks use) and diffs against the DB: returns missing (live, no entity) and
ghost (DB, not live). Read-only.
Companion to audit_knowledge_graph; the skill now runs both and treats the
remaining checks (misplaced parent, undeployed scripts, seed drift) as manual.
resolveProxmoxHostSlug trusted attributes.host verbatim, so a value polluted
with prose — lxc:teddycloud carried host="hubris (confirmed via pct config…)" —
became a slug that never resolved, leaving its checks 'down' despite a correct
`hosts` edge. Treat an attribute containing whitespace/parens as invalid and
fall back to the canonical hosts edge.
The audit now reports `polluted_attrs` — entities whose routing-critical
attributes carry prose — so this class is visible instead of a silent
resolution failure.
The B1 target-state filter `tgt.state NOT IN ('deprecated','destroyed')`
evaluates to NULL (unknown) when a target's state is NULL, which the WHERE
clause treats as false — so freshly-seeded entities without an explicit state
(the 20 TLS certificates) were silently dropped from ListEnabledCheckDefs and
never monitored. Treat NULL state as active (only explicit deprecated/
destroyed is excluded): `tgt.state IS NULL OR tgt.state NOT IN (...)`.
checkCertExpiry now accepts a `dial` address and sets ServerName to the
hostname — it connects to the terminator's IP while SNI/cert-read use the
hostname. The scheduler container has no mesh interface and the host resolver
doesn't know the split-horizon zone, so *.hubris.network can't be dialed by
name from there; dialing Caddy's lab IP (reachable on the LAN) makes the probe
work. The builder passes through a cert entity's `dial` attribute.
Re-seed the 20 *.hubris.network certificate entities with dial=192.168.8.175
(Caddy) and uses-certificate edges; cert-expiry monitoring now has real data.
disk_usage_check.sh built its mount list with `df`, which blocks on a wedged
filesystem (stale NFS export, a stuck ZFS pool) — and that stalled the whole
check past the scheduler's 30s budget, leaving host:hubris:4 perpetually down.
Build the mount list from /proc/mounts (a read that never stats anything), and
bound every per-mount `df` with `timeout 8` so a single stuck mount is skipped
instead of hanging the probe. Degrades to plain `df` on hosts without
`timeout`//proc/mounts (macOS), whose local mounts don't hang.
The cert-expiry builder and checkCertExpiry probe are correct, but the
scheduler container can't reach *.hubris.network:443 — its DNS forwards to the
host resolver, which doesn't know the split-horizon zone, and overriding the
container DNS would break docker service-name resolution. Seeding the 20 cert
entities now produced 20 false-down certificates.
Keep the builder (committed), drop the entities + edges until the scheduler can
reach Caddy (extra_hosts mapping, or a SNI-dial enhancement) — then re-add them.
