Adds a new 'dns' semantic monitoring kind that probes whether a DNS name
resolves. Uses net.LookupNS (NS records) with fallback to net.LookupHost
(A/AAAA). Supports an explicit server config for split-horizon resolution.
Changes:
- seeds/ontology.yaml: dns-zone monitoring: none → [dns] (was deferred
since 2026-06 with a comment 'no dns checker exists yet')
- seeds/inventory.yaml: host:netbird-vps monitoring: [http] (was none;
VPS was invisible for 7 days during the 2026-07-29 outage)
- internal/checkdefaults/defaults.go: add KindDNS, buildKind case for 'dns'
that creates a check_def at 5-minute intervals
- internal/scheduler/scheduler.go: add checkDNS probe + wire in executeCheck
The DNS checker catches stale/unreachable zones (e.g. matrix.hubris.network
pointing to a dead VPS IP). The VPS HTTP check probes the public endpoint
every 60s, closing the 7-day monitoring gap.
VMs declared monitoring [ping], but many block ICMP and lack a guest agent
(haos), so ping was the wrong probe — a powered-on VM reported "down". Add a
vm-status check: `qm status <pve_id>` on the VM's Proxmox host, which tests
"powered on" without needing the VM's network at all. vm type monitoring is
now [vm-status].
matrix.hubris.network is a public hostname (federation) resolving to
netbird-vps, not served by the lab Caddy — so its cert-expiry check's
dial=caddy IP failed. Drop the dial for matrix; it dials by name (DNS ->
public) like wget already proved works.
checkCertExpiry now accepts a `dial` address and sets ServerName to the
hostname — it connects to the terminator's IP while SNI/cert-read use the
hostname. The scheduler container has no mesh interface and the host resolver
doesn't know the split-horizon zone, so *.hubris.network can't be dialed by
name from there; dialing Caddy's lab IP (reachable on the LAN) makes the probe
work. The builder passes through a cert entity's `dial` attribute.
Re-seed the 20 *.hubris.network certificate entities with dial=192.168.8.175
(Caddy) and uses-certificate edges; cert-expiry monitoring now has real data.
A service check used to bake its hosting LXC's lan_ip and SSH it directly as
root, which failed because the scheduler key is authorized on the Proxmox hosts
but not inside every guest — leaving all 8 service process checks 'down' even
after the guest routing and scripts were fixed.
ResolveExecTargetForCheck now, for a non-guest target, walks the
provides/runs-on/hosts edges to the compute entity that runs it and routes
through that: pct/qm exec if the host is a guest, direct SSH with the host's
correct user (workstation `user` attr) if it's a machine. The guest-resolution
path is shared via resolveGuest, and the scheduler no longer needs an
isMachine special case — one resolver handles guest, machine, and service.
Older writeCheck inserts omitted target_type, so every seed-created check_def
had a NULL/empty target_type. checkSSHScript's IsGuest check then never matched,
and guest checks silently fell back to their baked (often mesh-only) address —
keeping them 'down' even after the pct-exec routing and deployed scripts were
in place. rclone stayed down for exactly this reason after the host-hop fix.
writeCheck now writes target_type, and checkSSHScript resolves the type from
the target_id when the column is blank (a runtime safety net for existing rows;
the seed rows were also backfilled in the live DB).
ListEnabledCheckDefs now LEFT JOINs the target entity and excludes rows whose
target is deprecated or destroyed, so retired things (secrets-issuance,
homelab-mcp, the dead secrets ingress route) stop generating permanent false
alarms instead of waiting for an operator to disable the check_def by hand.
coverageSweep's None() branch previously did nothing, so a type changed from
declared monitoring to `monitoring: none` (dns-zone) left its open
`unmonitored` signals lingering forever — a None() entity never gains a check,
so the hasCheck resolution path never fired. It now resolves those signals.
The scheduler SSHed each guest directly and assumed a deployed probe script
plus working root SSH at the guest's address — false for headless (nfs-export),
keyless (teddycloud), mesh-only (rclone), and macOS (mac-mini) targets, which
left 49 enabled checks stuck "down" on a healthy fleet.
Extract the MCP run tool's resolveExecTarget into a shared internal/remote
package and make it the single execution path for both the scheduler and MCP.
LXC/VM checks now host-hop via pct exec / qm guest exec through the owning
Proxmox host (no per-guest lan_ip, sshd, or authorized key needed); hosts and
workstations resolve their address and user live, so mac-mini's `user: dtoro`
is honored without a re-seed. Address preference now prefers public_ipv4 over
mesh, so netbird-vps is probeable from the scheduler container.
cpu_check.sh gains a real Darwin branch (it reported cpu_pct 0 before).
checkdefaults.resolveSSHUser reads the top-level `user` attribute too.
A machine-target resolution failure is now logged before falling back to baked
config, so a broken probe-config is distinguishable from a real outage.
Two things the entity window redesign surfaced but deliberately left alone.
**blast_radius answered the wrong question.** It walked source→target for every
relationship type, but which end of an edge is the dependent differs per type:
"machine hosts container" means the target breaks, while "service depends-on
service" and "ingress routes-to service" mean the SOURCE breaks. Walking
everything forwards was right for hosts/provides and backwards for everything
else — and swept in 2,800+ documents/involves/targets edges of pure bookkeeping,
so the result contained tasks and executions that cannot break.
Direction is now declared per relationship type in seeds/ontology.yaml
(blast_direction: forward | backward | none), the same shape as the entity
types' monitoring: declaration, and defaults to none so an undeclared edge
contributes nothing rather than a confidently wrong answer.
It also needed a modelling fix: `routes-to` names an ingress's BACKEND, so
nothing recorded that all 21 public hostnames are terminated by caddy. A
`served-by` edge type now says so.
pool:ludo-lvm 2 -> 23 (every container storing on it, then their services)
lxc:caddy 4 -> 22 (service:caddy, then all 21 ingress routes)
service:authentik 7 (what authenticates via it)
**Every ping check was reporting down.** Not a host:strong false positive: all
seven, including ws:mac-mini — the Docker host itself. The scheduler runs in
Docker on macOS, whose VM does not route ICMP to the LAN; loopback pings succeed
and every LAN ping fails. Under health aggregation each broken probe dragged its
entity to down.
The question the check exists to answer is "is it reachable", and ICMP is only
one way to ask it. checkPing now falls back to a TCP connect before concluding
anything, which restores an honest verdict for the four hosts that are genuinely
up while leaving the genuinely unreachable ones down.
TestBlastRadiusTerminatesOnCycles asserted the old direction (caddy=1,
authentik=2 — the cycle walked the wrong way); it now asserts the corrected
depths, and its exact-node-count check is relaxed because walking the right way
also surfaces the seed's own real dependents, which are correct answers.
Co-Authored-By: Claude <noreply@anthropic.com>
host:strong logged 226 health.changed events in one hour, oscillating
down/healthy while the host was fine throughout. host:hubris did it 126 times.
runCheck wrote entity_status.health on every check completion, so an entity's
health was simply whichever of its checks finished most recently. A host with
six checks reported whichever facet happened to be sampled last, and one
failing probe alternating with five passing ones flapped forever. resolveSignal
forced "healthy" too, a second path by which one passing probe erased another
probe's genuine failure.
On this fleet the trigger is a known false positive: the scheduler's network
vantage point cannot ICMP host:strong, so its ping check fails while every
ssh-script check succeeds. Under last-writer-wins that single probe declared
the whole host down, twice a minute.
Each check now records its own verdict (check_defs.last_health, migration 027)
and the entity's health is the worst across its enabled checks. A failing probe
now degrades the entity honestly and *stably*, without erasing what the other
five report, and health.changed fires only when that aggregate actually moves.
Checks that have never run are ignored rather than counted as unknown, so
adding a check cannot drag a known-good entity down before it has a verdict.
Also declares service:oikos in the seed. The previous commit re-pointed the mcp
ingress at it, but the entity only ever existed in the production database — so
a fresh seed (a new install, or a DR restore) failed on an unresolvable edge.
Caught by seeding an empty database rather than a copy of prod, which is the
only way that class of bug shows up.
Co-Authored-By: Claude <noreply@anthropic.com>
ListEnabledCheckDefs selected interval_s but never filtered on it, so every
enabled check ran on every 30s pass and the declared per-check intervals were
decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126
SSH connections every 30s (~363k/day) and `apt update` on every machine every
30 seconds — 14,400 mirror hits a day to answer a question that changes daily.
- check_defs.last_run_at (migration 026) + a due-ness predicate in the query.
A column rather than scheduler memory because this control plane restarts on
every deploy, and an in-memory map would re-fire every check on each restart.
- runCheck stamps last_run_at before processing the result, so a permanently
failing check backs off to its interval instead of re-running every pass.
- updates and backup-freshness drop to daily. Both answer questions whose
answers change about once a day; 60s was just the shared ssh-script default.
- last_run_at is seeded to a random offset within the interval so checks
created by the same seed do not stay in lockstep — otherwise ~165 probes
land in the same instant each minute instead of spread across it.
Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them.
Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day
(down from 14,400), with each 60s check landing at its own point in the minute.
Also renumbers 022→023, 023→024, 024→025: origin/main added its own
022_knowledge_revisions, and prod has already applied version 22. Left
colliding, prod would have skipped the monitoring_spec migration entirely and
then failed the seed on a missing column.
Co-Authored-By: Claude <noreply@anthropic.com>
Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by
discarded errors in checkdefaults:
- writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT
(slug) DO NOTHING, then wrote a check_defs row referencing it. On any
re-seed the slug already existed, the entity insert no-oped, and the FK
violated — aborting the ingest transaction and surfacing as an unrelated
failure several entities later. Re-seeding has been broken since; prod's
coverage was frozen at its first successful seed. This is what
TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting.
- shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed
to ".network" and overwrote each other; service:jellyfin collided with
lxc:jellyfin.
- The ssh-script checker never read the `args` config checkdefaults wrote, so
process_check.sh always ran without its unit name and returned "unknown".
Coverage is now 75/89. Monitoring is declared per entity type in
seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say
it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap.
coverageSweep raises an `unmonitored` signal only where a type declares
monitoring it lacks — 8 real gaps, no false positives.
Also:
- entity_types.attribute_schema was never ingested: the seed loader read
"attribute_schema" but the YAML says "attributes", so all 60 types stored
JSON null.
- ListExecutions ignored its declared target/action/correlation_id filters and
paginated on a non-unique target slug, dropping and repeating rows.
- started_at was captured but only written at terminal state, so a running
execution reported NULL for its whole life. The three MCP auto-run copies
wrote no timing at all; they are now one autoRun helper.
- SSH output was buffered to completion and discarded entirely on timeout.
Both sshExec copies now stream through a shared execlog sink into
execution_logs, and keep partial output when a command is cancelled.
- executions.correlation_id was a random per-execution uuid that correlated
nothing; it is now the chat session id, which is what lets the chat tail
live output.
- reversible_low had no auto-run branch despite policy declaring it
unattended. Since computeCommandRisk never returns it, the class only arises
when an agent declares it over a read_only command — so gating it penalised
candor without adding safety.
- backup-target gains a backup-freshness checker (portable find -mmin, since
the first target is on macOS), resolving its host by walking backs-up-to
backwards. The pre-deploy pg_dump is now a tracked backup target.
UI: an Executions section on entity detail with live output tailing, and
streamed output under a running `run` call in the chat timeline.
Migrations 022-024. Ops.svelte and context.ts exclude execution.output from
their refetch triggers, which would otherwise fire once a second per command.
Co-Authored-By: Claude <noreply@anthropic.com>
Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.
Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
cd.TargetID (falling back to the check's own id if unset) and write
status/metrics/events there. Signals stay keyed by the check entity,
unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
observation is older than 3x its fastest enabled check's interval
(floor 5m) is marked 'stale' and emits health.stale, so a stalled
scheduler or disabled check_def can no longer look like current data
forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
rows so dashboard/fleet-health rollups stop double-counting probes as
monitored entities. Historical metric_samples on check entities are
left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
the conversation instead of dropping them (previously only final text
was replayed, forcing the agent to re-derive fleet state every turn),
and inject a compact live fleet-health snapshot into the system prompt
each turn so it starts oriented instead of spending an iteration on
discovery.
Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.
Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- InsertMetricSample now includes ts=now() (TimescaleDB hypertable requires it)
- ssh-script: pass host and port separately (ssh uses -p flag, not host:port)
- ssh-script: use OIKOS_SSH_USER from config/env, default root
- Add -o LogLevel=ERROR to suppress SSH warnings polluting JSON output
- Use Output() (stdout only) instead of CombinedOutput()
- Set OIKOS_SSH_USER=root in scheduler docker-compose service
- Switch Dockerfile from distroless/static to alpine:3.21
- Install openssh-client-default in runtime image
- Mount SSH key in scheduler service (docker-compose)
- Add NET_RAW capability for ping checks
- Wire OIKOS_SSH_KEY_PATH and OIKOS_SSH_USER env vars in scheduler
- sshExec uses configured key path with StrictHostKeyChecking=no
- actuator/ssh.go: custom errorsAs chain broken — all SSH errors classified as SSHErrorOther.
Replaced with standard errors.As + errors.Is.
- scheduler/scheduler.go: all four check functions were stubs returning healthy.
Implemented real HTTP GET, TCP dial, unix.Statfs disk, and TLS cert expiry checks.
- learning/learning.go: uuid.NewV7() called unconditionally before ON CONFLICT upsert.
Now looks up existing pattern first, reuses entity_id.
- notifier/notifier.go: removed dead var_, fixed token regeneration every 15s.
Now skips if token_hash already set.
- phase3.go: removed dead GetPattern+dummy args call in PatchPattern.
- classify.go: removed unused var_ guard.
