fix: scheduler wrote health/metrics/events to probe entities, not targets
Problem: every host/service/lxc/etc. entity_status row was permanently stuck at 'unknown' since creation. Verified against the live DB: metric_samples had 17,559 rows, 100% attached to type='check' probe entities and 0% to any real monitored entity; only 25 check entities ever had real health written. check_defs.entity_id (the probe's own bookkeeping entity) and check_defs.target_id (the host/service actually being observed) were both real fields, but the scheduler wrote UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by entity_id instead of target_id — so every check ran and every result was real, it just landed on the wrong row. This is the mechanism behind observed drift: the agent's dashboard/health tools reported the internal probes' state, never the actual fleet. Change: - scheduler.go: runCheck/resolveSignal now resolve targetID from cd.TargetID (falling back to the check's own id if unset) and write status/metrics/events there. Signals stay keyed by the check entity, unchanged, matching their existing resolution logic. - Added a staleness sweep to housekeeping(): an entity whose last observation is older than 3x its fastest enabled check's interval (floor 5m) is marked 'stale' and emits health.stale, so a stalled scheduler or disabled check_def can no longer look like current data forever. - migrations/016: deletes the now-orphaned check-entity entity_status rows so dashboard/fleet-health rollups stop double-counting probes as monitored entities. Historical metric_samples on check entities are left as-is (time-series data, not safe to reattribute). - openapi.yaml + regenerated gen code: Entity gains health/last_check_at; 'stale' added to the health enum everywhere it's used. - dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool: exclude type='check' entities from rollups. - nomos/agent.go: replay prior turns' tool_use/tool_result pairs into the conversation instead of dropping them (previously only final text was replayed, forcing the agent to re-derive fleet state every turn), and inject a compact live fleet-health snapshot into the system prompt each turn so it starts oriented instead of spending an iteration on discovery. Risk: config_mutation (schema-adjacent — new migration, no destructive DDL, additive DELETE only on orphaned rows). No behavior change until oikos-api/oikos-scheduler/nomos are rebuilt and redeployed. Verification: go build/vet clean across the repo. Ran this worktree's own API binary against the live dev Postgres on an alternate port (read-only from the live containers' perspective) and confirmed /api/v1/entities now returns health/last_check_at, and the dashboard health rollup dropped from double-counting to an honest 168 unmonitored entities (matches reality pre-deploy — the live scheduler hasn't run the fixed code yet). Confirmed check_defs.target_id correctly maps multiple checks to host:hubris via direct psql query. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1914,6 +1914,21 @@ components:
|
||||
updated_at:
|
||||
type: string
|
||||
format: date-time
|
||||
health:
|
||||
type: string
|
||||
description: last observed health, when the entity is monitored
|
||||
nullable: true
|
||||
enum:
|
||||
- healthy
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
- stale
|
||||
last_check_at:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
description: when health was last observed
|
||||
EntityCreate:
|
||||
type: object
|
||||
required:
|
||||
@@ -2020,6 +2035,7 @@ components:
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
- stale
|
||||
EntityType:
|
||||
type: object
|
||||
required:
|
||||
@@ -3031,6 +3047,9 @@ components:
|
||||
type: integer
|
||||
unknown:
|
||||
type: integer
|
||||
stale:
|
||||
type: integer
|
||||
description: last observation older than the check's expected cadence
|
||||
entities:
|
||||
type: array
|
||||
items:
|
||||
@@ -3051,6 +3070,7 @@ components:
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
- stale
|
||||
trend:
|
||||
type: string
|
||||
enum:
|
||||
@@ -3100,6 +3120,9 @@ components:
|
||||
type: integer
|
||||
unknown:
|
||||
type: integer
|
||||
stale:
|
||||
type: integer
|
||||
description: last observation older than the check's expected cadence
|
||||
signals_by_severity:
|
||||
type: object
|
||||
description: open (non-resolved) signal counts keyed by severity
|
||||
|
||||
Reference in New Issue
Block a user