dtoro 1dca2cfd7a feat(observability): restore monitoring coverage, make gaps visible, stream executions
Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by
discarded errors in checkdefaults:

- writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT
  (slug) DO NOTHING, then wrote a check_defs row referencing it. On any
  re-seed the slug already existed, the entity insert no-oped, and the FK
  violated — aborting the ingest transaction and surfacing as an unrelated
  failure several entities later. Re-seeding has been broken since; prod's
  coverage was frozen at its first successful seed. This is what
  TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting.
- shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed
  to ".network" and overwrote each other; service:jellyfin collided with
  lxc:jellyfin.
- The ssh-script checker never read the `args` config checkdefaults wrote, so
  process_check.sh always ran without its unit name and returned "unknown".

Coverage is now 75/89. Monitoring is declared per entity type in
seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say
it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap.
coverageSweep raises an `unmonitored` signal only where a type declares
monitoring it lacks — 8 real gaps, no false positives.

Also:
- entity_types.attribute_schema was never ingested: the seed loader read
  "attribute_schema" but the YAML says "attributes", so all 60 types stored
  JSON null.
- ListExecutions ignored its declared target/action/correlation_id filters and
  paginated on a non-unique target slug, dropping and repeating rows.
- started_at was captured but only written at terminal state, so a running
  execution reported NULL for its whole life. The three MCP auto-run copies
  wrote no timing at all; they are now one autoRun helper.
- SSH output was buffered to completion and discarded entirely on timeout.
  Both sshExec copies now stream through a shared execlog sink into
  execution_logs, and keep partial output when a command is cancelled.
- executions.correlation_id was a random per-execution uuid that correlated
  nothing; it is now the chat session id, which is what lets the chat tail
  live output.
- reversible_low had no auto-run branch despite policy declaring it
  unattended. Since computeCommandRisk never returns it, the class only arises
  when an agent declares it over a read_only command — so gating it penalised
  candor without adding safety.
- backup-target gains a backup-freshness checker (portable find -mmin, since
  the first target is on macOS), resolving its host by walking backs-up-to
  backwards. The pre-deploy pg_dump is now a tracked backup target.

UI: an Executions section on entity detail with live output tailing, and
streamed output under a running `run` call in the chat timeline.

Migrations 022-024. Ops.svelte and context.ts exclude execution.output from
their refetch triggers, which would otherwise fire once a second per command.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 13:51:14 +02:00

Oikos

Agentic homelab operating system written in Go. Single binary (cmd/oikos), Docker-deployed on mac-mini, with a standalone Nomos MCP agent gateway (cmd/nomos). Manages the hubris Proxmox homelab autonomously — observes state, classifies actions against policy, executes approved procedures over SSH, learns from outcomes, and escalates when uncertain.

For agents running on enrolled clients: start with AGENTS.md. For client machines: see CLIENTS.md. For developers: see CONTRIBUTING.md.

Quick start

# Dev stack (postgres + api + scheduler + notifier). The api/nomos
# services need a shared token — every route requires a real bearer
# credential, there's no dev-open bypass.
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile dev up -d

# Full stack (adds Nomos agent gateway)
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile full up -d

# Build standalone binary
go build -o bin/oikos -tags timetzdata ./cmd/oikos

# Run all roles in one process (dev mode)
OIKOS_DATABASE_URL="postgres://oikos:oikos_dev@localhost:5432/oikos?sslmode=disable" \
  OIKOS_API_TOKEN=dev-token \
  go run ./cmd/oikos all

# Control-room SPA (separate from the Go binary — see web/)
cd web && OIKOS_API_TOKEN=dev-token npm run dev   # http://localhost:5173

Architecture

                  ┌──────────────────────────────────┐
                  │         mac-mini (Docker)         │
                  │                                   │
  Workstation ─── │  nomos (8092) ──MCP── api (8090) │
  (mesh)          │    MCP gateway      REST + MCP    │
                  │                                   │
                  │  scheduler ── notifier ── postgres │
                  │  (observe)    (Matrix)   (Timescale)│
                  └──────────────────────────────────┘
Component Port Role
oikos api 8090 REST API + MCP server (tool list in AGENTS.md §3)
oikos scheduler Probe runner, signal lifecycle, metrics
oikos notifier Approval tokens, Matrix alerts
nomos serve 8092 MCP client gateway, query routing

Phases

Phase Status Description
1 — Ontology + DB TimescaleDB, migrations, seeds, blast_radius
2 — API OpenAPI-first REST + MCP, auth, SSE, audit
3 — Control loop Scheduler, actuator, learning, classifier, notifier
4 — Nomos agent Standalone MCP client gateway, agent activity
5 — Secrets Infisical backend + SOPS fallback, rotation runbooks
6 — Deploy CI pipeline, cutover checklist, watchdog, rollback

Full plan: plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md.

Operations

API endpoints

curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
  http://localhost:8090/api/v1/entities?type=service  # fleet
curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
  http://localhost:8090/api/v1/health                  # fleet health
curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
  http://localhost:8090/api/v1/agent-activity          # agent log

Nomos queries

# Structured tool call
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
  -d '{"tool":"get_blast_radius","args":{"entity_id":"service:authentik"}}'

# Natural language
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
  -d '{"query":"what depends on authentik?"}'

CLI

oikos migrate     # apply DB migrations
oikos seed        # ingest ontology/inventory/policy seeds
oikos export      # export DB state to YAML
oikos api         # serve REST + MCP
oikos scheduler   # run observe loop
oikos notifier    # run notification loop
oikos all         # all roles in one process
oikos secret list # enumerate SOPS secrets
oikos secret migrate  # SOPS → Infisical

Web UI

web/ is a standalone Svelte 5 SPA — not embedded in the oikos binary, not part of docker-compose.yml. It talks to api/nomos over HTTP with a bearer token entered on first launch (see web/src/pages/Config.svelte). Build with make ui, deploy with make deploy-ui (Caddy serves the static output). A native desktop wrapper exists at cmd/desktop/ — see plans/done/2026-07-12-wails-desktop-app.md.

Repo layout

cmd/oikos/          Go entry point — single binary
cmd/nomos/          Nomos MCP client gateway
cmd/webhook/        Gitea deploy-webhook receiver (push-to-deploy on mac-mini)
cmd/desktop/        Wails desktop wrapper around the SPA
internal/           Go packages (actuator, checkdefaults, config, db, domain,
                    httpapi, knowledge, learning, mcp, notifier, observability,
                    ontology, policy, safego, scheduler, secrets)
web/                Control-room SPA (Svelte 5) — standalone, not embedded
api/openapi.yaml    API contract (OpenAPI 3.1)
migrations/         Forward-only SQL migrations (TimescaleDB)
seeds/              Bootstrap YAML (ontology, inventory, policy, knowledge)
compose/            Dockerfiles + Caddy config
scripts/            Deploy, watchdog, verification, rollback
checks/             Host health-check scripts run over SSH by the scheduler
tools/              Client auto-setup scripts (checks)
ssh/                Deploy keys + authorized_keys management
vps/                Caddy/TURN config templates for the netbird VPS
nomos/              Nomos config, persona, skills
.agents/            Agent instruction files, shared conventions, skills
archive/            Historical reference (legacy wiki, plans, SOPS backups)
plans/              Design documents (active + done)
docs/adr/           Architecture decision records
docs/operations/    Runbooks (rollback, etc.)

For agents

See AGENTS.md for the full orientation. Quick reference:

  • Source of truth: DB (runtime) then seeds (bootstrap). Old wiki is archived at archive/knowledge/ — use MCP search_knowledge instead.
  • Mutations: classify against policy, request approval for destructive/config_mutation
  • Secrets: Infisical (primary) or SOPS (fallback) — never hardcode
Description
Agentic OS for running a Homelab
Readme 37 MiB
Languages
Go 53.1%
Svelte 25.7%
TypeScript 14%
Shell 3.8%
Python 1.7%
Other 1.5%