dtoro 8950bada44
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
fix: sshExec had no timeout — a hung remote command blocked forever
Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).

Two compounding bugs made this unrecoverable without manual intervention:

1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
   NO execution timeout — `session.CombinedOutput()` blocks until the remote
   command exits, with no deadline. A hung remote process blocks the Go
   goroutine forever; the execution can never leave 'running', and the
   operator has no way to make it stop. Fixed: both now race the SSH call
   against a 10-minute hard timeout, closing the session/client and
   returning a clear "timed out after 10m0s" error if exceeded. (The
   mcp/server.go copy also still had the original "swallowed non-zero exit"
   bug from before that fix was applied to httpapi's copy only — fixed here
   too.)

2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
   connectivity — it doesn't; a black-holed network can make each call hang
   far past the resolver's nominal timeout, so the documented "~90s" budget
   was never real. Wrapped every attempt in `timeout 3` so the wall-clock
   budget is now actually enforced (~2min worst case), and the failure
   message now suggests checking the net0 gateway.

Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):

- InlineApproval's track() polling loop had its own ~6min ceiling and simply
  STOPPED polling after that — silently going stale before the backend (now
  correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
  with margin, and added a distinct 'stalled' state if that's ever exceeded
  (explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
  execution's created_at), the actual command being run, and the execution
  ID — previously just a static "this can take a minute" with zero
  information. Also added command display to the destructive pending-
  approval card for full transparency before confirming.

Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 10:45:47 +02:00

Oikos

Agentic homelab operating system written in Go. Single binary (cmd/oikos), Docker-deployed on mac-mini, with a standalone Nomos MCP agent gateway (cmd/nomos). Manages the hubris Proxmox homelab autonomously — observes state, classifies actions against policy, executes approved procedures over SSH, learns from outcomes, and escalates when uncertain.

For agents running on enrolled clients: start with AGENTS.md. For client machines: see CLIENTS.md. For developers: see CONTRIBUTING.md.

Quick start

# Dev stack (postgres + api + scheduler + notifier)
docker compose --profile dev up -d

# Full stack (adds Nomos agent gateway)
docker compose --profile full up -d

# Build standalone binary
go build -o bin/oikos -tags timetzdata ./cmd/oikos

# Run all roles in one process (dev mode)
OIKOS_DATABASE_URL="postgres://oikos:oikos_dev@localhost:5432/oikos?sslmode=disable" \
  go run ./cmd/oikos all

Architecture

                  ┌──────────────────────────────────┐
                  │         mac-mini (Docker)         │
                  │                                   │
  Workstation ─── │  nomos (8092) ──MCP── api (8090) │
  (mesh)          │    MCP gateway      REST + MCP    │
                  │                                   │
                  │  scheduler ── notifier ── postgres │
                  │  (observe)    (Matrix)   (Timescale)│
                  └──────────────────────────────────┘
Component Port Role
oikos api 8090 REST API + MCP server (15 tools)
oikos scheduler Probe runner, signal lifecycle, metrics
oikos notifier Approval tokens, Matrix alerts
nomos serve 8092 MCP client gateway, query routing

Phases

Phase Status Description
1 — Ontology + DB TimescaleDB, migrations, seeds, blast_radius
2 — API OpenAPI-first REST + MCP, auth, SSE, audit
3 — Control loop Scheduler, actuator, learning, classifier, notifier
4 — Nomos agent Standalone MCP client gateway, agent activity
5 — Secrets Infisical backend + SOPS fallback, rotation runbooks
6 — Deploy CI pipeline, cutover checklist, watchdog, rollback

Full plan: plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md.

Operations

API endpoints

curl http://localhost:8090/api/v1/entities?type=service  # fleet
curl http://localhost:8090/api/v1/health                  # fleet health
curl http://localhost:8090/api/v1/agent-activity          # agent log

Nomos queries

# Structured tool call
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
  -d '{"tool":"get_blast_radius","args":{"entity_id":"service:authentik"}}'

# Natural language
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
  -d '{"query":"what depends on authentik?"}'

CLI

oikos migrate     # apply DB migrations
oikos seed        # ingest ontology/inventory/policy seeds
oikos export      # export DB state to YAML
oikos api         # serve REST + MCP
oikos scheduler   # run observe loop
oikos notifier    # run notification loop
oikos all         # all roles in one process
oikos secret list # enumerate SOPS secrets
oikos secret migrate  # SOPS → Infisical

Repo layout

cmd/oikos/          Go entry point — single binary
cmd/nomos/          Nomos MCP client gateway
internal/           Go packages (httpapi, mcp, scheduler, actuator, learning,
                    notifier, policy, secrets, db, config, ontology, domain,
                    knowledge)
api/openapi.yaml    API contract (OpenAPI 3.1)
migrations/         Forward-only SQL migrations (TimescaleDB)
seeds/              Bootstrap YAML (ontology, inventory, policy, knowledge)
compose/            Dockerfiles + Caddy config
scripts/            Deploy, watchdog, verification, rollback
nomos/              Nomos config, persona, skills
.agents/            Agent instruction files, shared conventions, skills
archive/            Historical reference (legacy wiki, plans, SOPS backups)
plans/              Design documents (active + done)
docs/adr/           Architecture decision records

For agents

See AGENTS.md for the full orientation. Quick reference:

  • Source of truth: DB (runtime) then seeds (bootstrap). Old wiki is archived at archive/knowledge/ — use MCP search_knowledge instead.
  • Mutations: classify against policy, request approval for destructive/config_mutation
  • Secrets: Infisical (primary) or SOPS (fallback) — never hardcode
Description
Agentic OS for running a Homelab
Readme 37 MiB
Languages
Go 53.1%
Svelte 25.7%
TypeScript 14%
Shell 3.8%
Python 1.7%
Other 1.5%