From c9fb5fe5537d55112eb375062d5be9ad6dbc3e1d Mon Sep 17 00:00:00 2001 From: dtoro Date: Tue, 7 Jul 2026 17:54:17 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20README=20for=20Oikos=20identity=20?= =?UTF-8?q?=E2=80=94=20agentic=20homelab=20OS=20in=20Go?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rewrote README from 'living documentation' to 'Oikos — agentic homelab operating system written in Go.' - Added quick start, architecture diagram, component ports table. - Added Phase 1-6 status table with checkmarks. - Added API usage examples, Hermes query examples, CLI reference. - Added repo layout table. - Note: Gitea repo path (dtoro/Homelab-Docs → dtoro/oikos) requires Gitea UI rename — references in oikos/cards/, bootstrap.sh, and deploy scripts will need updating after the rename. --- README.md | 261 +++++++++++++++++++++--------------------------------- 1 file changed, 100 insertions(+), 161 deletions(-) diff --git a/README.md b/README.md index 40e636f..e525c13 100644 --- a/README.md +++ b/README.md @@ -1,190 +1,129 @@ -# Homelab OS +# Oikos -Living documentation for the **hubris** Proxmox homelab + Oikos operating system. +Agentic homelab operating system written in Go. Single binary (`cmd/oikos`), +Docker-deployed on mac-mini, with a standalone Hermes MCP agent gateway +(`cmd/hermes`). Manages the **hubris** Proxmox homelab autonomously — observes +state, classifies actions against policy, executes approved procedures over SSH, +learns from outcomes, and escalates when uncertain. -**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md), then [OIKOS.md](.agents/OIKOS.md). +**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md). ---- +## Quick start -## For Agents — Navigation & Entry Points +```bash +# Dev stack (postgres + api + scheduler + notifier) +docker compose --profile dev up -d -### You are running on a client enrolled in the hubris homelab +# Full stack (adds Hermes agent gateway) +docker compose --profile full up -d -1. **First:** Read [AGENTS.md](AGENTS.md) once. It explains who you are, the topology, available tools, conventions, and how to act. -2. **Before any mutation:** Read [OIKOS.md](.agents/OIKOS.md). It defines the operating model, risk classes, approval flow, and the ontology you'll consult. -3. **For specific workflows:** Load the matching skill from `.agents/skills//SKILL.md` (e.g., [service-health-check](.agents/skills/service-health-check/SKILL.md)). -4. **When in doubt:** Use MCP tools (`search_docs`, `get_page`, `explain`, `get_changelog`) — they're cheaper and more reliable than grepping. +# Build standalone binary +go build -o bin/oikos -tags timetzdata ./cmd/oikos -### Key References for Agents - -- **What am I?** → `/opt/homelab-context/hosts/.yaml` (read on first run) -- **Live topology** → `inventory.yaml` + `hosts/*.yaml` (canonical, always wins) -- **Risk & approval** → [oikos/policy.yaml](oikos/policy.yaml) (enforced, not advisory) -- **Runbooks & workflows** → [.agents/skills/](.agents/skills/) (risk class + verification checklist included) -- **State of Oikos** → [OIKOS.md build status](.agents/OIKOS.md#build-status-30-day-roadmap) (scheduled probes, drift detectors, signals, approval engine) - -### When to Use MCP vs Files vs Shell - -| Task | Use | Tool | -|------|-----|------| -| Resolve hostname → address | MCP | `get_host(name)` or `list_services()` | -| Search wiki by content | MCP | `search_docs(query)` | -| Read a wiki page | MCP or file | `get_page(path)` or `cat knowledge/wiki/.../...md` | -| Get changelog entries | MCP | `get_changelog(page, since?)` | -| Understand a service | MCP | `explain(service)` — compact context card, cheaper than search+read | -| Blast-radius query | MCP | `get_relations(entity)` (ontology walk) | -| List available secrets | MCP | `list_my_secrets()` (scoped to your age key) | -| Browse or grep | File | Raw `grep` when MCP unreachable, or exploratory browsing | - -**When MCP is unreachable:** fall back to grepping the clone at `/opt/homelab-context/`. The local files are the same; MCP is just an index. - ---- - -## Understanding the Operating Model - -Before you act, **classify your action against [oikos/policy.yaml](oikos/policy.yaml)**. - -### The Oikos OODA Loop + Decision Tree - -```mermaid -flowchart TD - Observe["**Observe**
probes, drift detectors, agent signals"] - Orient["**Orient**
ontology, context, state, entity relations"] - Decide{"**Decide**
classify against oikos/policy.yaml"} - Auto["Auto-act
(unattended)"] - Escalate["Escalate
homelab approval request"] - Act["**Act**
homelab CLI, runbooks, skills"] - Verify["**Verify**
checklist from SKILL.md"] - Ledger["**Ledger**
mutation record: who/what/risk"] - Document["**Document**
wiki update, same-session rule"] - - Observe --> Orient --> Decide - Decide -->|read_only, reversible_low| Auto - Decide -->|config_mutation, destructive| Escalate - Auto --> Act - Escalate -->|approval granted| Act - Act --> Verify --> Ledger --> Document - Document -.loop.-> Observe +# Run all roles in one process (dev mode) +OIKOS_DATABASE_URL="postgres://oikos:oikos_dev@localhost:5432/oikos?sslmode=disable" \ + go run ./cmd/oikos all ``` -### Risk Classes (enforced, not advisory) +## Architecture -From [oikos/policy.yaml](oikos/policy.yaml): +``` + ┌──────────────────────────────────┐ + │ mac-mini (Docker) │ + │ │ + Workstation ─── │ hermes (8092) ──MCP── api (8090) │ + (mesh) │ MCP gateway REST + MCP │ + │ │ + │ scheduler ── notifier ── postgres │ + │ (observe) (Matrix) (Timescale)│ + └──────────────────────────────────┘ +``` -- **read_only** — status, logs, docs, inventory queries. Unattended. MCP tools are all read_only. -- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger entry. -- **config_mutation** — tracked-config edits (commit+push, never local), deploys, upgrades, DNS/ingress changes. **Operator approval required.** -- **destructive** — destroy, format, wipe, rotate, revoke. **Approval + typed confirmation phrase.** +| Component | Port | Role | +|-----------|------|------| +| `oikos api` | 8090 | REST API + MCP server (15 tools) | +| `oikos scheduler` | — | Probe runner, signal lifecycle, metrics | +| `oikos notifier` | — | Approval tokens, Matrix alerts | +| `hermes serve` | 8092 | MCP client gateway, query routing | -### Decision Flow +## Phases -1. **Decide:** Use `homelab decide ` to classify (risk class × blast radius × confidence). -2. **Escalate if needed:** `homelab approval request` (Matrix-delivered to operator; see [operations/commands.md](.agents/operations/commands.md)). -3. **Execute:** Use `homelab` CLI (not ad-hoc SSH) — it enforces policy, logs mutations, and verifies outcomes. -4. **Document:** Update wiki in the same session (per [AGENTS.md §5](AGENTS.md#5-acting-on-the-homelab) and the [same-session rule](.agents/shared/page-templates.md#same-session-update-rule)). +| Phase | Status | Description | +|-------|--------|-------------| +| 1 — Ontology + DB | ✅ | TimescaleDB, migrations, seeds, blast_radius | +| 2 — API | ✅ | OpenAPI-first REST + MCP, auth, SSE, audit | +| 3 — Control loop | ✅ | Scheduler, actuator, learning, classifier, notifier | +| 4 — Hermes agent | ✅ | Standalone MCP client gateway, agent activity | +| 5 — Secrets | ✅ | Infisical backend + SOPS fallback, rotation runbooks | +| 6 — Deploy | ✅ | CI pipeline, cutover checklist, watchdog, rollback | -### The Ontology Graph +Full plan: [plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md). -Everything that can break, be changed, or hold data has an entity in `inventory.yaml` + `oikos/ontology.yaml`. Blast-radius questions ("what breaks if strong goes down?") are graph walks via `homelab node relations`, not doc archaeology. +## Operations -**See:** [OIKOS.md](.agents/OIKOS.md) (full operating model, OODA loop, primitives, lifecycle gates, build status). +### API endpoints ---- +```bash +curl http://localhost:8090/api/v1/entities?type=service # fleet +curl http://localhost:8090/api/v1/health # fleet health +curl http://localhost:8090/api/v1/agent-activity # agent log +``` -## Finding & Understanding Information +### Hermes queries -The narrative documentation is organized in **layers**: +```bash +# Structured tool call +curl -X POST localhost:8092/query -H "Content-Type: application/json" \ + -d '{"tool":"get_blast_radius","args":{"entity_id":"service:authentik"}}' -| Layer | What it is | Where | Immutable? | How agents use it | -|-------|-----------|-------|-----------|-------------------| -| **Sources** | Raw evidence: incidents, external refs, live state | `knowledge/sources/investigations/` | Yes | Read to understand root causes; do not rewrite | -| **Wiki** | Synthesized current-state: one page per node & per system | `knowledge/wiki/{containers,hosts,vms,infrastructure}/` | No | This is the reference layer — if wiki disagrees with live state, update it *in the same session* | -| **Index** | Pure listings — every page in scope with one-line summary | `index.md` / folder `README.md` | No | Navigation aid; keep it current when wiki restructures | -| **Log** | Append-only doc-maintenance record (restructures, ingests, lints) | `knowledge/log.md` | Yes (append-only) | Read to understand past doc changes; never edit directly | +# Natural language +curl -X POST localhost:8092/query -H "Content-Type: application/json" \ + -d '{"query":"what depends on authentik?"}' +``` -**Changelog ≠ Log:** Each wiki page ends with a `## Changelog` (infrastructure changes to that node, machine-parsed). That's not the Log; the Log records *doc operations* only. +### CLI -**See:** [llm-wiki.md](.agents/shared/llm-wiki.md) (full rules, page structure, immutability contract). +```bash +oikos migrate # apply DB migrations +oikos seed # ingest ontology/inventory/policy seeds +oikos export # export DB state to YAML +oikos api # serve REST + MCP +oikos scheduler # run observe loop +oikos notifier # run notification loop +oikos all # all roles in one process +oikos secret list # enumerate SOPS secrets +oikos secret migrate # SOPS → Infisical +``` ---- +## Repo layout -## Map & Quick Navigation +``` +cmd/oikos/ Go entry point — single binary +cmd/hermes/ Hermes MCP client gateway +internal/ Go packages (httpapi, mcp, scheduler, actuator, learning, + notifier, policy, secrets, db, config, ontology, domain) +api/openapi.yaml API contract (OpenAPI 3.1) +migrations/ Forward-only SQL migrations (TimescaleDB) +seeds/ Bootstrap YAML (ontology, inventory, policy) +compose/ Dockerfiles + Caddy config +scripts/ Deploy, watchdog, verification, rollback +hermes/ Hermes config, persona, skills +knowledge/wiki/ Narrative documentation (containers, hosts, infrastructure) +.agents/ Agent instruction files + skills +plans/ Design documents +``` -### Agent Entry Points (Start Here) +## For agents -- **You are an agent** → [AGENTS.md](AGENTS.md) (on deployed clients: `/opt/homelab-context/AGENTS.md`) -- **Operating model & risk policy** → [OIKOS.md](.agents/OIKOS.md) -- **Specific workflows** → [.agents/skills/](.agents/skills/) (load the matching SKILL.md before acting) -- **Operations cheatsheet** → [.agents/operations/commands.md](.agents/operations/commands.md) -- **Tools & MCP reference** → [AGENTS.md §3 — The MCP server](AGENTS.md#3-the-mcp-server) +See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference: -### Topology & Infrastructure +- **Source of truth:** DB (runtime) then repo (bootstrap + docs) +- **Mutations:** classify against policy, request approval for `destructive`/`config_mutation` +- **Wiki:** files under `knowledge/wiki/`, changelog at bottom of each page +- **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode -Node counts, IPs, and service lists change often — treat `inventory.yaml` and the index pages below as the source of truth, not this README. +## Related -- **Proxmox hosts** → [knowledge/wiki/hosts/index.md](knowledge/wiki/hosts/index.md) -- **VMs** → [knowledge/wiki/vms/index.md](knowledge/wiki/vms/index.md) -- **LXC containers** → [knowledge/wiki/containers/index.md](knowledge/wiki/containers/index.md) -- **Cross-cutting infrastructure** (DNS, ingress, mesh, backups, monitoring, auto-deploy, VPS) → [knowledge/wiki/infrastructure/index.md](knowledge/wiki/infrastructure/index.md) - -### Knowledge & References - -- **Glossary** — [GLOSSARY.md](knowledge/GLOSSARY.md) -- **Incidents & investigations** — [knowledge/sources/investigations/index.md](knowledge/sources/investigations/index.md) (active + [archive](knowledge/sources/investigations/archive/)) -- **Plans & design docs** — [plans/index.md](plans/index.md) -- **Hermes agent** (for Hermes-enrolled clients) — [HERMES.md](.agents/HERMES.md) - ---- - -## Conventions - -All pages follow: - -- **File naming.** Foundational docs (entry-points, agent instruction, references) are ALL-CAPS (`AGENTS.md`, `OIKOS.md`, `GLOSSARY.md`); containers use `-.md`; infrastructure pages use lowercase-with-dashes; plans and incidents use `YYYY-MM-DD-slug.md`; skills are `/SKILL.md`. See [page-templates.md](.agents/shared/page-templates.md#file-naming) for the full rules. -- **Voice & vocabulary.** Concise, technical, sysadmin-to-sysadmin. No marketing prose, no puffers (seamless, robust, leverage, etc.). Full rules in [writing-style.md](.agents/shared/writing-style.md). -- **Cross-linking is mandatory.** If a page references a node or system, link to it. Treat orphans as a bug. -- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running state, fix the wiki *and* add a changelog entry *in the same session*. -- **Tracked configs.** Pages for configs living in git repos (Caddy, Gitea, Artifacto, mule-image) must note the repo. Edits go through commit+push, never local changes. See [auto-deploy](knowledge/wiki/infrastructure/auto-deploy.md). -- **No secrets.** This is a private repo, but still: reference secret *paths*, never secret *values*. - -**For agents:** Read [caveman.md](.agents/shared/caveman.md) (terse communication standard). Use templates at [page-templates.md](.agents/shared/page-templates.md) when creating pages. - ---- - -## Updating the Wiki - -### When You Change Infrastructure - -1. Update the relevant page (config snapshot, ports, mounts, IP address). -2. Add a `### YYYY-MM-DD — title` entry to the page's `## Changelog` section (reverse chronological order). -3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link from the changelog. -4. If it's an incident, add a record to [`knowledge/sources/investigations/`](knowledge/sources/investigations/index.md). - -### When You Restructure the Wiki - -1. Update the relevant `index.md` / `README.md` in that section. -2. Add a single-line entry to [`knowledge/log.md`](knowledge/log.md): `## [YYYY-MM-DD] | ` (e.g., `## [2026-07-06] restructure | split infrastructure/dns into dns.md + dns-advanced.md`). - -### The Same-Session Update Rule - -**Any meaningful state change made in this session requires a wiki update before the session closes.** A change that touches a container page must also update: -- The `containers/index.md` table (IPs, host, mounts, status) -- The root `README.md` table (if affected) -- The Caddy page site list (if affects `*.hubris.network` routing) -- The DNS / ingress infrastructure pages (if affects routing) -- The `hosts/hubris.md` or `hosts/strong.md` page (if container count changes) -- The `inventory.yaml` host entry (source of truth for `hosts/*.yaml` generation) -- The `knowledge/wiki/infrastructure/topology.md` (regenerate if needed) - -Not updating all linked places is a bug. See [page-templates.md — same-session update rule](.agents/shared/page-templates.md#same-session-update-rule). - ---- - -## More Information - -- **For Hermes agents** → [HERMES.md](.agents/HERMES.md) (persona, source-of-truth hierarchy, token efficiency) -- **For manual workflows** → [.agents/operations/](.agents/operations/) (commands cheatsheet, agent enrollment, Hermes guide) -- **For skills/runbooks** → [.agents/skills/](.agents/skills/) (load the matching SKILL.md before acting; includes risk class + verification) -- **MCP tools** → [AGENTS.md §3](AGENTS.md#3-the-mcp-server) (available tools, when to use MCP vs files) -- **Page templates & voice** → [.agents/shared/](.agents/shared/) (page-templates.md, writing-style.md, caveman.md, llm-wiki.md) -- **Machine-readable substrate** → `inventory.yaml`, `oikos/policy.yaml`, `oikos/ontology.yaml` (not part of the wiki; see [llm-wiki.md](.agents/shared/llm-wiki.md#rules)) +- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology +- [plans/](plans/) — design documents and cutover checklist +- [docs/adr/](docs/adr/) — architecture decision records