diff --git a/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md b/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md index b1717bd..a98508e 100644 --- a/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md +++ b/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md @@ -1,220 +1,689 @@ -# Plan: Consolidate the Oikos control plane onto mac-mini +# Plan: Oikos — Docker-based agentic homelab OS on mac-mini -**Status:** Planned (2026-07-06) — approved, not yet executed. +**Status:** Planned (2026-07-06, revised) — supersedes the launchd-based consolidation plan. +**Predecessor:** The original plan re-platformed Oikos services from LXC 105 (apps) to +mac-mini via launchd plists. This revision pivots to a **Docker-based** architecture: +the repo becomes a containerized homelab OS, deployed via `docker compose` on mac-mini. -## Context +## Vision -The goal is an agentic OS that runs the homelab and keeps the operator in -control. Today the Oikos control plane is scattered and, worse, mis-placed: -`homelab-mcp`, `secrets-issuance`, and `oikos-console` all run on **LXC 105 -(apps) — a container inside hubris**, `oikos-scheduler` (the Observe stage) is -undeployed, and the Hermes agent runtime is spread across several clients. Two -problems follow: +Convert this repo into a **Docker-based agentic homelab OS**. The OS is a set of +containerized services that manage the homelab autonomously, with the operator in +control. Two actors: -- **The observer lives inside the observed.** A scheduler on apps/105 can't - report "hubris is down" — it dies with hubris. The Observe/Act loop needs an - always-on node *outside* both hypervisors. -- **Deploy is fragmented.** Three Gitea webhooks (ids 10/11/14); webhook 14 - (console) has been 403-broken for a cause never found, so console never - auto-redeploys. +- **Operator** (dtoro) — owns the homelab, expresses intent ("install X", "restart + Y"), approves destructive actions. Connects from any workstation via remote Hermes + or Matrix. +- **Agent** — Hermes core + custom homelab skills, running in Docker. Executes orders, + monitors the lab, escalates when unsure. -**Decision:** make **mac-mini** the single Oikos control node — it's always-on, -external to both hubris and strong, on the mesh, can SSH/curl both, and already -runs Hermes. Move *everything* (MCP, secrets-issuance, console, scheduler, the -new actuator, Hermes) there; apps/105 stops running Oikos. Because mac-mini -already has a launchd git-sync (5-min pull), we replace all three Gitea webhooks -with pull-based deploy — which eliminates the webhook-403 problem entirely. +All OS services run in Docker containers on mac-mini. The OS is deployed by a git +push (Gitea webhook → Docker rebuild). It's designed for mac-mini now, with a path +to multi-node later. -This is a **macOS re-platforming** (systemd → launchd) plus an **ingress -re-point**. The public URLs (`mcp.`/`secrets.`/`oikos.hubris.network`) and DNS -are unchanged — only the Caddy backend and the runtime host move. +## Decisions (from operator Q&A, 2026-07-06) + +| Question | Decision | +|---|---| +| Repo structure | One repo, reorganize internally. Wiki docs get restructured into the knowledge graph. | +| Hermes runtime | Runs inside Docker as part of the OS stack (gateway mode — connect from any workstation). | +| Agent → homelab access | Hybrid — mounted SSH keys for now, build actuator gateway incrementally. | +| Data storage | Migrate to PostgreSQL. `inventory.yaml` stays as declarative config; signals/ledger/state/knowledge go into DB. | +| Knowledge/context | Structured knowledge graph in Postgres — docs become typed entities linked to inventory entities. | +| Operator interface | Primary = remote Hermes from any workstation + Matrix. Console/UIs built later for specific tasks. | +| Host | mac-mini for now, designed to scale later. | +| Deploy | Git push → Gitea webhook → Docker rebuild + restart. | +| `bin/homelab` CLI | Replaced by an API. CLI becomes a thin client that calls the OS API. | +| Secrets | Migrate from SOPS+age to Infisical (open-source, API-first, Docker-native). | +| Matrix | Keep for now, abstract the notification layer for future channels. | +| MCP server | Merged into the unified API — one service, REST + MCP interfaces. | +| Agent type | Hermes core + custom homelab skills (standard Hermes, purpose-loaded). | +| apps/105 | Keep running as fallback until Docker OS is proven. | +| mac-mini cleanup | Start fresh in a new directory, clean up old artifacts later. | ## Target architecture ``` -Caddy (LXC 121, ingress) ──reverse_proxy over netbird──▶ mac-mini (always-on, macOS) - mcp.hubris.network ─────────────────────────────▶ homelab-mcp :9810 - secrets.hubris.network ─────────────────────────────▶ secrets-issuance :9820 - oikos.hubris.network ─────────────────────────────▶ oikos-console :8091 +┌──────────────────────────────────────────────────────────────────────┐ +│ mac-mini (Docker host, always-on) │ +│ │ +│ ┌──────────────┐ ┌────────────┐ ┌───────────────────────────┐ │ +│ │ PostgreSQL │ │ Infisical │ │ Hermes Agent (gateway) │ │ +│ │ │ │ (secrets) │ │ + homelab skills │ │ +│ │ • inventory │ │ │ │ + MCP client → API │ │ +│ │ • signals │ │ │ │ + SSH keys (mounted) │ │ +│ │ • ledger │ │ │ │ │ │ +│ │ • knowledge │ │ │ │ Any workstation connects │ │ +│ │ • state │ │ │ │ via Hermes gateway protocol │ │ +│ │ • approvals │ │ │ │ │ │ +│ └──────┬────────┘ └─────────────┘ └──────────┬────────────────┘ │ +│ │ │ │ +│ ┌──────┴─────────────────────────────────────────┴──────────────┐ │ +│ │ Oikos API (REST + MCP, one service) │ │ +│ │ │ │ +│ │ MCP interface (agent context): │ │ +│ │ get_host, list_services, search_knowledge, get_entity, │ │ +│ │ get_relations, get_change_history, get_state_snapshot │ │ +│ │ │ │ +│ │ REST interface (operations + operator): │ │ +│ │ GET /api/v1/hosts, /services, /signals, /approvals │ │ +│ │ POST /api/v1/exec — gated execution (actuator) │ │ +│ │ POST /api/v1/approve — operator approval │ │ +│ │ POST /api/v1/deploy — trigger service deploy │ │ +│ │ WS /api/v1/events — real-time signal/approval stream │ │ +│ │ │ │ +│ │ Policy enforcement, risk classification, ledger recording │ │ +│ └──────────────────────┬───────────────────────────────────────────┘ │ +│ │ │ +│ ┌──────────────────────┴───────────────────────────────────────────┐ │ +│ │ Scheduler (Observe) + Actuator (Act) │ │ +│ │ • Periodic probes (10-min) → signals in DB │ │ +│ │ • Reads open signals, classifies via decide.py │ │ +│ │ • Auto-acts (reversible_low) or escalates (approval) │ │ +│ │ • SSH to hubris/strong for probes + execution │ │ +│ └──────────────────────┬───────────────────────────────────────────┘ │ +│ │ │ +│ ┌──────────────────────┴───────────────────────────────────────────┐ │ +│ │ Notifier (abstracted, pluggable) │ │ +│ │ • Matrix (current) — alerts, approval requests │ │ +│ │ • Future: webhook, email, Slack, etc. │ │ +│ └──────────────────────────────────────────────────────────────────┘ │ +│ │ +│ Deploy: Gitea webhook → docker compose build + up -d │ +│ Volumes: pg-data, infisical-data, ssh-keys (ro mount), hermes-config │ +│ Network: Docker bridge (internal) + host network (SSH outbound) │ +└──────────────────────────────────────────────────────────────────────┘ -mac-mini also runs (no ingress, mesh/SSH outward): - oikos-scheduler + actuator (launchd StartInterval 600s) ──ssh/curl──▶ hubris, strong - Hermes agent runtime (already here) +External: + Caddy (LXC 121) → mac-mini mesh IP :8090 (API), future UIs + apps/105 → fallback, kept running until Docker OS proven + hubris/strong → SSH targets for scheduler/actuator + Any workstation → Hermes gateway (connects to agent in Docker) + Gitea (LXC 104) → webhook triggers Docker rebuild +``` -Single synced checkout: /opt/homelab-context (5-min launchd git pull = deploy trigger) -apps/105: Oikos services stopped + removed; keeps its non-Oikos roles (artifacto, plantuml) +## Repo reorganization + +The repo shifts from "wiki + code mixed" to "OS code + declarative config + docs as +ingest source." Proposed layout: + +``` +/ # repo root +├── docker-compose.yml # the OS stack definition +├── compose/ # per-service Dockerfiles + build context +│ ├── api/ # Oikos API (REST + MCP) — merges mcp/ + bin/homelab +│ │ └── Dockerfile +│ ├── scheduler/ # Observe + Act loop +│ │ └── Dockerfile +│ ├── hermes/ # Hermes agent runtime +│ │ └── Dockerfile +│ ├── postgres/ # init scripts, migrations +│ │ └── init.sql +│ └── infisical/ # Infisical config +│ └── config.yaml +├── oikos/ # Core OS Python package (reused, adapted) +│ ├── api/ # NEW — unified API server +│ │ ├── server.py # FastAPI app, REST routes + MCP protocol adapter +│ │ ├── routes/ # REST route modules +│ │ └── mcp.py # MCP protocol adapter (exposes same tools via MCP) +│ ├── db/ # NEW — database layer +│ │ ├── models.py # SQLAlchemy models: Inventory, Signal, Ledger, Knowledge, Approval +│ │ ├── migrations/ # Alembic migrations +│ │ └── ingest.py # Ingest inventory.yaml + docs into DB on deploy +│ ├── scheduler/ # Adapted from scheduler.py — reads/writes DB +│ ├── actuator/ # NEW — Act stage: reads signals, classifies, executes +│ │ └── act.py +│ ├── policy/ # Adapted — policy.yaml loader + classifier +│ ├── knowledge/ # NEW — knowledge graph ingestion from docs +│ │ └── ingest_docs.py +│ ├── notifier/ # NEW — abstracted notification (Matrix impl) +│ │ ├── base.py # Notifier interface +│ │ └── matrix.py # Matrix implementation +│ ├── decide.py # Reused — risk classifier (reads from DB) +│ ├── signal.py # Adapted — signals in DB instead of JSONL files +│ ├── approve.py # Adapted — approvals in DB +│ ├── ledger.py # Adapted — ledger in DB +│ ├── drift.py # Adapted — drift detection, writes signals to DB +│ ├── relations.py # Reused — ontology graph walk (reads from DB) +│ └── report.py # Adapted — generates reports from DB +├── hermes/ # Hermes agent config + skills (mounted into container) +│ ├── config.yaml # Hermes config (providers, models, gateway) +│ ├── SOUL.md # Agent persona (homelab-specific) +│ └── skills/ # Custom homelab skills +│ └── homelab-ops/ +│ └── SKILL.md +├── inventory.yaml # Declarative config — the "kernel data structure" (stays YAML) +├── ontology.yaml # Systems model (stays YAML) +├── policy.yaml # Risk & approval policy (stays YAML) +├── docs/ # Narrative docs — ingested into knowledge graph on deploy +│ ├── containers/ # LXC/VM documentation +│ ├── hosts/ # Host documentation +│ ├── infrastructure/ # Infrastructure docs +│ └── investigations/ # Incident records +├── secrets/ # SOPS-encrypted secrets (migrated to Infisical, then removed) +└── .sops.yaml # SOPS config (deprecated after Infisical migration) +``` + +**What stays YAML (declarative, human-edited, git-tracked):** +- `inventory.yaml` — topology, the kernel data structure +- `ontology.yaml` — entity types, relationships, lifecycle +- `policy.yaml` — risk classes, approval rules, autonomy settings +- `hermes/config.yaml` — agent configuration +- `hermes/SOUL.md` — agent persona + +**What moves to PostgreSQL (mutable, runtime state):** +- `signals/` → `signals` table +- `ledger/` → `ledger_entries` table +- `oikos/state.json` → `state_snapshots` table +- `approvals/` → `approvals` table +- `knowledge/wiki/` narrative docs → `knowledge_entities` + `knowledge_relations` tables + +**What gets removed (no longer needed):** +- `mcp/server.py` — merged into `oikos/api/server.py` +- `bin/homelab` — replaced by the API; thin CLI client can be generated later +- `secrets-issuance/` — replaced by Infisical +- `oikos/systemd/` — replaced by Docker containers +- `oikos/console/` — deferred (built later as a specific UI on top of the API) +- `mcp/deploy/`, `oikos/console/deploy/` — replaced by Docker build +- `hosts/*.yaml` — generated from inventory, now served by the API directly + +## Docker Compose stack + +```yaml +# docker-compose.yml (simplified — full version in the repo) +services: + postgres: + image: postgres:16-alpine + volumes: [pg-data:/var/lib/postgresql/data, ./compose/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql] + environment: + POSTGRES_DB: oikos + POSTGRES_PASSWORD_FILE: /run/secrets/pg_password + secrets: [pg_password] + healthcheck: ... + + infisical: + image: infisical/infisical:latest + volumes: [infisical-data:/var/lib/infisical] + environment: ... + # Secrets manager — replaces SOPS+age for runtime secret access + + api: + build: ./compose/api + depends_on: [postgres, infisical] + environment: + DATABASE_URL: postgresql://oikos@postgres/oikos + HOMELAB_CONTEXT_DIR: /opt/homelab-context + volumes: + - ./inventory.yaml:/app/inventory.yaml:ro + - ./ontology.yaml:/app/ontology.yaml:ro + - ./policy.yaml:/app/policy.yaml:ro + - ./docs:/app/docs:ro + ports: ["8090:8090"] # REST + MCP + # On startup: ingest inventory.yaml + docs into DB if changed + + scheduler: + build: ./compose/scheduler + depends_on: [api, postgres] + environment: + DATABASE_URL: postgresql://oikos@postgres/oikos + SSH_KEY: /run/secrets/ssh_key + volumes: + - ssh-keys:/run/secrets:ro + # Runs observe + act loop on a cron-like schedule (10-min interval) + + hermes: + build: ./compose/hermes + depends_on: [api, infisical] + environment: + OIKOS_API_URL: http://api:8090 + INFISICAL_TOKEN: ... + volumes: + - ./hermes:/root/.hermes:ro + - ssh-keys:/root/.ssh:ro + - hermes-data:/root/.hermes/data + ports: ["8092:8092"] # Hermes gateway port + # Agent runtime — connects to API via MCP, has SSH for direct host access + + notifier: + build: ./compose/scheduler # shares image with scheduler + depends_on: [api, postgres] + # Matrix bot — listens for approval requests, sends alerts + # Abstracted via notifier interface — Matrix impl now, pluggable later + +volumes: + pg-data: + infisical-data: + ssh-keys: # populated from host ~/.ssh at deploy time + hermes-data: ``` ## Workstreams -### 1. macOS runtime — launchd plists (`deploy/macos/`, new) +### 1. Database schema + models (`oikos/db/`, new) -One `LaunchDaemon` plist per service (run at boot, no GUI login), mirroring each -systemd unit's `ExecStart`/`Environment`, replacing `[Timer]` with launchd keys: +The foundational layer — everything else reads/writes through this. -| Service | plist | launchd equivalent of | -|---|---|---| -| `homelab-mcp` | `network.hubris.homelab-mcp.plist` | `mcp/deploy/homelab-mcp.service` (KeepAlive) | -| `secrets-issuance` | `network.hubris.secrets-issuance.plist` | its systemd unit (KeepAlive) | -| `oikos-console` | `network.hubris.oikos-console.plist` | `oikos/console/deploy/oikos-console.service` (KeepAlive) | -| `oikos-scheduler` | `network.hubris.oikos-scheduler.plist` | `oikos/systemd/oikos-scheduler.{service,timer}` (`StartInterval` 600) | +**Tables:** -- **Single checkout, shared venv.** All services run in-place from - `/opt/homelab-context` with one shared `.venv` (gitignored), instead of the - Linux per-service checkouts (`/opt/homelab-mcp`, `/opt/oikos-console`). This is - the concrete consolidation win: one clone, one sync, N daemons. -- **Confinement gap (tradeoff, document it):** the systemd units use - `ProtectSystem=strict`, `ReadOnlyPaths`, `NoNewPrivileges`, `PrivateTmp` — - launchd has no equivalent. Mitigate by running the daemons as a dedicated - non-root `_oikos` user (or `dtoro`) with least-privilege file perms, and note - the reduced isolation in the deploy README + OIKOS.md risk section. -- **`install-macos.sh`** — create `_oikos` user + venv (`pip install` the deps), - copy plists to `/Library/LaunchDaemons`, `launchctl bootstrap system …`. - Idempotent, mirroring `mcp/deploy/deploy.sh`'s shape. +```sql +-- Inventory entities (synced from inventory.yaml on deploy) +CREATE TABLE entities ( + id TEXT PRIMARY KEY, -- "host:hubris", "service:caddy" + type TEXT NOT NULL, -- host, service, lxc, vm, etc. + name TEXT NOT NULL, + domain TEXT, -- physical, compute, network, etc. + state TEXT DEFAULT 'active', -- provisioning, active, deprecated, destroyed + data JSONB NOT NULL, -- full entity record from inventory.yaml + updated_at TIMESTAMPTZ DEFAULT now() +); -### 2. macOS deploy = the existing pull-sync (retire the 3 webhooks) +-- Entity relationships (the ontology graph) +CREATE TABLE relations ( + source_id TEXT REFERENCES entities(id), + target_id TEXT REFERENCES entities(id), + type TEXT NOT NULL, -- hosts, provides, mounts, depends-on, etc. + data JSONB, + PRIMARY KEY (source_id, target_id, type) +); -- Extend mac-mini's existing launchd git-sync post-pull hook (the enrollment - `tools/post-pull.sh` path) to `launchctl kickstart -k` any service whose files - changed after a pull. Pull-based deploy replaces webhook-push deploy. -- **Retire Gitea webhooks 10, 11, 14** (operator step in Gitea UI). This removes - the never-solved console-webhook 403 by deleting the webhook, not fixing it. -- Update `knowledge/wiki/infrastructure/auto-deploy.md`: mac-mini services deploy - by pull-sync, not webhook; the webhook shape now applies only to remaining - in-lab config repos. +-- Signals (replaces signals/*.jsonl) +CREATE TABLE signals ( + id TEXT PRIMARY KEY, -- "sig-2026-07-06-0001" + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + kind TEXT NOT NULL, -- service-down, disk-threshold, drift, etc. + severity TEXT NOT NULL, -- info, warning, critical + entity_id TEXT REFERENCES entities(id), + evidence TEXT, + likely_cause TEXT, + recommended_action JSONB, + verification TEXT, + state TEXT NOT NULL DEFAULT 'raised', -- raised, acknowledged, acting, resolved, muted + mute_until TIMESTAMPTZ, + note TEXT, + updated_at TIMESTAMPTZ DEFAULT now() +); -### 3. Build the actuator + autonomy kill-switch (the Act stage) +-- Change ledger (replaces ledger/*.jsonl) +CREATE TABLE ledger_entries ( + id SERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + entity_id TEXT REFERENCES entities(id), + action TEXT NOT NULL, + risk_class TEXT NOT NULL, + result TEXT, -- ok, failed, escalated + approval_id TEXT, -- references approvals if gated + agent TEXT, -- agent identity (age pubkey or hostname) + notes TEXT +); -Still required — it's the control loop, now living on mac-mini. +-- Approvals (replaces approvals/*.jsonl) +CREATE TABLE approvals ( + id TEXT PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + entity_id TEXT REFERENCES entities(id), + action TEXT NOT NULL, + risk_class TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', -- pending, approved, denied, expired + ttl INTERVAL NOT NULL DEFAULT '1 hour', + decided_at TIMESTAMPTZ, + decided_by TEXT, + confirmation_phrase TEXT +); -- **`oikos/act.py`** (new): read open Signals (`oikos_signal.list_signals( - state="raised")`); for each with a `recommended_action`, call - `oikos_decide.classify()`. `auto-act` (reversible_low, contained, confident) - → `start_acting()` (the existing unused transition), execute via - `homelab restart -y` (reuses `cmd_restart` + its `_record_change` ledger - write), run the Signal's `verification`, then `resolve()` or leave open with a - `result=failed` ledger entry so the next pass escalates. Everything else → - `oikos_approve.request(...)` + `acknowledge(note="escalated: ")`. A - ledger-history loop-guard caps auto-retries per (entity, kind). -- **Autonomy kill-switch** in `oikos/policy.yaml` (`autonomy.auto_act: - off|reversible_low`, `never_auto_act:` entity list) + loaders in - `oikos/policy.py` — the "keep control" guarantee. -- Wire `python3 oikos/act.py run` into the scheduler's launchd pass (after the - observe pass) and commit `signals/` + `ledger/`. +-- State snapshots (replaces oikos/state.json) +CREATE TABLE state_snapshots ( + id SERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + entity_id TEXT REFERENCES entities(id), + health TEXT, -- healthy, degraded, down, unknown + data JSONB -- full probe response +); -### 4. Ingress re-point (Caddy backends: apps → mac-mini mesh) +-- Knowledge graph (replaces narrative wiki docs) +CREATE TABLE knowledge_entities ( + id TEXT PRIMARY KEY, -- "doc:105-apps", "investigation:2026-07-05-disk" + type TEXT NOT NULL, -- container-doc, host-doc, investigation, plan, runbook + entity_id TEXT REFERENCES entities(id), -- link to inventory entity if applicable + title TEXT NOT NULL, + content TEXT NOT NULL, -- markdown body + tags TEXT[], + source_path TEXT, -- original file path for traceability + updated_at TIMESTAMPTZ DEFAULT now() +); -- In `dtoro/caddy-conf` (separate repo, tracked-config commit+push): change the - three `reverse_proxy` backends for `mcp.`/`secrets.`/`oikos.hubris.network` - from `192.168.8.205:` to **mac-mini's netbird address** - (`mac-mini-234-17.netbird.selfhosted:`), since Caddy (8.x) and mac-mini - (178.x) are cross-subnet — mesh is the robust path. **DNS and public URLs - unchanged.** -- **Verify Caddy → mac-mini reachability first** (Caddy container must resolve + - reach the netbird name; confirm netbird is on the caddy LXC or route via - hubris). If mesh isn't reachable from Caddy, fall back to the 178.x LAN IP and - confirm hubris routes 8.x↔178.x (it's dual-homed: hubris 178.181, apps 8.205). +-- Knowledge relationships (cross-references between docs + entities) +CREATE TABLE knowledge_relations ( + source_id TEXT REFERENCES knowledge_entities(id), + target_id TEXT, -- can be knowledge_entity or inventory entity + type TEXT NOT NULL, -- describes, relates-to, runbook-for, etc. + PRIMARY KEY (source_id, target_id, type) +); +``` -### 5. Secrets & SSH keys on mac-mini +**Ingest pipeline** (`oikos/db/ingest.py`): +- On startup / deploy: parse `inventory.yaml` → upsert into `entities` + `relations` +- Walk `docs/` directory → parse markdown, extract frontmatter → upsert into `knowledge_entities` +- Link knowledge entities to inventory entities by path convention (`docs/containers/105-apps.md` → `entity:lxc:apps`) +- Idempotent — safe to re-run on every deploy -- **SOPS recipients:** grant mac-mini's `age_pubkey` - (`age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs`) on every - secret the moved services decrypt at runtime (secrets-issuance's issuing key, - approval HMAC `secrets/oikos-approval-hmac.yaml`, any service creds), then - `sops updatekeys`. Edit `.sops.yaml` recipient lists + re-key; commit via - `homelab client add mac-mini --finalize-pubkey …` path (comment-preserving). -- **SSH keys:** provision on mac-mini the `mcp-reader.key` restricted key the MCP - server SSH-proxies to hubris with (`HOMELAB_MCP_SSH_KEY`), plus root SSH access - to hubris + strong for the scheduler's disk probe (`_ssh` in - `oikos/scheduler.py:102`). Store under `/etc/homelab-mcp/` and `_oikos`'s - `~/.ssh` respectively. -- **Security note (document):** the age-key *issuer* now runs on a workstation. - Record the widened surface in OIKOS.md; least-privilege `_oikos` user + - keychain/file perms are the mitigation. +**Migration from existing data:** +- Write a one-time script to import existing `signals/*.jsonl`, `ledger/*.jsonl`, `approvals/*.jsonl` into the DB +- `oikos/state.json` → `state_snapshots` table -### 6. Inventory / topology truth (same-session rule) +### 2. Unified API server (`oikos/api/`, new — merges MCP + homelab CLI) -- `inventory.yaml`: set `services.{homelab_mcp,secrets_issuance}.backend: - mac-mini`; add `oikos_console` + `oikos_scheduler` service entries backed by - mac-mini; give `mac-mini` a `runs:` list (homelab-mcp, secrets-issuance, - oikos-console, oikos-scheduler); remove `homelab-mcp`/`secrets-issuance` from - `apps.runs`. Keep `systemd_unit`→ note launchd label. -- Regenerate derived views: `mcp/build_host_files.py` (hosts/*.yaml) and - `oikos/gen-topology.py` (cards + `topology.md`). Update - `knowledge/wiki/containers/105-apps.md` and add a mac-mini control-node page - under `knowledge/wiki/hosts/` (or note on the workstation). +One FastAPI service exposing two interfaces: -### 7. Decommission on apps/105 (operator steps, documented not executed) +**MCP interface** (`oikos/api/mcp.py`): +- Same tool surface as the current `mcp/server.py` (get_host, list_services, search_knowledge, etc.) +- Reads from PostgreSQL instead of filesystem +- `search_docs` → `search_knowledge` (queries the knowledge graph) +- New tools: `get_entity_relations`, `get_signal_history`, `get_ledger` +- Protocol: FastMCP over streamable HTTP (same as current) -`systemctl disable --now homelab-mcp secrets-issuance oikos-console` + their -`-deploy` webhook units on apps; remove `/opt/homelab-mcp`, `/opt/oikos-console` -checkouts. Captured in the deploy README as a cutover checklist. +**REST interface** (`oikos/api/routes/`): +- `GET /api/v1/hosts` — list hosts (replaces `homelab list`) +- `GET /api/v1/services` — list services +- `GET /api/v1/hosts/{name}` — host detail (replaces `homelab status`) +- `GET /api/v1/services/{name}/health` — service health (cache-first, `?live=true` for probe) +- `GET /api/v1/signals` — list signals (`?state=raised&severity=warning`) +- `POST /api/v1/signals/{id}/ack` — acknowledge +- `POST /api/v1/signals/{id}/resolve` — resolve +- `GET /api/v1/approvals` — pending approvals +- `POST /api/v1/approvals/{id}/decide` — approve/deny (operator only, Authentik-gated) +- `POST /api/v1/exec` — gated execution (the actuator entrypoint) + - Body: `{entity, action, approval_id?}` + - Classifies via `decide.py`, checks approval if needed, executes, records ledger +- `POST /api/v1/deploy/{service}` — trigger a service deploy +- `WS /api/v1/events` — real-time stream (signals, approvals, ledger entries) +- `GET /api/v1/knowledge/{entity_id}` — knowledge graph query for an entity +- `GET /api/v1/knowledge/search?q=...` — search knowledge graph -### 8. Docs +**Policy enforcement** is built into the API: +- Every mutating endpoint classifies the action via `oikos/decide.py` +- `config_mutation`/`destructive` actions require a valid approval ID +- All mutations write to the ledger automatically -`.agents/OIKOS.md` (control node = mac-mini; loop closed; risks), -`auto-deploy.md` (pull-sync), `deploy/macos/README.md` (cutover runbook), -`.agents/skills/autopilot-pass/SKILL.md` (supervised actuator run), host/service -cards (regenerated). +**Auth:** +- MCP interface: no auth (internal, container-to-container) +- REST interface: Authentik OIDC forward-auth (via Caddy) for operator endpoints +- Internal service-to-service: shared secret / mTLS (Docker network) -## Critical files +### 3. Scheduler + Actuator (`oikos/scheduler/`, `oikos/actuator/`) -| File | Change | -|---|---| -| `deploy/macos/*.plist` + `install-macos.sh` + `README.md` | **New** — launchd runtime + cutover | -| `oikos/act.py` | **New** — actuator (Act stage) | -| `oikos/policy.yaml`, `oikos/policy.py` | Autonomy kill-switch + loaders | -| `oikos/systemd/run-scheduler.sh` → macOS variant | Observe+Act pass, commit signals/+ledger/ | -| `inventory.yaml` | Move service backends apps→mac-mini; mac-mini `runs:` | -| `hosts/*.yaml`, `oikos/cards/*`, `topology.md` | Regenerated | -| `dtoro/caddy-conf` (external repo) | 3 reverse_proxy backends → mac-mini mesh | -| `.sops.yaml` | Grant mac-mini recipient on moved-service secrets; re-key | -| `bin/homelab` | Optional `autopilot` subcommand | -| Docs: `OIKOS.md`, `auto-deploy.md`, `105-apps.md` | Reflect consolidation | +**Scheduler (Observe)** — adapted from `oikos/scheduler.py`: +- Runs as a Docker service on a timer (10-min interval via a loop or cron sidecar) +- Probes service health (HTTP), disk usage (SSH to hubris/strong), drift detection +- Writes signals to PostgreSQL (not JSONL files) +- Writes state snapshots to DB +- No more git commit/push of signals — they're in the DB + +**Actuator (Act)** — new, `oikos/actuator/act.py`: +- Reads open signals from DB where `recommended_action` is set +- For each: classify via `decide.py` + - `auto-act` (reversible_low, contained, confident) → execute via SSH/API, verify, resolve signal, write ledger + - `escalate` → create approval request, notify via Matrix, acknowledge signal +- Loop-guard: check ledger history per (entity, action) to cap auto-retries +- Autonomy kill-switch: `policy.yaml` → `autonomy.auto_act: off|reversible_low`, `never_auto_act: [entity list]` +- **Phase 1**: direct SSH execution (keys mounted) +- **Phase 2**: execution through the API's `/exec` endpoint (gateway pattern) + +### 4. Hermes agent container (`compose/hermes/`) + +**What it is:** +- Hermes Agent runtime in a Docker container +- Runs in gateway mode — workstations connect to it remotely +- Config: `hermes/config.yaml` (providers, models, gateway port) +- Persona: `hermes/SOUL.md` (homelab-specific) +- Skills: `hermes/skills/homelab-ops/SKILL.md` (how to interact with the homelab) + +**What it has access to:** +- **Oikos API** via MCP (container network, no auth needed) +- **Infisical** for secret retrieval (API token) +- **SSH keys** mounted from host (`~/.ssh` → `/root/.ssh:ro`) — for direct host access (hybrid approach) +- **Hermes data volume** — persistent state, session history + +**How workstations connect:** +- Hermes gateway listens on port 8092 +- Any workstation with Hermes CLI can connect: `hermes gateway connect mac-mini:8092` +- Or via Caddy ingress: `hermes.hubris.network` → mac-mini:8092 (mesh) + +**Homelab skills** (loaded into Hermes): +- `homelab-ops` — how to use the Oikos API, classify actions, request approvals +- `homelab-deploy` — how to deploy/manage services +- `homelab-troubleshoot` — diagnostic procedures +- (Future skills added as needed) + +### 5. Infisical secrets migration + +**Phase 1 — Stand up Infisical:** +- Add Infisical to the Docker stack +- Configure with PostgreSQL backend (shared with Oikos DB or separate DB) +- Bootstrap admin credentials, create `hubris` project + environments (prod, staging) + +**Phase 2 — Migrate secrets:** +- Write a migration script: read all `secrets/*.yaml` (SOPS-encrypted), decrypt with age key, import into Infisical +- Map each secret to Infisical's key-value structure +- Verify all secrets are imported + +**Phase 3 — Wire services:** +- All OS services authenticate to Infisical via machine identity (service token) +- Services fetch secrets at startup or on-demand +- Remove `secrets/` directory and `.sops.yaml` once verified +- The age key on mac-mini stays for one-time decryption during migration, then is retired + +**Phase 4 — Retire SOPS:** +- Remove SOPS dependencies from the repo +- Update inventory to reflect Infisical as the secret store +- Document the new secret access pattern + +### 6. Knowledge graph ingestion (`oikos/knowledge/`) + +**What it does:** +- On deploy, walks `docs/` directory +- Parses each markdown file: + - Extracts frontmatter (if any) for metadata + - Infers entity relationships from path conventions: + - `docs/containers/105-apps.md` → links to `entity:lxc:apps` + - `docs/hosts/hubris.md` → links to `entity:host:hubris` + - `docs/infrastructure/dns.md` → links to `entity:service:dns` + - Extracts cross-references (links to other docs) → `knowledge_relations` + - Tags by directory: container-doc, host-doc, infrastructure, investigation, plan +- Upserts into `knowledge_entities` + `knowledge_relations` tables + +**Agent access:** +- MCP tool `search_knowledge(query)` — full-text search on knowledge graph +- MCP tool `get_entity_knowledge(entity_id)` — all docs related to an entity +- MCP tool `get_relations(entity_id)` — ontology blast-radius query (existing, adapted to DB) + +**Docs as source, DB as query layer:** +- Humans still write markdown docs in `docs/` +- The DB is the query/index layer — agents never grep files +- On deploy, docs are re-ingested (idempotent) +- Future: wiki-style editing UI that writes back to `docs/` via git + +### 7. Notifier abstraction (`oikos/notifier/`) + +**Interface:** +```python +class Notifier(Protocol): + def send_alert(self, signal: Signal) -> None: ... + def send_approval_request(self, approval: Approval) -> None: ... + def listen_for_decisions(self) -> Iterator[ApprovalDecision]: ... +``` + +**Matrix implementation** (current): +- Sends alerts to `@dtoro:avispero` via Synapse (LXC 118) +- Approval requests as messages with ✅/❌ reactions +- Listens for reactions to record decisions +- Reuses existing `oikos/approve.py` Matrix delivery logic + +**Future implementations** (pluggable): +- Webhook, email, Slack, etc. +- Registered in config, selected at runtime + +### 8. Docker build + deploy pipeline + +**Build:** +- `docker compose build` from the repo root +- Each service has a Dockerfile in `compose//` +- Shared base image: Python 3.12-slim + common deps (postgresql libs, etc.) +- Multi-stage builds for smaller images + +**Deploy trigger:** +- Gitea webhook on push to `main` → hits a deploy endpoint on mac-mini +- Deploy script: `git pull && docker compose build && docker compose up -d` +- Alternatively: a "deployer" sidecar container watches Gitea and triggers rebuilds +- The existing 5-min launchd git pull can serve as a fallback deploy mechanism initially + +**Health checks:** +- Each service has a Docker healthcheck +- `docker compose ps` shows health status +- API exposes `/health` for external monitoring + +### 9. Ingress re-point (Caddy) + +- In `dtoro/caddy-conf`: point `mcp.hubris.network` and `oikos.hubris.network` to + mac-mini's mesh address at port 8090 (API) +- Future: `hermes.hubris.network` → mac-mini:8092 (Hermes gateway) +- DNS and public URLs unchanged +- Verify Caddy → mac-mini reachability over netbird first (same as original plan) + +### 10. mac-mini host setup + +**Directory:** +- New directory: `~/oikos-os/` (or `/opt/oikos-os/`) — the repo clone +- This is where `docker compose` runs from +- Existing `/opt/homelab-context/` stays untouched until cleanup phase + +**Prerequisites:** +- Docker Desktop (or OrbStack) — already installed +- SSH keys accessible at `~/.ssh/` (mounted into containers) +- Age key at `/etc/age/key.txt` (for one-time SOPS migration) + +**Cleanup (deferred, after Docker OS is proven):** +- Stop launchd git-sync for `/opt/homelab-context` +- Remove native Hermes install at `~/.hermes/hermes-agent/` +- Remove `/opt/homelab-context` clone +- Remove any launchd plists for homelab services + +### 11. Decommission apps/105 (deferred) + +Keep apps/105 running all Oikos services (homelab-mcp, secrets-issuance, +oikos-console) as a fallback until the Docker OS is proven end-to-end. + +**Cutover checklist (executed when ready):** +1. Verify Docker OS serves all traffic correctly +2. `systemctl disable --now homelab-mcp secrets-issuance oikos-console` on apps/105 +3. Remove `/opt/homelab-mcp`, `/opt/oikos-console` checkouts on apps/105 +4. Update Caddy backends to point to mac-mini exclusively +5. Remove Gitea webhooks 10, 11, 14 (or repoint to Docker deploy trigger) ## Reuse (do not reimplement) -- Deploy scaffolding shape: `mcp/deploy/deploy.sh`, `oikos/console/deploy/`. -- Service definitions to port: `mcp/deploy/homelab-mcp.service`, - `oikos/console/deploy/oikos-console.service`, `oikos/systemd/*`. -- Actuator primitives: `oikos_decide.classify` (`decide.py:64`), signal - transitions incl. unused `start_acting` (`signal.py:125`), `cmd_restart` + - `_record_change` (`bin/homelab:603,190`), `oikos_approve.request` - (`bin/homelab:1370`), `oikos_ledger.history` (`ledger.py:59`). -- Comment-preserving inventory + `.sops.yaml` edits + `sops updatekeys`: - `homelab client add --finalize-pubkey` path in `bin/homelab`. +| Existing code | What it becomes | +|---|---| +| `oikos/decide.py` | Risk classifier — adapted to read from DB, logic unchanged | +| `oikos/signal.py` | Signal engine — adapted to DB, lifecycle logic unchanged | +| `oikos/approve.py` | Approval engine — adapted to DB, Matrix delivery reused | +| `oikos/ledger.py` | Ledger — adapted to DB | +| `oikos/policy.py` + `policy.yaml` | Policy loader — unchanged, still reads YAML | +| `oikos/drift.py` | Drift detectors — adapted to write signals to DB | +| `oikos/relations.py` | Ontology graph walk — adapted to DB | +| `oikos/report.py` | Report generator — adapted to read from DB | +| `mcp/server.py` | Merged into `oikos/api/` — MCP protocol adapter on top of DB | +| `bin/homelab` logic | Migrated into API routes — same operations, REST interface | +| `oikos/scheduler.py` | Adapted — probes unchanged, output goes to DB | +| `oikos/approve.py` Matrix delivery | Reused as the Matrix notifier implementation | -## Risks / assumptions +## Risks / trade-offs -- **Caddy → mac-mini reachability** over netbird is unverified — workstream 4 - verifies before cutover; LAN 178.x is the fallback (hubris is dual-homed). -- **Lost sandboxing** on macOS (no `ProtectSystem`/`ReadOnlyPaths`) — mitigated - by a least-privilege `_oikos` daemon user, not eliminated. -- **secrets-issuance on a workstation** widens the security surface — accepted - per scope decision; documented. -- **Single-checkout race:** a git pull mid-request could serve half-updated code - — post-pull `kickstart -k` restarts services after the pull completes. -- macOS `/opt/homelab-context` already exists (mac-mini is enrolled); daemons - need it readable by `_oikos`. +- **Hermes in Docker** — the agent's "world" is the container. Direct file access + to the host is limited to mounted volumes. SSH access is the bridge to the homelab. + Trade-off: less host integration, more isolation. Mitigated by the hybrid approach + (SSH now, actuator gateway later). +- **PostgreSQL as single point of failure** — if Postgres goes down, the OS loses + all state. Mitigated by Docker volume persistence + regular `pg_dump` backups + (can be automated by the scheduler itself once running). +- **Infisical bootstrapping** — migrating from SOPS to Infisical has a transition + period where both systems coexist. Keep SOPS as fallback until all services are + verified against Infisical. +- **Knowledge graph ingestion quality** — markdown-to-structured-data conversion + relies on path conventions and frontmatter. Docs without frontmatter may lose + metadata. Start simple, improve extraction over time. +- **Deploy downtime** — `docker compose up -d` may briefly interrupt services during + container replacement. Use `--scale` + health-gated rollout for zero-downtime + when needed (later). +- **Multi-agent concurrency** — PostgreSQL handles concurrent writers correctly, + but the actuator needs a lock mechanism to prevent two passes from acting on + the same signal simultaneously. Use `SELECT ... FOR UPDATE SKIP LOCKED`. ## Verification (end to end) -1. **Runtime:** `install-macos.sh` on mac-mini; `launchctl print system/…` shows - all four daemons running; `curl 127.0.0.1:{9810,9820,8091}` responds locally. -2. **Ingress:** after the caddy-conf push + reload, `curl -I - https://mcp.hubris.network/mcp`, `.../` (console → 302 Authentik), - `secrets.hubris.network/health` → 200/302 through Caddy to mac-mini. -3. **Observe+Act loop:** trigger a `service-down` Signal; confirm the launchd - scheduler pass runs it `raised→acting→resolved`, writes a `restart`/`result: - ok` ledger entry, and pushes `signals/`+`ledger/`. Kill-switch: set - `autonomy.auto_act: off` → next pass escalates instead. -4. **Deploy-by-pull:** `git push` a trivial change; within 5 min mac-mini pulls - and `kickstart`s the changed daemon (no webhook involved). -5. **Cutover:** after decommission, `systemctl is-active` on apps shows the three - services `inactive`; production traffic served only by mac-mini; MCP tools + - console + a test enrollment all still work. +1. **Database:** `docker compose up postgres` — `psql` shows all tables; ingest + script populates entities from `inventory.yaml` and knowledge from `docs/`. +2. **API:** `curl http://localhost:8090/api/v1/hosts` returns the fleet; + MCP `list_services` tool works via the same endpoint. +3. **Scheduler:** trigger a probe pass — signals appear in DB, state snapshot written. +4. **Actuator:** raise a test `service-down` signal → actuator classifies → + auto-acts (restart) or escalates (Matrix approval request) → ledger entry written. +5. **Hermes:** connect from another workstation via `hermes gateway connect` → + agent responds, can query the API via MCP, can SSH to hubris. +6. **Secrets:** Infisical running, services fetch secrets at startup, SOPS files + removed without breakage. +7. **Deploy:** `git push` → Gitea webhook → `docker compose build + up -d` → + changes live within minutes. +8. **Knowledge:** MCP `search_knowledge("caddy")` returns relevant docs linked + to `entity:service:caddy`. +9. **Notifier:** raise a `critical` signal → Matrix message received → + ✅ reaction → approval granted in DB. +10. **Cutover:** after all above passes, stop services on apps/105, verify + production traffic served only by Docker OS on mac-mini. -## Out of scope (per prior direction) +## Phasing -SSH-signed approval requests, Authentik step-up on the console POST, Prometheus / -temperature trend signals, and the generalize-to-any-homelab refactor. +**Phase 1 — Foundation (DB + API):** +- PostgreSQL schema + models + migrations +- Ingest pipeline (inventory.yaml → DB, docs → knowledge graph) +- Unified API server (REST + MCP) reading from DB +- Migrate existing signals/ledger/approvals data into DB + +**Phase 2 — Services (Scheduler + Actuator + Notifier):** +- Adapt scheduler to read/write DB +- Build actuator (Act stage) with kill-switch +- Matrix notifier (abstracted interface + impl) +- SSH-based execution (hybrid approach) + +**Phase 3 — Agent (Hermes container):** +- Hermes Docker image +- Gateway mode config +- Homelab skills (homelab-ops) +- Connect from workstation, verify MCP + SSH access + +**Phase 4 — Secrets (Infisical):** +- Stand up Infisical in the stack +- Migrate SOPS secrets +- Wire all services to Infisical +- Retire SOPS + +**Phase 5 — Deploy pipeline:** +- Docker Compose fully working +- Gitea webhook → rebuild trigger +- Caddy ingress re-point to mac-mini +- Health checks + monitoring + +**Phase 6 — Cutover + cleanup:** +- Verify end-to-end +- Stop apps/105 services +- Clean up mac-mini old artifacts +- Remove old deploy webhooks + +## Out of scope (for now) + +- Oikos Console web UI (deferred — built later on top of the API for specific tasks) +- Multi-node deployment (designed for, not implemented) +- Vector embeddings / semantic search (knowledge graph is structured-only for now) +- SSH-key-signed approval requests (same backlog as before) +- Prometheus / trend signals +- Actuator gateway pattern (Phase 2 of the hybrid — start with mounted SSH) +- Custom homelab skills beyond `homelab-ops` (add as needed)