dtoro
bead722fac
Supersedes the launchd-based consolidation plan. Key changes: - Docker-based deployment on mac-mini (docker compose) - PostgreSQL for all mutable state (signals, ledger, knowledge graph) - Infisical replaces SOPS+age for secrets management - Unified API merges MCP server + homelab CLI (REST + MCP interfaces) - Hermes agent runs in Docker (gateway mode, connect from any workstation) - Knowledge graph in Postgres replaces narrative wiki files as agent context - Structured entity relationships link docs to inventory entities - Hybrid SSH access (mounted keys now, actuator gateway later) - Git push → Gitea webhook → Docker rebuild = deploy trigger - 6-phase rollout: DB → services → agent → secrets → deploy → cutover
690 lines
34 KiB
Markdown
690 lines
34 KiB
Markdown
# Plan: Oikos — Docker-based agentic homelab OS on mac-mini
|
|
|
|
**Status:** Planned (2026-07-06, revised) — supersedes the launchd-based consolidation plan.
|
|
**Predecessor:** The original plan re-platformed Oikos services from LXC 105 (apps) to
|
|
mac-mini via launchd plists. This revision pivots to a **Docker-based** architecture:
|
|
the repo becomes a containerized homelab OS, deployed via `docker compose` on mac-mini.
|
|
|
|
## Vision
|
|
|
|
Convert this repo into a **Docker-based agentic homelab OS**. The OS is a set of
|
|
containerized services that manage the homelab autonomously, with the operator in
|
|
control. Two actors:
|
|
|
|
- **Operator** (dtoro) — owns the homelab, expresses intent ("install X", "restart
|
|
Y"), approves destructive actions. Connects from any workstation via remote Hermes
|
|
or Matrix.
|
|
- **Agent** — Hermes core + custom homelab skills, running in Docker. Executes orders,
|
|
monitors the lab, escalates when unsure.
|
|
|
|
All OS services run in Docker containers on mac-mini. The OS is deployed by a git
|
|
push (Gitea webhook → Docker rebuild). It's designed for mac-mini now, with a path
|
|
to multi-node later.
|
|
|
|
## Decisions (from operator Q&A, 2026-07-06)
|
|
|
|
| Question | Decision |
|
|
|---|---|
|
|
| Repo structure | One repo, reorganize internally. Wiki docs get restructured into the knowledge graph. |
|
|
| Hermes runtime | Runs inside Docker as part of the OS stack (gateway mode — connect from any workstation). |
|
|
| Agent → homelab access | Hybrid — mounted SSH keys for now, build actuator gateway incrementally. |
|
|
| Data storage | Migrate to PostgreSQL. `inventory.yaml` stays as declarative config; signals/ledger/state/knowledge go into DB. |
|
|
| Knowledge/context | Structured knowledge graph in Postgres — docs become typed entities linked to inventory entities. |
|
|
| Operator interface | Primary = remote Hermes from any workstation + Matrix. Console/UIs built later for specific tasks. |
|
|
| Host | mac-mini for now, designed to scale later. |
|
|
| Deploy | Git push → Gitea webhook → Docker rebuild + restart. |
|
|
| `bin/homelab` CLI | Replaced by an API. CLI becomes a thin client that calls the OS API. |
|
|
| Secrets | Migrate from SOPS+age to Infisical (open-source, API-first, Docker-native). |
|
|
| Matrix | Keep for now, abstract the notification layer for future channels. |
|
|
| MCP server | Merged into the unified API — one service, REST + MCP interfaces. |
|
|
| Agent type | Hermes core + custom homelab skills (standard Hermes, purpose-loaded). |
|
|
| apps/105 | Keep running as fallback until Docker OS is proven. |
|
|
| mac-mini cleanup | Start fresh in a new directory, clean up old artifacts later. |
|
|
|
|
## Target architecture
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────────┐
|
|
│ mac-mini (Docker host, always-on) │
|
|
│ │
|
|
│ ┌──────────────┐ ┌────────────┐ ┌───────────────────────────┐ │
|
|
│ │ PostgreSQL │ │ Infisical │ │ Hermes Agent (gateway) │ │
|
|
│ │ │ │ (secrets) │ │ + homelab skills │ │
|
|
│ │ • inventory │ │ │ │ + MCP client → API │ │
|
|
│ │ • signals │ │ │ │ + SSH keys (mounted) │ │
|
|
│ │ • ledger │ │ │ │ │ │
|
|
│ │ • knowledge │ │ │ │ Any workstation connects │ │
|
|
│ │ • state │ │ │ │ via Hermes gateway protocol │ │
|
|
│ │ • approvals │ │ │ │ │ │
|
|
│ └──────┬────────┘ └─────────────┘ └──────────┬────────────────┘ │
|
|
│ │ │ │
|
|
│ ┌──────┴─────────────────────────────────────────┴──────────────┐ │
|
|
│ │ Oikos API (REST + MCP, one service) │ │
|
|
│ │ │ │
|
|
│ │ MCP interface (agent context): │ │
|
|
│ │ get_host, list_services, search_knowledge, get_entity, │ │
|
|
│ │ get_relations, get_change_history, get_state_snapshot │ │
|
|
│ │ │ │
|
|
│ │ REST interface (operations + operator): │ │
|
|
│ │ GET /api/v1/hosts, /services, /signals, /approvals │ │
|
|
│ │ POST /api/v1/exec — gated execution (actuator) │ │
|
|
│ │ POST /api/v1/approve — operator approval │ │
|
|
│ │ POST /api/v1/deploy — trigger service deploy │ │
|
|
│ │ WS /api/v1/events — real-time signal/approval stream │ │
|
|
│ │ │ │
|
|
│ │ Policy enforcement, risk classification, ledger recording │ │
|
|
│ └──────────────────────┬───────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────────────────┴───────────────────────────────────────────┐ │
|
|
│ │ Scheduler (Observe) + Actuator (Act) │ │
|
|
│ │ • Periodic probes (10-min) → signals in DB │ │
|
|
│ │ • Reads open signals, classifies via decide.py │ │
|
|
│ │ • Auto-acts (reversible_low) or escalates (approval) │ │
|
|
│ │ • SSH to hubris/strong for probes + execution │ │
|
|
│ └──────────────────────┬───────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────────────────┴───────────────────────────────────────────┐ │
|
|
│ │ Notifier (abstracted, pluggable) │ │
|
|
│ │ • Matrix (current) — alerts, approval requests │ │
|
|
│ │ • Future: webhook, email, Slack, etc. │ │
|
|
│ └──────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Deploy: Gitea webhook → docker compose build + up -d │
|
|
│ Volumes: pg-data, infisical-data, ssh-keys (ro mount), hermes-config │
|
|
│ Network: Docker bridge (internal) + host network (SSH outbound) │
|
|
└──────────────────────────────────────────────────────────────────────┘
|
|
|
|
External:
|
|
Caddy (LXC 121) → mac-mini mesh IP :8090 (API), future UIs
|
|
apps/105 → fallback, kept running until Docker OS proven
|
|
hubris/strong → SSH targets for scheduler/actuator
|
|
Any workstation → Hermes gateway (connects to agent in Docker)
|
|
Gitea (LXC 104) → webhook triggers Docker rebuild
|
|
```
|
|
|
|
## Repo reorganization
|
|
|
|
The repo shifts from "wiki + code mixed" to "OS code + declarative config + docs as
|
|
ingest source." Proposed layout:
|
|
|
|
```
|
|
/ # repo root
|
|
├── docker-compose.yml # the OS stack definition
|
|
├── compose/ # per-service Dockerfiles + build context
|
|
│ ├── api/ # Oikos API (REST + MCP) — merges mcp/ + bin/homelab
|
|
│ │ └── Dockerfile
|
|
│ ├── scheduler/ # Observe + Act loop
|
|
│ │ └── Dockerfile
|
|
│ ├── hermes/ # Hermes agent runtime
|
|
│ │ └── Dockerfile
|
|
│ ├── postgres/ # init scripts, migrations
|
|
│ │ └── init.sql
|
|
│ └── infisical/ # Infisical config
|
|
│ └── config.yaml
|
|
├── oikos/ # Core OS Python package (reused, adapted)
|
|
│ ├── api/ # NEW — unified API server
|
|
│ │ ├── server.py # FastAPI app, REST routes + MCP protocol adapter
|
|
│ │ ├── routes/ # REST route modules
|
|
│ │ └── mcp.py # MCP protocol adapter (exposes same tools via MCP)
|
|
│ ├── db/ # NEW — database layer
|
|
│ │ ├── models.py # SQLAlchemy models: Inventory, Signal, Ledger, Knowledge, Approval
|
|
│ │ ├── migrations/ # Alembic migrations
|
|
│ │ └── ingest.py # Ingest inventory.yaml + docs into DB on deploy
|
|
│ ├── scheduler/ # Adapted from scheduler.py — reads/writes DB
|
|
│ ├── actuator/ # NEW — Act stage: reads signals, classifies, executes
|
|
│ │ └── act.py
|
|
│ ├── policy/ # Adapted — policy.yaml loader + classifier
|
|
│ ├── knowledge/ # NEW — knowledge graph ingestion from docs
|
|
│ │ └── ingest_docs.py
|
|
│ ├── notifier/ # NEW — abstracted notification (Matrix impl)
|
|
│ │ ├── base.py # Notifier interface
|
|
│ │ └── matrix.py # Matrix implementation
|
|
│ ├── decide.py # Reused — risk classifier (reads from DB)
|
|
│ ├── signal.py # Adapted — signals in DB instead of JSONL files
|
|
│ ├── approve.py # Adapted — approvals in DB
|
|
│ ├── ledger.py # Adapted — ledger in DB
|
|
│ ├── drift.py # Adapted — drift detection, writes signals to DB
|
|
│ ├── relations.py # Reused — ontology graph walk (reads from DB)
|
|
│ └── report.py # Adapted — generates reports from DB
|
|
├── hermes/ # Hermes agent config + skills (mounted into container)
|
|
│ ├── config.yaml # Hermes config (providers, models, gateway)
|
|
│ ├── SOUL.md # Agent persona (homelab-specific)
|
|
│ └── skills/ # Custom homelab skills
|
|
│ └── homelab-ops/
|
|
│ └── SKILL.md
|
|
├── inventory.yaml # Declarative config — the "kernel data structure" (stays YAML)
|
|
├── ontology.yaml # Systems model (stays YAML)
|
|
├── policy.yaml # Risk & approval policy (stays YAML)
|
|
├── docs/ # Narrative docs — ingested into knowledge graph on deploy
|
|
│ ├── containers/ # LXC/VM documentation
|
|
│ ├── hosts/ # Host documentation
|
|
│ ├── infrastructure/ # Infrastructure docs
|
|
│ └── investigations/ # Incident records
|
|
├── secrets/ # SOPS-encrypted secrets (migrated to Infisical, then removed)
|
|
└── .sops.yaml # SOPS config (deprecated after Infisical migration)
|
|
```
|
|
|
|
**What stays YAML (declarative, human-edited, git-tracked):**
|
|
- `inventory.yaml` — topology, the kernel data structure
|
|
- `ontology.yaml` — entity types, relationships, lifecycle
|
|
- `policy.yaml` — risk classes, approval rules, autonomy settings
|
|
- `hermes/config.yaml` — agent configuration
|
|
- `hermes/SOUL.md` — agent persona
|
|
|
|
**What moves to PostgreSQL (mutable, runtime state):**
|
|
- `signals/` → `signals` table
|
|
- `ledger/` → `ledger_entries` table
|
|
- `oikos/state.json` → `state_snapshots` table
|
|
- `approvals/` → `approvals` table
|
|
- `knowledge/wiki/` narrative docs → `knowledge_entities` + `knowledge_relations` tables
|
|
|
|
**What gets removed (no longer needed):**
|
|
- `mcp/server.py` — merged into `oikos/api/server.py`
|
|
- `bin/homelab` — replaced by the API; thin CLI client can be generated later
|
|
- `secrets-issuance/` — replaced by Infisical
|
|
- `oikos/systemd/` — replaced by Docker containers
|
|
- `oikos/console/` — deferred (built later as a specific UI on top of the API)
|
|
- `mcp/deploy/`, `oikos/console/deploy/` — replaced by Docker build
|
|
- `hosts/*.yaml` — generated from inventory, now served by the API directly
|
|
|
|
## Docker Compose stack
|
|
|
|
```yaml
|
|
# docker-compose.yml (simplified — full version in the repo)
|
|
services:
|
|
postgres:
|
|
image: postgres:16-alpine
|
|
volumes: [pg-data:/var/lib/postgresql/data, ./compose/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql]
|
|
environment:
|
|
POSTGRES_DB: oikos
|
|
POSTGRES_PASSWORD_FILE: /run/secrets/pg_password
|
|
secrets: [pg_password]
|
|
healthcheck: ...
|
|
|
|
infisical:
|
|
image: infisical/infisical:latest
|
|
volumes: [infisical-data:/var/lib/infisical]
|
|
environment: ...
|
|
# Secrets manager — replaces SOPS+age for runtime secret access
|
|
|
|
api:
|
|
build: ./compose/api
|
|
depends_on: [postgres, infisical]
|
|
environment:
|
|
DATABASE_URL: postgresql://oikos@postgres/oikos
|
|
HOMELAB_CONTEXT_DIR: /opt/homelab-context
|
|
volumes:
|
|
- ./inventory.yaml:/app/inventory.yaml:ro
|
|
- ./ontology.yaml:/app/ontology.yaml:ro
|
|
- ./policy.yaml:/app/policy.yaml:ro
|
|
- ./docs:/app/docs:ro
|
|
ports: ["8090:8090"] # REST + MCP
|
|
# On startup: ingest inventory.yaml + docs into DB if changed
|
|
|
|
scheduler:
|
|
build: ./compose/scheduler
|
|
depends_on: [api, postgres]
|
|
environment:
|
|
DATABASE_URL: postgresql://oikos@postgres/oikos
|
|
SSH_KEY: /run/secrets/ssh_key
|
|
volumes:
|
|
- ssh-keys:/run/secrets:ro
|
|
# Runs observe + act loop on a cron-like schedule (10-min interval)
|
|
|
|
hermes:
|
|
build: ./compose/hermes
|
|
depends_on: [api, infisical]
|
|
environment:
|
|
OIKOS_API_URL: http://api:8090
|
|
INFISICAL_TOKEN: ...
|
|
volumes:
|
|
- ./hermes:/root/.hermes:ro
|
|
- ssh-keys:/root/.ssh:ro
|
|
- hermes-data:/root/.hermes/data
|
|
ports: ["8092:8092"] # Hermes gateway port
|
|
# Agent runtime — connects to API via MCP, has SSH for direct host access
|
|
|
|
notifier:
|
|
build: ./compose/scheduler # shares image with scheduler
|
|
depends_on: [api, postgres]
|
|
# Matrix bot — listens for approval requests, sends alerts
|
|
# Abstracted via notifier interface — Matrix impl now, pluggable later
|
|
|
|
volumes:
|
|
pg-data:
|
|
infisical-data:
|
|
ssh-keys: # populated from host ~/.ssh at deploy time
|
|
hermes-data:
|
|
```
|
|
|
|
## Workstreams
|
|
|
|
### 1. Database schema + models (`oikos/db/`, new)
|
|
|
|
The foundational layer — everything else reads/writes through this.
|
|
|
|
**Tables:**
|
|
|
|
```sql
|
|
-- Inventory entities (synced from inventory.yaml on deploy)
|
|
CREATE TABLE entities (
|
|
id TEXT PRIMARY KEY, -- "host:hubris", "service:caddy"
|
|
type TEXT NOT NULL, -- host, service, lxc, vm, etc.
|
|
name TEXT NOT NULL,
|
|
domain TEXT, -- physical, compute, network, etc.
|
|
state TEXT DEFAULT 'active', -- provisioning, active, deprecated, destroyed
|
|
data JSONB NOT NULL, -- full entity record from inventory.yaml
|
|
updated_at TIMESTAMPTZ DEFAULT now()
|
|
);
|
|
|
|
-- Entity relationships (the ontology graph)
|
|
CREATE TABLE relations (
|
|
source_id TEXT REFERENCES entities(id),
|
|
target_id TEXT REFERENCES entities(id),
|
|
type TEXT NOT NULL, -- hosts, provides, mounts, depends-on, etc.
|
|
data JSONB,
|
|
PRIMARY KEY (source_id, target_id, type)
|
|
);
|
|
|
|
-- Signals (replaces signals/*.jsonl)
|
|
CREATE TABLE signals (
|
|
id TEXT PRIMARY KEY, -- "sig-2026-07-06-0001"
|
|
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
kind TEXT NOT NULL, -- service-down, disk-threshold, drift, etc.
|
|
severity TEXT NOT NULL, -- info, warning, critical
|
|
entity_id TEXT REFERENCES entities(id),
|
|
evidence TEXT,
|
|
likely_cause TEXT,
|
|
recommended_action JSONB,
|
|
verification TEXT,
|
|
state TEXT NOT NULL DEFAULT 'raised', -- raised, acknowledged, acting, resolved, muted
|
|
mute_until TIMESTAMPTZ,
|
|
note TEXT,
|
|
updated_at TIMESTAMPTZ DEFAULT now()
|
|
);
|
|
|
|
-- Change ledger (replaces ledger/*.jsonl)
|
|
CREATE TABLE ledger_entries (
|
|
id SERIAL PRIMARY KEY,
|
|
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
entity_id TEXT REFERENCES entities(id),
|
|
action TEXT NOT NULL,
|
|
risk_class TEXT NOT NULL,
|
|
result TEXT, -- ok, failed, escalated
|
|
approval_id TEXT, -- references approvals if gated
|
|
agent TEXT, -- agent identity (age pubkey or hostname)
|
|
notes TEXT
|
|
);
|
|
|
|
-- Approvals (replaces approvals/*.jsonl)
|
|
CREATE TABLE approvals (
|
|
id TEXT PRIMARY KEY,
|
|
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
entity_id TEXT REFERENCES entities(id),
|
|
action TEXT NOT NULL,
|
|
risk_class TEXT NOT NULL,
|
|
status TEXT NOT NULL DEFAULT 'pending', -- pending, approved, denied, expired
|
|
ttl INTERVAL NOT NULL DEFAULT '1 hour',
|
|
decided_at TIMESTAMPTZ,
|
|
decided_by TEXT,
|
|
confirmation_phrase TEXT
|
|
);
|
|
|
|
-- State snapshots (replaces oikos/state.json)
|
|
CREATE TABLE state_snapshots (
|
|
id SERIAL PRIMARY KEY,
|
|
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
entity_id TEXT REFERENCES entities(id),
|
|
health TEXT, -- healthy, degraded, down, unknown
|
|
data JSONB -- full probe response
|
|
);
|
|
|
|
-- Knowledge graph (replaces narrative wiki docs)
|
|
CREATE TABLE knowledge_entities (
|
|
id TEXT PRIMARY KEY, -- "doc:105-apps", "investigation:2026-07-05-disk"
|
|
type TEXT NOT NULL, -- container-doc, host-doc, investigation, plan, runbook
|
|
entity_id TEXT REFERENCES entities(id), -- link to inventory entity if applicable
|
|
title TEXT NOT NULL,
|
|
content TEXT NOT NULL, -- markdown body
|
|
tags TEXT[],
|
|
source_path TEXT, -- original file path for traceability
|
|
updated_at TIMESTAMPTZ DEFAULT now()
|
|
);
|
|
|
|
-- Knowledge relationships (cross-references between docs + entities)
|
|
CREATE TABLE knowledge_relations (
|
|
source_id TEXT REFERENCES knowledge_entities(id),
|
|
target_id TEXT, -- can be knowledge_entity or inventory entity
|
|
type TEXT NOT NULL, -- describes, relates-to, runbook-for, etc.
|
|
PRIMARY KEY (source_id, target_id, type)
|
|
);
|
|
```
|
|
|
|
**Ingest pipeline** (`oikos/db/ingest.py`):
|
|
- On startup / deploy: parse `inventory.yaml` → upsert into `entities` + `relations`
|
|
- Walk `docs/` directory → parse markdown, extract frontmatter → upsert into `knowledge_entities`
|
|
- Link knowledge entities to inventory entities by path convention (`docs/containers/105-apps.md` → `entity:lxc:apps`)
|
|
- Idempotent — safe to re-run on every deploy
|
|
|
|
**Migration from existing data:**
|
|
- Write a one-time script to import existing `signals/*.jsonl`, `ledger/*.jsonl`, `approvals/*.jsonl` into the DB
|
|
- `oikos/state.json` → `state_snapshots` table
|
|
|
|
### 2. Unified API server (`oikos/api/`, new — merges MCP + homelab CLI)
|
|
|
|
One FastAPI service exposing two interfaces:
|
|
|
|
**MCP interface** (`oikos/api/mcp.py`):
|
|
- Same tool surface as the current `mcp/server.py` (get_host, list_services, search_knowledge, etc.)
|
|
- Reads from PostgreSQL instead of filesystem
|
|
- `search_docs` → `search_knowledge` (queries the knowledge graph)
|
|
- New tools: `get_entity_relations`, `get_signal_history`, `get_ledger`
|
|
- Protocol: FastMCP over streamable HTTP (same as current)
|
|
|
|
**REST interface** (`oikos/api/routes/`):
|
|
- `GET /api/v1/hosts` — list hosts (replaces `homelab list`)
|
|
- `GET /api/v1/services` — list services
|
|
- `GET /api/v1/hosts/{name}` — host detail (replaces `homelab status`)
|
|
- `GET /api/v1/services/{name}/health` — service health (cache-first, `?live=true` for probe)
|
|
- `GET /api/v1/signals` — list signals (`?state=raised&severity=warning`)
|
|
- `POST /api/v1/signals/{id}/ack` — acknowledge
|
|
- `POST /api/v1/signals/{id}/resolve` — resolve
|
|
- `GET /api/v1/approvals` — pending approvals
|
|
- `POST /api/v1/approvals/{id}/decide` — approve/deny (operator only, Authentik-gated)
|
|
- `POST /api/v1/exec` — gated execution (the actuator entrypoint)
|
|
- Body: `{entity, action, approval_id?}`
|
|
- Classifies via `decide.py`, checks approval if needed, executes, records ledger
|
|
- `POST /api/v1/deploy/{service}` — trigger a service deploy
|
|
- `WS /api/v1/events` — real-time stream (signals, approvals, ledger entries)
|
|
- `GET /api/v1/knowledge/{entity_id}` — knowledge graph query for an entity
|
|
- `GET /api/v1/knowledge/search?q=...` — search knowledge graph
|
|
|
|
**Policy enforcement** is built into the API:
|
|
- Every mutating endpoint classifies the action via `oikos/decide.py`
|
|
- `config_mutation`/`destructive` actions require a valid approval ID
|
|
- All mutations write to the ledger automatically
|
|
|
|
**Auth:**
|
|
- MCP interface: no auth (internal, container-to-container)
|
|
- REST interface: Authentik OIDC forward-auth (via Caddy) for operator endpoints
|
|
- Internal service-to-service: shared secret / mTLS (Docker network)
|
|
|
|
### 3. Scheduler + Actuator (`oikos/scheduler/`, `oikos/actuator/`)
|
|
|
|
**Scheduler (Observe)** — adapted from `oikos/scheduler.py`:
|
|
- Runs as a Docker service on a timer (10-min interval via a loop or cron sidecar)
|
|
- Probes service health (HTTP), disk usage (SSH to hubris/strong), drift detection
|
|
- Writes signals to PostgreSQL (not JSONL files)
|
|
- Writes state snapshots to DB
|
|
- No more git commit/push of signals — they're in the DB
|
|
|
|
**Actuator (Act)** — new, `oikos/actuator/act.py`:
|
|
- Reads open signals from DB where `recommended_action` is set
|
|
- For each: classify via `decide.py`
|
|
- `auto-act` (reversible_low, contained, confident) → execute via SSH/API, verify, resolve signal, write ledger
|
|
- `escalate` → create approval request, notify via Matrix, acknowledge signal
|
|
- Loop-guard: check ledger history per (entity, action) to cap auto-retries
|
|
- Autonomy kill-switch: `policy.yaml` → `autonomy.auto_act: off|reversible_low`, `never_auto_act: [entity list]`
|
|
- **Phase 1**: direct SSH execution (keys mounted)
|
|
- **Phase 2**: execution through the API's `/exec` endpoint (gateway pattern)
|
|
|
|
### 4. Hermes agent container (`compose/hermes/`)
|
|
|
|
**What it is:**
|
|
- Hermes Agent runtime in a Docker container
|
|
- Runs in gateway mode — workstations connect to it remotely
|
|
- Config: `hermes/config.yaml` (providers, models, gateway port)
|
|
- Persona: `hermes/SOUL.md` (homelab-specific)
|
|
- Skills: `hermes/skills/homelab-ops/SKILL.md` (how to interact with the homelab)
|
|
|
|
**What it has access to:**
|
|
- **Oikos API** via MCP (container network, no auth needed)
|
|
- **Infisical** for secret retrieval (API token)
|
|
- **SSH keys** mounted from host (`~/.ssh` → `/root/.ssh:ro`) — for direct host access (hybrid approach)
|
|
- **Hermes data volume** — persistent state, session history
|
|
|
|
**How workstations connect:**
|
|
- Hermes gateway listens on port 8092
|
|
- Any workstation with Hermes CLI can connect: `hermes gateway connect mac-mini:8092`
|
|
- Or via Caddy ingress: `hermes.hubris.network` → mac-mini:8092 (mesh)
|
|
|
|
**Homelab skills** (loaded into Hermes):
|
|
- `homelab-ops` — how to use the Oikos API, classify actions, request approvals
|
|
- `homelab-deploy` — how to deploy/manage services
|
|
- `homelab-troubleshoot` — diagnostic procedures
|
|
- (Future skills added as needed)
|
|
|
|
### 5. Infisical secrets migration
|
|
|
|
**Phase 1 — Stand up Infisical:**
|
|
- Add Infisical to the Docker stack
|
|
- Configure with PostgreSQL backend (shared with Oikos DB or separate DB)
|
|
- Bootstrap admin credentials, create `hubris` project + environments (prod, staging)
|
|
|
|
**Phase 2 — Migrate secrets:**
|
|
- Write a migration script: read all `secrets/*.yaml` (SOPS-encrypted), decrypt with age key, import into Infisical
|
|
- Map each secret to Infisical's key-value structure
|
|
- Verify all secrets are imported
|
|
|
|
**Phase 3 — Wire services:**
|
|
- All OS services authenticate to Infisical via machine identity (service token)
|
|
- Services fetch secrets at startup or on-demand
|
|
- Remove `secrets/` directory and `.sops.yaml` once verified
|
|
- The age key on mac-mini stays for one-time decryption during migration, then is retired
|
|
|
|
**Phase 4 — Retire SOPS:**
|
|
- Remove SOPS dependencies from the repo
|
|
- Update inventory to reflect Infisical as the secret store
|
|
- Document the new secret access pattern
|
|
|
|
### 6. Knowledge graph ingestion (`oikos/knowledge/`)
|
|
|
|
**What it does:**
|
|
- On deploy, walks `docs/` directory
|
|
- Parses each markdown file:
|
|
- Extracts frontmatter (if any) for metadata
|
|
- Infers entity relationships from path conventions:
|
|
- `docs/containers/105-apps.md` → links to `entity:lxc:apps`
|
|
- `docs/hosts/hubris.md` → links to `entity:host:hubris`
|
|
- `docs/infrastructure/dns.md` → links to `entity:service:dns`
|
|
- Extracts cross-references (links to other docs) → `knowledge_relations`
|
|
- Tags by directory: container-doc, host-doc, infrastructure, investigation, plan
|
|
- Upserts into `knowledge_entities` + `knowledge_relations` tables
|
|
|
|
**Agent access:**
|
|
- MCP tool `search_knowledge(query)` — full-text search on knowledge graph
|
|
- MCP tool `get_entity_knowledge(entity_id)` — all docs related to an entity
|
|
- MCP tool `get_relations(entity_id)` — ontology blast-radius query (existing, adapted to DB)
|
|
|
|
**Docs as source, DB as query layer:**
|
|
- Humans still write markdown docs in `docs/`
|
|
- The DB is the query/index layer — agents never grep files
|
|
- On deploy, docs are re-ingested (idempotent)
|
|
- Future: wiki-style editing UI that writes back to `docs/` via git
|
|
|
|
### 7. Notifier abstraction (`oikos/notifier/`)
|
|
|
|
**Interface:**
|
|
```python
|
|
class Notifier(Protocol):
|
|
def send_alert(self, signal: Signal) -> None: ...
|
|
def send_approval_request(self, approval: Approval) -> None: ...
|
|
def listen_for_decisions(self) -> Iterator[ApprovalDecision]: ...
|
|
```
|
|
|
|
**Matrix implementation** (current):
|
|
- Sends alerts to `@dtoro:avispero` via Synapse (LXC 118)
|
|
- Approval requests as messages with ✅/❌ reactions
|
|
- Listens for reactions to record decisions
|
|
- Reuses existing `oikos/approve.py` Matrix delivery logic
|
|
|
|
**Future implementations** (pluggable):
|
|
- Webhook, email, Slack, etc.
|
|
- Registered in config, selected at runtime
|
|
|
|
### 8. Docker build + deploy pipeline
|
|
|
|
**Build:**
|
|
- `docker compose build` from the repo root
|
|
- Each service has a Dockerfile in `compose/<service>/`
|
|
- Shared base image: Python 3.12-slim + common deps (postgresql libs, etc.)
|
|
- Multi-stage builds for smaller images
|
|
|
|
**Deploy trigger:**
|
|
- Gitea webhook on push to `main` → hits a deploy endpoint on mac-mini
|
|
- Deploy script: `git pull && docker compose build && docker compose up -d`
|
|
- Alternatively: a "deployer" sidecar container watches Gitea and triggers rebuilds
|
|
- The existing 5-min launchd git pull can serve as a fallback deploy mechanism initially
|
|
|
|
**Health checks:**
|
|
- Each service has a Docker healthcheck
|
|
- `docker compose ps` shows health status
|
|
- API exposes `/health` for external monitoring
|
|
|
|
### 9. Ingress re-point (Caddy)
|
|
|
|
- In `dtoro/caddy-conf`: point `mcp.hubris.network` and `oikos.hubris.network` to
|
|
mac-mini's mesh address at port 8090 (API)
|
|
- Future: `hermes.hubris.network` → mac-mini:8092 (Hermes gateway)
|
|
- DNS and public URLs unchanged
|
|
- Verify Caddy → mac-mini reachability over netbird first (same as original plan)
|
|
|
|
### 10. mac-mini host setup
|
|
|
|
**Directory:**
|
|
- New directory: `~/oikos-os/` (or `/opt/oikos-os/`) — the repo clone
|
|
- This is where `docker compose` runs from
|
|
- Existing `/opt/homelab-context/` stays untouched until cleanup phase
|
|
|
|
**Prerequisites:**
|
|
- Docker Desktop (or OrbStack) — already installed
|
|
- SSH keys accessible at `~/.ssh/` (mounted into containers)
|
|
- Age key at `/etc/age/key.txt` (for one-time SOPS migration)
|
|
|
|
**Cleanup (deferred, after Docker OS is proven):**
|
|
- Stop launchd git-sync for `/opt/homelab-context`
|
|
- Remove native Hermes install at `~/.hermes/hermes-agent/`
|
|
- Remove `/opt/homelab-context` clone
|
|
- Remove any launchd plists for homelab services
|
|
|
|
### 11. Decommission apps/105 (deferred)
|
|
|
|
Keep apps/105 running all Oikos services (homelab-mcp, secrets-issuance,
|
|
oikos-console) as a fallback until the Docker OS is proven end-to-end.
|
|
|
|
**Cutover checklist (executed when ready):**
|
|
1. Verify Docker OS serves all traffic correctly
|
|
2. `systemctl disable --now homelab-mcp secrets-issuance oikos-console` on apps/105
|
|
3. Remove `/opt/homelab-mcp`, `/opt/oikos-console` checkouts on apps/105
|
|
4. Update Caddy backends to point to mac-mini exclusively
|
|
5. Remove Gitea webhooks 10, 11, 14 (or repoint to Docker deploy trigger)
|
|
|
|
## Reuse (do not reimplement)
|
|
|
|
| Existing code | What it becomes |
|
|
|---|---|
|
|
| `oikos/decide.py` | Risk classifier — adapted to read from DB, logic unchanged |
|
|
| `oikos/signal.py` | Signal engine — adapted to DB, lifecycle logic unchanged |
|
|
| `oikos/approve.py` | Approval engine — adapted to DB, Matrix delivery reused |
|
|
| `oikos/ledger.py` | Ledger — adapted to DB |
|
|
| `oikos/policy.py` + `policy.yaml` | Policy loader — unchanged, still reads YAML |
|
|
| `oikos/drift.py` | Drift detectors — adapted to write signals to DB |
|
|
| `oikos/relations.py` | Ontology graph walk — adapted to DB |
|
|
| `oikos/report.py` | Report generator — adapted to read from DB |
|
|
| `mcp/server.py` | Merged into `oikos/api/` — MCP protocol adapter on top of DB |
|
|
| `bin/homelab` logic | Migrated into API routes — same operations, REST interface |
|
|
| `oikos/scheduler.py` | Adapted — probes unchanged, output goes to DB |
|
|
| `oikos/approve.py` Matrix delivery | Reused as the Matrix notifier implementation |
|
|
|
|
## Risks / trade-offs
|
|
|
|
- **Hermes in Docker** — the agent's "world" is the container. Direct file access
|
|
to the host is limited to mounted volumes. SSH access is the bridge to the homelab.
|
|
Trade-off: less host integration, more isolation. Mitigated by the hybrid approach
|
|
(SSH now, actuator gateway later).
|
|
- **PostgreSQL as single point of failure** — if Postgres goes down, the OS loses
|
|
all state. Mitigated by Docker volume persistence + regular `pg_dump` backups
|
|
(can be automated by the scheduler itself once running).
|
|
- **Infisical bootstrapping** — migrating from SOPS to Infisical has a transition
|
|
period where both systems coexist. Keep SOPS as fallback until all services are
|
|
verified against Infisical.
|
|
- **Knowledge graph ingestion quality** — markdown-to-structured-data conversion
|
|
relies on path conventions and frontmatter. Docs without frontmatter may lose
|
|
metadata. Start simple, improve extraction over time.
|
|
- **Deploy downtime** — `docker compose up -d` may briefly interrupt services during
|
|
container replacement. Use `--scale` + health-gated rollout for zero-downtime
|
|
when needed (later).
|
|
- **Multi-agent concurrency** — PostgreSQL handles concurrent writers correctly,
|
|
but the actuator needs a lock mechanism to prevent two passes from acting on
|
|
the same signal simultaneously. Use `SELECT ... FOR UPDATE SKIP LOCKED`.
|
|
|
|
## Verification (end to end)
|
|
|
|
1. **Database:** `docker compose up postgres` — `psql` shows all tables; ingest
|
|
script populates entities from `inventory.yaml` and knowledge from `docs/`.
|
|
2. **API:** `curl http://localhost:8090/api/v1/hosts` returns the fleet;
|
|
MCP `list_services` tool works via the same endpoint.
|
|
3. **Scheduler:** trigger a probe pass — signals appear in DB, state snapshot written.
|
|
4. **Actuator:** raise a test `service-down` signal → actuator classifies →
|
|
auto-acts (restart) or escalates (Matrix approval request) → ledger entry written.
|
|
5. **Hermes:** connect from another workstation via `hermes gateway connect` →
|
|
agent responds, can query the API via MCP, can SSH to hubris.
|
|
6. **Secrets:** Infisical running, services fetch secrets at startup, SOPS files
|
|
removed without breakage.
|
|
7. **Deploy:** `git push` → Gitea webhook → `docker compose build + up -d` →
|
|
changes live within minutes.
|
|
8. **Knowledge:** MCP `search_knowledge("caddy")` returns relevant docs linked
|
|
to `entity:service:caddy`.
|
|
9. **Notifier:** raise a `critical` signal → Matrix message received →
|
|
✅ reaction → approval granted in DB.
|
|
10. **Cutover:** after all above passes, stop services on apps/105, verify
|
|
production traffic served only by Docker OS on mac-mini.
|
|
|
|
## Phasing
|
|
|
|
**Phase 1 — Foundation (DB + API):**
|
|
- PostgreSQL schema + models + migrations
|
|
- Ingest pipeline (inventory.yaml → DB, docs → knowledge graph)
|
|
- Unified API server (REST + MCP) reading from DB
|
|
- Migrate existing signals/ledger/approvals data into DB
|
|
|
|
**Phase 2 — Services (Scheduler + Actuator + Notifier):**
|
|
- Adapt scheduler to read/write DB
|
|
- Build actuator (Act stage) with kill-switch
|
|
- Matrix notifier (abstracted interface + impl)
|
|
- SSH-based execution (hybrid approach)
|
|
|
|
**Phase 3 — Agent (Hermes container):**
|
|
- Hermes Docker image
|
|
- Gateway mode config
|
|
- Homelab skills (homelab-ops)
|
|
- Connect from workstation, verify MCP + SSH access
|
|
|
|
**Phase 4 — Secrets (Infisical):**
|
|
- Stand up Infisical in the stack
|
|
- Migrate SOPS secrets
|
|
- Wire all services to Infisical
|
|
- Retire SOPS
|
|
|
|
**Phase 5 — Deploy pipeline:**
|
|
- Docker Compose fully working
|
|
- Gitea webhook → rebuild trigger
|
|
- Caddy ingress re-point to mac-mini
|
|
- Health checks + monitoring
|
|
|
|
**Phase 6 — Cutover + cleanup:**
|
|
- Verify end-to-end
|
|
- Stop apps/105 services
|
|
- Clean up mac-mini old artifacts
|
|
- Remove old deploy webhooks
|
|
|
|
## Out of scope (for now)
|
|
|
|
- Oikos Console web UI (deferred — built later on top of the API for specific tasks)
|
|
- Multi-node deployment (designed for, not implemented)
|
|
- Vector embeddings / semantic search (knowledge graph is structured-only for now)
|
|
- SSH-key-signed approval requests (same backlog as before)
|
|
- Prometheus / trend signals
|
|
- Actuator gateway pattern (Phase 2 of the hybrid — start with mounted SSH)
|
|
- Custom homelab skills beyond `homelab-ops` (add as needed)
|