oikos/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md

# Plan: Oikos — Docker-based agentic homelab OS on mac-mini

**Status:** Planned (2026-07-06, revised) — supersedes the launchd-based consolidation plan.
**Predecessor:** The original plan re-platformed Oikos services from LXC 105 (apps) to
mac-mini via launchd plists. This revision pivots to a **Docker-based** architecture:
the repo becomes a containerized homelab OS, deployed via `docker compose` on mac-mini.

## Vision

Convert this repo into a **Docker-based agentic homelab OS**. The OS is a set of
containerized services that manage the homelab autonomously, with the operator in
control. Two actors:

- **Operator** (dtoro) — owns the homelab, expresses intent ("install X", "restart
  Y"), approves destructive actions. Connects from any workstation via remote Hermes
  or Matrix.
- **Agent** — Hermes core + custom homelab skills, running in Docker. Executes orders,
  monitors the lab, escalates when unsure.

All OS services run in Docker containers on mac-mini. The OS is deployed by a git
push (Gitea webhook → Docker rebuild). It's designed for mac-mini now, with a path
to multi-node later.

## Decisions (from operator Q&A, 2026-07-06)

| Question | Decision |
|---|---|
| Repo structure | One repo, reorganize internally. Wiki docs get restructured into the knowledge graph. |
| Hermes runtime | Runs inside Docker as part of the OS stack (gateway mode — connect from any workstation). |
| Agent → homelab access | Hybrid — mounted SSH keys for now, build actuator gateway incrementally. |
| Data storage | Migrate to PostgreSQL. `inventory.yaml` stays as declarative config; signals/ledger/state/knowledge go into DB. |
| Knowledge/context | Structured knowledge graph in Postgres — docs become typed entities linked to inventory entities. |
| Operator interface | Primary = remote Hermes from any workstation + Matrix. Console/UIs built later for specific tasks. |
| Host | mac-mini for now, designed to scale later. |
| Deploy | Git push → Gitea webhook → Docker rebuild + restart. |
| `bin/homelab` CLI | Replaced by an API. CLI becomes a thin client that calls the OS API. |
| Secrets | Migrate from SOPS+age to Infisical (open-source, API-first, Docker-native). |
| Matrix | Keep for now, abstract the notification layer for future channels. |
| MCP server | Merged into the unified API — one service, REST + MCP interfaces. |
| Agent type | Hermes core + custom homelab skills (standard Hermes, purpose-loaded). |
| apps/105 | Keep running as fallback until Docker OS is proven. |
| mac-mini cleanup | Start fresh in a new directory, clean up old artifacts later. |

## Target architecture

### Container stack on mac-mini

```mermaid
graph TB
    subgraph mac-mini["mac-mini — Docker host, always-on"]
        subgraph services["Docker Compose"]
            PG["PostgreSQL\n• inventory (synced from YAML)\n• signals • ledger\n• knowledge graph\n• state snapshots\n• approvals"]
            INF["Infisical\n(secrets manager)"]
            HERMES["Hermes Agent — gateway mode\n+ homelab skills\n+ MCP client → API\n+ SSH keys (mounted)"]
            API["Oikos API — REST + MCP\n\nMCP: get_host, list_services,\nsearch_knowledge, get_relations\nREST: /hosts, /services, /signals,\n/exec, /approve, /deploy, /events\n\nPolicy enforcement + risk\nclassification + ledger"]
            SCHED["Scheduler (Observe)\n+ Actuator (Act)\n• 10-min probes → DB\n• Classify → auto-act or escalate\n• SSH to hubris/strong"]
            NOTIFIER["Notifier — abstracted\n• Matrix (current): alerts,\napproval requests\n• Future: webhook, email, etc."]
        end
        DEPLOY["Gitea webhook →\ndocker compose build + up -d"]
        VOLS["Volumes: pg-data, infisical-data,\nssh-keys (ro), hermes-config"]
    end

    HERMES -- MCP --> API
    API --> PG
    SCHED --> PG
    SCHED -- SSH --> HUBRIS
    SCHED -- SSH --> STRONG
    API -- "escalate" --> NOTIFIER
    NOTIFIER -- "alerts + approvals" --> MATRIX
    DEPLOY -- "rebuild" --> services

    subgraph external["External"]
        CADDY["Caddy (LXC 121)\n→ mac-mini mesh :8090"]
        APPS["apps/105 (fallback)\nkept running until proven"]
        HUBRIS["hubris (PVE)"]
        STRONG["strong (PVE)"]
        GITEA["Gitea (LXC 104)\nwebhook trigger"]
        MATRIX["Matrix (LXC 118)"]
        WS["Any workstation\nHermes remote → gateway"]
    end

    CADDY -- reverse_proxy --> API
    GITEA -- webhook --> DEPLOY
    WS -- "Hermes gateway" --> HERMES
    APPS -. "fallback" .- API
```

### OODA loop — the control flow

```mermaid
flowchart LR
    OBSERVE["Observe\nScheduler probes:\n• HTTP health\n• disk usage\n• drift detection"] --> ORIENT["Orient\nRelations graph walk:\n• blast radius\n• lifecycle state\n• runbook match"]
    ORIENT --> DECIDE["Decide\nRisk classifier:\nrisk × blast × confidence"]
    DECIDE -- "auto-act\n(reversible_low,\ncontained, confident)" --> ACT["Act\nExecute via SSH/API\n→ verify → resolve\n→ ledger entry"]
    DECIDE -- "escalate\n(approval needed)" --> APPROVE["Approval request\n→ Matrix ✅/❌\n→ grant or deny"]
    APPROVE -- "approved" --> ACT
    APPROVE -- "denied" --> RESOLVE["Resolve signal\nnote: denied"]
    ACT --> OBSERVE
```

### Knowledge graph — docs to DB

```mermaid
flowchart TB
    subgraph repo["Repo (git-tracked)"]
        INV["inventory.yaml\ndeclarative topology"]
        DOCS["docs/\n• containers/105-apps.md\n• hosts/hubris.md\n• infrastructure/dns.md\n• investigations/..."]
    end
    subgraph db["PostgreSQL"]
        ENT["entities table\n(host:hubris, service:caddy, ...)"]
        REL["relations table\n(hosts, provides, depends-on, ...)"]
        KNOW["knowledge_entities table\n(doc:105-apps, doc:hubris, ...)"]
        KNOWREL["knowledge_relations table\n(describes, relates-to, runbook-for)"]
    end
    INGEST["Ingest pipeline\n(runs on deploy)"]
    INV --> INGEST --> ENT
    INV --> INGEST --> REL
    DOCS --> INGEST --> KNOW
    DOCS --> INGEST --> KNOWREL
    KNOW -. "linked by entity_id" .- ENT
```

### Deploy flow

```mermaid
flowchart LR
    DEV["Operator\nedits repo"] --> PUSH["git push"] --> GITEA["Gitea\n(LXC 104)"]
    GITEA -- "webhook" --> MACMINI["mac-mini\ndeploy script"]
    MACMINI -- "git pull" --> REPO["repo clone"]
    MACMINI -- "docker compose\nbuild + up -d" --> STACK["OS containers\nrebuilt + restarted"]
    STACK -- "ingest" --> DB["PostgreSQL\nupdated"]
```

## Repo reorganization

The repo shifts from "wiki + code mixed" to "OS code + declarative config + docs as
ingest source." Proposed layout:

```
/                        # repo root
├── docker-compose.yml   # the OS stack definition
├── compose/             # per-service Dockerfiles + build context
│   ├── api/             # Oikos API (REST + MCP) — merges mcp/ + bin/homelab
│   │   └── Dockerfile
│   ├── scheduler/       # Observe + Act loop
│   │   └── Dockerfile
│   ├── hermes/          # Hermes agent runtime
│   │   └── Dockerfile
│   ├── postgres/        # init scripts, migrations
│   │   └── init.sql
│   └── infisical/       # Infisical config
│       └── config.yaml
├── oikos/               # Core OS Python package (reused, adapted)
│   ├── api/             # NEW — unified API server
│   │   ├── server.py    # FastAPI app, REST routes + MCP protocol adapter
│   │   ├── routes/      # REST route modules
│   │   └── mcp.py       # MCP protocol adapter (exposes same tools via MCP)
│   ├── db/              # NEW — database layer
│   │   ├── models.py    # SQLAlchemy models: Inventory, Signal, Ledger, Knowledge, Approval
│   │   ├── migrations/  # Alembic migrations
│   │   └── ingest.py    # Ingest inventory.yaml + docs into DB on deploy
│   ├── scheduler/       # Adapted from scheduler.py — reads/writes DB
│   ├── actuator/        # NEW — Act stage: reads signals, classifies, executes
│   │   └── act.py
│   ├── policy/          # Adapted — policy.yaml loader + classifier
│   ├── knowledge/       # NEW — knowledge graph ingestion from docs
│   │   └── ingest_docs.py
│   ├── notifier/        # NEW — abstracted notification (Matrix impl)
│   │   ├── base.py      # Notifier interface
│   │   └── matrix.py    # Matrix implementation
│   ├── decide.py        # Reused — risk classifier (reads from DB)
│   ├── signal.py        # Adapted — signals in DB instead of JSONL files
│   ├── approve.py       # Adapted — approvals in DB
│   ├── ledger.py        # Adapted — ledger in DB
│   ├── drift.py         # Adapted — drift detection, writes signals to DB
│   ├── relations.py     # Reused — ontology graph walk (reads from DB)
│   └── report.py        # Adapted — generates reports from DB
├── hermes/              # Hermes agent config + skills (mounted into container)
│   ├── config.yaml      # Hermes config (providers, models, gateway)
│   ├── SOUL.md          # Agent persona (homelab-specific)
│   └── skills/          # Custom homelab skills
│       └── homelab-ops/
│           └── SKILL.md
├── inventory.yaml       # Declarative config — the "kernel data structure" (stays YAML)
├── ontology.yaml        # Systems model (stays YAML)
├── policy.yaml          # Risk & approval policy (stays YAML)
├── docs/                # Narrative docs — ingested into knowledge graph on deploy
│   ├── containers/      # LXC/VM documentation
│   ├── hosts/           # Host documentation
│   ├── infrastructure/  # Infrastructure docs
│   └── investigations/  # Incident records
├── secrets/             # SOPS-encrypted secrets (migrated to Infisical, then removed)
└── .sops.yaml           # SOPS config (deprecated after Infisical migration)
```

**What stays YAML (declarative, human-edited, git-tracked):**
- `inventory.yaml` — topology, the kernel data structure
- `ontology.yaml` — entity types, relationships, lifecycle
- `policy.yaml` — risk classes, approval rules, autonomy settings
- `hermes/config.yaml` — agent configuration
- `hermes/SOUL.md` — agent persona

**What moves to PostgreSQL (mutable, runtime state):**
- `signals/` → `signals` table
- `ledger/` → `ledger_entries` table
- `oikos/state.json` → `state_snapshots` table
- `approvals/` → `approvals` table
- `knowledge/wiki/` narrative docs → `knowledge_entities` + `knowledge_relations` tables

**What gets removed (no longer needed):**
- `mcp/server.py` — merged into `oikos/api/server.py`
- `bin/homelab` — replaced by the API; thin CLI client can be generated later
- `secrets-issuance/` — replaced by Infisical
- `oikos/systemd/` — replaced by Docker containers
- `oikos/console/` — deferred (built later as a specific UI on top of the API)
- `mcp/deploy/`, `oikos/console/deploy/` — replaced by Docker build
- `hosts/*.yaml` — generated from inventory, now served by the API directly

## Docker Compose stack

```yaml
# docker-compose.yml (simplified — full version in the repo)
services:
  postgres:
    image: postgres:16-alpine
    volumes: [pg-data:/var/lib/postgresql/data, ./compose/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql]
    environment:
      POSTGRES_DB: oikos
      POSTGRES_PASSWORD_FILE: /run/secrets/pg_password
    secrets: [pg_password]
    healthcheck: ...

  infisical:
    image: infisical/infisical:latest
    volumes: [infisical-data:/var/lib/infisical]
    environment: ...
    # Secrets manager — replaces SOPS+age for runtime secret access

  api:
    build: ./compose/api
    depends_on: [postgres, infisical]
    environment:
      DATABASE_URL: postgresql://oikos@postgres/oikos
      HOMELAB_CONTEXT_DIR: /opt/homelab-context
    volumes:
      - ./inventory.yaml:/app/inventory.yaml:ro
      - ./ontology.yaml:/app/ontology.yaml:ro
      - ./policy.yaml:/app/policy.yaml:ro
      - ./docs:/app/docs:ro
    ports: ["8090:8090"]   # REST + MCP
    # On startup: ingest inventory.yaml + docs into DB if changed

  scheduler:
    build: ./compose/scheduler
    depends_on: [api, postgres]
    environment:
      DATABASE_URL: postgresql://oikos@postgres/oikos
      SSH_KEY: /run/secrets/ssh_key
    volumes:
      - ssh-keys:/run/secrets:ro
    # Runs observe + act loop on a cron-like schedule (10-min interval)

  hermes:
    build: ./compose/hermes
    depends_on: [api, infisical]
    environment:
      OIKOS_API_URL: http://api:8090
      INFISICAL_TOKEN: ...
    volumes:
      - ./hermes:/root/.hermes:ro
      - ssh-keys:/root/.ssh:ro
      - hermes-data:/root/.hermes/data
    ports: ["8092:8092"]   # Hermes gateway port
    # Agent runtime — connects to API via MCP, has SSH for direct host access

  notifier:
    build: ./compose/scheduler  # shares image with scheduler
    depends_on: [api, postgres]
    # Matrix bot — listens for approval requests, sends alerts
    # Abstracted via notifier interface — Matrix impl now, pluggable later

volumes:
  pg-data:
  infisical-data:
  ssh-keys:    # populated from host ~/.ssh at deploy time
  hermes-data:
```

## Workstreams

### 1. Database schema + models (`oikos/db/`, new)

The foundational layer — everything else reads/writes through this.

**Tables:**

```sql
-- Inventory entities (synced from inventory.yaml on deploy)
CREATE TABLE entities (
    id          TEXT PRIMARY KEY,      -- "host:hubris", "service:caddy"
    type        TEXT NOT NULL,         -- host, service, lxc, vm, etc.
    name        TEXT NOT NULL,
    domain      TEXT,                  -- physical, compute, network, etc.
    state       TEXT DEFAULT 'active', -- provisioning, active, deprecated, destroyed
    data        JSONB NOT NULL,        -- full entity record from inventory.yaml
    updated_at  TIMESTAMPTZ DEFAULT now()
);

-- Entity relationships (the ontology graph)
CREATE TABLE relations (
    source_id   TEXT REFERENCES entities(id),
    target_id   TEXT REFERENCES entities(id),
    type        TEXT NOT NULL,         -- hosts, provides, mounts, depends-on, etc.
    data        JSONB,
    PRIMARY KEY (source_id, target_id, type)
);

-- Signals (replaces signals/*.jsonl)
CREATE TABLE signals (
    id          TEXT PRIMARY KEY,      -- "sig-2026-07-06-0001"
    ts          TIMESTAMPTZ NOT NULL DEFAULT now(),
    kind        TEXT NOT NULL,         -- service-down, disk-threshold, drift, etc.
    severity    TEXT NOT NULL,         -- info, warning, critical
    entity_id   TEXT REFERENCES entities(id),
    evidence    TEXT,
    likely_cause TEXT,
    recommended_action JSONB,
    verification TEXT,
    state       TEXT NOT NULL DEFAULT 'raised',  -- raised, acknowledged, acting, resolved, muted
    mute_until  TIMESTAMPTZ,
    note        TEXT,
    updated_at  TIMESTAMPTZ DEFAULT now()
);

-- Change ledger (replaces ledger/*.jsonl)
CREATE TABLE ledger_entries (
    id          SERIAL PRIMARY KEY,
    ts          TIMESTAMPTZ NOT NULL DEFAULT now(),
    entity_id   TEXT REFERENCES entities(id),
    action      TEXT NOT NULL,
    risk_class  TEXT NOT NULL,
    result      TEXT,                  -- ok, failed, escalated
    approval_id TEXT,                  -- references approvals if gated
    agent       TEXT,                  -- agent identity (age pubkey or hostname)
    notes       TEXT
);

-- Approvals (replaces approvals/*.jsonl)
CREATE TABLE approvals (
    id          TEXT PRIMARY KEY,
    ts          TIMESTAMPTZ NOT NULL DEFAULT now(),
    entity_id   TEXT REFERENCES entities(id),
    action      TEXT NOT NULL,
    risk_class  TEXT NOT NULL,
    status      TEXT NOT NULL DEFAULT 'pending',  -- pending, approved, denied, expired
    ttl         INTERVAL NOT NULL DEFAULT '1 hour',
    decided_at  TIMESTAMPTZ,
    decided_by  TEXT,
    confirmation_phrase TEXT
);

-- State snapshots (replaces oikos/state.json)
CREATE TABLE state_snapshots (
    id          SERIAL PRIMARY KEY,
    ts          TIMESTAMPTZ NOT NULL DEFAULT now(),
    entity_id   TEXT REFERENCES entities(id),
    health      TEXT,                  -- healthy, degraded, down, unknown
    data        JSONB                   -- full probe response
);

-- Knowledge graph (replaces narrative wiki docs)
CREATE TABLE knowledge_entities (
    id          TEXT PRIMARY KEY,      -- "doc:105-apps", "investigation:2026-07-05-disk"
    type        TEXT NOT NULL,         -- container-doc, host-doc, investigation, plan, runbook
    entity_id   TEXT REFERENCES entities(id),  -- link to inventory entity if applicable
    title       TEXT NOT NULL,
    content     TEXT NOT NULL,          -- markdown body
    tags        TEXT[],
    source_path TEXT,                  -- original file path for traceability
    updated_at  TIMESTAMPTZ DEFAULT now()
);

-- Knowledge relationships (cross-references between docs + entities)
CREATE TABLE knowledge_relations (
    source_id   TEXT REFERENCES knowledge_entities(id),
    target_id   TEXT,                  -- can be knowledge_entity or inventory entity
    type        TEXT NOT NULL,         -- describes, relates-to, runbook-for, etc.
    PRIMARY KEY (source_id, target_id, type)
);
```

**Ingest pipeline** (`oikos/db/ingest.py`):
- On startup / deploy: parse `inventory.yaml` → upsert into `entities` + `relations`
- Walk `docs/` directory → parse markdown, extract frontmatter → upsert into `knowledge_entities`
- Link knowledge entities to inventory entities by path convention (`docs/containers/105-apps.md` → `entity:lxc:apps`)
- Idempotent — safe to re-run on every deploy

**Migration from existing data:**
- Write a one-time script to import existing `signals/*.jsonl`, `ledger/*.jsonl`, `approvals/*.jsonl` into the DB
- `oikos/state.json` → `state_snapshots` table

### 2. Unified API server (`oikos/api/`, new — merges MCP + homelab CLI)

One FastAPI service exposing two interfaces:

**MCP interface** (`oikos/api/mcp.py`):
- Same tool surface as the current `mcp/server.py` (get_host, list_services, search_knowledge, etc.)
- Reads from PostgreSQL instead of filesystem
- `search_docs` → `search_knowledge` (queries the knowledge graph)
- New tools: `get_entity_relations`, `get_signal_history`, `get_ledger`
- Protocol: FastMCP over streamable HTTP (same as current)

**REST interface** (`oikos/api/routes/`):
- `GET /api/v1/hosts` — list hosts (replaces `homelab list`)
- `GET /api/v1/services` — list services
- `GET /api/v1/hosts/{name}` — host detail (replaces `homelab status`)
- `GET /api/v1/services/{name}/health` — service health (cache-first, `?live=true` for probe)
- `GET /api/v1/signals` — list signals (`?state=raised&severity=warning`)
- `POST /api/v1/signals/{id}/ack` — acknowledge
- `POST /api/v1/signals/{id}/resolve` — resolve
- `GET /api/v1/approvals` — pending approvals
- `POST /api/v1/approvals/{id}/decide` — approve/deny (operator only, Authentik-gated)
- `POST /api/v1/exec` — gated execution (the actuator entrypoint)
  - Body: `{entity, action, approval_id?}`
  - Classifies via `decide.py`, checks approval if needed, executes, records ledger
- `POST /api/v1/deploy/{service}` — trigger a service deploy
- `WS /api/v1/events` — real-time stream (signals, approvals, ledger entries)
- `GET /api/v1/knowledge/{entity_id}` — knowledge graph query for an entity
- `GET /api/v1/knowledge/search?q=...` — search knowledge graph

**Policy enforcement** is built into the API:
- Every mutating endpoint classifies the action via `oikos/decide.py`
- `config_mutation`/`destructive` actions require a valid approval ID
- All mutations write to the ledger automatically

**Auth:**
- MCP interface: no auth (internal, container-to-container)
- REST interface: Authentik OIDC forward-auth (via Caddy) for operator endpoints
- Internal service-to-service: shared secret / mTLS (Docker network)

### 3. Scheduler + Actuator (`oikos/scheduler/`, `oikos/actuator/`)

**Scheduler (Observe)** — adapted from `oikos/scheduler.py`:
- Runs as a Docker service on a timer (10-min interval via a loop or cron sidecar)
- Probes service health (HTTP), disk usage (SSH to hubris/strong), drift detection
- Writes signals to PostgreSQL (not JSONL files)
- Writes state snapshots to DB
- No more git commit/push of signals — they're in the DB

**Actuator (Act)** — new, `oikos/actuator/act.py`:
- Reads open signals from DB where `recommended_action` is set
- For each: classify via `decide.py`
  - `auto-act` (reversible_low, contained, confident) → execute via SSH/API, verify, resolve signal, write ledger
  - `escalate` → create approval request, notify via Matrix, acknowledge signal
- Loop-guard: check ledger history per (entity, action) to cap auto-retries
- Autonomy kill-switch: `policy.yaml` → `autonomy.auto_act: off|reversible_low`, `never_auto_act: [entity list]`
- **Phase 1**: direct SSH execution (keys mounted)
- **Phase 2**: execution through the API's `/exec` endpoint (gateway pattern)

### 4. Hermes agent container (`compose/hermes/`)

**What it is:**
- Hermes Agent runtime in a Docker container
- Runs in gateway mode — workstations connect to it remotely
- Config: `hermes/config.yaml` (providers, models, gateway port)
- Persona: `hermes/SOUL.md` (homelab-specific)
- Skills: `hermes/skills/homelab-ops/SKILL.md` (how to interact with the homelab)

**What it has access to:**
- **Oikos API** via MCP (container network, no auth needed)
- **Infisical** for secret retrieval (API token)
- **SSH keys** mounted from host (`~/.ssh` → `/root/.ssh:ro`) — for direct host access (hybrid approach)
- **Hermes data volume** — persistent state, session history

**How workstations connect:**
- Hermes gateway listens on port 8092
- Any workstation with Hermes CLI can connect: `hermes gateway connect mac-mini:8092`
- Or via Caddy ingress: `hermes.hubris.network` → mac-mini:8092 (mesh)

**Homelab skills** (loaded into Hermes):
- `homelab-ops` — how to use the Oikos API, classify actions, request approvals
- `homelab-deploy` — how to deploy/manage services
- `homelab-troubleshoot` — diagnostic procedures
- (Future skills added as needed)

### 5. Infisical secrets migration

**Phase 1 — Stand up Infisical:**
- Add Infisical to the Docker stack
- Configure with PostgreSQL backend (shared with Oikos DB or separate DB)
- Bootstrap admin credentials, create `hubris` project + environments (prod, staging)

**Phase 2 — Migrate secrets:**
- Write a migration script: read all `secrets/*.yaml` (SOPS-encrypted), decrypt with age key, import into Infisical
- Map each secret to Infisical's key-value structure
- Verify all secrets are imported

**Phase 3 — Wire services:**
- All OS services authenticate to Infisical via machine identity (service token)
- Services fetch secrets at startup or on-demand
- Remove `secrets/` directory and `.sops.yaml` once verified
- The age key on mac-mini stays for one-time decryption during migration, then is retired

**Phase 4 — Retire SOPS:**
- Remove SOPS dependencies from the repo
- Update inventory to reflect Infisical as the secret store
- Document the new secret access pattern

### 6. Knowledge graph ingestion (`oikos/knowledge/`)

**What it does:**
- On deploy, walks `docs/` directory
- Parses each markdown file:
  - Extracts frontmatter (if any) for metadata
  - Infers entity relationships from path conventions:
    - `docs/containers/105-apps.md` → links to `entity:lxc:apps`
    - `docs/hosts/hubris.md` → links to `entity:host:hubris`
    - `docs/infrastructure/dns.md` → links to `entity:service:dns`
  - Extracts cross-references (links to other docs) → `knowledge_relations`
  - Tags by directory: container-doc, host-doc, infrastructure, investigation, plan
- Upserts into `knowledge_entities` + `knowledge_relations` tables

**Agent access:**
- MCP tool `search_knowledge(query)` — full-text search on knowledge graph
- MCP tool `get_entity_knowledge(entity_id)` — all docs related to an entity
- MCP tool `get_relations(entity_id)` — ontology blast-radius query (existing, adapted to DB)

**Docs as source, DB as query layer:**
- Humans still write markdown docs in `docs/`
- The DB is the query/index layer — agents never grep files
- On deploy, docs are re-ingested (idempotent)
- Future: wiki-style editing UI that writes back to `docs/` via git

### 7. Notifier abstraction (`oikos/notifier/`)

**Interface:**
```python
class Notifier(Protocol):
    def send_alert(self, signal: Signal) -> None: ...
    def send_approval_request(self, approval: Approval) -> None: ...
    def listen_for_decisions(self) -> Iterator[ApprovalDecision]: ...
```

**Matrix implementation** (current):
- Sends alerts to `@dtoro:avispero` via Synapse (LXC 118)
- Approval requests as messages with ✅/❌ reactions
- Listens for reactions to record decisions
- Reuses existing `oikos/approve.py` Matrix delivery logic

**Future implementations** (pluggable):
- Webhook, email, Slack, etc.
- Registered in config, selected at runtime

### 8. Docker build + deploy pipeline

**Build:**
- `docker compose build` from the repo root
- Each service has a Dockerfile in `compose/<service>/`
- Shared base image: Python 3.12-slim + common deps (postgresql libs, etc.)
- Multi-stage builds for smaller images

**Deploy trigger:**
- Gitea webhook on push to `main` → hits a deploy endpoint on mac-mini
- Deploy script: `git pull && docker compose build && docker compose up -d`
- Alternatively: a "deployer" sidecar container watches Gitea and triggers rebuilds
- The existing 5-min launchd git pull can serve as a fallback deploy mechanism initially

**Health checks:**
- Each service has a Docker healthcheck
- `docker compose ps` shows health status
- API exposes `/health` for external monitoring

### 9. Ingress re-point (Caddy)

- In `dtoro/caddy-conf`: point `mcp.hubris.network` and `oikos.hubris.network` to
  mac-mini's mesh address at port 8090 (API)
- Future: `hermes.hubris.network` → mac-mini:8092 (Hermes gateway)
- DNS and public URLs unchanged
- Verify Caddy → mac-mini reachability over netbird first (same as original plan)

### 10. mac-mini host setup

**Directory:**
- New directory: `~/oikos-os/` (or `/opt/oikos-os/`) — the repo clone
- This is where `docker compose` runs from
- Existing `/opt/homelab-context/` stays untouched until cleanup phase

**Prerequisites:**
- Docker Desktop (or OrbStack) — already installed
- SSH keys accessible at `~/.ssh/` (mounted into containers)
- Age key at `/etc/age/key.txt` (for one-time SOPS migration)

**Cleanup (deferred, after Docker OS is proven):**
- Stop launchd git-sync for `/opt/homelab-context`
- Remove native Hermes install at `~/.hermes/hermes-agent/`
- Remove `/opt/homelab-context` clone
- Remove any launchd plists for homelab services

### 11. Decommission apps/105 (deferred)

Keep apps/105 running all Oikos services (homelab-mcp, secrets-issuance,
oikos-console) as a fallback until the Docker OS is proven end-to-end.

**Cutover checklist (executed when ready):**
1. Verify Docker OS serves all traffic correctly
2. `systemctl disable --now homelab-mcp secrets-issuance oikos-console` on apps/105
3. Remove `/opt/homelab-mcp`, `/opt/oikos-console` checkouts on apps/105
4. Update Caddy backends to point to mac-mini exclusively
5. Remove Gitea webhooks 10, 11, 14 (or repoint to Docker deploy trigger)

## Reuse (do not reimplement)

| Existing code | What it becomes |
|---|---|
| `oikos/decide.py` | Risk classifier — adapted to read from DB, logic unchanged |
| `oikos/signal.py` | Signal engine — adapted to DB, lifecycle logic unchanged |
| `oikos/approve.py` | Approval engine — adapted to DB, Matrix delivery reused |
| `oikos/ledger.py` | Ledger — adapted to DB |
| `oikos/policy.py` + `policy.yaml` | Policy loader — unchanged, still reads YAML |
| `oikos/drift.py` | Drift detectors — adapted to write signals to DB |
| `oikos/relations.py` | Ontology graph walk — adapted to DB |
| `oikos/report.py` | Report generator — adapted to read from DB |
| `mcp/server.py` | Merged into `oikos/api/` — MCP protocol adapter on top of DB |
| `bin/homelab` logic | Migrated into API routes — same operations, REST interface |
| `oikos/scheduler.py` | Adapted — probes unchanged, output goes to DB |
| `oikos/approve.py` Matrix delivery | Reused as the Matrix notifier implementation |

## Risks / trade-offs

- **Hermes in Docker** — the agent's "world" is the container. Direct file access
  to the host is limited to mounted volumes. SSH access is the bridge to the homelab.
  Trade-off: less host integration, more isolation. Mitigated by the hybrid approach
  (SSH now, actuator gateway later).
- **PostgreSQL as single point of failure** — if Postgres goes down, the OS loses
  all state. Mitigated by Docker volume persistence + regular `pg_dump` backups
  (can be automated by the scheduler itself once running).
- **Infisical bootstrapping** — migrating from SOPS to Infisical has a transition
  period where both systems coexist. Keep SOPS as fallback until all services are
  verified against Infisical.
- **Knowledge graph ingestion quality** — markdown-to-structured-data conversion
  relies on path conventions and frontmatter. Docs without frontmatter may lose
  metadata. Start simple, improve extraction over time.
- **Deploy downtime** — `docker compose up -d` may briefly interrupt services during
  container replacement. Use `--scale` + health-gated rollout for zero-downtime
  when needed (later).
- **Multi-agent concurrency** — PostgreSQL handles concurrent writers correctly,
  but the actuator needs a lock mechanism to prevent two passes from acting on
  the same signal simultaneously. Use `SELECT ... FOR UPDATE SKIP LOCKED`.

## Verification (end to end)

1. **Database:** `docker compose up postgres` — `psql` shows all tables; ingest
   script populates entities from `inventory.yaml` and knowledge from `docs/`.
2. **API:** `curl http://localhost:8090/api/v1/hosts` returns the fleet;
   MCP `list_services` tool works via the same endpoint.
3. **Scheduler:** trigger a probe pass — signals appear in DB, state snapshot written.
4. **Actuator:** raise a test `service-down` signal → actuator classifies →
   auto-acts (restart) or escalates (Matrix approval request) → ledger entry written.
5. **Hermes:** connect from another workstation via `hermes gateway connect` →
   agent responds, can query the API via MCP, can SSH to hubris.
6. **Secrets:** Infisical running, services fetch secrets at startup, SOPS files
   removed without breakage.
7. **Deploy:** `git push` → Gitea webhook → `docker compose build + up -d` →
   changes live within minutes.
8. **Knowledge:** MCP `search_knowledge("caddy")` returns relevant docs linked
   to `entity:service:caddy`.
9. **Notifier:** raise a `critical` signal → Matrix message received →
   ✅ reaction → approval granted in DB.
10. **Cutover:** after all above passes, stop services on apps/105, verify
    production traffic served only by Docker OS on mac-mini.

## Phasing

**Phase 1 — Foundation (DB + API):**
- PostgreSQL schema + models + migrations
- Ingest pipeline (inventory.yaml → DB, docs → knowledge graph)
- Unified API server (REST + MCP) reading from DB
- Migrate existing signals/ledger/approvals data into DB

**Phase 2 — Services (Scheduler + Actuator + Notifier):**
- Adapt scheduler to read/write DB
- Build actuator (Act stage) with kill-switch
- Matrix notifier (abstracted interface + impl)
- SSH-based execution (hybrid approach)

**Phase 3 — Agent (Hermes container):**
- Hermes Docker image
- Gateway mode config
- Homelab skills (homelab-ops)
- Connect from workstation, verify MCP + SSH access

**Phase 4 — Secrets (Infisical):**
- Stand up Infisical in the stack
- Migrate SOPS secrets
- Wire all services to Infisical
- Retire SOPS

**Phase 5 — Deploy pipeline:**
- Docker Compose fully working
- Gitea webhook → rebuild trigger
- Caddy ingress re-point to mac-mini
- Health checks + monitoring

**Phase 6 — Cutover + cleanup:**
- Verify end-to-end
- Stop apps/105 services
- Clean up mac-mini old artifacts
- Remove old deploy webhooks

## Out of scope (for now)

- Oikos Console web UI (deferred — built later on top of the API for specific tasks)
- Multi-node deployment (designed for, not implemented)
- Vector embeddings / semantic search (knowledge graph is structured-only for now)
- SSH-key-signed approval requests (same backlog as before)
- Prometheus / trend signals
- Actuator gateway pattern (Phase 2 of the hybrid — start with mounted SSH)
- Custom homelab skills beyond `homelab-ops` (add as needed)