plans: pivot oikos consolidation to docker-based agentic homelab OS
Supersedes the launchd-based consolidation plan. Key changes: - Docker-based deployment on mac-mini (docker compose) - PostgreSQL for all mutable state (signals, ledger, knowledge graph) - Infisical replaces SOPS+age for secrets management - Unified API merges MCP server + homelab CLI (REST + MCP interfaces) - Hermes agent runs in Docker (gateway mode, connect from any workstation) - Knowledge graph in Postgres replaces narrative wiki files as agent context - Structured entity relationships link docs to inventory entities - Hybrid SSH access (mounted keys now, actuator gateway later) - Git push → Gitea webhook → Docker rebuild = deploy trigger - 6-phase rollout: DB → services → agent → secrets → deploy → cutover
This commit is contained in:
@@ -1,220 +1,689 @@
|
||||
# Plan: Consolidate the Oikos control plane onto mac-mini
|
||||
# Plan: Oikos — Docker-based agentic homelab OS on mac-mini
|
||||
|
||||
**Status:** Planned (2026-07-06) — approved, not yet executed.
|
||||
**Status:** Planned (2026-07-06, revised) — supersedes the launchd-based consolidation plan.
|
||||
**Predecessor:** The original plan re-platformed Oikos services from LXC 105 (apps) to
|
||||
mac-mini via launchd plists. This revision pivots to a **Docker-based** architecture:
|
||||
the repo becomes a containerized homelab OS, deployed via `docker compose` on mac-mini.
|
||||
|
||||
## Context
|
||||
## Vision
|
||||
|
||||
The goal is an agentic OS that runs the homelab and keeps the operator in
|
||||
control. Today the Oikos control plane is scattered and, worse, mis-placed:
|
||||
`homelab-mcp`, `secrets-issuance`, and `oikos-console` all run on **LXC 105
|
||||
(apps) — a container inside hubris**, `oikos-scheduler` (the Observe stage) is
|
||||
undeployed, and the Hermes agent runtime is spread across several clients. Two
|
||||
problems follow:
|
||||
Convert this repo into a **Docker-based agentic homelab OS**. The OS is a set of
|
||||
containerized services that manage the homelab autonomously, with the operator in
|
||||
control. Two actors:
|
||||
|
||||
- **The observer lives inside the observed.** A scheduler on apps/105 can't
|
||||
report "hubris is down" — it dies with hubris. The Observe/Act loop needs an
|
||||
always-on node *outside* both hypervisors.
|
||||
- **Deploy is fragmented.** Three Gitea webhooks (ids 10/11/14); webhook 14
|
||||
(console) has been 403-broken for a cause never found, so console never
|
||||
auto-redeploys.
|
||||
- **Operator** (dtoro) — owns the homelab, expresses intent ("install X", "restart
|
||||
Y"), approves destructive actions. Connects from any workstation via remote Hermes
|
||||
or Matrix.
|
||||
- **Agent** — Hermes core + custom homelab skills, running in Docker. Executes orders,
|
||||
monitors the lab, escalates when unsure.
|
||||
|
||||
**Decision:** make **mac-mini** the single Oikos control node — it's always-on,
|
||||
external to both hubris and strong, on the mesh, can SSH/curl both, and already
|
||||
runs Hermes. Move *everything* (MCP, secrets-issuance, console, scheduler, the
|
||||
new actuator, Hermes) there; apps/105 stops running Oikos. Because mac-mini
|
||||
already has a launchd git-sync (5-min pull), we replace all three Gitea webhooks
|
||||
with pull-based deploy — which eliminates the webhook-403 problem entirely.
|
||||
All OS services run in Docker containers on mac-mini. The OS is deployed by a git
|
||||
push (Gitea webhook → Docker rebuild). It's designed for mac-mini now, with a path
|
||||
to multi-node later.
|
||||
|
||||
This is a **macOS re-platforming** (systemd → launchd) plus an **ingress
|
||||
re-point**. The public URLs (`mcp.`/`secrets.`/`oikos.hubris.network`) and DNS
|
||||
are unchanged — only the Caddy backend and the runtime host move.
|
||||
## Decisions (from operator Q&A, 2026-07-06)
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| Repo structure | One repo, reorganize internally. Wiki docs get restructured into the knowledge graph. |
|
||||
| Hermes runtime | Runs inside Docker as part of the OS stack (gateway mode — connect from any workstation). |
|
||||
| Agent → homelab access | Hybrid — mounted SSH keys for now, build actuator gateway incrementally. |
|
||||
| Data storage | Migrate to PostgreSQL. `inventory.yaml` stays as declarative config; signals/ledger/state/knowledge go into DB. |
|
||||
| Knowledge/context | Structured knowledge graph in Postgres — docs become typed entities linked to inventory entities. |
|
||||
| Operator interface | Primary = remote Hermes from any workstation + Matrix. Console/UIs built later for specific tasks. |
|
||||
| Host | mac-mini for now, designed to scale later. |
|
||||
| Deploy | Git push → Gitea webhook → Docker rebuild + restart. |
|
||||
| `bin/homelab` CLI | Replaced by an API. CLI becomes a thin client that calls the OS API. |
|
||||
| Secrets | Migrate from SOPS+age to Infisical (open-source, API-first, Docker-native). |
|
||||
| Matrix | Keep for now, abstract the notification layer for future channels. |
|
||||
| MCP server | Merged into the unified API — one service, REST + MCP interfaces. |
|
||||
| Agent type | Hermes core + custom homelab skills (standard Hermes, purpose-loaded). |
|
||||
| apps/105 | Keep running as fallback until Docker OS is proven. |
|
||||
| mac-mini cleanup | Start fresh in a new directory, clean up old artifacts later. |
|
||||
|
||||
## Target architecture
|
||||
|
||||
```
|
||||
Caddy (LXC 121, ingress) ──reverse_proxy over netbird──▶ mac-mini (always-on, macOS)
|
||||
mcp.hubris.network ─────────────────────────────▶ homelab-mcp :9810
|
||||
secrets.hubris.network ─────────────────────────────▶ secrets-issuance :9820
|
||||
oikos.hubris.network ─────────────────────────────▶ oikos-console :8091
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ mac-mini (Docker host, always-on) │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌────────────┐ ┌───────────────────────────┐ │
|
||||
│ │ PostgreSQL │ │ Infisical │ │ Hermes Agent (gateway) │ │
|
||||
│ │ │ │ (secrets) │ │ + homelab skills │ │
|
||||
│ │ • inventory │ │ │ │ + MCP client → API │ │
|
||||
│ │ • signals │ │ │ │ + SSH keys (mounted) │ │
|
||||
│ │ • ledger │ │ │ │ │ │
|
||||
│ │ • knowledge │ │ │ │ Any workstation connects │ │
|
||||
│ │ • state │ │ │ │ via Hermes gateway protocol │ │
|
||||
│ │ • approvals │ │ │ │ │ │
|
||||
│ └──────┬────────┘ └─────────────┘ └──────────┬────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌──────┴─────────────────────────────────────────┴──────────────┐ │
|
||||
│ │ Oikos API (REST + MCP, one service) │ │
|
||||
│ │ │ │
|
||||
│ │ MCP interface (agent context): │ │
|
||||
│ │ get_host, list_services, search_knowledge, get_entity, │ │
|
||||
│ │ get_relations, get_change_history, get_state_snapshot │ │
|
||||
│ │ │ │
|
||||
│ │ REST interface (operations + operator): │ │
|
||||
│ │ GET /api/v1/hosts, /services, /signals, /approvals │ │
|
||||
│ │ POST /api/v1/exec — gated execution (actuator) │ │
|
||||
│ │ POST /api/v1/approve — operator approval │ │
|
||||
│ │ POST /api/v1/deploy — trigger service deploy │ │
|
||||
│ │ WS /api/v1/events — real-time signal/approval stream │ │
|
||||
│ │ │ │
|
||||
│ │ Policy enforcement, risk classification, ledger recording │ │
|
||||
│ └──────────────────────┬───────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────────┴───────────────────────────────────────────┐ │
|
||||
│ │ Scheduler (Observe) + Actuator (Act) │ │
|
||||
│ │ • Periodic probes (10-min) → signals in DB │ │
|
||||
│ │ • Reads open signals, classifies via decide.py │ │
|
||||
│ │ • Auto-acts (reversible_low) or escalates (approval) │ │
|
||||
│ │ • SSH to hubris/strong for probes + execution │ │
|
||||
│ └──────────────────────┬───────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────────┴───────────────────────────────────────────┐ │
|
||||
│ │ Notifier (abstracted, pluggable) │ │
|
||||
│ │ • Matrix (current) — alerts, approval requests │ │
|
||||
│ │ • Future: webhook, email, Slack, etc. │ │
|
||||
│ └──────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Deploy: Gitea webhook → docker compose build + up -d │
|
||||
│ Volumes: pg-data, infisical-data, ssh-keys (ro mount), hermes-config │
|
||||
│ Network: Docker bridge (internal) + host network (SSH outbound) │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
mac-mini also runs (no ingress, mesh/SSH outward):
|
||||
oikos-scheduler + actuator (launchd StartInterval 600s) ──ssh/curl──▶ hubris, strong
|
||||
Hermes agent runtime (already here)
|
||||
External:
|
||||
Caddy (LXC 121) → mac-mini mesh IP :8090 (API), future UIs
|
||||
apps/105 → fallback, kept running until Docker OS proven
|
||||
hubris/strong → SSH targets for scheduler/actuator
|
||||
Any workstation → Hermes gateway (connects to agent in Docker)
|
||||
Gitea (LXC 104) → webhook triggers Docker rebuild
|
||||
```
|
||||
|
||||
Single synced checkout: /opt/homelab-context (5-min launchd git pull = deploy trigger)
|
||||
apps/105: Oikos services stopped + removed; keeps its non-Oikos roles (artifacto, plantuml)
|
||||
## Repo reorganization
|
||||
|
||||
The repo shifts from "wiki + code mixed" to "OS code + declarative config + docs as
|
||||
ingest source." Proposed layout:
|
||||
|
||||
```
|
||||
/ # repo root
|
||||
├── docker-compose.yml # the OS stack definition
|
||||
├── compose/ # per-service Dockerfiles + build context
|
||||
│ ├── api/ # Oikos API (REST + MCP) — merges mcp/ + bin/homelab
|
||||
│ │ └── Dockerfile
|
||||
│ ├── scheduler/ # Observe + Act loop
|
||||
│ │ └── Dockerfile
|
||||
│ ├── hermes/ # Hermes agent runtime
|
||||
│ │ └── Dockerfile
|
||||
│ ├── postgres/ # init scripts, migrations
|
||||
│ │ └── init.sql
|
||||
│ └── infisical/ # Infisical config
|
||||
│ └── config.yaml
|
||||
├── oikos/ # Core OS Python package (reused, adapted)
|
||||
│ ├── api/ # NEW — unified API server
|
||||
│ │ ├── server.py # FastAPI app, REST routes + MCP protocol adapter
|
||||
│ │ ├── routes/ # REST route modules
|
||||
│ │ └── mcp.py # MCP protocol adapter (exposes same tools via MCP)
|
||||
│ ├── db/ # NEW — database layer
|
||||
│ │ ├── models.py # SQLAlchemy models: Inventory, Signal, Ledger, Knowledge, Approval
|
||||
│ │ ├── migrations/ # Alembic migrations
|
||||
│ │ └── ingest.py # Ingest inventory.yaml + docs into DB on deploy
|
||||
│ ├── scheduler/ # Adapted from scheduler.py — reads/writes DB
|
||||
│ ├── actuator/ # NEW — Act stage: reads signals, classifies, executes
|
||||
│ │ └── act.py
|
||||
│ ├── policy/ # Adapted — policy.yaml loader + classifier
|
||||
│ ├── knowledge/ # NEW — knowledge graph ingestion from docs
|
||||
│ │ └── ingest_docs.py
|
||||
│ ├── notifier/ # NEW — abstracted notification (Matrix impl)
|
||||
│ │ ├── base.py # Notifier interface
|
||||
│ │ └── matrix.py # Matrix implementation
|
||||
│ ├── decide.py # Reused — risk classifier (reads from DB)
|
||||
│ ├── signal.py # Adapted — signals in DB instead of JSONL files
|
||||
│ ├── approve.py # Adapted — approvals in DB
|
||||
│ ├── ledger.py # Adapted — ledger in DB
|
||||
│ ├── drift.py # Adapted — drift detection, writes signals to DB
|
||||
│ ├── relations.py # Reused — ontology graph walk (reads from DB)
|
||||
│ └── report.py # Adapted — generates reports from DB
|
||||
├── hermes/ # Hermes agent config + skills (mounted into container)
|
||||
│ ├── config.yaml # Hermes config (providers, models, gateway)
|
||||
│ ├── SOUL.md # Agent persona (homelab-specific)
|
||||
│ └── skills/ # Custom homelab skills
|
||||
│ └── homelab-ops/
|
||||
│ └── SKILL.md
|
||||
├── inventory.yaml # Declarative config — the "kernel data structure" (stays YAML)
|
||||
├── ontology.yaml # Systems model (stays YAML)
|
||||
├── policy.yaml # Risk & approval policy (stays YAML)
|
||||
├── docs/ # Narrative docs — ingested into knowledge graph on deploy
|
||||
│ ├── containers/ # LXC/VM documentation
|
||||
│ ├── hosts/ # Host documentation
|
||||
│ ├── infrastructure/ # Infrastructure docs
|
||||
│ └── investigations/ # Incident records
|
||||
├── secrets/ # SOPS-encrypted secrets (migrated to Infisical, then removed)
|
||||
└── .sops.yaml # SOPS config (deprecated after Infisical migration)
|
||||
```
|
||||
|
||||
**What stays YAML (declarative, human-edited, git-tracked):**
|
||||
- `inventory.yaml` — topology, the kernel data structure
|
||||
- `ontology.yaml` — entity types, relationships, lifecycle
|
||||
- `policy.yaml` — risk classes, approval rules, autonomy settings
|
||||
- `hermes/config.yaml` — agent configuration
|
||||
- `hermes/SOUL.md` — agent persona
|
||||
|
||||
**What moves to PostgreSQL (mutable, runtime state):**
|
||||
- `signals/` → `signals` table
|
||||
- `ledger/` → `ledger_entries` table
|
||||
- `oikos/state.json` → `state_snapshots` table
|
||||
- `approvals/` → `approvals` table
|
||||
- `knowledge/wiki/` narrative docs → `knowledge_entities` + `knowledge_relations` tables
|
||||
|
||||
**What gets removed (no longer needed):**
|
||||
- `mcp/server.py` — merged into `oikos/api/server.py`
|
||||
- `bin/homelab` — replaced by the API; thin CLI client can be generated later
|
||||
- `secrets-issuance/` — replaced by Infisical
|
||||
- `oikos/systemd/` — replaced by Docker containers
|
||||
- `oikos/console/` — deferred (built later as a specific UI on top of the API)
|
||||
- `mcp/deploy/`, `oikos/console/deploy/` — replaced by Docker build
|
||||
- `hosts/*.yaml` — generated from inventory, now served by the API directly
|
||||
|
||||
## Docker Compose stack
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml (simplified — full version in the repo)
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
volumes: [pg-data:/var/lib/postgresql/data, ./compose/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql]
|
||||
environment:
|
||||
POSTGRES_DB: oikos
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/pg_password
|
||||
secrets: [pg_password]
|
||||
healthcheck: ...
|
||||
|
||||
infisical:
|
||||
image: infisical/infisical:latest
|
||||
volumes: [infisical-data:/var/lib/infisical]
|
||||
environment: ...
|
||||
# Secrets manager — replaces SOPS+age for runtime secret access
|
||||
|
||||
api:
|
||||
build: ./compose/api
|
||||
depends_on: [postgres, infisical]
|
||||
environment:
|
||||
DATABASE_URL: postgresql://oikos@postgres/oikos
|
||||
HOMELAB_CONTEXT_DIR: /opt/homelab-context
|
||||
volumes:
|
||||
- ./inventory.yaml:/app/inventory.yaml:ro
|
||||
- ./ontology.yaml:/app/ontology.yaml:ro
|
||||
- ./policy.yaml:/app/policy.yaml:ro
|
||||
- ./docs:/app/docs:ro
|
||||
ports: ["8090:8090"] # REST + MCP
|
||||
# On startup: ingest inventory.yaml + docs into DB if changed
|
||||
|
||||
scheduler:
|
||||
build: ./compose/scheduler
|
||||
depends_on: [api, postgres]
|
||||
environment:
|
||||
DATABASE_URL: postgresql://oikos@postgres/oikos
|
||||
SSH_KEY: /run/secrets/ssh_key
|
||||
volumes:
|
||||
- ssh-keys:/run/secrets:ro
|
||||
# Runs observe + act loop on a cron-like schedule (10-min interval)
|
||||
|
||||
hermes:
|
||||
build: ./compose/hermes
|
||||
depends_on: [api, infisical]
|
||||
environment:
|
||||
OIKOS_API_URL: http://api:8090
|
||||
INFISICAL_TOKEN: ...
|
||||
volumes:
|
||||
- ./hermes:/root/.hermes:ro
|
||||
- ssh-keys:/root/.ssh:ro
|
||||
- hermes-data:/root/.hermes/data
|
||||
ports: ["8092:8092"] # Hermes gateway port
|
||||
# Agent runtime — connects to API via MCP, has SSH for direct host access
|
||||
|
||||
notifier:
|
||||
build: ./compose/scheduler # shares image with scheduler
|
||||
depends_on: [api, postgres]
|
||||
# Matrix bot — listens for approval requests, sends alerts
|
||||
# Abstracted via notifier interface — Matrix impl now, pluggable later
|
||||
|
||||
volumes:
|
||||
pg-data:
|
||||
infisical-data:
|
||||
ssh-keys: # populated from host ~/.ssh at deploy time
|
||||
hermes-data:
|
||||
```
|
||||
|
||||
## Workstreams
|
||||
|
||||
### 1. macOS runtime — launchd plists (`deploy/macos/`, new)
|
||||
### 1. Database schema + models (`oikos/db/`, new)
|
||||
|
||||
One `LaunchDaemon` plist per service (run at boot, no GUI login), mirroring each
|
||||
systemd unit's `ExecStart`/`Environment`, replacing `[Timer]` with launchd keys:
|
||||
The foundational layer — everything else reads/writes through this.
|
||||
|
||||
| Service | plist | launchd equivalent of |
|
||||
|---|---|---|
|
||||
| `homelab-mcp` | `network.hubris.homelab-mcp.plist` | `mcp/deploy/homelab-mcp.service` (KeepAlive) |
|
||||
| `secrets-issuance` | `network.hubris.secrets-issuance.plist` | its systemd unit (KeepAlive) |
|
||||
| `oikos-console` | `network.hubris.oikos-console.plist` | `oikos/console/deploy/oikos-console.service` (KeepAlive) |
|
||||
| `oikos-scheduler` | `network.hubris.oikos-scheduler.plist` | `oikos/systemd/oikos-scheduler.{service,timer}` (`StartInterval` 600) |
|
||||
**Tables:**
|
||||
|
||||
- **Single checkout, shared venv.** All services run in-place from
|
||||
`/opt/homelab-context` with one shared `.venv` (gitignored), instead of the
|
||||
Linux per-service checkouts (`/opt/homelab-mcp`, `/opt/oikos-console`). This is
|
||||
the concrete consolidation win: one clone, one sync, N daemons.
|
||||
- **Confinement gap (tradeoff, document it):** the systemd units use
|
||||
`ProtectSystem=strict`, `ReadOnlyPaths`, `NoNewPrivileges`, `PrivateTmp` —
|
||||
launchd has no equivalent. Mitigate by running the daemons as a dedicated
|
||||
non-root `_oikos` user (or `dtoro`) with least-privilege file perms, and note
|
||||
the reduced isolation in the deploy README + OIKOS.md risk section.
|
||||
- **`install-macos.sh`** — create `_oikos` user + venv (`pip install` the deps),
|
||||
copy plists to `/Library/LaunchDaemons`, `launchctl bootstrap system …`.
|
||||
Idempotent, mirroring `mcp/deploy/deploy.sh`'s shape.
|
||||
```sql
|
||||
-- Inventory entities (synced from inventory.yaml on deploy)
|
||||
CREATE TABLE entities (
|
||||
id TEXT PRIMARY KEY, -- "host:hubris", "service:caddy"
|
||||
type TEXT NOT NULL, -- host, service, lxc, vm, etc.
|
||||
name TEXT NOT NULL,
|
||||
domain TEXT, -- physical, compute, network, etc.
|
||||
state TEXT DEFAULT 'active', -- provisioning, active, deprecated, destroyed
|
||||
data JSONB NOT NULL, -- full entity record from inventory.yaml
|
||||
updated_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
|
||||
### 2. macOS deploy = the existing pull-sync (retire the 3 webhooks)
|
||||
-- Entity relationships (the ontology graph)
|
||||
CREATE TABLE relations (
|
||||
source_id TEXT REFERENCES entities(id),
|
||||
target_id TEXT REFERENCES entities(id),
|
||||
type TEXT NOT NULL, -- hosts, provides, mounts, depends-on, etc.
|
||||
data JSONB,
|
||||
PRIMARY KEY (source_id, target_id, type)
|
||||
);
|
||||
|
||||
- Extend mac-mini's existing launchd git-sync post-pull hook (the enrollment
|
||||
`tools/post-pull.sh` path) to `launchctl kickstart -k` any service whose files
|
||||
changed after a pull. Pull-based deploy replaces webhook-push deploy.
|
||||
- **Retire Gitea webhooks 10, 11, 14** (operator step in Gitea UI). This removes
|
||||
the never-solved console-webhook 403 by deleting the webhook, not fixing it.
|
||||
- Update `knowledge/wiki/infrastructure/auto-deploy.md`: mac-mini services deploy
|
||||
by pull-sync, not webhook; the webhook shape now applies only to remaining
|
||||
in-lab config repos.
|
||||
-- Signals (replaces signals/*.jsonl)
|
||||
CREATE TABLE signals (
|
||||
id TEXT PRIMARY KEY, -- "sig-2026-07-06-0001"
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
kind TEXT NOT NULL, -- service-down, disk-threshold, drift, etc.
|
||||
severity TEXT NOT NULL, -- info, warning, critical
|
||||
entity_id TEXT REFERENCES entities(id),
|
||||
evidence TEXT,
|
||||
likely_cause TEXT,
|
||||
recommended_action JSONB,
|
||||
verification TEXT,
|
||||
state TEXT NOT NULL DEFAULT 'raised', -- raised, acknowledged, acting, resolved, muted
|
||||
mute_until TIMESTAMPTZ,
|
||||
note TEXT,
|
||||
updated_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
|
||||
### 3. Build the actuator + autonomy kill-switch (the Act stage)
|
||||
-- Change ledger (replaces ledger/*.jsonl)
|
||||
CREATE TABLE ledger_entries (
|
||||
id SERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
entity_id TEXT REFERENCES entities(id),
|
||||
action TEXT NOT NULL,
|
||||
risk_class TEXT NOT NULL,
|
||||
result TEXT, -- ok, failed, escalated
|
||||
approval_id TEXT, -- references approvals if gated
|
||||
agent TEXT, -- agent identity (age pubkey or hostname)
|
||||
notes TEXT
|
||||
);
|
||||
|
||||
Still required — it's the control loop, now living on mac-mini.
|
||||
-- Approvals (replaces approvals/*.jsonl)
|
||||
CREATE TABLE approvals (
|
||||
id TEXT PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
entity_id TEXT REFERENCES entities(id),
|
||||
action TEXT NOT NULL,
|
||||
risk_class TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'pending', -- pending, approved, denied, expired
|
||||
ttl INTERVAL NOT NULL DEFAULT '1 hour',
|
||||
decided_at TIMESTAMPTZ,
|
||||
decided_by TEXT,
|
||||
confirmation_phrase TEXT
|
||||
);
|
||||
|
||||
- **`oikos/act.py`** (new): read open Signals (`oikos_signal.list_signals(
|
||||
state="raised")`); for each with a `recommended_action`, call
|
||||
`oikos_decide.classify()`. `auto-act` (reversible_low, contained, confident)
|
||||
→ `start_acting()` (the existing unused transition), execute via
|
||||
`homelab restart <svc> -y` (reuses `cmd_restart` + its `_record_change` ledger
|
||||
write), run the Signal's `verification`, then `resolve()` or leave open with a
|
||||
`result=failed` ledger entry so the next pass escalates. Everything else →
|
||||
`oikos_approve.request(...)` + `acknowledge(note="escalated: <id>")`. A
|
||||
ledger-history loop-guard caps auto-retries per (entity, kind).
|
||||
- **Autonomy kill-switch** in `oikos/policy.yaml` (`autonomy.auto_act:
|
||||
off|reversible_low`, `never_auto_act:` entity list) + loaders in
|
||||
`oikos/policy.py` — the "keep control" guarantee.
|
||||
- Wire `python3 oikos/act.py run` into the scheduler's launchd pass (after the
|
||||
observe pass) and commit `signals/` + `ledger/`.
|
||||
-- State snapshots (replaces oikos/state.json)
|
||||
CREATE TABLE state_snapshots (
|
||||
id SERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
entity_id TEXT REFERENCES entities(id),
|
||||
health TEXT, -- healthy, degraded, down, unknown
|
||||
data JSONB -- full probe response
|
||||
);
|
||||
|
||||
### 4. Ingress re-point (Caddy backends: apps → mac-mini mesh)
|
||||
-- Knowledge graph (replaces narrative wiki docs)
|
||||
CREATE TABLE knowledge_entities (
|
||||
id TEXT PRIMARY KEY, -- "doc:105-apps", "investigation:2026-07-05-disk"
|
||||
type TEXT NOT NULL, -- container-doc, host-doc, investigation, plan, runbook
|
||||
entity_id TEXT REFERENCES entities(id), -- link to inventory entity if applicable
|
||||
title TEXT NOT NULL,
|
||||
content TEXT NOT NULL, -- markdown body
|
||||
tags TEXT[],
|
||||
source_path TEXT, -- original file path for traceability
|
||||
updated_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
|
||||
- In `dtoro/caddy-conf` (separate repo, tracked-config commit+push): change the
|
||||
three `reverse_proxy` backends for `mcp.`/`secrets.`/`oikos.hubris.network`
|
||||
from `192.168.8.205:<port>` to **mac-mini's netbird address**
|
||||
(`mac-mini-234-17.netbird.selfhosted:<port>`), since Caddy (8.x) and mac-mini
|
||||
(178.x) are cross-subnet — mesh is the robust path. **DNS and public URLs
|
||||
unchanged.**
|
||||
- **Verify Caddy → mac-mini reachability first** (Caddy container must resolve +
|
||||
reach the netbird name; confirm netbird is on the caddy LXC or route via
|
||||
hubris). If mesh isn't reachable from Caddy, fall back to the 178.x LAN IP and
|
||||
confirm hubris routes 8.x↔178.x (it's dual-homed: hubris 178.181, apps 8.205).
|
||||
-- Knowledge relationships (cross-references between docs + entities)
|
||||
CREATE TABLE knowledge_relations (
|
||||
source_id TEXT REFERENCES knowledge_entities(id),
|
||||
target_id TEXT, -- can be knowledge_entity or inventory entity
|
||||
type TEXT NOT NULL, -- describes, relates-to, runbook-for, etc.
|
||||
PRIMARY KEY (source_id, target_id, type)
|
||||
);
|
||||
```
|
||||
|
||||
### 5. Secrets & SSH keys on mac-mini
|
||||
**Ingest pipeline** (`oikos/db/ingest.py`):
|
||||
- On startup / deploy: parse `inventory.yaml` → upsert into `entities` + `relations`
|
||||
- Walk `docs/` directory → parse markdown, extract frontmatter → upsert into `knowledge_entities`
|
||||
- Link knowledge entities to inventory entities by path convention (`docs/containers/105-apps.md` → `entity:lxc:apps`)
|
||||
- Idempotent — safe to re-run on every deploy
|
||||
|
||||
- **SOPS recipients:** grant mac-mini's `age_pubkey`
|
||||
(`age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs`) on every
|
||||
secret the moved services decrypt at runtime (secrets-issuance's issuing key,
|
||||
approval HMAC `secrets/oikos-approval-hmac.yaml`, any service creds), then
|
||||
`sops updatekeys`. Edit `.sops.yaml` recipient lists + re-key; commit via
|
||||
`homelab client add mac-mini --finalize-pubkey …` path (comment-preserving).
|
||||
- **SSH keys:** provision on mac-mini the `mcp-reader.key` restricted key the MCP
|
||||
server SSH-proxies to hubris with (`HOMELAB_MCP_SSH_KEY`), plus root SSH access
|
||||
to hubris + strong for the scheduler's disk probe (`_ssh` in
|
||||
`oikos/scheduler.py:102`). Store under `/etc/homelab-mcp/` and `_oikos`'s
|
||||
`~/.ssh` respectively.
|
||||
- **Security note (document):** the age-key *issuer* now runs on a workstation.
|
||||
Record the widened surface in OIKOS.md; least-privilege `_oikos` user +
|
||||
keychain/file perms are the mitigation.
|
||||
**Migration from existing data:**
|
||||
- Write a one-time script to import existing `signals/*.jsonl`, `ledger/*.jsonl`, `approvals/*.jsonl` into the DB
|
||||
- `oikos/state.json` → `state_snapshots` table
|
||||
|
||||
### 6. Inventory / topology truth (same-session rule)
|
||||
### 2. Unified API server (`oikos/api/`, new — merges MCP + homelab CLI)
|
||||
|
||||
- `inventory.yaml`: set `services.{homelab_mcp,secrets_issuance}.backend:
|
||||
mac-mini`; add `oikos_console` + `oikos_scheduler` service entries backed by
|
||||
mac-mini; give `mac-mini` a `runs:` list (homelab-mcp, secrets-issuance,
|
||||
oikos-console, oikos-scheduler); remove `homelab-mcp`/`secrets-issuance` from
|
||||
`apps.runs`. Keep `systemd_unit`→ note launchd label.
|
||||
- Regenerate derived views: `mcp/build_host_files.py` (hosts/*.yaml) and
|
||||
`oikos/gen-topology.py` (cards + `topology.md`). Update
|
||||
`knowledge/wiki/containers/105-apps.md` and add a mac-mini control-node page
|
||||
under `knowledge/wiki/hosts/` (or note on the workstation).
|
||||
One FastAPI service exposing two interfaces:
|
||||
|
||||
### 7. Decommission on apps/105 (operator steps, documented not executed)
|
||||
**MCP interface** (`oikos/api/mcp.py`):
|
||||
- Same tool surface as the current `mcp/server.py` (get_host, list_services, search_knowledge, etc.)
|
||||
- Reads from PostgreSQL instead of filesystem
|
||||
- `search_docs` → `search_knowledge` (queries the knowledge graph)
|
||||
- New tools: `get_entity_relations`, `get_signal_history`, `get_ledger`
|
||||
- Protocol: FastMCP over streamable HTTP (same as current)
|
||||
|
||||
`systemctl disable --now homelab-mcp secrets-issuance oikos-console` + their
|
||||
`-deploy` webhook units on apps; remove `/opt/homelab-mcp`, `/opt/oikos-console`
|
||||
checkouts. Captured in the deploy README as a cutover checklist.
|
||||
**REST interface** (`oikos/api/routes/`):
|
||||
- `GET /api/v1/hosts` — list hosts (replaces `homelab list`)
|
||||
- `GET /api/v1/services` — list services
|
||||
- `GET /api/v1/hosts/{name}` — host detail (replaces `homelab status`)
|
||||
- `GET /api/v1/services/{name}/health` — service health (cache-first, `?live=true` for probe)
|
||||
- `GET /api/v1/signals` — list signals (`?state=raised&severity=warning`)
|
||||
- `POST /api/v1/signals/{id}/ack` — acknowledge
|
||||
- `POST /api/v1/signals/{id}/resolve` — resolve
|
||||
- `GET /api/v1/approvals` — pending approvals
|
||||
- `POST /api/v1/approvals/{id}/decide` — approve/deny (operator only, Authentik-gated)
|
||||
- `POST /api/v1/exec` — gated execution (the actuator entrypoint)
|
||||
- Body: `{entity, action, approval_id?}`
|
||||
- Classifies via `decide.py`, checks approval if needed, executes, records ledger
|
||||
- `POST /api/v1/deploy/{service}` — trigger a service deploy
|
||||
- `WS /api/v1/events` — real-time stream (signals, approvals, ledger entries)
|
||||
- `GET /api/v1/knowledge/{entity_id}` — knowledge graph query for an entity
|
||||
- `GET /api/v1/knowledge/search?q=...` — search knowledge graph
|
||||
|
||||
### 8. Docs
|
||||
**Policy enforcement** is built into the API:
|
||||
- Every mutating endpoint classifies the action via `oikos/decide.py`
|
||||
- `config_mutation`/`destructive` actions require a valid approval ID
|
||||
- All mutations write to the ledger automatically
|
||||
|
||||
`.agents/OIKOS.md` (control node = mac-mini; loop closed; risks),
|
||||
`auto-deploy.md` (pull-sync), `deploy/macos/README.md` (cutover runbook),
|
||||
`.agents/skills/autopilot-pass/SKILL.md` (supervised actuator run), host/service
|
||||
cards (regenerated).
|
||||
**Auth:**
|
||||
- MCP interface: no auth (internal, container-to-container)
|
||||
- REST interface: Authentik OIDC forward-auth (via Caddy) for operator endpoints
|
||||
- Internal service-to-service: shared secret / mTLS (Docker network)
|
||||
|
||||
## Critical files
|
||||
### 3. Scheduler + Actuator (`oikos/scheduler/`, `oikos/actuator/`)
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `deploy/macos/*.plist` + `install-macos.sh` + `README.md` | **New** — launchd runtime + cutover |
|
||||
| `oikos/act.py` | **New** — actuator (Act stage) |
|
||||
| `oikos/policy.yaml`, `oikos/policy.py` | Autonomy kill-switch + loaders |
|
||||
| `oikos/systemd/run-scheduler.sh` → macOS variant | Observe+Act pass, commit signals/+ledger/ |
|
||||
| `inventory.yaml` | Move service backends apps→mac-mini; mac-mini `runs:` |
|
||||
| `hosts/*.yaml`, `oikos/cards/*`, `topology.md` | Regenerated |
|
||||
| `dtoro/caddy-conf` (external repo) | 3 reverse_proxy backends → mac-mini mesh |
|
||||
| `.sops.yaml` | Grant mac-mini recipient on moved-service secrets; re-key |
|
||||
| `bin/homelab` | Optional `autopilot` subcommand |
|
||||
| Docs: `OIKOS.md`, `auto-deploy.md`, `105-apps.md` | Reflect consolidation |
|
||||
**Scheduler (Observe)** — adapted from `oikos/scheduler.py`:
|
||||
- Runs as a Docker service on a timer (10-min interval via a loop or cron sidecar)
|
||||
- Probes service health (HTTP), disk usage (SSH to hubris/strong), drift detection
|
||||
- Writes signals to PostgreSQL (not JSONL files)
|
||||
- Writes state snapshots to DB
|
||||
- No more git commit/push of signals — they're in the DB
|
||||
|
||||
**Actuator (Act)** — new, `oikos/actuator/act.py`:
|
||||
- Reads open signals from DB where `recommended_action` is set
|
||||
- For each: classify via `decide.py`
|
||||
- `auto-act` (reversible_low, contained, confident) → execute via SSH/API, verify, resolve signal, write ledger
|
||||
- `escalate` → create approval request, notify via Matrix, acknowledge signal
|
||||
- Loop-guard: check ledger history per (entity, action) to cap auto-retries
|
||||
- Autonomy kill-switch: `policy.yaml` → `autonomy.auto_act: off|reversible_low`, `never_auto_act: [entity list]`
|
||||
- **Phase 1**: direct SSH execution (keys mounted)
|
||||
- **Phase 2**: execution through the API's `/exec` endpoint (gateway pattern)
|
||||
|
||||
### 4. Hermes agent container (`compose/hermes/`)
|
||||
|
||||
**What it is:**
|
||||
- Hermes Agent runtime in a Docker container
|
||||
- Runs in gateway mode — workstations connect to it remotely
|
||||
- Config: `hermes/config.yaml` (providers, models, gateway port)
|
||||
- Persona: `hermes/SOUL.md` (homelab-specific)
|
||||
- Skills: `hermes/skills/homelab-ops/SKILL.md` (how to interact with the homelab)
|
||||
|
||||
**What it has access to:**
|
||||
- **Oikos API** via MCP (container network, no auth needed)
|
||||
- **Infisical** for secret retrieval (API token)
|
||||
- **SSH keys** mounted from host (`~/.ssh` → `/root/.ssh:ro`) — for direct host access (hybrid approach)
|
||||
- **Hermes data volume** — persistent state, session history
|
||||
|
||||
**How workstations connect:**
|
||||
- Hermes gateway listens on port 8092
|
||||
- Any workstation with Hermes CLI can connect: `hermes gateway connect mac-mini:8092`
|
||||
- Or via Caddy ingress: `hermes.hubris.network` → mac-mini:8092 (mesh)
|
||||
|
||||
**Homelab skills** (loaded into Hermes):
|
||||
- `homelab-ops` — how to use the Oikos API, classify actions, request approvals
|
||||
- `homelab-deploy` — how to deploy/manage services
|
||||
- `homelab-troubleshoot` — diagnostic procedures
|
||||
- (Future skills added as needed)
|
||||
|
||||
### 5. Infisical secrets migration
|
||||
|
||||
**Phase 1 — Stand up Infisical:**
|
||||
- Add Infisical to the Docker stack
|
||||
- Configure with PostgreSQL backend (shared with Oikos DB or separate DB)
|
||||
- Bootstrap admin credentials, create `hubris` project + environments (prod, staging)
|
||||
|
||||
**Phase 2 — Migrate secrets:**
|
||||
- Write a migration script: read all `secrets/*.yaml` (SOPS-encrypted), decrypt with age key, import into Infisical
|
||||
- Map each secret to Infisical's key-value structure
|
||||
- Verify all secrets are imported
|
||||
|
||||
**Phase 3 — Wire services:**
|
||||
- All OS services authenticate to Infisical via machine identity (service token)
|
||||
- Services fetch secrets at startup or on-demand
|
||||
- Remove `secrets/` directory and `.sops.yaml` once verified
|
||||
- The age key on mac-mini stays for one-time decryption during migration, then is retired
|
||||
|
||||
**Phase 4 — Retire SOPS:**
|
||||
- Remove SOPS dependencies from the repo
|
||||
- Update inventory to reflect Infisical as the secret store
|
||||
- Document the new secret access pattern
|
||||
|
||||
### 6. Knowledge graph ingestion (`oikos/knowledge/`)
|
||||
|
||||
**What it does:**
|
||||
- On deploy, walks `docs/` directory
|
||||
- Parses each markdown file:
|
||||
- Extracts frontmatter (if any) for metadata
|
||||
- Infers entity relationships from path conventions:
|
||||
- `docs/containers/105-apps.md` → links to `entity:lxc:apps`
|
||||
- `docs/hosts/hubris.md` → links to `entity:host:hubris`
|
||||
- `docs/infrastructure/dns.md` → links to `entity:service:dns`
|
||||
- Extracts cross-references (links to other docs) → `knowledge_relations`
|
||||
- Tags by directory: container-doc, host-doc, infrastructure, investigation, plan
|
||||
- Upserts into `knowledge_entities` + `knowledge_relations` tables
|
||||
|
||||
**Agent access:**
|
||||
- MCP tool `search_knowledge(query)` — full-text search on knowledge graph
|
||||
- MCP tool `get_entity_knowledge(entity_id)` — all docs related to an entity
|
||||
- MCP tool `get_relations(entity_id)` — ontology blast-radius query (existing, adapted to DB)
|
||||
|
||||
**Docs as source, DB as query layer:**
|
||||
- Humans still write markdown docs in `docs/`
|
||||
- The DB is the query/index layer — agents never grep files
|
||||
- On deploy, docs are re-ingested (idempotent)
|
||||
- Future: wiki-style editing UI that writes back to `docs/` via git
|
||||
|
||||
### 7. Notifier abstraction (`oikos/notifier/`)
|
||||
|
||||
**Interface:**
|
||||
```python
|
||||
class Notifier(Protocol):
|
||||
def send_alert(self, signal: Signal) -> None: ...
|
||||
def send_approval_request(self, approval: Approval) -> None: ...
|
||||
def listen_for_decisions(self) -> Iterator[ApprovalDecision]: ...
|
||||
```
|
||||
|
||||
**Matrix implementation** (current):
|
||||
- Sends alerts to `@dtoro:avispero` via Synapse (LXC 118)
|
||||
- Approval requests as messages with ✅/❌ reactions
|
||||
- Listens for reactions to record decisions
|
||||
- Reuses existing `oikos/approve.py` Matrix delivery logic
|
||||
|
||||
**Future implementations** (pluggable):
|
||||
- Webhook, email, Slack, etc.
|
||||
- Registered in config, selected at runtime
|
||||
|
||||
### 8. Docker build + deploy pipeline
|
||||
|
||||
**Build:**
|
||||
- `docker compose build` from the repo root
|
||||
- Each service has a Dockerfile in `compose/<service>/`
|
||||
- Shared base image: Python 3.12-slim + common deps (postgresql libs, etc.)
|
||||
- Multi-stage builds for smaller images
|
||||
|
||||
**Deploy trigger:**
|
||||
- Gitea webhook on push to `main` → hits a deploy endpoint on mac-mini
|
||||
- Deploy script: `git pull && docker compose build && docker compose up -d`
|
||||
- Alternatively: a "deployer" sidecar container watches Gitea and triggers rebuilds
|
||||
- The existing 5-min launchd git pull can serve as a fallback deploy mechanism initially
|
||||
|
||||
**Health checks:**
|
||||
- Each service has a Docker healthcheck
|
||||
- `docker compose ps` shows health status
|
||||
- API exposes `/health` for external monitoring
|
||||
|
||||
### 9. Ingress re-point (Caddy)
|
||||
|
||||
- In `dtoro/caddy-conf`: point `mcp.hubris.network` and `oikos.hubris.network` to
|
||||
mac-mini's mesh address at port 8090 (API)
|
||||
- Future: `hermes.hubris.network` → mac-mini:8092 (Hermes gateway)
|
||||
- DNS and public URLs unchanged
|
||||
- Verify Caddy → mac-mini reachability over netbird first (same as original plan)
|
||||
|
||||
### 10. mac-mini host setup
|
||||
|
||||
**Directory:**
|
||||
- New directory: `~/oikos-os/` (or `/opt/oikos-os/`) — the repo clone
|
||||
- This is where `docker compose` runs from
|
||||
- Existing `/opt/homelab-context/` stays untouched until cleanup phase
|
||||
|
||||
**Prerequisites:**
|
||||
- Docker Desktop (or OrbStack) — already installed
|
||||
- SSH keys accessible at `~/.ssh/` (mounted into containers)
|
||||
- Age key at `/etc/age/key.txt` (for one-time SOPS migration)
|
||||
|
||||
**Cleanup (deferred, after Docker OS is proven):**
|
||||
- Stop launchd git-sync for `/opt/homelab-context`
|
||||
- Remove native Hermes install at `~/.hermes/hermes-agent/`
|
||||
- Remove `/opt/homelab-context` clone
|
||||
- Remove any launchd plists for homelab services
|
||||
|
||||
### 11. Decommission apps/105 (deferred)
|
||||
|
||||
Keep apps/105 running all Oikos services (homelab-mcp, secrets-issuance,
|
||||
oikos-console) as a fallback until the Docker OS is proven end-to-end.
|
||||
|
||||
**Cutover checklist (executed when ready):**
|
||||
1. Verify Docker OS serves all traffic correctly
|
||||
2. `systemctl disable --now homelab-mcp secrets-issuance oikos-console` on apps/105
|
||||
3. Remove `/opt/homelab-mcp`, `/opt/oikos-console` checkouts on apps/105
|
||||
4. Update Caddy backends to point to mac-mini exclusively
|
||||
5. Remove Gitea webhooks 10, 11, 14 (or repoint to Docker deploy trigger)
|
||||
|
||||
## Reuse (do not reimplement)
|
||||
|
||||
- Deploy scaffolding shape: `mcp/deploy/deploy.sh`, `oikos/console/deploy/`.
|
||||
- Service definitions to port: `mcp/deploy/homelab-mcp.service`,
|
||||
`oikos/console/deploy/oikos-console.service`, `oikos/systemd/*`.
|
||||
- Actuator primitives: `oikos_decide.classify` (`decide.py:64`), signal
|
||||
transitions incl. unused `start_acting` (`signal.py:125`), `cmd_restart` +
|
||||
`_record_change` (`bin/homelab:603,190`), `oikos_approve.request`
|
||||
(`bin/homelab:1370`), `oikos_ledger.history` (`ledger.py:59`).
|
||||
- Comment-preserving inventory + `.sops.yaml` edits + `sops updatekeys`:
|
||||
`homelab client add --finalize-pubkey` path in `bin/homelab`.
|
||||
| Existing code | What it becomes |
|
||||
|---|---|
|
||||
| `oikos/decide.py` | Risk classifier — adapted to read from DB, logic unchanged |
|
||||
| `oikos/signal.py` | Signal engine — adapted to DB, lifecycle logic unchanged |
|
||||
| `oikos/approve.py` | Approval engine — adapted to DB, Matrix delivery reused |
|
||||
| `oikos/ledger.py` | Ledger — adapted to DB |
|
||||
| `oikos/policy.py` + `policy.yaml` | Policy loader — unchanged, still reads YAML |
|
||||
| `oikos/drift.py` | Drift detectors — adapted to write signals to DB |
|
||||
| `oikos/relations.py` | Ontology graph walk — adapted to DB |
|
||||
| `oikos/report.py` | Report generator — adapted to read from DB |
|
||||
| `mcp/server.py` | Merged into `oikos/api/` — MCP protocol adapter on top of DB |
|
||||
| `bin/homelab` logic | Migrated into API routes — same operations, REST interface |
|
||||
| `oikos/scheduler.py` | Adapted — probes unchanged, output goes to DB |
|
||||
| `oikos/approve.py` Matrix delivery | Reused as the Matrix notifier implementation |
|
||||
|
||||
## Risks / assumptions
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Caddy → mac-mini reachability** over netbird is unverified — workstream 4
|
||||
verifies before cutover; LAN 178.x is the fallback (hubris is dual-homed).
|
||||
- **Lost sandboxing** on macOS (no `ProtectSystem`/`ReadOnlyPaths`) — mitigated
|
||||
by a least-privilege `_oikos` daemon user, not eliminated.
|
||||
- **secrets-issuance on a workstation** widens the security surface — accepted
|
||||
per scope decision; documented.
|
||||
- **Single-checkout race:** a git pull mid-request could serve half-updated code
|
||||
— post-pull `kickstart -k` restarts services after the pull completes.
|
||||
- macOS `/opt/homelab-context` already exists (mac-mini is enrolled); daemons
|
||||
need it readable by `_oikos`.
|
||||
- **Hermes in Docker** — the agent's "world" is the container. Direct file access
|
||||
to the host is limited to mounted volumes. SSH access is the bridge to the homelab.
|
||||
Trade-off: less host integration, more isolation. Mitigated by the hybrid approach
|
||||
(SSH now, actuator gateway later).
|
||||
- **PostgreSQL as single point of failure** — if Postgres goes down, the OS loses
|
||||
all state. Mitigated by Docker volume persistence + regular `pg_dump` backups
|
||||
(can be automated by the scheduler itself once running).
|
||||
- **Infisical bootstrapping** — migrating from SOPS to Infisical has a transition
|
||||
period where both systems coexist. Keep SOPS as fallback until all services are
|
||||
verified against Infisical.
|
||||
- **Knowledge graph ingestion quality** — markdown-to-structured-data conversion
|
||||
relies on path conventions and frontmatter. Docs without frontmatter may lose
|
||||
metadata. Start simple, improve extraction over time.
|
||||
- **Deploy downtime** — `docker compose up -d` may briefly interrupt services during
|
||||
container replacement. Use `--scale` + health-gated rollout for zero-downtime
|
||||
when needed (later).
|
||||
- **Multi-agent concurrency** — PostgreSQL handles concurrent writers correctly,
|
||||
but the actuator needs a lock mechanism to prevent two passes from acting on
|
||||
the same signal simultaneously. Use `SELECT ... FOR UPDATE SKIP LOCKED`.
|
||||
|
||||
## Verification (end to end)
|
||||
|
||||
1. **Runtime:** `install-macos.sh` on mac-mini; `launchctl print system/…` shows
|
||||
all four daemons running; `curl 127.0.0.1:{9810,9820,8091}` responds locally.
|
||||
2. **Ingress:** after the caddy-conf push + reload, `curl -I
|
||||
https://mcp.hubris.network/mcp`, `.../` (console → 302 Authentik),
|
||||
`secrets.hubris.network/health` → 200/302 through Caddy to mac-mini.
|
||||
3. **Observe+Act loop:** trigger a `service-down` Signal; confirm the launchd
|
||||
scheduler pass runs it `raised→acting→resolved`, writes a `restart`/`result:
|
||||
ok` ledger entry, and pushes `signals/`+`ledger/`. Kill-switch: set
|
||||
`autonomy.auto_act: off` → next pass escalates instead.
|
||||
4. **Deploy-by-pull:** `git push` a trivial change; within 5 min mac-mini pulls
|
||||
and `kickstart`s the changed daemon (no webhook involved).
|
||||
5. **Cutover:** after decommission, `systemctl is-active` on apps shows the three
|
||||
services `inactive`; production traffic served only by mac-mini; MCP tools +
|
||||
console + a test enrollment all still work.
|
||||
1. **Database:** `docker compose up postgres` — `psql` shows all tables; ingest
|
||||
script populates entities from `inventory.yaml` and knowledge from `docs/`.
|
||||
2. **API:** `curl http://localhost:8090/api/v1/hosts` returns the fleet;
|
||||
MCP `list_services` tool works via the same endpoint.
|
||||
3. **Scheduler:** trigger a probe pass — signals appear in DB, state snapshot written.
|
||||
4. **Actuator:** raise a test `service-down` signal → actuator classifies →
|
||||
auto-acts (restart) or escalates (Matrix approval request) → ledger entry written.
|
||||
5. **Hermes:** connect from another workstation via `hermes gateway connect` →
|
||||
agent responds, can query the API via MCP, can SSH to hubris.
|
||||
6. **Secrets:** Infisical running, services fetch secrets at startup, SOPS files
|
||||
removed without breakage.
|
||||
7. **Deploy:** `git push` → Gitea webhook → `docker compose build + up -d` →
|
||||
changes live within minutes.
|
||||
8. **Knowledge:** MCP `search_knowledge("caddy")` returns relevant docs linked
|
||||
to `entity:service:caddy`.
|
||||
9. **Notifier:** raise a `critical` signal → Matrix message received →
|
||||
✅ reaction → approval granted in DB.
|
||||
10. **Cutover:** after all above passes, stop services on apps/105, verify
|
||||
production traffic served only by Docker OS on mac-mini.
|
||||
|
||||
## Out of scope (per prior direction)
|
||||
## Phasing
|
||||
|
||||
SSH-signed approval requests, Authentik step-up on the console POST, Prometheus /
|
||||
temperature trend signals, and the generalize-to-any-homelab refactor.
|
||||
**Phase 1 — Foundation (DB + API):**
|
||||
- PostgreSQL schema + models + migrations
|
||||
- Ingest pipeline (inventory.yaml → DB, docs → knowledge graph)
|
||||
- Unified API server (REST + MCP) reading from DB
|
||||
- Migrate existing signals/ledger/approvals data into DB
|
||||
|
||||
**Phase 2 — Services (Scheduler + Actuator + Notifier):**
|
||||
- Adapt scheduler to read/write DB
|
||||
- Build actuator (Act stage) with kill-switch
|
||||
- Matrix notifier (abstracted interface + impl)
|
||||
- SSH-based execution (hybrid approach)
|
||||
|
||||
**Phase 3 — Agent (Hermes container):**
|
||||
- Hermes Docker image
|
||||
- Gateway mode config
|
||||
- Homelab skills (homelab-ops)
|
||||
- Connect from workstation, verify MCP + SSH access
|
||||
|
||||
**Phase 4 — Secrets (Infisical):**
|
||||
- Stand up Infisical in the stack
|
||||
- Migrate SOPS secrets
|
||||
- Wire all services to Infisical
|
||||
- Retire SOPS
|
||||
|
||||
**Phase 5 — Deploy pipeline:**
|
||||
- Docker Compose fully working
|
||||
- Gitea webhook → rebuild trigger
|
||||
- Caddy ingress re-point to mac-mini
|
||||
- Health checks + monitoring
|
||||
|
||||
**Phase 6 — Cutover + cleanup:**
|
||||
- Verify end-to-end
|
||||
- Stop apps/105 services
|
||||
- Clean up mac-mini old artifacts
|
||||
- Remove old deploy webhooks
|
||||
|
||||
## Out of scope (for now)
|
||||
|
||||
- Oikos Console web UI (deferred — built later on top of the API for specific tasks)
|
||||
- Multi-node deployment (designed for, not implemented)
|
||||
- Vector embeddings / semantic search (knowledge graph is structured-only for now)
|
||||
- SSH-key-signed approval requests (same backlog as before)
|
||||
- Prometheus / trend signals
|
||||
- Actuator gateway pattern (Phase 2 of the hybrid — start with mounted SSH)
|
||||
- Custom homelab skills beyond `homelab-ops` (add as needed)
|
||||
|
||||
Reference in New Issue
Block a user