test: e2e client lifecycle + ADRs with sequence diagrams
- client_lifecycle_test.go: full end-to-end integration test planned → provisioning (enroll) → active → migrating → active → deprecated → failed. Validates age keypair generation, attrs, context/secrets endpoints, invalid transition blocking, compute entity provisioning with relationship edges and status tracking. Also tests enrollment rejection for invalid states and duplicate slug rejection for provisioning. - adr/0011-client-lifecycle-flows.md: workstation self-enrollment, compute entity provisioning, deprecation/destruction flows with Mermaid sequence diagrams. Full lifecycle state diagram. Transition check enforcement documentation. - adr/0012-hermes-oikos-interactions.md: Hermes ↔ Oikos interaction flow through OODA loop phases. Thin client bootstrap. Internal component interactions (scheduler, actuator, notifier). Complete 30-tool ownership matrix. - Fix: migration 012 FK reference (executions.id → executions.entity_id) - Fix: provision handler null attributes JSONB - Fix: provisioning steps use entity_id for execution FK All 3 integration tests pass, go vet clean.
This commit is contained in:
233
docs/adr/0012-hermes-oikos-interactions.md
Normal file
233
docs/adr/0012-hermes-oikos-interactions.md
Normal file
@@ -0,0 +1,233 @@
|
||||
# ADR 0012 — Hermes/Oikos interaction architecture
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-08
|
||||
|
||||
## Context
|
||||
|
||||
Hermes (the AI agent) is the primary operator interface for the hubris
|
||||
homelab. It communicates with Oikos via the MCP protocol. The MCP tools
|
||||
map to the OODA loop phases (Observe, Orient, Decide, Act). A thin client
|
||||
model distributes agent context via API deltas instead of git clones.
|
||||
|
||||
## Decision
|
||||
|
||||
Hermes interacts with Oikos through three surface layers: MCP tools
|
||||
(agent-facing), REST API endpoints (operator-facing and agent-facing), and
|
||||
the SSH actuator (internal). All read paths go through the Postgres DB as
|
||||
the single source of truth. All writes go through the API with audit
|
||||
logging and policy classification.
|
||||
|
||||
## Hermes → Oikos interaction flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant H as Hermes (AI Agent)
|
||||
participant MCP as MCP Server (:8090/mcp)
|
||||
participant API as REST API (:8090/api/v1)
|
||||
participant DB as Postgres (TimescaleDB)
|
||||
participant Act as Actuator
|
||||
participant PVE as Proxmox Hosts
|
||||
participant Matrix as Matrix Notifier
|
||||
|
||||
Note over H,Matrix: ── OODA: Observe ──
|
||||
|
||||
H->>MCP: get_entity("service:caddy")
|
||||
MCP->>DB: SELECT * FROM entities WHERE slug=$1
|
||||
DB-->>MCP: {slug, type, state, health, attrs}
|
||||
MCP-->>H: entity record
|
||||
|
||||
H->>MCP: get_state_snapshot()
|
||||
MCP->>DB: SELECT e.slug, st.health, st.disk_usage_pct FROM entities e LEFT JOIN entity_status st
|
||||
DB-->>MCP: [{slug, health, disk_pct, drift_count}, ...]
|
||||
MCP-->>H: fleet health snapshot
|
||||
|
||||
H->>MCP: search_knowledge("jellyfin hardware acceleration")
|
||||
MCP->>DB: SELECT ... WHERE search @@ to_tsquery('jellyfin & hardware & acceleration')
|
||||
DB-->>MCP: [documents, runbooks]
|
||||
MCP-->>H: ranked FTS results
|
||||
|
||||
H->>MCP: get_blast_radius("service:caddy")
|
||||
MCP->>DB: SELECT blast_radius($1, 3) -- recursive CTE
|
||||
DB-->>MCP: [{entity, depth}, ...]
|
||||
MCP-->>H: what breaks if caddy goes down
|
||||
|
||||
Note over H,Matrix: ── OODA: Orient ──
|
||||
|
||||
H->>MCP: explain("lxc:jellyfin")
|
||||
MCP->>DB: SELECT e.*, st.health, st.last_check FROM entities e LEFT JOIN entity_status st
|
||||
MCP->>DB: SELECT r.type, se.slug, te.slug FROM relationships r WHERE ...
|
||||
DB-->>MCP: compact context card
|
||||
MCP-->>H: {type, state, health, relations, version, updated_at}
|
||||
|
||||
H->>MCP: preflight("lxc:jellyfin", "restart")
|
||||
MCP->>DB: SELECT risk_class, approval FROM classification_for($1, $2)
|
||||
DB-->>MCP: {risk_class: "reversible_low", approval: "auto-act"}
|
||||
MCP-->>H: safe to auto-act
|
||||
|
||||
H->>MCP: preflight("lxc:jellyfin", "deploy")
|
||||
MCP->>DB: ...
|
||||
DB-->>MCP: {risk_class: "config_mutation", approval: "operator-approval"}
|
||||
MCP-->>H: needs operator approval
|
||||
|
||||
Note over H,Matrix: ── OODA: Decide ──
|
||||
|
||||
alt reversible_low (auto-act)
|
||||
H->>MCP: request_execution("lxc:caddy", "restart")
|
||||
MCP->>API: POST /executions {action:"restart", target:"lxc:caddy"}
|
||||
API->>DB: INSERT executions (auto_approved)
|
||||
API->>Act: queue execution
|
||||
Act->>PVE: ssh systemctl restart caddy
|
||||
Act->>DB: UPDATE execution status→completed
|
||||
MCP-->>H: execution {status: completed}
|
||||
else config_mutation (escalate)
|
||||
H->>MCP: request_execution("lxc:jellyfin", "deploy")
|
||||
MCP->>API: POST /executions
|
||||
API->>DB: INSERT executions (proposed, needs approval)
|
||||
API->>Matrix: send approval request via notifier
|
||||
Matrix->>Operator: "Approve deploy lxc:jellyfin? ✅/❌"
|
||||
Operator->>Matrix: ✅
|
||||
Matrix->>API: POST /approvals/{id}/approve
|
||||
API->>DB: UPDATE execution status→approved
|
||||
API->>Act: queue execution
|
||||
Act->>PVE: run deploy procedure
|
||||
MCP-->>H: execution {status: completed}
|
||||
end
|
||||
|
||||
Note over H,Matrix: ── OODA: Act (mutation gated) ──
|
||||
|
||||
H->>MCP: tail_log("caddy", lines=200)
|
||||
MCP->>PVE: ssh journalctl -u caddy -n 200
|
||||
PVE-->>MCP: log lines
|
||||
MCP-->>H: caddy logs
|
||||
|
||||
H->>MCP: get_service_status("caddy")
|
||||
MCP->>PVE: ssh systemctl show caddy
|
||||
PVE-->>MCP: {ActiveState, SubState, ...}
|
||||
MCP-->>H: service status
|
||||
|
||||
H->>MCP: list_lxcs()
|
||||
MCP->>DB: SELECT * FROM entity_status WHERE type='lxc'
|
||||
DB-->>MCP: [lxc:caddy, lxc:jellyfin, ...]
|
||||
MCP-->>H: all LXCs with state
|
||||
|
||||
H->>MCP: get_lxc_state("lxc:caddy")
|
||||
MCP->>PVE: ssh pct status {vmid} --verbose
|
||||
PVE-->>MCP: RAM, CPU, disk, uptime
|
||||
MCP-->>H: LXC resource state
|
||||
```
|
||||
|
||||
## Thin client bootstrap flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant New as New Client (bare machine)
|
||||
participant Gitea as Gitea (raw URL)
|
||||
participant API as Oikos API
|
||||
participant DB as Postgres
|
||||
|
||||
New->>Gitea: curl bootstrap.sh
|
||||
Gitea-->>New: bootstrap.sh
|
||||
|
||||
New->>Gitea: fetch CLIENTS.md, AGENTS.md, OIKOS.md, tools/
|
||||
Gitea-->>New: agent orientation files
|
||||
|
||||
New->>API: POST /clients/enroll {slug, hostname, mesh_ip}
|
||||
API->>DB: validate state, mesh IP
|
||||
API->>API: generate age keypair
|
||||
API->>DB: store pubkey, transition→provisioning
|
||||
API-->>New: {age_private_key, age_public_key, infisical_identity}
|
||||
|
||||
New->>New: write /etc/age/key.txt, /etc/infisical/identity
|
||||
New->>New: install context-poller (launchd/systemd, every 5min)
|
||||
|
||||
loop Every 5 minutes
|
||||
New->>API: GET /clients/ws:{hostname}/context?since={timestamp}
|
||||
API->>DB: SELECT files changed since {timestamp}
|
||||
API-->>New: {agent_files_changed, sops_config_changed, tools_changed}
|
||||
New->>Gitea: fetch only changed files
|
||||
end
|
||||
```
|
||||
|
||||
## Internal Oikos component interactions
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Sched as Scheduler
|
||||
participant API as API Server
|
||||
participant Notif as Notifier
|
||||
participant Act as Actuator
|
||||
participant DB as Postgres
|
||||
|
||||
Note over Sched,DB: The OODA loop (internal)
|
||||
|
||||
Sched->>DB: probe endpoints (HTTP, TCP, disk, cert-expiry)
|
||||
DB-->>Sched: results
|
||||
Sched->>DB: INSERT signals (dedup, flap suppression)
|
||||
Sched->>DB: UPDATE entity_status (health, disk, drift_count)
|
||||
|
||||
Act->>DB: poll executions with status=auto_approved
|
||||
DB-->>Act: pending executions
|
||||
Act->>Act: circuit breaker check
|
||||
Act->>SSH: execute procedure
|
||||
Act->>DB: UPDATE execution status+result
|
||||
|
||||
Notif->>DB: poll pending approvals
|
||||
DB-->>Notif: [approval requests]
|
||||
Notif->>Matrix: send approval messages
|
||||
|
||||
API->>DB: INSERT audit_log (every mutation)
|
||||
API->>DB: NOTIFY oikos_events (SSE streaming)
|
||||
```
|
||||
|
||||
## Tool ownership matrix
|
||||
|
||||
| Tool | Interface | Package | DB query | SSH |
|
||||
|------|-----------|---------|----------|-----|
|
||||
| `get_entity` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `list_entities` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `get_relations` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `get_blast_radius` | MCP + REST | both | CTE function | — |
|
||||
| `search_knowledge` | MCP | `internal/mcp/server.go` | FTS query | — |
|
||||
| `get_health_summary` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `whoami` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `explain` | MCP | `internal/mcp/server.go` | DIRECT + JOIN | — |
|
||||
| `preflight` | MCP | `internal/mcp/server.go` | CASE expression | — |
|
||||
| `get_change_history` | MCP | `internal/mcp/server.go` | audit_log query | — |
|
||||
| `get_state_snapshot` | MCP | `internal/mcp/server.go` | DIRECT + JOIN | — |
|
||||
| `list_my_secrets` | MCP | `internal/mcp/server.go` | attributes query | — |
|
||||
| `tail_log` | MCP | `internal/mcp/server.go` | — | journalctl |
|
||||
| `get_service_status` | MCP | `internal/mcp/server.go` | — | systemctl show |
|
||||
| `list_lxcs` | MCP | `internal/mcp/server.go` | entity_status query | — |
|
||||
| `get_lxc_state` | MCP | `internal/mcp/server.go` | relationship query | pct status |
|
||||
| `ping_service` | MCP | `internal/mcp/server.go` | — | probe |
|
||||
| `request_execution` | MCP | `internal/mcp/server.go` | executions INSERT | SSH via actuator |
|
||||
| `get_agent_activity` | MCP | `internal/mcp/server.go` | agent_activity query | — |
|
||||
| `get_signal_history` | MCP | `internal/mcp/server.go` | signals query | — |
|
||||
| `get_patterns` | MCP | `internal/mcp/server.go` | patterns query | — |
|
||||
| `get_skills` | MCP | `internal/mcp/server.go` | skills query | — |
|
||||
| `get_audit_trail` | MCP | `internal/mcp/server.go` | audit_log query | — |
|
||||
| `get_trend` | MCP | `internal/mcp/server.go` | metrics query | — |
|
||||
| `get_event_timeline` | MCP | `internal/mcp/server.go` | events query | — |
|
||||
| `query_metrics` | MCP | `internal/mcp/server.go` | metric_samples query | — |
|
||||
| `enroll` | REST | `internal/httpapi/impl.go` | entities+audit+events | — |
|
||||
| `context` | REST | `internal/httpapi/impl.go` | context_files query | — |
|
||||
| `secrets` | REST | `internal/httpapi/impl.go` | secrets.Manager.List | — |
|
||||
| `provision` | REST | `internal/httpapi/impl.go` | entities+steps+relations | SSH via actuator |
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Hermes is the primary operator interface.** All operator actions flow
|
||||
through Hermes → MCP → Oikos. The old `bin/homelab` CLI is dead.
|
||||
- **MCP tools are read-only by design.** Mutations go through
|
||||
`request_execution`, which is policy-gated and requires operator
|
||||
approval for `config_mutation` and `destructive` actions.
|
||||
- **The actuator holds the SSH key.** Hermes has no direct SSH access.
|
||||
The security boundary is Hermes → MCP → API → execution queue →
|
||||
actuator → SSH.
|
||||
- **Thin clients poll for context deltas.** No git clones on workstations.
|
||||
The 5-minute poll replaces `git pull` with HTTP queries to
|
||||
`GET /clients/{slug}/context`.
|
||||
- **The DB is the single source of truth.** All state transitions,
|
||||
audit entries, and event emissions go through Postgres. The scheduler,
|
||||
actuator, notifier, and API all read/write the same tables.
|
||||
Reference in New Issue
Block a user