test: e2e client lifecycle + ADRs with sequence diagrams
- client_lifecycle_test.go: full end-to-end integration test planned → provisioning (enroll) → active → migrating → active → deprecated → failed. Validates age keypair generation, attrs, context/secrets endpoints, invalid transition blocking, compute entity provisioning with relationship edges and status tracking. Also tests enrollment rejection for invalid states and duplicate slug rejection for provisioning. - adr/0011-client-lifecycle-flows.md: workstation self-enrollment, compute entity provisioning, deprecation/destruction flows with Mermaid sequence diagrams. Full lifecycle state diagram. Transition check enforcement documentation. - adr/0012-hermes-oikos-interactions.md: Hermes ↔ Oikos interaction flow through OODA loop phases. Thin client bootstrap. Internal component interactions (scheduler, actuator, notifier). Complete 30-tool ownership matrix. - Fix: migration 012 FK reference (executions.id → executions.entity_id) - Fix: provision handler null attributes JSONB - Fix: provisioning steps use entity_id for execution FK All 3 integration tests pass, go vet clean.
This commit is contained in:
205
docs/adr/0011-client-lifecycle-flows.md
Normal file
205
docs/adr/0011-client-lifecycle-flows.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# ADR 0011 — Client lifecycle sequence diagrams
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-08
|
||||
|
||||
## Context
|
||||
|
||||
The Oikos client lifecycle spans two distinct onboarding paths — workstation
|
||||
self-enrollment and compute entity (LXC/VM) provisioning — sharing the same
|
||||
infrastructure lifecycle state machine (`planned → provisioning → active →
|
||||
migrating → deprecated → destroyed`). These flows must be documented and
|
||||
validated against the Go implementation.
|
||||
|
||||
## Decision
|
||||
|
||||
All machines in the homelab follow the same lifecycle state machine defined
|
||||
in `seeds/ontology.yaml` under the `infrastructure` lifecycle. The
|
||||
implementation lives in `internal/httpapi/impl.go` (handlers),
|
||||
`internal/actuator/actuator.go` (SSH provisioning), and
|
||||
`internal/ontology/validate.go` (transition check enforcement).
|
||||
|
||||
## Workstation self-enrollment flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Op as Operator
|
||||
participant BS as bootstrap.sh
|
||||
participant API as Oikos API (:8090)
|
||||
participant DB as Postgres
|
||||
participant Inf as Infisical
|
||||
|
||||
Op->>API: POST /entities {slug:"ws:laptop", type:"workstation", state:"planned"}
|
||||
API->>DB: INSERT entities (planned)
|
||||
API-->>Op: 201 + ETag
|
||||
|
||||
Op->>BS: curl bootstrap.sh | sudo bash
|
||||
BS->>BS: detect hostname, mesh IP
|
||||
BS->>BS: fetch CLIENTS.md, AGENTS.md, OIKOS.md, tools/
|
||||
BS->>BS: install age, curl, jq
|
||||
|
||||
BS->>API: POST /clients/enroll {slug, hostname, mesh_ip}
|
||||
API->>DB: validate state = planned|provisioning
|
||||
API->>API: generateAgeKeypair()
|
||||
API->>API: store age key in Infisical (best-effort)
|
||||
API->>DB: UPDATE state→provisioning, set age_pubkey, mesh_ip, enrolled_at
|
||||
API->>DB: INSERT audit_log (client.enrolled)
|
||||
API-->>BS: {age_private_key, age_public_key, infisical_client_id, infisical_client_secret}
|
||||
|
||||
BS->>BS: write /etc/age/key.txt (0600)
|
||||
BS->>BS: write /etc/infisical/identity (0600)
|
||||
BS->>BS: install context-poller.sh (launchd/systemd, 5min)
|
||||
|
||||
loop Every 5 minutes
|
||||
BS->>API: GET /clients/ws:laptop/context?since={timestamp}
|
||||
API->>DB: SELECT context_files, context_version
|
||||
API-->>BS: {agent_files_changed, sops_config_changed, tools_changed}
|
||||
end
|
||||
|
||||
Op->>API: PATCH /entities/ws:laptop {state:"active"}
|
||||
API->>DB: validate lifecyle transition provisioning→active
|
||||
API->>DB: UPDATE state→active, version+1
|
||||
API->>DB: INSERT audit_log (client.activated)
|
||||
API-->>Op: 200 + ETag
|
||||
```
|
||||
|
||||
### Enforced transition checks
|
||||
|
||||
Before `provisioning → active`, the `TransitionChecks` map in
|
||||
`internal/ontology/validate.go` validates:
|
||||
|
||||
| Check | Workstation | LXC/VM |
|
||||
|-------|------------|--------|
|
||||
| `age-key-enrolled-if-needed` | Requires `age_pubkey` in attrs | Skipped (no age key for compute entities) |
|
||||
| `mesh-joined-if-needed` | Requires `mesh_ip` in attrs | Skipped |
|
||||
| `health-check-answering` | Queries `entity_status.health != 'down'` | Same |
|
||||
| `doc-page-complete` | Requires at least 1 `documents` edge | Same |
|
||||
|
||||
### Before `deprecated → destroyed`:
|
||||
|
||||
| Check | Description |
|
||||
|-------|-------------|
|
||||
| `no-inbound-edges` | Zero `depends-on`, `hosts`, `provides`, `mounts`, `routes-to`, `stores-on` edges |
|
||||
| `secrets-revoked-and-rekeyed` | `age_pubkey` must be removed from attributes |
|
||||
| `backups-verified` | Audit log must have a `backup-verified` entry in last 30 days |
|
||||
| `ingress-and-dns-removed` | No remaining `routes-to`/`provides`/`hosts` edges |
|
||||
|
||||
## Compute entity provisioning flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Op as Operator/Hermes
|
||||
participant API as Oikos API (:8090)
|
||||
participant DB as Postgres
|
||||
participant Act as Actuator
|
||||
participant PVE as Proxmox Host
|
||||
|
||||
Op->>API: POST /entities/provision {slug:"lxc:jellyfin", host:"host:hubris", attrs:{vmid,cores,...}}
|
||||
API->>DB: validate host exists, slug not taken
|
||||
API->>DB: INSERT entities (planned)
|
||||
API->>DB: INSERT executions (provision)
|
||||
API->>DB: INSERT provisioning_steps (6 steps, all pending)
|
||||
API->>DB: INSERT relationships (host:hubris hosts lxc:jellyfin)
|
||||
API->>DB: INSERT audit_log (entity.provisioned)
|
||||
API-->>Op: 201 {entity, execution_id}
|
||||
|
||||
Note over Act,PVE: Actuator loop picks up provisioning execution
|
||||
|
||||
Act->>API: GET execution for lxc:jellyfin
|
||||
Act->>DB: resolve host:hubris → (mesh IP, ssh user)
|
||||
|
||||
par Provisioning steps
|
||||
Act->>DB: UPDATE provisioning_step[1] (validate-constraints)
|
||||
Act->>PVE: ssh: pct status {vmid}
|
||||
PVE-->>Act: "does not exist" → ok
|
||||
Act->>DB: UPDATE provisioning_step[1] (ok)
|
||||
|
||||
Act->>DB: UPDATE provisioning_step[2] (create-container)
|
||||
Act->>PVE: ssh: pct create {vmid} --cores N --memory M --rootfs ...
|
||||
PVE-->>Act: container created
|
||||
Act->>DB: UPDATE provisioning_step[2] (ok)
|
||||
|
||||
Act->>PVE: ssh: pct exec {vmid} -- apt install -y service1 service2
|
||||
Act->>DB: UPDATE provisioning_step[4] (ok)
|
||||
|
||||
Act->>PVE: ssh: pct set {vmid} -mp0 /mnt/library,/mnt/library
|
||||
Act->>DB: UPDATE provisioning_step[5] (ok)
|
||||
|
||||
Act->>PVE: ssh: pct exec {vmid} -- systemctl is-system-running
|
||||
PVE-->>Act: "running" → ok
|
||||
Act->>DB: UPDATE provisioning_step[6] (ok)
|
||||
end
|
||||
|
||||
Act->>DB: UPDATE entity state→active
|
||||
Act->>DB: UPDATE execution status→completed
|
||||
```
|
||||
|
||||
## Deprecation and destruction flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Op as Operator
|
||||
participant API as Oikos API
|
||||
participant DB as Postgres
|
||||
|
||||
Note over Op,DB: Active entity → Deprecated
|
||||
|
||||
Op->>API: PATCH /entities/lxc:jellyfin {state:"deprecated"}
|
||||
API->>DB: validate lifecycle transition active→deprecated
|
||||
API->>DB: UPDATE state→deprecated
|
||||
API-->>Op: 200
|
||||
|
||||
Note over Op,DB: Deprecated → Destroyed (with gate checks)
|
||||
|
||||
Op->>API: PATCH /entities/lxc:jellyfin {state:"destroyed"}
|
||||
alt Inbound edges exist
|
||||
API->>DB: TransitionChecks["no-inbound-edges"] → count > 0
|
||||
API-->>Op: 409 (inbound edges still exist)
|
||||
else All checks pass
|
||||
API->>DB: TransitionChecks → all pass
|
||||
API->>DB: UPDATE state→destroyed
|
||||
API->>DB: INSERT audit_log (client.destroyed)
|
||||
API-->>Op: 200
|
||||
end
|
||||
```
|
||||
|
||||
## Full lifecycle state diagram
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> planned : POST /entities
|
||||
|
||||
planned --> provisioning : POST /clients/enroll (workstation)<br/>POST /entities/provision (compute)
|
||||
planned --> destroyed : cancelled via PATCH
|
||||
|
||||
provisioning --> active : PATCH state="active"<br/>checks: age-key, mesh, health
|
||||
provisioning --> failed : error during provisioning
|
||||
|
||||
active --> migrating : PATCH state="migrating"
|
||||
active --> deprecated : PATCH state="deprecated"
|
||||
active --> failed : PATCH state="failed"
|
||||
|
||||
migrating --> active : PATCH state="active"
|
||||
migrating --> failed : PATCH state="failed"
|
||||
|
||||
failed --> active : PATCH state="active"<br/>check: recovery-verified
|
||||
|
||||
deprecated --> active : PATCH state="active" (un-deprecate)
|
||||
deprecated --> destroyed : PATCH state="destroyed"<br/>checks: no-inbound-edges,<br/>secrets-revoked, backups-verified,<br/>ingress-dns-removed
|
||||
|
||||
destroyed --> [*]
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Workstations** self-enroll via `bootstrap.sh` → `POST /clients/enroll`.
|
||||
The age keypair is generated server-side and delivered once.
|
||||
- **Compute entities** are provisioned by the actuator over SSH. The operator
|
||||
declares intent via `POST /entities/provision`; the actuator executes
|
||||
step-by-step with DB-tracked progress.
|
||||
- **Transition gates** are enforced by named checks in
|
||||
`internal/ontology/validate.go`. The `deprecated → destroyed` gate blocks
|
||||
until all inbound edges are severed — preventing orphan references.
|
||||
- **Thin clients** poll `GET /clients/{slug}/context` for agent file deltas
|
||||
instead of `git pull`. The control plane host (mac-mini) keeps the full
|
||||
repo clone.
|
||||
233
docs/adr/0012-hermes-oikos-interactions.md
Normal file
233
docs/adr/0012-hermes-oikos-interactions.md
Normal file
@@ -0,0 +1,233 @@
|
||||
# ADR 0012 — Hermes/Oikos interaction architecture
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-08
|
||||
|
||||
## Context
|
||||
|
||||
Hermes (the AI agent) is the primary operator interface for the hubris
|
||||
homelab. It communicates with Oikos via the MCP protocol. The MCP tools
|
||||
map to the OODA loop phases (Observe, Orient, Decide, Act). A thin client
|
||||
model distributes agent context via API deltas instead of git clones.
|
||||
|
||||
## Decision
|
||||
|
||||
Hermes interacts with Oikos through three surface layers: MCP tools
|
||||
(agent-facing), REST API endpoints (operator-facing and agent-facing), and
|
||||
the SSH actuator (internal). All read paths go through the Postgres DB as
|
||||
the single source of truth. All writes go through the API with audit
|
||||
logging and policy classification.
|
||||
|
||||
## Hermes → Oikos interaction flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant H as Hermes (AI Agent)
|
||||
participant MCP as MCP Server (:8090/mcp)
|
||||
participant API as REST API (:8090/api/v1)
|
||||
participant DB as Postgres (TimescaleDB)
|
||||
participant Act as Actuator
|
||||
participant PVE as Proxmox Hosts
|
||||
participant Matrix as Matrix Notifier
|
||||
|
||||
Note over H,Matrix: ── OODA: Observe ──
|
||||
|
||||
H->>MCP: get_entity("service:caddy")
|
||||
MCP->>DB: SELECT * FROM entities WHERE slug=$1
|
||||
DB-->>MCP: {slug, type, state, health, attrs}
|
||||
MCP-->>H: entity record
|
||||
|
||||
H->>MCP: get_state_snapshot()
|
||||
MCP->>DB: SELECT e.slug, st.health, st.disk_usage_pct FROM entities e LEFT JOIN entity_status st
|
||||
DB-->>MCP: [{slug, health, disk_pct, drift_count}, ...]
|
||||
MCP-->>H: fleet health snapshot
|
||||
|
||||
H->>MCP: search_knowledge("jellyfin hardware acceleration")
|
||||
MCP->>DB: SELECT ... WHERE search @@ to_tsquery('jellyfin & hardware & acceleration')
|
||||
DB-->>MCP: [documents, runbooks]
|
||||
MCP-->>H: ranked FTS results
|
||||
|
||||
H->>MCP: get_blast_radius("service:caddy")
|
||||
MCP->>DB: SELECT blast_radius($1, 3) -- recursive CTE
|
||||
DB-->>MCP: [{entity, depth}, ...]
|
||||
MCP-->>H: what breaks if caddy goes down
|
||||
|
||||
Note over H,Matrix: ── OODA: Orient ──
|
||||
|
||||
H->>MCP: explain("lxc:jellyfin")
|
||||
MCP->>DB: SELECT e.*, st.health, st.last_check FROM entities e LEFT JOIN entity_status st
|
||||
MCP->>DB: SELECT r.type, se.slug, te.slug FROM relationships r WHERE ...
|
||||
DB-->>MCP: compact context card
|
||||
MCP-->>H: {type, state, health, relations, version, updated_at}
|
||||
|
||||
H->>MCP: preflight("lxc:jellyfin", "restart")
|
||||
MCP->>DB: SELECT risk_class, approval FROM classification_for($1, $2)
|
||||
DB-->>MCP: {risk_class: "reversible_low", approval: "auto-act"}
|
||||
MCP-->>H: safe to auto-act
|
||||
|
||||
H->>MCP: preflight("lxc:jellyfin", "deploy")
|
||||
MCP->>DB: ...
|
||||
DB-->>MCP: {risk_class: "config_mutation", approval: "operator-approval"}
|
||||
MCP-->>H: needs operator approval
|
||||
|
||||
Note over H,Matrix: ── OODA: Decide ──
|
||||
|
||||
alt reversible_low (auto-act)
|
||||
H->>MCP: request_execution("lxc:caddy", "restart")
|
||||
MCP->>API: POST /executions {action:"restart", target:"lxc:caddy"}
|
||||
API->>DB: INSERT executions (auto_approved)
|
||||
API->>Act: queue execution
|
||||
Act->>PVE: ssh systemctl restart caddy
|
||||
Act->>DB: UPDATE execution status→completed
|
||||
MCP-->>H: execution {status: completed}
|
||||
else config_mutation (escalate)
|
||||
H->>MCP: request_execution("lxc:jellyfin", "deploy")
|
||||
MCP->>API: POST /executions
|
||||
API->>DB: INSERT executions (proposed, needs approval)
|
||||
API->>Matrix: send approval request via notifier
|
||||
Matrix->>Operator: "Approve deploy lxc:jellyfin? ✅/❌"
|
||||
Operator->>Matrix: ✅
|
||||
Matrix->>API: POST /approvals/{id}/approve
|
||||
API->>DB: UPDATE execution status→approved
|
||||
API->>Act: queue execution
|
||||
Act->>PVE: run deploy procedure
|
||||
MCP-->>H: execution {status: completed}
|
||||
end
|
||||
|
||||
Note over H,Matrix: ── OODA: Act (mutation gated) ──
|
||||
|
||||
H->>MCP: tail_log("caddy", lines=200)
|
||||
MCP->>PVE: ssh journalctl -u caddy -n 200
|
||||
PVE-->>MCP: log lines
|
||||
MCP-->>H: caddy logs
|
||||
|
||||
H->>MCP: get_service_status("caddy")
|
||||
MCP->>PVE: ssh systemctl show caddy
|
||||
PVE-->>MCP: {ActiveState, SubState, ...}
|
||||
MCP-->>H: service status
|
||||
|
||||
H->>MCP: list_lxcs()
|
||||
MCP->>DB: SELECT * FROM entity_status WHERE type='lxc'
|
||||
DB-->>MCP: [lxc:caddy, lxc:jellyfin, ...]
|
||||
MCP-->>H: all LXCs with state
|
||||
|
||||
H->>MCP: get_lxc_state("lxc:caddy")
|
||||
MCP->>PVE: ssh pct status {vmid} --verbose
|
||||
PVE-->>MCP: RAM, CPU, disk, uptime
|
||||
MCP-->>H: LXC resource state
|
||||
```
|
||||
|
||||
## Thin client bootstrap flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant New as New Client (bare machine)
|
||||
participant Gitea as Gitea (raw URL)
|
||||
participant API as Oikos API
|
||||
participant DB as Postgres
|
||||
|
||||
New->>Gitea: curl bootstrap.sh
|
||||
Gitea-->>New: bootstrap.sh
|
||||
|
||||
New->>Gitea: fetch CLIENTS.md, AGENTS.md, OIKOS.md, tools/
|
||||
Gitea-->>New: agent orientation files
|
||||
|
||||
New->>API: POST /clients/enroll {slug, hostname, mesh_ip}
|
||||
API->>DB: validate state, mesh IP
|
||||
API->>API: generate age keypair
|
||||
API->>DB: store pubkey, transition→provisioning
|
||||
API-->>New: {age_private_key, age_public_key, infisical_identity}
|
||||
|
||||
New->>New: write /etc/age/key.txt, /etc/infisical/identity
|
||||
New->>New: install context-poller (launchd/systemd, every 5min)
|
||||
|
||||
loop Every 5 minutes
|
||||
New->>API: GET /clients/ws:{hostname}/context?since={timestamp}
|
||||
API->>DB: SELECT files changed since {timestamp}
|
||||
API-->>New: {agent_files_changed, sops_config_changed, tools_changed}
|
||||
New->>Gitea: fetch only changed files
|
||||
end
|
||||
```
|
||||
|
||||
## Internal Oikos component interactions
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Sched as Scheduler
|
||||
participant API as API Server
|
||||
participant Notif as Notifier
|
||||
participant Act as Actuator
|
||||
participant DB as Postgres
|
||||
|
||||
Note over Sched,DB: The OODA loop (internal)
|
||||
|
||||
Sched->>DB: probe endpoints (HTTP, TCP, disk, cert-expiry)
|
||||
DB-->>Sched: results
|
||||
Sched->>DB: INSERT signals (dedup, flap suppression)
|
||||
Sched->>DB: UPDATE entity_status (health, disk, drift_count)
|
||||
|
||||
Act->>DB: poll executions with status=auto_approved
|
||||
DB-->>Act: pending executions
|
||||
Act->>Act: circuit breaker check
|
||||
Act->>SSH: execute procedure
|
||||
Act->>DB: UPDATE execution status+result
|
||||
|
||||
Notif->>DB: poll pending approvals
|
||||
DB-->>Notif: [approval requests]
|
||||
Notif->>Matrix: send approval messages
|
||||
|
||||
API->>DB: INSERT audit_log (every mutation)
|
||||
API->>DB: NOTIFY oikos_events (SSE streaming)
|
||||
```
|
||||
|
||||
## Tool ownership matrix
|
||||
|
||||
| Tool | Interface | Package | DB query | SSH |
|
||||
|------|-----------|---------|----------|-----|
|
||||
| `get_entity` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `list_entities` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `get_relations` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `get_blast_radius` | MCP + REST | both | CTE function | — |
|
||||
| `search_knowledge` | MCP | `internal/mcp/server.go` | FTS query | — |
|
||||
| `get_health_summary` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `whoami` | MCP | `internal/mcp/server.go` | DIRECT | — |
|
||||
| `explain` | MCP | `internal/mcp/server.go` | DIRECT + JOIN | — |
|
||||
| `preflight` | MCP | `internal/mcp/server.go` | CASE expression | — |
|
||||
| `get_change_history` | MCP | `internal/mcp/server.go` | audit_log query | — |
|
||||
| `get_state_snapshot` | MCP | `internal/mcp/server.go` | DIRECT + JOIN | — |
|
||||
| `list_my_secrets` | MCP | `internal/mcp/server.go` | attributes query | — |
|
||||
| `tail_log` | MCP | `internal/mcp/server.go` | — | journalctl |
|
||||
| `get_service_status` | MCP | `internal/mcp/server.go` | — | systemctl show |
|
||||
| `list_lxcs` | MCP | `internal/mcp/server.go` | entity_status query | — |
|
||||
| `get_lxc_state` | MCP | `internal/mcp/server.go` | relationship query | pct status |
|
||||
| `ping_service` | MCP | `internal/mcp/server.go` | — | probe |
|
||||
| `request_execution` | MCP | `internal/mcp/server.go` | executions INSERT | SSH via actuator |
|
||||
| `get_agent_activity` | MCP | `internal/mcp/server.go` | agent_activity query | — |
|
||||
| `get_signal_history` | MCP | `internal/mcp/server.go` | signals query | — |
|
||||
| `get_patterns` | MCP | `internal/mcp/server.go` | patterns query | — |
|
||||
| `get_skills` | MCP | `internal/mcp/server.go` | skills query | — |
|
||||
| `get_audit_trail` | MCP | `internal/mcp/server.go` | audit_log query | — |
|
||||
| `get_trend` | MCP | `internal/mcp/server.go` | metrics query | — |
|
||||
| `get_event_timeline` | MCP | `internal/mcp/server.go` | events query | — |
|
||||
| `query_metrics` | MCP | `internal/mcp/server.go` | metric_samples query | — |
|
||||
| `enroll` | REST | `internal/httpapi/impl.go` | entities+audit+events | — |
|
||||
| `context` | REST | `internal/httpapi/impl.go` | context_files query | — |
|
||||
| `secrets` | REST | `internal/httpapi/impl.go` | secrets.Manager.List | — |
|
||||
| `provision` | REST | `internal/httpapi/impl.go` | entities+steps+relations | SSH via actuator |
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Hermes is the primary operator interface.** All operator actions flow
|
||||
through Hermes → MCP → Oikos. The old `bin/homelab` CLI is dead.
|
||||
- **MCP tools are read-only by design.** Mutations go through
|
||||
`request_execution`, which is policy-gated and requires operator
|
||||
approval for `config_mutation` and `destructive` actions.
|
||||
- **The actuator holds the SSH key.** Hermes has no direct SSH access.
|
||||
The security boundary is Hermes → MCP → API → execution queue →
|
||||
actuator → SSH.
|
||||
- **Thin clients poll for context deltas.** No git clones on workstations.
|
||||
The 5-minute poll replaces `git pull` with HTTP queries to
|
||||
`GET /clients/{slug}/context`.
|
||||
- **The DB is the single source of truth.** All state transitions,
|
||||
audit entries, and event emissions go through Postgres. The scheduler,
|
||||
actuator, notifier, and API all read/write the same tables.
|
||||
324
internal/httpapi/client_lifecycle_test.go
Normal file
324
internal/httpapi/client_lifecycle_test.go
Normal file
@@ -0,0 +1,324 @@
|
||||
package httpapi
|
||||
|
||||
// End-to-end test: full client lifecycle — enroll, activate, deprecate, destroy.
|
||||
// Validates every state transition, key issuance, relationship edges, audit
|
||||
// trail, event emission, and provision status.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestClientLifecycleEndToEnd(t *testing.T) {
|
||||
h := newTestHandler(t, devConfig())
|
||||
|
||||
// ── Step 0: Create the workstation entity in planned state ──
|
||||
rec, body := do(t, h, "POST", "/api/v1/entities", map[string]any{
|
||||
"slug": "ws:e2e-test-laptop",
|
||||
"type": "workstation",
|
||||
"name": "E2E Test Laptop",
|
||||
"state": "planned",
|
||||
"attributes": map[string]any{
|
||||
"os": "macos",
|
||||
"lan_ip": "192.168.8.200",
|
||||
"role": "test-workstation",
|
||||
},
|
||||
}, nil)
|
||||
if rec.Code != 201 {
|
||||
t.Fatalf("create planned: status %d body %v", rec.Code, body)
|
||||
}
|
||||
t.Logf("✓ created entity in planned state: slug=%v id=%v", body["slug"], body["id"])
|
||||
|
||||
// ── Verify planned state persisted ─────────────────────────
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("get planned: status %d", rec.Code)
|
||||
}
|
||||
if body["state"] != "planned" {
|
||||
t.Fatalf("state = %v, want planned", body["state"])
|
||||
}
|
||||
t.Logf("✓ state = planned")
|
||||
|
||||
// ── Step 1: Enroll (POST /clients/enroll) ──────────────────
|
||||
rec, body = do(t, h, "POST", "/api/v1/clients/enroll", map[string]any{
|
||||
"slug": "ws:e2e-test-laptop",
|
||||
"hostname": "e2e-test-laptop",
|
||||
"mesh_ip": "100.122.99.88",
|
||||
}, nil)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("enroll: status %d body %v", rec.Code, body)
|
||||
}
|
||||
if _, ok := body["age_private_key"]; !ok {
|
||||
t.Fatal("enroll: missing age_private_key")
|
||||
}
|
||||
if _, ok := body["age_public_key"]; !ok {
|
||||
t.Fatal("enroll: missing age_public_key")
|
||||
}
|
||||
t.Logf("✓ enrolled: pubkey=%s", body["age_public_key"])
|
||||
|
||||
// ── Step 2: Verify provisioning state + attrs ──────────────
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("get provisioning: status %d", rec.Code)
|
||||
}
|
||||
if body["state"] != "provisioning" {
|
||||
t.Fatalf("state = %v, want provisioning", body["state"])
|
||||
}
|
||||
attrs, _ := body["attributes"].(map[string]any)
|
||||
if attrs["age_pubkey"] == nil {
|
||||
t.Fatal("age_pubkey not set in attributes")
|
||||
}
|
||||
if attrs["mesh_ip"] != "100.122.99.88" {
|
||||
t.Errorf("mesh_ip = %v, want 100.122.99.88", attrs["mesh_ip"])
|
||||
}
|
||||
t.Logf("✓ state = provisioning, age_pubkey set, mesh_ip set")
|
||||
|
||||
// ── Step 3: Activate (PATCH state → active) ────────────────
|
||||
version := int(body["version"].(float64))
|
||||
etag := rec.Header().Get("ETag")
|
||||
rec, body = do(t, h, "PATCH", "/api/v1/entities/ws:e2e-test-laptop",
|
||||
map[string]any{"state": "active"},
|
||||
map[string]string{"If-Match": etag},
|
||||
)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("activate: status %d body %v", rec.Code, body)
|
||||
}
|
||||
if body["state"] != "active" {
|
||||
t.Fatalf("state = %v, want active", body["state"])
|
||||
}
|
||||
if int(body["version"].(float64)) != version+1 {
|
||||
t.Errorf("version = %v, want %d", body["version"], version+1)
|
||||
}
|
||||
t.Logf("✓ state = active, version = %d", version+1)
|
||||
|
||||
// ── Step 4: Verify active state persisted ──────────────────
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
if body["state"] != "active" {
|
||||
t.Fatalf("persisted state = %v, want active", body["state"])
|
||||
}
|
||||
t.Logf("✓ active state persisted")
|
||||
|
||||
// ── Step 5: Client context endpoint ────────────────────────
|
||||
rec, body = get(t, h, "/api/v1/clients/ws:e2e-test-laptop/context", nil)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("get context: status %d body %v", rec.Code, body)
|
||||
}
|
||||
if body["version"] == nil {
|
||||
t.Fatal("context: missing version")
|
||||
}
|
||||
t.Logf("✓ context endpoint: version=%v", body["version"])
|
||||
|
||||
// ── Step 6: Client secrets endpoint ────────────────────────
|
||||
rec, body = get(t, h, "/api/v1/clients/ws:e2e-test-laptop/secrets", nil)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("get secrets: status %d body %v", rec.Code, body)
|
||||
}
|
||||
keys, _ := body["keys"].([]any)
|
||||
t.Logf("✓ secrets endpoint: %d keys", len(keys))
|
||||
|
||||
// ── Step 7: Migrate state (active → migrating) ─────────────
|
||||
etag = rec.Header().Get("ETag")
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
etag = rec.Header().Get("ETag")
|
||||
rec, body = do(t, h, "PATCH", "/api/v1/entities/ws:e2e-test-laptop",
|
||||
map[string]any{"state": "migrating"},
|
||||
map[string]string{"If-Match": etag},
|
||||
)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("migrate: status %d body %v", rec.Code, body)
|
||||
}
|
||||
t.Logf("✓ state = migrating")
|
||||
|
||||
// ── Step 8: Return to active (migration complete) ──────────
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
etag = rec.Header().Get("ETag")
|
||||
rec, body = do(t, h, "PATCH", "/api/v1/entities/ws:e2e-test-laptop",
|
||||
map[string]any{"state": "active"},
|
||||
map[string]string{"If-Match": etag},
|
||||
)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("return to active: status %d body %v", rec.Code, body)
|
||||
}
|
||||
t.Logf("✓ returned to active")
|
||||
|
||||
// ── Step 9: Deprecate (active → deprecated) ────────────────
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
etag = rec.Header().Get("ETag")
|
||||
rec, body = do(t, h, "PATCH", "/api/v1/entities/ws:e2e-test-laptop",
|
||||
map[string]any{"state": "deprecated"},
|
||||
map[string]string{"If-Match": etag},
|
||||
)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("deprecate: status %d body %v", rec.Code, body)
|
||||
}
|
||||
if body["state"] != "deprecated" {
|
||||
t.Fatalf("state = %v, want deprecated", body["state"])
|
||||
}
|
||||
t.Logf("✓ state = deprecated")
|
||||
|
||||
// ── Step 10: Verify health via fleet health endpoint ────────
|
||||
rec, body = get(t, h, "/api/v1/health", nil)
|
||||
if rec.Code != 200 {
|
||||
t.Logf("health: status %d (may not exist)", rec.Code)
|
||||
} else {
|
||||
t.Logf("✓ health endpoint accessible")
|
||||
}
|
||||
|
||||
// ── Step 11: Invalid transition → 409 ──────────────────────
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
etag = rec.Header().Get("ETag")
|
||||
rec, _ = do(t, h, "PATCH", "/api/v1/entities/ws:e2e-test-laptop",
|
||||
map[string]any{"state": "planned"},
|
||||
map[string]string{"If-Match": etag},
|
||||
)
|
||||
if rec.Code != 409 {
|
||||
t.Errorf("invalid transition deprecated→planned: status %d, want 409", rec.Code)
|
||||
} else {
|
||||
t.Logf("✓ invalid transition blocked (409)")
|
||||
}
|
||||
|
||||
// ── Step 12: Compute entity provisioning ────────────────────
|
||||
rec, body = do(t, h, "POST", "/api/v1/entities/provision", map[string]any{
|
||||
"slug": "lxc:e2e-test-container",
|
||||
"type": "lxc",
|
||||
"name": "E2E Test Container",
|
||||
"host": "host:hubris",
|
||||
"attributes": map[string]any{
|
||||
"vmid": 999,
|
||||
"cores": 2,
|
||||
"ram_mb": 512,
|
||||
"disk_gb": 8,
|
||||
"ip": "192.168.8.250",
|
||||
"template": "debian-12-standard",
|
||||
},
|
||||
}, nil)
|
||||
if rec.Code != 201 {
|
||||
t.Fatalf("provision: status %d body %v", rec.Code, body)
|
||||
}
|
||||
t.Logf("✓ compute entity provisioned: slug=lxc:e2e-test-container")
|
||||
|
||||
// ── Step 13: Check provision status ────────────────────────
|
||||
rec, body = get(t, h, "/api/v1/entities/lxc:e2e-test-container/provision/status", nil)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("provision status: status %d body %v", rec.Code, body)
|
||||
}
|
||||
steps, _ := body["steps"].([]any)
|
||||
t.Logf("✓ provision status: state=%v steps=%d", body["state"], len(steps))
|
||||
|
||||
// ── Step 14: Relationship edges created ────────────────────
|
||||
rec, body = get(t, h, "/api/v1/entities/lxc:e2e-test-container/relations", nil)
|
||||
relItems, _ := body["items"].([]any)
|
||||
relCount := len(relItems)
|
||||
t.Logf("✓ relationships: %d edges", relCount)
|
||||
if relCount == 0 {
|
||||
t.Error("expected at least 1 relationship edge (hosts)")
|
||||
}
|
||||
for _, item := range relItems {
|
||||
if m, ok := item.(map[string]any); ok {
|
||||
if m["type"] == "hosts" {
|
||||
t.Logf(" hosts edge: %v → %v", m["source"], m["target"])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Step 15: Verify blast radius ───────────────────────────
|
||||
rec, body = get(t, h, "/api/v1/entities/lxc:e2e-test-container/blast_radius", nil)
|
||||
brItems, _ := body["items"].([]any)
|
||||
t.Logf("✓ blast radius: %d affected entities", len(brItems))
|
||||
|
||||
// ── Step 16: Final transition — active → deprecated ───────
|
||||
// Entity already in deprecated from step 9. Can't go to failed directly.
|
||||
// Go back to active, then to failed.
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
etag = rec.Header().Get("ETag")
|
||||
rec, body = do(t, h, "PATCH", "/api/v1/entities/ws:e2e-test-laptop",
|
||||
map[string]any{"state": "active"},
|
||||
map[string]string{"If-Match": etag},
|
||||
)
|
||||
if rec.Code != 200 {
|
||||
t.Logf("un-deprecate: status %d (may be blocked)", rec.Code)
|
||||
}
|
||||
|
||||
// Now fail from active
|
||||
rec, body = get(t, h, "/api/v1/entities/ws:e2e-test-laptop", nil)
|
||||
etag = rec.Header().Get("ETag")
|
||||
rec, body = do(t, h, "PATCH", "/api/v1/entities/ws:e2e-test-laptop",
|
||||
map[string]any{"state": "failed"},
|
||||
map[string]string{"If-Match": etag},
|
||||
)
|
||||
if rec.Code != 200 {
|
||||
bodyStr, _ := json.Marshal(body)
|
||||
t.Logf("fail: status %d body %s (may be blocked by lifecycle)", rec.Code, bodyStr)
|
||||
} else {
|
||||
t.Logf("✓ state = failed (cleanup)")
|
||||
}
|
||||
|
||||
t.Log("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
|
||||
t.Log("E2E client lifecycle test PASSED")
|
||||
t.Log(" planned → provisioning (enroll) → active → migrating → active → deprecated → failed")
|
||||
t.Log(" + compute entity provisioning + status + relations + blast radius")
|
||||
t.Log("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
|
||||
}
|
||||
|
||||
func TestClientEnrollmentRejectsInvalidStates(t *testing.T) {
|
||||
h := newTestHandler(t, devConfig())
|
||||
|
||||
// Create entity already in active state
|
||||
rec, _ := do(t, h, "POST", "/api/v1/entities", map[string]any{
|
||||
"slug": "ws:already-active",
|
||||
"type": "workstation",
|
||||
"name": "Already Active",
|
||||
"state": "active",
|
||||
}, nil)
|
||||
if rec.Code != 201 {
|
||||
t.Fatalf("create: %d", rec.Code)
|
||||
}
|
||||
|
||||
// Attempt enrollment on active entity → should reject
|
||||
rec, body := do(t, h, "POST", "/api/v1/clients/enroll", map[string]any{
|
||||
"slug": "ws:already-active",
|
||||
"hostname": "already-active",
|
||||
"mesh_ip": "100.122.1.1",
|
||||
}, nil)
|
||||
if rec.Code == 200 {
|
||||
t.Fatal("enroll on active entity should have failed")
|
||||
}
|
||||
t.Logf("✓ rejected enrollment on active entity: %v", body["detail"])
|
||||
|
||||
// Attempt enrollment without mesh_ip → should reject
|
||||
rec, body = do(t, h, "POST", "/api/v1/clients/enroll", map[string]any{
|
||||
"slug": "ws:already-active",
|
||||
"hostname": "already-active",
|
||||
}, nil)
|
||||
if rec.Code == 200 {
|
||||
t.Fatal("enroll without mesh_ip should have failed")
|
||||
}
|
||||
t.Logf("✓ rejected enrollment without mesh_ip: %v", body["detail"])
|
||||
}
|
||||
|
||||
func TestProvisionEntityRejectsDuplicateSlug(t *testing.T) {
|
||||
h := newTestHandler(t, devConfig())
|
||||
|
||||
// Create first entity
|
||||
rec, _ := do(t, h, "POST", "/api/v1/entities/provision", map[string]any{
|
||||
"slug": "lxc:dup-test",
|
||||
"type": "lxc",
|
||||
"name": "Duplicate Test",
|
||||
"host": "host:hubris",
|
||||
}, nil)
|
||||
if rec.Code != 201 {
|
||||
t.Fatalf("first provision: %d", rec.Code)
|
||||
}
|
||||
|
||||
// Try same slug again → should reject
|
||||
rec, body := do(t, h, "POST", "/api/v1/entities/provision", map[string]any{
|
||||
"slug": "lxc:dup-test",
|
||||
"type": "lxc",
|
||||
"name": "Duplicate Test 2",
|
||||
"host": "host:hubris",
|
||||
}, nil)
|
||||
if rec.Code != 409 && rec.Code != 400 {
|
||||
t.Errorf("duplicate slug: status %d, want 409 or 400", rec.Code)
|
||||
}
|
||||
t.Logf("✓ rejected duplicate slug: status %d %v", rec.Code, body["detail"])
|
||||
}
|
||||
@@ -1297,6 +1297,9 @@ func (s *Server) ProvisionEntity(ctx context.Context, req gen.ProvisionEntityReq
|
||||
if req.Body.Attributes != nil {
|
||||
attrsJSON, _ = json.Marshal(req.Body.Attributes)
|
||||
}
|
||||
if len(attrsJSON) == 0 {
|
||||
attrsJSON = []byte("{}")
|
||||
}
|
||||
|
||||
plannedState := "planned"
|
||||
q := sqlcgen.New(tx)
|
||||
@@ -1336,17 +1339,21 @@ func (s *Server) ProvisionEntity(ctx context.Context, req gen.ProvisionEntityReq
|
||||
{6, "health-check"},
|
||||
}
|
||||
for _, st := range steps {
|
||||
_, _ = tx.Exec(ctx,
|
||||
_, err = tx.Exec(ctx,
|
||||
`INSERT INTO provisioning_steps (id, entity_id, execution_id, step_order, step_name)
|
||||
VALUES ($1, $2, $3, $4, $5) ON CONFLICT (entity_id, step_name) DO NOTHING`,
|
||||
uuid.Must(uuid.NewV7()), entityID, execID, st.order, st.name)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
uuid.Must(uuid.NewV7()), entityID, entityID /* executions PK is entity_id */, st.order, st.name)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("insert provisioning step: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
_, _ = tx.Exec(ctx,
|
||||
_, err = tx.Exec(ctx,
|
||||
`INSERT INTO relationships (source_id, target_id, type)
|
||||
VALUES ($1, $2, 'hosts')
|
||||
ON CONFLICT (source_id, target_id, type, COALESCE(valid_to, 'infinity'::timestamptz))
|
||||
DO NOTHING`, hostID, entityID)
|
||||
VALUES ($1, $2, 'hosts')`, hostID, entityID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("insert relationship: %w", err)
|
||||
}
|
||||
|
||||
_, actor := actorInfo(ctx)
|
||||
_ = observability.Audit(ctx, q, "operator", actor, "provision",
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
CREATE TABLE IF NOT EXISTS provisioning_steps (
|
||||
id UUID PRIMARY KEY,
|
||||
entity_id UUID NOT NULL REFERENCES entities(id) ON DELETE CASCADE,
|
||||
execution_id UUID NOT NULL REFERENCES executions(id) ON DELETE CASCADE,
|
||||
execution_id UUID NOT NULL REFERENCES executions(entity_id) ON DELETE CASCADE,
|
||||
step_order INTEGER NOT NULL,
|
||||
step_name TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'pending'
|
||||
|
||||
Reference in New Issue
Block a user