diff --git a/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md b/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md index 12f629b..854f8cf 100644 --- a/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md +++ b/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md @@ -506,11 +506,11 @@ flowchart LR graph TB subgraph mac-mini["mac-mini — Docker host, always-on"] subgraph services["Docker Compose — Go binaries"] - PG["PostgreSQL 16\n• entity/relationship store\n• signals • ledger\n• knowledge graph\n• patterns + skills\n• policy + ontology metadata"] + PG["PostgreSQL 16 + TimescaleDB\n• entity/relationship store\n• signals • ledger\n• knowledge graph\n• patterns + skills\n• policy + ontology metadata\n• metric_samples (hypertable)\n• audit_log (hypertable)\n• events (hypertable)\n• agent_activity (hypertable)"] INF["Infisical\n(secrets manager)"] HERMES["Hermes Agent — gateway mode\n+ homelab skills\n+ MCP client → API\n+ SSH keys (mounted)"] - API["Oikos API — Go (Gin)\n\nMCP: get_host, list_services,\nsearch_knowledge, get_relations\nREST: /hosts, /services, /signals,\n/exec, /approve, /deploy, /events\n\nPolicy enforcement + risk\nclassification + ledger"] - SCHED["Scheduler (Observe) — Go\n+ Actuator (Act) — Go\n• 10-min probes → DB\n• Classify → auto-act or escalate\n• Execution → feedback → patterns\n• SSH to hubris/strong"] + API["Oikos API — Go (Gin)\n\nMCP: get_host, list_services,\nsearch_knowledge, get_relations,\nquery_metrics, get_trend,\nget_audit_trail, get_health_summary\nREST: /hosts, /services, /signals,\n/exec, /approve, /deploy, /events,\n/metrics, /audit, /health\n\nPolicy enforcement + risk\nclassification + ledger\nAudit middleware + event emitter"] + SCHED["Scheduler (Observe) — Go\n+ Actuator (Act) — Go\n• 10-min probes → DB + metrics\n• Classify → auto-act or escalate\n• Execution → feedback → patterns\n• Correlation ID propagation\n• SSH to hubris/strong"] NOTIFIER["Notifier — Go\n• Matrix (current)\n• Future: webhook, email"] end DEPLOY["Gitea webhook →\ndocker compose build + up -d"] @@ -519,6 +519,9 @@ graph TB HERMES -- MCP --> API API --> PG SCHED --> PG + SCHED -- "metrics + events + audit" --> PG + API -- "audit + events" --> PG + HERMES -- "agent_activity" --> PG SCHED -- SSH --> HUBRIS SCHED -- SSH --> STRONG API -- "escalate" --> NOTIFIER @@ -540,6 +543,75 @@ graph TB WS -- "Hermes gateway" --> HERMES ``` +### Observability — data capture and query flow + +```mermaid +flowchart TB + subgraph sources["Data sources"] + SCHED_PROBE["Scheduler probes\nhealth, disk, latency, drift"] + API_CALLS["API calls\nrequest/response metrics"] + AGENT_CALLS["Agent activity\nMCP tools, SSH, reasoning"] + EXEC["Executions\nSSH commands, verification"] + LEARN["Learning engine\npatterns, skills, confidence"] + DEPLOY_EVT["Deploy events\nwebhook, build, restart"] + end + + subgraph capture["Capture layer (Go)"] + METRICS_W["metrics.go\n→ metric_samples"] + AUDIT_W["audit.go\n→ audit_log"] + EVENTS_W["events.go\n→ events table"] + AGENT_W["agent.go\n→ agent_activity"] + end + + subgraph store["PostgreSQL + TimescaleDB"] + MS["metric_samples\n(hypertable, 90d raw,\n1y rollups via CAGGs)"] + AL["audit_log\n(hypertable, 1y retention)"] + EV["events\n(hypertable, 90d retention)"] + AA["agent_activity\n(hypertable, 90d retention)"] + end + + subgraph query["Query layer (API)"] + REST["REST endpoints\n/metrics, /audit,\n/events, /health,\n/trends, /agent-activity"] + MCP["MCP tools\nquery_metrics, get_trend,\nget_audit_trail, get_event_timeline,\nget_agent_activity, get_health_summary"] + WS["WebSocket\n/api/v1/events\n(live stream)"] + end + + subgraph consumers["Consumers"] + AGENT["Agent\nqueries trends, audits\nits own history"] + FUTURE["Future: Grafana,\ndashboards, notebooks,\nSIEM, compliance tools"] + end + + SCHED_PROBE --> METRICS_W + API_CALLS --> METRICS_W + API_CALLS --> AUDIT_W + AGENT_CALLS --> AGENT_W + EXEC --> EVENTS_W + EXEC --> AUDIT_W + LEARN --> METRICS_W + LEARN --> EVENTS_W + DEPLOY_EVT --> EVENTS_W + + METRICS_W --> MS + AUDIT_W --> AL + EVENTS_W --> EV + AGENT_W --> AA + + MS --> REST + AL --> REST + EV --> REST + EV --> WS + AA --> REST + MS --> MCP + AL --> MCP + EV --> MCP + AA --> MCP + + REST --> AGENT + MCP --> AGENT + REST --> FUTURE + WS --> FUTURE +``` + ### OODA loop — with learning feedback ```mermaid @@ -669,6 +741,13 @@ flowchart TB │ ├── notifier/ # notification abstraction │ │ ├── notifier.go # interface │ │ └── matrix.go # Matrix implementation +│ ├── observability/ # logging, metrics, audit, events (NEW) +│ │ ├── logging.go # slog structured logging setup +│ │ ├── metrics.go # metric recording (writes to metric_samples) +│ │ ├── audit.go # audit middleware (writes to audit_log) +│ │ ├── events.go # event emitter (writes to events table) +│ │ ├── agent.go # agent activity recording +│ │ └── correlation.go # correlation ID propagation (context-based) │ └── config/ # config loading (env, files) │ └── config.go ├── migrations/ # SQL migrations (golang-migrate format) @@ -677,7 +756,8 @@ flowchart TB │ ├── 002_instances.up.sql # entities, relationships │ ├── 003_operations.up.sql # signals, approvals, executions, feedback │ ├── 004_learning.up.sql # patterns, skills -│ └── 005_policy.up.sql # policies, risk_classes, autonomy +│ ├── 005_policy.up.sql # policies, risk_classes, autonomy +│ └── 006_observability.up.sql # metrics, audit_log, events, agent_activity ├── seeds/ # YAML seed manifests (bootstrap + DR) │ ├── ontology.yaml # entity types, relationship types, lifecycles │ ├── inventory.yaml # entity instances (hosts, services, etc.) @@ -966,6 +1046,198 @@ CREATE TABLE autonomy_settings ( ); ``` +### Migration 6: Observability (metrics, audit log, event log) + +Uses **TimescaleDB** (PostgreSQL extension) for time-series data. Hypertables +auto-partition by time, continuous aggregates provide rollups, and retention policies +auto-drop old data. No separate database needed — everything stays in Postgres. + +```sql +-- Enable TimescaleDB +CREATE EXTENSION IF NOT EXISTS timescaledb; + +-- ─── Time-series metrics ────────────────────────────────────────────── +-- Generic metric store. Every probe, health check, and system measurement +-- writes here. Designed for high insert volume, time-range queries, and +-- continuous-aggregate rollups. + +CREATE TABLE metric_samples ( + ts TIMESTAMPTZ NOT NULL, + entity_id TEXT NOT NULL, -- which entity this metric is about + metric TEXT NOT NULL, -- 'health', 'disk_usage_pct', 'probe_latency_ms', + -- 'api_p99_ms', 'goroutines', 'db_connections', + -- 'pattern_confidence', 'skill_success_rate', ... + value DOUBLE PRECISION NOT NULL, + tags JSONB DEFAULT '{}'::JSONB -- arbitrary key-value labels: + -- {probe: "http", target: "192.168.8.121"}, + -- {host: "hubris", mount: "/mnt/library"}, ... +); + +-- Hypertable: partition by time, 1-week chunks +SELECT create_hypertable('metric_samples', 'ts', chunk_time_interval => INTERVAL '7 days'); + +-- Indexes for common query patterns +CREATE INDEX idx_metrics_entity_ts ON metric_samples(entity_id, ts DESC); +CREATE INDEX idx_metrics_metric_ts ON metric_samples(metric, ts DESC); +CREATE INDEX idx_metrics_tags ON metric_samples USING GIN(tags); + +-- Retention: drop raw metrics older than 90 days (continuous aggregates keep rollups) +SELECT add_retention_policy('metric_samples', INTERVAL '90 days'); + +-- Continuous aggregate: 1-hour rollups (mean, min, max, count) +CREATE MATERIALIZED VIEW metric_rollups_1h +WITH (timescaledb.continuous) AS + SELECT + time_bucket('1 hour', ts) AS bucket, + entity_id, + metric, + avg(value) AS avg_value, + min(value) AS min_value, + max(value) AS max_value, + count(*) AS sample_count, + (array_agg(tags))[1] AS representative_tags + FROM metric_samples + GROUP BY bucket, entity_id, metric; + +-- Refresh policy: refresh every 1 hour, keep 1 year of rollups +SELECT add_continuous_aggregate_policy('metric_rollups_1h', + start_offset => INTERVAL '2 hours', + end_offset => INTERVAL '5 minutes', + schedule_interval => INTERVAL '1 hour'); + +CREATE MATERIALIZED VIEW metric_rollups_1d +WITH (timescaledb.continuous) AS + SELECT + time_bucket('1 day', ts) AS bucket, + entity_id, + metric, + avg(value) AS avg_value, + min(value) AS min_value, + max(value) AS max_value, + count(*) AS sample_count + FROM metric_samples + GROUP BY bucket, entity_id, metric; + +SELECT add_continuous_aggregate_policy('metric_rollups_1d', + start_offset => INTERVAL '2 days', + end_offset => INTERVAL '1 hour', + schedule_interval => INTERVAL '1 day'); + +-- ─── Audit log ──────────────────────────────────────────────────────── +-- Every mutating action — by the OS, by an agent, or by an operator — +-- gets an immutable audit entry. This is the "who did what when" trail +-- that the ledger doesn't fully capture (ledger records OS decisions and +-- executions; audit captures ALL API calls including reads-that-matter +-- and operator interactions). + +CREATE TABLE audit_log ( + id BIGSERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + actor_type TEXT NOT NULL, -- 'agent', 'operator', 'system', 'scheduler' + actor_id TEXT, -- entity ID of the actor (agent entity, person entity) + action TEXT NOT NULL, -- 'api.call', 'entity.create', 'policy.update', + -- 'approval.decide', 'exec.request', 'deploy.trigger' + entity_id TEXT, -- entity affected (if any) + method TEXT, -- 'GET', 'POST', 'PATCH', 'DELETE', 'MCP', 'SSH' + path TEXT, -- API path or MCP tool name or SSH command + status_code INTEGER, -- HTTP status or 0 for non-HTTP + detail JSONB DEFAULT '{}'::JSONB, -- request body, response summary, extra context + source_ip TEXT, -- where the call came from + correlation_id TEXT -- links to execution_id / signal_id for tracing +); + +SELECT create_hypertable('audit_log', 'ts', chunk_time_interval => INTERVAL '7 days'); +CREATE INDEX idx_audit_actor ON audit_log(actor_type, actor_id, ts DESC); +CREATE INDEX idx_audit_entity ON audit_log(entity_id, ts DESC); +CREATE INDEX idx_audit_action ON audit_log(action, ts DESC); +CREATE INDEX idx_audit_correlation ON audit_log(correlation_id); + +-- Retention: keep 1 year of audit logs +SELECT add_retention_policy('audit_log', INTERVAL '365 days'); + +-- ─── Event log ──────────────────────────────────────────────────────── +-- Structured event stream — the "news feed" of the OS. Every significant +-- state change is an event: signal raised/resolved, execution started/completed, +-- approval requested/granted, deploy triggered, pattern validated, skill refined, +-- config changed, entity lifecycle transition. The WebSocket /api/v1/events +-- streams from this table; agents can also query it historically. + +CREATE TABLE events ( + id BIGSERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + type TEXT NOT NULL, -- 'signal.raised', 'signal.resolved', + -- 'execution.started', 'execution.completed', + -- 'approval.requested', 'approval.decided', + -- 'deploy.triggered', 'deploy.completed', + -- 'pattern.validated', 'skill.refined', + -- 'entity.created', 'entity.state_changed', + -- 'policy.changed', 'config.changed' + entity_id TEXT, -- primary entity involved + severity TEXT DEFAULT 'info', -- info, warning, critical + source TEXT NOT NULL, -- 'scheduler', 'actuator', 'api', 'hermes', + -- 'deploy', 'notifier', 'learning' + data JSONB DEFAULT '{}'::JSONB, -- event-specific payload + correlation_id TEXT -- links to signal/exec/approval for tracing +); + +SELECT create_hypertable('events', 'ts', chunk_time_interval => INTERVAL '7 days'); +CREATE INDEX idx_events_type_ts ON events(type, ts DESC); +CREATE INDEX idx_events_entity_ts ON events(entity_id, ts DESC); +CREATE INDEX idx_events_severity_ts ON events(severity, ts DESC); +CREATE INDEX idx_events_correlation ON events(correlation_id); + +-- Retention: keep 90 days of events (signals/ledger have their own tables +-- for permanent records; events are the transient feed) +SELECT add_retention_policy('events', INTERVAL '90 days'); + +-- ─── Agent activity log ────────────────────────────────────────────── +-- Records what the agent (Hermes) does: tool calls, reasoning, decisions, +-- token usage, latency. This is for agent behavior auditing and trend +-- analysis ("is the agent getting more efficient?"). + +CREATE TABLE agent_activity ( + id BIGSERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + agent_id TEXT NOT NULL, -- entity ID of the agent + session_id TEXT, -- Hermes session ID + activity_type TEXT NOT NULL, -- 'tool_call', 'reasoning', 'decision', + -- 'mcp_query', 'ssh_command', 'escalation' + tool_name TEXT, -- MCP tool or CLI command called + entity_id TEXT, -- entity acted upon (if any) + input_summary TEXT, -- truncated input (first 500 chars) + output_summary TEXT, -- truncated output (first 500 chars) + duration_ms INTEGER, + token_count INTEGER, -- LLM tokens consumed (if applicable) + success BOOLEAN, + correlation_id TEXT +); + +SELECT create_hypertable('agent_activity', 'ts', chunk_time_interval => INTERVAL '7 days'); +CREATE INDEX idx_agent_activity_agent_ts ON agent_activity(agent_id, ts DESC); +CREATE INDEX idx_agent_activity_type_ts ON agent_activity(activity_type, ts DESC); +CREATE INDEX idx_agent_activity_entity ON agent_activity(entity_id, ts DESC); +CREATE INDEX idx_agent_activity_correlation ON agent_activity(correlation_id); + +-- Retention: keep 90 days of agent activity +SELECT add_retention_policy('agent_activity', INTERVAL '90 days'); +``` + +**What gets captured where:** + +| Data source | Table | Retention | Purpose | +|---|---|---|---| +| Scheduler probes (HTTP health, disk, drift) | `metric_samples` | 90 days raw, 1 year rollups | Trend analysis, anomaly detection | +| API response latencies | `metric_samples` | same | Performance monitoring | +| Go runtime metrics (goroutines, mem, GC) | `metric_samples` | same | OS self-monitoring | +| Learning model metrics (pattern confidence, skill success rate) | `metric_samples` | same | Learning trend tracking | +| All mutating API calls | `audit_log` | 1 year | Auditing, compliance, forensics | +| Operator actions (approval decisions, entity edits, policy changes) | `audit_log` | 1 year | Operator accountability | +| Agent (Hermes) tool calls + reasoning | `agent_activity` | 90 days | Agent behavior auditing, efficiency tracking | +| Signal/exec/approval/deploy state changes | `events` | 90 days | Event stream (WebSocket), timeline reconstruction | +| Signals (permanent record) | `signals` | no expiry | Signal lifecycle tracking | +| Ledger entries (permanent record) | `ledger_entries` | no expiry | Change history | +| Executions (permanent record) | `executions` | no expiry | Execution audit trail | + ## Workstreams ### 1. Ontology definition + seed manifests (`seeds/`, `internal/ontology/`) @@ -1006,6 +1278,11 @@ One Go binary (Gin web framework) exposing REST + MCP from the same codebase. - Tools: `get_host`, `list_services`, `search_knowledge`, `get_entity`, `get_relations`, `get_blast_radius`, `get_signal_history`, `get_ledger`, `get_patterns`, `get_skills`, `get_state_snapshot` +- **Observability tools**: `query_metrics` (time-series for an entity/metric), + `get_trend` (trend analysis with slope + anomaly flags), `get_audit_trail` + (who-did-what for an entity or actor), `get_event_timeline` (structured event + feed for an entity or time range), `get_agent_activity` (what the agent did), + `get_health_summary` (current health across the fleet with trend indicators) - All read from PostgreSQL **REST interface** (`internal/api/routes/`): @@ -1026,6 +1303,26 @@ One Go binary (Gin web framework) exposing REST + MCP from the same codebase. - `GET /api/v1/ontology` — list entity types, relationship types, lifecycles - `POST /api/v1/ontology/entity_types` — create entity type (extend the schema) - `WS /api/v1/events` — real-time stream (signals, approvals, executions, feedback) +- **Observability routes** (`internal/api/routes/observability.go`): + - `GET /api/v1/metrics` — query time-series: `?entity_id=&metric=&from=&to=&interval=` + - Returns raw samples or rollups (auto-selects 1h/1d aggregates based on range) + - `?rollup=1h|1d|raw` to force a specific resolution + - Supports multiple metrics: `?metric=disk_usage_pct&metric=probe_latency_ms` + - `GET /api/v1/metrics/{entity_id}/{metric}` — single metric for one entity + - `?from=2026-07-01T00:00:00Z&to=2026-07-06T00:00:00Z&rollup=1h` + - Returns: `{entity_id, metric, samples: [{ts, avg, min, max, count}], trend: {slope, direction, anomaly}}` + - `GET /api/v1/trends/{entity_id}` — trend analysis for all metrics on an entity + - Returns slope (improving/degrading/stable), recent anomalies, forecast (simple linear) + - `GET /api/v1/audit` — audit log: `?actor_type=&actor_id=&entity_id=&action=&from=&to=` + - Paginated, ordered by ts DESC + - `?correlation_id=` to trace a full execution chain + - `GET /api/v1/events` — historical events: `?type=&entity_id=&severity=&from=&to=` + - Same data as the WebSocket stream, but queryable historically + - `GET /api/v1/agent-activity` — agent behavior log: `?agent_id=&activity_type=&entity_id=&from=&to=` + - Includes token usage, latency, success/failure per tool call + - `GET /api/v1/health` — fleet health summary with trend indicators + - Returns: `{entities: [{id, type, health, trend, last_probe}], summary: {healthy, degraded, down, unknown}}` + - `GET /api/v1/export` — export current DB state as YAML (for DR / version control) **Policy enforcement** (`internal/policy/classify.go`): - Every mutating endpoint classifies the action via the policy DB @@ -1184,6 +1481,96 @@ Keep apps/105 running as fallback. Cutover checklist when ready: 4. Update Caddy backends exclusively to mac-mini 5. Remove/retarget Gitea webhooks +### 14. Observability + data capture — Go (`internal/observability/`) + +The OS captures three classes of data, all in PostgreSQL (TimescaleDB for +time-series, regular tables for audit/events): + +**A. Metrics (time-series)** — `internal/observability/metrics.go`: +- **Infrastructure probes**: every scheduler probe writes metrics: + - `health` (0=down, 1=degraded, 2=healthy) per service + - `disk_usage_pct` per mount point per host + - `probe_latency_ms` per probe target + - `drift_count` per drift check +- **OS self-metrics**: the API server exposes a `/metrics` endpoint and also + writes to the DB: + - `api_request_count` (per route, per status code) + - `api_latency_ms` (p50, p99) + - `db_connections_active` + - `db_query_duration_ms` + - `goroutines` (Go runtime) + - `gc_pause_ms` + - `memory_alloc_mb` +- **Learning metrics**: the learning engine writes: + - `pattern_confidence` per pattern ID + - `skill_success_rate` per skill ID + - `auto_act_count` vs `escalation_count` (autonomy ratio over time) + - `execution_duration_ms` per (entity_type, action) +- **Agent metrics**: Hermes activity is logged: + - `agent_token_count` per session + - `agent_tool_call_count` per session + - `agent_decision_latency_ms` +- All metrics are written to `metric_samples` as hypertable with 1h and 1d + continuous aggregates. Raw data retained 90 days, rollups 1 year. + +**B. Audit log (immutable)** — `internal/observability/audit.go`: +- Gin middleware: every mutating API call (POST/PATCH/DELETE) writes to + `audit_log` with actor, action, entity, method, path, status, detail, source_ip +- MCP tool calls also audited (actor_type='agent') +- SSH commands by the actuator audited (actor_type='system', method='SSH') +- Policy mutations get special audit entries with a hash of the before/after state +- `correlation_id` propagated through the call chain (API → actuator → SSH → result) + so a full execution chain can be reconstructed: "signal → classification → + execution → SSH command → verification → feedback" all linked by correlation_id + +**C. Event stream** — `internal/observability/events.go`: +- Every significant state change emits an event to the `events` table: + - Signal lifecycle: `signal.raised`, `signal.acknowledged`, `signal.acting`, + `signal.resolved`, `signal.muted` + - Execution: `execution.started`, `execution.completed`, `execution.failed` + - Approval: `approval.requested`, `approval.decided` + - Deploy: `deploy.triggered`, `deploy.completed` + - Learning: `pattern.validated`, `skill.refined` + - Entity: `entity.created`, `entity.state_changed` + - Policy: `policy.changed` +- The WebSocket `/api/v1/events` streams from this table (new events pushed to + subscribers, historical events queryable via REST) +- Events carry `correlation_id` for end-to-end tracing + +**D. Agent activity** — `internal/observability/agent.go`: +- Hermes MCP tool calls logged to `agent_activity` with tool name, entity + acted upon, duration, token count, success/failure +- SSH commands by the agent logged separately +- Reasoning/decision audit: when the agent makes a classification decision, + the reasoning is recorded (input, classification result, route, why) +- This is the "what is the agent doing and is it getting better?" dataset + +**E. Structured logging** — `internal/observability/logging.go`: +- All services use Go's `slog` (structured JSON logging) +- Every log line has: `ts`, `level`, `service`, `msg`, `correlation_id` + (when applicable), `entity_id` (when applicable) +- Logs go to stdout (Docker captures them, `docker compose logs` for access) +- Debug mode: `debug=true` env var enables verbose probe payloads, SQL queries, + classification reasoning in logs + +**F. Data availability for agents**: +- The agent can query its own history: "what actions have I taken on service:caddy + in the last 30 days, and what were the outcomes?" +- The agent can see trends: "is disk usage on hubris trending upward?" +- The agent can audit: "who changed the policy for service:caddy and when?" +- The agent can self-assess: "am I getting more efficient? (token usage trend)" +- All through MCP tools: `query_metrics`, `get_trend`, `get_audit_trail`, + `get_event_timeline`, `get_agent_activity`, `get_health_summary` + +**G. Future visualization plug-in points**: +- The `/api/v1/metrics` REST endpoint returns JSON time-series — any tool + (Grafana, custom dashboard, notebook) can consume it +- The data model is compatible with Grafana's PostgreSQL data source + (time column + metric name + value + tags as labels) +- The event stream (`/api/v1/events` + WebSocket) can feed a live dashboard +- The audit log can feed a SIEM or compliance tool +- No UI built now — the APIs are the contract; visualization plugs in later + ## Phasing **Phase 0 — Ontology design (no code):** @@ -1193,21 +1580,27 @@ Keep apps/105 running as fallback. Cutover checklist when ready: **Phase 1 — Foundation (Go + DB):** - Go module setup, project structure -- PostgreSQL migrations (all 5) +- PostgreSQL + TimescaleDB setup +- Migrations 1-6 (ontology, instances, operations, learning, policy, observability) - Seed ingest pipeline (YAML → DB) - sqlc queries for core operations +- Structured logging setup (slog) - Import existing signals/ledger data **Phase 2 — API (Go):** - Gin server with REST routes -- MCP protocol adapter +- MCP protocol adapter (including observability tools) - Policy enforcement middleware +- Audit middleware (every mutating call → audit_log) +- Event emitter (every state change → events table) - Knowledge graph ingestion from docs/ +- Observability routes (metrics, trends, audit, events, health) **Phase 3 — Control loop (Go):** -- Scheduler (Observe) — probes, signals, state snapshots -- Actuator (Act) — classify, execute, verify -- Learning engine — feedback, patterns, skills +- Scheduler (Observe) — probes, signals, state snapshots, **metric recording** +- Actuator (Act) — classify, execute, verify, **correlation ID propagation** +- Learning engine — feedback, patterns, skills, **learning metrics** +- Circuit breaker per target host **Phase 4 — Agent (Hermes container):** - Hermes Docker image, gateway config @@ -1352,26 +1745,39 @@ should be addressed during the phase they belong to.** 1. **Ontology:** `seeds/ontology.yaml` ingested — `SELECT * FROM entity_types` shows all types across 3 layers; lifecycle definitions match the state machine diagrams. -2. **DB:** `docker compose up postgres` — all 5 migrations applied; seed ingest - populates entities + relationships from `inventory.yaml`. +2. **DB:** `docker compose up postgres` — all 6 migrations applied (including + TimescaleDB extension + hypertables); seed ingest populates entities + + relationships from `inventory.yaml`. 3. **API:** `curl http://localhost:8090/api/v1/entities?type=service` returns the fleet; MCP `list_services` works via the same endpoint. -4. **Scheduler:** trigger a probe pass — signals in DB, state snapshots written. +4. **Scheduler:** trigger a probe pass — signals in DB, state snapshots written, + **metrics in `metric_samples`** (health, probe_latency_ms, disk_usage_pct). 5. **Actuator:** raise a test `service-down` signal → actuator classifies → - auto-acts (restart) or escalates (Matrix) → execution recorded → feedback generated. + auto-acts (restart) or escalates (Matrix) → execution recorded → feedback generated + → **event emitted + audit entry written + correlation_id links the full chain**. 6. **Learning:** after N executions of the same (entity_type, action), a pattern appears with confidence score; after enough evidence, a skill is created. + **`pattern_confidence` metric visible in time-series.** 7. **Classifier with learning:** set `autonomy.auto_act: reversible_low` → next similar signal: classifier checks pattern confidence → auto-acts if high, escalates if low. Kill-switch (`auto_act: off`) → always escalates. 8. **Hermes:** connect from another workstation → agent responds, queries API via - MCP, can SSH to hubris. + MCP, can SSH to hubris. **Agent activity logged to `agent_activity` table.** 9. **Secrets:** Infisical running, Go services fetch secrets, SOPS files removed. 10. **Deploy:** `git push` → Gitea webhook → `docker compose build + up -d` → changes live, seed ingest syncs any YAML changes to DB. + **`deploy.triggered` + `deploy.completed` events in the event log.** 11. **Knowledge:** MCP `search_knowledge("caddy")` returns docs linked to `entity:service:caddy`. -12. **Cutover:** stop apps/105, verify production traffic only from Docker OS. +12. **Observability:** `curl /api/v1/metrics?entity_id=host:hubris&metric=disk_usage_pct` + returns time-series with trend. `curl /api/v1/audit?entity_id=service:caddy` + returns the full audit trail. `curl /api/v1/health` returns fleet summary + with trend indicators. MCP `get_trend("host:hubris", "disk_usage_pct")` + returns slope + anomaly detection. +13. **Correlation tracing:** follow a `correlation_id` from signal → classification → + execution → SSH command → verification → feedback → pattern update, all + linked in the audit_log + events table. +14. **Cutover:** stop apps/105, verify production traffic only from Docker OS. ## Out of scope (for now)