dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

plans: add observability — metrics, audit log, events, agent activity

Browse Source 
Adds comprehensive data capture layer to the OS plan:

1. TimescaleDB — PostgreSQL extension for time-series data. No separate
   database. Hypertables auto-partition, continuous aggregates provide
   1h/1d rollups, retention policies auto-drop old data.

2. Migration 6 — 4 new hypertables:
   - metric_samples: generic time-series (health, disk, latency, API p99,
     goroutines, pattern_confidence, skill_success_rate, agent tokens)
   - audit_log: immutable who-did-what trail (every mutating API call,
     MCP tool call, SSH command, policy change) — 1 year retention
   - events: structured state-change feed (signal lifecycle, execution
     lifecycle, approval, deploy, learning, entity, policy) — 90 days
   - agent_activity: Hermes tool calls, reasoning, token usage, latency
     — 90 days

3. Correlation IDs — propagated through the full call chain (signal →
   classification → execution → SSH → verification → feedback → pattern)
   so any action chain can be reconstructed end-to-end.

4. 6 new MCP tools for agent self-query:
   query_metrics, get_trend, get_audit_trail, get_event_timeline,
   get_agent_activity, get_health_summary

5. 8 new REST endpoints for metrics/audit/events/health/trends/export

6. Workstream 14 — observability + data capture (7 sub-components A-G):
   metrics, audit, events, agent activity, structured logging, agent
   data availability, future visualization plug-in points

7. New Mermaid diagram — observability data capture and query flow

8. Updated architecture diagram to show observability data flows

9. Updated phasing — observability woven into phases 1-3

10. Updated verification — 14 end-to-end checks (was 12), including
    metrics querying, audit trail, correlation tracing
This commit is contained in:
authentik Default Admin
2026-07-06 23:12:34 +02:00
parent 2d75544362
commit 3a35289f46
1 changed files with 421 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

436
plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md
View File

@@ -506,11 +506,11 @@ flowchart LR
graph TB graph TB
subgraph mac-mini["mac-mini — Docker host, always-on"] subgraph mac-mini["mac-mini — Docker host, always-on"]
subgraph services["Docker Compose — Go binaries"] subgraph services["Docker Compose — Go binaries"]
PG["PostgreSQL 16\n• entity/relationship store\n• signals • ledger\n• knowledge graph\n• patterns + skills\n• policy + ontology metadata"] PG["PostgreSQL 16 + TimescaleDB\n• entity/relationship store\n• signals • ledger\n• knowledge graph\n• patterns + skills\n• policy + ontology metadata\n• metric_samples (hypertable)\n• audit_log (hypertable)\n• events (hypertable)\n• agent_activity (hypertable)"]
INF["Infisical\n(secrets manager)"] INF["Infisical\n(secrets manager)"]
HERMES["Hermes Agent — gateway mode\n+ homelab skills\n+ MCP client → API\n+ SSH keys (mounted)"] HERMES["Hermes Agent — gateway mode\n+ homelab skills\n+ MCP client → API\n+ SSH keys (mounted)"]
API["Oikos API — Go (Gin)\n\nMCP: get_host, list_services,\nsearch_knowledge, get_relations\nREST: /hosts, /services, /signals,\n/exec, /approve, /deploy, /events\n\nPolicy enforcement + risk\nclassification + ledger"] API["Oikos API — Go (Gin)\n\nMCP: get_host, list_services,\nsearch_knowledge, get_relations,\nquery_metrics, get_trend,\nget_audit_trail, get_health_summary\nREST: /hosts, /services, /signals,\n/exec, /approve, /deploy, /events,\n/metrics, /audit, /health\n\nPolicy enforcement + risk\nclassification + ledger\nAudit middleware + event emitter"]
SCHED["Scheduler (Observe) — Go\n+ Actuator (Act) — Go\n• 10-min probes → DB\n• Classify → auto-act or escalate\n• Execution → feedback → patterns\n• SSH to hubris/strong"] SCHED["Scheduler (Observe) — Go\n+ Actuator (Act) — Go\n• 10-min probes → DB + metrics\n• Classify → auto-act or escalate\n• Execution → feedback → patterns\n• Correlation ID propagation\n• SSH to hubris/strong"]
NOTIFIER["Notifier — Go\n• Matrix (current)\n• Future: webhook, email"] NOTIFIER["Notifier — Go\n• Matrix (current)\n• Future: webhook, email"]
end end
DEPLOY["Gitea webhook →\ndocker compose build + up -d"] DEPLOY["Gitea webhook →\ndocker compose build + up -d"]
@@ -519,6 +519,9 @@ graph TB
HERMES -- MCP --> API HERMES -- MCP --> API
API --> PG API --> PG
SCHED --> PG SCHED --> PG
SCHED -- "metrics + events + audit" --> PG
API -- "audit + events" --> PG
HERMES -- "agent_activity" --> PG
SCHED -- SSH --> HUBRIS SCHED -- SSH --> HUBRIS
SCHED -- SSH --> STRONG SCHED -- SSH --> STRONG
API -- "escalate" --> NOTIFIER API -- "escalate" --> NOTIFIER
@@ -540,6 +543,75 @@ graph TB
WS -- "Hermes gateway" --> HERMES WS -- "Hermes gateway" --> HERMES
``` ```
### Observability — data capture and query flow
```mermaid
flowchart TB
subgraph sources["Data sources"]
SCHED_PROBE["Scheduler probes\nhealth, disk, latency, drift"]
API_CALLS["API calls\nrequest/response metrics"]
AGENT_CALLS["Agent activity\nMCP tools, SSH, reasoning"]
EXEC["Executions\nSSH commands, verification"]
LEARN["Learning engine\npatterns, skills, confidence"]
DEPLOY_EVT["Deploy events\nwebhook, build, restart"]
end
subgraph capture["Capture layer (Go)"]
METRICS_W["metrics.go\n→ metric_samples"]
AUDIT_W["audit.go\n→ audit_log"]
EVENTS_W["events.go\n→ events table"]
AGENT_W["agent.go\n→ agent_activity"]
end
subgraph store["PostgreSQL + TimescaleDB"]
MS["metric_samples\n(hypertable, 90d raw,\n1y rollups via CAGGs)"]
AL["audit_log\n(hypertable, 1y retention)"]
EV["events\n(hypertable, 90d retention)"]
AA["agent_activity\n(hypertable, 90d retention)"]
end
subgraph query["Query layer (API)"]
REST["REST endpoints\n/metrics, /audit,\n/events, /health,\n/trends, /agent-activity"]
MCP["MCP tools\nquery_metrics, get_trend,\nget_audit_trail, get_event_timeline,\nget_agent_activity, get_health_summary"]
WS["WebSocket\n/api/v1/events\n(live stream)"]
end
subgraph consumers["Consumers"]
AGENT["Agent\nqueries trends, audits\nits own history"]
FUTURE["Future: Grafana,\ndashboards, notebooks,\nSIEM, compliance tools"]
end
SCHED_PROBE --> METRICS_W
API_CALLS --> METRICS_W
API_CALLS --> AUDIT_W
AGENT_CALLS --> AGENT_W
EXEC --> EVENTS_W
EXEC --> AUDIT_W
LEARN --> METRICS_W
LEARN --> EVENTS_W
DEPLOY_EVT --> EVENTS_W
METRICS_W --> MS
AUDIT_W --> AL
EVENTS_W --> EV
AGENT_W --> AA
MS --> REST
AL --> REST
EV --> REST
EV --> WS
AA --> REST
MS --> MCP
AL --> MCP
EV --> MCP
AA --> MCP
REST --> AGENT
MCP --> AGENT
REST --> FUTURE
WS --> FUTURE
```
### OODA loop — with learning feedback ### OODA loop — with learning feedback
```mermaid ```mermaid
@@ -669,6 +741,13 @@ flowchart TB
│ ├── notifier/ # notification abstraction │ ├── notifier/ # notification abstraction
│ │ ├── notifier.go # interface │ │ ├── notifier.go # interface
│ │ └── matrix.go # Matrix implementation │ │ └── matrix.go # Matrix implementation
│ ├── observability/ # logging, metrics, audit, events (NEW)
│ │ ├── logging.go # slog structured logging setup
│ │ ├── metrics.go # metric recording (writes to metric_samples)
│ │ ├── audit.go # audit middleware (writes to audit_log)
│ │ ├── events.go # event emitter (writes to events table)
│ │ ├── agent.go # agent activity recording
│ │ └── correlation.go # correlation ID propagation (context-based)
│ └── config/ # config loading (env, files) │ └── config/ # config loading (env, files)
│ └── config.go │ └── config.go
├── migrations/ # SQL migrations (golang-migrate format) ├── migrations/ # SQL migrations (golang-migrate format)
@@ -677,7 +756,8 @@ flowchart TB
│ ├── 002_instances.up.sql # entities, relationships │ ├── 002_instances.up.sql # entities, relationships
│ ├── 003_operations.up.sql # signals, approvals, executions, feedback │ ├── 003_operations.up.sql # signals, approvals, executions, feedback
│ ├── 004_learning.up.sql # patterns, skills │ ├── 004_learning.up.sql # patterns, skills
│ └── 005_policy.up.sql # policies, risk_classes, autonomy │ ├── 005_policy.up.sql # policies, risk_classes, autonomy
│ └── 006_observability.up.sql # metrics, audit_log, events, agent_activity
├── seeds/ # YAML seed manifests (bootstrap + DR) ├── seeds/ # YAML seed manifests (bootstrap + DR)
│ ├── ontology.yaml # entity types, relationship types, lifecycles │ ├── ontology.yaml # entity types, relationship types, lifecycles
│ ├── inventory.yaml # entity instances (hosts, services, etc.) │ ├── inventory.yaml # entity instances (hosts, services, etc.)
@@ -966,6 +1046,198 @@ CREATE TABLE autonomy_settings (
); );
``` ```
### Migration 6: Observability (metrics, audit log, event log)
Uses **TimescaleDB** (PostgreSQL extension) for time-series data. Hypertables
auto-partition by time, continuous aggregates provide rollups, and retention policies
auto-drop old data. No separate database needed — everything stays in Postgres.
```sql
-- Enable TimescaleDB
CREATE EXTENSION IF NOT EXISTS timescaledb;
-- ─── Time-series metrics ──────────────────────────────────────────────
-- Generic metric store. Every probe, health check, and system measurement
-- writes here. Designed for high insert volume, time-range queries, and
-- continuous-aggregate rollups.
CREATE TABLE metric_samples (
ts TIMESTAMPTZ NOT NULL,
entity_id TEXT NOT NULL, -- which entity this metric is about
metric TEXT NOT NULL, -- 'health', 'disk_usage_pct', 'probe_latency_ms',
-- 'api_p99_ms', 'goroutines', 'db_connections',
-- 'pattern_confidence', 'skill_success_rate', ...
value DOUBLE PRECISION NOT NULL,
tags JSONB DEFAULT '{}'::JSONB -- arbitrary key-value labels:
-- {probe: "http", target: "192.168.8.121"},
-- {host: "hubris", mount: "/mnt/library"}, ...
);
-- Hypertable: partition by time, 1-week chunks
SELECT create_hypertable('metric_samples', 'ts', chunk_time_interval => INTERVAL '7 days');
-- Indexes for common query patterns
CREATE INDEX idx_metrics_entity_ts ON metric_samples(entity_id, ts DESC);
CREATE INDEX idx_metrics_metric_ts ON metric_samples(metric, ts DESC);
CREATE INDEX idx_metrics_tags ON metric_samples USING GIN(tags);
-- Retention: drop raw metrics older than 90 days (continuous aggregates keep rollups)
SELECT add_retention_policy('metric_samples', INTERVAL '90 days');
-- Continuous aggregate: 1-hour rollups (mean, min, max, count)
CREATE MATERIALIZED VIEW metric_rollups_1h
WITH (timescaledb.continuous) AS
SELECT
time_bucket('1 hour', ts) AS bucket,
entity_id,
metric,
avg(value) AS avg_value,
min(value) AS min_value,
max(value) AS max_value,
count(*) AS sample_count,
(array_agg(tags))[1] AS representative_tags
FROM metric_samples
GROUP BY bucket, entity_id, metric;
-- Refresh policy: refresh every 1 hour, keep 1 year of rollups
SELECT add_continuous_aggregate_policy('metric_rollups_1h',
start_offset => INTERVAL '2 hours',
end_offset => INTERVAL '5 minutes',
schedule_interval => INTERVAL '1 hour');
CREATE MATERIALIZED VIEW metric_rollups_1d
WITH (timescaledb.continuous) AS
SELECT
time_bucket('1 day', ts) AS bucket,
entity_id,
metric,
avg(value) AS avg_value,
min(value) AS min_value,
max(value) AS max_value,
count(*) AS sample_count
FROM metric_samples
GROUP BY bucket, entity_id, metric;
SELECT add_continuous_aggregate_policy('metric_rollups_1d',
start_offset => INTERVAL '2 days',
end_offset => INTERVAL '1 hour',
schedule_interval => INTERVAL '1 day');
-- ─── Audit log ────────────────────────────────────────────────────────
-- Every mutating action — by the OS, by an agent, or by an operator —
-- gets an immutable audit entry. This is the "who did what when" trail
-- that the ledger doesn't fully capture (ledger records OS decisions and
-- executions; audit captures ALL API calls including reads-that-matter
-- and operator interactions).
CREATE TABLE audit_log (
id BIGSERIAL PRIMARY KEY,
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
actor_type TEXT NOT NULL, -- 'agent', 'operator', 'system', 'scheduler'
actor_id TEXT, -- entity ID of the actor (agent entity, person entity)
action TEXT NOT NULL, -- 'api.call', 'entity.create', 'policy.update',
-- 'approval.decide', 'exec.request', 'deploy.trigger'
entity_id TEXT, -- entity affected (if any)
method TEXT, -- 'GET', 'POST', 'PATCH', 'DELETE', 'MCP', 'SSH'
path TEXT, -- API path or MCP tool name or SSH command
status_code INTEGER, -- HTTP status or 0 for non-HTTP
detail JSONB DEFAULT '{}'::JSONB, -- request body, response summary, extra context
source_ip TEXT, -- where the call came from
correlation_id TEXT -- links to execution_id / signal_id for tracing
);
SELECT create_hypertable('audit_log', 'ts', chunk_time_interval => INTERVAL '7 days');
CREATE INDEX idx_audit_actor ON audit_log(actor_type, actor_id, ts DESC);
CREATE INDEX idx_audit_entity ON audit_log(entity_id, ts DESC);
CREATE INDEX idx_audit_action ON audit_log(action, ts DESC);
CREATE INDEX idx_audit_correlation ON audit_log(correlation_id);
-- Retention: keep 1 year of audit logs
SELECT add_retention_policy('audit_log', INTERVAL '365 days');
-- ─── Event log ────────────────────────────────────────────────────────
-- Structured event stream — the "news feed" of the OS. Every significant
-- state change is an event: signal raised/resolved, execution started/completed,
-- approval requested/granted, deploy triggered, pattern validated, skill refined,
-- config changed, entity lifecycle transition. The WebSocket /api/v1/events
-- streams from this table; agents can also query it historically.
CREATE TABLE events (
id BIGSERIAL PRIMARY KEY,
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
type TEXT NOT NULL, -- 'signal.raised', 'signal.resolved',
-- 'execution.started', 'execution.completed',
-- 'approval.requested', 'approval.decided',
-- 'deploy.triggered', 'deploy.completed',
-- 'pattern.validated', 'skill.refined',
-- 'entity.created', 'entity.state_changed',
-- 'policy.changed', 'config.changed'
entity_id TEXT, -- primary entity involved
severity TEXT DEFAULT 'info', -- info, warning, critical
source TEXT NOT NULL, -- 'scheduler', 'actuator', 'api', 'hermes',
-- 'deploy', 'notifier', 'learning'
data JSONB DEFAULT '{}'::JSONB, -- event-specific payload
correlation_id TEXT -- links to signal/exec/approval for tracing
);
SELECT create_hypertable('events', 'ts', chunk_time_interval => INTERVAL '7 days');
CREATE INDEX idx_events_type_ts ON events(type, ts DESC);
CREATE INDEX idx_events_entity_ts ON events(entity_id, ts DESC);
CREATE INDEX idx_events_severity_ts ON events(severity, ts DESC);
CREATE INDEX idx_events_correlation ON events(correlation_id);
-- Retention: keep 90 days of events (signals/ledger have their own tables
-- for permanent records; events are the transient feed)
SELECT add_retention_policy('events', INTERVAL '90 days');
-- ─── Agent activity log ──────────────────────────────────────────────
-- Records what the agent (Hermes) does: tool calls, reasoning, decisions,
-- token usage, latency. This is for agent behavior auditing and trend
-- analysis ("is the agent getting more efficient?").
CREATE TABLE agent_activity (
id BIGSERIAL PRIMARY KEY,
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
agent_id TEXT NOT NULL, -- entity ID of the agent
session_id TEXT, -- Hermes session ID
activity_type TEXT NOT NULL, -- 'tool_call', 'reasoning', 'decision',
-- 'mcp_query', 'ssh_command', 'escalation'
tool_name TEXT, -- MCP tool or CLI command called
entity_id TEXT, -- entity acted upon (if any)
input_summary TEXT, -- truncated input (first 500 chars)
output_summary TEXT, -- truncated output (first 500 chars)
duration_ms INTEGER,
token_count INTEGER, -- LLM tokens consumed (if applicable)
success BOOLEAN,
correlation_id TEXT
);
SELECT create_hypertable('agent_activity', 'ts', chunk_time_interval => INTERVAL '7 days');
CREATE INDEX idx_agent_activity_agent_ts ON agent_activity(agent_id, ts DESC);
CREATE INDEX idx_agent_activity_type_ts ON agent_activity(activity_type, ts DESC);
CREATE INDEX idx_agent_activity_entity ON agent_activity(entity_id, ts DESC);
CREATE INDEX idx_agent_activity_correlation ON agent_activity(correlation_id);
-- Retention: keep 90 days of agent activity
SELECT add_retention_policy('agent_activity', INTERVAL '90 days');
```
**What gets captured where:**
| Data source | Table | Retention | Purpose |
|---|---|---|---|
| Scheduler probes (HTTP health, disk, drift) | `metric_samples` | 90 days raw, 1 year rollups | Trend analysis, anomaly detection |
| API response latencies | `metric_samples` | same | Performance monitoring |
| Go runtime metrics (goroutines, mem, GC) | `metric_samples` | same | OS self-monitoring |
| Learning model metrics (pattern confidence, skill success rate) | `metric_samples` | same | Learning trend tracking |
| All mutating API calls | `audit_log` | 1 year | Auditing, compliance, forensics |
| Operator actions (approval decisions, entity edits, policy changes) | `audit_log` | 1 year | Operator accountability |
| Agent (Hermes) tool calls + reasoning | `agent_activity` | 90 days | Agent behavior auditing, efficiency tracking |
| Signal/exec/approval/deploy state changes | `events` | 90 days | Event stream (WebSocket), timeline reconstruction |
| Signals (permanent record) | `signals` | no expiry | Signal lifecycle tracking |
| Ledger entries (permanent record) | `ledger_entries` | no expiry | Change history |
| Executions (permanent record) | `executions` | no expiry | Execution audit trail |
## Workstreams ## Workstreams
### 1. Ontology definition + seed manifests (`seeds/`, `internal/ontology/`) ### 1. Ontology definition + seed manifests (`seeds/`, `internal/ontology/`)
@@ -1006,6 +1278,11 @@ One Go binary (Gin web framework) exposing REST + MCP from the same codebase.
- Tools: `get_host`, `list_services`, `search_knowledge`, `get_entity`, - Tools: `get_host`, `list_services`, `search_knowledge`, `get_entity`,
`get_relations`, `get_blast_radius`, `get_signal_history`, `get_ledger`, `get_relations`, `get_blast_radius`, `get_signal_history`, `get_ledger`,
`get_patterns`, `get_skills`, `get_state_snapshot` `get_patterns`, `get_skills`, `get_state_snapshot`
- **Observability tools**: `query_metrics` (time-series for an entity/metric),
`get_trend` (trend analysis with slope + anomaly flags), `get_audit_trail`
(who-did-what for an entity or actor), `get_event_timeline` (structured event
feed for an entity or time range), `get_agent_activity` (what the agent did),
`get_health_summary` (current health across the fleet with trend indicators)
- All read from PostgreSQL - All read from PostgreSQL
**REST interface** (`internal/api/routes/`): **REST interface** (`internal/api/routes/`):
@@ -1026,6 +1303,26 @@ One Go binary (Gin web framework) exposing REST + MCP from the same codebase.
- `GET /api/v1/ontology` — list entity types, relationship types, lifecycles - `GET /api/v1/ontology` — list entity types, relationship types, lifecycles
- `POST /api/v1/ontology/entity_types` — create entity type (extend the schema) - `POST /api/v1/ontology/entity_types` — create entity type (extend the schema)
- `WS /api/v1/events` — real-time stream (signals, approvals, executions, feedback) - `WS /api/v1/events` — real-time stream (signals, approvals, executions, feedback)
- **Observability routes** (`internal/api/routes/observability.go`):
- `GET /api/v1/metrics` — query time-series: `?entity_id=&metric=&from=&to=&interval=`
- Returns raw samples or rollups (auto-selects 1h/1d aggregates based on range)
- `?rollup=1h|1d|raw` to force a specific resolution
- Supports multiple metrics: `?metric=disk_usage_pct&metric=probe_latency_ms`
- `GET /api/v1/metrics/{entity_id}/{metric}` — single metric for one entity
- `?from=2026-07-01T00:00:00Z&to=2026-07-06T00:00:00Z&rollup=1h`
- Returns: `{entity_id, metric, samples: [{ts, avg, min, max, count}], trend: {slope, direction, anomaly}}`
- `GET /api/v1/trends/{entity_id}` — trend analysis for all metrics on an entity
- Returns slope (improving/degrading/stable), recent anomalies, forecast (simple linear)
- `GET /api/v1/audit` — audit log: `?actor_type=&actor_id=&entity_id=&action=&from=&to=`
- Paginated, ordered by ts DESC
- `?correlation_id=` to trace a full execution chain
- `GET /api/v1/events` — historical events: `?type=&entity_id=&severity=&from=&to=`
- Same data as the WebSocket stream, but queryable historically
- `GET /api/v1/agent-activity` — agent behavior log: `?agent_id=&activity_type=&entity_id=&from=&to=`
- Includes token usage, latency, success/failure per tool call
- `GET /api/v1/health` — fleet health summary with trend indicators
- Returns: `{entities: [{id, type, health, trend, last_probe}], summary: {healthy, degraded, down, unknown}}`
- `GET /api/v1/export` — export current DB state as YAML (for DR / version control)
**Policy enforcement** (`internal/policy/classify.go`): **Policy enforcement** (`internal/policy/classify.go`):
- Every mutating endpoint classifies the action via the policy DB - Every mutating endpoint classifies the action via the policy DB
@@ -1184,6 +1481,96 @@ Keep apps/105 running as fallback. Cutover checklist when ready:
4. Update Caddy backends exclusively to mac-mini 4. Update Caddy backends exclusively to mac-mini
5. Remove/retarget Gitea webhooks 5. Remove/retarget Gitea webhooks
### 14. Observability + data capture — Go (`internal/observability/`)
The OS captures three classes of data, all in PostgreSQL (TimescaleDB for
time-series, regular tables for audit/events):
**A. Metrics (time-series)** — `internal/observability/metrics.go`:
- **Infrastructure probes**: every scheduler probe writes metrics:
- `health` (0=down, 1=degraded, 2=healthy) per service
- `disk_usage_pct` per mount point per host
- `probe_latency_ms` per probe target
- `drift_count` per drift check
- **OS self-metrics**: the API server exposes a `/metrics` endpoint and also
writes to the DB:
- `api_request_count` (per route, per status code)
- `api_latency_ms` (p50, p99)
- `db_connections_active`
- `db_query_duration_ms`
- `goroutines` (Go runtime)
- `gc_pause_ms`
- `memory_alloc_mb`
- **Learning metrics**: the learning engine writes:
- `pattern_confidence` per pattern ID
- `skill_success_rate` per skill ID
- `auto_act_count` vs `escalation_count` (autonomy ratio over time)
- `execution_duration_ms` per (entity_type, action)
- **Agent metrics**: Hermes activity is logged:
- `agent_token_count` per session
- `agent_tool_call_count` per session
- `agent_decision_latency_ms`
- All metrics are written to `metric_samples` as hypertable with 1h and 1d
continuous aggregates. Raw data retained 90 days, rollups 1 year.
**B. Audit log (immutable)** — `internal/observability/audit.go`:
- Gin middleware: every mutating API call (POST/PATCH/DELETE) writes to
`audit_log` with actor, action, entity, method, path, status, detail, source_ip
- MCP tool calls also audited (actor_type='agent')
- SSH commands by the actuator audited (actor_type='system', method='SSH')
- Policy mutations get special audit entries with a hash of the before/after state
- `correlation_id` propagated through the call chain (API → actuator → SSH → result)
so a full execution chain can be reconstructed: "signal → classification →
execution → SSH command → verification → feedback" all linked by correlation_id
**C. Event stream** — `internal/observability/events.go`:
- Every significant state change emits an event to the `events` table:
- Signal lifecycle: `signal.raised`, `signal.acknowledged`, `signal.acting`,
`signal.resolved`, `signal.muted`
- Execution: `execution.started`, `execution.completed`, `execution.failed`
- Approval: `approval.requested`, `approval.decided`
- Deploy: `deploy.triggered`, `deploy.completed`
- Learning: `pattern.validated`, `skill.refined`
- Entity: `entity.created`, `entity.state_changed`
- Policy: `policy.changed`
- The WebSocket `/api/v1/events` streams from this table (new events pushed to
subscribers, historical events queryable via REST)
- Events carry `correlation_id` for end-to-end tracing
**D. Agent activity** — `internal/observability/agent.go`:
- Hermes MCP tool calls logged to `agent_activity` with tool name, entity
acted upon, duration, token count, success/failure
- SSH commands by the agent logged separately
- Reasoning/decision audit: when the agent makes a classification decision,
the reasoning is recorded (input, classification result, route, why)
- This is the "what is the agent doing and is it getting better?" dataset
**E. Structured logging** — `internal/observability/logging.go`:
- All services use Go's `slog` (structured JSON logging)
- Every log line has: `ts`, `level`, `service`, `msg`, `correlation_id`
(when applicable), `entity_id` (when applicable)
- Logs go to stdout (Docker captures them, `docker compose logs` for access)
- Debug mode: `debug=true` env var enables verbose probe payloads, SQL queries,
classification reasoning in logs
**F. Data availability for agents**:
- The agent can query its own history: "what actions have I taken on service:caddy
in the last 30 days, and what were the outcomes?"
- The agent can see trends: "is disk usage on hubris trending upward?"
- The agent can audit: "who changed the policy for service:caddy and when?"
- The agent can self-assess: "am I getting more efficient? (token usage trend)"
- All through MCP tools: `query_metrics`, `get_trend`, `get_audit_trail`,
`get_event_timeline`, `get_agent_activity`, `get_health_summary`
**G. Future visualization plug-in points**:
- The `/api/v1/metrics` REST endpoint returns JSON time-series — any tool
(Grafana, custom dashboard, notebook) can consume it
- The data model is compatible with Grafana's PostgreSQL data source
(time column + metric name + value + tags as labels)
- The event stream (`/api/v1/events` + WebSocket) can feed a live dashboard
- The audit log can feed a SIEM or compliance tool
- No UI built now — the APIs are the contract; visualization plugs in later
## Phasing ## Phasing
**Phase 0 — Ontology design (no code):** **Phase 0 — Ontology design (no code):**
@@ -1193,21 +1580,27 @@ Keep apps/105 running as fallback. Cutover checklist when ready:
**Phase 1 — Foundation (Go + DB):** **Phase 1 — Foundation (Go + DB):**
- Go module setup, project structure - Go module setup, project structure
- PostgreSQL migrations (all 5) - PostgreSQL + TimescaleDB setup
- Migrations 1-6 (ontology, instances, operations, learning, policy, observability)
- Seed ingest pipeline (YAML → DB) - Seed ingest pipeline (YAML → DB)
- sqlc queries for core operations - sqlc queries for core operations
- Structured logging setup (slog)
- Import existing signals/ledger data - Import existing signals/ledger data
**Phase 2 — API (Go):** **Phase 2 — API (Go):**
- Gin server with REST routes - Gin server with REST routes
- MCP protocol adapter - MCP protocol adapter (including observability tools)
- Policy enforcement middleware - Policy enforcement middleware
- Audit middleware (every mutating call → audit_log)
- Event emitter (every state change → events table)
- Knowledge graph ingestion from docs/ - Knowledge graph ingestion from docs/
- Observability routes (metrics, trends, audit, events, health)
**Phase 3 — Control loop (Go):** **Phase 3 — Control loop (Go):**
- Scheduler (Observe) — probes, signals, state snapshots - Scheduler (Observe) — probes, signals, state snapshots, **metric recording**
- Actuator (Act) — classify, execute, verify - Actuator (Act) — classify, execute, verify, **correlation ID propagation**
- Learning engine — feedback, patterns, skills - Learning engine — feedback, patterns, skills, **learning metrics**
- Circuit breaker per target host
**Phase 4 — Agent (Hermes container):** **Phase 4 — Agent (Hermes container):**
- Hermes Docker image, gateway config - Hermes Docker image, gateway config
@@ -1352,26 +1745,39 @@ should be addressed during the phase they belong to.**
1. **Ontology:** `seeds/ontology.yaml` ingested — `SELECT * FROM entity_types` shows 1. **Ontology:** `seeds/ontology.yaml` ingested — `SELECT * FROM entity_types` shows
all types across 3 layers; lifecycle definitions match the state machine diagrams. all types across 3 layers; lifecycle definitions match the state machine diagrams.
2. **DB:** `docker compose up postgres` — all 5 migrations applied; seed ingest 2. **DB:** `docker compose up postgres` — all 6 migrations applied (including
populates entities + relationships from `inventory.yaml`. TimescaleDB extension + hypertables); seed ingest populates entities +
relationships from `inventory.yaml`.
3. **API:** `curl http://localhost:8090/api/v1/entities?type=service` returns the 3. **API:** `curl http://localhost:8090/api/v1/entities?type=service` returns the
fleet; MCP `list_services` works via the same endpoint. fleet; MCP `list_services` works via the same endpoint.
4. **Scheduler:** trigger a probe pass — signals in DB, state snapshots written. 4. **Scheduler:** trigger a probe pass — signals in DB, state snapshots written,
**metrics in `metric_samples`** (health, probe_latency_ms, disk_usage_pct).
5. **Actuator:** raise a test `service-down` signal → actuator classifies → 5. **Actuator:** raise a test `service-down` signal → actuator classifies →
auto-acts (restart) or escalates (Matrix) → execution recorded → feedback generated. auto-acts (restart) or escalates (Matrix) → execution recorded → feedback generated
→ **event emitted + audit entry written + correlation_id links the full chain**.
6. **Learning:** after N executions of the same (entity_type, action), a pattern 6. **Learning:** after N executions of the same (entity_type, action), a pattern
appears with confidence score; after enough evidence, a skill is created. appears with confidence score; after enough evidence, a skill is created.
**`pattern_confidence` metric visible in time-series.**
7. **Classifier with learning:** set `autonomy.auto_act: reversible_low` → next 7. **Classifier with learning:** set `autonomy.auto_act: reversible_low` → next
similar signal: classifier checks pattern confidence → auto-acts if high, similar signal: classifier checks pattern confidence → auto-acts if high,
escalates if low. Kill-switch (`auto_act: off`) → always escalates. escalates if low. Kill-switch (`auto_act: off`) → always escalates.
8. **Hermes:** connect from another workstation → agent responds, queries API via 8. **Hermes:** connect from another workstation → agent responds, queries API via
MCP, can SSH to hubris. MCP, can SSH to hubris. **Agent activity logged to `agent_activity` table.**
9. **Secrets:** Infisical running, Go services fetch secrets, SOPS files removed. 9. **Secrets:** Infisical running, Go services fetch secrets, SOPS files removed.
10. **Deploy:** `git push` → Gitea webhook → `docker compose build + up -d` → 10. **Deploy:** `git push` → Gitea webhook → `docker compose build + up -d` →
changes live, seed ingest syncs any YAML changes to DB. changes live, seed ingest syncs any YAML changes to DB.
**`deploy.triggered` + `deploy.completed` events in the event log.**
11. **Knowledge:** MCP `search_knowledge("caddy")` returns docs linked to 11. **Knowledge:** MCP `search_knowledge("caddy")` returns docs linked to
`entity:service:caddy`. `entity:service:caddy`.
12. **Cutover:** stop apps/105, verify production traffic only from Docker OS. 12. **Observability:** `curl /api/v1/metrics?entity_id=host:hubris&metric=disk_usage_pct`
returns time-series with trend. `curl /api/v1/audit?entity_id=service:caddy`
returns the full audit trail. `curl /api/v1/health` returns fleet summary
with trend indicators. MCP `get_trend("host:hubris", "disk_usage_pct")`
returns slope + anomaly detection.
13. **Correlation tracing:** follow a `correlation_id` from signal → classification →
execution → SSH command → verification → feedback → pattern update, all
linked in the audit_log + events table.
14. **Cutover:** stop apps/105, verify production traffic only from Docker OS.
## Out of scope (for now) ## Out of scope (for now)