diff --git a/adr/entity-model.md b/adr/entity-model.md new file mode 100644 index 0000000..552c23b --- /dev/null +++ b/adr/entity-model.md @@ -0,0 +1,486 @@ +# Oikos Entity Model — Types, Relationships & Interactions + +**Status:** Adopted +**Date:** 2026-07-08 +**Scope:** Full inventory of every entity type, relationship, state machine, and +cognition pipeline — with clear markers for what is **code-real** vs **schema-only**. + +--- + +## 1. Entity Type Hierarchy (56 types) + +``` +layer: meta + entity ★ (abstract root) + +layer: infrastructure ────────────────────────────────────────────────── + domain: physical + site ups sensor peripheral + + domain: compute + compute-entity ★ (abstract) + machine ★ (abstract) + proxmox-host standalone-server workstation appliance + vm + container ★ (abstract) + lxc docker-container + hypervisor + + domain: network + network ★ (abstract) + lan mesh vlan + network-interface dns-zone dns-record + ingress-route certificate firewall-rule + + domain: storage + storage-pool volume backup-target dataset + + domain: software + service application config-repo deploy-pipeline + package-set cluster compose-stack + + domain: external + domain-registration cloud-service isp-link vendor-dependency + +layer: governance ────────────────────────────────────────────────────── + domain: identity + person agent identity-provider account + secret key access-grant + +layer: cognition ── the OODA loop ────────────────────────────────────── + domain: cognition + check signal classification execution feedback + pattern skill approval + document runbook investigation + + ★ = abstract (cannot be instantiated; acts as polymorphic target for relationships) +``` + +### Concrete instances (88 active entities) + +| Type | Count | Examples | +|------|-------|---------| +| `lxc` | 19 | jellyfin, caddy, dns, gitea, nextcloud, matrix, arriman… | +| `service` | 25 | caddy, authentik, dns, jellyfin, paperless, matrix… | +| `ingress-route` | 21 | *.hubris.network | +| `config-repo` | 6 | caddy-conf, gitea-customizations, mule-image… | +| `proxmox-host` | 2 | hubris, strong | +| `workstation` | 2 | mac-mini, republic-laptop | +| `standalone-server` | 1 | netbird-vps | +| `vm` | 2 | zimaos, haos | +| `storage-pool` | 3 | local-lvm-hubris, library-hubris, ludo-lvm | +| `volume` | 2 | library, media-local | +| + sites, networks, agents, documents, destroyed… | | | + +--- + +## 2. Core Sequence: Machine Onboarding + +``` + Operator Oikos API DB Scheduler Target Machine + ┌────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ POST │ │ │ │ │ │ │ │ │ + │/entities│────>│Create │ │ │ │ │ │ │ + │ │ │Entity() │ │ │ │ │ │ │ + │ │ │ │────>│INSERT │ │ │ │ │ + │ │ │ │ │entities │ │ │ │ │ + │ │ │ │ │ │ │ │ │ │ + │ │ │ensure │ │ │ │ │ │ │ + │ │ │Default │────>│INSERT │ │ │ │ │ + │ │ │Checks() │ │check_defs│ │ │ │ │ + │ │ │→ ping │ │×6 │ │ │ │ │ + │ │ │→ cpu │ │ │ │ │ │ │ + │ │ │→ memory │ │ │ │ │ │ │ + │ │ │→ load │ │(target_id│ │ │ │ │ + │ │ │→ disk │ │ set) │ │ │ │ │ + │ │ │→ updates │ │ │ │ │ │ │ + │ │ │ │ │ │ │ │ │ │ + │ │<────│201 │ │ │ │ │ │ │ + │ │ │Created │ │ │ │ │ │ │ + │ │ │ │ │ │ │ │ │ + │ │ │ │ │ │ ── 30s tick ─>│ │ │ + │ │ │ │ │ │ loads │ │ │ + │ │ │ │ │ │ check_defs │ │ │ + │ │ │ │ │ │ │──SSH────>│ │ + │ │ │ │ │ │ │ /opt/ │ │ + │ │ │ │ │ │ │ oikos/ │ │ + │ │ │ │ │ │ │ checks/ │ │ + │ │ │ │ │ │ │ cpu.sh │ │ + │ │ │ │ │ │ │<──JSON───│ │ + │ │ │ │ │<─────────│INSERT │ │ │ + │ │ │ │ │metric │metric_samples │ │ │ + │ │ │ │ │samples │ │ │ │ + │ │ │ │ │ │ │ │ │ + │ │ │ │ │<─────────│UPSERT │ │ │ + │ │ │ │ │entity │entity_status │ │ │ + │ │ │ │ │status │(health) │ │ │ + │ │ │ │ │ │ │ │ │ +``` + +**What's code-real here:** +- `CreateEntity()` at `internal/httpapi/impl.go:811` — handles POST, validates type, calls `ensureDefaultChecks()` +- `ensureDefaultChecks()` → `internal/checkdefaults/defaults.go:144` — resolves host IP, SSH user, creates 6 check_defs rows with target_id +- Scheduler at `internal/scheduler/scheduler.go:26` — loads `ListEnabledCheckDefs`, dispatches by kind, writes metrics + signals + +--- + +## 3. Core Sequence: The OODA Loop (observe → orient → decide → act) + +``` + ┌─────────────────────────────────────────────────────────────────────┐ + │ OBSERVE (Scheduler) │ + │ │ + │ Every 30s: │ + │ ┌──────────┐ ListEnabledCheckDefs ┌──────────┐ │ + │ │scheduler │─────────────────────────>│ Postgres │ │ + │ │.go:54 │ │ │ │ + │ └──────────┘ └──────────┘ │ + │ │ │ + │ ├── ping ──> exec.Command("ping", host) │ + │ ├── http ──> http.Get(url) │ + │ ├── tcp ──> net.DialTimeout("tcp", addr) │ + │ ├── disk ──> unix.Statfs(path) │ + │ ├── cert-expiry ──> tls.Dial + cert.NotAfter │ + │ └── ssh-script ──> exec.Command("ssh", host, script) │ + │ │ │ + │ ┌───────┘ │ + │ ▼ │ + │ ┌─────────────┐ │ + │ │ checkResult │ {health, signalKind, evidence, metrics}│ + │ └─────────────┘ │ + │ │ │ + │ ┌──────────┼──────────┐ │ + │ ▼ ▼ ▼ │ + │ metric_samples signals entity_status │ + │ INSERT UPSERT UPSERT │ + │ (every cycle) (dedup by (health + last_check_at) │ + │ target+kind) │ + └─────────────────────────────────────────────────────────────────────┘ + + ┌─────────────────────────────────────────────────────────────────────┐ + │ ORIENT (Classification) │ + │ │ + │ ┌──────────────────────────────────────────────────────┐ │ + │ │ For each open signal: │ │ + │ │ │ │ + │ │ classify_by_policy(signal, entity, blast_radius) │ │ + │ │ │ │ │ + │ │ ├── read_only ───────────> route: auto_act │ │ + │ │ ├── reversible_low ──────> route: auto_act │ │ + │ │ │ (if global.auto_act=on + not in never_auto_act)│ │ + │ │ ├── config_mutation ─────> route: escalate │ │ + │ │ └── destructive ─────────> route: hold │ │ + │ │ │ │ + │ │ INSERT INTO classifications │ │ + │ │ edge: classifies → signal │ │ + │ └──────────────────────────────────────────────────────┘ │ + │ │ + │ ⚠ classification creation: schema defined, NOT yet wired │ + │ (policy.ClassifySignal exists but scheduler doesn't call it) │ + └─────────────────────────────────────────────────────────────────────┘ + + ┌─────────────────────────────────────────────────────────────────────┐ + │ DECIDE (Approval Gate) │ + │ │ + │ For route=auto_act: │ + │ skip approval, execute immediately │ + │ │ + │ For route=escalate (config_mutation): │ + │ POST /api/v1/executions ──> INSERT approval (status=pending) │ + │ notifier.go sends Matrix alert with HMAC token │ + │ operator replies ✅ or ❌ │ + │ DecideApproval() → systemctl restart / apt upgrade │ + │ │ + │ For route=hold (destructive): │ + │ queued for operator, requires explicit confirmation │ + │ (never auto-executed even with global.auto_act=on) │ + └─────────────────────────────────────────────────────────────────────┘ + + ┌─────────────────────────────────────────────────────────────────────┐ + │ ACT (Execution) │ + │ │ + │ ┌──────────┐ request_execution ┌──────────┐ │ + │ │ Nomos │───────────────────────>│ MCP tool │ │ + │ │ (agent) │ │ server.go │ │ + │ └──────────┘ └──────────┘ │ + │ │ │ + │ ┌───────────┼───────────┐ │ + │ ▼ ▼ ▼ │ + │ reversible config_ destructive │ + │ _low mutation │ + │ │ │ │ │ + │ ▼ ▼ ▼ │ + │ immediate approval hold │ + │ execute queue (never auto) │ + │ │ │ │ + │ ▼ ▼ │ + │ actuator. Matrix │ + │ Execute() alert → │ + │ (SSH exec) operator │ + │ → approves │ + │ → actuator.Execute() │ + └─────────────────────────────────────────────────────────────────────┘ + + ┌─────────────────────────────────────────────────────────────────────┐ + │ LEARN (Patterns + Skills) │ + │ │ + │ execution ──produces──> feedback ──contributes-to──> pattern │ + │ │ │ + │ informs │ + │ ▼ │ + │ skill │ + │ │ + │ ⚠ Schema defined, NOT yet wired: │ + │ - No code writes feedback records │ + │ - No code transitions patterns hypothesized→validated │ + │ - Skill execution against JSON procedure definitions not built │ + └─────────────────────────────────────────────────────────────────────┘ +``` + +### What's code-real in the OODA loop + +| Phase | Table | Code | Status | +|-------|-------|------|--------| +| Observe | `check_defs`, `metric_samples` | `scheduler.go:26-215` | ✅ fully wired, 6 probe kinds | +| Observe → Orient | `signals` | `scheduler.go:130-141` (UpsertSignal) | ✅ dedup, severity, events | +| Orient | `classifications` | `policy/classify.go` (function exists) | ⚠ function defined but scheduler never calls it | +| Decide | `approvals` | `server.go:311-367` (request_execution) | ✅ escalation gate works | +| Act | `executions` | `actuator/exec.go` (SSH exec) | ✅ systemctl, apt, pct | +| Learn | `feedback`, `patterns`, `skills` | tables + list endpoints only | ⚠ schema only, no write path | + +--- + +## 4. Relationship Types — The Edge Catalog (34 edges) + +### Infrastructure Topology +``` + host:hubris ──hosts──> lxc:jellyfin, lxc:caddy, lxc:dns, ... (machine provisions LXCs) + host:strong ──hosts──> lxc:jellyfin, lxc:arriman, ... (migrated LXCs) + host:hubris ──member-of──> cluster:homelab + host:strong ──member-of──> cluster:homelab + lxc:caddy ──provides──> service:caddy + lxc:gitea ──provides──> service:gitea + lxc:dns ──provides──> service:dns + host:hubris ──mounts──> volume:library (attrs: mount_point=/mnt/library) + host:hubris ──stores-on──> pool:library-hubris +``` + +### Network +``` + ingress:paperless.hubris.network ──routes-to──> service:paperless + ingress:paperless.hubris.network ──secured-by──> idp:authentik + ingress:paperless.hubris.network ──uses-certificate──> cert:*.hubris.network + service:jellyfin ──authenticates-via──> idp:authentik (OIDC) + dns:paperless ──in-zone──> zone:hubris.network + dns:paperless ──resolves-to──> lxc:caddy (caddy terminates) + host:hubris ──connects-via──> lan:lab + host:strong ──connects-via──> lan:household +``` + +### Service Dependencies +``` + service:jellyfin ──depends-on──> service:authentik (OIDC auth) + service:paperless ──depends-on──> service:authentik + service:arr-stack ──depends-on──> service:jellyfin + (depends-on edges feed blast_radius() — recursive CTE) +``` + +### Cognition (OODA edges) +``` + check:ssh-script:d419257d ──checks──> host:hubris + check:ssh-script:d419257d ──raises──> signal:cpu-pressure (when unhealthy) + signal:cpu-pressure ──about──> host:hubris + classification:xyz ──classifies──> signal:cpu-pressure + classification:xyz ──precedes──> execution:restart-xyz + execution:restart-xyz ──targets──> host:hubris + execution:restart-xyz ──performs──> agent:nomos +``` + +### Governance +``` + person:dtoro ──owns──> agent:nomos + person:dtoro ──decides──> approval:xyz + idp:authentik ──authenticates──> person:dtoro +``` + +--- + +## 5. Lifecycle State Machines + +### Infrastructure (15 concrete types use this) +``` + planned ──> provisioning ──> active ──> migrating ──> active + │ │ │ │ + │ │ └── failed ──┘ + │ │ └── deprecated ──> destroyed + │ │ + │ └── failed ──> active (recovery) + │ + └── destroyed (cancelled) + + Terminal: [destroyed] + Default: active +``` + +**Real precondition checks** (code in `impl.go:1494-1579`): +| Transition | Precondition | How it's checked | +|------------|-------------|-----------------| +| provisioning→active | `health-check-answering` | `SELECT health FROM entity_status WHERE entity_id=$1` — must be healthy | +| provisioning→active | `age-key-enrolled-if-needed` | Checks `attributes->>'age_pubkey'` (workstation only) | +| provisioning→active | `mesh-joined-if-needed` | Checks `attributes->>'mesh_ip'` (workstation only) | +| provisioning→active | `doc-page-complete` | `SELECT count(*) FROM relationships WHERE target_id=$1 AND type='documents'` | +| deprecated→destroyed | `no-inbound-edges` | `SELECT count(*) FROM relationships WHERE target_id=$1 AND valid_to IS NULL` | +| any → terminated | `backups-verified` | Checks flag in entity attributes | +| any → terminated | `secrets-revoked` | Checks flag in entity attributes | + +**Soft preconditions** (always pass — operator-confirmed): `inventory-entry`, `ip-reserved`, `preflight-passed`, `backup-verified`, `replacement-live`, `caddy-backends-checked`, `un-deprecate-note`, etc. + +### Signal +``` + raised ──> acknowledged ──> acting ──> resolved + │ │ │ + ├── muted ├── muted ├── raised (retry budget) + │ │ │ + └── resolved└── resolved └── failed ──> acknowledged (operator-retry) + + Terminal: [resolved] + Default: raised +``` + +**Implemented preconditions:** +- `raised → muted`: requires `mute_until` set (MuteSignal handler, `impl.go:615-689`) +- `acting → resolved`: requires `verification-passed` (soft — operator confirms) + +**Dedup mechanism:** `UNIQUE INDEX uq_signals_open ON signals(target_entity_id, kind) WHERE state NOT IN ('resolved','failed')` — at most one open signal per (entity, kind). Repeated failures call `UpsertSignal` which increments `occurrence_count` on the existing row. + +--- + +## 6. What's Code-Real vs Schema-Only + +### ✅ Fully Implemented (code exists, running in production) + +| Component | File(s) | What it does | +|-----------|---------|-------------| +| Entity CRUD | `impl.go:811-966` | Create, read, patch, list entities | +| Lifecycle transitions | `impl.go:1494-1579` | Precondition checks + state transitions | +| Relationship management | `seed.go` (ingest) | Create edges with `valid_from/valid_to` | +| Client enrollment | `impl.go:1134-1236` | `POST /clients/enroll` — age keypair, Infisical, state: provisioning | +| Check definitions | `phase3.go:265-376` | CreateCheck, ListChecks, PatchCheck | +| Scheduler observe | `scheduler.go:26-215` | 6 probe kinds, metric_samples, signals, entity_status | +| Signals | `scheduler.go:81-161` | UpsertSignal (dedup), ResolveSignal, severity evaluation | +| Executions | `server.go:286-411` | request_execution MCP tool — reversible_low/config_mutation/destructive | +| Approvals | `server.go:970-1052` | createApproval, DecideApproval → executeApprovedAction | +| Notifier | `notifier/notifier.go` | Matrix alerts for pending approvals | +| Patterns | `phase3.go:1100+` | ListPatterns, PatchPattern (status/quarantine) | +| Skills | `phase3.go:1300+` | ListSkills, PatchSkill, ListSkillVersions | +| Default checks | `checkdefaults/defaults.go` | Auto-create checks on entity creation/enrollment/seed | +| TimescaleDB metrics | `metric_samples` table | Hypertable with 1h/1d continuous aggregates, 90-day retention | +| Events + SSE | `events` table + pg_notify | Real-time UI updates via SSE endpoint | +| Audit log | `audit_log` hypertable | Every mutation with actor + action | +| Knowledge entities | `knowledge_entities` | Documents, runbooks, investigations with FTS | +| MCP tools | `server.go` | 24 tools for observe/orient/decide/act | +| Blast radius | `blast_radius()` fn | Recursive CTE — depends-on + hosts + routes-to edges | + +### ⚠ Schema Defined, Not Yet Wired (table exists, no active code path creates rows) + +| Component | What's Missing | +|-----------|---------------| +| `classifications` auto-creation | `policy.ClassifySignal()` exists but scheduler never calls it. Signals are raised but never automatically classified. The `GetOpenSignalsForAutoAct` query would return signals with auto-act classification, but the classify step is manual-only. | +| `feedback` records | No code writes to the `feedback` table. Execution results are not analyzed for patterns. | +| Pattern auto-learning | No code transitions patterns from `hypothesized → validated`. The lifecycle requires `evidence-count≥5 + confidence≥0.7` but no aggregation runs. | +| Skill execution | Skill entities carry a JSON `procedure` field but no execution engine reads or runs it. | +| `drift` check kind | Defined in OpenAPI and `check_defs.kind` enum, but no scheduler implementation exists. | + +### 📋 Defined in Seeds Only (ontology.yaml references, no DB schema) + +| Item | Notes | +|------|-------| +| Relationship type `cluster` | Mentioned in inventory but not in ontology relationship_types | +| `certificate` entity type | Referenced in `uses-certificate` edges but no concrete certificates in inventory | +| Relationship type `powers` / `monitors` | Not defined in relationship_types | + +--- + +## 7. Database Physical Schema (Key Tables) + +``` + entity_types ──FK──> lifecycle_defs + │ + │ FK (entities.type) + ▼ + entities ──FK──> entity_types + │ + ├──FK──> entity_status (dual) + ├──FK──> check_defs (dual; check_defs.target_id → entities) + ├──FK──> signals (dual; signals.target_entity_id → entities) + ├──FK──> classifications (dual) + ├──FK──> executions (dual; executions.target_entity_id → entities) + ├──FK──> feedback (dual) + ├──FK──> patterns (dual) + ├──FK──> skills (dual) + ├──FK──> approvals (dual; approvals.subject_entity_id → entities) + ├──FK──> knowledge_entities (dual) + └──>→ relationships (source_id, target_id → entities) + + relationship_types ──FK──> entity_types (source_type, target_type) + │ + │ FK (relationships.type) + ▼ + relationships ──FK──> entities (source_id, target_id) + │ + └── unique index: (source_id, target_id, type) WHERE valid_to IS NULL + + approval_rules ──FK──> entity_types (entity_type) + autonomy_settings (key/value, no FKs) + risk_classes (standalone) + + metric_samples (TimescaleDB hypertable — ts dimension) + events (TimescaleDB hypertable — ts dimension, pg_notify trigger for SSE) + audit_log (TimescaleDB hypertable — ts dimension) + agent_activity (TimescaleDB hypertable — ts dimension) +``` + +**Key architectural patterns:** +- **Dual entities:** `check_defs`, `signals`, `classifications`, `executions`, `feedback`, `patterns`, `skills`, `approvals`, `knowledge_entities` — all have `entity_id UUID PK REFERENCES entities(id)`. Every row is also an entity. +- **Partial unique indexes:** `relationships` (current edges), `signals` (open signals), `patterns` (per-type action) — all use `WHERE` clauses for snapshot semantics. +- **TimescaleDB:** 4 hypertables with continuous aggregates and retention policies. +- **SSE fan-out:** `pg_notify('oikos_events', ...)` trigger on `events` INSERT → Go listener fan-out → SSE connections. + +--- + +## 8. How Nomos Queries Thermals — End-to-End Trace + +``` + User: "what are the thermals of hubris?" + │ + ▼ + Nomos calls MCP: query_metrics(metric=["cpu_pct","cpu_temp"]) + │ + ▼ + server.go:getMetricHistory() + │ + ▼ + SELECT time_bucket('1h', ts) AS bucket, + avg(value), min(value), max(value) + FROM metric_samples + WHERE metric IN ('cpu_pct', 'cpu_temp') + AND entity_id = (SELECT id FROM entities WHERE slug = 'host:hubris') + GROUP BY bucket + │ + ▼ + Returns: cpu_pct ≈ 15%, cpu_temp ≈ 48°C (from TimescaleDB continuous aggregate) + │ + ▼ + Nomos formats and presents results to user +``` + +**What made this possible (chronologically):** +1. `scheduler.go` refactored to return metrics map → `checkResult{metrics}` +2. `ssh-script` check kind implemented → SSH exec to remote host +3. `cpu_check.sh` deployed to hubris → returns `{"metrics":{"cpu_pct":2.5,"cpu_temp":48}}` +4. Check created: `POST /checks {"kind":"ssh-script","target":"host:hubris","config":{"host":"192.168.8.77","script":"cpu_check.sh"}}` +5. Fixed: `InsertMetricSample` missing `ts` column → `now()` literal +6. Fixed: SSH port/user parsing bugs +7. Fixed: SSH warnings polluting JSON output +8. Scheduler loop → metrics written to TimescaleDB every 60s +9. MCP `query_metrics` reads from TimescaleDB → Nomos gets live data