Files
oikos/docs/adr/0014-entity-model.md
dtoro 551497e0b3
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
adr: move to docs/adr/, renumber 0013 + 0014, update README index
2026-07-08 22:18:58 +02:00

487 lines
30 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Oikos Entity Model — Types, Relationships & Interactions
**Status:** Adopted
**Date:** 2026-07-08
**Scope:** Full inventory of every entity type, relationship, state machine, and
cognition pipeline — with clear markers for what is **code-real** vs **schema-only**.
---
## 1. Entity Type Hierarchy (56 types)
```
layer: meta
entity ★ (abstract root)
layer: infrastructure ──────────────────────────────────────────────────
domain: physical
site ups sensor peripheral
domain: compute
compute-entity ★ (abstract)
machine ★ (abstract)
proxmox-host standalone-server workstation appliance
vm
container ★ (abstract)
lxc docker-container
hypervisor
domain: network
network ★ (abstract)
lan mesh vlan
network-interface dns-zone dns-record
ingress-route certificate firewall-rule
domain: storage
storage-pool volume backup-target dataset
domain: software
service application config-repo deploy-pipeline
package-set cluster compose-stack
domain: external
domain-registration cloud-service isp-link vendor-dependency
layer: governance ──────────────────────────────────────────────────────
domain: identity
person agent identity-provider account
secret key access-grant
layer: cognition ── the OODA loop ──────────────────────────────────────
domain: cognition
check signal classification execution feedback
pattern skill approval
document runbook investigation
★ = abstract (cannot be instantiated; acts as polymorphic target for relationships)
```
### Concrete instances (88 active entities)
| Type | Count | Examples |
|------|-------|---------|
| `lxc` | 19 | jellyfin, caddy, dns, gitea, nextcloud, matrix, arriman… |
| `service` | 25 | caddy, authentik, dns, jellyfin, paperless, matrix… |
| `ingress-route` | 21 | *.hubris.network |
| `config-repo` | 6 | caddy-conf, gitea-customizations, mule-image… |
| `proxmox-host` | 2 | hubris, strong |
| `workstation` | 2 | mac-mini, republic-laptop |
| `standalone-server` | 1 | netbird-vps |
| `vm` | 2 | zimaos, haos |
| `storage-pool` | 3 | local-lvm-hubris, library-hubris, ludo-lvm |
| `volume` | 2 | library, media-local |
| + sites, networks, agents, documents, destroyed… | | |
---
## 2. Core Sequence: Machine Onboarding
```
Operator Oikos API DB Scheduler Target Machine
┌────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ POST │ │ │ │ │ │ │ │ │
│/entities│────>│Create │ │ │ │ │ │ │
│ │ │Entity() │ │ │ │ │ │ │
│ │ │ │────>│INSERT │ │ │ │ │
│ │ │ │ │entities │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ensure │ │ │ │ │ │ │
│ │ │Default │────>│INSERT │ │ │ │ │
│ │ │Checks() │ │check_defs│ │ │ │ │
│ │ │→ ping │ │×6 │ │ │ │ │
│ │ │→ cpu │ │ │ │ │ │ │
│ │ │→ memory │ │ │ │ │ │ │
│ │ │→ load │ │(target_id│ │ │ │ │
│ │ │→ disk │ │ set) │ │ │ │ │
│ │ │→ updates │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │<────│201 │ │ │ │ │ │ │
│ │ │Created │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ │ │ │ ── 30s tick ─>│ │ │
│ │ │ │ │ │ loads │ │ │
│ │ │ │ │ │ check_defs │ │ │
│ │ │ │ │ │ │──SSH────>│ │
│ │ │ │ │ │ │ /opt/ │ │
│ │ │ │ │ │ │ oikos/ │ │
│ │ │ │ │ │ │ checks/ │ │
│ │ │ │ │ │ │ cpu.sh │ │
│ │ │ │ │ │ │<──JSON───│ │
│ │ │ │ │<─────────│INSERT │ │ │
│ │ │ │ │metric │metric_samples │ │ │
│ │ │ │ │samples │ │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ │ │<─────────│UPSERT │ │ │
│ │ │ │ │entity │entity_status │ │ │
│ │ │ │ │status │(health) │ │ │
│ │ │ │ │ │ │ │ │
```
**What's code-real here:**
- `CreateEntity()` at `internal/httpapi/impl.go:811` — handles POST, validates type, calls `ensureDefaultChecks()`
- `ensureDefaultChecks()``internal/checkdefaults/defaults.go:144` — resolves host IP, SSH user, creates 6 check_defs rows with target_id
- Scheduler at `internal/scheduler/scheduler.go:26` — loads `ListEnabledCheckDefs`, dispatches by kind, writes metrics + signals
---
## 3. Core Sequence: The OODA Loop (observe → orient → decide → act)
```
┌─────────────────────────────────────────────────────────────────────┐
│ OBSERVE (Scheduler) │
│ │
│ Every 30s: │
│ ┌──────────┐ ListEnabledCheckDefs ┌──────────┐ │
│ │scheduler │─────────────────────────>│ Postgres │ │
│ │.go:54 │ │ │ │
│ └──────────┘ └──────────┘ │
│ │ │
│ ├── ping ──> exec.Command("ping", host) │
│ ├── http ──> http.Get(url) │
│ ├── tcp ──> net.DialTimeout("tcp", addr) │
│ ├── disk ──> unix.Statfs(path) │
│ ├── cert-expiry ──> tls.Dial + cert.NotAfter │
│ └── ssh-script ──> exec.Command("ssh", host, script) │
│ │ │
│ ┌───────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ checkResult │ {health, signalKind, evidence, metrics}│
│ └─────────────┘ │
│ │ │
│ ┌──────────┼──────────┐ │
│ ▼ ▼ ▼ │
│ metric_samples signals entity_status │
│ INSERT UPSERT UPSERT │
│ (every cycle) (dedup by (health + last_check_at) │
│ target+kind) │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ ORIENT (Classification) │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ For each open signal: │ │
│ │ │ │
│ │ classify_by_policy(signal, entity, blast_radius) │ │
│ │ │ │ │
│ │ ├── read_only ───────────> route: auto_act │ │
│ │ ├── reversible_low ──────> route: auto_act │ │
│ │ │ (if global.auto_act=on + not in never_auto_act)│ │
│ │ ├── config_mutation ─────> route: escalate │ │
│ │ └── destructive ─────────> route: hold │ │
│ │ │ │
│ │ INSERT INTO classifications │ │
│ │ edge: classifies → signal │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ ⚠ classification creation: schema defined, NOT yet wired │
│ (policy.ClassifySignal exists but scheduler doesn't call it) │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ DECIDE (Approval Gate) │
│ │
│ For route=auto_act: │
│ skip approval, execute immediately │
│ │
│ For route=escalate (config_mutation): │
│ POST /api/v1/executions ──> INSERT approval (status=pending) │
│ notifier.go sends Matrix alert with HMAC token │
│ operator replies ✅ or ❌ │
│ DecideApproval() → systemctl restart / apt upgrade │
│ │
│ For route=hold (destructive): │
│ queued for operator, requires explicit confirmation │
│ (never auto-executed even with global.auto_act=on) │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ ACT (Execution) │
│ │
│ ┌──────────┐ request_execution ┌──────────┐ │
│ │ Nomos │───────────────────────>│ MCP tool │ │
│ │ (agent) │ │ server.go │ │
│ └──────────┘ └──────────┘ │
│ │ │
│ ┌───────────┼───────────┐ │
│ ▼ ▼ ▼ │
│ reversible config_ destructive │
│ _low mutation │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ immediate approval hold │
│ execute queue (never auto) │
│ │ │ │
│ ▼ ▼ │
│ actuator. Matrix │
│ Execute() alert → │
│ (SSH exec) operator │
│ → approves │
│ → actuator.Execute() │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ LEARN (Patterns + Skills) │
│ │
│ execution ──produces──> feedback ──contributes-to──> pattern │
│ │ │
│ informs │
│ ▼ │
│ skill │
│ │
│ ⚠ Schema defined, NOT yet wired: │
│ - No code writes feedback records │
│ - No code transitions patterns hypothesized→validated │
│ - Skill execution against JSON procedure definitions not built │
└─────────────────────────────────────────────────────────────────────┘
```
### What's code-real in the OODA loop
| Phase | Table | Code | Status |
|-------|-------|------|--------|
| Observe | `check_defs`, `metric_samples` | `scheduler.go:26-215` | ✅ fully wired, 6 probe kinds |
| Observe → Orient | `signals` | `scheduler.go:130-141` (UpsertSignal) | ✅ dedup, severity, events |
| Orient | `classifications` | `policy/classify.go` (function exists) | ⚠ function defined but scheduler never calls it |
| Decide | `approvals` | `server.go:311-367` (request_execution) | ✅ escalation gate works |
| Act | `executions` | `actuator/exec.go` (SSH exec) | ✅ systemctl, apt, pct |
| Learn | `feedback`, `patterns`, `skills` | tables + list endpoints only | ⚠ schema only, no write path |
---
## 4. Relationship Types — The Edge Catalog (34 edges)
### Infrastructure Topology
```
host:hubris ──hosts──> lxc:jellyfin, lxc:caddy, lxc:dns, ... (machine provisions LXCs)
host:strong ──hosts──> lxc:jellyfin, lxc:arriman, ... (migrated LXCs)
host:hubris ──member-of──> cluster:homelab
host:strong ──member-of──> cluster:homelab
lxc:caddy ──provides──> service:caddy
lxc:gitea ──provides──> service:gitea
lxc:dns ──provides──> service:dns
host:hubris ──mounts──> volume:library (attrs: mount_point=/mnt/library)
host:hubris ──stores-on──> pool:library-hubris
```
### Network
```
ingress:paperless.hubris.network ──routes-to──> service:paperless
ingress:paperless.hubris.network ──secured-by──> idp:authentik
ingress:paperless.hubris.network ──uses-certificate──> cert:*.hubris.network
service:jellyfin ──authenticates-via──> idp:authentik (OIDC)
dns:paperless ──in-zone──> zone:hubris.network
dns:paperless ──resolves-to──> lxc:caddy (caddy terminates)
host:hubris ──connects-via──> lan:lab
host:strong ──connects-via──> lan:household
```
### Service Dependencies
```
service:jellyfin ──depends-on──> service:authentik (OIDC auth)
service:paperless ──depends-on──> service:authentik
service:arr-stack ──depends-on──> service:jellyfin
(depends-on edges feed blast_radius() — recursive CTE)
```
### Cognition (OODA edges)
```
check:ssh-script:d419257d ──checks──> host:hubris
check:ssh-script:d419257d ──raises──> signal:cpu-pressure (when unhealthy)
signal:cpu-pressure ──about──> host:hubris
classification:xyz ──classifies──> signal:cpu-pressure
classification:xyz ──precedes──> execution:restart-xyz
execution:restart-xyz ──targets──> host:hubris
execution:restart-xyz ──performs──> agent:nomos
```
### Governance
```
person:dtoro ──owns──> agent:nomos
person:dtoro ──decides──> approval:xyz
idp:authentik ──authenticates──> person:dtoro
```
---
## 5. Lifecycle State Machines
### Infrastructure (15 concrete types use this)
```
planned ──> provisioning ──> active ──> migrating ──> active
│ │ │ │
│ │ └── failed ──┘
│ │ └── deprecated ──> destroyed
│ │
│ └── failed ──> active (recovery)
└── destroyed (cancelled)
Terminal: [destroyed]
Default: active
```
**Real precondition checks** (code in `impl.go:1494-1579`):
| Transition | Precondition | How it's checked |
|------------|-------------|-----------------|
| provisioning→active | `health-check-answering` | `SELECT health FROM entity_status WHERE entity_id=$1` — must be healthy |
| provisioning→active | `age-key-enrolled-if-needed` | Checks `attributes->>'age_pubkey'` (workstation only) |
| provisioning→active | `mesh-joined-if-needed` | Checks `attributes->>'mesh_ip'` (workstation only) |
| provisioning→active | `doc-page-complete` | `SELECT count(*) FROM relationships WHERE target_id=$1 AND type='documents'` |
| deprecated→destroyed | `no-inbound-edges` | `SELECT count(*) FROM relationships WHERE target_id=$1 AND valid_to IS NULL` |
| any → terminated | `backups-verified` | Checks flag in entity attributes |
| any → terminated | `secrets-revoked` | Checks flag in entity attributes |
**Soft preconditions** (always pass — operator-confirmed): `inventory-entry`, `ip-reserved`, `preflight-passed`, `backup-verified`, `replacement-live`, `caddy-backends-checked`, `un-deprecate-note`, etc.
### Signal
```
raised ──> acknowledged ──> acting ──> resolved
│ │ │
├── muted ├── muted ├── raised (retry budget)
│ │ │
└── resolved└── resolved └── failed ──> acknowledged (operator-retry)
Terminal: [resolved]
Default: raised
```
**Implemented preconditions:**
- `raised → muted`: requires `mute_until` set (MuteSignal handler, `impl.go:615-689`)
- `acting → resolved`: requires `verification-passed` (soft — operator confirms)
**Dedup mechanism:** `UNIQUE INDEX uq_signals_open ON signals(target_entity_id, kind) WHERE state NOT IN ('resolved','failed')` — at most one open signal per (entity, kind). Repeated failures call `UpsertSignal` which increments `occurrence_count` on the existing row.
---
## 6. What's Code-Real vs Schema-Only
### ✅ Fully Implemented (code exists, running in production)
| Component | File(s) | What it does |
|-----------|---------|-------------|
| Entity CRUD | `impl.go:811-966` | Create, read, patch, list entities |
| Lifecycle transitions | `impl.go:1494-1579` | Precondition checks + state transitions |
| Relationship management | `seed.go` (ingest) | Create edges with `valid_from/valid_to` |
| Client enrollment | `impl.go:1134-1236` | `POST /clients/enroll` — age keypair, Infisical, state: provisioning |
| Check definitions | `phase3.go:265-376` | CreateCheck, ListChecks, PatchCheck |
| Scheduler observe | `scheduler.go:26-215` | 6 probe kinds, metric_samples, signals, entity_status |
| Signals | `scheduler.go:81-161` | UpsertSignal (dedup), ResolveSignal, severity evaluation |
| Executions | `server.go:286-411` | request_execution MCP tool — reversible_low/config_mutation/destructive |
| Approvals | `server.go:970-1052` | createApproval, DecideApproval → executeApprovedAction |
| Notifier | `notifier/notifier.go` | Matrix alerts for pending approvals |
| Patterns | `phase3.go:1100+` | ListPatterns, PatchPattern (status/quarantine) |
| Skills | `phase3.go:1300+` | ListSkills, PatchSkill, ListSkillVersions |
| Default checks | `checkdefaults/defaults.go` | Auto-create checks on entity creation/enrollment/seed |
| TimescaleDB metrics | `metric_samples` table | Hypertable with 1h/1d continuous aggregates, 90-day retention |
| Events + SSE | `events` table + pg_notify | Real-time UI updates via SSE endpoint |
| Audit log | `audit_log` hypertable | Every mutation with actor + action |
| Knowledge entities | `knowledge_entities` | Documents, runbooks, investigations with FTS |
| MCP tools | `server.go` | 24 tools for observe/orient/decide/act |
| Blast radius | `blast_radius()` fn | Recursive CTE — depends-on + hosts + routes-to edges |
### ⚠ Schema Defined, Not Yet Wired (table exists, no active code path creates rows)
| Component | What's Missing |
|-----------|---------------|
| `classifications` auto-creation | `policy.ClassifySignal()` exists but scheduler never calls it. Signals are raised but never automatically classified. The `GetOpenSignalsForAutoAct` query would return signals with auto-act classification, but the classify step is manual-only. |
| `feedback` records | No code writes to the `feedback` table. Execution results are not analyzed for patterns. |
| Pattern auto-learning | No code transitions patterns from `hypothesized → validated`. The lifecycle requires `evidence-count≥5 + confidence≥0.7` but no aggregation runs. |
| Skill execution | Skill entities carry a JSON `procedure` field but no execution engine reads or runs it. |
| `drift` check kind | Defined in OpenAPI and `check_defs.kind` enum, but no scheduler implementation exists. |
### 📋 Defined in Seeds Only (ontology.yaml references, no DB schema)
| Item | Notes |
|------|-------|
| Relationship type `cluster` | Mentioned in inventory but not in ontology relationship_types |
| `certificate` entity type | Referenced in `uses-certificate` edges but no concrete certificates in inventory |
| Relationship type `powers` / `monitors` | Not defined in relationship_types |
---
## 7. Database Physical Schema (Key Tables)
```
entity_types ──FK──> lifecycle_defs
│ FK (entities.type)
entities ──FK──> entity_types
├──FK──> entity_status (dual)
├──FK──> check_defs (dual; check_defs.target_id → entities)
├──FK──> signals (dual; signals.target_entity_id → entities)
├──FK──> classifications (dual)
├──FK──> executions (dual; executions.target_entity_id → entities)
├──FK──> feedback (dual)
├──FK──> patterns (dual)
├──FK──> skills (dual)
├──FK──> approvals (dual; approvals.subject_entity_id → entities)
├──FK──> knowledge_entities (dual)
└──>→ relationships (source_id, target_id → entities)
relationship_types ──FK──> entity_types (source_type, target_type)
│ FK (relationships.type)
relationships ──FK──> entities (source_id, target_id)
└── unique index: (source_id, target_id, type) WHERE valid_to IS NULL
approval_rules ──FK──> entity_types (entity_type)
autonomy_settings (key/value, no FKs)
risk_classes (standalone)
metric_samples (TimescaleDB hypertable — ts dimension)
events (TimescaleDB hypertable — ts dimension, pg_notify trigger for SSE)
audit_log (TimescaleDB hypertable — ts dimension)
agent_activity (TimescaleDB hypertable — ts dimension)
```
**Key architectural patterns:**
- **Dual entities:** `check_defs`, `signals`, `classifications`, `executions`, `feedback`, `patterns`, `skills`, `approvals`, `knowledge_entities` — all have `entity_id UUID PK REFERENCES entities(id)`. Every row is also an entity.
- **Partial unique indexes:** `relationships` (current edges), `signals` (open signals), `patterns` (per-type action) — all use `WHERE` clauses for snapshot semantics.
- **TimescaleDB:** 4 hypertables with continuous aggregates and retention policies.
- **SSE fan-out:** `pg_notify('oikos_events', ...)` trigger on `events` INSERT → Go listener fan-out → SSE connections.
---
## 8. How Nomos Queries Thermals — End-to-End Trace
```
User: "what are the thermals of hubris?"
Nomos calls MCP: query_metrics(metric=["cpu_pct","cpu_temp"])
server.go:getMetricHistory()
SELECT time_bucket('1h', ts) AS bucket,
avg(value), min(value), max(value)
FROM metric_samples
WHERE metric IN ('cpu_pct', 'cpu_temp')
AND entity_id = (SELECT id FROM entities WHERE slug = 'host:hubris')
GROUP BY bucket
Returns: cpu_pct ≈ 15%, cpu_temp ≈ 48°C (from TimescaleDB continuous aggregate)
Nomos formats and presents results to user
```
**What made this possible (chronologically):**
1. `scheduler.go` refactored to return metrics map → `checkResult{metrics}`
2. `ssh-script` check kind implemented → SSH exec to remote host
3. `cpu_check.sh` deployed to hubris → returns `{"metrics":{"cpu_pct":2.5,"cpu_temp":48}}`
4. Check created: `POST /checks {"kind":"ssh-script","target":"host:hubris","config":{"host":"192.168.8.77","script":"cpu_check.sh"}}`
5. Fixed: `InsertMetricSample` missing `ts` column → `now()` literal
6. Fixed: SSH port/user parsing bugs
7. Fixed: SSH warnings polluting JSON output
8. Scheduler loop → metrics written to TimescaleDB every 60s
9. MCP `query_metrics` reads from TimescaleDB → Nomos gets live data