diff --git a/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md b/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md index 14c8c0b..1f03463 100644 --- a/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md +++ b/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md @@ -1,79 +1,135 @@ # Plan: Oikos — Docker-based agentic homelab OS on mac-mini -**Status:** Planned (2026-07-06, rev 2) — supersedes the launchd-based plan and the -Python/Docker rev 1. This revision introduces: Go instead of Python, ontology-first -design with systems modeling, DB-native config (inventory/ontology/policy as graph -metadata, not YAML files), and a feedback loop where the agent learns from execution. +**Status:** Planned (2026-07-06, **rev 3**) — supersedes rev 2. Rev 3 consolidates the +rev-2 audit + remediation layers into one self-consistent spec (no more "read the +migration, then read the fix section") and closes newly found gaps. This document is +the single source of truth for implementation; an agent should be able to implement +phase by phase from this file alone. + +## Rev 3 changelog + +Consolidation: +- All rev-2 HIGH/MEDIUM remediations (S1–S10, SA1–SA10, SG1–SG18, A1–A7, O1–O7, + D1–D7, P1–P7, M1–M7) are now merged inline. Appendix A maps every finding ID to + where it is resolved in this document. + +New in rev 3 (gaps found in external review): +- **R3-1 Ontology inheritance in the meta-schema.** The BDD uses generalization + heavily (`ComputeEntity <|-- Machine <|-- ProxmoxHost`), and relationships hang off + abstract types (`ComputeEntity provides Service`), but the rev-2 `entity_types` + table was flat. Added `parent_type` + `is_abstract`; relationship endpoint + validation walks the type hierarchy. +- **R3-2 Contract-first API.** `api/openapi.yaml` (OpenAPI 3.1) is the source of + truth; Go server stubs generated with `oapi-codegen` (chi router, strict server). + Future UIs get a generated TypeScript client. Gin dropped. +- **R3-3 API semantics for machines and UIs.** RFC 9457 problem+json errors, uniform + list envelope, cursor pagination, `Idempotency-Key` on unsafe POSTs, ETag/If-Match + optimistic concurrency, role scopes (operator/viewer/agent), CORS policy, and a + `/graph` endpoint for UI visualization. +- **R3-4 Single binary, role subcommands.** One `cmd/oikos` binary (`oikos api| + scheduler|notifier|all`), one Docker image, roles selected by compose `command` + (Loki/Temporal pattern). Guarantees version consistency; trivial local dev. +- **R3-5 IDs.** UUIDv7 primary keys (app-generated, time-ordered) + unique human + `slug` (`host:hubris`). API accepts either. Resolves D1 properly. +- **R3-6 P5 actually fixed.** `state_snapshots` (unbounded growth) replaced by a + single-row-per-entity `entity_status` table; health *history* lives in + `metric_samples` (already retained/rolled up). +- **R3-7 Checks as data.** Probe definitions (`check_defs`) live in the DB and are + ontology-attached — adding a probe is an API call, not a code deploy. +- **R3-8 Signal dedup + flap suppression + maintenance mode.** Partial unique index + guarantees one open signal per (entity, kind); occurrence counting; hold-down on + flapping; `maintenance_until` on entities suppresses signals/auto-act. +- **R3-9 Executable skill format.** `skills.procedure` is a JSON-schema-validated + structure (steps/verify/rollback/params) the actuator can run deterministically; + markdown is rendered *from* it for humans. +- **R3-10 MCP modernized.** Official `github.com/modelcontextprotocol/go-sdk`, + Streamable HTTP transport (SSE-only transport is deprecated in the MCP spec), + static bearer token auth. Tools are thin wrappers over the same service layer as + REST — one behavior, two protocols. +- **R3-11 Ledger simplified.** `ledger_entries` table dropped; `ledger` is a SQL view + over `executions ⋈ classifications ⋈ approvals`. One less write path to keep + consistent; `audit_log` already covers operator mutations. +- **R3-12 Release + rollback made concrete.** Images tagged with git SHA, last 5 + kept; rollback = redeploy previous tag + `pg_restore` of pre-deploy dump. +- **R3-13 macOS deployment realities + dual-path networking.** Docker-in-VM + (OrbStack), no host networking, sleep/auto-restart settings, launch-at-login. + Outbound SSH goes direct over the LAN; inbound is mesh-primary with a LAN + break-glass binding for the API (rev 3.1, operator decision). Previously silent. +- **R3-14 SSE event stream.** Server→client push is all we need; SSE is simpler than + WebSocket through Caddy and for browser UIs. WebSocket deferred. +- **R3-15 Phases now carry acceptance criteria** ("done when" + verification + commands) so an implementing agent knows when to stop. +- **R3-16 ADRs.** `docs/adr/` with MADR template; the Decisions table below seeds the + initial ADRs. Future architecture changes are recorded, not re-litigated. ## Vision -Convert this repo into a **Docker-based agentic homelab OS** written in **Go**. The OS -is a set of containerized services that manage the homelab autonomously, with the +Convert this repo into a **Docker-based agentic homelab OS** written in **Go**. The +OS is a set of containerized services that manage the homelab autonomously, with the operator in control. Two actors: - **Operator** (dtoro) — owns the homelab, expresses intent ("install X", "restart Y"), approves destructive actions. Connects from any workstation via remote Hermes or Matrix. -- **Agent** — Hermes core + custom homelab skills, running in Docker. Executes orders, - monitors the lab, escalates when unsure, and **learns from every action** to improve - over time. +- **Agent** — Hermes core + custom homelab skills, running in Docker. Executes + orders, monitors the lab, escalates when unsure, and **learns from every action**. -All OS services run in Docker containers on mac-mini. The OS is deployed by a git -push (Gitea webhook → Docker rebuild). It's designed for mac-mini now, with a path -to multi-node later. +All OS services run in Docker on mac-mini, deployed by git push (Gitea webhook → +image build → restart). Designed for mac-mini now with a path to multi-node later +(see "Multi-node path"). -## Decisions (from operator Q&A, 2026-07-06) +## Decisions | Question | Decision | |---|---| -| Repo structure | One repo, reorganize internally. | +| Repo structure | One repo, reorganized (see Repo layout). | | Language | **Go** — compiled, type-safe, small containers, goroutines for concurrent probes. | -| Hermes runtime | Runs inside Docker as part of the OS stack (gateway mode). | -| Agent → homelab access | Hybrid — mounted SSH keys now, actuator gateway built incrementally. | -| Data storage | PostgreSQL. Inventory/ontology/policy become DB-native graph metadata. | -| Config (inventory/ontology/policy) | **DB-native** — YAML files are seed manifests only (bootstrap + DR). The DB is the runtime source of truth. Editable via API, future frontend. | -| Knowledge/context | Structured knowledge graph in Postgres — ontology-defined entity types, typed relationships, linked to operational data. | -| Operator interface | Primary = remote Hermes from any workstation + Matrix. Console/UIs built later for specific tasks. | -| Host | mac-mini for now, designed to scale later. | -| Deploy | Git push → Gitea webhook → Docker rebuild + restart. | -| `bin/homelab` CLI | Replaced by an API. CLI becomes a thin Go client that calls the OS API. | -| Secrets | Migrate from SOPS+age to Infisical. | -| Matrix | Keep for now, abstract the notification layer for future channels. | -| MCP server | Merged into the unified API — one Go service, REST + MCP interfaces. | -| Agent type | Hermes core + custom homelab skills. | -| Feedback loop | **Agent learns from execution** — outcomes feed back as patterns and skills that inform future decisions. | -| Ontology | **Developed first**, before knowledge graph ingestion. Systems modeling: entities, connections, lifecycles. | -| apps/105 | Keep running as fallback until Docker OS is proven. | -| mac-mini cleanup | Start fresh in a new directory, clean up old artifacts later. | +| Packaging | **Single binary** `oikos` with role subcommands; one Docker image (R3-4). | +| API style | **OpenAPI-first** — `api/openapi.yaml` is the contract; oapi-codegen + chi; RFC 9457 errors (R3-2/3). | +| Hermes runtime | Docker container, gateway mode. | +| Agent → homelab access | Hybrid — restricted SSH key in the **actuator only** now; full actuator gateway in Phase 3. Hermes never holds SSH keys. | +| Data storage | PostgreSQL 16 + TimescaleDB. DB is the runtime source of truth. | +| Config (inventory/ontology/policy) | **DB-native**; YAML files are seed manifests (bootstrap + DR) with round-trip export. | +| IDs | UUIDv7 PK + unique `slug` for humans/API (R3-5). | +| Operator interface | Remote Hermes + Matrix now; UIs later on top of the API. | +| Deploy | Git push → CI green → Gitea webhook → SHA-tagged image build → compose up. | +| `bin/homelab` CLI | Thin Go client generated from the OpenAPI spec. | +| Secrets | Migrate SOPS+age → Infisical (one age key kept for DR fallback). | +| Notifications | Matrix now, behind a `Notifier` interface; DB is the rendezvous (no service-to-service calls). | +| MCP | Same binary/service layer as REST; official Go SDK, Streamable HTTP (R3-10). | +| Event stream | SSE (`/api/v1/events/stream`); WebSocket deferred (R3-14). | +| Feedback loop | Agent learns from execution; **pattern activation and any autonomy expansion require operator approval** (anti-poisoning). | +| Ontology | Developed first; meta-schema supports inheritance + abstract types (R3-1). | +| apps/105 | Keeps running (read-only toward shared state) as fallback until cutover. | +| ADRs | `docs/adr/` (MADR format); this table seeds ADR-0001…0010 (R3-16). | ## Ontology — the systems model -The ontology is the foundational layer of the OS. It defines what exists, how things -connect, how they change over time, and how the OS learns about them. It is stored -**in the database** as metadata (entity types, relationship types, lifecycle -definitions). YAML seed files bootstrap it on first deploy; after that, the DB is +The ontology defines what exists, how things connect, how they change over time, and +how the OS learns. It is stored **in the database** (entity types, relationship +types, lifecycle definitions). YAML seeds bootstrap it; after that the DB is authoritative and editable via API. -### Design principles (systems modeling) +### Design principles 1. **Three layers** — Infrastructure (the managed world), Governance (who controls - what), Cognition (the OS's own behavior + learning). Dependencies flow upward: + what), Cognition (the OS's behavior + learning). Dependencies flow downward: Cognition depends on Governance depends on Infrastructure. 2. **Everything is an entity** — if it can break, be changed, or hold data, it has an - entity type and edges. The OS's own objects (signals, changes, skills) are - first-class entities, not second-class records. -3. **Typed relationships with cardinality** — edges carry semantics. `hosts` is - one-to-many; `depends-on` is many-to-many; `documents` is one-to-one. The graph - is queryable for blast radius, dependency chains, and knowledge lookup. -4. **Lifecycles are state machines** — every entity type has a lifecycle. Infrastructure - entities move through `planned → active → destroyed`. Operational entities have - their own lifecycles (signals, approvals, patterns, skills). Transitions can require - preconditions. -5. **Policy is attached to the ontology** — risk classes and approval rules link to - entity types and actions. The policy IS part of the model, not a separate file. -6. **Learning is modeled** — executions produce outcomes, outcomes accumulate into - patterns, patterns refine skills, skills inform future decisions. This is an - explicit, queryable part of the graph. + entity type and edges. The OS's own objects (signals, executions, skills) are + first-class entities. +3. **Typed relationships with cardinality** — edges carry semantics and are + queryable (blast radius, dependency chains, knowledge lookup). +4. **Inheritance is part of the model** — entity types form an is-a hierarchy with + abstract types (`ComputeEntity`); relationship endpoint constraints may name + abstract types and validation walks the hierarchy (R3-1). +5. **Lifecycles are state machines** — every entity type has a lifecycle with + explicit terminal states and (named, code-implemented) transition preconditions. +6. **Policy attaches to the ontology** — risk classes and approval rules link to + entity types and actions. +7. **Learning is modeled but never self-authorizing** — executions → feedback → + patterns → skills is an explicit, queryable graph, but the learning engine only + *proposes* governance changes; the operator approves them (S4/SA2). ### Layer map @@ -82,16 +138,16 @@ graph TB cognition -- "observes, acts on, learns about" --> infra governance -- "governs access to" --> infra governance -- "constrains" --> cognition - cognition -- "creates + refines" --> governance + cognition -. "proposes changes (operator approves)" .-> governance subgraph cognition["Layer 3 — Cognition (OS behavior + learning)"] direction LR - OBS["Observation\nsignal, state-snapshot"] - DEC["Decision\nclassification, risk-assessment"] + OBS["Observation\nsignal, check, entity-status"] + DEC["Decision\nclassification"] ACT["Action\nexecution, verification"] - GOV["Governance\napproval-request, approval-decision"] + APPR["Approvals\napproval-request, approval-decision"] KNOW["Knowledge\ndocument, runbook"] - LEARN["Learning\npattern, skill, feedback"] + LEARN["Learning\nfeedback, pattern, skill"] end subgraph governance["Layer 2 — Governance (who controls what)"] @@ -107,21 +163,20 @@ graph TB COMP["Compute\nmachine, vm, container\n(lxc, docker)"] NET["Network\nlan, mesh, dns-zone,\ningress-route, certificate"] STOR["Storage\nstorage-pool, volume,\nmount, backup-target"] - SOFT["Software\nservice, application,\nconfig-repo, deploy-pipeline"] + SOFT["Software\nservice, application, cluster,\ncompose-stack, config-repo,\ndeploy-pipeline"] end ``` -### Block definition diagram (SysML BDD) +Note the dashed arrow: the learning engine **cannot write** to governance tables. A +validated pattern that would expand autonomy becomes an approval request; only an +operator decision changes policy (SA2, S4). -Uses Mermaid class diagram syntax following SysML BDD conventions: -- `«abstract»` = abstract block (cannot be instantiated) -- `<\|--` = generalization (is-a) -- `*--` = composition (whole-part, lifecycle dependency) -- `o--` = aggregation (whole-part, independent lifecycle) -- `--` = association (typed link) -- Multiplicity: `"1"`, `"0..1"`, `"1..*"`, `"0..*"`, `"*"` +### Block definition diagrams (SysML BDD) -**Infrastructure layer — compute, storage, network:** +Conventions: `«abstract»` = cannot be instantiated; `<|--` generalization; `*--` +composition; `o--` aggregation; `-->` association; multiplicities as labeled. + +**Infrastructure — compute, storage, network:** ```mermaid classDiagram @@ -130,46 +185,18 @@ classDiagram +state lifecycle +attributes jsonb } - class Machine { - +cpu_arch - +ram_gb - } - class VirtualMachine { - +vcpus - +memory_mb - +disk_gb - } - class Container { - <> - +runtime - } - class LXC { - +pve_id - +rootfs - } - class DockerContainer { - +image - +compose_stack - } - class ProxmoxHost { - +pve_version - +cluster_member - } - class StandaloneServer { - +hypervisor - } - class Workstation { - +os - +user - } - class Appliance { - +vendor - +model - } - class Hypervisor { - +type - +version - } + class Machine { +cpu_arch +ram_gb } + class VirtualMachine { +vcpus +memory_mb +disk_gb } + class Container { <> +runtime } + class LXC { +pve_id +rootfs } + class DockerContainer { +image +compose_stack } + class ProxmoxHost { +pve_version +cluster_member } + class StandaloneServer { +hypervisor +provider +control_level } + class Workstation { +os +user } + class Appliance { +vendor +model } + class Hypervisor { +type +version } + class Cluster { +quorum +members } + class ComposeStack { +path +services } ComputeEntity <|-- Machine ComputeEntity <|-- VirtualMachine @@ -184,34 +211,22 @@ classDiagram Machine "1" *-- "0..1" Hypervisor : runs Hypervisor "1" o-- "0..*" VirtualMachine : hosts Hypervisor "1" o-- "0..*" Container : hosts + ProxmoxHost "0..*" --> "0..1" Cluster : member-of + DockerContainer "0..*" --> "0..1" ComposeStack : part-of - class StoragePool { - +type lvm, zfs, nfs - +capacity_gb - } - class Volume { - +name - +size_gb - } - class Mount { - +mount_point - +options - } + class StoragePool { +type lvm, zfs, nfs +capacity_gb } + class Volume { +name +size_gb } + class Mount { +mount_point +options } StoragePool "1" *-- "0..*" Volume : contains ComputeEntity "1" o-- "0..*" Mount : has Mount "0..*" --> "1" Volume : mounts - class NetworkInterface { - +mac - +ip - } - class Network { - <> - } - class LAN { +subnet} - class Mesh { +provider} - class VLAN { +tag} + class NetworkInterface { +mac +ip } + class Network { <> } + class LAN { +subnet } + class Mesh { +provider } + class VLAN { +tag } ComputeEntity "1" *-- "0..*" NetworkInterface : has NetworkInterface "0..*" --> "1" Network : connects-to @@ -220,43 +235,18 @@ classDiagram Network <|-- VLAN ``` -**Software + services layer:** +**Software + services:** ```mermaid classDiagram - class Service { - +port - +health_url - +risk_notes - } - class Application { - +version - +config - } - class ConfigRepo { - +url - +branch - } - class DeployPipeline { - +trigger - +target_path - } - class IngressRoute { - +pattern - +upstream - } - class Certificate { - +issuer - +expires - } - class DNSZone { - +zone - } - class DNSRecord { - +name - +record_type - +value - } + class Service { +port +health_url +risk_notes } + class Application { +version +config } + class ConfigRepo { +url +branch } + class DeployPipeline { +trigger +target_path } + class IngressRoute { +pattern +upstream } + class Certificate { +issuer +expires } + class DNSZone { +zone } + class DNSRecord { +name +record_type +value } ComputeEntity "1" o-- "0..*" Service : provides Service "1" *-- "0..*" Application : runs @@ -270,60 +260,40 @@ classDiagram DNSRecord "0..*" --> "0..1" IngressRoute : resolves-to ``` -**Cognition layer — operations + learning:** +**Governance — identity (new in rev 2 remediation, kept):** ```mermaid classDiagram - class Signal { - +kind - +severity - +state lifecycle - +evidence - +recommended_action - } - class Execution { - +status - +result - +duration_ms - +verified - } - class Feedback { - +outcome - +observation - +lesson - } - class Pattern { - +confidence - +evidence_count - +status - } - class Skill { - +procedure - +version - +success_rate - } - class Classification { - +risk - +route - +reasoning - } - class Approval { - +status - +ttl - } - class Document { - +title - +content - +source_path - } - class Runbook { - +steps - +risk_class - +verification - } + class Person { +matrix_id +oidc_sub } + class Agent { +provider +model +gateway_port } + class IdentityProvider { +issuer +client_id +auth_mode } + class Secret { +path +rotation_days } + class AccessGrant { +scope +expires } - ComputeEntity "1" o-- "0..*" Signal : monitored-by - Signal "0..1" --> "0..*" Execution : triggers + Person "0..1" --> "0..*" Agent : owns + IdentityProvider "1" o-- "0..*" Person : authenticates + AccessGrant "0..*" --> "1" Secret : grants + Agent "0..*" --> "0..*" AccessGrant : holds +``` + +**Cognition — operations + learning:** + +```mermaid +classDiagram + class CheckDef { +kind +config +interval } + class Signal { +kind +severity +state +evidence +occurrences } + class Classification { +risk +route +reasoning } + class Execution { +status +result +duration_ms +verified } + class Feedback { +outcome +observation +lesson } + class Pattern { +confidence +evidence_count +status } + class Skill { +procedure +version +success_rate } + class Approval { +status +ttl } + class Document { +title +content +source_path } + class Runbook { +steps +risk_class +verification } + + CheckDef "0..*" --> "1" Entity : checks + CheckDef "1" o-- "0..*" Signal : raises + Signal "1" --> "0..*" Classification : classified-by Classification "1" --> "0..1" Execution : precedes Execution "1" *-- "0..1" Feedback : produces Feedback "0..*" --> "0..*" Pattern : contributes-to @@ -332,2032 +302,1238 @@ classDiagram Execution "0..1" --> "0..1" Approval : requires Agent "0..*" --> "0..*" Execution : performs Person "0..1" --> "0..*" Approval : decides - Person "0..1" --> "0..*" Agent : owns - Entity "1" o-- "0..*" Document : documented-by Entity "1" o-- "0..*" Runbook : procedure-for ``` -### Design notes — validated assumptions +Design notes carried from rev 2 (validated): VMs and LXCs share storage pools via +`Mount`→`Volume` on abstract `ComputeEntity`; not every machine is a Proxmox host +(fleet today: 2 PVE hosts, 2 workstations, 1 VPS, 2 VMs, 19 LXCs); Docker containers +are first-class (the OS models itself); services attach to any compute entity; +documents/runbooks attach to any entity via the root abstract type. -**Can VMs mount storage pools?** Yes. In Proxmox, VMs have virtual disks on storage -pools (LVM, ZFS, NFS). LXCs have bind mounts and mount points. Both use the same -storage pools. The model reflects this: `ComputeEntity` (abstract) has `Mount` edges -to `Volume`, regardless of whether the compute entity is a VM, LXC, or machine. +### Lifecycles -**Not every machine is a Proxmox host.** The current homelab has: 2 Proxmox hosts, 2 -workstations, 1 external VPS, 2 VMs, 19 LXCs. The model uses `Machine` as the base -type with specializations (`ProxmoxHost`, `StandaloneServer`, `Workstation`, -`Appliance`). Only `ProxmoxHost` runs PVE; `StandaloneServer` could run KVM/libvirt; -`Workstation` runs desktop OS + optionally Docker. A `Hypervisor` is software that -runs on a `Machine` and hosts VMs/containers — it's not always present (workstations -and appliances may not have one). +All lifecycles have explicit terminal states and recovery paths (SA4). Transition +preconditions are **named checks implemented in Go** and referenced by ID from +`lifecycle_defs.transitions` (e.g. `no-inbound-edges`, `backup-verified`) — the DB +stores which checks gate a transition; the code implements them. -**Docker containers are first-class compute entities.** The OS itself runs in Docker -containers, and services like Jellyfin's MariaDB sidecar run in Docker within LXCs. -`DockerContainer` is a specialization of `Container` with `image`, `compose_stack` -attributes. This lets the OS model its own infrastructure. - -**Services run on any compute entity.** The homelab has services on LXCs (caddy, -gitea), VMs (zimaos, haos), workstations (mac-mini will run the OS), and external -hosts (authentik on netbird-vps). The `provides` relationship is from -`ComputeEntity` (abstract), not from a specific compute type. - -**Documents and runbooks describe any entity.** The old ER diagram had -`Document → Service` only. In practice, docs describe hosts, containers, network -infrastructure, storage, and investigations. The model uses `Entity` (the root -abstract type) for `documented-by` and `procedure-for`, so any entity can have docs -and runbooks. - -### Infrastructure lifecycle +**Infrastructure:** ```mermaid stateDiagram-v2 [*] --> planned : operator creates entity planned --> provisioning : IP reserved, storage chosen, doc stub - provisioning --> active : mesh joined, health-check answering, doc complete + planned --> destroyed : cancelled + provisioning --> active : mesh joined, health answering, doc complete + provisioning --> failed : provision failed active --> migrating : preflight + backup verified - migrating --> active : post-verify, caddy checked, mounts checked + migrating --> active : post-verify (ingress, mounts checked) + migrating --> failed : migration failed + failed --> active : recovered + failed --> deprecated : written off active --> deprecated : replacement live or role retired - deprecated --> destroyed : backups verified, secrets revoked, ingress removed + deprecated --> active : un-deprecate (replacement failed) + deprecated --> destroyed : backups verified, secrets revoked,\ningress removed, zero inbound edges destroyed --> [*] : archaeology entry recorded - - note right of deprecated - Complete only when - zero inbound edges remain - (no depends-on, no routes-to) - end note ``` -### Signal lifecycle +**Signal:** ```mermaid stateDiagram-v2 - [*] --> raised : scheduler probe or agent finding + [*] --> raised : check fails / agent finding raised --> acknowledged : agent or operator sees it raised --> muted : operator suppresses (TTL) - acknowledged --> acting : actuator starts execution - acting --> resolved : action succeeded, verification passed - acting --> raised : action failed, re-escalated raised --> resolved : condition cleared (auto-resolve) - muted --> raised : TTL expired + acknowledged --> acting : actuator starts execution + acknowledged --> resolved : manual resolve + acknowledged --> muted + acting --> resolved : action + verification passed + acting --> raised : action failed, re-escalated (retry budget left) + acting --> failed : permanent failure — needs operator + failed --> acknowledged : operator retries + muted --> raised : TTL expired and condition persists resolved --> [*] ``` -### Execution + learning lifecycle +**Execution:** ```mermaid stateDiagram-v2 - [*] --> proposed : signal + recommended_action + [*] --> proposed : classification produced an action proposed --> approved : operator approves (if gated) proposed --> auto_approved : risk class allows auto-act - approved --> executing : actuator runs - auto_approved --> executing : actuator runs - executing --> verified : verification command passed + proposed --> denied : operator denies + approved --> expired : approval TTL ran out + approved --> executing + auto_approved --> executing + executing --> verified : verification passed executing --> failed : execution or verification failed - executing --> timed_out : exceeded duration limit + executing --> timed_out + executing --> cancelled : operator abort + timed_out --> verifying : check if command completed anyway + verifying --> verified + verifying --> failed failed --> rolled_back : rollback procedure executed - verified --> [*] : feedback recorded, pattern updated - failed --> [*] : feedback recorded, pattern updated - rolled_back --> [*] : feedback recorded, pattern updated - timed_out --> [*] : feedback recorded, pattern updated + failed --> rollback_failed : rollback also failed — page operator + verified --> [*] : feedback recorded + failed --> [*] : feedback recorded + rolled_back --> [*] : feedback recorded + rollback_failed --> [*] : feedback recorded + cancelled --> [*] + denied --> [*] + expired --> [*] ``` -### Feedback / learning model — the cognition loop +**Approval:** `pending → approved | denied | expired`; `approved → revoked` +(operator changes mind before execution starts). + +**Pattern:** `hypothesized → validated → active → deprecated`; `hypothesized → +invalidated` (disproven, terminal); `active → invalidated` (new evidence +contradicts). **`validated → active` requires operator approval** (S4). + +**Skill:** `drafted → tested → active`; `active → refined → active` (new version); +`tested → failed → drafted`; `drafted|active → deprecated`. + +### Cognition loop (observe → decide → act → learn) ```mermaid flowchart TB subgraph observe["Observe"] - SIG["Signal raised\n(service down, disk full, drift)"] + SIG["Signal raised\n(check failed, drift, agent finding)"] end - subgraph decide["Decide"] - CLASS["Classification\nrisk × blast radius × confidence"] - SKILL_LOOKUP["Skill lookup\nbest-known procedure for\nthis entity type + action"] + CLASS["Classification\nrisk × blast radius × confidence\n+ recommended action"] + SKILL_LOOKUP["Skill lookup\nbest-known procedure for\n(entity type, action)"] CLASS --> SKILL_LOOKUP end - subgraph act["Act"] - EXEC["Execution\nfollow skill procedure\n(or escalate if no skill)"] - VERIFY["Verification\ncheck if action succeeded"] + EXEC["Execution\nrun skill procedure\n(or escalate if none)"] + VERIFY["Verification"] EXEC --> VERIFY end - - subgraph learn["Learn (feedback loop)"] - OUTCOME["Outcome evaluation\nsuccess / failure / partial / unexpected"] - FEEDBACK["Feedback record\nwhat happened vs expected\nextractable lesson"] - PATTERN["Pattern extraction\naccumulate feedback on\nsimilar entity+action pairs"] - SKILL_REFINE["Skill refinement\nupdate or create skill\nbased on validated patterns"] - - OUTCOME --> FEEDBACK - FEEDBACK --> PATTERN - PATTERN --> SKILL_REFINE + subgraph learn["Learn"] + OUTCOME["Outcome evaluation"] + FEEDBACK["Feedback record"] + PATTERN["Pattern extraction"] + SKILL_REFINE["Skill refinement\n(operator gates activation)"] + OUTCOME --> FEEDBACK --> PATTERN --> SKILL_REFINE end - SIG --> CLASS SKILL_LOOKUP --> EXEC VERIFY --> OUTCOME - SKILL_REFINE -. "informs next decision" .-> SKILL_LOOKUP - - subgraph escalation["Escalation path"] - APPROVAL["Approval request\n→ Matrix ✅/❌"] - end - - CLASS -- "needs approval" --> APPROVAL - APPROVAL -- "approved" --> EXEC - APPROVAL -- "denied" --> RESOLVE["Resolve signal\nnote: denied"] + SKILL_REFINE -. informs next decision .-> SKILL_LOOKUP + CLASS -- needs approval --> APPROVAL["Approval request → Matrix ✅/❌"] + APPROVAL -- approved --> EXEC + APPROVAL -- denied --> RESOLVE["Resolve signal (denied)"] ``` -### Policy model — how risk attaches to entities - -```mermaid -flowchart LR - subgraph ontology["Ontology (DB metadata)"] - ET["entity_types\nhost, service, lxc, vm, ..."] - RT["relationship_types\nhosts, provides, depends-on, ..."] - LD["lifecycle_defs\nstates + transitions"] - end - - subgraph policy["Policy (DB records)"] - RC["risk_classes\nread_only, reversible_low,\nconfig_mutation, destructive"] - RULES["approval_rules\nentity_type + action → risk_class\n+ approval_required + autonomy_level"] - AUTO["autonomy_settings\nglobal: auto_act on/off\nper-entity: never_auto_act"] - end - - subgraph instances["Instances (DB data)"] - ENT["entities\nhost:hubris, service:caddy, ..."] - REL["relationships\nhubris hosts lxc:apps"] - end - - ET --> RULES - RC --> RULES - RULES --> ENT - AUTO --> ENT - ET --> ENT - RT --> REL - LD --> ENT -``` +`recommended_action` lives on the **classification**, not the signal (SA6): checks +raise facts; the classifier decides what to do about them. ## Target architecture ### Container stack on mac-mini +One image (`oikos:`), three long-running roles + init jobs, three Docker +networks as trust boundaries (S10): + +- `net-front` — Caddy-facing: `api` only. +- `net-data` — Postgres + everything that needs it. +- `net-ops` — SSH egress: **actuator role only** (Hermes and API have no SSH). + ```mermaid graph TB - subgraph mac-mini["mac-mini — Docker host, always-on"] - subgraph services["Docker Compose — Go binaries"] - PG["PostgreSQL 16 + TimescaleDB\n• entity/relationship store\n• signals • ledger\n• knowledge graph\n• patterns + skills\n• policy + ontology metadata\n• metric_samples (hypertable)\n• audit_log (hypertable)\n• events (hypertable)\n• agent_activity (hypertable)"] - INF["Infisical\n(secrets manager)"] - HERMES["Hermes Agent — gateway mode\n+ homelab skills\n+ MCP client → API\n+ SSH keys (mounted)"] - API["Oikos API — Go (Gin)\n\nMCP: get_host, list_services,\nsearch_knowledge, get_relations,\nquery_metrics, get_trend,\nget_audit_trail, get_health_summary\nREST: /hosts, /services, /signals,\n/exec, /approve, /deploy, /events,\n/metrics, /audit, /health\n\nPolicy enforcement + risk\nclassification + ledger\nAudit middleware + event emitter"] - SCHED["Scheduler (Observe) — Go\n+ Actuator (Act) — Go\n• 10-min probes → DB + metrics\n• Classify → auto-act or escalate\n• Execution → feedback → patterns\n• Correlation ID propagation\n• SSH to hubris/strong"] - NOTIFIER["Notifier — Go\n• Matrix (current)\n• Future: webhook, email"] + subgraph mac-mini["mac-mini — Docker host (OrbStack), always-on"] + subgraph stack["Docker Compose — image oikos:<sha>"] + MIGRATE["init: oikos migrate + seed\n(one-shot, DDL user)"] + PG["PostgreSQL 16 + TimescaleDB\nentities/relationships • signals\nexecutions • patterns/skills\npolicy • ontology • hypertables"] + INF["Infisical (secrets)"] + API["oikos api\nREST (OpenAPI) + MCP (streamable HTTP)\npolicy enforcement • audit • events\nSSE stream"] + SCHED["oikos scheduler\nchecks (from check_defs) → signals\n+ actuator: classify → execute/escalate\n+ learning engine\nSSH (restricted key) → fleet"] + NOTIF["oikos notifier\nMatrix alerts + approval reactions\nDB rendezvous (no RPC)"] + HERMES["Hermes agent (gateway :8092)\nMCP client → api\nNO SSH keys"] end - DEPLOY["Gitea webhook →\ndocker compose build + up -d"] + DEPLOY["deploy webhook listener\n(HMAC-verified, non-root)"] end - - HERMES -- MCP --> API + subgraph external["External"] + CADDY["Caddy (LXC 121)"] + GITEA["Gitea (LXC 104) + CI"] + MATRIX["Matrix (LXC 118)"] + FLEET["hubris / strong / LXCs / VMs"] + WS["Workstations (Hermes remote)"] + WATCHDOG["Watchdog cron on apps/105\ncurl /healthz → Matrix ping"] + end + MIGRATE --> PG API --> PG SCHED --> PG - SCHED -- "metrics + events + audit" --> PG - API -- "audit + events" --> PG - HERMES -- "agent_activity" --> PG - SCHED -- SSH --> HUBRIS - SCHED -- SSH --> STRONG - API -- "escalate" --> NOTIFIER - NOTIFIER -- "alerts + approvals" --> MATRIX - DEPLOY -- "rebuild" --> services - - subgraph external["External"] - CADDY["Caddy (LXC 121)\n→ mac-mini mesh :8090"] - APPS["apps/105 (fallback)"] - HUBRIS["hubris (PVE)"] - STRONG["strong (PVE)"] - GITEA["Gitea (LXC 104)"] - MATRIX["Matrix (LXC 118)"] - WS["Any workstation\nHermes remote → gateway"] - end - - CADDY -- reverse_proxy --> API - GITEA -- webhook --> DEPLOY - WS -- "Hermes gateway" --> HERMES + NOTIF --> PG + HERMES -- "MCP (bearer token)" --> API + SCHED -- "SSH (restricted key)" --> FLEET + NOTIF --- MATRIX + CADDY -- "reverse_proxy + forward-auth" --> API + GITEA -- "webhook (HMAC, CI-gated)" --> DEPLOY + WS -- "mesh, mTLS/token" --> HERMES + WATCHDOG -. "external heartbeat" .-> API ``` -### Observability — data capture and query flow +Container hardening: read-only root filesystems, non-root users, `restart: always`, +`stop_grace_period: 30s`, images pinned by digest, `gcr.io/distroless/static` runtime +base, `CGO_ENABLED=0`, `pgx` (pure Go). -```mermaid -flowchart TB - subgraph sources["Data sources"] - SCHED_PROBE["Scheduler probes\nhealth, disk, latency, drift"] - API_CALLS["API calls\nrequest/response metrics"] - AGENT_CALLS["Agent activity\nMCP tools, SSH, reasoning"] - EXEC["Executions\nSSH commands, verification"] - LEARN["Learning engine\npatterns, skills, confidence"] - DEPLOY_EVT["Deploy events\nwebhook, build, restart"] - end +### macOS host realities (R3-13) - subgraph capture["Capture layer (Go)"] - METRICS_W["metrics.go\n→ metric_samples"] - AUDIT_W["audit.go\n→ audit_log"] - EVENTS_W["events.go\n→ events table"] - AGENT_W["agent.go\n→ agent_activity"] - end +Docker on macOS runs in a lightweight VM (use **OrbStack**: fast, auto-starts at +login, stable networking). Consequences the deploy must respect: - subgraph store["PostgreSQL + TimescaleDB"] - MS["metric_samples\n(hypertable, 90d raw,\n1y rollups via CAGGs)"] - AL["audit_log\n(hypertable, 1y retention)"] - EV["events\n(hypertable, 90d retention)"] - AA["agent_activity\n(hypertable, 90d retention)"] - end +- **No `network_mode: host`.** All inbound reachability is via published ports. +- **Outbound (actuator SSH → hubris/strong/LXCs): direct over the LAN.** Containers + reach the LAN via Docker NAT with no special config; SSH provides its own + encryption, so there is no reason to route this hop through the mesh. +- **Inbound: mesh-primary, LAN break-glass.** NetBird runs on the macOS host. + - Primary: Caddy and workstations reach the API and Hermes gateway on ports + published on the **mesh IP** (`:8090` API, `:8092` Hermes gateway) — + WireGuard-encrypted Caddy→backend hop, mesh membership as a network-level + filter, stable addressing, works for roaming workstations. + - Break-glass: the **API port is also published on the mac-mini's LAN IP** + (`:8090`; give mac-mini a DHCP reservation). Safe because the API + authenticates every request itself (OIDC JWT / bearer token — S6); the network + you arrive from is defense-in-depth, not the auth. This keeps the control plane + reachable from inside the house if NetBird's management plane is down after a + reboot, and lets the watchdog test the API independently of the mesh. The + Hermes gateway stays **mesh-only** (no LAN binding — it has the weakest + application-layer auth story and no break-glass need). + - Never bind published ports to `0.0.0.0`; enumerate mesh IP, LAN IP (API only), + and localhost explicitly. +- Host prep: disable sleep (`sudo pmset -a sleep 0 displaysleep 10`), auto-restart + after power failure (`sudo pmset -a autorestart 1`), auto-login enabled so OrbStack + starts, macOS auto-updates deferred/scheduled (O7). +- Volumes: keep Postgres data on a named Docker volume (VM-native filesystem), not a + bind mount — bind mounts cross the VM boundary and are slow. - subgraph query["Query layer (API)"] - REST["REST endpoints\n/metrics, /audit,\n/events, /health,\n/trends, /agent-activity"] - MCP["MCP tools\nquery_metrics, get_trend,\nget_audit_trail, get_event_timeline,\nget_agent_activity, get_health_summary"] - WS["WebSocket\n/api/v1/events\n(live stream)"] - end +### Multi-node path (designed for, not built) - subgraph consumers["Consumers"] - AGENT["Agent\nqueries trends, audits\nits own history"] - FUTURE["Future: Grafana,\ndashboards, notebooks,\nSIEM, compliance tools"] - end +All roles are stateless; Postgres is the only stateful service. Scaling later means: +run `oikos scheduler` on another node pointed at the same DB (checks can be +partitioned by a `zone` attribute on `check_defs`); run `oikos api` behind Caddy on +N nodes. `SELECT … FOR UPDATE SKIP LOCKED` + advisory locks already make the work +queue multi-consumer-safe. Postgres remains the accepted SPOF (mitigated by +backup/DR, below), with an upgrade path to streaming replication if ever needed. - SCHED_PROBE --> METRICS_W - API_CALLS --> METRICS_W - API_CALLS --> AUDIT_W - AGENT_CALLS --> AGENT_W - EXEC --> EVENTS_W - EXEC --> AUDIT_W - LEARN --> METRICS_W - LEARN --> EVENTS_W - DEPLOY_EVT --> EVENTS_W +## API contract - METRICS_W --> MS - AUDIT_W --> AL - EVENTS_W --> EV - AGENT_W --> AA +### Contract-first (R3-2) - MS --> REST - AL --> REST - EV --> REST - EV --> WS - AA --> REST - MS --> MCP - AL --> MCP - EV --> MCP - AA --> MCP +- `api/openapi.yaml` (OpenAPI 3.1) is the **source of truth**. CI fails if handlers + drift from the spec. +- Server: `oapi-codegen` strict-server stubs on `chi` + stdlib `net/http`. +- Clients: the `homelab` CLI and future web UIs consume generated clients (Go / + TypeScript via `openapi-typescript`). The spec is served at `GET /api/v1/openapi.yaml` + and human docs at `GET /api/v1/docs` (Redoc/Scalar static page) — a UI developer + needs nothing but the running API. - REST --> AGENT - MCP --> AGENT - REST --> FUTURE - WS --> FUTURE -``` +### Conventions (R3-3) -### OODA loop — with learning feedback +- **Versioning:** everything under `/api/v1`. Additive-only within v1 (new fields, + new endpoints); breaking changes ship as `/api/v2` side by side with a deprecation + window. +- **Errors:** RFC 9457 `application/problem+json`: + `{"type":"https://oikos.dev/errors/invalid-transition","title":"invalid lifecycle transition","status":409,"detail":"...","instance":"/api/v1/entities/…","errors":[{field,reason}]}`. + Domain sentinel errors map centrally: `ErrNotFound→404`, `ErrInvalidTransition→409`, + `ErrApprovalRequired→403`, `ErrAutonomyBlocked→403`, `ErrConflict→409`, + `ErrCircuitOpen→503`, validation → 422. +- **Lists:** uniform envelope `{"items":[…],"next_cursor":"…"}`. Cursor pagination + everywhere (`?cursor=&limit=`, default 50, max 200): keyset on `(created_at,id)` + for entity-ish tables, on `ts` for hypertables. MCP tools take the same `limit`. +- **Idempotency:** unsafe POSTs (`/executions`, `/approvals/{id}/decision`) accept an + `Idempotency-Key` header; keys + response snapshots stored 24h; replay returns the + original response. Agents retry safely (R3-3). +- **Optimistic concurrency:** mutable resources carry a `version`; `GET` returns + `ETag`; `PATCH`/`PUT` require `If-Match`, mismatch → 412. UIs can safely edit. +- **Timestamps:** `TIMESTAMPTZ` in DB, RFC 3339 UTC on the wire. +- **CORS:** config-driven origin allowlist (empty by default; future UI origins added + via config, not code). +- **Rate limiting (M2):** token bucket per authenticated actor; tighter budget on + `/executions`; per-(entity,action) cooldown enforced in the actuator besides. -```mermaid -flowchart LR - OBSERVE["Observe\nScheduler probes:\n• HTTP health\n• disk usage\n• drift detection"] --> ORIENT["Orient\nRelations graph walk:\n• blast radius\n• lifecycle state\n• runbook match"] - ORIENT --> DECIDE["Decide\nRisk classifier +\nskill lookup:\nrisk × blast × confidence"] - DECIDE -- "auto-act" --> ACT["Act\nExecute via SSH\n→ verify → feedback"] - DECIDE -- "escalate" --> APPROVE["Approval\n→ Matrix ✅/❌"] - APPROVE -- "approved" --> ACT - ACT --> LEARN["Learn\nOutcome → feedback\n→ pattern → skill"] - LEARN -. "improves confidence" .-> DECIDE - LEARN --> OBSERVE -``` +### AuthN/AuthZ -### Deploy flow +| Caller | Mechanism | Scope | +|---|---|---| +| Operator (browser/CLI) | Authentik OIDC; Caddy forward-auth **and** JWT validated in API middleware (defense in depth, S6/SA10) | role `operator` (full) or `viewer` (read-only) | +| Hermes (MCP) | Static bearer token from Infisical, dedicated Docker network, HMAC on requests (S2) | role `agent` — read tools + `POST /executions` (which is always policy-gated) | +| Internal roles (scheduler/notifier) | Direct DB with least-privilege DB users; no API hop | n/a | +| `/healthz`, `/metrics` | No auth, no audit; not exposed via Caddy (SG18) | n/a | -```mermaid -flowchart LR - DEV["Operator\nedits repo"] --> PUSH["git push"] --> GITEA["Gitea\n(LXC 104)"] - GITEA -- "webhook" --> MACMINI["mac-mini\ndeploy script"] - MACMINI -- "git pull" --> REPO["repo clone"] - MACMINI -- "go build +\ndocker compose up -d" --> STACK["OS containers\nrebuilt + restarted"] - STACK -- "seed ingest" --> DB["PostgreSQL\nontology + inventory + policy\nsynced from YAML seeds"] -``` +Roles are claims checked per-route in generated middleware; the spec annotates each +operation with its required scope, so future UIs can render capability-aware. + +### REST surface (summary; the OpenAPI file is normative) + +Inventory + ontology: +- `GET/POST /api/v1/entities`, `GET/PATCH /api/v1/entities/{id-or-slug}` + (PATCH covers attribute edits and lifecycle transitions; transition legality + validated against `lifecycle_defs`) +- `GET /api/v1/entities/{id}/relations`, `GET /api/v1/graph?root=&depth=&rel_type=` + → `{nodes:[…],edges:[…]}` for UI visualization (R3-3) +- `GET /api/v1/ontology` (types, relationship types, lifecycles), + `POST /api/v1/ontology/entity-types`, `PATCH /api/v1/ontology/entity-types/{name}` + (policy-gated `config_mutation`; deprecate-not-delete while instances exist, D3) + +Operations: +- `GET /api/v1/signals`, `POST /api/v1/signals/{id}/ack|resolve|mute` +- `GET /api/v1/checks`, `POST /api/v1/checks`, `PATCH /api/v1/checks/{id}` (R3-7) +- `GET /api/v1/approvals`, `POST /api/v1/approvals/{id}/decision` +- `POST /api/v1/executions` (classify → approval check → enqueue; SG15), + `GET /api/v1/executions/{id}`, `POST /api/v1/executions/{id}/cancel` +- `GET /api/v1/classifications?signal_id=…` + +Learning (operator safety valves, SG7): +- `GET /api/v1/patterns`, `PATCH /api/v1/patterns/{id}` (activate/invalidate — + policy-gated `config_mutation`, audit-logged) +- `GET /api/v1/skills`, `GET /api/v1/skills/{id}/versions`, `PATCH /api/v1/skills/{id}` + +Policy (dual-control, S3): +- `GET /api/v1/policy/risk-classes|approval-rules|autonomy` +- `PATCH` on any policy resource creates a **meta-approval**; the change applies only + after operator approval; before/after hash audit-logged. + +Knowledge + observability: +- `GET /api/v1/knowledge/search?q=`, `GET /api/v1/knowledge/{entity_id}` +- `GET /api/v1/metrics?entity_id=&metric=&from=&to=&rollup=raw|1h|1d` (auto-selects + resolution by range), `GET /api/v1/trends/{entity_id}` +- `GET /api/v1/audit?actor=&entity_id=&action=&correlation_id=&from=&to=` +- `GET /api/v1/events?type=&entity_id=&severity=&from=&to=` and + `GET /api/v1/events/stream` (SSE; `Last-Event-ID` resume; heartbeat comments; + bounded per-subscriber buffers, drop-oldest, P6) +- `GET /api/v1/agent-activity`, `GET /api/v1/health` (fleet summary + trends) +- `GET /api/v1/export` — regenerates the three seed YAMLs from DB (round-trip tested, + D6) + +### MCP interface (R3-10) + +- Official `github.com/modelcontextprotocol/go-sdk`, **Streamable HTTP** transport, + mounted at `/mcp` on the same binary, bearer-token auth, dedicated network. +- Tools delegate to the identical service layer as REST (one behavior, two + protocols): `get_entity`, `list_entities`, `get_relations`, `get_blast_radius`, + `search_knowledge`, `get_signal_history`, `get_patterns`, `get_skills`, + `request_execution`, `query_metrics`, `get_trend`, `get_audit_trail`, + `get_event_timeline`, `get_agent_activity`, `get_health_summary`. +- Docs/runbooks additionally exposed as **MCP resources** (URI = entity slug) so + Hermes can attach them as context without a tool round-trip. ## DB-native configuration -The three YAML files — `inventory.yaml`, `ontology.yaml`, `policy.yaml` — become -**seed manifests**. They bootstrap the DB on first deploy. After that, the DB is the -runtime source of truth, editable via the API. A future frontend can edit all three -directly. +`seeds/ontology.yaml`, `seeds/inventory.yaml`, `seeds/policy.yaml` bootstrap the DB +and serve DR; afterwards the DB is authoritative and editable via API. Ingest runs in +the one-shot init container (A4): each seed file applies in a single transaction; a +`seed_versions` table records applied (file, content-hash) so unchanged seeds are +skipped; `GET /api/v1/export` regenerates the YAMLs for commit. Round-trip +(seed → DB → export → DB) must be byte-stable — tested in CI. -### How it works +Config hierarchy (A5): compiled defaults → config file → env vars → Infisical +(**secrets only** — never plain config). Each role's required keys documented in +`docs/operations/config.md`. -```mermaid -flowchart TB - subgraph seeds["Seed manifests (git-tracked, YAML)"] - ONTO_YAML["seeds/ontology.yaml\nentity types, relationship types,\nlifecycle definitions"] - INV_YAML["seeds/inventory.yaml\nentity instances (hosts, services,\nnetworks, storage)"] - POL_YAML["seeds/policy.yaml\nrisk classes, approval rules,\nautonomy settings"] - end - - subgraph db["PostgreSQL (runtime source of truth)"] - META["entity_types table\nrelationship_types table\nlifecycle_defs table"] - INST["entities table\nrelationships table"] - POLDB["policies table\nrisk_classes table\nautonomy_settings table"] - end - - INGEST["Seed ingest (on deploy)\nidempotent upsert"] - ONTO_YAML --> INGEST --> META - INV_YAML --> INGEST --> INST - POL_YAML --> INGEST --> POLDB - - API_EDIT["API edits\n(POST/PUT/PATCH)"] - API_EDIT --> META - API_EDIT --> INST - API_EDIT --> POLDB - - EXPORT["Export to YAML\n(for DR / version control)"] - META --> EXPORT - INST --> EXPORT - POLDB --> EXPORT -``` - -### Why DB-native - -- **Querying** — the agent can ask "what services depend on authentik?" as a graph - query, not a YAML parse. Blast-radius walks are SQL, not file reads. -- **Mutation** — adding a service, updating a lifecycle state, changing a policy rule - are DB transactions with audit trail, not file edits + git commits. -- **Consistency** — the ontology, inventory, and policy are always in sync (same DB, - same transaction). No drift between what the YAML says and what the runtime sees. -- **Future frontend** — a UI can edit entities, relationships, and policies directly - via the API. No need to generate/edit YAML files. -- **Version control** — seed YAML files are still git-tracked for bootstrap and DR. - The API can export the current DB state back to YAML for commit. - -## Repo layout (Go project) +## Repo layout ``` -/ # repo root -├── docker-compose.yml # the OS stack definition -├── Makefile # build, test, deploy targets -├── go.mod # Go module definition -├── go.sum -├── cmd/ # binary entrypoints (one per service) -│ ├── api/ # Oikos API server -│ │ └── main.go -│ ├── scheduler/ # Observe + Act loop -│ │ └── main.go -│ └── notifier/ # Notification service -│ └── main.go -├── internal/ # private packages (not importable) -│ ├── db/ # database layer -│ │ ├── queries/ # sqlc SQL queries -│ │ ├── models.go # generated Go types -│ │ └── db.go # connection pool, migrations -│ ├── ontology/ # ontology types + meta-schema -│ │ ├── types.go # EntityType, RelationshipType, LifecycleDef -│ │ ├── graph.go # graph traversal (blast radius, dependencies) -│ │ └── ingest.go # YAML seed → DB ingest -│ ├── api/ # HTTP + MCP server -│ │ ├── server.go # Gin app setup -│ │ ├── routes/ # REST handlers -│ │ │ ├── hosts.go -│ │ │ ├── services.go -│ │ │ ├── signals.go -│ │ │ ├── approvals.go -│ │ │ ├── exec.go -│ │ │ └── knowledge.go -│ │ └── mcp.go # MCP protocol adapter (JSON-RPC over SSE) -│ ├── policy/ # risk classification + approval -│ │ ├── classify.go # risk × blast × confidence -│ │ ├── approve.go # approval request + grant lifecycle -│ │ └── autonomy.go # kill-switch, never-auto-act list -│ ├── scheduler/ # Observe stage -│ │ ├── probe.go # HTTP health, disk, drift -│ │ └── signal.go # raise/resolve signals in DB -│ ├── actuator/ # Act stage -│ │ ├── act.go # read signals, classify, execute or escalate -│ │ ├── execute.go # SSH execution + verification -│ │ └── guard.go # loop-guard, retry caps -│ ├── learning/ # feedback loop (the learning model) -│ │ ├── feedback.go # record outcome + lesson from execution -│ │ ├── pattern.go # extract/validate patterns from feedback -│ │ └── skill.go # create/refine skills from patterns -│ ├── notifier/ # notification abstraction -│ │ ├── notifier.go # interface -│ │ └── matrix.go # Matrix implementation -│ ├── observability/ # logging, metrics, audit, events (NEW) -│ │ ├── logging.go # slog structured logging setup -│ │ ├── metrics.go # metric recording (writes to metric_samples) -│ │ ├── audit.go # audit middleware (writes to audit_log) -│ │ ├── events.go # event emitter (writes to events table) -│ │ ├── agent.go # agent activity recording -│ │ └── correlation.go # correlation ID propagation (context-based) -│ └── config/ # config loading (env, files) -│ └── config.go -├── migrations/ # SQL migrations (golang-migrate format) -│ ├── 001_ontology.up.sql # meta-schema (entity_types, relationship_types, ...) -│ ├── 001_ontology.down.sql -│ ├── 002_instances.up.sql # entities, relationships -│ ├── 003_operations.up.sql # signals, approvals, executions, feedback -│ ├── 004_learning.up.sql # patterns, skills -│ ├── 005_policy.up.sql # policies, risk_classes, autonomy -│ └── 006_observability.up.sql # metrics, audit_log, events, agent_activity -├── seeds/ # YAML seed manifests (bootstrap + DR) -│ ├── ontology.yaml # entity types, relationship types, lifecycles -│ ├── inventory.yaml # entity instances (hosts, services, etc.) -│ └── policy.yaml # risk classes, approval rules, autonomy -├── docs/ # narrative docs (ingested into knowledge graph) -│ ├── containers/ -│ ├── hosts/ -│ ├── infrastructure/ -│ └── investigations/ -├── hermes/ # Hermes agent config + skills -│ ├── config.yaml -│ ├── SOUL.md -│ └── skills/ -│ └── homelab-ops/ -│ └── SKILL.md -├── compose/ # Docker build contexts -│ ├── api/Dockerfile -│ ├── scheduler/Dockerfile -│ ├── hermes/Dockerfile -│ └── postgres/init.sql -└── scripts/ # utility scripts - ├── migrate-sops.sh # one-time SOPS → Infisical migration - └── import-legacy.sh # import existing signals/ledger JSONL +/ +├── docker-compose.yml +├── Makefile # build, test, lint, generate, deploy targets +├── go.mod / go.sum +├── sqlc.yaml +├── .golangci.yml +├── api/ +│ └── openapi.yaml # THE API contract (R3-2) +├── cmd/ +│ └── oikos/ # single binary: api | scheduler | notifier | all | +│ └── main.go # migrate | seed | export (R3-4) +├── internal/ +│ ├── domain/ # pure domain types + state machines + sentinel errors +│ │ ├── entity.go signal.go execution.go classification.go +│ │ ├── pattern.go skill.go approval.go check.go errors.go +│ ├── db/ # pgx pool, sqlc output, repositories (models never escape) +│ │ └── queries/ # sqlc SQL +│ ├── ontology/ # type hierarchy, validation, graph traversal, seed ingest/export +│ ├── httpapi/ # oapi-codegen server impl, middleware (auth, audit, +│ │ # idempotency, rate-limit, problem+json mapping), SSE +│ ├── mcp/ # MCP server (official SDK) over the same services +│ ├── service/ # shared service layer used by httpapi + mcp + loops +│ ├── policy/ # classify, approve (tokens), autonomy, meta-approval +│ ├── scheduler/ # check runner (check_defs → signals/metrics), dedup, flap +│ ├── actuator/ # queue consumer, SSH exec, verify, circuit breaker, locks +│ ├── learning/ # feedback, pattern extraction, skill refinement +│ ├── notifier/ # interface + matrix impl (DB rendezvous) +│ ├── observability/ # slog setup, metrics, audit, events, correlation +│ └── config/ +├── migrations/ # golang-migrate, embedded, forward-only (D5/O1) +├── seeds/ # ontology.yaml, inventory.yaml, policy.yaml +├── docs/ +│ ├── adr/ # MADR records (R3-16) +│ ├── operations/ # backup-restore.md, dr.md, config.md, runbooks +│ └── … # narrative docs (ingested into knowledge graph) +├── hermes/ # config.yaml, SOUL.md, skills/homelab-ops/SKILL.md +├── compose/ +│ ├── oikos/Dockerfile # one multi-stage Dockerfile for the binary +│ └── postgres/ # timescale/timescaledb:2-pg16 config +└── scripts/ # migrate-sops.sh, import-legacy.sh, deploy.sh, watchdog.sh ``` +Developer experience: `make generate` (oapi-codegen + sqlc), `make test` +(unit + testcontainers), `make dev` (`docker compose --profile dev up` with seeded +fake data + `oikos all`), `.env.example` committed. + ## Database schema -The schema is the ontology made concrete. Five migration groups, each adding a layer. +Consolidated migrations — all rev-2 fixes applied inline. Forward-only (no +`down.sql`; compensating migrations for rollback, plus pre-deploy dumps). Runner: +`golang-migrate` via `oikos migrate` in the init container with a DDL-only DB user; +runtime roles get DML-only users (SA9). `updated_at` maintained by a shared trigger. -### Migration 1: Ontology meta-schema +### 001 — Ontology meta-schema (with inheritance, R3-1) ```sql --- The meta-graph: defines what entity types and relationship types can exist. --- This IS the ontology, stored in the DB, editable via API. +CREATE TABLE lifecycle_defs ( + id TEXT PRIMARY KEY, -- 'infrastructure', 'signal', ... + states TEXT[] NOT NULL, + default_state TEXT NOT NULL, + terminal_states TEXT[] NOT NULL DEFAULT '{}', + transitions JSONB NOT NULL, -- {"from":{"to":{"requires":["no-inbound-edges",...]}}} + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); CREATE TABLE entity_types ( - name TEXT PRIMARY KEY, -- 'host', 'service', 'signal', 'pattern' - domain TEXT NOT NULL, -- 'physical', 'compute', 'network', ... - layer TEXT NOT NULL, -- 'infrastructure', 'governance', 'cognition' + name TEXT PRIMARY KEY, -- 'compute-entity', 'machine', 'proxmox-host' + parent_type TEXT REFERENCES entity_types(name), -- is-a hierarchy (R3-1) + is_abstract BOOLEAN NOT NULL DEFAULT false, -- abstract types can't be instantiated + domain TEXT NOT NULL, -- 'physical','compute','network','storage', + -- 'software','identity','policy','cognition' + layer TEXT NOT NULL CHECK (layer IN ('infrastructure','governance','cognition')), description TEXT, - lifecycle_id TEXT, -- FK to lifecycle_defs (nullable = no lifecycle) - attribute_schema JSONB, -- JSON Schema for validating entity.attributes - status TEXT NOT NULL DEFAULT 'active', -- 'active', 'deprecated' (no hard delete while instances exist) - created_at TIMESTAMPTZ DEFAULT now(), - updated_at TIMESTAMPTZ DEFAULT now() + lifecycle_id TEXT REFERENCES lifecycle_defs(id), + attribute_schema JSONB, -- JSON Schema for entities.attributes (D2) + schema_version INTEGER NOT NULL DEFAULT 1, + status TEXT NOT NULL DEFAULT 'active', -- 'active'|'deprecated'; no hard delete + -- while instances exist (D3) + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE TABLE relationship_types ( - name TEXT PRIMARY KEY, -- 'hosts', 'provides', 'depends-on' - inverse TEXT, -- 'runs-on', 'provided-by' - source_type TEXT REFERENCES entity_types(name), - target_type TEXT REFERENCES entity_types(name), - cardinality TEXT NOT NULL, -- 'one-to-one', 'one-to-many', 'many-to-many' + name TEXT PRIMARY KEY, -- 'hosts', 'provides', 'depends-on' + inverse TEXT, + source_type TEXT NOT NULL REFERENCES entity_types(name), -- MAY be abstract; + target_type TEXT NOT NULL REFERENCES entity_types(name), -- validation walks hierarchy + cardinality TEXT NOT NULL CHECK (cardinality IN + ('one-to-one','one-to-many','many-to-many')), description TEXT, - created_at TIMESTAMPTZ DEFAULT now() + created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); -CREATE TABLE lifecycle_defs ( - id TEXT PRIMARY KEY, -- 'infrastructure', 'signal', 'execution', ... - states TEXT[] NOT NULL, -- ordered states - default_state TEXT NOT NULL, - transitions JSONB NOT NULL, -- {"from": {"to": {"requires": [...]}}} - created_at TIMESTAMPTZ DEFAULT now() +CREATE TABLE seed_versions ( -- A4: skip unchanged seed files + file TEXT PRIMARY KEY, + content_hash TEXT NOT NULL, + applied_at TIMESTAMPTZ NOT NULL DEFAULT now() ); ``` -### Migration 2: Entity instances (the inventory graph) +Validation semantics (app layer, `internal/ontology`): +- Instantiating an `is_abstract` type is rejected. +- A relationship `(s, t, type)` is valid iff `type_of(s)` is `source_type` **or a + descendant of it** (same for target). The hierarchy is small; resolved in-memory + with a cached type tree. +- Cardinality enforced by partial unique indexes where expressible + (`one-to-many` → `UNIQUE (target_id, type)`; `one-to-one` → unique on both ends) + plus app-layer checks for the rest. + +### 002 — Entity instances (UUIDv7 + slug, R3-5/D1) ```sql --- Entity instances — the actual hosts, services, signals, patterns, etc. --- This replaces inventory.yaml as the runtime source of truth. - CREATE TABLE entities ( - id TEXT PRIMARY KEY, -- 'host:hubris', 'service:caddy', 'sig:2026-07-06-0001' + id UUID PRIMARY KEY, -- UUIDv7 generated in Go (time-ordered) + slug TEXT NOT NULL UNIQUE, -- 'host:hubris', 'service:caddy' — human/API handle type TEXT NOT NULL REFERENCES entity_types(name), - name TEXT NOT NULL, -- 'hubris', 'caddy', 'disk-threshold' - state TEXT, -- lifecycle state (e.g. 'active', 'raised') - attributes JSONB NOT NULL DEFAULT '{}', -- type-specific data (IP, mesh addr, port, ...) - parent_id TEXT REFERENCES entities(id), -- for hierarchical entities (LXC on host) - created_at TIMESTAMPTZ DEFAULT now(), - updated_at TIMESTAMPTZ DEFAULT now() + name TEXT NOT NULL, + state TEXT, -- lifecycle state + attributes JSONB NOT NULL DEFAULT '{}', -- validated against attribute_schema + maintenance_until TIMESTAMPTZ, -- R3-8: suppress signals + auto-act while set + version INTEGER NOT NULL DEFAULT 1, -- optimistic lock / ETag source + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE (type, name) ); - -CREATE INDEX idx_entities_type ON entities(type); +CREATE INDEX idx_entities_type ON entities(type); CREATE INDEX idx_entities_state ON entities(state); -CREATE INDEX idx_entities_attributes ON entities USING GIN(attributes); +CREATE INDEX idx_entities_attrs ON entities USING GIN(attributes); --- Relationship instances — the typed edges of the graph CREATE TABLE relationships ( - source_id TEXT NOT NULL REFERENCES entities(id), - target_id TEXT NOT NULL REFERENCES entities(id), + source_id UUID NOT NULL REFERENCES entities(id) ON DELETE RESTRICT, + target_id UUID NOT NULL REFERENCES entities(id) ON DELETE RESTRICT, type TEXT NOT NULL REFERENCES relationship_types(name), attributes JSONB, - created_at TIMESTAMPTZ DEFAULT now(), - PRIMARY KEY (source_id, target_id, type) + valid_from TIMESTAMPTZ NOT NULL DEFAULT now(), -- D7: temporal edges + valid_to TIMESTAMPTZ, -- NULL = current + PRIMARY KEY (source_id, target_id, type, valid_from) ); +CREATE INDEX idx_rel_source ON relationships(source_id) WHERE valid_to IS NULL; +CREATE INDEX idx_rel_target ON relationships(target_id) WHERE valid_to IS NULL; +CREATE INDEX idx_rel_type ON relationships(type) WHERE valid_to IS NULL; -CREATE INDEX idx_rel_source ON relationships(source_id); -CREATE INDEX idx_rel_target ON relationships(target_id); -CREATE INDEX idx_rel_type ON relationships(type); - --- Recursive graph traversal function (blast radius, dependency chains) -CREATE OR REPLACE FUNCTION blast_radius(start_id TEXT, max_depth INT DEFAULT 3) -RETURNS TABLE(entity_id TEXT, depth INT) AS $$ +-- Cycle-safe traversal (P1): path accumulator prevents revisits; depth capped. +CREATE OR REPLACE FUNCTION blast_radius(start_id UUID, max_depth INT DEFAULT 3, + rel_types TEXT[] DEFAULT NULL) +RETURNS TABLE(entity_id UUID, depth INT) AS $$ WITH RECURSIVE walk AS ( - SELECT start_id::TEXT AS entity_id, 0::INT AS depth - UNION - SELECT r.target_id::TEXT, w.depth + 1 + SELECT start_id AS entity_id, 0 AS depth, ARRAY[start_id] AS path + UNION ALL + SELECT r.target_id, w.depth + 1, w.path || r.target_id FROM relationships r JOIN walk w ON r.source_id = w.entity_id - WHERE w.depth < max_depth + WHERE w.depth < LEAST(max_depth, 5) + AND r.valid_to IS NULL + AND NOT r.target_id = ANY(w.path) + AND (rel_types IS NULL OR r.type = ANY(rel_types)) ) - SELECT DISTINCT entity_id, MIN(depth) FROM walk GROUP BY entity_id; + SELECT entity_id, MIN(depth) FROM walk GROUP BY entity_id; $$ LANGUAGE sql STABLE; ``` -### Migration 3: Operations (signals, approvals, ledger, state) +Entities are never hard-deleted while edges exist (`ON DELETE RESTRICT`); +decommission is the lifecycle path (`… → destroyed`), and destroyed entities remain +as archaeology. + +**Dual-entity pattern (SA1):** every cognition object (signal, classification, +execution, feedback, pattern, skill, approval, check) gets an `entities` row (so the +graph is traversable: `triggers`, `produces`, `contributes-to` edges live in +`relationships`) **and** a typed table below whose PK references `entities(id)` for +indexed querying. + +### 003 — Operations (signals, checks, approvals, status) ```sql --- Signals — now entities in the graph, with a dedicated table for indexed querying --- (the entity row is the canonical record; this table is a fast lookup) +CREATE TABLE check_defs ( -- R3-7: probes as data + entity_id UUID PRIMARY KEY REFERENCES entities(id), + target_id UUID REFERENCES entities(id), -- what it checks (NULL + target_type = type-scoped) + target_type TEXT REFERENCES entity_types(name), + kind TEXT NOT NULL, -- 'http','tcp','disk','cert-expiry','drift','ssh-script' + config JSONB NOT NULL DEFAULT '{}', -- validated per-kind JSON Schema + interval_s INTEGER NOT NULL DEFAULT 600, + timeout_s INTEGER NOT NULL DEFAULT 10, + zone TEXT, -- multi-node partitioning later + enabled BOOLEAN NOT NULL DEFAULT true, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + CREATE TABLE signals ( - entity_id TEXT PRIMARY KEY REFERENCES entities(id), + entity_id UUID PRIMARY KEY REFERENCES entities(id), kind TEXT NOT NULL, - severity TEXT NOT NULL, -- info, warning, critical - target_entity_id TEXT REFERENCES entities(id), -- the infrastructure entity this is about + severity TEXT NOT NULL CHECK (severity IN ('info','warning','critical')), + target_entity_id UUID REFERENCES entities(id), + check_id UUID REFERENCES check_defs(entity_id), evidence TEXT, likely_cause TEXT, - recommended_action JSONB, - verification TEXT, state TEXT NOT NULL DEFAULT 'raised', + occurrence_count INTEGER NOT NULL DEFAULT 1, -- R3-8: dedup counting + first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), + last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), + flap_count INTEGER NOT NULL DEFAULT 0, -- resolve→re-raise cycles + hold_down_until TIMESTAMPTZ, -- flap suppression window mute_until TIMESTAMPTZ, - created_at TIMESTAMPTZ DEFAULT now(), - updated_at TIMESTAMPTZ DEFAULT now() + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() ); - +-- R3-8: at most ONE open signal per (target, kind) — repeats update the open row +CREATE UNIQUE INDEX uq_signals_open ON signals(target_entity_id, kind) + WHERE state NOT IN ('resolved','failed'); CREATE INDEX idx_signals_state ON signals(state); -CREATE INDEX idx_signals_target ON signals(target_entity_id); -CREATE INDEX idx_signals_severity ON signals(severity); --- Approvals CREATE TABLE approvals ( - id TEXT PRIMARY KEY, - ts TIMESTAMPTZ DEFAULT now(), - entity_id TEXT REFERENCES entities(id), -- entity to act on + entity_id UUID PRIMARY KEY REFERENCES entities(id), + subject_entity_id UUID REFERENCES entities(id), -- entity to act on action TEXT NOT NULL, risk_class TEXT NOT NULL, - status TEXT NOT NULL DEFAULT 'pending', -- pending, approved, denied, expired - ttl INTERVAL NOT NULL DEFAULT '1 hour', + kind TEXT NOT NULL DEFAULT 'execution', -- 'execution'|'policy-change'|'pattern-activation' + payload JSONB, -- e.g. the proposed policy diff + status TEXT NOT NULL DEFAULT 'pending', + token_hash TEXT, -- S5: single-use HMAC token, stored hashed + expires_at TIMESTAMPTZ NOT NULL, decided_at TIMESTAMPTZ, - decided_by TEXT REFERENCES entities(id), -- person entity - confirmation_phrase TEXT + decided_by UUID REFERENCES entities(id), -- person entity + created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); --- Change ledger (high-level record, links to execution for detail) -CREATE TABLE ledger_entries ( - id SERIAL PRIMARY KEY, - ts TIMESTAMPTZ DEFAULT now(), - entity_id TEXT REFERENCES entities(id), - action TEXT NOT NULL, - risk_class TEXT NOT NULL, - result TEXT, -- ok, failed, escalated - approval_id TEXT REFERENCES approvals(id), - execution_id INTEGER, -- FK to executions (migration 4) - agent_id TEXT REFERENCES entities(id), - notes TEXT +CREATE TABLE entity_status ( -- R3-6: replaces state_snapshots (P5) + entity_id UUID PRIMARY KEY REFERENCES entities(id), + health TEXT NOT NULL DEFAULT 'unknown', -- healthy|degraded|down|unknown + last_check_at TIMESTAMPTZ, + details JSONB NOT NULL DEFAULT '{}', + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() ); +-- health HISTORY is the 'health' metric in metric_samples (retained + rolled up) --- State snapshots (replaces oikos/state.json) -CREATE TABLE state_snapshots ( - id SERIAL PRIMARY KEY, - ts TIMESTAMPTZ DEFAULT now(), - entity_id TEXT REFERENCES entities(id), - health TEXT, -- healthy, degraded, down, unknown - data JSONB +CREATE TABLE idempotency_keys ( -- R3-3 + key TEXT NOT NULL, + actor TEXT NOT NULL, + request_hash TEXT NOT NULL, + response_code INTEGER, + response_body JSONB, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (actor, key) ); +-- pruned by the scheduler after 24h ``` -### Migration 4: Learning model (executions, feedback, patterns, skills) +Approval tokens (S5): token = HMAC(approval_id ‖ subject ‖ action ‖ risk_class ‖ +nonce, secret), single-use, stored hashed, TTL-bound; Matrix carries only the +approval ID + decision; verification is server-side. + +### 004 — Cognition (classifications, executions, learning) ```sql --- Executions — detailed record of each action the OS performs -CREATE TABLE executions ( - id SERIAL PRIMARY KEY, - ts TIMESTAMPTZ DEFAULT now(), - signal_entity_id TEXT REFERENCES entities(id), -- signal that triggered this - target_entity_id TEXT REFERENCES entities(id), -- entity acted upon +CREATE TABLE classifications ( -- SA5: every autonomous decision persisted + entity_id UUID PRIMARY KEY REFERENCES entities(id), + signal_entity_id UUID REFERENCES signals(entity_id), + target_entity_id UUID REFERENCES entities(id), action TEXT NOT NULL, + recommended_action JSONB, -- SA6: lives here, not on the signal risk_class TEXT NOT NULL, - approval_id TEXT REFERENCES approvals(id), - agent_id TEXT REFERENCES entities(id), -- who/what executed - skill_id TEXT REFERENCES entities(id), -- skill used (if any) - status TEXT NOT NULL DEFAULT 'queued', -- queued, running, completed, failed, timed-out - result JSONB, -- detailed result data - duration_ms INTEGER, - verified BOOLEAN DEFAULT false, - started_at TIMESTAMPTZ, - completed_at TIMESTAMPTZ + route TEXT NOT NULL CHECK (route IN ('auto-act','escalate','hold')), + blast_radius UUID[], + pattern_confidence REAL, + skill_id UUID, -- skill entity matched (if any) + autonomy_check TEXT, -- 'allowed' | 'blocked: ' + reasoning JSONB NOT NULL, + correlation_id TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); +CREATE TABLE executions ( + entity_id UUID PRIMARY KEY REFERENCES entities(id), + classification_id UUID REFERENCES classifications(entity_id), + signal_entity_id UUID REFERENCES signals(entity_id), + target_entity_id UUID REFERENCES entities(id), + action TEXT NOT NULL, + risk_class TEXT NOT NULL, + approval_id UUID REFERENCES approvals(entity_id), + agent_id UUID REFERENCES entities(id), + skill_id UUID, -- + version pinned at execution time (SG9) + skill_version INTEGER, + status TEXT NOT NULL DEFAULT 'proposed', + result JSONB, + duration_ms INTEGER, + verified BOOLEAN NOT NULL DEFAULT false, + correlation_id TEXT NOT NULL, + started_at TIMESTAMPTZ, + completed_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); CREATE INDEX idx_exec_target ON executions(target_entity_id); CREATE INDEX idx_exec_status ON executions(status); -CREATE INDEX idx_exec_action ON executions(action); --- Feedback — what was learned from an execution CREATE TABLE feedback ( - id SERIAL PRIMARY KEY, - execution_id INTEGER REFERENCES executions(id), - ts TIMESTAMPTZ DEFAULT now(), - outcome TEXT NOT NULL, -- success, failure, partial, unexpected - observation TEXT, -- what happened vs what was expected - lesson TEXT, -- extractable lesson + entity_id UUID PRIMARY KEY REFERENCES entities(id), + execution_id UUID NOT NULL REFERENCES executions(entity_id), + outcome TEXT NOT NULL CHECK (outcome IN ('success','failure','partial','unexpected')), + observation TEXT, + lesson TEXT, unexpected_side_effects TEXT[], - tags TEXT[] + tags TEXT[], + created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); +CREATE INDEX idx_feedback_ts ON feedback(created_at); -- P4: watermark scans -CREATE INDEX idx_feedback_execution ON feedback(execution_id); -CREATE INDEX idx_feedback_outcome ON feedback(outcome); - --- Patterns — generalized rules extracted from accumulated feedback CREATE TABLE patterns ( - id TEXT PRIMARY KEY, -- 'pat-2026-07-06-001' - ts TIMESTAMPTZ DEFAULT now(), - entity_type TEXT REFERENCES entity_types(name), -- applies to this type - action TEXT NOT NULL, -- 'restart', 'deploy', etc. - pattern TEXT NOT NULL, -- 'service X recovers within 30s after restart' - confidence REAL DEFAULT 0.5, -- 0.0 to 1.0 - evidence_count INTEGER DEFAULT 1, -- how many executions support this - success_count INTEGER DEFAULT 0, - failure_count INTEGER DEFAULT 0, - status TEXT DEFAULT 'hypothesized', -- hypothesized, validated, active, deprecated - last_validated_at TIMESTAMPTZ + entity_id UUID PRIMARY KEY REFERENCES entities(id), + applies_type TEXT NOT NULL REFERENCES entity_types(name), + action TEXT NOT NULL, + pattern TEXT NOT NULL, + confidence REAL NOT NULL DEFAULT 0, -- Wilson lower bound, capped by sample size (S4) + evidence_count INTEGER NOT NULL DEFAULT 0, + success_count INTEGER NOT NULL DEFAULT 0, + failure_count INTEGER NOT NULL DEFAULT 0, + status TEXT NOT NULL DEFAULT 'hypothesized', + quarantined BOOLEAN NOT NULL DEFAULT false, -- S4: anomalous feedback bursts + version INTEGER NOT NULL DEFAULT 1, -- D4: optimistic lock + last_validated_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE (applies_type, action) ); +-- Counters updated atomically: UPDATE … SET evidence_count = evidence_count + 1 (D4) -CREATE INDEX idx_patterns_type_action ON patterns(entity_type, action); -CREATE INDEX idx_patterns_status ON patterns(status); - --- Skills — codified procedures refined through feedback CREATE TABLE skills ( - id TEXT PRIMARY KEY, -- 'skill-restart-service', 'skill-deploy-lxc' - name TEXT NOT NULL, - ts TIMESTAMPTZ DEFAULT now(), - procedure TEXT NOT NULL, -- the codified steps (markdown or structured) - applies_to TEXT REFERENCES entity_types(name), - pattern_ids TEXT[], -- patterns that inform this skill - status TEXT DEFAULT 'drafted', -- drafted, tested, active, refined, deprecated - version INTEGER DEFAULT 1, - success_rate REAL, -- rolling success rate - last_used_at TIMESTAMPTZ + entity_id UUID NOT NULL REFERENCES entities(id), + version INTEGER NOT NULL DEFAULT 1, -- SG9: history preserved + name TEXT NOT NULL, + procedure JSONB NOT NULL, -- R3-9: structured, schema-validated (below) + applies_type TEXT REFERENCES entity_types(name), + action TEXT NOT NULL, + pattern_ids UUID[], + status TEXT NOT NULL DEFAULT 'drafted', + success_rate REAL, + changed_by UUID, -- agent or person entity + change_reason TEXT, + last_used_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (entity_id, version) ); - -CREATE INDEX idx_skills_type ON skills(applies_to); -CREATE INDEX idx_skills_status ON skills(status); ``` -### Migration 5: Policy (DB-native risk + approval rules) +**Skill procedure format (R3-9)** — validated by JSON Schema at write time; the +actuator executes it deterministically; markdown for humans is rendered from it: + +```json +{ + "params_schema": { "type": "object", "properties": { "unit": {"type":"string"} }, "required": ["unit"] }, + "steps": [ + { "name": "restart unit", + "runner": "ssh", "target": "{{ .host }}", + "command": "systemctl restart {{ .unit }}", + "timeout_s": 60 } + ], + "verify": [ + { "runner": "ssh", "target": "{{ .host }}", + "command": "systemctl is-active {{ .unit }}", + "expect": { "exit_code": 0, "stdout_contains": "active" }, + "retry": { "attempts": 3, "delay_s": 10 } } + ], + "rollback": [], + "expected_duration_s": 30, + "known_failure_modes": ["unit masked", "dependency service down"] +} +``` + +Templates are Go `text/template` over validated params; the actuator refuses any +command not generated from a stored skill/runbook step (no free-form agent shell). + +### 005 — Policy ```sql --- Risk classes — the four-level safety model CREATE TABLE risk_classes ( - name TEXT PRIMARY KEY, -- 'read_only', 'reversible_low', etc. + name TEXT PRIMARY KEY, -- read_only, reversible_low, config_mutation, destructive description TEXT, - approval_required TEXT NOT NULL DEFAULT 'none', -- none, operator, operator_confirmed - ledger BOOLEAN DEFAULT false, - autonomy_allowed BOOLEAN DEFAULT false -- can agent auto-act at this risk level? + approval_required TEXT NOT NULL DEFAULT 'none', -- none|operator|operator_confirmed + autonomy_allowed BOOLEAN NOT NULL DEFAULT false ); --- Approval rules — entity_type + action → risk_class + requirements CREATE TABLE approval_rules ( - id SERIAL PRIMARY KEY, - entity_type TEXT REFERENCES entity_types(name), -- applies to this entity type - action TEXT NOT NULL, -- 'restart', 'deploy', 'destroy' + id UUID PRIMARY KEY, + entity_type TEXT REFERENCES entity_types(name), -- may be abstract (R3-1) + action TEXT NOT NULL, risk_class TEXT NOT NULL REFERENCES risk_classes(name), - autonomy_level TEXT NOT NULL DEFAULT 'auto', -- 'auto', 'escalate', 'never' - scope_entity TEXT REFERENCES entities(id), -- optional: specific entity only - created_at TIMESTAMPTZ DEFAULT now(), - updated_at TIMESTAMPTZ DEFAULT now(), - UNIQUE(entity_type, action) + autonomy_level TEXT NOT NULL DEFAULT 'escalate' CHECK + (autonomy_level IN ('auto','escalate','never')), + scope_entity UUID REFERENCES entities(id), -- optional per-entity override + version INTEGER NOT NULL DEFAULT 1, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE (entity_type, action, scope_entity) ); --- Autonomy settings — global kill-switch + per-entity overrides CREATE TABLE autonomy_settings ( - key TEXT PRIMARY KEY, -- 'global.auto_act', 'never_auto_act.caddy' - value TEXT NOT NULL, -- 'off', 'reversible_low', 'true', 'false' - updated_at TIMESTAMPTZ DEFAULT now() + key TEXT PRIMARY KEY, -- 'global.auto_act', 'never_auto_act.' + value TEXT NOT NULL, + version INTEGER NOT NULL DEFAULT 1, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() ); ``` -### Migration 6: Observability (metrics, audit log, event log) +Rule resolution: most-specific wins (scope_entity > concrete type > ancestor type via +the hierarchy). Policy mutations are dual-controlled (S3): the API writes a +`policy-change` approval; on operator approval the change applies in one transaction +with a before/after hash written to `audit_log`; at startup each role verifies the +policy hash against last-known-good and raises a critical signal on mismatch. -Uses **TimescaleDB** (PostgreSQL extension) for time-series data. Hypertables -auto-partition by time, continuous aggregates provide rollups, and retention policies -auto-drop old data. No separate database needed — everything stays in Postgres. +### 006 — Observability (TimescaleDB) + +Hypertable PKs include the time column (SG1); TimescaleDB DDL is idempotent +(`if_not_exists => TRUE`, exception-guarded policies — SG3); CAGGs avoid +`array_agg` (SG2). ```sql --- Enable TimescaleDB CREATE EXTENSION IF NOT EXISTS timescaledb; --- ─── Time-series metrics ────────────────────────────────────────────── --- Generic metric store. Every probe, health check, and system measurement --- writes here. Designed for high insert volume, time-range queries, and --- continuous-aggregate rollups. - CREATE TABLE metric_samples ( ts TIMESTAMPTZ NOT NULL, - entity_id TEXT NOT NULL, -- which entity this metric is about - metric TEXT NOT NULL, -- 'health', 'disk_usage_pct', 'probe_latency_ms', - -- 'api_p99_ms', 'goroutines', 'db_connections', - -- 'pattern_confidence', 'skill_success_rate', ... + entity_id UUID NOT NULL, + metric TEXT NOT NULL, -- 'health','disk_usage_pct','probe_latency_ms', + -- 'api_latency_ms','pattern_confidence',… value DOUBLE PRECISION NOT NULL, - tags JSONB DEFAULT '{}'::JSONB -- arbitrary key-value labels: - -- {probe: "http", target: "192.168.8.121"}, - -- {host: "hubris", mount: "/mnt/library"}, ... + tags JSONB NOT NULL DEFAULT '{}' ); - --- Hypertable: partition by time, 1-week chunks -SELECT create_hypertable('metric_samples', 'ts', chunk_time_interval => INTERVAL '7 days'); - --- Indexes for common query patterns +SELECT create_hypertable('metric_samples','ts', + chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE); CREATE INDEX idx_metrics_entity_ts ON metric_samples(entity_id, ts DESC); CREATE INDEX idx_metrics_metric_ts ON metric_samples(metric, ts DESC); -CREATE INDEX idx_metrics_tags ON metric_samples USING GIN(tags); +DO $$ BEGIN + PERFORM add_retention_policy('metric_samples', INTERVAL '90 days'); +EXCEPTION WHEN OTHERS THEN NULL; END $$; --- Retention: drop raw metrics older than 90 days (continuous aggregates keep rollups) -SELECT add_retention_policy('metric_samples', INTERVAL '90 days'); - --- Continuous aggregate: 1-hour rollups (mean, min, max, count) -CREATE MATERIALIZED VIEW metric_rollups_1h -WITH (timescaledb.continuous) AS - SELECT - time_bucket('1 hour', ts) AS bucket, - entity_id, - metric, - avg(value) AS avg_value, - min(value) AS min_value, - max(value) AS max_value, - count(*) AS sample_count, - (array_agg(tags))[1] AS representative_tags - FROM metric_samples - GROUP BY bucket, entity_id, metric; - --- Refresh policy: refresh every 1 hour, keep 1 year of rollups -SELECT add_continuous_aggregate_policy('metric_rollups_1h', - start_offset => INTERVAL '2 hours', - end_offset => INTERVAL '5 minutes', - schedule_interval => INTERVAL '1 hour'); - -CREATE MATERIALIZED VIEW metric_rollups_1d -WITH (timescaledb.continuous) AS - SELECT - time_bucket('1 day', ts) AS bucket, - entity_id, - metric, - avg(value) AS avg_value, - min(value) AS min_value, - max(value) AS max_value, - count(*) AS sample_count - FROM metric_samples - GROUP BY bucket, entity_id, metric; - -SELECT add_continuous_aggregate_policy('metric_rollups_1d', - start_offset => INTERVAL '2 days', - end_offset => INTERVAL '1 hour', - schedule_interval => INTERVAL '1 day'); - --- ─── Audit log ──────────────────────────────────────────────────────── --- Every mutating action — by the OS, by an agent, or by an operator — --- gets an immutable audit entry. This is the "who did what when" trail --- that the ledger doesn't fully capture (ledger records OS decisions and --- executions; audit captures ALL API calls including reads-that-matter --- and operator interactions). +CREATE MATERIALIZED VIEW metric_rollups_1h WITH (timescaledb.continuous) AS + SELECT time_bucket('1 hour', ts) AS bucket, entity_id, metric, + avg(value) AS avg_value, min(value) AS min_value, + max(value) AS max_value, count(*) AS sample_count + FROM metric_samples GROUP BY bucket, entity_id, metric; +-- + metric_rollups_1d identically; refresh policies 1h/1d; rollups kept 1 year CREATE TABLE audit_log ( - id BIGSERIAL PRIMARY KEY, + id BIGINT GENERATED ALWAYS AS IDENTITY, ts TIMESTAMPTZ NOT NULL DEFAULT now(), - actor_type TEXT NOT NULL, -- 'agent', 'operator', 'system', 'scheduler' - actor_id TEXT, -- entity ID of the actor (agent entity, person entity) - action TEXT NOT NULL, -- 'api.call', 'entity.create', 'policy.update', - -- 'approval.decide', 'exec.request', 'deploy.trigger' - entity_id TEXT, -- entity affected (if any) - method TEXT, -- 'GET', 'POST', 'PATCH', 'DELETE', 'MCP', 'SSH' - path TEXT, -- API path or MCP tool name or SSH command - status_code INTEGER, -- HTTP status or 0 for non-HTTP - detail JSONB DEFAULT '{}'::JSONB, -- request body, response summary, extra context - source_ip TEXT, -- where the call came from - correlation_id TEXT -- links to execution_id / signal_id for tracing + actor_type TEXT NOT NULL, -- agent|operator|system|scheduler + actor_id UUID, -- resolves to a real entity (SA3) + action TEXT NOT NULL, + entity_id UUID, + method TEXT, path TEXT, status_code INTEGER, + detail JSONB NOT NULL DEFAULT '{}', + source_ip TEXT, + correlation_id TEXT, + PRIMARY KEY (id, ts) -- SG1 ); - -SELECT create_hypertable('audit_log', 'ts', chunk_time_interval => INTERVAL '7 days'); -CREATE INDEX idx_audit_actor ON audit_log(actor_type, actor_id, ts DESC); -CREATE INDEX idx_audit_entity ON audit_log(entity_id, ts DESC); -CREATE INDEX idx_audit_action ON audit_log(action, ts DESC); -CREATE INDEX idx_audit_correlation ON audit_log(correlation_id); - --- Retention: keep 1 year of audit logs -SELECT add_retention_policy('audit_log', INTERVAL '365 days'); - --- ─── Event log ──────────────────────────────────────────────────────── --- Structured event stream — the "news feed" of the OS. Every significant --- state change is an event: signal raised/resolved, execution started/completed, --- approval requested/granted, deploy triggered, pattern validated, skill refined, --- config changed, entity lifecycle transition. The WebSocket /api/v1/events --- streams from this table; agents can also query it historically. +SELECT create_hypertable('audit_log','ts', + chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE); +-- indexes on (actor_type,actor_id,ts), (entity_id,ts), (correlation_id); 365d retention CREATE TABLE events ( - id BIGSERIAL PRIMARY KEY, + id BIGINT GENERATED ALWAYS AS IDENTITY, ts TIMESTAMPTZ NOT NULL DEFAULT now(), - type TEXT NOT NULL, -- 'signal.raised', 'signal.resolved', - -- 'execution.started', 'execution.completed', - -- 'approval.requested', 'approval.decided', - -- 'deploy.triggered', 'deploy.completed', - -- 'pattern.validated', 'skill.refined', - -- 'entity.created', 'entity.state_changed', - -- 'policy.changed', 'config.changed' - entity_id TEXT, -- primary entity involved - severity TEXT DEFAULT 'info', -- info, warning, critical - source TEXT NOT NULL, -- 'scheduler', 'actuator', 'api', 'hermes', - -- 'deploy', 'notifier', 'learning' - data JSONB DEFAULT '{}'::JSONB, -- event-specific payload - correlation_id TEXT -- links to signal/exec/approval for tracing + type TEXT NOT NULL, -- 'signal.raised','execution.completed',… + entity_id UUID, + severity TEXT NOT NULL DEFAULT 'info', + source TEXT NOT NULL, + data JSONB NOT NULL DEFAULT '{}', + correlation_id TEXT, + PRIMARY KEY (id, ts) ); - -SELECT create_hypertable('events', 'ts', chunk_time_interval => INTERVAL '7 days'); -CREATE INDEX idx_events_type_ts ON events(type, ts DESC); -CREATE INDEX idx_events_entity_ts ON events(entity_id, ts DESC); -CREATE INDEX idx_events_severity_ts ON events(severity, ts DESC); -CREATE INDEX idx_events_correlation ON events(correlation_id); - --- Retention: keep 90 days of events (signals/ledger have their own tables --- for permanent records; events are the transient feed) -SELECT add_retention_policy('events', INTERVAL '90 days'); - --- ─── Agent activity log ────────────────────────────────────────────── --- Records what the agent (Hermes) does: tool calls, reasoning, decisions, --- token usage, latency. This is for agent behavior auditing and trend --- analysis ("is the agent getting more efficient?"). +SELECT create_hypertable('events','ts', + chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE); +-- 90d retention; NOTIFY trigger fires after commit for the SSE stream (SG10) CREATE TABLE agent_activity ( - id BIGSERIAL PRIMARY KEY, - ts TIMESTAMPTZ NOT NULL DEFAULT now(), - agent_id TEXT NOT NULL, -- entity ID of the agent - session_id TEXT, -- Hermes session ID - activity_type TEXT NOT NULL, -- 'tool_call', 'reasoning', 'decision', - -- 'mcp_query', 'ssh_command', 'escalation' - tool_name TEXT, -- MCP tool or CLI command called - entity_id TEXT, -- entity acted upon (if any) - input_summary TEXT, -- truncated input (first 500 chars) - output_summary TEXT, -- truncated output (first 500 chars) - duration_ms INTEGER, - token_count INTEGER, -- LLM tokens consumed (if applicable) - success BOOLEAN, - correlation_id TEXT + id BIGINT GENERATED ALWAYS AS IDENTITY, + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + agent_id UUID NOT NULL, + session_id TEXT, + activity_type TEXT NOT NULL, -- tool_call|reasoning|decision|mcp_query|escalation + tool_name TEXT, entity_id UUID, + input_summary TEXT, output_summary TEXT, -- truncated 500 chars + duration_ms INTEGER, token_count INTEGER, success BOOLEAN, + correlation_id TEXT, + PRIMARY KEY (id, ts) ); +SELECT create_hypertable('agent_activity','ts', + chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE); +-- 90d retention -SELECT create_hypertable('agent_activity', 'ts', chunk_time_interval => INTERVAL '7 days'); -CREATE INDEX idx_agent_activity_agent_ts ON agent_activity(agent_id, ts DESC); -CREATE INDEX idx_agent_activity_type_ts ON agent_activity(activity_type, ts DESC); -CREATE INDEX idx_agent_activity_entity ON agent_activity(entity_id, ts DESC); -CREATE INDEX idx_agent_activity_correlation ON agent_activity(correlation_id); - --- Retention: keep 90 days of agent activity -SELECT add_retention_policy('agent_activity', INTERVAL '90 days'); +-- R3-11: ledger is a VIEW, not a fourth write path +CREATE VIEW ledger AS +SELECT e.created_at AS ts, e.entity_id AS execution_id, e.target_entity_id, + e.action, e.risk_class, e.status, e.verified, + c.route, c.reasoning, a.status AS approval_status, a.decided_by, + e.agent_id, e.correlation_id +FROM executions e +LEFT JOIN classifications c ON c.entity_id = e.classification_id +LEFT JOIN approvals a ON a.entity_id = e.approval_id; ``` -**What gets captured where:** +**Retention summary:** metrics 90d raw / 1y rollups; audit 1y; events 90d; agent +activity 90d; signals/executions/classifications/feedback permanent (they're the +learning corpus and small); `idempotency_keys` 24h (scheduler prune job). -| Data source | Table | Retention | Purpose | -|---|---|---|---| -| Scheduler probes (HTTP health, disk, drift) | `metric_samples` | 90 days raw, 1 year rollups | Trend analysis, anomaly detection | -| API response latencies | `metric_samples` | same | Performance monitoring | -| Go runtime metrics (goroutines, mem, GC) | `metric_samples` | same | OS self-monitoring | -| Learning model metrics (pattern confidence, skill success rate) | `metric_samples` | same | Learning trend tracking | -| All mutating API calls | `audit_log` | 1 year | Auditing, compliance, forensics | -| Operator actions (approval decisions, entity edits, policy changes) | `audit_log` | 1 year | Operator accountability | -| Agent (Hermes) tool calls + reasoning | `agent_activity` | 90 days | Agent behavior auditing, efficiency tracking | -| Signal/exec/approval/deploy state changes | `events` | 90 days | Event stream (WebSocket), timeline reconstruction | -| Signals (permanent record) | `signals` | no expiry | Signal lifecycle tracking | -| Ledger entries (permanent record) | `ledger_entries` | no expiry | Change history | -| Executions (permanent record) | `executions` | no expiry | Execution audit trail | +## Control loop -## Workstreams +### Scheduler (Observe) -### 1. Ontology definition + seed manifests (`seeds/`, `internal/ontology/`) +- Loads enabled `check_defs`; each check runs on its own interval with jitter; a + bounded worker pool (`errgroup.SetLimit`) caps concurrency; per-check timeouts (P2). +- Check kinds implemented in Go, config-driven: `http`, `tcp`, `disk` (SSH `df`), + `cert-expiry`, `drift` (DB inventory vs live state — mismatches raise + `drift` signals with the observed diff as evidence), `ssh-script` (allowlisted). +- Every run writes metrics (`health`, `probe_latency_ms`, …) and updates + `entity_status` in place (R3-6). +- **Signal dedup (R3-8):** failure upserts against the partial unique index — an + existing open signal gets `occurrence_count+1`, `last_seen_at=now()`. Recovery + auto-resolves. A resolve→re-raise cycle increments `flap_count`; after 3 cycles/1h + the signal enters hold-down (no notifications, no auto-act) and a `flapping` + meta-signal is raised for the operator. +- Entities with `maintenance_until > now()` still get metrics but no signals and are + excluded from auto-act. +- Housekeeping jobs: daily `pg_dump` + rclone push, idempotency-key prune, feedback + watermark advance, secrets-expiry check (M5). -**Before any code**, finalize the ontology. The current `ontology.yaml` has 8 domains -and 14 relationship types. The new ontology adds: +### Actuator (Act) -- **Layer 3 entities** (cognition): signal, execution, feedback, pattern, skill, - approval, classification, document, runbook -- **New relationships**: `triggers`, `produces`, `contributes-to`, `informs`, `guides`, - `precedes`, `performs`, `procedure-for`, `learned-from` -- **Lifecycle definitions** for operational entities (signals, executions, patterns, - skills) — not just infrastructure +- Consumes signals with `route='auto-act'` classifications via + `SELECT … FOR UPDATE SKIP LOCKED`; per-target serialization with + `pg_advisory_xact_lock(hashtext(target_entity_id::text))` (SG5). +- Executes only stored skill/runbook procedures (R3-9) over SSH with a **restricted + key**: dedicated keypair, `command=`/`from=` constrained in `authorized_keys` on + targets, mounted read-only into the scheduler/actuator container only (S1). Phase 3 + formalizes this as the gateway: all execution flows through `/executions`; Hermes + never touches SSH. +- Context-aware SSH (SG13): session started in a goroutine, `ctx.Done()` closes + session+client to unblock; SSH errors classified (network → retryable/circuit, + auth → fatal alert, non-zero exit → failed, timeout → `timed_out → verifying`). +- **Circuit breaker per target host** (M4): N consecutive failures → open circuit, + exponential backoff, raise `target-unreachable` signal instead of piling failed + executions into the learning corpus. +- Loop guard: retry budget per (entity, action) from execution history; rate cooldown + per (entity, action). +- Autonomy kill-switch consulted every pass (`global.auto_act`, + `never_auto_act.`). +- Graceful shutdown (SG4): `signal.NotifyContext`; stop intake → 30s drain → if an + execution is in flight, mark `failed` ("shutdown interrupted") + emit feedback → + close pool. Compose sets `stop_grace_period: 30s`. -**Deliverables:** -- `seeds/ontology.yaml` — entity types, relationship types, lifecycle definitions - (seeded into `entity_types`, `relationship_types`, `lifecycle_defs` on deploy) -- `seeds/inventory.yaml` — adapted from current `inventory.yaml` (seeded into - `entities` + `relationships`) -- `seeds/policy.yaml` — adapted from current `policy.yaml` (seeded into `risk_classes`, - `approval_rules`, `autonomy_settings`) -- `internal/ontology/ingest.go` — idempotent seed → DB ingest +### Learning engine -### 2. Database layer + migrations (`migrations/`, `internal/db/`) +Concrete algorithm (P4 + S4 guardrails): -- Write the 5 migrations above -- Set up sqlc for type-safe Go database access -- Connection pool, migration runner -- Graph traversal queries (blast radius, dependency chains, knowledge lookup) -- Import script for existing `signals/*.jsonl`, `ledger/*.jsonl` → DB +1. Hourly, read feedback past the watermark, joined to executions, grouped by + `(applies_type, action)`. +2. Update pattern counters atomically; recompute confidence as the **Wilson score + lower bound** of success rate (conservative for small N), additionally capped by + `min(confidence, evidence_count/5)` so nothing looks confident before 5 samples. +3. Status: `hypothesized` (N<5) → `validated` (N≥5 and confidence ≥ 0.7, emits + `pattern.validated` event + notification) → **`active` only via operator PATCH** + (policy-gated `config_mutation`). +4. Anomaly quarantine: >10 identical-outcome feedback rows within 1h for one + (type, action) → `quarantined=true`, meta-signal for review. +5. Skill refinement: an `active` pattern with confidence > 0.7 drafts/refines a skill + (new version row; `changed_by`, `change_reason` recorded). Skills follow their own + lifecycle; activation is operator-gated. **No skill ever auto-promotes an action + into `destructive` autonomy** — hard-coded, not policy data. +6. Classifier consumption: pattern confidence and skill existence feed the + auto-act/escalate score; frequent-failure patterns *lower* confidence. -### 3. Unified API server — Go (`cmd/api/`, `internal/api/`) +### Notifier -One Go binary (Gin web framework) exposing REST + MCP from the same codebase. +`Notifier` interface (SendAlert / SendApprovalRequest / decision intake); Matrix +implementation posts approvals with ✅/❌ reactions and writes decisions **directly +to the approvals table** — the DB is the rendezvous, no service-to-service calls, so +pending approvals survive restarts of either side (SA7/A7). If the notifier is down, +the operator's alternative path is the REST `/approvals` endpoint. -**MCP interface** (`internal/api/mcp.go`): -- JSON-RPC over SSE, compatible with Hermes MCP client -- Tools: `get_host`, `list_services`, `search_knowledge`, `get_entity`, - `get_relations`, `get_blast_radius`, `get_signal_history`, `get_ledger`, - `get_patterns`, `get_skills`, `get_state_snapshot` -- **Observability tools**: `query_metrics` (time-series for an entity/metric), - `get_trend` (trend analysis with slope + anomaly flags), `get_audit_trail` - (who-did-what for an entity or actor), `get_event_timeline` (structured event - feed for an entity or time range), `get_agent_activity` (what the agent did), - `get_health_summary` (current health across the fleet with trend indicators) -- All read from PostgreSQL +## Security model -**REST interface** (`internal/api/routes/`): -- `GET /api/v1/entities` — list entities (filter by type, state, domain) -- `GET /api/v1/entities/{id}` — entity detail + relationships -- `POST /api/v1/entities` — create entity (creates inventory entry) -- `PATCH /api/v1/entities/{id}` — update entity (state transition, attributes) -- `GET /api/v1/signals` — list signals (filter by state, severity, entity) -- `POST /api/v1/signals/{id}/ack` — acknowledge -- `POST /api/v1/signals/{id}/resolve` — resolve -- `GET /api/v1/approvals` — pending approvals -- `POST /api/v1/approvals/{id}/decide` — approve/deny (Authentik-gated) -- `POST /api/v1/exec` — gated execution (classify → check approval → execute → feedback) -- `GET /api/v1/patterns` — list patterns (filter by entity_type, action, status) -- `GET /api/v1/skills` — list skills (filter by applies_to, status) -- `GET /api/v1/knowledge/{entity_id}` — knowledge graph query -- `GET /api/v1/knowledge/search?q=...` — search knowledge graph -- `GET /api/v1/ontology` — list entity types, relationship types, lifecycles -- `POST /api/v1/ontology/entity_types` — create entity type (extend the schema) -- `WS /api/v1/events` — real-time stream (signals, approvals, executions, feedback) -- **Observability routes** (`internal/api/routes/observability.go`): - - `GET /api/v1/metrics` — query time-series: `?entity_id=&metric=&from=&to=&interval=` - - Returns raw samples or rollups (auto-selects 1h/1d aggregates based on range) - - `?rollup=1h|1d|raw` to force a specific resolution - - Supports multiple metrics: `?metric=disk_usage_pct&metric=probe_latency_ms` - - `GET /api/v1/metrics/{entity_id}/{metric}` — single metric for one entity - - `?from=2026-07-01T00:00:00Z&to=2026-07-06T00:00:00Z&rollup=1h` - - Returns: `{entity_id, metric, samples: [{ts, avg, min, max, count}], trend: {slope, direction, anomaly}}` - - `GET /api/v1/trends/{entity_id}` — trend analysis for all metrics on an entity - - Returns slope (improving/degrading/stable), recent anomalies, forecast (simple linear) - - `GET /api/v1/audit` — audit log: `?actor_type=&actor_id=&entity_id=&action=&from=&to=` - - Paginated, ordered by ts DESC - - `?correlation_id=` to trace a full execution chain - - `GET /api/v1/events` — historical events: `?type=&entity_id=&severity=&from=&to=` - - Same data as the WebSocket stream, but queryable historically - - `GET /api/v1/agent-activity` — agent behavior log: `?agent_id=&activity_type=&entity_id=&from=&to=` - - Includes token usage, latency, success/failure per tool call - - `GET /api/v1/health` — fleet health summary with trend indicators - - Returns: `{entities: [{id, type, health, trend, last_probe}], summary: {healthy, degraded, down, unknown}}` - - `GET /api/v1/export` — export current DB state as YAML (for DR / version control) +Threat model (documented in `docs/adr/0007-threat-model.md`): -**Policy enforcement** (`internal/policy/classify.go`): -- Every mutating endpoint classifies the action via the policy DB -- Risk class → approval check → autonomy check -- All mutations write to the ledger automatically +| Boundary | Mechanism | +|---|---| +| Internet/mesh → API | Caddy (TLS) + Authentik forward-auth **and** in-API OIDC JWT validation (Caddy compromise ≠ API compromise) | +| LAN → API (break-glass) | Same in-API auth (OIDC JWT / bearer) — network origin is defense-in-depth, never the auth. Plaintext hop accepted for emergency/watchdog use only; routine traffic uses the mesh | +| Workstation → Hermes gateway | Mesh membership (network) + gateway token/mTLS (application); **mesh-only, no LAN binding** | +| Hermes → API (MCP) | Dedicated Docker network + static bearer token + HMAC | +| Roles → Postgres | Least-privilege DB users (DDL only in init; DML per role), TLS on the Docker network (S7) | +| Actuator → fleet | Restricted SSH key (`command=`,`from=`), actuator container only; full gateway in Phase 3 (S1/S10) | +| Gitea → deploy | HMAC-signed webhook, localhost/mesh-bound listener, non-root deploy user, CI-gated (S8/M1) | +| Learning → policy | Structurally impossible: learning role's DB user has **no write grants** on policy tables; changes route through approvals (S3/S4) | +| Approvals | Single-use HMAC tokens, hashed at rest, TTL (S5) | +| Secrets | Infisical with machine identities; bootstrap root of trust = master key in mac-mini Keychain, backed up offline; one SOPS age key retained for DR until a restore drill passes (S9) | -**Auth:** -- MCP interface: no auth (internal, container-to-container) -- REST interface: Authentik OIDC forward-auth (via Caddy) for operator endpoints -- Internal: shared secret (Docker network) +Rotation cadences (M5): SSH actuator key 6mo, Infisical machine tokens 90d, MCP +bearer 90d, webhook HMAC 1y — each with a documented procedure and a scheduler check +that raises a signal 2 weeks before expiry. Supply chain (M6): images pinned by +digest, `govulncheck` + `golangci-lint` in CI, distroless runtime. -### 4. Scheduler + Actuator — Go (`cmd/scheduler/`, `internal/scheduler/`, `internal/actuator/`) +## Observability -**Scheduler (Observe)** — Go service with goroutines for concurrent probes: -- HTTP health probes (concurrent, with timeouts) -- Disk usage probes (SSH to hubris/strong) -- Drift detection (inventory vs live state) -- Writes signals + state snapshots to DB -- Runs on a 10-min ticker +All capture goes to Postgres (hypertables above) through `internal/observability`: -**Actuator (Act)** — Go service, the control loop: -- Reads open signals with `recommended_action` -- For each: classify via `internal/policy/classify.go` - - **auto-act**: look up skill for (entity_type, action) → follow procedure → execute - via SSH → verify → record execution → generate feedback → update patterns - - **escalate**: create approval request → notify via Matrix → acknowledge signal -- Loop-guard: check execution history per (entity, action) to cap auto-retries - (`SELECT ... FOR UPDATE SKIP LOCKED` for concurrency safety) -- Autonomy kill-switch: check `autonomy_settings` table +- **Metrics** — check results, API request count/latency, Go runtime stats, learning + metrics (pattern confidence, skill success rate, auto-act vs escalation ratio), + agent token/tool-call counts. Also exposed as a Prometheus-format `/metrics` + endpoint (internal only) so Grafana/Prometheus can attach later without schema + work. +- **Audit** — middleware on every mutating REST call + MCP tool call + actuator SSH + command; policy changes carry before/after hashes; operator identity from OIDC + claims (M3). +- **Events** — emitted **in the same transaction** as the state change (SG10); + post-commit `NOTIFY` feeds the SSE stream; in-process bus covers API-local events + (SG8). SSE subscribers get bounded buffers with drop-oldest + heartbeats (P6); + delivery is best-effort, history via `GET /events`. +- **Correlation** — a `correlation_id` is minted at signal creation (or API request) + and propagated via `context.Context` through classification → approval → execution + → SSH → verification → feedback, linking audit + events end-to-end. +- **Logging** — `slog` JSON to stdout; every line carries `service`, + `correlation_id`, `entity_id` where applicable; `debug=true` enables probe + payloads/SQL/classification reasoning. +- **Agent self-inspection** — MCP tools let Hermes query its own history, trends, + audit trail, and efficiency (token usage over time). -### 5. Learning engine — Go (`internal/learning/`) +SLOs (M7): check interval 10min ±1min; API p99 < 200ms; deploy < 5min; alert +delivery < 30s; watchdog detection < 5min. -The feedback loop that makes the agent improve over time. +## Operations -**Feedback recording** (`internal/learning/feedback.go`): -- After every execution, evaluate the outcome: - - Did the verification command pass? → success - - Did it fail? → failure - - Did it partially work? → partial - - Did something unexpected happen? → unexpected -- Record a feedback entry with: outcome, observation (what happened vs expected), - lesson (extractable insight), unexpected_side_effects +### Backup / restore / DR (A3, O3, O4) -**Pattern extraction** (`internal/learning/pattern.go`): -- Periodically scan accumulated feedback for (entity_type, action) pairs -- When N+ executions share a similar outcome, extract a pattern: - - "Restarting service:X typically takes 15s and succeeds" - - "Deploying to LXC:Y via webhook has 30% failure rate, retry helps" -- Patterns start as `hypothesized`, move to `validated` after enough evidence, - then `active` (used by the decision classifier) -- Confidence score = success_count / evidence_count, adjusted by recency +- **Daily `pg_dump`** (custom format, compressed) + WAL archiving for PITR; pushed + off-host to **Proton Drive via rclone** (reuse existing rclone credentials from + LXC 132 setup). Retention 30 daily + 12 monthly. Pre-deploy dump before every + migration run. +- Infisical: native backup + secrets exported to one SOPS-age-encrypted file as + fallback (the retained age key is the DR escape hatch). +- **Monthly automated restore drill**: scratch container, `pg_restore`, run the + export round-trip check, alert on failure. +- DR targets: **RTO 4h / RPO 24h**. Cold-start runbook + (`docs/operations/dr.md`): fresh machine → OrbStack + Docker → clone repo → + restore Infisical → `pg_restore` → `docker compose up -d` → verify `/healthz` + + fleet health. -**Skill management** (`internal/learning/skill.go`): -- When a pattern reaches `active` status with confidence > 0.7, create or refine - a skill for that (entity_type, action) pair -- Skills codify the best-known procedure (what steps to take, what to verify, - expected duration, known failure modes) -- Skills are versioned — each refinement increments the version -- The actuator looks up skills before executing: if a skill exists, follow it; - if not, use the default procedure and generate feedback for future pattern extraction +### Watchdog (O2) -**How the classifier uses learning** (`internal/policy/classify.go`): -- Confidence scoring now checks patterns + skills, not just raw ledger history: - - If a pattern exists for (entity_type, action) with high confidence → boost - auto-act confidence - - If patterns show frequent failures → lower confidence, escalate - - If a skill exists → higher confidence (proven procedure available) -- This is the closed loop: **execution → feedback → pattern → skill → classification - → execution** (better informed each time) +Cron on apps/105 (outside the stack): every 5min, curl `/healthz` on **both paths** — +`http://:8090/healthz` (LAN, tests the API itself) and +`http://:8090/healthz` (mesh, tests the path Caddy and workstations use) — +plus `pg_isready`; on any failure, post directly to the Matrix webhook naming which +path failed (LAN-down = stack problem; mesh-down-LAN-up = NetBird problem). This is +the orthogonal "who watches the watcher" channel. -### 6. Knowledge graph ingestion (`internal/ontology/ingest.go`) +### Deploy, release, rollback (O1, O5, M1, R3-12) -After the ontology is defined and the DB schema is in place: +- CI (Gitea Actions): `go vet`, `golangci-lint run`, `go test ./... -race -cover` + (coverage gates: ≥80% `internal/policy` + `internal/learning`, ≥60% elsewhere), + `oapi-codegen`/`sqlc` diff check (generated code committed and clean), + `govulncheck`, `docker build`. +- Deploy: webhook (HMAC-verified, gated on green CI) → `deploy.sh`: `git pull` → + pre-deploy `pg_dump` → build image `oikos:` (multi-stage; **no host `go + build`**, P7) → `oikos migrate` init job → `docker compose up -d --no-deps` app + roles (Postgres container never recreated on routine deploys) → healthcheck-gated. +- Migration compatibility: additive-only per deploy window (new columns nullable); + new code tolerates previous schema for one deploy. +- Rollback: retag compose to the previous SHA (last 5 images kept) → if the migration + was the problem, `pg_restore` the pre-deploy dump → `docker compose up -d`. + Runbook in `docs/operations/rollback.md`. +- Health checks (O6): API `/healthz` (DB ping); scheduler freshness ("last successful + check pass < 15min") exposed via `entity_status` self-row; notifier "last poll < + 60s"; Hermes gateway ping — all wired into compose `healthcheck` + watchdog. -- On deploy, walk `docs/` directory -- Parse each markdown file: - - Extract frontmatter for metadata (entity type, tags, relations) - - Infer entity relationships from path conventions: - `docs/containers/105-apps.md` → relationship to `entity:lxc:apps` - - Extract cross-references (markdown links) → relationships -- Create knowledge entities in the `entities` table (type = `document`, `runbook`, - `investigation`, etc.) with `documents` / `procedure-for` edges to infrastructure - entities -- Idempotent — safe to re-run on every deploy +### Coexistence + cutover (A6) -**Agent access:** -- MCP tool `search_knowledge(query)` — full-text search on knowledge entities -- MCP tool `get_entity_knowledge(entity_id)` — all docs related to an entity -- MCP tool `get_relations(entity_id)` — graph traversal (blast radius, dependencies) +During coexistence apps/105 is **read-only toward shared state** (its scheduler +disabled once the new stack's checks are verified). Cutover: verify traffic on the +new stack → disable apps/105 services → retarget Caddy + Gitea webhooks → cleanup. +Rollback after cutover = re-enable apps/105 (its JSONL state frozen at cutover; +reconciliation = re-import from Postgres if ever needed). -### 7. Hermes agent container (`compose/hermes/`) +## Phasing — with acceptance criteria (R3-15) -- Hermes Agent runtime in a Docker container, gateway mode -- Config: `hermes/config.yaml` (providers, models, gateway port) -- Persona: `hermes/SOUL.md` (homelab-specific) -- Skills: `hermes/skills/homelab-ops/SKILL.md` — how to use the Oikos API, classify - actions, request approvals, query the knowledge graph -- Access: Oikos API via MCP (container network) + SSH keys mounted (hybrid) -- Gateway port 8092 — workstations connect remotely -- Hermes data volume for persistent state +Each phase ends with explicit "done when" checks an implementing agent can run. -### 8. Infisical secrets migration +**Phase 0 — Ontology + contract (no service code):** +- Finalize entity types (incl. abstract hierarchy, Person/Agent/IdentityProvider, + Cluster/ComposeStack), relationship types (with endpoint types + cardinality), + lifecycles (all terminal states + named preconditions). +- Write `seeds/*.yaml`; write `api/openapi.yaml` v1 for the full REST surface; write + ADRs 0001–0010 from the Decisions table. +- **Done when:** seeds lint against a meta-schema validator script; `openapi.yaml` + passes `redocly lint`; operator has reviewed the lifecycle diagrams + spec. -Same as rev 1: -- Stand up Infisical in the Docker stack -- Migrate SOPS secrets (one-time decrypt + import) -- Wire all Go services to Infisical via machine identity -- Retire SOPS + age keys +**Phase 1 — Foundation (DB + skeleton):** +- Go module `github.com/dtoro/oikos`; `cmd/oikos` skeleton with role subcommands; + domain layer + sentinel errors; migrations 001–006; `oikos migrate` + `oikos seed` + (idempotent, transactional, `seed_versions`); sqlc + repositories; slog; + testcontainers harness; daily backup job + watchdog cron installed. +- **Done when:** `make test` green including: seed ingest twice = no-op; export + round-trip byte-stable; `blast_radius` correct on a cyclic fixture; abstract-type + instantiation rejected; relationship endpoint validation honors inheritance; + hypertables + CAGGs + retention created idempotently; legacy `signals/*.jsonl` + + `ledger/*.jsonl` imported by `scripts/import-legacy.sh`. -### 9. Notifier — Go (`cmd/notifier/`, `internal/notifier/`) +**Phase 2 — API:** +- oapi-codegen server; auth middleware (OIDC JWT + roles, bearer for agent); + problem+json mapping; idempotency; ETag/If-Match; rate limiting; audit middleware; + transactional event emitter + SSE stream; knowledge ingestion from `docs/` + (content-hash skip, P3); MCP server (official SDK) over the shared service layer; + pattern/skill/policy endpoints with dual-control approvals; CI pipeline live. +- **Done when:** spec-conformance tests pass (schemathesis or generated-client round + trip); `curl /api/v1/entities?type=service` returns the fleet; MCP `list_entities` + returns the same data; a mutating call without `If-Match` on a stale version → + 412; replayed `Idempotency-Key` returns the cached response; SSE stream shows an + `entity.created` event; audit rows carry the OIDC sub. -**Interface (Go):** -```go -type Notifier interface { - SendAlert(ctx context.Context, signal Signal) error - SendApprovalRequest(ctx context.Context, approval Approval) error - ListenForDecisions(ctx context.Context) (<-chan ApprovalDecision, error) -} -``` +**Phase 3 — Control loop:** +- Scheduler with `check_defs` runner, dedup/flap/maintenance logic, metrics + + `entity_status`; actuator with restricted SSH key, advisory locks, circuit breaker, + retry budgets, graceful shutdown; learning engine per the algorithm above; approval + tokens; notifier (Matrix, DB rendezvous). +- **Done when:** killing a probed service raises exactly one signal (repeats + increment `occurrence_count`); a flapping fixture enters hold-down; a + `service-down` signal on a `reversible_low` target auto-restarts, verifies, and + the full correlation chain (signal → classification → execution → SSH audit → + feedback) is queryable by one `correlation_id`; a `destructive` action produces a + Matrix approval whose ✅ token is single-use; after 5 successful executions a + pattern reaches `validated` and **stays there** until operator PATCH; kill-switch + `global.auto_act=off` forces escalation. -**Matrix implementation:** -- Sends alerts to `@dtoro:avispero` via Synapse (LXC 118) -- Approval requests as messages with ✅/❌ reactions -- Listens for reactions to record decisions -- Pluggable — future implementations (webhook, email) register via config - -### 10. Docker build + deploy pipeline - -- Multi-stage Dockerfiles: Go build stage → minimal runtime image (alpine or scratch) -- `docker compose build` from repo root -- Gitea webhook on push to `main` → deploy script on mac-mini -- Deploy script: `git pull && go build ./... && docker compose build && docker compose up -d` -- Seed ingest runs as part of the API startup (idempotent) -- Health checks on each service - -### 11. Ingress re-point (Caddy) - -- `dtoro/caddy-conf`: point `mcp.hubris.network` + `oikos.hubris.network` → - mac-mini mesh IP :8090 (API) -- Future: `hermes.hubris.network` → mac-mini:8092 -- DNS and public URLs unchanged - -### 12. mac-mini host setup - -- New directory: `~/oikos-os/` — repo clone + `docker compose` working dir -- Existing `/opt/homelab-context/` stays untouched until cleanup phase -- Prerequisites: Docker (or OrbStack), Go toolchain (for local dev), SSH keys -- Cleanup (deferred): stop launchd git-sync, remove native Hermes, remove old clone - -### 13. Decommission apps/105 (deferred) - -Keep apps/105 running as fallback. Cutover checklist when ready: -1. Verify Docker OS serves all traffic -2. `systemctl disable --now` Oikos services on apps/105 -3. Remove old checkouts -4. Update Caddy backends exclusively to mac-mini -5. Remove/retarget Gitea webhooks - -### 14. Observability + data capture — Go (`internal/observability/`) - -The OS captures three classes of data, all in PostgreSQL (TimescaleDB for -time-series, regular tables for audit/events): - -**A. Metrics (time-series)** — `internal/observability/metrics.go`: -- **Infrastructure probes**: every scheduler probe writes metrics: - - `health` (0=down, 1=degraded, 2=healthy) per service - - `disk_usage_pct` per mount point per host - - `probe_latency_ms` per probe target - - `drift_count` per drift check -- **OS self-metrics**: the API server exposes a `/metrics` endpoint and also - writes to the DB: - - `api_request_count` (per route, per status code) - - `api_latency_ms` (p50, p99) - - `db_connections_active` - - `db_query_duration_ms` - - `goroutines` (Go runtime) - - `gc_pause_ms` - - `memory_alloc_mb` -- **Learning metrics**: the learning engine writes: - - `pattern_confidence` per pattern ID - - `skill_success_rate` per skill ID - - `auto_act_count` vs `escalation_count` (autonomy ratio over time) - - `execution_duration_ms` per (entity_type, action) -- **Agent metrics**: Hermes activity is logged: - - `agent_token_count` per session - - `agent_tool_call_count` per session - - `agent_decision_latency_ms` -- All metrics are written to `metric_samples` as hypertable with 1h and 1d - continuous aggregates. Raw data retained 90 days, rollups 1 year. - -**B. Audit log (immutable)** — `internal/observability/audit.go`: -- Gin middleware: every mutating API call (POST/PATCH/DELETE) writes to - `audit_log` with actor, action, entity, method, path, status, detail, source_ip -- MCP tool calls also audited (actor_type='agent') -- SSH commands by the actuator audited (actor_type='system', method='SSH') -- Policy mutations get special audit entries with a hash of the before/after state -- `correlation_id` propagated through the call chain (API → actuator → SSH → result) - so a full execution chain can be reconstructed: "signal → classification → - execution → SSH command → verification → feedback" all linked by correlation_id - -**C. Event stream** — `internal/observability/events.go`: -- Every significant state change emits an event to the `events` table: - - Signal lifecycle: `signal.raised`, `signal.acknowledged`, `signal.acting`, - `signal.resolved`, `signal.muted` - - Execution: `execution.started`, `execution.completed`, `execution.failed` - - Approval: `approval.requested`, `approval.decided` - - Deploy: `deploy.triggered`, `deploy.completed` - - Learning: `pattern.validated`, `skill.refined` - - Entity: `entity.created`, `entity.state_changed` - - Policy: `policy.changed` -- The WebSocket `/api/v1/events` streams from this table (new events pushed to - subscribers, historical events queryable via REST) -- Events carry `correlation_id` for end-to-end tracing - -**D. Agent activity** — `internal/observability/agent.go`: -- Hermes MCP tool calls logged to `agent_activity` with tool name, entity - acted upon, duration, token count, success/failure -- SSH commands by the agent logged separately -- Reasoning/decision audit: when the agent makes a classification decision, - the reasoning is recorded (input, classification result, route, why) -- This is the "what is the agent doing and is it getting better?" dataset - -**E. Structured logging** — `internal/observability/logging.go`: -- All services use Go's `slog` (structured JSON logging) -- Every log line has: `ts`, `level`, `service`, `msg`, `correlation_id` - (when applicable), `entity_id` (when applicable) -- Logs go to stdout (Docker captures them, `docker compose logs` for access) -- Debug mode: `debug=true` env var enables verbose probe payloads, SQL queries, - classification reasoning in logs - -**F. Data availability for agents**: -- The agent can query its own history: "what actions have I taken on service:caddy - in the last 30 days, and what were the outcomes?" -- The agent can see trends: "is disk usage on hubris trending upward?" -- The agent can audit: "who changed the policy for service:caddy and when?" -- The agent can self-assess: "am I getting more efficient? (token usage trend)" -- All through MCP tools: `query_metrics`, `get_trend`, `get_audit_trail`, - `get_event_timeline`, `get_agent_activity`, `get_health_summary` - -**G. Future visualization plug-in points**: -- The `/api/v1/metrics` REST endpoint returns JSON time-series — any tool - (Grafana, custom dashboard, notebook) can consume it -- The data model is compatible with Grafana's PostgreSQL data source - (time column + metric name + value + tags as labels) -- The event stream (`/api/v1/events` + WebSocket) can feed a live dashboard -- The audit log can feed a SIEM or compliance tool -- No UI built now — the APIs are the contract; visualization plugs in later - -## Phasing - -**Phase 0 — Ontology design (no code):** -- Finalize entity types, relationship types, lifecycles -- Write seed manifests (`seeds/ontology.yaml`, `seeds/inventory.yaml`, `seeds/policy.yaml`) -- Review diagrams with operator - -**Phase 1 — Foundation (Go + DB):** -- Go module setup, project structure -- PostgreSQL + TimescaleDB setup -- Migrations 1-6 (ontology, instances, operations, learning, policy, observability) -- Seed ingest pipeline (YAML → DB) -- sqlc queries for core operations -- Structured logging setup (slog) -- Import existing signals/ledger data - -**Phase 2 — API (Go):** -- Gin server with REST routes -- MCP protocol adapter (including observability tools) -- Policy enforcement middleware -- Audit middleware (every mutating call → audit_log) -- Event emitter (every state change → events table) -- Knowledge graph ingestion from docs/ -- Observability routes (metrics, trends, audit, events, health) - -**Phase 3 — Control loop (Go):** -- Scheduler (Observe) — probes, signals, state snapshots, **metric recording** -- Actuator (Act) — classify, execute, verify, **correlation ID propagation** -- Learning engine — feedback, patterns, skills, **learning metrics** -- Circuit breaker per target host - -**Phase 4 — Agent (Hermes container):** -- Hermes Docker image, gateway config -- Homelab skills -- Connect from workstation, verify MCP + SSH +**Phase 4 — Agent (Hermes):** +- Hermes container (gateway :8092, mesh-published, token/mTLS), homelab skills, + MCP wiring, agent-activity logging. No SSH keys in this container. +- **Done when:** from a workstation over mesh, Hermes answers "what depends on + authentik?" via `get_blast_radius`, requests an execution that routes through + `/executions` policy gating, and its tool calls appear in `agent_activity`. **Phase 5 — Secrets (Infisical):** -- Stand up Infisical, migrate SOPS, wire services +- Infisical up; SOPS migrated; services on machine identities; rotation checks; + SOPS-age DR fallback exported. +- **Done when:** no service reads SOPS at runtime; restore drill of Infisical backup + passes; rotation runbooks written. **Phase 6 — Deploy + cutover:** -- Docker Compose, Gitea webhook, Caddy re-point -- End-to-end verification -- Stop apps/105, clean up mac-mini - -## Reuse (logic carried over, rewritten in Go) - -| Existing Python | What it becomes in Go | -|---|---| -| `oikos/decide.py` | `internal/policy/classify.go` — same scoring logic, reads from DB + patterns | -| `oikos/signal.py` | `internal/scheduler/signal.go` — same lifecycle, DB-backed | -| `oikos/approve.py` | `internal/policy/approve.go` — same grant lifecycle, DB-backed | -| `oikos/ledger.py` | `internal/db/` — ledger_entries table + sqlc queries | -| `oikos/policy.py` + `policy.yaml` | `internal/policy/` + `seeds/policy.yaml` → DB tables | -| `oikos/drift.py` | `internal/scheduler/` — drift detection, writes signals to DB | -| `oikos/relations.py` | `internal/ontology/graph.go` — SQL graph traversal | -| `oikos/report.py` | `internal/api/routes/` — report endpoints, reads from DB | -| `mcp/server.py` | `internal/api/mcp.go` — MCP adapter on top of DB | -| `bin/homelab` logic | `internal/api/routes/` — same operations, REST interface | -| `oikos/scheduler.py` | `internal/scheduler/` — same probes, goroutines for concurrency | -| `oikos/approve.py` Matrix delivery | `internal/notifier/matrix.go` | -| *(new)* | `internal/learning/` — feedback, patterns, skills (no Python equivalent) | - -## Risks / trade-offs - -- **Go rewrite** — the existing Python code (~4400 lines) is replaced. The logic and - design patterns carry over, but it's a full rewrite. Mitigated by the fact that the - Python code is well-documented and the Go structure mirrors it. -- **Hermes in Docker** — agent's world is the container. SSH access is the bridge. - Hybrid approach (mounted keys now, actuator gateway later). -- **PostgreSQL as SPOF** — mitigated by Docker volume persistence + automated - `pg_dump` backups (the scheduler can do this once running). -- **Learning model cold start** — no patterns/skills exist initially. The agent starts - cautious (escalates everything), accumulates feedback, and gradually becomes more - autonomous as patterns validate. This is by design — trust is earned. -- **Infisical bootstrapping** — SOPS coexists during transition. Keep SOPS as fallback. -- **Multi-agent concurrency** — `SELECT ... FOR UPDATE SKIP LOCKED` prevents two - actuator passes from acting on the same signal. -- **Ontology evolution** — as the homelab changes, entity types and relationship types - need to be added/modified. The DB-native approach makes this an API call, not a - file edit + redeploy. - -## Audit findings (best practices, security, performance, sanity) - -A full audit was performed against the plan. Findings are organized by category and -severity. **High-severity items must be addressed before implementation; medium items -should be addressed during the phase they belong to.** - -### Security - -| ID | Severity | Finding | Recommendation | -|---|---|---|---| -| S1 | **HIGH** | SSH private keys mounted into Hermes container — a compromised container has unrestricted SSH to all hosts | Use a dedicated, restricted SSH key with `command=` in authorized_keys. Time-box the stopgap. Build the actuator gateway (which brokers SSH per-execution) sooner rather than later. | -| S2 | **HIGH** | MCP interface has no auth — any container on the Docker network has full read access to the control plane | Add a shared secret or mTLS between API and Hermes. Bind MCP to a dedicated Docker network, not the default bridge. Never expose the MCP port via Caddy without auth. | -| S3 | **HIGH** | Policy DB is mutable via the same API it governs — a compromised API can rewrite its own approval rules (e.g. flip `destructive` to auto-act) | Policy mutations require a meta-approval (dual-control). Add an immutable audit log of policy changes. Startup self-check: alert if policy hash differs from a known-good baseline. | -| S4 | **HIGH** | Learning model poisoning — flapping services or misconfigured probes can inject biased feedback to push patterns past the confidence threshold and unlock auto-act for destructive actions | Require human confirmation before pattern transitions to `active`. Cap confidence by sample size (require N≥5 executions). Detect anomalous feedback bursts and quarantine. Never let a skill auto-promote to destructive risk class. | -| S5 | **HIGH** | `confirmation_phrase` is a weak auth primitive — if reused, stored in plaintext, or transmitted via Matrix, it's replayable | Make it single-use (one phrase per approval). Store hashed. Transmit only the decision + HMAC, not the phrase. Prefer signed approval tokens. | -| S6 | MEDIUM | REST `/exec` and `/decide` endpoints rely on Authentik forward-auth only — if the API port is reachable directly on the mesh, auth disappears | Enforce OIDC JWT validation in the API middleware too (defense in depth). Bind API port to localhost + Caddy only. | -| S7 | MEDIUM | No TLS between internal services — Postgres connections are plaintext on the Docker network | TLS to Postgres (server cert verification). Use dedicated Docker networks per trust boundary. Document the threat model: is the Docker network trusted? | -| S8 | MEDIUM | Webhook (Gitea → mac-mini deploy) has no auth specified — anyone who can reach the endpoint can trigger arbitrary code execution via `go build` | HMAC signature verification on the webhook. Bind listener to localhost (Gitea reaches via mesh). Run deploy script as non-root. Verify commit signatures before `git pull`. | -| S9 | MEDIUM | Infisical bootstrapping has a chicken-and-egg — Infisical's own master key must come from somewhere | Document the bootstrap root of trust explicitly: where the master key lives (mac-mini keychain), how it's backed up, revocation path. Keep SOPS as fallback until Infisical has a tested restore-from-backup drill. | -| S10 | MEDIUM | Blast radius of a compromised container is wide — Hermes has SSH + MCP + gateway port + data volume; actuator has SSH + writes to policy/learning tables | Principle of least privilege per container: only the actuator should have SSH, not Hermes. Split networks: data (PG), ops (SSH egress), front (Caddy). Use Docker user namespaces and read-only root filesystems. | - -### Performance - -| ID | Severity | Finding | Recommendation | -|---|---|---|---| -| P1 | MEDIUM | Recursive CTE `blast_radius` has no cycle guard — cycles (A→B→A) inflate work exponentially with depth | Add a visited-set guard (array accumulator in the recursion). Cap `max_depth` at 3. Add `LIMIT` on the outer query. Consider a closure table for hot-path queries. | -| P2 | MEDIUM | Concurrent probes with no concurrency cap — unbounded goroutines could exhaust FDs or hammer slow targets | Bounded worker pool (`errgroup.SetLimit`). Per-target probe timeout. Jitter to avoid thundering herd on shared backends. | -| P3 | MEDIUM | Knowledge graph re-ingestion on every deploy is wasteful if no docs changed | Content-hash each doc (store hash on the entity). Skip ingestion if hash unchanged. Run in a single transaction with deferred FK checks. | -| P4 | MEDIUM | Pattern extraction frequency unspecified — either too tight (scans whole table) or too loose (patterns lag) | Define cadence (hourly). Use a high-watermark on `feedback.ts`. Add index on `feedback(ts)`. Process only new feedback. | -| P5 | **HIGH** | Table growth unaddressed — `state_snapshots` (every entity every 10 min) and `executions`/`feedback` grow indefinitely | Partition `state_snapshots` and `signals` by month. Define TTLs (snapshots > 90 days → aggregate, raw → archive). Add a `prune_*` job in the scheduler. | -| P6 | MEDIUM | WebSocket `/events` has no backpressure — a stalled client pins a goroutine and accumulates memory | Bounded channel per subscriber with drop-oldest-on-full. Max subscribers cap. Heartbeat/timeout. Document: best-effort vs. guaranteed delivery. | -| P7 | LOW | Deploy script runs `go build` on host AND in Docker — double build, host Go toolchain is a deploy dependency | Standardize on multi-stage Docker build only. Drop host `go build` from deploy script. Keep host toolchain for local dev only. | - -### Architecture - -| ID | Severity | Finding | Recommendation | -|---|---|---|---| -| A1 | **HIGH** | No testing strategy — 13 workstreams, no mention of unit/integration/property tests | Add a testing workstream: unit tests for `classify.go`, `pattern.go`, `skill.go`; integration tests with testcontainers Postgres; golden-file tests for seed ingest; property test for blast-radius (cycles, depth caps). Coverage gates per package. | -| A2 | **HIGH** | No observability — no structured logging, metrics, or traces. Can't inspect why the classifier escalated or how long probes took | At minimum: structured JSON logs (slog) with correlation IDs per execution. A `/metrics` endpoint (even before full Prometheus). `debug=true` flag for full probe payloads. Reconsider Prometheus deferral — "who watches the watcher" requires metrics. | -| A3 | **HIGH** | DB backup strategy is a one-liner — the DB is the entire control-plane state (ontology, policy, inventory, learning, ledger) | Define: `pg_dump` cadence (daily + WAL archiving for PITR), off-host storage (push to hubris or object storage, not the same mac-mini volume), encryption at rest, tested restore procedure (monthly drill), retention (30 daily + 12 monthly). This is Phase 1, not "once running." | -| A4 | MEDIUM | Seed ingest transactional safety — if API crashes mid-ingest, DB could be in partial state | Run migrations + seed ingest as a distinct init container before the API starts. Wrap each seed file in a single transaction. Add a `seed_version` table to skip already-applied seed versions. | -| A5 | MEDIUM | Config management underspecified — unclear what's env vs. DB vs. file vs. Infisical | Define a config hierarchy: defaults → file → env → Infisical (secrets only). Document each service's required config keys. Avoid putting non-secret config in Infisical. | -| A6 | MEDIUM | apps/105 fallback has no cutover safety — both stacks can't write to the same state without conflict | Define cutover mode: apps/105 is read-only during coexistence. If rollback needed after new OS has been writing to Postgres, document the reconciliation plan. | -| A7 | LOW | Notifier as a separate service is over-engineered initially | Either keep in-process (split later), or document the failure mode: if notifier is down, the API must retry/queue and the operator needs an alternative approval path. | - -### Data model - -| ID | Severity | Finding | Recommendation | -|---|---|---|---| -| D1 | **HIGH** | `entities.id` as TEXT with manual naming is fragile — renames break the ID; signal IDs require a date-string generator that's race-prone | Use UUID or SERIAL for PK. Keep `name` + `type` as a unique composite for human lookup. Signal IDs: use a DB sequence. | -| D2 | MEDIUM | Ontology flexibility sacrifices type safety — `attributes JSONB` has no schema enforcement per entity type | Add a `schema JSONB` column to `entity_types` (JSON Schema). Validate `attributes` against it on insert/update via a trigger or app-layer check. | -| D3 | MEDIUM | Entity type evolution has no migration story — renaming/removing an entity type cascades through many FK references | Add `status` (`active`/`deprecated`) to `entity_types`. Forbid hard deletes while instances exist. Provide a `merge` endpoint for renames. | -| D4 | MEDIUM | Concurrent writer safety — counters on `patterns` (evidence_count, success_count) have read-modify-write races | Use atomic `UPDATE ... SET evidence_count = evidence_count + 1`. Add `version` optimistic-lock column on `skills` and `patterns`. | -| D5 | MEDIUM | Migration rollback not addressed — only one `.down.sql` shown | Commit to forward-only migrations (compensating migrations for rollbacks) and document it, or write and test every `.down.sql`. | -| D6 | MEDIUM | Data export for DR is asserted but no endpoint or format defined | Add `GET /api/v1/export` (returns the three YAML files regenerated from DB). Test round-trip: seed → DB → export → seed → DB yields identical state. | -| D7 | LOW | `relationships` has no temporal data — can't answer "what depended on authentik last month?" | Add `valid_from`/`valid_to` (nullable = current) for `depends-on` and `hosts` edges. Low priority but cheap to add now. | - -### Operational - -| ID | Severity | Finding | Recommendation | -|---|---|---|---| -| O1 | **HIGH** | No rollback strategy — if a deploy breaks the API (bad migration, schema bug), recovery is "revert the commit" but the migration may have already run | Define: pre-deploy DB backup snapshot; migration compatibility policy (new code tolerates old schema for one deploy); tested rollback runbook per phase. | -| O2 | **HIGH** | "Who watches the watcher" is unresolved — if the OS is down, no alerts fire. No external/orthogonal monitor for the mac-mini or the OS containers | A minimal external watchdog: a cron on apps/105 (or hubris) that curls the API `/healthz` every 5 min and Matrix-pings the operator directly if it fails. This must be *outside* the Docker stack. | -| O3 | **HIGH** | No backup/restore runbook — "pg_dump" is mentioned but no procedure, tested restore, or definition of what "restored" means | Write `docs/operations/backup-restore.md`: what's backed up (DB, Infisical, Hermes volume, seed YAMLs), where, how often, how to restore each, quarterly restore drill. | -| O4 | **HIGH** | No disaster recovery plan — if the mac-mini dies (disk, theft, water), what's the RTO/RPO? | Define RTO/RPO targets (homelab: RTO 4h, RPO 24h). Name the off-host backup target. Write the DR runbook: fresh mac-mini → install Docker → clone repo → restore Infisical → restore DB → `docker compose up`. | -| O5 | MEDIUM | Deploy downtime — `docker compose up -d` recreates containers; API has a brief gap | Use `docker compose up -d --no-deps` for app services. Don't recreate the Postgres container on routine deploys (pin its image). Healthcheck-gated rollout. | -| O6 | MEDIUM | Health checks asserted but not specified | Define per-service: API `/healthz` (DB ping), scheduler "last successful probe < 15min ago", notifier "last poll < 60s ago", Hermes "gateway responding". Wire into Docker healthcheck + alerting. | -| O7 | MEDIUM | mac-mini single-host failure = total outage — disk failure, macOS update reboot, Docker daemon crash | Automated macOS update deferral/scheduling. Docker `restart: always` on all services. Monitoring heartbeat from external host. Documented cold-start runbook (what comes up first, in what order). | - -### Missing - -| ID | Severity | Finding | Recommendation | -|---|---|---|---| -| M1 | **HIGH** | No CI/CD — deploys are git-push → webhook → build. No linting, no tests before deploy, no gated merges | Add CI stage (Gitea Actions): `go vet`, `golangci-lint`, `go test ./...`, `docker build` (no push). Gate the webhook on green CI. At minimum, deploy script runs `go test` before `docker compose up`. | -| M2 | MEDIUM | No rate limiting on the API — a runaway agent loop or misconfigured skill can hammer the API and DB | Per-caller rate limiting (token bucket) on mutating endpoints. `/exec` needs a per-entity/per-action rate cap to prevent actuator loops. | -| M3 | MEDIUM | No access audit for human activity — the ledger records OS actions, but not operator API actions (entity edits, policy changes, approval decisions) | Add an `audit_log` table for all mutating REST calls, including operator identity from OIDC. | -| M4 | MEDIUM | No circuit breaker for the actuator — if a target host is unreachable, the actuator keeps attempting SSH and generating failed executions, poisoning the learning model | Circuit breaker per target: after N consecutive failures, back off (exponential) and raise a "target unreachable" signal instead of continuing to execute. | -| M5 | MEDIUM | No secret rotation story — SSH keys, Infisical machine tokens, API shared secrets need rotation policies | Define rotation cadences. Document the rotation procedure for each secret class. Add a "secrets expiring" check to the scheduler. | -| M6 | LOW | No dependency/supply-chain hygiene — Go modules, Docker base images, Infisical image not pinned or scanned | Pin base images by digest. Run `govulncheck` in CI. Periodically audit `go.sum`. Use distroless or scratch runtime images. | -| M7 | LOW | No documented SLOs — no quantitative success criteria (probe latency, API p99, deploy time, alert delivery) | Add a small SLO table: probe interval 10min ±1min, API p99 < 200ms, deploy < 5min, alert delivery < 30s. | - -### Top-priority items to address before implementation - -1. **S1 + S10** — SSH keys in containers / wide blast radius → build the actuator gateway first, not incrementally -2. **S3 + S4** — Mutable policy DB + learning model poisoning → these compound: a compromised container can rewrite its own rules and inject feedback to unlock auto-act -3. **A1 + A2 + O2** — No tests, no observability, no external watchdog → can't safely run an autonomous agent without all three -4. **A3 + O3 + O4** — Backup/restore/DR is a one-liner for the SPOF Postgres -5. **O1 + M1** — No rollback strategy and no CI gate on deploys +- Full pipeline (CI-gated webhook, SHA images, init migrate, healthcheck rollout); + Caddy re-point (`mcp.hubris.network`, `oikos.hubris.network` → mac-mini mesh + :8090); end-to-end verification below; apps/105 disabled per cutover checklist; + first monthly restore drill executed. +- **Done when:** all 14 verification checks pass; watchdog alert fires when the API + is stopped manually; rollback drill (previous SHA + pg_restore) rehearsed once. ## Verification (end to end) -1. **Ontology:** `seeds/ontology.yaml` ingested — `SELECT * FROM entity_types` shows - all types across 3 layers; lifecycle definitions match the state machine diagrams. -2. **DB:** `docker compose up postgres` — all 6 migrations applied (including - TimescaleDB extension + hypertables); seed ingest populates entities + - relationships from `inventory.yaml`. -3. **API:** `curl http://localhost:8090/api/v1/entities?type=service` returns the - fleet; MCP `list_services` works via the same endpoint. -4. **Scheduler:** trigger a probe pass — signals in DB, state snapshots written, - **metrics in `metric_samples`** (health, probe_latency_ms, disk_usage_pct). -5. **Actuator:** raise a test `service-down` signal → actuator classifies → - auto-acts (restart) or escalates (Matrix) → execution recorded → feedback generated - → **event emitted + audit entry written + correlation_id links the full chain**. -6. **Learning:** after N executions of the same (entity_type, action), a pattern - appears with confidence score; after enough evidence, a skill is created. - **`pattern_confidence` metric visible in time-series.** -7. **Classifier with learning:** set `autonomy.auto_act: reversible_low` → next - similar signal: classifier checks pattern confidence → auto-acts if high, - escalates if low. Kill-switch (`auto_act: off`) → always escalates. -8. **Hermes:** connect from another workstation → agent responds, queries API via - MCP, can SSH to hubris. **Agent activity logged to `agent_activity` table.** -9. **Secrets:** Infisical running, Go services fetch secrets, SOPS files removed. -10. **Deploy:** `git push` → Gitea webhook → `docker compose build + up -d` → - changes live, seed ingest syncs any YAML changes to DB. - **`deploy.triggered` + `deploy.completed` events in the event log.** -11. **Knowledge:** MCP `search_knowledge("caddy")` returns docs linked to - `entity:service:caddy`. -12. **Observability:** `curl /api/v1/metrics?entity_id=host:hubris&metric=disk_usage_pct` - returns time-series with trend. `curl /api/v1/audit?entity_id=service:caddy` - returns the full audit trail. `curl /api/v1/health` returns fleet summary - with trend indicators. MCP `get_trend("host:hubris", "disk_usage_pct")` - returns slope + anomaly detection. -13. **Correlation tracing:** follow a `correlation_id` from signal → classification → - execution → SSH command → verification → feedback → pattern update, all - linked in the audit_log + events table. -14. **Cutover:** stop apps/105, verify production traffic only from Docker OS. +1. **Ontology:** `SELECT * FROM entity_types` shows the 3-layer hierarchy incl. + abstract types; lifecycle defs match the diagrams; abstract instantiation is + rejected via API (422). +2. **DB:** migrations 001–006 apply idempotently; seeds ingest; re-ingest is a no-op. +3. **API:** REST + MCP return identical data for the fleet; problem+json on errors; + pagination envelope everywhere. +4. **Scheduler:** check pass writes metrics + `entity_status`; one open signal per + (entity, kind) under repeated failure. +5. **Actuator:** classify → auto-act or escalate → execute → verify → feedback, all + correlation-linked in audit + events. +6. **Learning:** after N=5 similar executions a pattern is `validated` with a + Wilson-bounded confidence; operator PATCH activates it; a skill version appears. +7. **Classifier + autonomy:** high-confidence pattern → auto-act on + `reversible_low`; kill-switch off → always escalate; `never_auto_act.` + honored. +8. **Hermes:** remote workstation session; MCP tools work; activity logged. +9. **Secrets:** services fetch from Infisical; SOPS retired (except DR key). +10. **Deploy:** push → CI → webhook → SHA image → migrate init → rolling restart; + `deploy.triggered`/`deploy.completed` events emitted. +11. **Knowledge:** `search_knowledge("caddy")` returns docs edged to + `service:caddy`. +12. **Observability:** metrics/trends/audit/health endpoints return correct shapes; + Grafana can read `metric_samples` directly. +13. **Correlation tracing:** one `correlation_id` reconstructs signal → + classification → execution → SSH → verification → feedback. +14. **Cutover:** apps/105 stopped; watchdog still alive; production traffic served + solely by the Docker stack. + +## Risks / trade-offs + +- **Go rewrite** (~4,400 Python lines replaced) — logic carries over 1:1 (see reuse + table); mitigated by the phase gates and the contract-first spec. +- **Postgres SPOF** — accepted; mitigated by backups/PITR/DR drills; replication is + the future path. +- **Learning cold start** — by design: the agent escalates everything until patterns + validate and the operator activates them; trust is earned. +- **Single-host mac-mini** — watchdog + pmset hardening + documented cold start; + multi-node path exists when it matters. +- **Infisical bootstrap** — SOPS fallback retained until a restore drill passes. + +### Reuse map (Python → Go) + +| Existing | Becomes | +|---|---| +| `oikos/decide.py` | `internal/policy/classify.go` (+ pattern/skill inputs) | +| `oikos/signal.py` | `internal/scheduler` signal lifecycle (DB-backed, dedup added) | +| `oikos/approve.py` | `internal/policy/approve.go` (tokens) + `internal/notifier/matrix.go` | +| `oikos/ledger.py` | `executions` + `ledger` view + `audit_log` | +| `oikos/policy.py` + `policy.yaml` | `internal/policy` + `seeds/policy.yaml` → tables | +| `oikos/drift.py` | `drift` check kind | +| `oikos/relations.py` | `internal/ontology/graph.go` (SQL traversal) | +| `oikos/report.py` | report endpoints over DB | +| `mcp/server.py` | `internal/mcp` (official SDK) | +| `bin/homelab` | generated OpenAPI Go client | +| `oikos/scheduler.py` | `internal/scheduler` (goroutine worker pool, checks-as-data) | +| *(new)* | `internal/learning`, `internal/observability`, `internal/httpapi` | ## Out of scope (for now) -- Oikos Console web UI (deferred — built later on top of the API) -- Multi-node deployment (designed for, not implemented) -- Vector embeddings / semantic search (structured graph only for now) -- SSH-key-signed approval requests -- Actuator gateway pattern (Phase 2 of hybrid — start with restricted SSH key, - build gateway in Phase 3 alongside the actuator) -- Automated skill extraction via LLM (patterns extracted statistically for now; - LLM-assisted skill refinement is a future enhancement) - -## Audit remediation — HIGH priority + architect/developer review - -This section addresses all HIGH-severity audit findings and the 18 new findings -from the systems architect + senior Go developer review. Each fix references the -finding ID. - -### Schema fixes (CRITICAL/HIGH) - -**SA1 + SG1 — Cognition entities + broken FKs + hypertable PKs:** - -The core issue: the BDD models cognition objects (Signal, Execution, Feedback, -Pattern, Skill, Classification) as graph entities with typed edges, but the schema -only makes Signal a dual entity. `executions.skill_id REFERENCES entities(id)` is a -broken FK (skills live in `skills(id)`, not `entities(id)`). Additionally, hypertable -PKs (`id BIGSERIAL PRIMARY KEY`) don't include the time column — TimescaleDB rejects -this. - -**Resolution — dual entity pattern for all cognition objects:** - -Every cognition object gets an `entities` row (type = `signal`, `execution`, etc.) -AND a typed table for indexed querying. The typed table's PK references -`entities(id)`. Graph relationships (triggers, produces, contributes-to) live in the -`relationships` table, making the cognition loop traversable as a graph. - -```sql --- Fix: hypertable PKs must include the time column -CREATE TABLE audit_log ( - id BIGSERIAL, - ts TIMESTAMPTZ NOT NULL DEFAULT now(), - -- ... columns ... - PRIMARY KEY (id, ts) -- was: id BIGSERIAL PRIMARY KEY -); --- Same fix for events and agent_activity: PRIMARY KEY (id, ts) - --- Fix: executions.skill_id → references skills, not entities --- (or if using dual-entity pattern, reference entities(id) where type='skill') -CREATE TABLE executions ( - id SERIAL PRIMARY KEY, - -- ... existing columns ... - skill_id TEXT, -- entity ID of skill used - skill_version INTEGER, -- SG9: snapshot version at exec time - -- ... -); - --- Fix: skills versioning preserves history (SG9) --- Change PK to composite so old versions remain queryable -CREATE TABLE skills ( - id TEXT NOT NULL, -- 'skill-restart-service' - version INTEGER NOT NULL DEFAULT 1, - name TEXT NOT NULL, - ts TIMESTAMPTZ DEFAULT now(), - procedure TEXT NOT NULL, - applies_to TEXT REFERENCES entity_types(name), - pattern_ids TEXT[], - status TEXT DEFAULT 'drafted', - success_rate REAL, - last_used_at TIMESTAMPTZ, - PRIMARY KEY (id, version) -- was: id TEXT PRIMARY KEY -); -CREATE TABLE skill_versions_audit ( -- SG9: track procedure changes - id SERIAL PRIMARY KEY, - skill_id TEXT NOT NULL, - version INTEGER NOT NULL, - changed_at TIMESTAMPTZ DEFAULT now(), - changed_by TEXT, -- agent or operator entity - diff TEXT, -- diff of procedure field - reason TEXT -); -``` - -**SA5 — Classification entity must be persisted:** - -Add a `classifications` table. Every classifier decision is recorded with the full -reasoning — this is the audit trail for autonomous decisions. - -```sql -CREATE TABLE classifications ( - id SERIAL PRIMARY KEY, - ts TIMESTAMPTZ DEFAULT now(), - signal_entity_id TEXT REFERENCES entities(id), - entity_id TEXT REFERENCES entities(id), - action TEXT NOT NULL, - risk_class TEXT NOT NULL, - route TEXT NOT NULL, -- 'auto-act' or 'escalate' - blast_radius TEXT[], - pattern_confidence REAL, - skill_match TEXT, -- skill entity ID if matched - autonomy_check TEXT, -- 'allowed' or 'blocked: ' - reasoning JSONB NOT NULL, -- full decision explanation - correlation_id TEXT -); -CREATE INDEX idx_class_signal ON classifications(signal_entity_id); -CREATE INDEX idx_class_entity ON classifications(entity_id); -``` - -**SG2 — CAGG array_agg unsupported:** - -Drop `representative_tags` from the 1h continuous aggregate. Query tags from raw -data when needed. - -```sql -CREATE MATERIALIZED VIEW metric_rollups_1h -WITH (timescaledb.continuous) AS - SELECT - time_bucket('1 hour', ts) AS bucket, - entity_id, - metric, - avg(value) AS avg_value, - min(value) AS min_value, - max(value) AS max_value, - count(*) AS sample_count - -- removed: (array_agg(tags))[1] AS representative_tags - FROM metric_samples - GROUP BY bucket, entity_id, metric; -``` - -**SG3 — TimescaleDB functions not idempotent:** - -Use `if_not_exists => TRUE` and exception guards: - -```sql -SELECT create_hypertable('metric_samples', 'ts', - chunk_time_interval => INTERVAL '7 days', - if_not_exists => TRUE); - -DO $$ BEGIN - PERFORM add_retention_policy('metric_samples', INTERVAL '90 days'); -EXCEPTION WHEN OTHERS THEN NULL; -END $$; -``` - -### Ontology fixes (HIGH) - -**SA2 — Layer boundary "cognition creates governance" arrow:** - -Remove the unsupported arrow. The learning engine does NOT write to governance -(policy/autonomy) tables. Instead, validated patterns *propose* autonomy changes as -approval-request entities — the operator must accept. This keeps governance -authoritative and prevents the learning-poisoning vector (S4). - -Also rename the cognition subgraph node from "Governance" to "Approvals" to avoid -the layer-name collision. - -**SA3 — Missing entity types (Person, Agent, IdentityProvider):** - -Add BDD definitions and entity_type seeds: - -``` -Person: matrix_id, oidc_sub -Agent: provider, model, gateway_port -IdentityProvider: issuer, client_id, auth_mode (oidc/forward-auth) -``` - -These are first-class governance-layer entities. `audit_log.actor_id` now resolves -to a real entity. - -**SA4 — Lifecycle dead-ends and missing transitions:** - -Updated infrastructure lifecycle (add `failed` state): -``` -planned → provisioning → active → migrating → deprecated → destroyed - ↘ failed ↗ ↘ failed ↗ -failed → active (recovery) | failed → deprecated (write-off) -planned → destroyed (cancel) -deprecated → active (un-deprecate if replacement fails) -``` - -Updated signal lifecycle (add terminal states): -``` -raised → acknowledged → acting → resolved | failed -acknowledged → resolved (manual resolve without acting) -acknowledged → muted -acting → failed (permanent failure, terminal — needs operator) -failed → acknowledged (operator retries) -``` - -Updated execution lifecycle (add cancellation + rollback failure): -``` -approved → expired (TTL ran out) -executing → cancelled (operator abort) -failed → rolled_back | rollback_failed -timed_out → verifying (check if the command actually completed despite timeout) -``` - -Updated pattern lifecycle: -``` -hypothesized → validated → active → deprecated -hypothesized → invalidated (disproven, terminal) -active → invalidated (new evidence contradicts) -``` - -Updated skill lifecycle: -``` -drafted → tested → active → refined → active (new version) -drafted → deprecated (abandoned) -tested → failed → drafted (back to drawing board) -active → deprecated (superseded or unsafe) -``` - -Approval lifecycle (new diagram — was missing): -``` -pending → approved | denied | expired -approved → revoked (operator changes mind before execution) -``` - -**SA6 — recommended_action data source:** - -Move `recommended_action` from `signals` to `classifications`. The probe raises a -signal (kind, severity, evidence). The classifier populates the classification with -the recommended action based on signal kind + skill lookup. The actuator reads the -classification, not the signal, for the action to take. - -**SA8 — Missing entities (Cluster, ComposeStack, ManagedHost):** - -Add to ontology: -- `Cluster` entity (software domain) — `ProxmoxHost` `member-of` `Cluster` -- `ComposeStack` entity (software domain) — `DockerContainer` `part-of` `ComposeStack` -- Add `provider`, `control_level` (`full`/`partial`/`none`) to `StandaloneServer` - -### Security remediation (HIGH) - -**S1 — SSH keys in containers:** - -Dual approach (operator decision): -1. **Phase 1-2 (immediate):** Restricted SSH key — dedicated key pair with - `command="..."` and `from="..."` restrictions in `authorized_keys` on hubris/ - strong. The key can only run specific commands (pct, qm, df, systemctl status), - not arbitrary shells. Mounted read-only into the actuator container only (not - Hermes). -2. **Phase 3 (actuator build):** Full actuator gateway — the API's `/exec` endpoint - brokers all SSH. The actuator holds the keys, executes per-action, logs every - command. Hermes never touches SSH. - -**S2 — MCP auth:** -- Shared secret between API and Hermes (HMAC-signed requests) -- MCP bound to a dedicated Docker network (not the default bridge) -- Never exposed via Caddy without auth - -**S3 — Policy DB mutability:** -- Policy mutations (`risk_classes`, `approval_rules`, `autonomy_settings`) require a - meta-approval: the operator must approve the policy change itself (dual-control) -- Immutable audit log of all policy changes with before/after hash -- Startup self-check: compute policy hash, alert if differs from last-known-good - -**S4 — Learning model poisoning:** -- Pattern transitions to `active` require operator confirmation (`PATCH /api/v1/ - patterns/{id}` with `status=active` — policy-gated as `config_mutation`) -- Confidence capped by sample size: `confidence = min(raw_confidence, N/5)` where N - = evidence_count (requires N≥5 for confidence > 0.2) -- Anomaly detection: if >10 identical-outcome feedback entries arrive within 1 hour - for the same (entity_type, action), quarantine the pattern for review -- Skills can never auto-promote to destructive risk class — always escalate - -**S5 — confirmation_phrase replacement:** -- Replace with single-use signed approval tokens -- Token = HMAC(approval_id + entity_id + action + risk_class + nonce, shared_secret) -- Stored hashed in `approvals` table -- Transmitted via Matrix as the approval ID + decision; token verified server-side - -**SA10 — Gateway + Caddy trust boundary:** -- Port 8092 (Hermes gateway): mTLS or token auth. Mesh membership is the network - boundary; gateway auth is the application boundary. -- Caddy is an explicit trust root. API validates OIDC JWTs in middleware (not just - trusting Caddy headers). Documented: compromising Caddy ≠ compromising the API. - -### Operational remediation (HIGH) - -**A3 + O3 + O4 — Backup, restore, DR:** - -Backup strategy: -- **Daily `pg_dump`** (compressed, custom format) + WAL archiving for PITR -- **Off-host storage:** Proton Drive (cloud object storage, encrypted at rest) -- **Push via rclone** from the scheduler container (already have rclone LXC in the - fleet — reuse credentials) -- **Retention:** 30 daily + 12 monthly snapshots -- **Infisical backup:** Infisical has its own backup mechanism; also export secrets - to an encrypted SOPS file as a fallback (chicken-and-egg: keep one age key for - this purpose) -- **Hermes volume:** backed up with `pg_dump` of agent_activity + session data - -Restore procedure (`docs/operations/backup-restore.md`): -1. Restore Postgres: `pg_restore -d oikos < dump.psql` -2. Verify seed ingest matches (run `GET /api/v1/export` and diff against seed YAML) -3. Restore Infisical from its backup -4. `docker compose up -d` -5. Monthly restore drill (scheduled, automated, alert if restore fails) - -DR plan: -- **RTO:** 4 hours (fresh machine → Docker → restore → running) -- **RPO:** 24 hours (last daily backup) -- **Cold-start runbook:** install Docker → clone repo → restore Infisical → - restore DB → `docker compose up -d` → verify health -- **Off-host backup target:** Proton Drive (encrypted, offsite) - -**O1 — Rollback strategy:** -- **Forward-only migrations** (no `down.sql` beyond development). Compensating - migrations for production rollbacks. -- **Pre-deploy DB backup:** the deploy script runs `pg_dump` before `docker compose - up -d` -- **Migration compatibility:** new code must tolerate old schema for one deploy - window (additive migrations only — new columns nullable, new tables optional) -- **Rollback runbook:** revert git commit → `pg_restore` from pre-deploy backup → - `docker compose up -d` with old image - -**O2 — External watchdog:** -- Cron job on apps/105 (outside the Docker stack): `curl -sf - http://mac-mini:8090/healthz || curl -X POST matrix-webhook ...` -- Runs every 5 minutes -- Alerts operator directly via Matrix if the API is unreachable -- Also checks: Docker daemon running (`docker info`), Postgres accepting - connections (`pg_isready`) - -**M1 — CI/CD:** -- Gitea Actions (or simple webhook + script): - - `go vet ./...` - - `golangci-lint run` - - `go test ./... -race -cover` - - `docker build` (no push — just verify it builds) -- Webhook deploy gated on green CI -- Deploy script runs `go test` as a final safety check before `docker compose up` - -### Architecture remediation (HIGH) - -**A1 — Testing strategy:** - -Add testing workstream with specific tests: - -| Package | Test type | What to test | -|---|---|---| -| `internal/policy/` | Unit | Classifier scoring: risk × blast × confidence. Table-driven: every (risk_class, blast_radius, confidence) combination. Edge: unknown action, ambiguous entity. | -| `internal/policy/` | Unit | Approval lifecycle: token issue, verify, single-use enforcement, TTL expiry. | -| `internal/learning/` | Unit | Pattern confidence calculation. Skill versioning. Feedback → pattern extraction. | -| `internal/ontology/` | Unit | Lifecycle transition validation: every legal transition succeeds, every illegal one fails. Graph traversal (mock relationships). | -| `internal/db/` | Integration | testcontainers Postgres: seed ingest idempotency, blast_radius CTE (with cycles), hypertable insert + query, continuous aggregate refresh. | -| `internal/api/` | Integration | testcontainers: REST routes return correct status codes, MCP tools return expected shapes, audit middleware records entries, pagination works. | -| `internal/actuator/` | Integration | Mock SSH: execution → verification → feedback → pattern update. Loop-guard prevents retry storms. Circuit breaker trips after N failures. | -| `migrations/` | Property | Every migration is forward-only. `blast_radius` returns correct results on cyclic graphs. Hypertable retention doesn't drop data younger than threshold. | -| `internal/observability/` | Unit | Correlation ID propagation through context. Event emitter transactional with state change. Metric recording. | - -Coverage gate: ≥80% on `internal/policy/` and `internal/learning/` (the autonomy- -granting code). ≥60% on everything else. - -**A2 — Observability:** Already addressed (Migration 6, Workstream 14). Update audit -status to resolved. - -**SG4 — Graceful shutdown:** - -Every `cmd/*/main.go` implements: -1. `signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)` -2. Context propagated to all long-running loops and HTTP server -3. Shutdown sequence: stop accepting new work → wait for in-flight (30s deadline) → - for actuator: if execution in-flight, mark `failed` with "shutdown interrupted" + - emit feedback → close DB pool -4. `docker-compose.yml`: `stop_grace_period: 30s` on actuator, `stop_signal: SIGTERM` - on all services - -**SG5 — Entity-level concurrency:** - -```sql --- Per-entity advisory lock during execution (prevents concurrent actions --- on the same target entity, e.g., restart + deploy on the same service) -SELECT pg_advisory_xact_lock(hashtext($1)); -- $1 = target_entity_id --- ... execute, verify, feedback ... --- lock released on transaction commit/rollback -``` - -**SG6 — Domain layer:** - -Add `internal/domain/` package: -``` -internal/domain/ -├── entity.go # Entity, EntityType, Relationship domain types -├── signal.go # Signal domain type + lifecycle transition logic -├── execution.go # Execution domain type + state machine -├── classification.go # Classification domain type -├── pattern.go # Pattern domain type + confidence calculation -├── skill.go # Skill domain type + versioning -├── approval.go # Approval domain type + token verification -└── errors.go # Sentinel errors: ErrNotFound, ErrInvalidTransition, - # ErrApprovalRequired, ErrAutonomyBlocked, ErrConflict -``` - -DB ↔ domain mapping in `internal/db/` (repository pattern). API handlers accept/ -return domain types. sqlc models never escape `internal/db/`. - -**SG11 — Error handling:** - -```go -// internal/domain/errors.go -var ( - ErrNotFound = errors.New("entity not found") - ErrInvalidTransition = errors.New("invalid lifecycle transition") - ErrApprovalRequired = errors.New("operator approval required") - ErrAutonomyBlocked = errors.New("autonomy policy blocks this action") - ErrConflict = errors.New("concurrent modification conflict") - ErrCircuitOpen = errors.New("circuit breaker open for target") -) -``` - -HTTP mapping middleware: `ErrNotFound → 404`, `ErrInvalidTransition → 409`, -`ErrApprovalRequired → 403`, `ErrAutonomyBlocked → 403`, `ErrConflict → 409`, -`ErrCircuitOpen → 503`. - -SSH error classification in `internal/actuator/execute.go`: -- Network unreachable → retryable, circuit breaker -- Auth failure → fatal, alert operator -- Command exit non-zero → execution failed, feedback -- Command timeout → timed_out, feedback - -DB retry: serialization failures (SQLSTATE 40001, 40P01) → retry with exponential -backoff (max 3 retries). - -### Go implementation fixes (MEDIUM) - -**SA9 — TimescaleDB Docker image + migration runner:** -- Image: `timescale/timescaledb:2.x-pg16` (not `postgres:16`) -- Migrations run in a one-shot init container (`compose/migrate/Dockerfile`) with a - dedicated DB user that has DDL but no runtime data privileges -- API's DB user gets DML only (least privilege) -- `golang-migrate` Go API with `//go:embed migrations/*.up.sql` - -**SG7 — Pattern/skill management endpoints:** -- `PATCH /api/v1/patterns/{id}` — state transition (validate/invalidate/deprecate), - policy-gated as `config_mutation`, audit-logged -- `PATCH /api/v1/skills/{id}` — state transition + version pin, same gating -- This is the operator's manual safety valve for learning-model issues (S4) - -**SG8 — WebSocket push mechanism:** -- In-process event bus (Go channel pub/sub) for events written by the API itself - (zero-latency push to WebSocket subscribers) -- Postgres `LISTEN/NOTIFY` for events written by other services (scheduler, - actuator) — trigger on `events` table fires NOTIFY after commit -- Both feed the WebSocket handler - -**SG10 — Transactional event emission:** -- Event + audit entries written in the same DB transaction as the state change -- If transaction rolls back, events are discarded (never emitted) -- `LISTEN/NOTIFY` fires after commit — subscribers only see committed events - -**SG13 — Context-aware SSH:** -```go -func runSSH(ctx context.Context, client *ssh.Client, cmd string) ([]byte, error) { - session, err := client.NewSession() - if err != nil { return nil, err } - defer session.Close() - type result struct { out []byte; err error } - ch := make(chan result, 1) - go func() { out, err := session.CombinedOutput(cmd); ch <- result{out, err} }() - select { - case r := <-ch: return r.out, r.err - case <-ctx.Done(): - session.Close() // unblocks CombinedOutput - client.Close() - return nil, ctx.Err() - } -} -``` - -**SG14 — Connection pool sizing:** -- API: 15 connections, scheduler: 5, actuator: 5, learning: 3 = 28 total -- Postgres `max_connections` set to 80 -- Monitor `db_connections_active`, alert if >80% of pool - -**SG15 — RESTful exec endpoint:** -- `POST /api/v1/executions` (was `POST /api/v1/exec`) — creates an execution - resource. Handler classifies, checks approval, creates execution in `proposed` - state. -- `GET /api/v1/executions/{id}` — status -- `POST /api/v1/executions/{id}/cancel` — cancellation - -**SG16 — Pagination:** -- All hypertable-backed endpoints: cursor-based (`?cursor=&limit=50`) -- `entities` and other small tables: keyset pagination (`?after=&limit=50`) -- Default limit: 50, max: 200 -- MCP tools support `limit` parameter - -**SG17 — Go tooling:** -- `sqlc.yaml` added to repo layout -- Module path: `github.com/dtoro/oikos` -- `CGO_ENABLED=0` in Dockerfiles, `pgx` (pure Go), `gcr.io/distroless/static` - runtime image -- Migrations embedded with `//go:embed` - -**SG18 — Health/metrics bypass auth:** -- `/healthz` and `/metrics` on a separate Gin router group, no auth, no audit -- `/healthz`: `SELECT 1` against DB -- `/metrics`: internal-only (not exposed via Caddy), or token-protected - -**SA7 — Notifier decoupling:** -- API writes `approvals` row (status=pending) + emits `approval.requested` event -- Notifier polls pending approvals, sends to Matrix, writes decision directly to - `approvals` table (has DB access, not API access) -- API polls `approvals.status` -- No service-to-service calls in either direction — DB is the rendezvous point -- Pending approvals survive Notifier restart - -**SA9 + SA10 — Threat model documentation:** -- Caddy is an explicit trust root (compromise = API compromise, mitigated by JWT - validation in API middleware) -- Mesh membership is the network boundary for Hermes gateway -- Docker network is the trust boundary for internal services (mTLS between API and - Hermes, TLS to Postgres) -- The actuator is the only container with SSH egress (not Hermes, not the API) - -### Updated phasing (incorporating remediation) - -**Phase 0 — Ontology design (no code):** -- Finalize entity types (including Person, Agent, IdentityProvider, Cluster, - ComposeStack) + relationship types + lifecycles (all with terminal states) -- Write seed manifests -- Review diagrams + lifecycle completeness with operator - -**Phase 1 — Foundation (Go + DB):** -- Go module setup (`github.com/dtoro/oikos`), project structure with domain layer -- PostgreSQL + TimescaleDB (`timescale/timescaledb:2.x-pg16`) -- Migrations 1-6 (with fixed hypertable PKs, idempotent TimescaleDB calls, dual - entity pattern for cognition objects, classifications table) -- Migration runner as init container (DDL-only DB user) -- Seed ingest pipeline (transactional, init container) -- sqlc + domain layer (repository pattern) -- Structured logging (slog) + error sentinels + HTTP error mapping -- **Testing foundation:** testcontainers setup, unit test framework, coverage gates -- **Backup setup:** daily `pg_dump` + rclone push to Proton Drive, WAL archiving -- **External watchdog:** cron on apps/105 - -**Phase 2 — API (Go):** -- Gin server with REST routes (resource-oriented, paginated, `/executions` not - `/exec`) -- MCP protocol adapter (shared-secret auth, dedicated network) -- Policy enforcement middleware (OIDC JWT validation, not just Caddy headers) -- Audit middleware (transactional with state changes) -- Event emitter (transactional, in-process bus + LISTEN/NOTIFY) -- Observability routes (metrics, trends, audit, events, health, agent-activity) -- WebSocket (in-process bus + LISTEN/NOTIFY, bounded channels, backpressure) -- Pattern/skill management endpoints (operator override for learning model) -- Domain layer fully fleshed out -- **CI:** Gitea Actions (go vet, golangci-lint, go test -race -cover, docker build) - -**Phase 3 — Control loop (Go):** -- Scheduler (Observe) — probes, signals, state snapshots, metric recording -- Actuator (Act) — classify, execute, verify, correlation ID propagation - - **Restricted SSH key** (command= in authorized_keys, actuator-only) - - **Entity-level advisory locks** (pg_advisory_xact_lock) - - **Context-aware SSH** (context cancellation, hard timeout) - - **Circuit breaker** per target host - - **Graceful shutdown** (SIGTERM, in-flight protection, stop_grace_period: 30s) -- Learning engine — feedback, patterns, skills, learning metrics - - **Pattern activation requires operator confirmation** (PATCH endpoint) - - **Confidence capped by sample size** (N≥5) - - **Anomaly detection** for feedback bursts -- Approval tokens (single-use HMAC, not confirmation phrases) -- Policy meta-approval (dual-control for policy mutations) -- Notifier (DB rendezvous, no service-to-service calls, Matrix impl) - -**Phase 4 — Agent (Hermes container):** -- Hermes Docker image, gateway config (mTLS on port 8092) -- Homelab skills -- Agent activity logging -- Connect from workstation, verify MCP (shared-secret auth) + SSH via actuator - -**Phase 5 — Secrets (Infisical):** -- Stand up Infisical, migrate SOPS, wire services -- Infisical backup + SOPS fallback (one age key kept for DR) - -**Phase 6 — Deploy + cutover:** -- Docker Compose (timescale image, init containers, pool sizing, stop_grace_period) -- Gitea webhook (HMAC auth, non-root deploy user, CI-gated) -- Caddy re-point + JWT validation in API -- End-to-end verification (14 checks) -- Stop apps/105, clean up mac-mini -- **Restore drill** (monthly, automated) +- Web UI (the OpenAPI contract + SSE + `/graph` endpoint are built for it; the UI + itself comes later). +- Multi-node deployment (designed for; see "Multi-node path"). +- Vector embeddings / semantic search (Postgres FTS now; `pgvector` is a schema-only + addition later). +- WebSocket stream (SSE covers current needs). +- LLM-assisted skill extraction (patterns are statistical for now). + +--- + +## Appendix A — audit resolution ledger + +Every rev-2 finding and where rev 3 resolves it. (Full finding text in git history, +rev 2 of this file.) + +| Finding | Resolution | +|---|---| +| S1, S10 | Security model — restricted SSH key in actuator only; network trust zones; Hermes keyless | +| S2 | MCP bearer token + dedicated network (API contract / Security) | +| S3 | Policy dual-control + hash audit + startup self-check (005 / Security) | +| S4 | Operator-gated pattern activation, Wilson + N/5 cap, quarantine, no destructive auto-promotion (Learning engine) | +| S5 | Single-use HMAC approval tokens, hashed (003) | +| S6, SA10 | In-API OIDC JWT validation; Caddy = explicit trust root (AuthN/AuthZ) | +| S7 | TLS to Postgres; per-role DB users (Security) | +| S8 | HMAC webhook, non-root deploy, CI gate (Deploy) | +| S9 | Infisical bootstrap root of trust + SOPS DR fallback (Security / Phase 5) | +| P1 | Cycle-safe `blast_radius` with path accumulator + depth cap (002) | +| P2 | Bounded worker pool, jitter, timeouts (Scheduler) | +| P3 | Content-hash skip on knowledge ingestion (Phase 2) | +| P4 | Hourly watermark-based pattern extraction + `feedback(created_at)` index (004 / Learning) | +| P5 | `entity_status` replaces `state_snapshots`; history in retained metrics (R3-6) | +| P6 | SSE bounded buffers, drop-oldest, heartbeats (Observability) | +| P7 | Docker-only builds; no host `go build` in deploy (Deploy) | +| A1 | Testing strategy embedded in phase gates + CI coverage gates | +| A2 | Observability section + migration 006 | +| A3, O3, O4 | Backup/restore/DR section with drills, RTO/RPO | +| A4 | Init-container migrate/seed, per-file transactions, `seed_versions` | +| A5 | Config hierarchy (DB-native configuration) | +| A6 | Read-only coexistence + cutover/rollback plan | +| A7, SA7 | Notifier via DB rendezvous | +| D1 | UUIDv7 + slug (R3-5) | +| D2 | `attribute_schema` JSON Schema validation | +| D3 | entity_type `status`, deprecate-not-delete, ancestor-aware rules | +| D4 | Atomic counter updates + `version` optimistic locks | +| D5, O1 | Forward-only migrations + pre-deploy dump + rollback runbook | +| D6 | `GET /api/v1/export` + byte-stable round-trip test | +| D7 | `valid_from`/`valid_to` on relationships | +| O2 | External watchdog cron on apps/105 | +| O5 | `--no-deps` rollout, pinned Postgres container | +| O6 | Per-role health checks wired to compose + watchdog | +| O7 | pmset hardening, OrbStack autostart, update scheduling (R3-13) | +| M1 | Gitea Actions CI, gated webhook | +| M2 | Per-actor token bucket + `/executions` budget + actuator cooldowns | +| M3 | `audit_log` covers operator REST mutations with OIDC identity | +| M4 | Per-target circuit breaker | +| M5 | Rotation cadences + expiry signals | +| M6 | Digest-pinned images, `govulncheck`, distroless | +| M7 | SLO table (Observability) | +| SA1, SG1 | Dual-entity pattern for all cognition objects; hypertable PKs include `ts` | +| SA2 | Learning cannot write governance; proposes via approvals (Layer map) | +| SA3 | Person/Agent/IdentityProvider entity types (Governance BDD) | +| SA4 | All lifecycles have terminal states + recovery paths | +| SA5 | `classifications` table (004) | +| SA6 | `recommended_action` on classification, not signal | +| SA8 | Cluster, ComposeStack, StandaloneServer attrs (BDD) | +| SA9 | timescale image, init-container migrations, DDL/DML user split | +| SG2, SG3 | CAGGs without `array_agg`; idempotent TimescaleDB DDL | +| SG4 | Graceful shutdown spec (Actuator) | +| SG5 | Per-entity advisory locks | +| SG6 | `internal/domain` + repository pattern (Repo layout) | +| SG7 | Pattern/skill PATCH endpoints (API) | +| SG8, SG10 | Transactional events + post-commit NOTIFY + in-process bus | +| SG9 | Skill versions as composite PK + change metadata | +| SG11 | Sentinel errors + problem+json mapping | +| SG13 | Context-aware SSH | +| SG14 | Pool sizing: api 15 / scheduler 5 / actuator 5 / learning 3; `max_connections=80`; alert at 80% | +| SG15 | `POST /api/v1/executions` resource style | +| SG16 | Cursor pagination + envelope | +| SG17 | sqlc.yaml, module path, CGO off, pgx, distroless, embedded migrations | +| SG18 | Unauthenticated internal-only `/healthz` + `/metrics` | + +## Appendix B — initial ADRs to write (Phase 0) + +0001 Go + single-binary role packaging · 0002 Postgres+TimescaleDB as the only +datastore · 0003 DB-native ontology with YAML seeds · 0004 OpenAPI-first API · +0005 UUIDv7 + slug identity · 0006 learning is proposal-only (no self-authorization) +· 0007 threat model + trust zones · 0008 forward-only migrations · 0009 SSE over +WebSocket · 0010 Infisical with SOPS DR fallback.