Files
oikos/OIKOS.md
dtoro 2084a1583e Oikos Week 3: scheduler, drift detectors, signals, classifier, approvals
New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:

- oikos/scheduler.py — Observe stage: HTTP health probes for every
  service, disk-usage probes on hubris/strong, writes oikos/state.json
  (gitignored — regenerates every run). `homelab service <name> health`
  is now cache-first; `--live` forces a fresh probe. Deploys via
  oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.

- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
  detectors (fully local, no SSH) plus pct-list and Caddy-backend
  detectors (best-effort SSH, degrade to an info finding when
  unreachable rather than a false drift alarm). Found real, currently-
  true drift on first run: republic-laptop's age key granted on every
  secret but missing from inventory.yaml, grimmory missing from
  hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
  recorded in OIKOS.md for the operator, not auto-fixed (each is a
  config_mutation/destructive decision).

- oikos/signal.py — the attention layer: raised -> acknowledged ->
  acting -> resolved|muted lifecycle, severity-based routing, dedup via
  open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.

- oikos/decide.py — the Decide-stage classifier: risk class x blast
  radius x ledger-history confidence -> auto-act/escalate. Adds an
  action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
  service_name from the entity for per-service policy overrides.
  `homelab decide <action> <entity>`.

- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
  in this homelab, so this is the repo-side half only: request/reply/
  grant lifecycle with short-TTL HMAC-signed tokens (new secret
  secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
  delivery is Hermes's existing @dtoro:avispero send path (documented
  integration contract in the module docstring), not a new bot.
  `homelab restart` now mechanically refuses config_mutation/destructive
  services without a valid --approval-id, regardless of -y/interactivity.

- oikos/report.py — daily brief + weekly report from signal/approval/
  ledger state (no Prometheus yet, so point-in-time counts only).

- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
  not provisioned: no pve_id is guessed here since Proxmox assigns real
  IDs at creation time, and drift already found an unclaimed ID (131) to
  investigate first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 23:29:39 +02:00

7.3 KiB
Raw Blame History

Oikos — the operating model

Oikos (Greek: household) is the agent operating system layered on this repo. It is not new infrastructure: inventory.yaml is the kernel data structure, the homelab CLI and MCP server are the syscall surface, and this page defines the rules everything above them follows.

Read this after AGENTS.md. Machine-readable companions: oikos/ontology.yaml (systems model), oikos/policy.yaml (risk & approval).

The kernel loop: OODA

Every Oikos activity — scheduled probe, agent task, operator request — is one pass through Observe → Orient → Decide → Act:

  1. Observe — probes, drift detectors, and agent findings produce Signals (structured records, not loose messages): pending updates, high temperature, low disk, service down, cert expiry, stale backup, inventory drift.
  2. Orient — walk the ontology graph: what entity is affected, what depends on it (blast radius), its lifecycle state, whether a runbook matches, what the ledger says about past attempts.
  3. Decide — the classifier scores risk class × blast radius × confidence and routes:
    • auto-act: within autonomy policy, high confidence, contained radius
    • escalate: operator approval via Matrix (/ reaction; destructive actions additionally need a typed confirmation phrase)
    • queue: informational — console + reports The classifier can only lower autonomy relative to policy, never raise it. When in doubt, escalate.
  4. Act — execute through homelab commands or runbooks (never ad-hoc SSH), then verify with the action's verification command, write a ledger entry, resolve the Signal, and update docs in the same session.

Primitives

Primitive What it is Lives in
Host / Service topology entities inventory.yaml (+ generated hosts/*.yaml)
Secret SOPS+age encrypted value, per-client recipients secrets/ + .sops.yaml
Runbook executable workflow with risk class + verification runbooks/ (Week 2)
Signal something needing attention, with lifecycle signals/ ledger (Week 3)
Change one mutation: who, what, risk, approval, verification ledger/ (Week 2)
Approval short-TTL signed grant for a gated action approval engine (Week 3)
Incident investigation narrative investigations/
Plan design doc for non-trivial work plans/
Agent enrolled client identity = its age pubkey inventory.yaml + .sops.yaml

Risk classes (enforced, not advisory)

From oikos/policy.yaml:

  • read_only — status, logs, docs, inventory. Unattended.
  • reversible_low — restart, cache clear, sync pull. Unattended + ledger.
  • config_mutation — tracked-config edits (commit+push, never local), deploys, upgrades, DNS/ingress changes. Operator approval.
  • destructive — destroy, format, wipe, rotate, revoke. Approval + typed confirmation phrase.

Lifecycle gates modify these: provisioning nodes are freely mutable (nothing depends on them); deprecated nodes accept no new dependents; anything touching a destroyed node is drift.

The systems model

Eight domains — physical, compute, network, storage, software, identity & access, operations, external — cover everything in the lab; entities are connected by typed edges (hosts, provides, mounts, stores-on, routes-to, can-decrypt, depends-on, backs-up-to, …) defined in oikos/ontology.yaml. Rule of completeness: if it can break, be changed, or hold data, it has an entity and edges. Blast-radius questions ("what breaks if strong goes down?") are graph walks, not doc archaeology.

Nodes move through an explicit lifecycle — planned → provisioning → active → migrating → deprecated → destroyed — stored as state: in inventory (absent = active). Destroyed nodes live in the archaeology: section. Each transition is a runbook checklist; deprecation completes only when inbound edges reach zero.

Generated views: infrastructure/topology.md (Mermaid, regenerated from inventory).

Conventions carried forward

  • Inventory is the truth; live state wins over narrative docs.
  • Prefer homelab CLI and MCP over ad-hoc SSH.
  • Meaningful changes update docs in the same session.
  • Secrets are decrypted locally via per-client keys; never into docs/comments.
  • Tracked configs change by commit + push, not local edits.
  • Netbird is the preferred mesh path for new traffic.
  • Agents are terse (CAVEMAN.md), verify claims, and fix collateral drift when found.

Build status (30-day roadmap, started 2026-07-05)

  • Week 1: policy, ontology, service contract, archaeology, topology generator, this brief. Shipped.
  • Week 2: context cards, homelab service <name> …, change ledger, node relations, runbooks. Shipped.
  • Week 3: ops scheduler + state cache (homelab service <name> health is cache-first, --live forces a probe), drift detectors, signal engine (homelab signal …), decision classifier (homelab decide …), approval engine (homelab approval … — shared-HMAC grants; Matrix delivery is Hermes's existing @dtoro:avispero send path, not a new bot, see oikos/approve.py), daily brief + weekly report (oikos/report.py). Shipped, except: Prometheus is still planned (see plans/2026-07-05-oikos-prometheus-lxc.md) — trend signals (disk-full prediction, temp creep) wait on that LXC; the scheduler's disk check today is point-in-time only, and CPU/NVMe temperature isn't probed at all yet (no confirmed sensor path on hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness drift checks are also deferred (see oikos/drift.py docstring).
  • Week 4: Oikos Console (oikos.hubris.network, behind Authentik with step-up re-auth on approvals), per-agent age-key-signed approval requests (upgrading from Week 3's shared-HMAC), docs pass, 60/90-day backlog.

Real drift found while building Week 3 (unresolved, needs operator action)

The drift detectors surfaced genuine, currently-true findings on first run against production — recorded here rather than silently fixed, since each is a config_mutation/destructive-class decision:

  • republic-laptop has no age_pubkey: in inventory.yaml, but its real age key is granted on nearly every shared secret in .sops.yaml (age1vf8h7...) — the enrollment write-back to inventory never happened. Fix: homelab client add republic-laptop --finalize-pubkey age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6.
  • grimmory has an age_pubkey in inventory but is missing from secrets/hello.yaml's recipient list — incomplete enrollment the other direction. Fix: re-run homelab client add grimmory --finalize-pubkey <its key>.
  • pve_id 131 exists live on hubris (pct list) with no inventory entry — investigate before assuming it's a stale ID (see the Prometheus LXC plan doc above, which flags this explicitly).
  • Three lifecycle-pve-id-reuse info findings (100, 106, 107 each shared between an active host and an archaeology entry) — expected/benign ID reuse after destroy, no action needed.