New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:
- oikos/scheduler.py — Observe stage: HTTP health probes for every
service, disk-usage probes on hubris/strong, writes oikos/state.json
(gitignored — regenerates every run). `homelab service <name> health`
is now cache-first; `--live` forces a fresh probe. Deploys via
oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.
- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
detectors (fully local, no SSH) plus pct-list and Caddy-backend
detectors (best-effort SSH, degrade to an info finding when
unreachable rather than a false drift alarm). Found real, currently-
true drift on first run: republic-laptop's age key granted on every
secret but missing from inventory.yaml, grimmory missing from
hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
recorded in OIKOS.md for the operator, not auto-fixed (each is a
config_mutation/destructive decision).
- oikos/signal.py — the attention layer: raised -> acknowledged ->
acting -> resolved|muted lifecycle, severity-based routing, dedup via
open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.
- oikos/decide.py — the Decide-stage classifier: risk class x blast
radius x ledger-history confidence -> auto-act/escalate. Adds an
action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
service_name from the entity for per-service policy overrides.
`homelab decide <action> <entity>`.
- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
in this homelab, so this is the repo-side half only: request/reply/
grant lifecycle with short-TTL HMAC-signed tokens (new secret
secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
delivery is Hermes's existing @dtoro:avispero send path (documented
integration contract in the module docstring), not a new bot.
`homelab restart` now mechanically refuses config_mutation/destructive
services without a valid --approval-id, regardless of -y/interactivity.
- oikos/report.py — daily brief + weekly report from signal/approval/
ledger state (no Prometheus yet, so point-in-time counts only).
- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
not provisioned: no pve_id is guessed here since Proxmox assigns real
IDs at creation time, and drift already found an unclaimed ID (131) to
investigate first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
7.3 KiB
Oikos — the operating model
Oikos (Greek: household) is the agent operating system layered on this
repo. It is not new infrastructure: inventory.yaml is the kernel data
structure, the homelab CLI and MCP server are the syscall surface, and
this page defines the rules everything above them follows.
Read this after AGENTS.md. Machine-readable companions: oikos/ontology.yaml (systems model), oikos/policy.yaml (risk & approval).
The kernel loop: OODA
Every Oikos activity — scheduled probe, agent task, operator request — is one pass through Observe → Orient → Decide → Act:
- Observe — probes, drift detectors, and agent findings produce Signals (structured records, not loose messages): pending updates, high temperature, low disk, service down, cert expiry, stale backup, inventory drift.
- Orient — walk the ontology graph: what entity is affected, what depends on it (blast radius), its lifecycle state, whether a runbook matches, what the ledger says about past attempts.
- Decide — the classifier scores risk class × blast radius ×
confidence and routes:
- auto-act: within autonomy policy, high confidence, contained radius
- escalate: operator approval via Matrix (✅/❌ reaction; destructive actions additionally need a typed confirmation phrase)
- queue: informational — console + reports The classifier can only lower autonomy relative to policy, never raise it. When in doubt, escalate.
- Act — execute through
homelabcommands or runbooks (never ad-hoc SSH), then verify with the action's verification command, write a ledger entry, resolve the Signal, and update docs in the same session.
Primitives
| Primitive | What it is | Lives in |
|---|---|---|
| Host / Service | topology entities | inventory.yaml (+ generated hosts/*.yaml) |
| Secret | SOPS+age encrypted value, per-client recipients | secrets/ + .sops.yaml |
| Runbook | executable workflow with risk class + verification | runbooks/ (Week 2) |
| Signal | something needing attention, with lifecycle | signals/ ledger (Week 3) |
| Change | one mutation: who, what, risk, approval, verification | ledger/ (Week 2) |
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
| Incident | investigation narrative | investigations/ |
| Plan | design doc for non-trivial work | plans/ |
| Agent | enrolled client identity = its age pubkey | inventory.yaml + .sops.yaml |
Risk classes (enforced, not advisory)
From oikos/policy.yaml:
- read_only — status, logs, docs, inventory. Unattended.
- reversible_low — restart, cache clear, sync pull. Unattended + ledger.
- config_mutation — tracked-config edits (commit+push, never local), deploys, upgrades, DNS/ingress changes. Operator approval.
- destructive — destroy, format, wipe, rotate, revoke. Approval + typed confirmation phrase.
Lifecycle gates modify these: provisioning nodes are freely mutable
(nothing depends on them); deprecated nodes accept no new dependents;
anything touching a destroyed node is drift.
The systems model
Eight domains — physical, compute, network, storage, software,
identity & access, operations, external — cover everything in the lab;
entities are connected by typed edges (hosts, provides, mounts,
stores-on, routes-to, can-decrypt, depends-on, backs-up-to, …)
defined in oikos/ontology.yaml. Rule of
completeness: if it can break, be changed, or hold data, it has an
entity and edges. Blast-radius questions ("what breaks if strong goes
down?") are graph walks, not doc archaeology.
Nodes move through an explicit lifecycle —
planned → provisioning → active → migrating → deprecated → destroyed —
stored as state: in inventory (absent = active). Destroyed nodes live in
the archaeology: section. Each transition is a runbook checklist;
deprecation completes only when inbound edges reach zero.
Generated views: infrastructure/topology.md (Mermaid, regenerated from inventory).
Conventions carried forward
- Inventory is the truth; live state wins over narrative docs.
- Prefer
homelabCLI and MCP over ad-hoc SSH. - Meaningful changes update docs in the same session.
- Secrets are decrypted locally via per-client keys; never into docs/comments.
- Tracked configs change by commit + push, not local edits.
- Netbird is the preferred mesh path for new traffic.
- Agents are terse (CAVEMAN.md), verify claims, and fix collateral drift when found.
Build status (30-day roadmap, started 2026-07-05)
- Week 1: policy, ontology, service contract, archaeology, topology generator, this brief. Shipped.
- Week 2: context cards,
homelab service <name> …, change ledger,node relations, runbooks. Shipped. - Week 3: ops scheduler + state cache (
homelab service <name> healthis cache-first,--liveforces a probe), drift detectors, signal engine (homelab signal …), decision classifier (homelab decide …), approval engine (homelab approval …— shared-HMAC grants; Matrix delivery is Hermes's existing@dtoro:avisperosend path, not a new bot, seeoikos/approve.py), daily brief + weekly report (oikos/report.py). Shipped, except: Prometheus is stillplanned(see plans/2026-07-05-oikos-prometheus-lxc.md) — trend signals (disk-full prediction, temp creep) wait on that LXC; the scheduler's disk check today is point-in-time only, and CPU/NVMe temperature isn't probed at all yet (no confirmed sensor path on hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness drift checks are also deferred (seeoikos/drift.pydocstring). - Week 4: Oikos Console (oikos.hubris.network, behind Authentik with step-up re-auth on approvals), per-agent age-key-signed approval requests (upgrading from Week 3's shared-HMAC), docs pass, 60/90-day backlog.
Real drift found while building Week 3 (unresolved, needs operator action)
The drift detectors surfaced genuine, currently-true findings on first
run against production — recorded here rather than silently fixed, since
each is a config_mutation/destructive-class decision:
republic-laptophas noage_pubkey:ininventory.yaml, but its real age key is granted on nearly every shared secret in.sops.yaml(age1vf8h7...) — the enrollment write-back to inventory never happened. Fix:homelab client add republic-laptop --finalize-pubkey age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6.grimmoryhas anage_pubkeyin inventory but is missing fromsecrets/hello.yaml's recipient list — incomplete enrollment the other direction. Fix: re-runhomelab client add grimmory --finalize-pubkey <its key>.pve_id 131exists live on hubris (pct list) with no inventory entry — investigate before assuming it's a stale ID (see the Prometheus LXC plan doc above, which flags this explicitly).- Three
lifecycle-pve-id-reuseinfo findings (100, 106, 107 each shared between an active host and an archaeology entry) — expected/benign ID reuse after destroy, no action needed.