New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:
- oikos/scheduler.py — Observe stage: HTTP health probes for every
service, disk-usage probes on hubris/strong, writes oikos/state.json
(gitignored — regenerates every run). `homelab service <name> health`
is now cache-first; `--live` forces a fresh probe. Deploys via
oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.
- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
detectors (fully local, no SSH) plus pct-list and Caddy-backend
detectors (best-effort SSH, degrade to an info finding when
unreachable rather than a false drift alarm). Found real, currently-
true drift on first run: republic-laptop's age key granted on every
secret but missing from inventory.yaml, grimmory missing from
hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
recorded in OIKOS.md for the operator, not auto-fixed (each is a
config_mutation/destructive decision).
- oikos/signal.py — the attention layer: raised -> acknowledged ->
acting -> resolved|muted lifecycle, severity-based routing, dedup via
open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.
- oikos/decide.py — the Decide-stage classifier: risk class x blast
radius x ledger-history confidence -> auto-act/escalate. Adds an
action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
service_name from the entity for per-service policy overrides.
`homelab decide <action> <entity>`.
- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
in this homelab, so this is the repo-side half only: request/reply/
grant lifecycle with short-TTL HMAC-signed tokens (new secret
secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
delivery is Hermes's existing @dtoro:avispero send path (documented
integration contract in the module docstring), not a new bot.
`homelab restart` now mechanically refuses config_mutation/destructive
services without a valid --approval-id, regardless of -y/interactivity.
- oikos/report.py — daily brief + weekly report from signal/approval/
ledger state (no Prometheus yet, so point-in-time counts only).
- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
not provisioned: no pve_id is guessed here since Proxmox assigns real
IDs at creation time, and drift already found an unclaimed ID (131) to
investigate first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
141 lines
7.3 KiB
Markdown
141 lines
7.3 KiB
Markdown
# Oikos — the operating model
|
||
|
||
Oikos (Greek: *household*) is the agent operating system layered on this
|
||
repo. It is not new infrastructure: `inventory.yaml` is the kernel data
|
||
structure, the `homelab` CLI and MCP server are the syscall surface, and
|
||
this page defines the rules everything above them follows.
|
||
|
||
Read this after [AGENTS.md](AGENTS.md). Machine-readable companions:
|
||
[oikos/ontology.yaml](oikos/ontology.yaml) (systems model),
|
||
[oikos/policy.yaml](oikos/policy.yaml) (risk & approval).
|
||
|
||
## The kernel loop: OODA
|
||
|
||
Every Oikos activity — scheduled probe, agent task, operator request — is
|
||
one pass through **Observe → Orient → Decide → Act**:
|
||
|
||
1. **Observe** — probes, drift detectors, and agent findings produce
|
||
**Signals** (structured records, not loose messages): pending updates,
|
||
high temperature, low disk, service down, cert expiry, stale backup,
|
||
inventory drift.
|
||
2. **Orient** — walk the ontology graph: what entity is affected, what
|
||
depends on it (blast radius), its lifecycle state, whether a runbook
|
||
matches, what the ledger says about past attempts.
|
||
3. **Decide** — the classifier scores **risk class × blast radius ×
|
||
confidence** and routes:
|
||
- **auto-act**: within autonomy policy, high confidence, contained radius
|
||
- **escalate**: operator approval via Matrix (✅/❌ reaction; destructive
|
||
actions additionally need a typed confirmation phrase)
|
||
- **queue**: informational — console + reports
|
||
The classifier can only *lower* autonomy relative to policy, never raise
|
||
it. When in doubt, escalate.
|
||
4. **Act** — execute through `homelab` commands or runbooks (never ad-hoc
|
||
SSH), then **verify** with the action's verification command, write a
|
||
**ledger** entry, resolve the Signal, and update docs in the same session.
|
||
|
||
## Primitives
|
||
|
||
| Primitive | What it is | Lives in |
|
||
|---|---|---|
|
||
| Host / Service | topology entities | `inventory.yaml` (+ generated `hosts/*.yaml`) |
|
||
| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` |
|
||
| Runbook | executable workflow with risk class + verification | `runbooks/` (Week 2) |
|
||
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
|
||
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
|
||
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
|
||
| Incident | investigation narrative | `investigations/` |
|
||
| Plan | design doc for non-trivial work | `plans/` |
|
||
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
|
||
|
||
## Risk classes (enforced, not advisory)
|
||
|
||
From [oikos/policy.yaml](oikos/policy.yaml):
|
||
|
||
- **read_only** — status, logs, docs, inventory. Unattended.
|
||
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger.
|
||
- **config_mutation** — tracked-config edits (commit+push, never local),
|
||
deploys, upgrades, DNS/ingress changes. Operator approval.
|
||
- **destructive** — destroy, format, wipe, rotate, revoke. Approval +
|
||
typed confirmation phrase.
|
||
|
||
Lifecycle gates modify these: `provisioning` nodes are freely mutable
|
||
(nothing depends on them); `deprecated` nodes accept no new dependents;
|
||
anything touching a `destroyed` node is drift.
|
||
|
||
## The systems model
|
||
|
||
Eight domains — physical, compute, network, storage, software,
|
||
identity & access, operations, external — cover everything in the lab;
|
||
entities are connected by typed edges (`hosts`, `provides`, `mounts`,
|
||
`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …)
|
||
defined in [oikos/ontology.yaml](oikos/ontology.yaml). Rule of
|
||
completeness: **if it can break, be changed, or hold data, it has an
|
||
entity and edges.** Blast-radius questions ("what breaks if strong goes
|
||
down?") are graph walks, not doc archaeology.
|
||
|
||
Nodes move through an explicit lifecycle —
|
||
`planned → provisioning → active → migrating → deprecated → destroyed` —
|
||
stored as `state:` in inventory (absent = active). Destroyed nodes live in
|
||
the `archaeology:` section. Each transition is a runbook checklist;
|
||
deprecation completes only when inbound edges reach zero.
|
||
|
||
Generated views: [infrastructure/topology.md](infrastructure/topology.md)
|
||
(Mermaid, regenerated from inventory).
|
||
|
||
## Conventions carried forward
|
||
|
||
- Inventory is the truth; live state wins over narrative docs.
|
||
- Prefer `homelab` CLI and MCP over ad-hoc SSH.
|
||
- Meaningful changes update docs in the same session.
|
||
- Secrets are decrypted locally via per-client keys; never into docs/comments.
|
||
- Tracked configs change by commit + push, not local edits.
|
||
- Netbird is the preferred mesh path for new traffic.
|
||
- Agents are terse ([CAVEMAN.md](CAVEMAN.md)), verify claims, and fix
|
||
collateral drift when found.
|
||
|
||
## Build status (30-day roadmap, started 2026-07-05)
|
||
|
||
- **Week 1**: policy, ontology, service contract, archaeology, topology
|
||
generator, this brief. Shipped.
|
||
- **Week 2**: context cards, `homelab service <name> …`, change ledger,
|
||
`node relations`, runbooks. Shipped.
|
||
- **Week 3**: ops scheduler + state cache (`homelab service <name> health`
|
||
is cache-first, `--live` forces a probe), drift detectors, signal engine
|
||
(`homelab signal …`), decision classifier (`homelab decide …`), approval
|
||
engine (`homelab approval …` — shared-HMAC grants; Matrix delivery is
|
||
Hermes's existing `@dtoro:avispero` send path, not a new bot, see
|
||
`oikos/approve.py`), daily brief + weekly report (`oikos/report.py`).
|
||
Shipped, except: Prometheus is still `planned` (see
|
||
[plans/2026-07-05-oikos-prometheus-lxc.md](plans/2026-07-05-oikos-prometheus-lxc.md)) —
|
||
trend signals (disk-full prediction, temp creep) wait on that LXC; the
|
||
scheduler's disk check today is point-in-time only, and CPU/NVMe
|
||
temperature isn't probed at all yet (no confirmed sensor path on
|
||
hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness
|
||
drift checks are also deferred (see `oikos/drift.py` docstring).
|
||
- **Week 4**: Oikos Console (oikos.hubris.network, behind Authentik with
|
||
step-up re-auth on approvals), per-agent age-key-signed approval
|
||
requests (upgrading from Week 3's shared-HMAC), docs pass, 60/90-day
|
||
backlog.
|
||
|
||
### Real drift found while building Week 3 (unresolved, needs operator action)
|
||
|
||
The drift detectors surfaced genuine, currently-true findings on first
|
||
run against production — recorded here rather than silently fixed, since
|
||
each is a `config_mutation`/`destructive`-class decision:
|
||
|
||
- `republic-laptop` has no `age_pubkey:` in `inventory.yaml`, but its real
|
||
age key is granted on nearly every shared secret in `.sops.yaml`
|
||
(`age1vf8h7...`) — the enrollment write-back to inventory never
|
||
happened. Fix: `homelab client add republic-laptop --finalize-pubkey
|
||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6`.
|
||
- `grimmory` has an `age_pubkey` in inventory but is missing from
|
||
`secrets/hello.yaml`'s recipient list — incomplete enrollment the
|
||
other direction. Fix: re-run `homelab client add grimmory
|
||
--finalize-pubkey <its key>`.
|
||
- `pve_id 131` exists live on hubris (`pct list`) with no inventory entry
|
||
— investigate before assuming it's a stale ID (see the Prometheus LXC
|
||
plan doc above, which flags this explicitly).
|
||
- Three `lifecycle-pve-id-reuse` info findings (100, 106, 107 each shared
|
||
between an active host and an archaeology entry) — expected/benign ID
|
||
reuse after destroy, no action needed.
|