Files
oikos/OIKOS.md
dtoro 205d8a1a43 Oikos Week 4: Console v0, approval hardening, docs pass, backlog
Oikos Console v0 (oikos/console/) — read-mostly, server-rendered FastAPI
+ Jinja2 web UI, no SPA build chain. Signals landing page, service grid
+ detail, node/blast-radius view, live Mermaid relationship graph, drift
findings, approvals queue (approve/deny, destructive confirmation-phrase
enforced), daily/weekly reports. Tested end-to-end via the preview tools
against live production data, including a real click-through of the
approve/deny flow.

Found and fixed two bugs during that testing:
- Severity-dot CSS classes didn't match the actual severity strings
  (dot-warn/dot-crit vs "warning"/"critical") — warning-severity signals
  rendered with no visible indicator at all.
- The console's sys.path setup pointed at its own webhook checkout
  (/opt/oikos-console) rather than /opt/homelab-context, which would have
  made its oikos.* imports resolve to a SEPARATE copy of oikos/signal.py
  etc. than the scheduler and CLI use — silently forking signal/approval
  data into two locations in production. Fixed to match mcp/server.py's
  CONTEXT_DIR pattern. Also added _commit_push() so the console's writes
  (approval replies, signal ack/resolve) don't sit uncommitted against
  the 5-min-synced clone.

Split oikos/gen_topology_lib.py out of oikos/gen-topology.py (hyphenated
filenames aren't importable) so the console's /graph route can render
live without shelling out.

oikos/console/deploy/ — third webhook on dtoro/Homelab-Docs (port 9831),
matching the homelab-mcp/secrets-issuance precedent. README documents the
Caddy route and Gitea webhook registration this repo can't do for itself,
and that Authentik step-up on /approvals needs a live instance to
configure.

Approval hardening: grants are now single-use (oikos/approve.py
check_grant marks the request "executed" atomically, so a second call
for the same id fails even within the TTL) — verified with a test. Per-
agent age-key-signed requests, as originally planned, turned out not to
be buildable as stated: age is encryption-only, no signing primitive.
Documented the real alternative (SSH-key signing) and moved it to the
60/90-day backlog pending an inventory schema gap (no SSH pubkeys
recorded today).

Docs pass: added the Oikos command surface to operations/commands.md,
new MCP tools to AGENTS.md. Found two more stale references while at
it — commands.md and AGENTS.md both still pointed DNS at the destroyed
LXC 124/dnsmasq instead of Technitium on dns (107), and a claudio-monitor
reference deprecated since 2026-06-04 — fixed both.

60/90-day backlog written into OIKOS.md, derived from gaps actually
observed this month, not guesswork.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 00:03:51 +02:00

12 KiB
Raw Blame History

Oikos — the operating model

Oikos (Greek: household) is the agent operating system layered on this repo. It is not new infrastructure: inventory.yaml is the kernel data structure, the homelab CLI and MCP server are the syscall surface, and this page defines the rules everything above them follows.

Read this after AGENTS.md. Machine-readable companions: oikos/ontology.yaml (systems model), oikos/policy.yaml (risk & approval).

The kernel loop: OODA

Every Oikos activity — scheduled probe, agent task, operator request — is one pass through Observe → Orient → Decide → Act:

  1. Observe — probes, drift detectors, and agent findings produce Signals (structured records, not loose messages): pending updates, high temperature, low disk, service down, cert expiry, stale backup, inventory drift.
  2. Orient — walk the ontology graph: what entity is affected, what depends on it (blast radius), its lifecycle state, whether a runbook matches, what the ledger says about past attempts.
  3. Decide — the classifier scores risk class × blast radius × confidence and routes:
    • auto-act: within autonomy policy, high confidence, contained radius
    • escalate: operator approval via Matrix (/ reaction) or the Oikos Console's /approvals page (destructive actions additionally need a typed confirmation phrase either way)
    • queue: informational — console + reports The classifier can only lower autonomy relative to policy, never raise it. When in doubt, escalate.
  4. Act — execute through homelab commands or runbooks (never ad-hoc SSH), then verify with the action's verification command, write a ledger entry, resolve the Signal, and update docs in the same session.

Primitives

Primitive What it is Lives in
Host / Service topology entities inventory.yaml (+ generated hosts/*.yaml)
Secret SOPS+age encrypted value, per-client recipients secrets/ + .sops.yaml
Runbook executable workflow with risk class + verification runbooks/ (Week 2)
Signal something needing attention, with lifecycle signals/ ledger (Week 3)
Change one mutation: who, what, risk, approval, verification ledger/ (Week 2)
Approval short-TTL signed grant for a gated action approval engine (Week 3)
Incident investigation narrative investigations/
Plan design doc for non-trivial work plans/
Agent enrolled client identity = its age pubkey inventory.yaml + .sops.yaml

Risk classes (enforced, not advisory)

From oikos/policy.yaml:

  • read_only — status, logs, docs, inventory. Unattended.
  • reversible_low — restart, cache clear, sync pull. Unattended + ledger.
  • config_mutation — tracked-config edits (commit+push, never local), deploys, upgrades, DNS/ingress changes. Operator approval.
  • destructive — destroy, format, wipe, rotate, revoke. Approval + typed confirmation phrase.

Lifecycle gates modify these: provisioning nodes are freely mutable (nothing depends on them); deprecated nodes accept no new dependents; anything touching a destroyed node is drift.

The systems model

Eight domains — physical, compute, network, storage, software, identity & access, operations, external — cover everything in the lab; entities are connected by typed edges (hosts, provides, mounts, stores-on, routes-to, can-decrypt, depends-on, backs-up-to, …) defined in oikos/ontology.yaml. Rule of completeness: if it can break, be changed, or hold data, it has an entity and edges. Blast-radius questions ("what breaks if strong goes down?") are graph walks, not doc archaeology.

Nodes move through an explicit lifecycle — planned → provisioning → active → migrating → deprecated → destroyed — stored as state: in inventory (absent = active). Destroyed nodes live in the archaeology: section. Each transition is a runbook checklist; deprecation completes only when inbound edges reach zero.

Generated views: infrastructure/topology.md (Mermaid, regenerated from inventory) and the live, clickable version at oikos.hubris.network/graph once the Console is deployed.

Conventions carried forward

  • Inventory is the truth; live state wins over narrative docs.
  • Prefer homelab CLI and MCP over ad-hoc SSH.
  • Meaningful changes update docs in the same session.
  • Secrets are decrypted locally via per-client keys; never into docs/comments.
  • Tracked configs change by commit + push, not local edits.
  • Netbird is the preferred mesh path for new traffic.
  • Agents are terse (CAVEMAN.md), verify claims, and fix collateral drift when found.

Build status (30-day roadmap, started 2026-07-05)

  • Week 1: policy, ontology, service contract, archaeology, topology generator, this brief. Shipped.
  • Week 2: context cards, homelab service <name> …, change ledger, node relations, runbooks. Shipped.
  • Week 3: ops scheduler + state cache (homelab service <name> health is cache-first, --live forces a probe), drift detectors, signal engine (homelab signal …), decision classifier (homelab decide …), approval engine (homelab approval … — shared-HMAC grants; Matrix delivery is Hermes's existing @dtoro:avispero send path, not a new bot, see oikos/approve.py), daily brief + weekly report (oikos/report.py). Shipped, except: Prometheus is still planned (see plans/2026-07-05-oikos-prometheus-lxc.md) — trend signals (disk-full prediction, temp creep) wait on that LXC; the scheduler's disk check today is point-in-time only, and CPU/NVMe temperature isn't probed at all yet (no confirmed sensor path on hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness drift checks are also deferred (see oikos/drift.py docstring).
  • Week 4: Oikos Console v0 shipped — signals landing page, service grid + detail, node/blast-radius view, live Mermaid graph, drift view, approvals queue (approve/deny, destructive confirmation-phrase enforced), daily/weekly reports. Server-rendered FastAPI + Jinja2, no SPA build chain, tested end-to-end against live production data (see oikos/console/). Deploys as a third webhook on dtoro/Homelab-Docs (/opt/oikos-console, port :9831) — see oikos/console/deploy/README.md for the Caddy route and Gitea webhook registration this repo can't do for itself. Approval grants are now single-use (a second check_grant call for the same request fails even within the TTL) and already exact-bound to request id + entity + action. Not shipped as originally planned: per-agent age-key-signed request authentication — age has no signing primitive (it's an encryption-only keypair format), so "age-key-signed" wasn't buildable as stated. The real alternative (SSH-key signing via ssh-keygen -Y sign/-Y verify, using each host's already-provisioned SSH key) is real and buildable, but needs SSH public keys recorded in inventory first — not there today. Moved to the 60/90-day backlog. Authentik step-up re-auth on the approve/deny route is documented but needs a live Authentik instance to configure — also backlog. Docs pass done (this file, AGENTS.md, operations/commands.md); found and fixed two more stale references while at it (DNS section still pointed at destroyed LXC 124/dnsmasq instead of Technitium on 107, and a claudio-monitor reference that's been deprecated since 2026-06-04).

Real drift found while building Week 3 (unresolved, needs operator action)

The drift detectors surfaced genuine, currently-true findings on first run against production — recorded here rather than silently fixed, since each is a config_mutation/destructive-class decision:

  • republic-laptop has no age_pubkey: in inventory.yaml, but its real age key is granted on nearly every shared secret in .sops.yaml (age1vf8h7...) — the enrollment write-back to inventory never happened. Fix: homelab client add republic-laptop --finalize-pubkey age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6.
  • grimmory has an age_pubkey in inventory but is missing from secrets/hello.yaml's recipient list — incomplete enrollment the other direction. Fix: re-run homelab client add grimmory --finalize-pubkey <its key>.
  • pve_id 131 exists live on hubris (pct list) with no inventory entry — investigate before assuming it's a stale ID (see the Prometheus LXC plan doc above, which flags this explicitly).
  • Three lifecycle-pve-id-reuse info findings (100, 106, 107 each shared between an active host and an archaeology entry) — expected/benign ID reuse after destroy, no action needed.

60/90-day backlog

Derived from gaps observed while building the 30-day roadmap, not guesswork. Roughly ordered by what unblocks the most:

  • SSH-key-signed approval requests. Replaces the design note in Week 4: age keys can't sign (encryption-only format), so per-agent request authentication needs ssh-keygen -Y sign/-Y verify against each host's existing SSH key. Blocked on a schema gap: inventory doesn't record SSH public keys today, only ports/users. First step is populating that field on enrollment, then wiring oikos/approve.py to require and verify a signature over the request payload.
  • Authentik step-up re-auth on the Console's /approvals POST route — needs a live Authentik PromptStage/reauth flow scoped to that path; not configurable without a running instance to test against.
  • Prometheus provisioning (see plans/2026-07-05-oikos-prometheus-lxc.md) — unblocks trend signals (disk-full prediction, temp creep) and real sparklines in the Console; investigate the undocumented pve_id 131 on hubris first.
  • CPU/NVMe temperature probing in the scheduler — needs a confirmed sensor path on hubris and strong (lm-sensors vs vendor tool) before a real check can be written; guessing one risks a probe that silently never fires.
  • DNS-vs-inventory drift check — compare Technitium zone records against services.*.url/public_host; not implemented (oikos/drift.py has no Technitium API wiring yet).
  • Generic tracked-config-cleanliness drift check — today only caddy's /etc/caddy git-checkout path is hardcoded in oikos/drift.py; every other service with a config_repo needs its local checkout path recorded (a mutation_path-style field, same gap Week 1's service contract flagged but didn't backfill) before this generalizes.
  • Per-service policy overrides (oikos/policy.yaml service_overrides) — schema is ready (caddy/dns already use it); populate more as specific services turn out to need non-default risk classes.
  • Incident timeline generator — stitch ledger + signal history into a single narrative for investigations/ entries instead of writing them by hand.
  • Secret access audit — who-can-decrypt-what report from .sops.yaml + inventory age_pubkeys, extending what oikos/drift.py's SOPS check already partially does.
  • Restore drills — exercise backs-up-to (once populated) by actually restoring from a backup target on a schedule, not just checking freshness.
  • Multi-agent delegation model — more than one agent acting concurrently; needs the ledger's agent field to carry real identity (age pubkey, not just hostname) consistently, which it mostly does already but hasn't been stress-tested with concurrent writers.
  • Grafana — only if the Console's own Prometheus-backed sparklines turn out to be insufficient once Prometheus ships.
  • "Generalize later" extraction — the original decision was personal- first, generalize-later (see Week 1). Once patterns stabilize, extract a config-driven Oikos core with no hubris.network/hubris/strong hardcoding, so it's installable on a different homelab.