Files

dtoro 2084a1583e Oikos Week 3: scheduler, drift detectors, signals, classifier, approvals

New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:

- oikos/scheduler.py — Observe stage: HTTP health probes for every
  service, disk-usage probes on hubris/strong, writes oikos/state.json
  (gitignored — regenerates every run). `homelab service <name> health`
  is now cache-first; `--live` forces a fresh probe. Deploys via
  oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.

- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
  detectors (fully local, no SSH) plus pct-list and Caddy-backend
  detectors (best-effort SSH, degrade to an info finding when
  unreachable rather than a false drift alarm). Found real, currently-
  true drift on first run: republic-laptop's age key granted on every
  secret but missing from inventory.yaml, grimmory missing from
  hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
  recorded in OIKOS.md for the operator, not auto-fixed (each is a
  config_mutation/destructive decision).

- oikos/signal.py — the attention layer: raised -> acknowledged ->
  acting -> resolved|muted lifecycle, severity-based routing, dedup via
  open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.

- oikos/decide.py — the Decide-stage classifier: risk class x blast
  radius x ledger-history confidence -> auto-act/escalate. Adds an
  action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
  service_name from the entity for per-service policy overrides.
  `homelab decide <action> <entity>`.

- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
  in this homelab, so this is the repo-side half only: request/reply/
  grant lifecycle with short-TTL HMAC-signed tokens (new secret
  secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
  delivery is Hermes's existing @dtoro:avispero send path (documented
  integration contract in the module docstring), not a new bot.
  `homelab restart` now mechanically refuses config_mutation/destructive
  services without a valid --approval-id, regardless of -y/interactivity.

- oikos/report.py — daily brief + weekly report from signal/approval/
  ledger state (no Prometheus yet, so point-in-time counts only).

- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
  not provisioned: no pve_id is guessed here since Proxmox assigns real
  IDs at creation time, and drift already found an unclaimed ID (131) to
  investigate first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-05 23:29:39 +02:00

3.4 KiB

Raw Blame History

Oikos metrics stack — Prometheus LXC (planned)

Lifecycle state: planned (see oikos/ontology.yaml lifecycle). No LXC exists yet — this is the plan doc that state requires before provisioning starts. Do not add a hosts: entry with a guessed pve_id until the LXC is actually created; Proxmox assigns the real ID at pct create time.

Why

Week-3 reliability layer (see OIKOS.md) wants trend signals — "disk full in ~9 days at current rate", temperature creep — which need a real time-series store. The scheduler (oikos/scheduler.py) currently does point-in-time threshold checks only; Prometheus is the one new piece of infrastructure the 30-day roadmap calls for.

Note: undocumented LXC 131 on hubris

Oikos's drift detector (oikos/drift.py) found pve_id 131 live on hubris (via pct list) with no inventory.yaml entry — created outside the provision-node runbook, identity unknown from this repo. Investigate what 131 is before assuming any pve_id is free; don't let Proxmox auto-assign into a range you haven't confirmed is actually unused end-to-end.

Plan

Host: hubris (per the earlier decision: new LXC, not co-located).
Role: metrics (Prometheus + local TSDB retention; no Grafana yet — the Week-4 Oikos Console renders its own sparklines from the Prometheus HTTP API, per the plan's Week-3 scope decision).
Networking: LAN + mesh-gated only, no public ingress (matches homelab_mcp/secrets_issuance's MESH_SUBNETS pattern) — Prometheus exposes host/service metadata that shouldn't be public.
Scrape targets: node_exporter on hubris and strong (Proxmox hosts)
- any LXC the scheduler needs disk/temp trend data from beyond what pct/df already gives (start with hubris + strong only; expand only if a specific signal needs it).
Storage: local LXC rootfs is sufficient (metrics-only workload, short retention — no /mnt/library mount needed).

Provisioning steps (once pve_id is assigned)

Follow runbooks/lifecycle-provision-node.md:

pct create <new-id> ... on hubris — confirm the assigned ID doesn't collide with 131 or anything else live.
homelab client add metrics (or the chosen name) with state: provisioning, stub containers/<id>-metrics.md.
Install Prometheus + node_exporter (Debian package or binary release — decide at implementation time; no strong preference recorded here).
Point node_exporter at hubris + strong (either install locally on each, or scrape via SSH-tunneled metrics — install locally is simpler and is the standard approach).
Follow runbooks/lifecycle-activate-node.md to flip to active, complete the doc page, regenerate hosts/*.yaml + infrastructure/topology.md.
Extend oikos/scheduler.py's disk/temp probes to query Prometheus rate() instead of (or alongside) the current live SSH df probe, so drift/threshold Signals gain trend evidence ("full in ~9 days") instead of only a point-in-time percentage.

Open question for the operator

Package choice (apt prometheus vs upstream binary release) and exact scrape interval aren't decided here — pick at implementation time based on what's easiest to keep patched via the existing homelab apt-audit/ apt-upgrade fleet tooling if using the Debian package.

3.4 KiB Raw Blame History