Oikos Week 3: scheduler, drift detectors, signals, classifier, approvals

New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:

- oikos/scheduler.py — Observe stage: HTTP health probes for every
  service, disk-usage probes on hubris/strong, writes oikos/state.json
  (gitignored — regenerates every run). `homelab service <name> health`
  is now cache-first; `--live` forces a fresh probe. Deploys via
  oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.

- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
  detectors (fully local, no SSH) plus pct-list and Caddy-backend
  detectors (best-effort SSH, degrade to an info finding when
  unreachable rather than a false drift alarm). Found real, currently-
  true drift on first run: republic-laptop's age key granted on every
  secret but missing from inventory.yaml, grimmory missing from
  hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
  recorded in OIKOS.md for the operator, not auto-fixed (each is a
  config_mutation/destructive decision).

- oikos/signal.py — the attention layer: raised -> acknowledged ->
  acting -> resolved|muted lifecycle, severity-based routing, dedup via
  open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.

- oikos/decide.py — the Decide-stage classifier: risk class x blast
  radius x ledger-history confidence -> auto-act/escalate. Adds an
  action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
  service_name from the entity for per-service policy overrides.
  `homelab decide <action> <entity>`.

- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
  in this homelab, so this is the repo-side half only: request/reply/
  grant lifecycle with short-TTL HMAC-signed tokens (new secret
  secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
  delivery is Hermes's existing @dtoro:avispero send path (documented
  integration contract in the module docstring), not a new bot.
  `homelab restart` now mechanically refuses config_mutation/destructive
  services without a valid --approval-id, regardless of -y/interactivity.

- oikos/report.py — daily brief + weekly report from signal/approval/
  ledger state (no Prometheus yet, so point-in-time counts only).

- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
  not provisioned: no pve_id is guessed here since Proxmox assigns real
  IDs at creation time, and drift already found an unclaimed ID (131) to
  investigate first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

plans/2026-07-05-oikos-prometheus-lxc.md

@@ -0,0 +1,69 @@
+# Oikos metrics stack — Prometheus LXC (planned)
+
+Lifecycle state: **planned** (see [oikos/ontology.yaml](../oikos/ontology.yaml)
+lifecycle). No LXC exists yet — this is the plan doc that state requires
+before provisioning starts. Do not add a `hosts:` entry with a guessed
+`pve_id` until the LXC is actually created; Proxmox assigns the real ID
+at `pct create` time.
+
+## Why
+
+Week-3 reliability layer (see [OIKOS.md](../OIKOS.md)) wants trend
+signals — "disk full in ~9 days at current rate", temperature creep —
+which need a real time-series store. The scheduler
+([oikos/scheduler.py](../oikos/scheduler.py)) currently does point-in-time
+threshold checks only; Prometheus is the one new piece of infrastructure
+the 30-day roadmap calls for.
+
+## Note: undocumented LXC 131 on hubris
+
+Oikos's drift detector ([oikos/drift.py](../oikos/drift.py)) found
+`pve_id 131` live on hubris (via `pct list`) with no `inventory.yaml`
+entry — created outside the provision-node runbook, identity unknown
+from this repo. **Investigate what 131 is before assuming any pve_id is
+free**; don't let Proxmox auto-assign into a range you haven't confirmed
+is actually unused end-to-end.
+
+## Plan
+
+- **Host:** hubris (per the earlier decision: new LXC, not co-located).
+- **Role:** `metrics` (Prometheus + local TSDB retention; no Grafana yet —
+  the Week-4 Oikos Console renders its own sparklines from the Prometheus
+  HTTP API, per the plan's Week-3 scope decision).
+- **Networking:** LAN + mesh-gated only, no public ingress (matches
+  `homelab_mcp`/`secrets_issuance`'s `MESH_SUBNETS` pattern) — Prometheus
+  exposes host/service metadata that shouldn't be public.
+- **Scrape targets:** node_exporter on hubris and strong (Proxmox hosts)
+  + any LXC the scheduler needs disk/temp trend data from beyond what
+  `pct`/`df` already gives (start with hubris + strong only; expand only
+  if a specific signal needs it).
+- **Storage:** local LXC rootfs is sufficient (metrics-only workload,
+  short retention — no `/mnt/library` mount needed).
+
+## Provisioning steps (once pve_id is assigned)
+
+Follow [runbooks/lifecycle-provision-node.md](../runbooks/lifecycle-provision-node.md):
+
+1. `pct create <new-id> ...` on hubris — confirm the assigned ID doesn't
+   collide with 131 or anything else live.
+2. `homelab client add metrics` (or the chosen name) with `state:
+   provisioning`, stub `containers/<id>-metrics.md`.
+3. Install Prometheus + node_exporter (Debian package or binary release —
+   decide at implementation time; no strong preference recorded here).
+4. Point node_exporter at hubris + strong (either install locally on each,
+   or scrape via SSH-tunneled metrics — install locally is simpler and is
+   the standard approach).
+5. Follow [runbooks/lifecycle-activate-node.md](../runbooks/lifecycle-activate-node.md)
+   to flip to `active`, complete the doc page, regenerate
+   `hosts/*.yaml` + `infrastructure/topology.md`.
+6. Extend `oikos/scheduler.py`'s disk/temp probes to query Prometheus
+   rate() instead of (or alongside) the current live SSH `df` probe, so
+   drift/threshold Signals gain trend evidence ("full in ~9 days") instead
+   of only a point-in-time percentage.
+
+## Open question for the operator
+
+Package choice (apt `prometheus` vs upstream binary release) and exact
+scrape interval aren't decided here — pick at implementation time based
+on what's easiest to keep patched via the existing `homelab apt-audit`/
+`apt-upgrade` fleet tooling if using the Debian package.