Oikos Week 3: scheduler, drift detectors, signals, classifier, approvals

New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:

- oikos/scheduler.py — Observe stage: HTTP health probes for every
  service, disk-usage probes on hubris/strong, writes oikos/state.json
  (gitignored — regenerates every run). `homelab service <name> health`
  is now cache-first; `--live` forces a fresh probe. Deploys via
  oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.

- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
  detectors (fully local, no SSH) plus pct-list and Caddy-backend
  detectors (best-effort SSH, degrade to an info finding when
  unreachable rather than a false drift alarm). Found real, currently-
  true drift on first run: republic-laptop's age key granted on every
  secret but missing from inventory.yaml, grimmory missing from
  hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
  recorded in OIKOS.md for the operator, not auto-fixed (each is a
  config_mutation/destructive decision).

- oikos/signal.py — the attention layer: raised -> acknowledged ->
  acting -> resolved|muted lifecycle, severity-based routing, dedup via
  open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.

- oikos/decide.py — the Decide-stage classifier: risk class x blast
  radius x ledger-history confidence -> auto-act/escalate. Adds an
  action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
  service_name from the entity for per-service policy overrides.
  `homelab decide <action> <entity>`.

- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
  in this homelab, so this is the repo-side half only: request/reply/
  grant lifecycle with short-TTL HMAC-signed tokens (new secret
  secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
  delivery is Hermes's existing @dtoro:avispero send path (documented
  integration contract in the module docstring), not a new bot.
  `homelab restart` now mechanically refuses config_mutation/destructive
  services without a valid --approval-id, regardless of -y/interactivity.

- oikos/report.py — daily brief + weekly report from signal/approval/
  ledger state (no Prometheus yet, so point-in-time counts only).

- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
  not provisioned: no pve_id is guessed here since Proxmox assigns real
  IDs at creation time, and drift already found an unclaimed ID (131) to
  investigate first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-05 23:29:39 +02:00
parent 48debc0911
commit 2084a1583e
17 changed files with 1791 additions and 15 deletions

View File

@@ -95,13 +95,46 @@ Generated views: [infrastructure/topology.md](infrastructure/topology.md)
## Build status (30-day roadmap, started 2026-07-05)
- **Week 1 (this)**: policy, ontology, service contract, archaeology,
topology generator, this brief.
- **Week 1**: policy, ontology, service contract, archaeology, topology
generator, this brief. Shipped.
- **Week 2**: context cards, `homelab service <name> …`, change ledger,
`node relations`, runbooks.
- **Week 3**: Prometheus + node_exporter, ops scheduler + state cache,
drift detectors, signal engine, decision classifier, approval engine
(Matrix ✅/❌), daily brief.
`node relations`, runbooks. Shipped.
- **Week 3**: ops scheduler + state cache (`homelab service <name> health`
is cache-first, `--live` forces a probe), drift detectors, signal engine
(`homelab signal …`), decision classifier (`homelab decide …`), approval
engine (`homelab approval …` — shared-HMAC grants; Matrix delivery is
Hermes's existing `@dtoro:avispero` send path, not a new bot, see
`oikos/approve.py`), daily brief + weekly report (`oikos/report.py`).
Shipped, except: Prometheus is still `planned` (see
[plans/2026-07-05-oikos-prometheus-lxc.md](plans/2026-07-05-oikos-prometheus-lxc.md)) —
trend signals (disk-full prediction, temp creep) wait on that LXC; the
scheduler's disk check today is point-in-time only, and CPU/NVMe
temperature isn't probed at all yet (no confirmed sensor path on
hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness
drift checks are also deferred (see `oikos/drift.py` docstring).
- **Week 4**: Oikos Console (oikos.hubris.network, behind Authentik with
step-up re-auth on approvals), per-agent age-key-signed approval
requests, docs pass, 60/90-day backlog.
requests (upgrading from Week 3's shared-HMAC), docs pass, 60/90-day
backlog.
### Real drift found while building Week 3 (unresolved, needs operator action)
The drift detectors surfaced genuine, currently-true findings on first
run against production — recorded here rather than silently fixed, since
each is a `config_mutation`/`destructive`-class decision:
- `republic-laptop` has no `age_pubkey:` in `inventory.yaml`, but its real
age key is granted on nearly every shared secret in `.sops.yaml`
(`age1vf8h7...`) — the enrollment write-back to inventory never
happened. Fix: `homelab client add republic-laptop --finalize-pubkey
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6`.
- `grimmory` has an `age_pubkey` in inventory but is missing from
`secrets/hello.yaml`'s recipient list — incomplete enrollment the
other direction. Fix: re-run `homelab client add grimmory
--finalize-pubkey <its key>`.
- `pve_id 131` exists live on hubris (`pct list`) with no inventory entry
— investigate before assuming it's a stale ID (see the Prometheus LXC
plan doc above, which flags this explicitly).
- Three `lifecycle-pve-id-reuse` info findings (100, 106, 107 each shared
between an active host and an archaeology entry) — expected/benign ID
reuse after destroy, no action needed.