dtoro
43aaf2a318
Plan #4 done. Audit inventory verified against codebase: - 9 superseded oikos/*.py files deleted (only gen-topology.py remains) - bin/homelab deleted, bin/oikos deleted, oikos/cards/ deleted - .hermes/plans/ already archived to archive/hermes-plans/ (all 7 files) - TRMNL plan already in Done table - seanime + romm documented in seeds/knowledge.yaml (DB-native, no wiki needed) - Traefik references valid (VPS still runs traefik for public termination) - ADR-0011 exists (client lifecycle); consolidation plan is Go rewrite record - Prometheus plan updated: Python refs replaced with Go scheduler, check_defs, MCP request_execution; LXC 131 identified as teddycloud Remaining items (cutover, Infisical, watchdog, rollback, apps/105) belong to consolidation plan (#1). 4 of 6 plans now Done.
72 lines
3.5 KiB
Markdown
72 lines
3.5 KiB
Markdown
# Oikos metrics stack — Prometheus LXC (planned)
|
|
|
|
Lifecycle state: **planned** (see [seeds/ontology.yaml](../seeds/ontology.yaml)
|
|
lifecycle). No LXC exists yet — this is the plan doc that state requires
|
|
before provisioning starts. Do not add a host entry with a guessed
|
|
`pve_id` until the LXC is actually created; Proxmox assigns the real ID
|
|
at `pct create` time.
|
|
|
|
## Why
|
|
|
|
Week-3 reliability layer (see [OIKOS.md](../.agents/OIKOS.md)) wants trend
|
|
signals — "disk full in ~9 days at current rate", temperature creep —
|
|
which need a real time-series store. The Go scheduler
|
|
([internal/scheduler/scheduler.go](../internal/scheduler/scheduler.go))
|
|
currently does point-in-time threshold checks only; Prometheus is the one
|
|
new piece of infrastructure the 30-day roadmap calls for.
|
|
|
|
## Note: LXC 131 is teddycloud
|
|
|
|
The drift detector (`drift` check kind in `check_defs`) found
|
|
`pve_id 131` live on hubris — this is **teddycloud** (`lxc:teddycloud`),
|
|
which is now documented in `seeds/knowledge.yaml`. The pve_id range is
|
|
fully accounted for (101-134). Next available VMIDs start at 135.
|
|
|
|
## Plan
|
|
|
|
- **Host:** hubris (per the earlier decision: new LXC, not co-located).
|
|
- **Role:** `metrics` (Prometheus + local TSDB retention; no Grafana yet —
|
|
the Week-4 Oikos Console renders its own sparklines from the Prometheus
|
|
HTTP API, per the plan's Week-3 scope decision).
|
|
- **Networking:** LAN + mesh-gated only, no public ingress (matches
|
|
`mcp` / `secrets-issuance` `MESH_SUBNETS` pattern) — Prometheus
|
|
exposes host/service metadata that shouldn't be public.
|
|
- **Scrape targets:** node_exporter on hubris and strong (Proxmox hosts)
|
|
+ any LXC the scheduler needs disk/temp trend data from beyond what
|
|
`pct`/`df` already gives (start with hubris + strong only; expand only
|
|
if a specific signal needs it).
|
|
- **Storage:** local LXC rootfs is sufficient (metrics-only workload,
|
|
short retention — no `/mnt/library` mount needed).
|
|
|
|
## Provisioning steps (once pve_id is assigned)
|
|
|
|
1. Add entity to `seeds/inventory.yaml` with state `planned` and the
|
|
assigned pve_id. Run `oikos seed` to ingest.
|
|
2. `pct create <new-id> ...` on hubris — next available PVE ID is 135.
|
|
3. Install Prometheus + node_exporter (Debian package or binary release —
|
|
decide at implementation time; no strong preference recorded here).
|
|
4. Point node_exporter at hubris + strong (either install locally on each,
|
|
or scrape via SSH-tunneled metrics — install locally is simpler).
|
|
5. Transition to `active` via `PATCH /api/v1/entities/{slug}` or
|
|
operator approval through the execution flow.
|
|
6. Add a `prometheus` check kind to `check_defs` in the scheduler
|
|
(`internal/scheduler/scheduler.go`) so trend signals can query
|
|
Prometheus `rate()` alongside the current SSH `df` probe, giving
|
|
"full in ~9 days" predictions instead of only point-in-time
|
|
percentages. Register a `prometheus` check def via `POST /api/v1/checks`.
|
|
7. Run `oikos export` to regenerate `seeds/inventory.yaml` for git.
|
|
|
|
## Open question for the operator
|
|
|
|
Package choice (apt `prometheus` vs upstream binary release) and exact
|
|
scrape interval aren't decided here — pick at implementation time based
|
|
on what's easiest to keep patched via MCP `request_execution(action="apt_upgrade")`.
|
|
|
|
## Changelog
|
|
|
|
### 2026-07-08 — Go references updated
|
|
Replaced Python references (`oikos/scheduler.py`, `oikos/drift.py`,
|
|
`bin/homelab`, `homelab client add`) with Go equivalents: `internal/scheduler/`,
|
|
`check_defs` drift kind, MCP `request_execution`, `seeds/inventory.yaml`.
|
|
LXC 131 identified as teddycloud (PVE range 101-134 fully accounted for).
|