Oikos Week 2: Service Console v0, change ledger, node relations, runbooks

Adds the shared kernel modules (oikos/policy.py, oikos/relations.py,
oikos/ledger.py) that let every surface — CLI, MCP, context-card
generator — agree on risk classification and ontology graph walks
from one implementation.

homelab CLI: `service <name> explain|health|docs|log|actions|history`
(Service Console v0), `change preflight <service>`, `node <name>
relations`. Restart and client add/remove now append change-ledger
entries (ledger/*.jsonl, committed alongside the change they record).

mcp/server.py mirrors explain/preflight/get_relations/get_change_history
as MCP tools, card-first so agent orientation is one call instead of
several search_docs/get_page round-trips.

oikos/gen-topology.py now also emits a compact context card per host
and service (oikos/cards/*.md) — identity, blast radius, safe actions +
risk class, doc pointer, recent ledger history.

runbooks/*.md: service health check, config change + deploy, client
enrollment, incident investigation, and the five node lifecycle
transitions (provision/activate/migrate/deprecate/destroy), each with
machine-readable frontmatter (risk class, inputs, verification,
docs-update checklist). Wired into HERMES.md so agents load these
instead of rediscovering topology per-task.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

runbooks/client-enrollment.md

@@ -0,0 +1,35 @@
+---
+name: client-enrollment
+risk_class: config_mutation
+inputs: [hostname, kind, role]
+verification: "homelab doctor (on the new client)"
+docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
+---
+
+# Client enrollment
+
+Goal: bring a new host (workstation, LXC, VM) into the mesh, inventory,
+and secrets model. This wraps the existing `homelab client add` flow —
+see [operations/agent-enrollment.md](../operations/agent-enrollment.md)
+for the full walkthrough; this runbook is the risk/lifecycle framing.
+
+1. On any enrolled client: `homelab client add <hostname>` — appends a
+   `hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned`
+   → `provisioning`, per [oikos/ontology.yaml](../oikos/ontology.yaml)),
+   commits + pushes.
+2. Join the new host to Netbird (out-of-band, console or setup key).
+3. On the new host: run `bootstrap.sh` (add `--with-hermes` to also
+   enroll the Hermes agent). This provisions `/etc/age/key.txt`, the
+   sync timer, and prints an age pubkey.
+4. Back on an enrolled client: `homelab client add <hostname>
+   --finalize-pubkey <age1...>` — sets `age_pubkey`, grants shared
+   secrets, re-keys SOPS, commits + pushes. This is the
+   `provisioning → active` transition.
+5. Verify: `homelab doctor` on the new client should show all checks
+   green (clone, sync timer, age key, CLI symlink, MCP reachable).
+
+Docs-update checklist: if the new host is an LXC/VM, add its narrative
+page under `containers/` or `vms/` and set `doc_page` in its inventory
+entry (host-level cards don't have a `doc_page` field yet — services do;
+narrative pages are still found via the generated `see_also` in
+`hosts/<name>.yaml`).

runbooks/config-change-deploy.md

@@ -0,0 +1,34 @@
+---
+name: config-change-deploy
+risk_class: config_mutation
+inputs: [service_name, change_description]
+verification: "curl -sf <service_url> (or homelab service <name> health)"
+docs_update_checklist: [doc_page, changelog]
+---
+
+# Config change + deploy
+
+Goal: change a tracked config repo (Caddy, Gitea customizations, an app's
+own repo) and get it live, safely.
+
+1. `homelab change preflight <service>` — current health, the service's
+   `config_repo`, its risk class, and the verification command to run
+   after. If risk class requires approval (`config_mutation` or
+   `destructive`), stop and get operator sign-off before editing — see
+   `oikos/policy.yaml`.
+2. Clone/pull the `config_repo` (never edit the backend's working tree
+   directly — tracked configs change by commit + push, per
+   [OIKOS.md](../OIKOS.md) conventions).
+3. Make the change, commit, push to `main`.
+4. The Gitea webhook fires the deploy pipeline for that repo (see
+   [infrastructure/auto-deploy.md](../infrastructure/auto-deploy.md) for
+   the exact receiver/reload for this service).
+5. Run the preflight's verification command. If it fails, check
+   `homelab service <name> log` for the reload/restart error.
+6. Record the change: once `oikos/ledger.py` is wired into deploy tooling
+   (Week 3), this is automatic; until then, note the change and outcome
+   in the relevant investigation/plan doc.
+
+Docs-update checklist: update the service's `doc_page` if the change
+alters its behavior, ingress route, or ownership; add a changelog entry
+if the page has one.

runbooks/incident-investigation.md

@@ -0,0 +1,34 @@
+---
+name: incident-investigation
+risk_class: read_only
+inputs: [symptom, affected_entity]
+verification: "n/a — investigation produces a written record, not a state change"
+docs_update_checklist: [investigations_entry]
+---
+
+# Incident investigation
+
+Goal: understand what broke and why, before touching anything.
+
+1. `homelab service <name> explain` (or `homelab node <name> relations`
+   if the affected entity is a host) — get the blast radius and doc
+   pointer first. Don't start pulling logs blind.
+2. `homelab service <name> health` + `homelab service <name> log` (or
+   MCP `get_service_status` / `tail_log`) for the affected service.
+3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
+   `authentik`, or the backend host itself)? `homelab node <name>
+   relations` shows "affected by" — check those first.
+4. `homelab apt-audit` if the symptom looks like a dpkg/upgrade
+   interaction.
+5. Check the change ledger for recent mutations to the affected entity
+   or anything upstream of it: `homelab service <name> history` (once
+   populated) or grep `ledger/*.jsonl`.
+6. Write findings to a new `investigations/<date>-<slug>.md` — symptom,
+   timeline, root cause, fix applied, prevention. This is the durable
+   record; don't rely on chat history.
+
+Docs-update checklist: always create the investigation entry. If the
+root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,
+or `backend` that didn't match reality — this happened during Week 1
+kernel work, see the `authentik` backend fix), correct `inventory.yaml`
+in the same session.

runbooks/lifecycle-activate-node.md

@@ -0,0 +1,36 @@
+---
+name: lifecycle-activate-node
+risk_class: config_mutation
+inputs: [node_name]
+verification: "homelab service <name> health (if it hosts a service); homelab doctor (if it's a client)"
+docs_update_checklist: [doc_page_complete]
+transition: "provisioning -> active"
+---
+
+# Lifecycle: activate a node
+
+Per [oikos/ontology.yaml](../oikos/ontology.yaml). Requires: age key
+enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
+ingress live if public, health check answering, doc page complete,
+ledger entry.
+
+1. If the node is a `homelab` client: finish enrollment per
+   [client-enrollment.md](client-enrollment.md) (`--finalize-pubkey`,
+   mesh join, `homelab doctor` green).
+2. If it hosts a public service: add the `services:` entry in
+   `inventory.yaml` (backend, url, doc_page, config_repo, risk_notes —
+   see the Week-1 service contract fields) and wire the Caddy route in
+   `dtoro/caddy-conf`.
+3. Confirm the health check answers: `homelab service <name> health` or
+   a direct `curl`.
+4. Flip `state: provisioning` → `state: active` (or delete the `state:`
+   field — `active` is the default) in `inventory.yaml`.
+5. Complete the doc page (stub → full narrative: role, specs, how it's
+   configured, dependencies).
+6. Record the activation: `oikos/ledger.py append host:<name> activate
+   config_mutation --result ok` (or let the CLI wrapper do this once
+   Week 3's runbook automation lands).
+
+Regenerate derived data: `python3 mcp/build_host_files.py && python3
+oikos/gen-topology.py` so `hosts/<name>.yaml`, the topology diagram, and
+the context card all reflect the new state.

runbooks/lifecycle-deprecate-node.md

@@ -0,0 +1,35 @@
+---
+name: lifecycle-deprecate-node
+risk_class: config_mutation
+inputs: [node_name, replacement_node_or_reason]
+verification: "homelab node <name> relations — 'affected by' must be empty before completing"
+docs_update_checklist: [doc_page_deprecation_note]
+transition: "active -> deprecated"
+---
+
+# Lifecycle: deprecate a node
+
+Per [oikos/ontology.yaml](../oikos/ontology.yaml): a node keeps running
+but takes no new dependents. **Completion condition: zero remaining
+inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a
+suggestion; `oikos/policy.yaml` `lifecycle_overrides.deprecated.refuse`
+lists `new-inbound-edges` as refused going forward.
+
+1. Set `state: deprecated` on the node.
+2. `homelab node <name> relations` — read `affected_by`. Every entry
+   there is something still relying on this node.
+3. Migrate or retire each dependent one at a time (point its `backend`/
+   `config_repo`/ingress route elsewhere, or deprecate it too if it's
+   being retired alongside).
+4. Re-run `homelab node <name> relations` after each dependent is moved.
+   The transition to `destroyed` is only safe once `affected_by` is
+   empty — check this every time, don't assume from memory.
+5. Note the deprecation on the doc page: reason, replacement (if any),
+   date.
+
+If step 2 shows dependents you didn't expect, stop and investigate
+before proceeding — that's exactly the kind of drift the Week-3 detector
+will catch automatically, but until then this manual check is the gate.
+
+Next (once `affected_by` is empty):
+[lifecycle-destroy-node.md](lifecycle-destroy-node.md).

runbooks/lifecycle-destroy-node.md

@@ -0,0 +1,42 @@
+---
+name: lifecycle-destroy-node
+risk_class: destructive
+inputs: [node_name]
+verification: "homelab node <name> relations returns unknown-entity; pct list on the backend no longer shows it"
+docs_update_checklist: [archaeology_entry, containers_index_update]
+transition: "deprecated -> destroyed"
+---
+
+# Lifecycle: destroy a node
+
+**Destructive.** Requires operator approval + typed confirmation phrase
+per `oikos/policy.yaml`. Requires (ontology): backups verified, secrets
+recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
+ledger entry.
+
+1. Confirm the node is `deprecated` with zero `affected_by` edges
+   (`homelab node <name> relations`) — do not skip this even if the
+   deprecation runbook was followed recently; state can drift.
+2. If it's an enrolled client: `homelab client remove <name>` — revokes
+   the age key, re-keys SOPS, removes the inventory entry. This is
+   already destructive-class and confirmed in the CLI.
+3. Remove any ingress route (Caddy config repo) and DNS record still
+   pointing at it.
+4. Verify backups of anything on it are retained per policy before the
+   disk goes away (see `backs-up-to`).
+5. Destroy the LXC/VM (`pct destroy` / `qm destroy`).
+6. Move the `hosts.<name>:` block (if any inventory remnant survives
+   `client remove`, e.g. infra-only LXCs with no age key) into
+   inventory.yaml's `archaeology:` section: `pve_id`, `destroyed` date,
+   `reason`. Add a row to `containers/index.md` "Recently destroyed"
+   table (kept for human-readable browsing alongside the structured
+   data).
+7. `oikos/ledger.py append host:<name> destroy destructive --result ok`.
+8. Regenerate: `python3 mcp/build_host_files.py && python3
+   oikos/gen-topology.py` — the node drops out of `hosts/*.yaml` and
+   appears in the topology doc's archaeology table.
+
+If the destroy fails partway (e.g. secrets revoked but pct destroy
+errors), do not re-run step 2 — `client remove` is not idempotent
+against a second revocation attempt on the issuance server. Finish the
+remaining steps manually and note the partial state in an investigation.

runbooks/lifecycle-migrate-node.md

@@ -0,0 +1,39 @@
+---
+name: lifecycle-migrate-node
+risk_class: config_mutation
+inputs: [node_name, source_host, target_host]
+verification: "homelab node <name> relations (re-check blast radius); homelab service <svc> health for every hosted service"
+docs_update_checklist: [doc_page_migration_note, inventory_host_and_lan_ip]
+transition: "active -> migrating -> active"
+---
+
+# Lifecycle: migrate a node
+
+Modeled on the strong Phase 1+2 migration
+([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
+Requires (ontology): preflight + backup-verified before migrating;
+post-verify + Caddy backends checked + mounts checked + docs updated
+before returning to `active`.
+
+1. `homelab change preflight <every service the node hosts>` — capture
+   current health as a baseline.
+2. Verify backups are current for anything with data at rest on the
+   node (see `backs-up-to` edges once populated).
+3. Set `state: migrating` in `inventory.yaml`.
+4. Perform the migration (pct/qm move, or create-on-target +
+   data-copy + destroy-source, per the specific case).
+5. Update `inventory.yaml`: new `host:`, `lan_ip`, `mesh` addresses for
+   the node; update every `services:` entry whose `backend` pointed at
+   it if the backend name itself changes (usually it doesn't — only the
+   `host:`/`lan_ip` on the guest entry moves).
+6. Post-verify: re-run the Week-1 drift check by hand — confirm Caddy's
+   backend IP for each affected service matches the new `lan_ip`
+   (automatic in Week 3's drift detector), confirm mounts still resolve.
+7. `homelab service <name> health` for every service the node hosts.
+8. Set `state: active`. Add a migration note to the node's doc page
+   (old host/IP → new, date, phase reference) — this repo's convention
+   for every past migration (see `containers/101-jellyfin.md`,
+   `containers/129-house.md`).
+
+Regenerate: `python3 mcp/build_host_files.py && python3
+oikos/gen-topology.py`.

runbooks/lifecycle-provision-node.md

@@ -0,0 +1,33 @@
+---
+name: lifecycle-provision-node
+risk_class: config_mutation
+inputs: [node_name, kind, storage_pool]
+verification: "grep 'state: provisioning' hosts/<name>.yaml"
+docs_update_checklist: [doc_page_stub]
+transition: "planned -> provisioning"
+---
+
+# Lifecycle: provision a node
+
+Per [oikos/ontology.yaml](../oikos/ontology.yaml) `lifecycle.transitions`.
+Policy note: `provisioning` nodes get a lifecycle override —
+`config_mutation` actions downgrade to `reversible_low` because nothing
+depends on the node yet (see `oikos/policy.yaml` `lifecycle_overrides`).
+
+Requires (from ontology): inventory entry, IP reserved, storage pool
+chosen, doc page stub.
+
+1. Create the LXC/VM on its target Proxmox host (`pct create` /
+   `qm create`), choosing the storage pool deliberately — record it as
+   the `storage:` field once populated (Week 1 schema; not yet backfilled
+   for existing nodes).
+2. Add the inventory entry: `homelab client add <name>` for anything that
+   will run the `homelab` CLI, or a direct `hosts.<name>:` block with
+   `state: provisioning`, `kind`, `host`, `pve_id`, `lan_ip` for
+   infra-only LXCs that won't self-enroll.
+3. Stub the doc page (`containers/<pve_id>-<name>.md` or
+   `vms/<pve_id>-<name>.md`) — even a one-line "provisioning, see plan X"
+   is enough to satisfy the transition requirement.
+4. Reserve the IP in DNS/DHCP notes if it's a fixed LAN address.
+
+Next: [lifecycle-activate-node.md](lifecycle-activate-node.md).

runbooks/service-health-check.md

@@ -0,0 +1,30 @@
+---
+name: service-health-check
+risk_class: read_only
+inputs: [service_name]
+verification: "homelab service <name> health"
+docs_update_checklist: []
+---
+
+# Service health check
+
+Goal: determine whether a service is actually healthy, without ad-hoc SSH.
+
+1. `homelab service <name> explain` — read the context card: backend,
+   blast radius, doc pointer, risk notes.
+2. `homelab service <name> health` — live health probe (HTTP code against
+   the service's `url`/`endpoint`). Once the Week-3 scheduler ships, this
+   reads a cached snapshot by default; pass `--live` to force a fresh probe.
+3. If unhealthy, `homelab service <name> log` (or MCP `tail_log`) for the
+   last 200 lines.
+4. Cross-check blast radius: `homelab node <name> relations` — is this
+   entity's own backend host healthy? A downstream failure (e.g. `strong`
+   down) will show up here before the service's own logs explain anything.
+5. If the fix is a restart: classify first (`oikos/policy.yaml` —
+   `service-restart` is `reversible_low` unless the service has a
+   `service_overrides` entry, e.g. `caddy`/`dns` are `config_mutation`).
+   Unattended agents may act on `reversible_low` without approval.
+
+Docs-update checklist: none for a pure health check. If the investigation
+reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml` in
+the same session.