---
name: incident-investigation
risk_class: read_only
inputs: [symptom, affected_entity]
verification: "n/a — investigation produces a written record, not a state change"
docs_update_checklist: [investigations_entry]
---

# Incident investigation

Goal: understand what broke and why, before touching anything.

1. `homelab service <name> explain` (or `homelab node <name> relations`
   if the affected entity is a host) — get the blast radius and doc
   pointer first. Don't start pulling logs blind.
2. `homelab service <name> health` + `homelab service <name> log` (or
   MCP `get_service_status` / `tail_log`) for the affected service.
3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
   `authentik`, or the backend host itself)? `homelab node <name>
   relations` shows "affected by" — check those first.
4. `homelab apt-audit` if the symptom looks like a dpkg/upgrade
   interaction.
5. Check the change ledger for recent mutations to the affected entity
   or anything upstream of it: `homelab service <name> history` (once
   populated) or grep `ledger/*.jsonl`.
6. Write findings to a new `investigations/<date>-<slug>.md` — symptom,
   timeline, root cause, fix applied, prevention. This is the durable
   record; don't rely on chat history.

Docs-update checklist: always create the investigation entry. If the
root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,
or `backend` that didn't match reality — this happened during Week 1
kernel work, see the `authentik` backend fix), correct `inventory.yaml`
in the same session.