---
name: incident-investigation
risk_class: read_only
inputs: [symptom, affected_entity]
verification: "n/a — investigation produces a written record, not a state change"
docs_update_checklist: [investigations_entry]
---

# Incident investigation

Goal: understand what broke and why, before touching anything.

1. MCP `explain` (or `get_blast_radius` if the affected entity is a
   host) — get the blast radius and doc pointer first. Don't start
   pulling logs blind.
2. MCP `get_service_status` + `tail_log` for the affected service.
3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
   `authentik`, or the backend host itself)? MCP `get_blast_radius`
   shows "affected by" — check those first.
4. If the symptom looks like a dpkg/upgrade interaction, see
   [runbook-dpkg-interrupted](../runbook-dpkg-interrupted/SKILL.md) —
   there's no fleet-wide apt-audit tool anymore, check the host directly.
5. Check the change ledger for recent mutations to the affected entity
   or anything upstream of it: MCP `get_change_history` or `get_audit_trail`.
6. Write findings via MCP `upsert_knowledge` (`kind: investigation`) —
   symptom, timeline, root cause, fix applied, prevention, `about` set to
   the affected entity's slug. The DB is the durable record now, not a
   markdown file — `search_knowledge`/`get_entity_knowledge` read it back;
   a chat message alone is forgotten.

Docs-update checklist: always create the investigation entry. If the
root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,
or `backend` that didn't match reality — this happened during Week 1
kernel work, see the `authentik` backend fix), correct `inventory.yaml`
in the same session.