Files
oikos/.agents/skills/incident-investigation/SKILL.md
dtoro d80a394b7f
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
docs: fix plan/repo drift, retire dead Goose+Nomos and Caveman tooling
Documentation and repo-hygiene pass following the client/server split:

Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
  described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
  refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
  deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
  to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.

Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).

Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).

Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.

Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.

Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
  places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
  GetClientContext handler) since the mechanism's introduction on
  2026-06-02 — never matched any real filename, so no client has ever
  picked up an auto-setup script via git-pull or the context-poller sync.
  Fixed all three; the Go server-side fix is the one that actually matters
  since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
  (parsed, never consumed) left over from an earlier clone-based model.

Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 18:19:41 +02:00

1.7 KiB

name, risk_class, inputs, verification, docs_update_checklist
name risk_class inputs verification docs_update_checklist
incident-investigation read_only
symptom
affected_entity
n/a — investigation produces a written record, not a state change
investigations_entry

Incident investigation

Goal: understand what broke and why, before touching anything.

  1. MCP explain (or get_blast_radius if the affected entity is a host) — get the blast radius and doc pointer first. Don't start pulling logs blind.
  2. MCP get_service_status + tail_log for the affected service.
  3. Walk the blast radius: is a shared dependency down (caddy, dns, authentik, or the backend host itself)? MCP get_blast_radius shows "affected by" — check those first.
  4. If the symptom looks like a dpkg/upgrade interaction, see runbook-dpkg-interrupted — there's no fleet-wide apt-audit tool anymore, check the host directly.
  5. Check the change ledger for recent mutations to the affected entity or anything upstream of it: MCP get_change_history or get_audit_trail.
  6. Write findings via MCP upsert_knowledge (kind: investigation) — symptom, timeline, root cause, fix applied, prevention, about set to the affected entity's slug. The DB is the durable record now, not a markdown file — search_knowledge/get_entity_knowledge read it back; a chat message alone is forgotten.

Docs-update checklist: always create the investigation entry. If the root cause was stale/wrong inventory data (a doc_page, config_repo, or backend that didn't match reality — this happened during Week 1 kernel work, see the authentik backend fix), correct inventory.yaml in the same session.