docs: reshape runbooks into .agents/skills/<name>/SKILL.md (phase 4)

Problem: runbooks are agent-executable procedures but lived at the repo root,
separate from the other agent instruction now under .agents/.

Change:
- Move runbooks/<name>.md -> .agents/skills/<name>/SKILL.md (folder per skill,
  matching the wiki-hq skills layout). Frontmatter (name, risk_class, inputs,
  verification, docs_update_checklist, transition) preserved.
- Rewrite links (inbound from plans; between-skill siblings) via the move map.
- Update prose references in AGENTS.md, HERMES.md, .agents/OIKOS.md, and the
  operations schema; fix a pre-existing stale link to operations/commands.md.

No code consumed runbooks/ by path, so nothing else changes.

Verification: all SKILL.md frontmatter parses with valid risk_class; every
lifecycle transition resolves to an oikos/ontology.yaml state; broken-link
count 127 -> 126 (fixed one, introduced none).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-06 14:39:31 +02:00
parent 8a6422bd7d
commit 5c5016b3c7
16 changed files with 23 additions and 22 deletions

View File

@@ -40,7 +40,7 @@ one pass through **Observe → Orient → Decide → Act**:
|---|---|---|
| Host / Service | topology entities | `inventory.yaml` (+ generated `hosts/*.yaml`) |
| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` |
| Runbook | executable workflow with risk class + verification | `runbooks/` (Week 2) |
| Runbook | executable workflow with risk class + verification | `.agents/skills/<name>/SKILL.md` |
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |

View File

@@ -20,8 +20,8 @@ versioned, reviewable, and reachable by MCP `get_page`/`search_docs` like any ot
## Runbooks
Repeatable procedures live in `runbooks/<name>.md` with YAML front-matter that the Oikos policy and
lifecycle machinery reads:
Repeatable procedures are skills — one folder per skill at `.agents/skills/<name>/SKILL.md`, with
YAML front-matter that the Oikos policy and lifecycle machinery reads:
```yaml
---

View File

@@ -0,0 +1,43 @@
---
name: client-enrollment
risk_class: config_mutation
inputs: [hostname, kind, role]
verification: "homelab doctor (on the new client)"
docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
---
# Client enrollment
Goal: bring a new host (workstation, LXC, VM) into inventory and the
secrets model, with mesh membership only where it's actually needed.
This wraps the existing `homelab client add` flow — see
[operations/agent-enrollment.md](../../../operations/agent-enrollment.md) for
the full walkthrough; this runbook is the risk/lifecycle framing.
1. On any enrolled client: `homelab client add <hostname>` — appends a
`hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned`
`provisioning`, per [oikos/ontology.yaml](../../../oikos/ontology.yaml)),
commits + pushes.
2. Netbird join is **optional, not a required step** — only needed for
hosts that must be reachable off-LAN (workstations that roam, e.g.
`republic-laptop`, `mac-mini`). A node reachable on the household LAN
(192.168.8.0/24 — most LXCs/VMs) doesn't need it: it's already
reachable directly, and off-LAN clients reach it too via hubris's
routed `192.168.8.0/24` Netbird network resource. Skip this step for
LAN-only nodes; do it (out-of-band, console or setup key) only for
hosts that need independent off-LAN reachability.
3. On the new host: run `bootstrap.sh` (add `--with-hermes` to also
enroll the Hermes agent). This provisions `/etc/age/key.txt`, the
sync timer, and prints an age pubkey.
4. Back on an enrolled client: `homelab client add <hostname>
--finalize-pubkey <age1...>` — sets `age_pubkey`, grants shared
secrets, re-keys SOPS, commits + pushes. This is the
`provisioning → active` transition.
5. Verify: `homelab doctor` on the new client should show all checks
green (clone, sync timer, age key, CLI symlink, MCP reachable).
Docs-update checklist: if the new host is an LXC/VM, add its narrative
page under `containers/` or `vms/` and set `doc_page` in its inventory
entry (host-level cards don't have a `doc_page` field yet — services do;
narrative pages are still found via the generated `see_also` in
`hosts/<name>.yaml`).

View File

@@ -0,0 +1,34 @@
---
name: config-change-deploy
risk_class: config_mutation
inputs: [service_name, change_description]
verification: "curl -sf <service_url> (or homelab service <name> health)"
docs_update_checklist: [doc_page, changelog]
---
# Config change + deploy
Goal: change a tracked config repo (Caddy, Gitea customizations, an app's
own repo) and get it live, safely.
1. `homelab change preflight <service>` — current health, the service's
`config_repo`, its risk class, and the verification command to run
after. If risk class requires approval (`config_mutation` or
`destructive`), stop and get operator sign-off before editing — see
`oikos/policy.yaml`.
2. Clone/pull the `config_repo` (never edit the backend's working tree
directly — tracked configs change by commit + push, per
[OIKOS.md](../../../OIKOS.md) conventions).
3. Make the change, commit, push to `main`.
4. The Gitea webhook fires the deploy pipeline for that repo (see
[infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for
the exact receiver/reload for this service).
5. Run the preflight's verification command. If it fails, check
`homelab service <name> log` for the reload/restart error.
6. Record the change: once `oikos/ledger.py` is wired into deploy tooling
(Week 3), this is automatic; until then, note the change and outcome
in the relevant investigation/plan doc.
Docs-update checklist: update the service's `doc_page` if the change
alters its behavior, ingress route, or ownership; add a changelog entry
if the page has one.

View File

@@ -0,0 +1,34 @@
---
name: incident-investigation
risk_class: read_only
inputs: [symptom, affected_entity]
verification: "n/a — investigation produces a written record, not a state change"
docs_update_checklist: [investigations_entry]
---
# Incident investigation
Goal: understand what broke and why, before touching anything.
1. `homelab service <name> explain` (or `homelab node <name> relations`
if the affected entity is a host) — get the blast radius and doc
pointer first. Don't start pulling logs blind.
2. `homelab service <name> health` + `homelab service <name> log` (or
MCP `get_service_status` / `tail_log`) for the affected service.
3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
`authentik`, or the backend host itself)? `homelab node <name>
relations` shows "affected by" — check those first.
4. `homelab apt-audit` if the symptom looks like a dpkg/upgrade
interaction.
5. Check the change ledger for recent mutations to the affected entity
or anything upstream of it: `homelab service <name> history` (once
populated) or grep `ledger/*.jsonl`.
6. Write findings to a new `investigations/<date>-<slug>.md` — symptom,
timeline, root cause, fix applied, prevention. This is the durable
record; don't rely on chat history.
Docs-update checklist: always create the investigation entry. If the
root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,
or `backend` that didn't match reality — this happened during Week 1
kernel work, see the `authentik` backend fix), correct `inventory.yaml`
in the same session.

View File

@@ -0,0 +1,36 @@
---
name: lifecycle-activate-node
risk_class: config_mutation
inputs: [node_name]
verification: "homelab service <name> health (if it hosts a service); homelab doctor (if it's a client)"
docs_update_checklist: [doc_page_complete]
transition: "provisioning -> active"
---
# Lifecycle: activate a node
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml). Requires: age key
enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
ingress live if public, health check answering, doc page complete,
ledger entry.
1. If the node is a `homelab` client: finish enrollment per
[client-enrollment.md](../client-enrollment/SKILL.md) (`--finalize-pubkey`,
mesh join, `homelab doctor` green).
2. If it hosts a public service: add the `services:` entry in
`inventory.yaml` (backend, url, doc_page, config_repo, risk_notes —
see the Week-1 service contract fields) and wire the Caddy route in
`dtoro/caddy-conf`.
3. Confirm the health check answers: `homelab service <name> health` or
a direct `curl`.
4. Flip `state: provisioning``state: active` (or delete the `state:`
field — `active` is the default) in `inventory.yaml`.
5. Complete the doc page (stub → full narrative: role, specs, how it's
configured, dependencies).
6. Record the activation: `oikos/ledger.py append host:<name> activate
config_mutation --result ok` (or let the CLI wrapper do this once
Week 3's runbook automation lands).
Regenerate derived data: `python3 mcp/build_host_files.py && python3
oikos/gen-topology.py` so `hosts/<name>.yaml`, the topology diagram, and
the context card all reflect the new state.

View File

@@ -0,0 +1,35 @@
---
name: lifecycle-deprecate-node
risk_class: config_mutation
inputs: [node_name, replacement_node_or_reason]
verification: "homelab node <name> relations — 'affected by' must be empty before completing"
docs_update_checklist: [doc_page_deprecation_note]
transition: "active -> deprecated"
---
# Lifecycle: deprecate a node
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml): a node keeps running
but takes no new dependents. **Completion condition: zero remaining
inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a
suggestion; `oikos/policy.yaml` `lifecycle_overrides.deprecated.refuse`
lists `new-inbound-edges` as refused going forward.
1. Set `state: deprecated` on the node.
2. `homelab node <name> relations` — read `affected_by`. Every entry
there is something still relying on this node.
3. Migrate or retire each dependent one at a time (point its `backend`/
`config_repo`/ingress route elsewhere, or deprecate it too if it's
being retired alongside).
4. Re-run `homelab node <name> relations` after each dependent is moved.
The transition to `destroyed` is only safe once `affected_by` is
empty — check this every time, don't assume from memory.
5. Note the deprecation on the doc page: reason, replacement (if any),
date.
If step 2 shows dependents you didn't expect, stop and investigate
before proceeding — that's exactly the kind of drift the Week-3 detector
will catch automatically, but until then this manual check is the gate.
Next (once `affected_by` is empty):
[lifecycle-destroy-node.md](../lifecycle-destroy-node/SKILL.md).

View File

@@ -0,0 +1,42 @@
---
name: lifecycle-destroy-node
risk_class: destructive
inputs: [node_name]
verification: "homelab node <name> relations returns unknown-entity; pct list on the backend no longer shows it"
docs_update_checklist: [archaeology_entry, containers_index_update]
transition: "deprecated -> destroyed"
---
# Lifecycle: destroy a node
**Destructive.** Requires operator approval + typed confirmation phrase
per `oikos/policy.yaml`. Requires (ontology): backups verified, secrets
recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
ledger entry.
1. Confirm the node is `deprecated` with zero `affected_by` edges
(`homelab node <name> relations`) — do not skip this even if the
deprecation runbook was followed recently; state can drift.
2. If it's an enrolled client: `homelab client remove <name>` — revokes
the age key, re-keys SOPS, removes the inventory entry. This is
already destructive-class and confirmed in the CLI.
3. Remove any ingress route (Caddy config repo) and DNS record still
pointing at it.
4. Verify backups of anything on it are retained per policy before the
disk goes away (see `backs-up-to`).
5. Destroy the LXC/VM (`pct destroy` / `qm destroy`).
6. Move the `hosts.<name>:` block (if any inventory remnant survives
`client remove`, e.g. infra-only LXCs with no age key) into
inventory.yaml's `archaeology:` section: `pve_id`, `destroyed` date,
`reason`. Add a row to `containers/index.md` "Recently destroyed"
table (kept for human-readable browsing alongside the structured
data).
7. `oikos/ledger.py append host:<name> destroy destructive --result ok`.
8. Regenerate: `python3 mcp/build_host_files.py && python3
oikos/gen-topology.py` — the node drops out of `hosts/*.yaml` and
appears in the topology doc's archaeology table.
If the destroy fails partway (e.g. secrets revoked but pct destroy
errors), do not re-run step 2 — `client remove` is not idempotent
against a second revocation attempt on the issuance server. Finish the
remaining steps manually and note the partial state in an investigation.

View File

@@ -0,0 +1,39 @@
---
name: lifecycle-migrate-node
risk_class: config_mutation
inputs: [node_name, source_host, target_host]
verification: "homelab node <name> relations (re-check blast radius); homelab service <svc> health for every hosted service"
docs_update_checklist: [doc_page_migration_note, inventory_host_and_lan_ip]
transition: "active -> migrating -> active"
---
# Lifecycle: migrate a node
Modeled on the strong Phase 1+2 migration
([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
Requires (ontology): preflight + backup-verified before migrating;
post-verify + Caddy backends checked + mounts checked + docs updated
before returning to `active`.
1. `homelab change preflight <every service the node hosts>` — capture
current health as a baseline.
2. Verify backups are current for anything with data at rest on the
node (see `backs-up-to` edges once populated).
3. Set `state: migrating` in `inventory.yaml`.
4. Perform the migration (pct/qm move, or create-on-target +
data-copy + destroy-source, per the specific case).
5. Update `inventory.yaml`: new `host:`, `lan_ip`, `mesh` addresses for
the node; update every `services:` entry whose `backend` pointed at
it if the backend name itself changes (usually it doesn't — only the
`host:`/`lan_ip` on the guest entry moves).
6. Post-verify: re-run the Week-1 drift check by hand — confirm Caddy's
backend IP for each affected service matches the new `lan_ip`
(automatic in Week 3's drift detector), confirm mounts still resolve.
7. `homelab service <name> health` for every service the node hosts.
8. Set `state: active`. Add a migration note to the node's doc page
(old host/IP → new, date, phase reference) — this repo's convention
for every past migration (see `containers/101-jellyfin.md`,
`containers/129-house.md`).
Regenerate: `python3 mcp/build_host_files.py && python3
oikos/gen-topology.py`.

View File

@@ -0,0 +1,33 @@
---
name: lifecycle-provision-node
risk_class: config_mutation
inputs: [node_name, kind, storage_pool]
verification: "grep 'state: provisioning' hosts/<name>.yaml"
docs_update_checklist: [doc_page_stub]
transition: "planned -> provisioning"
---
# Lifecycle: provision a node
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml) `lifecycle.transitions`.
Policy note: `provisioning` nodes get a lifecycle override —
`config_mutation` actions downgrade to `reversible_low` because nothing
depends on the node yet (see `oikos/policy.yaml` `lifecycle_overrides`).
Requires (from ontology): inventory entry, IP reserved, storage pool
chosen, doc page stub.
1. Create the LXC/VM on its target Proxmox host (`pct create` /
`qm create`), choosing the storage pool deliberately — record it as
the `storage:` field once populated (Week 1 schema; not yet backfilled
for existing nodes).
2. Add the inventory entry: `homelab client add <name>` for anything that
will run the `homelab` CLI, or a direct `hosts.<name>:` block with
`state: provisioning`, `kind`, `host`, `pve_id`, `lan_ip` for
infra-only LXCs that won't self-enroll.
3. Stub the doc page (`containers/<pve_id>-<name>.md` or
`vms/<pve_id>-<name>.md`) — even a one-line "provisioning, see plan X"
is enough to satisfy the transition requirement.
4. Reserve the IP in DNS/DHCP notes if it's a fixed LAN address.
Next: [lifecycle-activate-node.md](../lifecycle-activate-node/SKILL.md).

View File

@@ -0,0 +1,209 @@
---
name: budget-import-from-csv
risk_class: config_mutation
inputs: [csv_file]
references: [containers/129-house.md]
---
# Runbook: Budget import from N26 CSV → Yuvomi
Distil a bank-export CSV into Yuvomi's Budget and Subscriptions modules using
the `yuvomi-mcp` tools. Run this whenever a new CSV period needs to be
summarised into targets and fixed costs.
---
## Prerequisites
- `yuvomi-mcp` is running on LXC 129 and connected as an MCP server in Claude.
- The CSV is an N26 export (columns: Booking Date, Value Date, Partner Name,
Partner Iban, Type, Payment Reference, Account Name, Amount (EUR), …).
- API token: `homelab secret yuvomi-api-token` (decrypts on any enrolled client).
- Direct API base: `https://house.hubris.network/api/v1`
---
## API quirks (Yuvomi ≤ 0.77.x)
- **Subscriptions live under `/budget/subscriptions`**, NOT `/subscriptions/`.
A top-level `/subscriptions` route returns 404.
- `GET /budget/subscriptions``{ data: { subscriptions: [...], summary: {...} } }`
- `GET /budget/subscriptions/meta``{ data: { categories: [...], payment_methods: [...] } }`
- `POST /budget/subscriptions` → create a subscription (name, amount, billing_cycle,
cycle_interval, next_payment_date, currency, category_id, payment_method_id required)
- `GET /budget/` (no month) → returns only **non-recurring** base entries.
Use `GET /budget/?month=YYYY-MM` to get all entries (recurring + one-time) for a month.
- `GET /budget/categories` → expense category keys + income category names (German keys
like `"Erwerbseinkommen"`, `"Sozialleistungen"`, `"Geschenke & Transfers"`).
- Budget entries: `amount` positive = income, negative = expense.
- Recurring entries: set `is_recurring: 1` + `recurrence_interval: "monthly"`.
The `date` field sets the start month.
- `recurrence_virtual: 1` smooths non-monthly amounts across all months in the summary
(e.g. 55.08 € quarterly → shows as ~18.36 €/month).
- Custom RRULE strings (`recurrence_rule`) are **not accepted** by the API — use
`cycle_interval` on the subscription instead, or `recurrence_interval` on budget entries.
---
## Subscription category IDs (as of 2026-06-26)
| id | name | budget_subcategory_key |
|---|---|---|
| 1 | Entertainment | subscription_entertainment |
| 2 | Productivity | subscription_productivity |
| 3 | Utilities | subscription_utilities |
| 4 | Health | subscription_health |
| 5 | Education | subscription_education |
| 6 | Other | subscription_other |
## Payment method IDs
| id | name |
|---|---|
| 1 | Credit Card |
| 2 | Debit Card |
| 3 | PayPal |
| 6 | Bank Transfer / SEPA |
| 7 | Other |
---
## Step 1 — Categorise the transactions
Skip these as internal/already-covered:
- Fixed costs you'll enter as **subscriptions** (Miete, SWM, SYNVIA, Hundefutter,
Netflix, Grover, Rundfunk ARD, KuKita, Lillydoo)
- Internal transfers (The Joy Pot ↔ Cookie, Hauptkonto, Tagesgeldkonto splits)
- Identified income (Cookie Share, Kindergeld, Pocket Money credits, Distributor)
- Fun Money pass-throughs (in and out same month → net zero)
**Variable expense taxonomy:**
| Category key | Subcategory key | Examples |
|---|---|---|
| `food` | `groceries` | E-Center, Knuspr, EDEKA, Tegut, VollCorner, Lidl, Netto, REWE, KoRo, Roast Market |
| `food` | `restaurants_bars` | Restaurants, Lieferando, Cafes, Zeit für Brot, Baobab, Höflinger |
| `personal_health` | `beauty_cosmetics` | DM, Rossmann |
| `personal_health` | `pharmacy` | Apotheke, MVZ Dermatologie |
| `transport` | `apps_taxi` | Uber, RYD GMBH, MVG, Handyparken |
| `shopping_clothing` | `gifts` | Children products: Schlummersack, Catchy Kids, SP EVERY., Dukal, Berger-Lernwelt |
| `shopping_clothing` | `clothes_shoes` | Zalando, Ernsting's, Schuhmair, Thalia, Vinted, Airbnb, Hotel at Booking.com |
| `shopping_clothing` | `electronics` | Amazon, AMZN Mktp DE |
| `housing` | `renovation_maintenance` | IKEA, Markus Festl, Granit, Sostrene Grene, Mol* tischdecken, Gaertnerei, Dehner |
| `education` | `courses_college` | Kathrin Orlob (PEKiP), Nerina Aupperle |
| `leisure` | `streaming` | WOW wowtv.de |
| `financial_other` | `bank_fees` | Unidentified PayPal, Ratepay, N26 fees |
| `Geschenke & Transfers` | *(income)* | One-off incoming transfers |
---
## Step 2 — Create subscriptions
```
get_subscriptions_meta() ← get category_id and payment_method_id
```
**Standard Cookie household subscriptions (as of 2026-07):**
| Name | Amount | billing_cycle | cycle_interval | category_id | payment_method_id |
|---|---|---|---|---|---|
| Miete | 1080.00 | monthly | 1 | 6 (Other) | 6 (Bank Transfer) |
| Strom (SWM) | 79.00 | monthly | 1 | 3 (Utilities) | 6 |
| Internet / TV / Telefon | 29.99 | monthly | 1 | 3 (Utilities) | 6 |
| Hundefutter | 75.00 | monthly | 1 | 6 (Other) | 6 |
| Netflix | 8.00 | monthly | 1 | 1 (Entertainment) | 6 |
| Grover | 16.90 | monthly | 1 | 6 (Other) | 2 (Debit Card) |
| Rundfunk ARD / ZDF | 55.08 | monthly | 3 | 1 (Entertainment) | 6 |
| KuKita Daycare (Leon) | 503.00 | monthly | 1 | 5 (Education) | 6 |
| Lillydoo diapers | 56.70 | monthly | 2 | 4 (Health) | 3 (PayPal) |
Monthly equivalent total: **1,838.60 €** (Yuvomi applies cycle_interval to prorate).
---
## Step 3 — Add recurring income entries
```
stage_add_budget_entry(
title="Kindergeld",
amount=55.00,
category="Sozialleistungen",
date="YYYY-MM-01",
is_recurring=True,
recurrence_interval="monthly",
)
commit_pending(pending_id)
```
**Standard recurring income:**
| Title | Amount | category |
|---|---|---|
| Kindergeld | +55.00 | Sozialleistungen |
| Cookie Share | +2650.00 | Erwerbseinkommen *(see recommended amount below)* |
---
## Step 4 — Post variable transactions
For each non-skipped CSV row, call `stage_add_budget_entry` with the mapped
category/subcategory and the actual transaction amount and date. Use the Partner
Name + Payment Reference as the title (truncate to 100 chars).
---
## Step 5 — Verify
```
get_budget_summary("YYYY-MM")
list_subscriptions()
```
Expected for a full month with KuKita:
- Fixed expenses ≥ 1,838 € (subscriptions)
- Variable expenses ≥ 500 € (groceries alone)
---
## Cookie Share: how much to transfer monthly
Calculated from JanJun 2026 data (Cookie account, one-offs stripped):
| | €/month |
|---|---|
| **Fixed costs (subscriptions)** | **1,839** |
| Miete | 1,080 |
| KuKita *(permanent from Jul 2026)* | 503 |
| Strom + SYNVIA + Rundfunk + Netflix + Grover + Hundefutter + Lillydoo | 256 |
| **Variable (6-month averages)** | **1,032** |
| Groceries | 595 |
| Children products | 142 |
| Dining & cafes | 100 |
| Transport | 66 |
| Drugstore | 52 |
| Clothing, Amazon, Pharmacy | 77 |
| **Total monthly spend** | **≈ 2,871** |
| Minus Kindergeld (fixed income) | 55 |
| Minus Pocket Money (conservative ~600 €) | 600 |
| **→ Recommended Cookie Share** | **≈ 2,650 €** |
| With 200 € buffer | **≈ 2,850 €** |
**Current Cookie Share (Jun 2026): 1,995 € — shortfall ~655 €.**
The gap was covered by irregular Pocket Money top-ups (avg 962 €/mo over 6 months, but
highly variable: 121 €3,000 €). KuKita starting in June is the biggest step-up; raising
Cookie Share to **2,650 €** makes the budget self-sufficient without relying on top-ups.
---
## Changelog
### 2026-06-29 — Corrections from first real import
- Subscriptions endpoint is `/budget/subscriptions`, NOT `/subscriptions/` (404).
- `recurrence_rule` RRULE strings are rejected by the API; use `cycle_interval` instead.
- `GET /budget/` (no filter) returns only non-recurring entries; use `?month=` for full view.
- Added Cookie Share recommendation (2,650 €/month) based on 6-month expense analysis.
- Added full category taxonomy table.
### 2026-06-29 — Initial runbook
Created from JanJun 2026 N26 Cookie account analysis.

View File

@@ -0,0 +1,114 @@
---
name: recover-dpkg-interrupted
risk_class: reversible_low
verification: "dpkg --audit (should be clean); apt-get check"
---
# Runbook — recover from dpkg-interrupted state
You're here because an apt run got killed mid-transaction and the target now
has packages that are **unpacked but not configured**. Symptoms:
- `apt` refuses to do anything new: `Error: dpkg was interrupted, you must
manually run 'dpkg --configure -a' to correct the problem.`
- `dpkg --audit` lists packages with header
`The following packages have been unpacked but not yet configured.`
- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host.
The system is still running the **old** binaries (still in memory), but the
**new** binaries are unpacked and waiting for their postinst to run. Two
worst-case manifestations from the 2026-05-21 sweep:
- LXC 121 caddy: leftover state from a prior aborted apt run; caddy itself was
still serving but the new caddy binary on disk hadn't been wired up.
- hubris: ssh master died mid-Wave-6 → 135 packages unpacked-not-configured,
including `systemd`, `openssh-server`, `sudo`, `netbird`. The half-
configured netbird daemon dropped the mesh peer, and we got locked out
until we recovered from the PVE web UI Shell.
**Do not reboot until dpkg is clean.** A reboot tries to start the new
binaries' services, which may fail because postinst never ran (missing users,
config dirs, capabilities, etc.). The system might not come back up cleanly.
## Path A — target is still reachable over ssh (preferred)
```
homelab ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
```
Or for an LXC by name:
```
homelab pct <lxc> exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
```
When that returns, confirm:
```
homelab apt-audit --target <host>
```
Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally
deferred (kernel/PVE on hubris, 0 elsewhere).
## Path B — target locked out (mesh broken / ssh dead)
Most common for hubris when netbird itself went half-configured: the daemon
crashed on the new binary, the mesh peer dropped, port 22022 stopped listening,
and you can't ssh in.
1. Open `https://proxmox.hubris.network` in a browser.
2. Datacenter → node `hubris` → `>_ Shell` (or `_ Console`). That's a root
shell on hubris served by the PVE web UI, independent of the netbird mesh.
3. Run the recovery one-liner:
```
DEBIAN_FRONTEND=noninteractive dpkg --configure -a \
&& DEBIAN_FRONTEND=noninteractive apt -y -o Dpkg::Options::=--force-confold upgrade \
&& systemctl restart netbird \
&& dpkg --audit \
&& echo RECOVERY_OK
```
Wait for `RECOVERY_OK`. The `systemctl restart netbird` is the bit that
heals the mesh — once netbird's daemon comes back up clean, your client's
peer state moves from `Connecting` to `Connected` within ~30 seconds and
the rest of your tooling works again.
4. For an **LXC** that's locked out (less common — LXCs reach the world via
netbird routed through hubris, so unless hubris itself is broken, you can
still `pct enter` from the hubris shell):
From the PVE web UI shell on hubris:
```
pct enter <id>
DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y upgrade
exit
```
## Prevention
The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect`
unit on the target, so it survives ssh teardown — the failure mode that put
hubris into this state in the first place is no longer reachable through the
standard tool. If you absolutely need to run apt manually over ssh, wrap it:
```
ssh <host> systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade'
```
Then `systemctl status apt-recovery` from a fresh ssh to check progress.
## Related
- [Operations cheatsheet](../../../operations/commands.md)
- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md)
- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md)
## Changelog
### 2026-05-21 — initial page
Documents the dpkg-interrupted recovery path that came out of the
fleet apt sweep (Wave 6 killed mid-transaction; hubris recovered via PVE
web Shell).

View File

@@ -0,0 +1,30 @@
---
name: service-health-check
risk_class: read_only
inputs: [service_name]
verification: "homelab service <name> health"
docs_update_checklist: []
---
# Service health check
Goal: determine whether a service is actually healthy, without ad-hoc SSH.
1. `homelab service <name> explain` — read the context card: backend,
blast radius, doc pointer, risk notes.
2. `homelab service <name> health` — live health probe (HTTP code against
the service's `url`/`endpoint`). Once the Week-3 scheduler ships, this
reads a cached snapshot by default; pass `--live` to force a fresh probe.
3. If unhealthy, `homelab service <name> log` (or MCP `tail_log`) for the
last 200 lines.
4. Cross-check blast radius: `homelab node <name> relations` — is this
entity's own backend host healthy? A downstream failure (e.g. `strong`
down) will show up here before the service's own logs explain anything.
5. If the fix is a restart: classify first (`oikos/policy.yaml`
`service-restart` is `reversible_low` unless the service has a
`service_overrides` entry, e.g. `caddy`/`dns` are `config_mutation`).
Unattended agents may act on `reversible_low` without approval.
Docs-update checklist: none for a pure health check. If the investigation
reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml` in
the same session.