docs: fix plan/repo drift, retire dead Goose+Nomos and Caveman tooling
Documentation and repo-hygiene pass following the client/server split:
Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.
Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).
Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).
Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.
Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.
Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
GetClientContext handler) since the mechanism's introduction on
2026-06-02 — never matched any real filename, so no client has ever
picked up an auto-setup script via git-pull or the context-poller sync.
Fixed all three; the Go server-side fix is the one that actually matters
since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
(parsed, never consumed) left over from an earlier clone-based model.
Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -2,39 +2,46 @@
|
||||
name: client-enrollment
|
||||
risk_class: config_mutation
|
||||
inputs: [hostname, kind, role]
|
||||
verification: "homelab doctor (on the new client)"
|
||||
verification: "MCP whoami(hostname) shows the entity active"
|
||||
docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
|
||||
---
|
||||
|
||||
# Client enrollment
|
||||
|
||||
Goal: bring a new host (workstation, LXC, VM) into inventory and the
|
||||
secrets model, with mesh membership only where it's actually needed.
|
||||
This wraps the existing `homelab client add` flow — see
|
||||
[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for
|
||||
the full walkthrough; this runbook is the risk/lifecycle framing.
|
||||
secrets model, with mesh membership only where it's actually needed. See
|
||||
[CLIENTS.md](../../../CLIENTS.md#enrollment) for the actual current
|
||||
flow and [operations/agent-enrollment.md](../../operations/agent-enrollment.md)
|
||||
for operational notes; this runbook is the risk/lifecycle framing.
|
||||
|
||||
1. On any enrolled client: `homelab client add <hostname>` — appends a
|
||||
`hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned`
|
||||
→ `provisioning`, per [seeds/ontology.yaml](../../../seeds/ontology.yaml)),
|
||||
commits + pushes.
|
||||
1. The entity must exist in `planned`/`provisioning` state before the new
|
||||
host can self-enroll — add a `hosts.<name>:` block to
|
||||
`seeds/inventory.yaml` and `oikos seed` to ingest it (lifecycle
|
||||
`planned` → `provisioning`, per
|
||||
[seeds/ontology.yaml](../../../seeds/ontology.yaml)).
|
||||
2. Netbird join is **optional, not a required step** — only needed for
|
||||
hosts that must be reachable off-LAN (workstations that roam, e.g.
|
||||
`republic-laptop`, `mac-mini`). A node reachable on the household LAN
|
||||
(192.168.8.0/24 — most LXCs/VMs) doesn't need it: it's already
|
||||
reachable directly, and off-LAN clients reach it too via hubris's
|
||||
routed `192.168.8.0/24` Netbird network resource. Skip this step for
|
||||
LAN-only nodes; do it (out-of-band, console or setup key) only for
|
||||
hosts that need independent off-LAN reachability.
|
||||
3. On the new host: run `bootstrap.sh` (add `--with-nomos` to also
|
||||
enroll the Hermes agent). This provisions `/etc/age/key.txt`, the
|
||||
sync timer, and prints an age pubkey.
|
||||
4. Back on an enrolled client: `homelab client add <hostname>
|
||||
--finalize-pubkey <age1...>` — sets `age_pubkey`, grants shared
|
||||
secrets, re-keys SOPS, commits + pushes. This is the
|
||||
`provisioning → active` transition.
|
||||
5. Verify: `homelab doctor` on the new client should show all checks
|
||||
green (clone, sync timer, age key, CLI symlink, MCP reachable).
|
||||
`mac-mini`). A node reachable on the household LAN (192.168.8.0/24 —
|
||||
most LXCs/VMs) doesn't need it. Skip for LAN-only nodes; do it
|
||||
(out-of-band, console or setup key) only for hosts that need
|
||||
independent off-LAN reachability.
|
||||
3. On the new host: run `bootstrap.sh`. This calls
|
||||
`POST /api/v1/clients/enroll`, which validates the entity exists and
|
||||
the mesh IP is in an expected subnet, then returns an age keypair and
|
||||
Infisical machine identity in one response — provisions
|
||||
`/etc/age/key.txt`, `/etc/infisical/identity`, and the context poller.
|
||||
4. **Known gap, confirmed 2026-07-12: `provisioning → active` has no
|
||||
working path.** `EnrollClient` (`internal/httpapi/impl.go`) sets the
|
||||
entity's state to `provisioning`, never `active`. `bootstrap.sh` prints
|
||||
`POST /api/v1/clients/ws:$HNAME/activate` as the next step, but that
|
||||
route doesn't exist — `api/openapi.yaml` only has `/clients/enroll`,
|
||||
`/clients/{slug}/context`, `/clients/{slug}/secrets`. Until this is
|
||||
fixed (add the route, or use the generic entity PATCH to flip `state`),
|
||||
a freshly-enrolled client is stuck in `provisioning` — MCP `preflight`
|
||||
and policy's `lifecycle_overrides` for `provisioning` still apply, but
|
||||
nothing transitions it onward automatically.
|
||||
5. Verify: MCP `whoami(hostname)` shows the entity in `active` state with
|
||||
its peers and health.
|
||||
|
||||
Docs-update checklist: if the new host is an LXC/VM, add its narrative
|
||||
page under `containers/` or `vms/` and set `doc_page` in its inventory
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
name: config-change-deploy
|
||||
risk_class: config_mutation
|
||||
inputs: [service_name, change_description]
|
||||
verification: "curl -sf <service_url> (or homelab service <name> health)"
|
||||
verification: "curl -sf <service_url> (or MCP get_service_status)"
|
||||
docs_update_checklist: [doc_page, changelog]
|
||||
---
|
||||
|
||||
@@ -11,11 +11,10 @@ docs_update_checklist: [doc_page, changelog]
|
||||
Goal: change a tracked config repo (Caddy, Gitea customizations, an app's
|
||||
own repo) and get it live, safely.
|
||||
|
||||
1. `homelab change preflight <service>` — current health, the service's
|
||||
`config_repo`, its risk class, and the verification command to run
|
||||
after. If risk class requires approval (`config_mutation` or
|
||||
`destructive`), stop and get operator sign-off before editing — see
|
||||
`seeds/policy.yaml`.
|
||||
1. MCP `preflight` — current health, the service's `config_repo`, its
|
||||
risk class, and the verification command to run after. If risk class
|
||||
requires approval (`config_mutation` or `destructive`), stop and get
|
||||
operator sign-off before editing — see `seeds/policy.yaml`.
|
||||
2. Clone/pull the `config_repo` (never edit the backend's working tree
|
||||
directly — tracked configs change by commit + push, per
|
||||
[OIKOS.md](../../OIKOS.md) conventions).
|
||||
@@ -24,10 +23,10 @@ own repo) and get it live, safely.
|
||||
[infrastructure/auto-deploy.md](../../../archive/knowledge/infrastructure/auto-deploy.md) for
|
||||
the exact receiver/reload for this service).
|
||||
5. Run the preflight's verification command. If it fails, check
|
||||
`homelab service <name> log` for the reload/restart error.
|
||||
6. Record the change: once `oikos/ledger.py` is wired into deploy tooling
|
||||
(Week 3), this is automatic; until then, note the change and outcome
|
||||
in the relevant investigation/plan doc.
|
||||
MCP `tail_log` for the reload/restart error.
|
||||
6. No manual record-keeping step needed — mutations made through the API
|
||||
(e.g. via the `run` MCP tool) are recorded automatically in the
|
||||
`audit_log` table.
|
||||
|
||||
Docs-update checklist: update the service's `doc_page` if the change
|
||||
alters its behavior, ingress route, or ownership; add a changelog entry
|
||||
|
||||
@@ -10,22 +10,23 @@ docs_update_checklist: [investigations_entry]
|
||||
|
||||
Goal: understand what broke and why, before touching anything.
|
||||
|
||||
1. `homelab service <name> explain` (or `homelab node <name> relations`
|
||||
if the affected entity is a host) — get the blast radius and doc
|
||||
pointer first. Don't start pulling logs blind.
|
||||
2. `homelab service <name> health` + `homelab service <name> log` (or
|
||||
MCP `get_service_status` / `tail_log`) for the affected service.
|
||||
1. MCP `explain` (or `get_blast_radius` if the affected entity is a
|
||||
host) — get the blast radius and doc pointer first. Don't start
|
||||
pulling logs blind.
|
||||
2. MCP `get_service_status` + `tail_log` for the affected service.
|
||||
3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
|
||||
`authentik`, or the backend host itself)? `homelab node <name>
|
||||
relations` shows "affected by" — check those first.
|
||||
4. `homelab apt-audit` if the symptom looks like a dpkg/upgrade
|
||||
interaction.
|
||||
`authentik`, or the backend host itself)? MCP `get_blast_radius`
|
||||
shows "affected by" — check those first.
|
||||
4. If the symptom looks like a dpkg/upgrade interaction, see
|
||||
[runbook-dpkg-interrupted](../runbook-dpkg-interrupted/SKILL.md) —
|
||||
there's no fleet-wide apt-audit tool anymore, check the host directly.
|
||||
5. Check the change ledger for recent mutations to the affected entity
|
||||
or anything upstream of it: `homelab service <name> history` (once
|
||||
populated) or grep `ledger/*.jsonl`.
|
||||
6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom,
|
||||
timeline, root cause, fix applied, prevention. This is the durable
|
||||
record; don't rely on chat history.
|
||||
or anything upstream of it: MCP `get_change_history` or `get_audit_trail`.
|
||||
6. Write findings via MCP `upsert_knowledge` (`kind: investigation`) —
|
||||
symptom, timeline, root cause, fix applied, prevention, `about` set to
|
||||
the affected entity's slug. The DB is the durable record now, not a
|
||||
markdown file — `search_knowledge`/`get_entity_knowledge` read it back;
|
||||
a chat message alone is forgotten.
|
||||
|
||||
Docs-update checklist: always create the investigation entry. If the
|
||||
root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
name: lifecycle-activate-node
|
||||
risk_class: config_mutation
|
||||
inputs: [node_name]
|
||||
verification: "homelab service <name> health (if it hosts a service); homelab doctor (if it's a client)"
|
||||
verification: "MCP get_service_status (if it hosts a service); MCP whoami (if it's a client)"
|
||||
docs_update_checklist: [doc_page_complete]
|
||||
transition: "provisioning -> active"
|
||||
---
|
||||
@@ -14,23 +14,22 @@ enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
|
||||
ingress live if public, health check answering, doc page complete,
|
||||
ledger entry.
|
||||
|
||||
1. If the node is a `homelab` client: finish enrollment per
|
||||
[client-enrollment.md](../client-enrollment/SKILL.md) (`--finalize-pubkey`,
|
||||
mesh join, `homelab doctor` green).
|
||||
1. If the node self-enrolls as a client: finish enrollment per
|
||||
[CLIENTS.md](../../../CLIENTS.md#enrollment) (`bootstrap.sh` →
|
||||
`/api/v1/clients/enroll`, mesh join, MCP `whoami` returns the entity).
|
||||
2. If it hosts a public service: add the `services:` entry in
|
||||
`inventory.yaml` (backend, url, doc_page, config_repo, risk_notes —
|
||||
see the Week-1 service contract fields) and wire the Caddy route in
|
||||
`dtoro/caddy-conf`.
|
||||
3. Confirm the health check answers: `homelab service <name> health` or
|
||||
a direct `curl`.
|
||||
`seeds/inventory.yaml` (backend, url, doc_page, config_repo,
|
||||
risk_notes) and wire the Caddy route in `dtoro/caddy-conf`.
|
||||
3. Confirm the health check answers: MCP `get_service_status` or a
|
||||
direct `curl`.
|
||||
4. Flip `state: provisioning` → `state: active` (or delete the `state:`
|
||||
field — `active` is the default) in `inventory.yaml`.
|
||||
field — `active` is the default) in `seeds/inventory.yaml`, then
|
||||
`oikos seed` to ingest.
|
||||
5. Complete the doc page (stub → full narrative: role, specs, how it's
|
||||
configured, dependencies).
|
||||
6. Record the activation: `oikos/ledger.py append host:<name> activate
|
||||
config_mutation --result ok` (or let the CLI wrapper do this once
|
||||
Week 3's runbook automation lands).
|
||||
6. No manual record-keeping step needed — the activation (via whatever
|
||||
API call flipped the state) is recorded automatically in `audit_log`.
|
||||
|
||||
Regenerate derived data: `python3 mcp/build_host_files.py && python3
|
||||
inventory.yaml` so `inventory.yaml`, the topology diagram, and
|
||||
the context card all reflect the new state.
|
||||
Regenerate: `oikos seed` re-ingests `seeds/inventory.yaml`; `oikos export`
|
||||
writes DB state back out to the YAML if you mutated via the API/MCP
|
||||
instead of editing the file directly.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
name: lifecycle-deprecate-node
|
||||
risk_class: config_mutation
|
||||
inputs: [node_name, replacement_node_or_reason]
|
||||
verification: "homelab node <name> relations — 'affected by' must be empty before completing"
|
||||
verification: "MCP get_blast_radius — 'affected by' must be empty before completing"
|
||||
docs_update_checklist: [doc_page_deprecation_note]
|
||||
transition: "active -> deprecated"
|
||||
---
|
||||
@@ -16,14 +16,14 @@ suggestion; `seeds/policy.yaml` `lifecycle_overrides.deprecated.refuse`
|
||||
lists `new-inbound-edges` as refused going forward.
|
||||
|
||||
1. Set `state: deprecated` on the node.
|
||||
2. `homelab node <name> relations` — read `affected_by`. Every entry
|
||||
there is something still relying on this node.
|
||||
2. MCP `get_blast_radius` — read `affected_by`. Every entry there is
|
||||
something still relying on this node.
|
||||
3. Migrate or retire each dependent one at a time (point its `backend`/
|
||||
`config_repo`/ingress route elsewhere, or deprecate it too if it's
|
||||
being retired alongside).
|
||||
4. Re-run `homelab node <name> relations` after each dependent is moved.
|
||||
The transition to `destroyed` is only safe once `affected_by` is
|
||||
empty — check this every time, don't assume from memory.
|
||||
4. Re-run MCP `get_blast_radius` after each dependent is moved. The
|
||||
transition to `destroyed` is only safe once `affected_by` is empty —
|
||||
check this every time, don't assume from memory.
|
||||
5. Note the deprecation on the doc page: reason, replacement (if any),
|
||||
date.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
name: lifecycle-destroy-node
|
||||
risk_class: destructive
|
||||
inputs: [node_name]
|
||||
verification: "homelab node <name> relations returns unknown-entity; pct list on the backend no longer shows it"
|
||||
verification: "MCP get_blast_radius returns unknown-entity; pct list on the backend no longer shows it"
|
||||
docs_update_checklist: [archaeology_entry, containers_index_update]
|
||||
transition: "deprecated -> destroyed"
|
||||
---
|
||||
@@ -15,28 +15,35 @@ recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
|
||||
ledger entry.
|
||||
|
||||
1. Confirm the node is `deprecated` with zero `affected_by` edges
|
||||
(`homelab node <name> relations`) — do not skip this even if the
|
||||
deprecation runbook was followed recently; state can drift.
|
||||
2. If it's an enrolled client: `homelab client remove <name>` — revokes
|
||||
the age key, re-keys SOPS, removes the inventory entry. This is
|
||||
already destructive-class and confirmed in the CLI.
|
||||
(MCP `get_blast_radius`) — do not skip this even if the deprecation
|
||||
runbook was followed recently; state can drift.
|
||||
2. **If it's an enrolled client: no current tool for revoking its age key /
|
||||
removing its Infisical identity.** The old `homelab client remove`
|
||||
(age key revocation + SOPS re-key + inventory removal, all one
|
||||
destructive-class CLI call) is retired along with the rest of that CLI
|
||||
and hasn't been re-verified against the current enrollment
|
||||
architecture (`POST /api/v1/clients/enroll` + Infisical machine
|
||||
identities) — see the "Open questions" section in
|
||||
[agent-enrollment.md](../../operations/agent-enrollment.md). Until
|
||||
that's confirmed, treat key/identity revocation as a manual step:
|
||||
at minimum remove the client's `age_pubkey` from any SOPS recipient
|
||||
lists and rotate credentials whose ciphertext it already decrypted.
|
||||
3. Remove any ingress route (Caddy config repo) and DNS record still
|
||||
pointing at it.
|
||||
4. Verify backups of anything on it are retained per policy before the
|
||||
disk goes away (see `backs-up-to`).
|
||||
5. Destroy the LXC/VM (`pct destroy` / `qm destroy`).
|
||||
6. Move the `hosts.<name>:` block (if any inventory remnant survives
|
||||
`client remove`, e.g. infra-only LXCs with no age key) into
|
||||
inventory.yaml's `archaeology:` section: `pve_id`, `destroyed` date,
|
||||
`reason`. Add a row to `containers/index.md` "Recently destroyed"
|
||||
table (kept for human-readable browsing alongside the structured
|
||||
data).
|
||||
7. `oikos/ledger.py append host:<name> destroy destructive --result ok`.
|
||||
8. Regenerate: `python3 mcp/build_host_files.py && python3
|
||||
inventory.yaml` — the node drops out of `inventory.yaml` and
|
||||
appears in the topology doc's archaeology table.
|
||||
6. Update the entity's `state` to `destroyed` in `seeds/inventory.yaml`
|
||||
(or move it to an `archaeology:`-style section if the schema still has
|
||||
one) — `pve_id`, `destroyed` date, `reason` — then `oikos seed` to
|
||||
ingest. Add a row to `containers/index.md` "Recently destroyed" table
|
||||
(kept for human-readable browsing alongside the structured data).
|
||||
7. No manual ledger step — mutations through the API are recorded
|
||||
automatically in the `audit_log` table (MCP `get_audit_trail`,
|
||||
`get_change_history`). The old `oikos/ledger.py append` was retired
|
||||
when this became automatic.
|
||||
|
||||
If the destroy fails partway (e.g. secrets revoked but pct destroy
|
||||
errors), do not re-run step 2 — `client remove` is not idempotent
|
||||
against a second revocation attempt on the issuance server. Finish the
|
||||
remaining steps manually and note the partial state in an investigation.
|
||||
If the destroy fails partway (e.g. secrets not fully revoked but pct
|
||||
destroy errors), finish the remaining steps manually and note the
|
||||
partial state in an investigation (MCP `upsert_knowledge`,
|
||||
`kind: investigation`).
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
name: lifecycle-migrate-node
|
||||
risk_class: config_mutation
|
||||
inputs: [node_name, source_host, target_host]
|
||||
verification: "homelab node <name> relations (re-check blast radius); homelab service <svc> health for every hosted service"
|
||||
verification: "MCP get_blast_radius (re-check blast radius); MCP get_service_status for every hosted service"
|
||||
docs_update_checklist: [doc_page_migration_note, inventory_host_and_lan_ip]
|
||||
transition: "active -> migrating -> active"
|
||||
---
|
||||
@@ -10,12 +10,12 @@ transition: "active -> migrating -> active"
|
||||
# Lifecycle: migrate a node
|
||||
|
||||
Modeled on the strong Phase 1+2 migration
|
||||
([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
|
||||
([archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
|
||||
Requires (ontology): preflight + backup-verified before migrating;
|
||||
post-verify + Caddy backends checked + mounts checked + docs updated
|
||||
before returning to `active`.
|
||||
|
||||
1. `homelab change preflight <every service the node hosts>` — capture
|
||||
1. MCP `preflight` for every service the node hosts — capture
|
||||
current health as a baseline.
|
||||
2. Verify backups are current for anything with data at rest on the
|
||||
node (see `backs-up-to` edges once populated).
|
||||
@@ -29,11 +29,12 @@ before returning to `active`.
|
||||
6. Post-verify: re-run the Week-1 drift check by hand — confirm Caddy's
|
||||
backend IP for each affected service matches the new `lan_ip`
|
||||
(automatic in Week 3's drift detector), confirm mounts still resolve.
|
||||
7. `homelab service <name> health` for every service the node hosts.
|
||||
7. MCP `get_service_status` for every service the node hosts.
|
||||
8. Set `state: active`. Add a migration note to the node's doc page
|
||||
(old host/IP → new, date, phase reference) — this repo's convention
|
||||
for every past migration (see `archive/knowledge/containers/101-jellyfin.md`,
|
||||
`containers/129-house.md`).
|
||||
|
||||
Regenerate: `python3 mcp/build_host_files.py && python3
|
||||
inventory.yaml`.
|
||||
Regenerate: `oikos seed` (re-ingests `seeds/inventory.yaml` into the DB —
|
||||
the DB is authoritative at runtime, the YAML is the source of truth
|
||||
on disk).
|
||||
|
||||
@@ -21,10 +21,12 @@ chosen, doc page stub.
|
||||
`qm create`), choosing the storage pool deliberately — record it as
|
||||
the `storage:` field once populated (Week 1 schema; not yet backfilled
|
||||
for existing nodes).
|
||||
2. Add the inventory entry: `homelab client add <name>` for anything that
|
||||
will run the `homelab` CLI, or a direct `hosts.<name>:` block with
|
||||
`state: provisioning`, `kind`, `host`, `pve_id`, `lan_ip` for
|
||||
infra-only LXCs that won't self-enroll.
|
||||
2. Add the inventory entry: a `hosts.<name>:` block in
|
||||
`seeds/inventory.yaml` with `state: provisioning`, `kind`, `host`,
|
||||
`pve_id`, `lan_ip`, then `oikos seed` to ingest it. For anything that
|
||||
will self-enroll as a client afterward (see
|
||||
[CLIENTS.md](../../../CLIENTS.md#enrollment)), the entity must exist in
|
||||
`planned`/`provisioning` state before `bootstrap.sh` runs there.
|
||||
3. Stub the doc page (`containers/<pve_id>-<name>.md` or
|
||||
`vms/<pve_id>-<name>.md`) — even a one-line "provisioning, see plan X"
|
||||
is enough to satisfy the transition requirement.
|
||||
|
||||
@@ -18,7 +18,7 @@ summarised into targets and fixed costs.
|
||||
- `yuvomi-mcp` is running on LXC 129 and connected as an MCP server in Claude.
|
||||
- The CSV is an N26 export (columns: Booking Date, Value Date, Partner Name,
|
||||
Partner Iban, Type, Payment Reference, Account Name, Amount (EUR), …).
|
||||
- API token: `homelab secret yuvomi-api-token` (decrypts on any enrolled client).
|
||||
- API token: `yuvomi-api-token`, via Infisical (primary) or `oikos secret` (SOPS fallback).
|
||||
- Direct API base: `https://house.hubris.network/api/v1`
|
||||
|
||||
---
|
||||
|
||||
@@ -13,7 +13,8 @@ has packages that are **unpacked but not configured**. Symptoms:
|
||||
manually run 'dpkg --configure -a' to correct the problem.`
|
||||
- `dpkg --audit` lists packages with header
|
||||
`The following packages have been unpacked but not yet configured.`
|
||||
- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host.
|
||||
- `dpkg --audit` on the host directly shows unpacked-not-configured packages
|
||||
(there's no fleet-wide audit tool anymore — check per-host).
|
||||
|
||||
The system is still running the **old** binaries (still in memory), but the
|
||||
**new** binaries are unpacked and waiting for their postinst to run. Two
|
||||
@@ -33,19 +34,20 @@ config dirs, capabilities, etc.). The system might not come back up cleanly.
|
||||
## Path A — target is still reachable over ssh (preferred)
|
||||
|
||||
```
|
||||
homelab ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
|
||||
ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
|
||||
```
|
||||
|
||||
Or for an LXC by name:
|
||||
Or for an LXC by name (via the MCP `run` tool, or directly on the Proxmox
|
||||
host):
|
||||
|
||||
```
|
||||
homelab pct <lxc> exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
|
||||
pct exec <lxc> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
|
||||
```
|
||||
|
||||
When that returns, confirm:
|
||||
|
||||
```
|
||||
homelab apt-audit --target <host>
|
||||
ssh <host> -- dpkg --audit
|
||||
```
|
||||
|
||||
Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally
|
||||
@@ -89,10 +91,11 @@ DEBIAN_FRONTEND=noninteractive dpkg --configure -a \
|
||||
|
||||
## Prevention
|
||||
|
||||
The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect`
|
||||
unit on the target, so it survives ssh teardown — the failure mode that put
|
||||
hubris into this state in the first place is no longer reachable through the
|
||||
standard tool. If you absolutely need to run apt manually over ssh, wrap it:
|
||||
The old `homelab apt-upgrade` wrapper (retired along with the rest of the
|
||||
`homelab` CLI) used to launch apt inside a `systemd-run --collect` unit on
|
||||
the target so it survived ssh teardown — that's the failure mode that put
|
||||
hubris into this state in the first place. There's no fleet-wide wrapper
|
||||
anymore; if you run apt manually over ssh, wrap it yourself the same way:
|
||||
|
||||
```
|
||||
ssh <host> systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade'
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
name: service-health-check
|
||||
risk_class: read_only
|
||||
inputs: [service_name]
|
||||
verification: "homelab service <name> health"
|
||||
verification: "MCP get_service_status"
|
||||
docs_update_checklist: []
|
||||
---
|
||||
|
||||
@@ -10,15 +10,15 @@ docs_update_checklist: []
|
||||
|
||||
Goal: determine whether a service is actually healthy, without ad-hoc SSH.
|
||||
|
||||
1. `homelab service <name> explain` — read the context card: backend,
|
||||
blast radius, doc pointer, risk notes.
|
||||
2. `homelab service <name> health` — live health probe (HTTP code against
|
||||
the service's `url`/`endpoint`). Once the Week-3 scheduler ships, this
|
||||
reads a cached snapshot by default; pass `--live` to force a fresh probe.
|
||||
3. If unhealthy, `homelab service <name> log` (or MCP `tail_log`) for the
|
||||
last 200 lines.
|
||||
4. Cross-check blast radius: `homelab node <name> relations` — is this
|
||||
entity's own backend host healthy? A downstream failure (e.g. `strong`
|
||||
1. MCP `explain` — read the context card: backend, blast radius, doc
|
||||
pointer, risk notes.
|
||||
2. MCP `get_service_status` — live health probe (HTTP code against the
|
||||
service's `url`/`endpoint`); the scheduler also probes on its own
|
||||
interval, so this may reflect a recent cached result, not necessarily
|
||||
a fresh one.
|
||||
3. If unhealthy, `tail_log` for the last 200 lines.
|
||||
4. Cross-check blast radius: MCP `get_blast_radius` — is this entity's
|
||||
own backend host healthy? A downstream failure (e.g. a Proxmox host
|
||||
down) will show up here before the service's own logs explain anything.
|
||||
5. If the fix is a restart: classify first (`seeds/policy.yaml` —
|
||||
`service-restart` is `reversible_low` unless the service has a
|
||||
|
||||
Reference in New Issue
Block a user