docs: add client lifecycle plan, cleanup stale files, document repo for 3 audiences

Problem: Repo had no developer guide, no client onboarding doc, no agent dev instructions. Stale files (675KB SQL dump, one-off convert script, legacy MCP builder) cluttered the tree. Client enrollment was a documented intention with no Go implementation. Changes: - New docs: CONTRIBUTING.md (dev setup), CLIENTS.md (client onboarding), .agents/dev/CONTRIBUTING.md (agent codebase map) - New plan: plans/2026-07-07-client-lifecycle-in-go.md — full client lifecycle (planned→provisioning→active→deprecated→destroyed) in Go, replacing archived Python secrets-issuance, adding client API endpoints and 6 missing MCP tools - Cleanup: deleted archive/convert-wiki.py (one-off), archive/mcp/ build_host_files.py (legacy), backups/pre-deploy-7f7d039.sql (local) - Fixes: plans/index.md duplicate row removed, README.md repo layout updated for current state, AGENTS.md header points to new guides Risk: low. Docs only + stale file deletion. No code changes. New plan is proposal, not implementation. Verification: git diff reviewed, all changes are prose/docs/plans.
2026-07-07 23:45:32 +02:00
parent 85b541a1cc
commit 638e313c66
9 changed files with 921 additions and 563 deletions
--- a/plans/2026-07-07-client-lifecycle-in-go.md
+++ b/plans/2026-07-07-client-lifecycle-in-go.md
@@ -0,0 +1,451 @@
+# Plan: Client lifecycle — enrollment through deprecation in Oikos Go
+
+**Status:** Planned (2026-07-07)
+
+## Goal
+
+Define and implement the complete lifecycle of a homelab client in the Oikos Go
+runtime: how a new machine is provisioned, enrolled, given secrets, synced,
+operated, and eventually deprecated (or decommissioned or destroyed). Every
+state transition feeds the Postgres DB as the authoritative source of truth.
+No step depends on the archived Python `secrets-issuance` server or the
+non-existent `bin/homelab` CLI.
+
+## Current state — what exists vs. what runs
+
+| Component | Exists? | Runs? | Notes |
+|-----------|---------|-------|-------|
+| `bootstrap.sh` (684 lines) | ✅ repo | ⚠️ references dead endpoints | Calls `https://secrets.hubris.network/issue` (Python server, stopped per Phase 6). Symlinks `bin/homelab` (file absent). References `tools/*.setup.sh` (files absent). |
+| `archive/secrets-issuance/server.py` | ✅ archive | ❌ stopped (Phase 6) | Issued age keys, validated mesh IP. No Go replacement. |
+| `archive/mcp/` (build_host_files.py, deleted) | ❌ deleted | ❌ | Legacy host file builder. |
+| `inventory.yaml` (root) | ✅ | ⚠️ edited manually | Flat `hosts:` + `services:` layout. Diverges from `seeds/inventory.yaml` entity-relationship format. |
+| `seeds/inventory.yaml` | ✅ | ✅ ingested into DB | Entity-relationship format with slugs (`host:hubris`, `ws:mac-mini`). No translation path from root format. |
+| Go entity API (`POST/GET/PATCH /entities`) | ✅ | ✅ | Generic CRUD. No client-specific validation, no key issuance, no lifecycle gating. |
+| Go MCP server | ✅ | ✅ | 15 tools. Missing `whoami`, `explain`, `preflight`, `get_change_history`, `get_state_snapshot`. |
+| `oikos secret` CLI (Infisical/SOPS) | ✅ | ✅ | Secrets read/migrate/export. No client-key provisioning. |
+| Sync timer (post-pull.sh) | ✅ | ⚠️ partially broken | References `tools/*.setup.sh` (glob returns zero files). `setup-caveman.sh` and `setup-hermes-soul.sh` documented but absent. |
+
+**Takeaway**: Enrollment today runs on shell scripts calling archived Python
+services. The Go runtime has zero awareness of client lifecycle. This plan
+closes that gap — the DB becomes the sole engine for client identity,
+secrets, state, and lifecycle transitions.
+
+## Target architecture
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│                    NEW CLIENT (bare machine)                  │
+│                                                              │
+│  1. curl bootstrap.sh | sudo bash                            │
+│     → clones repo, installs sync timer                        │
+│     → calls POST /api/v1/clients/enroll (new endpoint)        │
+│     → receives age keypair from Oikos API                     │
+│     → writes /etc/age/key.txt                                 │
+│     → sync timer starts pulling every 5 min                   │
+└──────────────────────────┬───────────────────────────────────┘
+                           │ POST /api/v1/clients/enroll
+                           ▼
+┌──────────────────────────────────────────────────────────────┐
+│                    OIKOS API (Go, :8090)                      │
+│                                                              │
+│  POST /api/v1/clients/enroll     — issue age key, set state   │
+│  POST /api/v1/clients/{slug}/activate — provisioning→active  │
+│  POST /api/v1/clients/{slug}/deprecate — active→deprecated   │
+│  POST /api/v1/clients/{slug}/destroy  — deprecated→destroyed │
+│  GET  /api/v1/clients/{slug}/secrets — client's accessible   │
+│        secrets (Infisical lookup by machine identity)         │
+│  MCP  whoami(hostname)           — client self-introspection │
+│  MCP  explain(service)           — compact context card      │
+│  MCP  preflight(service)         — risk classification       │
+│  MCP  get_change_history(entity) — ledger entries            │
+│  MCP  get_state_snapshot()       — last scheduler pass       │
+└──────────────────────────┬───────────────────────────────────┘
+                           │ writes
+                           ▼
+┌──────────────────────────────────────────────────────────────┐
+│                    POSTGRES (TimescaleDB)                     │
+│                                                              │
+│  entities table: slug, type, name, state, attributes (JSONB) │
+│  entity_status: health, disk, drift count (scheduler)        │
+│  audit_log: every state transition, enrollment, revocation   │
+│  executions: approved actions, results                       │
+│  secrets (via Infisical): age keys, API tokens               │
+└──────────────────────────────────────────────────────────────┘
+```
+
+## Client lifecycle: state machine
+
+```
+  [planned] ──→ provisioning ──→ active ──→ migrating ──→ active
+      │              │               │
+      │              │               ├──→ deprecated ──→ destroyed
+      │              │               │
+      │              └──→ failed     └──→ failed
+      │
+      └──→ destroyed (cancelled)
+```
+
+### State: `planned`
+
+The operator declares intent. A client entity exists in the DB with state
+`planned` but has no host, no keys, no sync.
+
+**Entry condition**: Operator creates the entity via API or seed file.
+
+**Required attributes**:
+- `slug` — `ws:<hostname>` for workstations, `host:<hostname>` for servers
+- `type` — `workstation`, `standalone-server`, or `proxmox-host`
+- `name` — human-readable name
+- `lan_ip` — expected LAN IP (reserved in DHCP)
+- `os` — `linux` or `macos`
+- `role` — free-text description of what this machine does
+- `mesh.expected_type` — `netbird` or `tailscale` (which mesh it will join)
+- `ssh.user` — login user (default `root`)
+
+**Allowed transitions**: `→ provisioning` (operator triggers), `→ destroyed` (cancelled).
+
+### State: `provisioning`
+
+The machine has been declared. Operator runs `bootstrap.sh` on the target,
+which calls the enrollment API. The API validates identity (mesh IP matches
+expected subnet, hostname matches slug), issues an age keypair, and records
+the public key. The sync timer starts pulling the repo.
+
+**Go API**: `POST /api/v1/clients/enroll`
+```json
+{
+  "slug": "ws:new-laptop",
+  "hostname": "new-laptop",
+  "mesh_ip": "100.122.x.x"
+}
+```
+
+**What the enrollment endpoint does**:
+1. Looks up entity by slug — must exist, must be in state `planned` or `provisioning`
+2. Validates mesh IP is in `100.122.0.0/16` (Netbird) or `100.64.0.0/10` (Tailscale) or `192.168.8.0/24` (LAN)
+3. Validates hostname has no conflicting mesh IP already recorded
+4. Generates an age keypair (`age-keygen`)
+5. Stores the private key in Infisical under path `/clients/<slug>/age-key`
+6. Creates an Infisical machine identity for the client (UniversalAuth)
+7. Updates entity `attributes` with `age_pubkey`, `mesh_ip`, `enrolled_at`
+8. Writes audit log: `client.enrolled`
+9. Returns the age private key, Infisical client ID + secret, and machine identity token
+
+**Response** (to bootstrap.sh, over mesh — TLS + mesh IP validation):
+```json
+{
+  "age_private_key": "AGE-SECRET-KEY-...",
+  "age_public_key": "age1...",
+  "infisical_client_id": "...",
+  "infisical_client_secret": "...",
+  "machine_identity_token": "..."
+}
+```
+
+**Bootstrap script changes**:
+- Remove call to `https://secrets.hubris.network/issue`
+- Replace with `POST /api/v1/clients/enroll` to `https://oikos.hubris.network`
+- Remove `--no-secrets` / `--no-mesh` flags (or keep as escape hatches with degraded state)
+- Remove symlink to `bin/homelab` (file doesn't exist)
+- After receiving keys, bootstrap.sh writes `/etc/age/key.txt` (0600) and `/etc/infisical/identity` (0600)
+
+**Pre-built bootstrap**: The bootstrap.sh is served from the Gitea repo raw URL
+(already the case). After this plan, it calls Oikos API instead of the dead
+Python service.
+
+**Allowed transition**: `→ active` (when `age-key-enrolled`, `mesh-joined`,
+`doc-page-complete` checks pass).
+
+### State: `active`
+
+Normal operation. The client pulls the repo every 5 minutes, uses its age key
+to decrypt SOPS secrets (fallback), and authenticates to Infisical via its
+machine identity (primary). The MCP `whoami(hostname)` tool returns its
+entity record, peer list, accessible secrets, and current health.
+
+**Go enforcement of transition checks** (`provisioning → active`):
+- `age-key-enrolled-if-needed`: entity.attributes.age_pubkey is non-empty
+- `mesh-joined-if-needed`: entity.attributes.mesh_ip is non-empty
+- `ingress-live-if-public`: skipped for workstations (no public ingress)
+- `health-check-answering`: scheduler probe passes for this entity
+- `doc-page-complete`: entity has at least one `documents` edge
+- `inventory-in-db`: entity exists in DB with all required attributes
+
+**API**: `POST /api/v1/clients/{slug}/activate`
+- Validates all `provisioning → active` transition checks
+- Sets state to `active`
+- Writes audit log: `client.activated`
+
+**MCP tools active clients get**:
+- `whoami(hostname)` — returns entity record, peers, secrets list, health
+- `list_my_secrets(caller_pubkey?)` — secrets this client can decrypt
+
+**Allowed transitions**: `→ migrating`, `→ deprecated`, `→ failed`.
+
+### State: `migrating`
+
+Client is being moved — OS reinstall, hardware swap, role change. Inbound edges
+still exist; no deprovisioning has started.
+
+**Allowed transition**: `→ active` (migration complete, post-verify passes).
+
+**API**: `POST /api/v1/clients/{slug}/migrate` (sets state, links migration plan).
+
+### State: `deprecated`
+
+Client is being phased out. Services moved off, mesh disconnected, secrets
+rotation started. The deprecation gate (`no-inbound-edges`) blocks `→ destroyed`
+until all `depends-on`, `hosts`, `provides`, and `mounts` edges are gone.
+
+**API**: `POST /api/v1/clients/{slug}/deprecate`
+- Validates `replacement-live-or-role-retired`: operator confirms replacement exists or role is no longer needed
+- Sets state to `deprecated`
+- Writes audit log: `client.deprecated`
+
+**Allowed transitions**: `→ active` (un-deprecate), `→ destroyed`.
+
+### State: `destroyed`
+
+Client is gone. All edges removed, secrets revoked, archaeology entry written.
+
+**API**: `POST /api/v1/clients/{slug}/destroy`
+- Validates all `deprecated → destroyed` transition checks:
+  - `backups-verified`: any data on this client was backed up
+  - `secrets-revoked-and-rekeyed`: age key removed from Infisical, SOPS recipients updated, machine identity deleted
+  - `ingress-and-dns-removed`: no remaining DNS records or Caddy backends
+  - `no-inbound-edges`: zero `depends-on`, `hosts`, `provides`, `mounts` edges pointing to this entity
+  - `archaeology-entry`: writes a record explaining why and when
+- Sets state to `destroyed`
+- Revokes Infisical machine identity
+- Removes age public key from `.sops.yaml`
+- Writes audit log: `client.destroyed`
+
+### State: `failed`
+
+Something went wrong during provisioning or operation. Requires operator
+intervention. Treated as informational — no automatic recovery.
+
+**API**: `POST /api/v1/clients/{slug}/fail`
+- Sets state to `failed`
+- Requires `reason` field explaining what broke
+- Writes audit log: `client.failed`
+
+## Secrets integration
+
+### Age key lifecycle
+
+```
+planned ────────────→ no key exists
+provisioning ───────→ keypair generated, pubkey stored in entity attributes,
+                      private key delivered to client via enroll response,
+                      private key stored in Infisical under /clients/<slug>/age-key
+active ─────────────→ key used for SOPS decryption fallback, authenticated
+                      to Infisical via machine identity for primary secrets
+deprecated ─────────→ key still valid, but rotation initiated
+destroyed ──────────→ key revoked from Infisical, removed from .sops.yaml,
+                      machine identity deleted
+```
+
+### Infisical machine identity
+
+Each client gets an Infisical machine identity during enrollment. This is the
+primary secrets path — the age key is fallback for SOPS-encrypted DR files.
+
+- **Client ID + Secret** returned in enroll response
+- **Scoped to paths**: `/clients/<slug>/*`, `/shared/*`
+- **Revoked on destroy**: identity deleted, access gone
+
+### SOPS fallback
+
+The age public key is added to `.sops.yaml` recipients during enrollment.
+On destroy, it is removed via `oikos secret export-sops` regeneration.
+
+### bootstrap.sh changes
+
+```diff
+- # calls https://secrets.hubris.network/issue (Python, dead)
+- AGE_KEY=$(curl -s -X POST "$ISSUANCE_URL" ...)
+- 
+ # calls Oikos API enrollment endpoint
+ ENROLL_RESP=$(curl -s -X POST "$OIKOS_URL/api/v1/clients/enroll" \
+   -H "Content-Type: application/json" \
+   -d "{\"slug\":\"ws:$HNAME\",\"hostname\":\"$HNAME\",\"mesh_ip\":\"$MESH_IP\"}")
+ AGE_PRIVKEY=$(echo "$ENROLL_RESP" | jq -r '.age_private_key')
+```
+
+## DB integration
+
+### New migration
+
+`012_client_enrollment.up.sql`:
+```sql
+-- No new tables needed — entities table already holds clients.
+-- Add enrollment-specific attributes validation via check constraints
+-- or application-level validation.
+
+-- Enforce slug format for machine entities
+-- ws:<hostname> for workstations, host:<hostname> for servers
+-- (application-level validation in Go, not a DB constraint)
+
+-- Add index for slug-based client lookups
+CREATE INDEX IF NOT EXISTS idx_entities_slug_type
+  ON entities (slug, type)
+  WHERE type IN ('workstation', 'standalone-server', 'proxmox-host');
+```
+
+### Entity attributes schema (for `machine` types)
+
+```json
+{
+  "cpu_arch": "arm64",
+  "ram_gb": 16,
+  "os": "macos",
+  "lan_ip": "192.168.8.175",
+  "mesh": {
+    "netbird": {"ip": "100.122.x.x", "fqdn": "hostname.netbird.selfhosted"}
+  },
+  "ssh": {"user": "dtoro"},
+  "age_pubkey": "age1...",
+  "enrolled_at": "2026-07-07T12:00:00Z",
+  "enrolled_by": "ws:mac-mini",
+  "infisical_identity_id": "identity_abc123"
+}
+```
+
+All attributes are stored in the `attributes` JSONB column on the `entities`
+table. Validation happens at the application layer (Go) using the schema
+defined in `seeds/ontology.yaml`.
+
+## MCP tools to add
+
+These are documented in AGENTS.md section 3 but not implemented in the Go MCP
+server. Implementation: register in `internal/mcp/server.go`.
+
+| Tool | Input | Output | Implementation |
+|------|-------|--------|----------------|
+| `whoami` | `hostname` | Entity record, peers, accessible secrets, health | DB lookup by slug derived from hostname |
+| `list_my_secrets` | `caller_pubkey?` | Secrets this client can decrypt | Infisical list + SOPS `.sops.yaml` match |
+| `explain` | `service_slug` | Compact context card: type, state, health, relations, last change | DB join: entity + entity_status + audit_log |
+| `preflight` | `service_slug` | Risk class, approval requirement, verification command | Policy classifier on the entity's type |
+| `get_change_history` | `entity_slug`, `limit` | Last N audit_log entries for entity | DB query on audit_log table |
+| `get_state_snapshot` | none | Last scheduler Observe pass: health, disk, drift count | DB query on entity_status + signals |
+
+## API endpoints to add
+
+Add to `api/openapi.yaml`, regenerate with `make generate`, implement in
+`internal/httpapi/impl.go`.
+
+| Method | Path | Scope | Purpose |
+|--------|------|-------|---------|
+| `POST` | `/api/v1/clients/enroll` | agent | Issue age key, validate mesh, set state → provisioning |
+| `POST` | `/api/v1/clients/{slug}/activate` | operator | Run transition checks, state → active |
+| `POST` | `/api/v1/clients/{slug}/deprecate` | operator | State → deprecated |
+| `POST` | `/api/v1/clients/{slug}/destroy` | operator | Run destroy checks, revoke secrets, state → destroyed |
+| `POST` | `/api/v1/clients/{slug}/fail` | operator | State → failed with reason |
+| `GET` | `/api/v1/clients/{slug}/secrets` | agent | List secrets this client can access |
+
+## Files changed
+
+| File | Change |
+|------|--------|
+| `bootstrap.sh` | Replace `secrets.hubris.network/issue` call with `POST /api/v1/clients/enroll`. Remove dead symlinks. |
+| `api/openapi.yaml` | Add client enrollment, lifecycle, and secret endpoints |
+| `internal/httpapi/impl.go` | Implement client lifecycle handlers |
+| `internal/db/queries/clients.sql` | Add client-specific sqlc queries |
+| `internal/mcp/server.go` | Register whoami, explain, preflight, get_change_history, get_state_snapshot, list_my_secrets |
+| `internal/secrets/infisical.go` | Add `CreateMachineIdentity`, `DeleteMachineIdentity`, `StoreClientKey` |
+| `internal/ontology/validate.go` | Implement lifecycle transition checks for infrastructure lifecycle |
+| `seeds/ontology.yaml` | Add client-specific attributes schema for machine types |
+| `migrations/012_client_enrollment.up.sql` | Index for slug+type lookups |
+| `AGENTS.md` | Update MCP tool list to match actual implementation |
+| `CLIENTS.md` | Update enrollment flow to reference Oikos API, not Python issuance |
+| `CONTRIBUTING.md` | Add client lifecycle as a documented extension point |
+
+## Files deleted or deprecated
+
+| File | Disposition |
+|------|-------------|
+| `archive/secrets-issuance/` | Already archived. Add deprecation notice referencing this plan. |
+| `archive/secrets-sops-backup/` | Keep for DR. Add note that new clients use Infisical, SOPS is fallback. |
+| Any reference to `bin/homelab` | Delete or comment out in bootstrap.sh; CLI doesn't exist. |
+| `tools/*.setup.sh` references | Either create the files or remove the auto-setup convention from post-pull.sh. |
+
+## Phased implementation
+
+### Phase 1 — API + DB (P0, this week)
+
+1. Write `migrations/012_client_enrollment.up.sql`
+2. Add client endpoints to `api/openapi.yaml`
+3. Run `make generate`
+4. Implement enrollment handler (`POST /api/v1/clients/enroll`):
+   - Age key generation
+   - Infisical machine identity creation
+   - Entity attribute update
+   - Audit log write
+5. Implement lifecycle transition handlers (activate, deprecate, destroy, fail)
+6. Implement `GET /api/v1/clients/{slug}/secrets`
+7. Update `seeds/ontology.yaml` with client attribute schemas
+8. Add sqlc queries in `internal/db/queries/clients.sql`
+
+### Phase 2 — MCP tools (P1, next week)
+
+1. Register `whoami(hostname)` in `internal/mcp/server.go`
+2. Register `explain(service)` — compact context card from DB
+3. Register `preflight(service)` — risk classification
+4. Register `get_change_history(entity, limit)`
+5. Register `get_state_snapshot()`
+6. Register `list_my_secrets(caller_pubkey?)`
+
+### Phase 3 — Bootstrap script cleanup (P1, next week)
+
+1. Replace secrets issuance URL with Oikos API endpoint
+2. Remove `--no-secrets` / `--no-mesh` or rewire them to degraded modes
+3. Remove `bin/homelab` symlink
+4. Update Infisical identity file creation
+5. Test full enrollment on a fresh machine
+
+### Phase 4 — Transition check enforcement (P2, within 2 weeks)
+
+1. Implement all `provisioning → active` checks in `internal/ontology/validate.go`
+2. Implement all `deprecated → destroyed` checks
+3. Wire checks into lifecycle transition handlers
+4. Test that `POST /activate` fails when checks don't pass
+5. Test that `POST /destroy` fails when inbound edges exist
+
+### Phase 5 — Cleanup (P2, within 2 weeks)
+
+1. Delete or comment-out dead code in bootstrap.sh
+2. Recreate `tools/setup-caveman.sh` and `tools/setup-hermes-soul.sh` (or remove references)
+3. Update AGENTS.md MCP tool list
+4. Update CLIENTS.md enrollment flow
+5. Archive Python secrets-issuance with final deprecation note
+6. Run `make generate-check` and full test suite
+
+## Verification
+
+- Fresh machine with no prior state: `curl bootstrap.sh | sudo bash` → machine
+  shows up in DB as `provisioning` with age pubkey, Infisical identity, and sync
+  timer running
+- `POST /api/v1/clients/ws:test-machine/activate` → state → `active`, all checks pass
+- `POST /api/v1/clients/ws:test-machine/deprecate` → state → `deprecated`
+- `POST /api/v1/clients/ws:test-machine/destroy` → fails if edges exist; succeeds
+  after edges removed, secrets revoked
+- MCP `whoami(ws:test-machine)` returns client record with peers and health
+- MCP `explain(service:caddy)` returns context card with relations and risk class
+- `GET /api/v1/clients/ws:test-machine/secrets` returns secrets list scoped to client
+- Existing clients continue working through the sync timer (no regression)
+- `make test test-db generate-check` passes
+
+## Related
+
+- [2026-07-07-migrate-bin-homelab-to-go.md](2026-07-07-migrate-bin-homelab-to-go.md) — MCP tool completion plan (whoami, explain, preflight)
+- [seeds/ontology.yaml](../seeds/ontology.yaml) — lifecycle definitions, entity type hierarchy
+- [seeds/policy.yaml](../seeds/policy.yaml) — risk classes, approval rules
+- [CLIENTS.md](../CLIENTS.md) — client onboarding guide (update after this plan)
+- [bootstrap.sh](../bootstrap.sh) — current enrollment script (rewrite in Phase 3)
+
+## Changelog
+
+- 2026-07-07 — initial plan. Replaces Python secrets-issuance, defines full
+  lifecycle in Go, adds client API endpoints, MCP tools, and Infisical
+  machine identity integration.
--- a/plans/index.md
+++ b/plans/index.md
@@ -10,10 +10,10 @@ went sideways, open an investigation.
 | ---- | ----- | ------ |
 | 2026-07-05 | [Oikos Prometheus LXC](2026-07-05-oikos-prometheus-lxc.md) | Planned |
 | 2026-07-06 | [Consolidate Oikos control plane onto mac-mini](2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md) | In Progress (Phase 1-6 implemented, pending cutover) |
+| 2026-07-07 | [Client lifecycle in Go — enrollment through deprecation](2026-07-07-client-lifecycle-in-go.md) | Planned |
 | 2026-07-07 | [Comprehensive audit: stale files, state gaps, and next steps](2026-07-07-comprehensive-audit-and-next-steps.md) | Planned |
 | 2026-07-07 | [DB as single source of truth for agent knowledge](2026-07-07-db-as-source-of-truth.md) | Proposed |
 | 2026-07-07 | [Migrate bin/homelab CLI to Go oikos homelab](2026-07-07-migrate-bin-homelab-to-go.md) | Planned |
-| 2026-07-07 | [Migrate bin/homelab CLI to Go oikos homelab](2026-07-07-migrate-bin-homelab-to-go.md) | Planned |

 ## Done