From 638e313c66bf6912ea25b623d22e29f15ce153ad Mon Sep 17 00:00:00 2001 From: dtoro Date: Tue, 7 Jul 2026 23:45:32 +0200 Subject: [PATCH] docs: add client lifecycle plan, cleanup stale files, document repo for 3 audiences MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem: Repo had no developer guide, no client onboarding doc, no agent dev instructions. Stale files (675KB SQL dump, one-off convert script, legacy MCP builder) cluttered the tree. Client enrollment was a documented intention with no Go implementation. Changes: - New docs: CONTRIBUTING.md (dev setup), CLIENTS.md (client onboarding), .agents/dev/CONTRIBUTING.md (agent codebase map) - New plan: plans/2026-07-07-client-lifecycle-in-go.md — full client lifecycle (planned→provisioning→active→deprecated→destroyed) in Go, replacing archived Python secrets-issuance, adding client API endpoints and 6 missing MCP tools - Cleanup: deleted archive/convert-wiki.py (one-off), archive/mcp/ build_host_files.py (legacy), backups/pre-deploy-7f7d039.sql (local) - Fixes: plans/index.md duplicate row removed, README.md repo layout updated for current state, AGENTS.md header points to new guides Risk: low. Docs only + stale file deletion. No code changes. New plan is proposal, not implementation. Verification: git diff reviewed, all changes are prose/docs/plans. --- .agents/dev/CONTRIBUTING.md | 179 ++++++++ AGENTS.md | 3 + CLIENTS.md | 117 ++++++ CONTRIBUTING.md | 155 +++++++ README.md | 23 +- archive/convert-wiki.py | 392 ------------------ archive/mcp/build_host_files.py | 162 -------- plans/2026-07-07-client-lifecycle-in-go.md | 451 +++++++++++++++++++++ plans/index.md | 2 +- 9 files changed, 921 insertions(+), 563 deletions(-) create mode 100644 .agents/dev/CONTRIBUTING.md create mode 100644 CLIENTS.md create mode 100644 CONTRIBUTING.md delete mode 100644 archive/convert-wiki.py delete mode 100755 archive/mcp/build_host_files.py create mode 100644 plans/2026-07-07-client-lifecycle-in-go.md diff --git a/.agents/dev/CONTRIBUTING.md b/.agents/dev/CONTRIBUTING.md new file mode 100644 index 0000000..d2617d5 --- /dev/null +++ b/.agents/dev/CONTRIBUTING.md @@ -0,0 +1,179 @@ +# Agent developer guide + +Instructions for AI agents working on the Oikos codebase. Read this after +[AGENTS.md](../../AGENTS.md) and [OIKOS.md](../OIKOS.md). Human developers: +see [CONTRIBUTING.md](../../CONTRIBUTING.md) for a human-friendly version. + +## Codebase map + +``` +cmd/oikos/main.go Entry point. Subcommands: api, scheduler, notifier, migrate, + seed, export, secret, all +cmd/hermes/main.go Hermes MCP client gateway (standalone binary) +internal/httpapi/ REST + MCP server. Chi router. OpenAPI-generated types from + internal/httpapi/gen/api.gen.go. Strict server in impl.go. +internal/mcp/ MCP tool implementations (get_entity, search_knowledge, etc.) +internal/db/ Connection pool (pool.go), seed ingestion (seed.go), DB→YAML + export (export.go), type hierarchy (typetree.go) +internal/db/queries/ SQL query files → sqlc generates internal/db/sqlcgen/ +internal/scheduler/ Observe loop: probes, signals, check_defs +internal/actuator/ SSH execution with circuit breaker + retry +internal/learning/ Pattern extraction, anomaly detection +internal/notifier/ Matrix notification + approval token generation +internal/policy/ Risk classifier (read policy.yaml → classify action) +internal/secrets/ Backend abstraction: Infisical (primary) + SOPS (fallback) +internal/domain/ Core types: entities, approvals, executions, signals, patterns +internal/ontology/ Type hierarchy validation, relationship checks +internal/knowledge/ Knowledge YAML seed ingestion +internal/config/ Config loading from env vars +api/openapi.yaml REST API contract. Source of truth for endpoints. +api/codegen.yaml oapi-codegen config → generates internal/httpapi/gen/ +migrations/ Forward-only SQL. Format: NNN_name.up.sql. No down migrations. +seeds/ Bootstrap YAML. ontology.yaml, inventory.yaml, policy.yaml, + knowledge.yaml. Regenerated from DB via oikos export. +compose/ Dockerfiles. oikos/ (multi-stage), hermes/ (distroless). + Caddy config at compose/caddy/Caddyfile.oikos. +scripts/ Deploy, rollback, watchdog, verification, cutover checklist. +hermes/ Hermes config.yaml, SOUL.md, skills. +.agents/ Agent instruction files, domains, shared conventions, skills. +plans/ Design documents. active/ + done/. +docs/adr/ Architecture decision records. Numbered, prefix-sorted. +``` + +## Development loop + +```bash +# Start dependencies +make dev + +# Generate code after API/SQL changes +make generate + +# Build +make build + +# Run tests +make test # all unit tests +make test-db # integration tests (needs compose Postgres) + +# Lint +make lint + +# CI drift guard (run before commit) +make generate-check +``` + +## Adding a feature or phase + +Oikos features follow a phase model (read [OIKOS.md](../OIKOS.md) for the +current phase status). To add a new capability: + +1. **ADR first.** Write an architecture decision record in `docs/adr/` with + the next sequence number. Document the decision, context, alternatives + considered, and consequences. +2. **Plan.** If the change is non-trivial, create a plan in `plans/` following + the template in [page-templates.md](../shared/page-templates.md). +3. **Schema.** If the feature needs new DB tables, write a forward-only + migration in `migrations/`. Use `IF NOT EXISTS` for idempotency. +4. **API.** If the feature exposes endpoints, define them in + `api/openapi.yaml` first, then run `make generate`, then implement. +5. **Domain.** Add types to `internal/domain/` before adding logic. +6. **Tests.** Write tests alongside implementation. Integration tests go in + `*_test.go` in the relevant package, using the compose Postgres. +7. **Policy.** If the feature introduces new mutation types, update + `seeds/policy.yaml` and the classifier in `internal/policy/`. +8. **Run `make generate-check`** before commit to ensure generated code is + current. + +## SQL conventions + +- Queries live in `internal/db/queries/*.sql` with `-- name: FuncName :exec` + annotations for sqlc +- Use `pgx/v5` driver. UUIDs use `pgtype.UUID`, timestamps use `time.Time` +- CTEs for graph traversals (blast radius, dependency chains) +- CAGGs and retention policies for TimescaleDB hypertables +- FTS via `tsvector` + `tsquery` for knowledge search (migration 011) + +## OpenAPI codegen + +- Config: `api/codegen.yaml`. Uses `oapi-codegen/v2` with Chi server template +- Generated output: `internal/httpapi/gen/api.gen.go` — never hand-edit +- Strict server interface: `api.gen.go` generates the `StrictServerInterface`; + implement it in `internal/httpapi/impl.go` +- Problem+JSON errors via `internal/httpapi/problem.go` — RFC 9457 format +- Cursor pagination, If-Match/ETag, idempotency keys, SSE streaming + +## Testing philosophy + +- **Race detector always on.** `make test` runs `go test -race -cover ./...` +- **Integration tests** use the compose Postgres. Run with `make test-db`. + Each test creates + tears down its own schema namespace. +- **Coverage gates** in CI: policy + learning ≥ 80%, others ≥ 60% +- Tests use `testing.T` directly, no assertion library +- Table-driven tests for validation and classification logic + +## Migration rules + +- Forward-only. No down migrations (ADR 0008) +- Idempotent: use `IF NOT EXISTS`, `DO $$ BEGIN ... END $$` blocks +- Sequence numbers are sequential integers (001, 002, ...) +- Each migration file is `NNN_name.up.sql` +- Migrations are embedded in the binary via `migrations/embed.go` + +## Seed files + +- `seeds/ontology.yaml` — entity types, relationship types, lifecycles + (validated against schema in `internal/ontology/`) +- `seeds/inventory.yaml` — hosts, services, entities (the topology) +- `seeds/policy.yaml` — risk classes, approval rules, autonomy settings +- `seeds/knowledge.yaml` — documents, investigations, runbooks (DB is source + of truth; this file is the DR export) +- After DB changes via the API, run `make export` to regenerate seeds + +## Secrets handling + +- No secrets in code, config, or commits +- Dev secrets in `.env` (gitignored) +- Primary: Infisical (`internal/secrets/infisical.go`) +- Fallback: SOPS + age (`internal/secrets/sops.go`) +- Backend interface: `internal/secrets/backend.go` +- Machine identities via Infisical UniversalAuth +- In-memory cache with TTL for performance + +## Staging and deployment + +- CI pipeline: `.gitea/workflows/ci.yml` — lint, vet, vulncheck, test, docker build +- Deploy: `scripts/deploy.sh` — git pull → docker build → compose up → health check +- Watchdog: `scripts/watchdog.sh` — 2-minute cron, Matrix alert on failure +- Rollback: `scripts/rollback.sh` — checkout SHA + pg_restore +- Cutover checklist: `scripts/cutover-checklist.md` + +## Writing conventions + +Apply [writing-style.md](../shared/writing-style.md) for all committed prose. +Terse, reference-style, no marketing vocabulary. Code comments explain intent +and trade-offs, not mechanics. + +Apply [caveman.md](../shared/caveman.md) for agent communication. The caveman +standard applies to agent *chat responses*, not committed documentation. + +## Skills + +Agent skills live under `.agents/skills//SKILL.md`. Each skill has a +frontmatter description that tools match against tasks. To add a skill: + +1. Create `.agents/skills//SKILL.md` +2. Include frontmatter with description field +3. Document the procedure following the runbook template +4. Reference relevant files, commands, and policy classes + +Skills that require code (e.g. linting) may include companion scripts in the +same directory. + +## When in doubt + +- Query MCP tools first (search_knowledge, get_entity) +- Read the relevant ADR in `docs/adr/` +- Grep the codebase: `rg internal/` +- Check `plans/` for in-progress work that may conflict +- Classify any new mutation against `seeds/policy.yaml` before suggesting it \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 901e315..6a79902 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,6 +4,9 @@ You are running on a machine that is part of the **hubris** homelab. The full context is in this checkout at `/opt/homelab-context/`. This file is the entry point. Read it once at start, then keep working. +- **New client?** Read [CLIENTS.md](CLIENTS.md) first. +- **Developing on this repo?** Also read [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md). + The operating model — OODA loop, risk classes, approval rules, the ontology, and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation, classify the action against `seeds/policy.yaml`; when the class requires diff --git a/CLIENTS.md b/CLIENTS.md new file mode 100644 index 0000000..c5af7f0 --- /dev/null +++ b/CLIENTS.md @@ -0,0 +1,117 @@ +# Oikos — client guide + +If you are a homelab machine, this is what Oikos is and what it gives you. +If you are an AI agent running on this machine, also read [AGENTS.md](AGENTS.md). + +## What is Oikos? + +Oikos is the agentic operating system for the **hubris** Proxmox homelab. It +observes state, classifies actions against policy, executes approved +procedures, learns from outcomes, and notifies the operator. It runs as a +Docker stack on mac-mini and exposes an MCP server + REST API. + +## What Oikos provides + +| Capability | How you access it | +|------------|-------------------| +| Entity query (topology, blast radius) | MCP `get_entity`, `get_blast_radius` | +| Full-text knowledge search | MCP `search_knowledge` | +| Service status + logs | MCP `get_service_status`, `tail_log` | +| LXC inventory + state | MCP `list_lxcs`, `get_lxc_state` | +| Context cards | MCP `explain` | +| Pre-flight risk classification | MCP `preflight` | +| Change history | MCP `get_change_history` | +| State snapshot (health, disk, drift) | MCP `get_state_snapshot` | +| Secrets (Infisical) | REST API + `oikos secret` CLI | +| Approval tokens | Matrix via notifier | + +All MCP tools are read-only. Mutations use the `homelab` CLI with operator +approval. + +## Enrollment + +Enrolled clients have a checkout at `/opt/homelab-context/`. If this +directory does not exist, the client is not enrolled. + +To enroll: +```bash +# Run from an existing enrolled client +homelab client add +``` + +This runs `bootstrap.sh` on the target, which: +1. Clones the repo to `/opt/homelab-context/` +2. Configures the auto-sync timer +3. Provisions agent persona from `hermes/SOUL.md` (on Hermes agents) +4. Installs Caveman tooling for terse communication + +## After enrollment + +### What changes on your machine + +- `/opt/homelab-context/` — the repo checkout, your source of truth +- `/opt/homelab-context/inventory.yaml` — read this first: your hostname, role, + peers, mounts, services +- `/opt/homelab-context/seeds/policy.yaml` — rules for what actions you can + take autonomously vs. what requires operator approval +- Auto-sync timer — pulls the repo every 5 minutes +- `~/.hermes/SOUL.md` — agent persona (on Hermes agents) +- `~/bin/caveman_wrapper.sh` — terse communication tooling + +### Your identity + +Your identity in the homelab is defined in `inventory.yaml`. Run `hostname` +(Linux) or `scutil --get LocalHostName` (macOS), then look up your entry. +It tells you your role, what services you host, what's mounted, and your +mesh address. + +### Source of truth hierarchy + +1. **Postgres database** (runtime) — authoritative for entities, knowledge, + signals, ledger. Query via MCP or REST API. +2. **Repo at `/opt/homelab-context/`** — bootstrap seeds + documentation. + On disk, available offline. +3. **Never guess.** If data is missing, query MCP. If MCP is down, grep the + clone. + +## The sync timer + +Every 5 minutes, `systemd` (Linux) or `launchd` (macOS) runs: + +1. `git pull` via `tools/post-pull.sh` +2. Any `tools/*.setup.sh` scripts that need to run + +This keeps your checkout current and applies tooling updates automatically. +To trigger sync manually: `sudo homelab sync`. + +## Making changes + +- **Read state**: use MCP tools or the repo checkout +- **Mutate state** (restart, edit config, deploy): classify the action against + `seeds/policy.yaml`: + - `read_only` / `reversible_low` — execute directly + - `config_mutation` / `destructive` — request operator approval via the + `homelab` CLI +- **Secrets**: use Infisical (primary) or SOPS (fallback). Never hardcode. +- **Knowledge**: if you observe a discrepancy between docs and live state, + update the DB via the API in the same session. Run `oikos export` to + regenerate seeds. + +## MCP endpoint + +``` +https://mcp.hubris.network/mcp +``` + +Available tools are listed in [AGENTS.md](AGENTS.md#3-the-mcp-server). + +## Communication mode + +Apply [.agents/shared/caveman.md](.agents/shared/caveman.md) — terse, +fragment-heavy communication. Drop filler, keep substance. + +## Related + +- [AGENTS.md](AGENTS.md) — full agent orientation (read this first) +- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, risk classes +- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide for the Oikos codebase \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..75b620e --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,155 @@ +# Contributing to Oikos + +Developer guide for the Oikos codebase. If you are a homelab client consuming +Oikos, see [CLIENTS.md](CLIENTS.md). If you are an AI agent working on the +repo, see [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md). + +## Dev setup + +- **Go 1.26+** (see `go.mod` for pinned version) +- **PostgreSQL with TimescaleDB** — the compose stack includes `timescale/timescaledb:2.17.2-pg16` +- **Docker** for the full dev stack + +```bash +# Start dependencies (Postgres + Redis) +docker compose --profile dev up -d + +# Run all tests +make test + +# Run integration tests (needs compose Postgres) +make test-db + +# Build the binary +make build +``` + +## Project structure + +``` +cmd/oikos/ Single-binary entry point +cmd/hermes/ Hermes MCP client gateway +internal/ All Go packages + httpapi/ REST + MCP server (OpenAPI-generated) + mcp/ MCP tool implementations + db/ Connection pool, migrations, seeds, sqlc queries + scheduler/ Observe loop, probes, signals + actuator/ SSH execution + learning/ Pattern recognition, anomaly detection + notifier/ Matrix notifications, approval tokens + policy/ Risk classifier + secrets/ Infisical + SOPS backend + domain/ Core types: entities, approvals, signals, patterns + ontology/ Type hierarchy, relationship validation + knowledge/ Knowledge YAML seed ingestion +api/openapi.yaml API contract — the source of truth for endpoints +migrations/ Forward-only SQL migrations (TimescaleDB) +seeds/ Bootstrap YAML: ontology, inventory, policy, knowledge +compose/ Dockerfiles + Caddy config +scripts/ Deploy, watchdog, rollback +hermes/ Hermes config, persona, skills +.agents/ Agent instruction files + skills +plans/ Design documents +docs/adr/ Architecture decision records +``` + +## Commands + +| Command | Purpose | +|---------|---------| +| `make build` | Build `oikos` binary | +| `make test` | Run all tests with race detection | +| `make test-db` | Run integration tests against compose Postgres | +| `make lint` | `go vet` + `golangci-lint` | +| `make generate` | Regenerate OpenAPI + sqlc code | +| `make generate-check` | CI drift guard — fail if generated code is stale | +| `make migrate` | Apply DB migrations | +| `make seed` | Ingest seeds into DB | +| `make export` | Export DB state to YAML seeds | +| `make dev` | Start compose dev stack | +| `make clean` | Remove binary + test cache | + +## Conventions + +### APIs are OpenAPI-first + +The REST API is defined in `api/openapi.yaml`. Server code is generated with +`oapi-codegen` into `internal/httpapi/gen/`. To add an endpoint: + +1. Add the path + schema to `api/openapi.yaml` +2. Run `make generate` +3. Implement the handler in `internal/httpapi/impl.go` +4. Add tests in `internal/httpapi/api_test.go` + +Never hand-edit `internal/httpapi/gen/api.gen.go`. + +### Database access is sqlc-first + +SQL queries live in `internal/db/queries/*.sql`. Go code is generated with +`sqlc` into `internal/db/sqlcgen/`. Config in `sqlc.yaml`. + +- Queries target pgx/v5 with UUID + timestamptz overrides +- Never hand-edit generated sqlc code + +### Migrations are forward-only + +SQL migrations live in `migrations/` as `NNN_name.up.sql`. There are no down +migrations (see [ADR 0008](docs/adr/0008-forward-only-migrations.md)). +Migrations are idempotent where possible (`IF NOT EXISTS`, `DO $$` blocks). + +To add a migration: + +1. Create `migrations/NNN_name.up.sql` with the next sequence number +2. Write the DDL +3. Run `make migrate` to apply + +### Seeds are DB-generated + +`seeds/*.yaml` are the bootstrap files used by `oikos seed`. After making +changes via the API, run `make export` to regenerate the seed files. These +files are version-controlled and serve as DR fallback. + +### Writing style + +Follow [.agents/shared/writing-style.md](.agents/shared/writing-style.md). +Documentation is reference prose, not marketing. Banned vocabulary includes +"robust", "seamless", "leverage", "utilize", "delve", "cutting-edge". + +### Risk classification + +Every mutation is classified against `seeds/policy.yaml` before execution. +Four risk classes: `read_only`, `reversible_low`, `config_mutation`, +`destructive`. The classifier can only lower autonomy relative to policy, +never raise it. When in doubt, escalate. + +## CI + +Gitea Actions runs on push to `main` and pull requests (`ci.yml`): + +1. `go vet` + `golangci-lint` + `govulncheck` +2. Generated code drift check (`make generate-check`) +3. Build (`go build ./...`) +4. Test with race detector + coverage +5. Docker build verification (no push) + +Coverage gates: policy + learning packages ≥ 80%, others ≥ 60%. + +## PR workflow + +1. Create a branch from `main` +2. Make changes, write tests +3. Run `make lint test generate-check` +4. Commit with a message following: problem → change → risk → verification +5. Push to Gitea; CI gates PRs on green + +## Secrets + +Secrets are managed by Infisical (primary) with SOPS as DR fallback. Never +hardcode secrets. Use environment variables from `.env` for local dev. +The `.env` and `.infisical-credentials` files are gitignored. + +## Related + +- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology +- [CLIENTS.md](CLIENTS.md) — for homelab clients consuming Oikos +- [docs/adr/](docs/adr/) — architecture decision records \ No newline at end of file diff --git a/README.md b/README.md index a15e98b..ac2d758 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,8 @@ state, classifies actions against policy, executes approved procedures over SSH, learns from outcomes, and escalates when uncertain. **For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md). +**For client machines:** see [CLIENTS.md](CLIENTS.md). +**For developers:** see [CONTRIBUTING.md](CONTRIBUTING.md). ## Quick start @@ -101,29 +103,34 @@ oikos secret migrate # SOPS → Infisical cmd/oikos/ Go entry point — single binary cmd/hermes/ Hermes MCP client gateway internal/ Go packages (httpapi, mcp, scheduler, actuator, learning, - notifier, policy, secrets, db, config, ontology, domain) + notifier, policy, secrets, db, config, ontology, domain, + knowledge) api/openapi.yaml API contract (OpenAPI 3.1) migrations/ Forward-only SQL migrations (TimescaleDB) -seeds/ Bootstrap YAML (ontology, inventory, policy) +seeds/ Bootstrap YAML (ontology, inventory, policy, knowledge) compose/ Dockerfiles + Caddy config scripts/ Deploy, watchdog, verification, rollback hermes/ Hermes config, persona, skills -archive/knowledge/ Narrative documentation (containers, hosts, infrastructure) -.agents/ Agent instruction files + skills -plans/ Design documents +.agents/ Agent instruction files, shared conventions, skills +archive/ Historical reference (legacy wiki, plans, SOPS backups) +plans/ Design documents (active + done) +docs/adr/ Architecture decision records ``` ## For agents See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference: -- **Source of truth:** DB (runtime) then repo (bootstrap + docs) -- **Mutations:** classify against policy, request approval for `destructive`/`config_mutation` -- **Wiki:** files under `archive/knowledge/`, changelog at bottom of each page +- **Source of truth:** DB (runtime) then seeds (bootstrap). Old wiki is + archived at `archive/knowledge/` — use MCP `search_knowledge` instead. +- **Mutations:** classify against policy, request approval for + `destructive`/`config_mutation` - **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode ## Related - [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology +- [CLIENTS.md](CLIENTS.md) — client onboarding guide +- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide - [plans/](plans/) — design documents and cutover checklist - [docs/adr/](docs/adr/) — architecture decision records diff --git a/archive/convert-wiki.py b/archive/convert-wiki.py deleted file mode 100644 index d830876..0000000 --- a/archive/convert-wiki.py +++ /dev/null @@ -1,392 +0,0 @@ -#!/usr/bin/env python3 -"""One-shot: convert knowledge/wiki/ to seeds/knowledge.yaml.""" - -import os, re, yaml -from pathlib import Path -from hashlib import sha256 - -REPO = Path("/Users/dtoro/Projects/oikos") -WIKI = REPO / "archive" / "knowledge" -SOURCES = REPO / "archive" / "knowledge" -GLOSSARY = REPO / "archive" / "knowledge" / "GLOSSARY.md" - -# Maps wiki path components to entity slugs -# Format: (path_pattern, entity_slug) -PATH_TO_ENTITY = { - # Containers - "containers/101-jellyfin": "lxc:jellyfin", - "containers/102-nfs-export": "lxc:nfs-export", - "containers/103-paperless": "lxc:paperless", - "containers/104-gitea": "lxc:gitea", - "containers/105-apps": "lxc:apps", - "containers/106-auth-outpost": "lxc:auth-outpost", - "containers/107-dns": "lxc:dns", - "containers/114-nextcloud": "lxc:nextcloud", - "containers/118-elementsynapse": "lxc:elementsynapse", - "containers/119-sophia": "lxc:sophia", - "containers/120-mule-images": "lxc:mule-images", - "containers/121-caddy": "lxc:caddy", - "containers/122-arriman": "lxc:arriman", - "containers/128-trmnl": "lxc:trmnl", - "containers/129-house": "lxc:house", - "containers/130-grimmory": "lxc:grimmory", - "containers/131-teddycloud": "lxc:teddycloud", - "containers/132-rclone": "lxc:rclone", - "containers/133-seanime": "lxc:seanime", - "containers/134-romm": "lxc:romm", - # Hosts - "hosts/hubris": "host:hubris", - "hosts/strong": "host:strong", - # VMs - "vms/100-zimaos": "vm:zimaos", - "vms/108-haos": "vm:haos", - # Infrastructure → services - "infrastructure/auto-deploy": None, - "infrastructure/backups": None, - "infrastructure/dns": "service:dns", - "infrastructure/homelab-context": "service:homelab-mcp", - "infrastructure/ingress": "service:caddy", - "infrastructure/media-permissions": "service:jellyfin", - "infrastructure/mesh": None, - "infrastructure/monitoring": None, - "infrastructure/network": None, - "infrastructure/ssh-access": None, - "infrastructure/topology": None, - "infrastructure/vps-hardening": "host:netbird-vps", -} - -def parse_page(path): - """Parse a wiki page into structured sections.""" - if not path.exists(): - return None - text = path.read_text() - lines = text.split('\n') - - # Title is first H1 - title = "" - for line in lines: - if line.startswith('# ') and not line.startswith('## '): - title = line[2:].strip() - break - - # Find sections by H2 headings - sections = {} - current_heading = "_preamble" - current_content = [] - - for line in lines: - if line.startswith('## ') and not line.startswith('### '): - if current_content: - sections[current_heading] = '\n'.join(current_content).strip() - current_heading = line[3:].strip().lower() - current_content = [] - else: - current_content.append(line) - if current_content: - sections[current_heading] = '\n'.join(current_content).strip() - - # Parse at-a-glance - at_glance = {} - ag_text = sections.get('at a glance', '') - for line in ag_text.split('\n'): - line = line.strip() - # Strip leading bullet - line = re.sub(r'^[-*]\s+', '', line) - # Match **Key:** value or **Key Word:** value - m = re.match(r'\*\*([^*]+?):?\*\*\s+(.+)', line) - if not m: - m = re.match(r'([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*):\s+(.+)', line) - if m: - key = m.group(1).lower().strip().replace(' ', '_').replace('/', '_') - val = m.group(2).strip() - # Strip trailing parenthetical notes - val = re.sub(r'\s*\([^)]*\)$', '', val) - # Strip markdown formatting from value - val = re.sub(r'\*\*([^*]+)\*\*', r'\1', val) - val = re.sub(r'`([^`]+)`', r'\1', val) - # Simplify link text - val = re.sub(r'\[([^\]]+)\]\([^)]+\)', r'\1', val) - val = re.sub(r'↔', '', val).strip() - # Normalize keys - key_map = { - 'cores': 'cores', 'core': 'cores', - 'ram': 'ram', 'memory': 'ram', - 'mounts': 'mounts', 'mount': 'mounts', - 'host': 'host', 'ip': 'ip', - 'public_host': 'public_host', 'public_hostname': 'public_host', - 'lan_ip': 'lan_ip', - 'os': 'os', 'kind': 'kind', - 'runtime': 'runtime', 'role': 'role', - 'pve_id': 'pve_id', 'privilege': 'privileged', - 'resources': 'resources', 'gpu': 'gpu', - 'swap': 'swap', 'rootfs': 'rootfs', - 'version': 'version', 'hardware': 'hardware', - } - key = key_map.get(key, key) - at_glance[key] = val - - # Parse changelog - changelog = [] - cl_text = sections.get('changelog', '') - current_entry = None - for line in cl_text.split('\n'): - m = re.match(r'###\s+(\d{4}-\d{2}-\d{2})\s+[—–-]\s+(.+)', line) - if m: - if current_entry: - changelog.append(current_entry) - current_entry = {'date': m.group(1), 'title': m.group(2).strip(), 'body': ''} - elif current_entry is not None: - stripped = line.strip() - if stripped and not stripped.startswith('#'): - if current_entry['body']: - current_entry['body'] += ' ' - current_entry['body'] += stripped - if current_entry: - changelog.append(current_entry) - - # Tags from path - parts = path.relative_to(REPO).parts - tags = [] - if 'containers' in parts: - tags.append('container') - elif 'hosts' in parts: - tags.append('host') - elif 'vms' in parts: - tags.append('vm') - elif 'infrastructure' in parts: - tags.append('infrastructure') - - # Determine slug from relative path - rel = str(path.relative_to(WIKI)) - slug = rel.replace('.md', '') - - # Entity mapping - entity_slug = PATH_TO_ENTITY.get(slug, None) - - return { - 'slug': slug, - 'title': title, - 'content': text, - 'entity_slug': entity_slug, - 'tags': tags, - 'at_glance': at_glance, - 'changelog': changelog, - 'is_investigation': 'investigations' in rel, - } - - -def parse_investigation(path): - """Parse an investigation page.""" - if not path.exists(): - return None - text = path.read_text() - lines = text.split('\n') - - title = "" - for line in lines: - if line.startswith('# '): - title = line[2:].strip() - break - - # Extract date from title or filename - date = "" - status = "resolved" - duration = "" - - for line in lines[:30]: - m = re.search(r'(\d{4}-\d{2}-\d{2})', line) - if m: - date = m.group(1) - break - - for line in lines: - if '**Status:**' in line: - status = line.split('**Status:**')[-1].strip().lower() - if '**Duration:**' in line: - duration = line.split('**Duration:**')[-1].strip() - - # Extract entity references for about_slugs - about_slugs = [] - entity_patterns = [ - (r'\bcaddy\b', 'service:caddy'), - (r'\bauthentik\b', 'service:authentik'), - (r'\bdns\b', 'service:dns'), - (r'\bgitea\b', 'service:gitea'), - (r'\bjellyfin\b', 'service:jellyfin'), - (r'\bmatrix\b', 'service:matrix'), - (r'\bpaperless\b', 'service:paperless'), - (r'\bnextcloud\b', 'service:nextcloud'), - (r'\bartifacto\b', 'service:artifacto'), - (r'\barriman\b', 'lxc:arriman'), - (r'\btrmnl\b', 'service:trmnl'), - (r'\bmac-mini\b', 'ws:mac-mini'), - (r'\bhubris\b', 'host:hubris'), - (r'\bstrong\b', 'host:strong'), - ] - for pattern, slug in entity_patterns: - if re.search(pattern, text, re.IGNORECASE): - about_slugs.append(slug) - - rel = str(path.relative_to(WIKI)) - slug = rel.replace('.md', '') - - return { - 'slug': slug, - 'title': title, - 'date': date, - 'status': status, - 'duration': duration, - 'content': text, - 'about_slugs': about_slugs, - 'tags': ['investigation'], - } - - -def main(): - documents = [] - investigations = [] - runbooks = [] - - # Container pages - containers_dir = WIKI / "containers" - for f in sorted(containers_dir.glob("*.md")): - if 'index' in f.name: - continue - if f.parent.name == 'archive': - continue - result = parse_page(f) - if result and result['title']: - documents.append(result) - print(f" document: {result['slug']} → {result['entity_slug']}") - - # Host pages - hosts_dir = WIKI / "hosts" - for f in sorted(hosts_dir.glob("*.md")): - if 'index' in f.name: - continue - result = parse_page(f) - if result and result['title']: - documents.append(result) - print(f" document: {result['slug']} → {result['entity_slug']}") - - # VM pages - vms_dir = WIKI / "vms" - for f in sorted(vms_dir.glob("*.md")): - if 'index' in f.name: - continue - result = parse_page(f) - if result and result['title']: - documents.append(result) - print(f" document: {result['slug']} → {result['entity_slug']}") - - # Infrastructure pages - infra_dir = WIKI / "infrastructure" - for f in sorted(infra_dir.glob("*.md")): - if 'index' in f.name: - continue - result = parse_page(f) - if result and result['title']: - documents.append(result) - print(f" document: {result['slug']} → {result['entity_slug']}") - - # Investigation pages - inv_dir = SOURCES / "investigations" - for f in sorted(inv_dir.glob("*.md")): - if 'index' in f.name: - continue - result = parse_investigation(f) - if result and result['title']: - investigations.append(result) - print(f" investigation: {result['slug']} → {result['about_slugs']}") - - # Archive investigations too - inv_archive = inv_dir / "archive" - if inv_archive.exists(): - for f in sorted(inv_archive.glob("*.md")): - result = parse_investigation(f) - if result and result['title']: - investigations.append(result) - print(f" investigation: {result['slug']} → {result['about_slugs']}") - - # Runbooks from .agents/skills/ - skills_dir = REPO / ".agents" / "skills" - for skill_dir in sorted(skills_dir.iterdir()): - if not skill_dir.is_dir(): - continue - skill_file = skill_dir / "SKILL.md" - if not skill_file.exists(): - continue - text = skill_file.read_text() - lines = text.split('\n') - title = "" - for line in lines: - if line.startswith('# '): - title = line[2:].strip() - break - - # Extract risk_class and entity_type from frontmatter - risk_class = "read_only" - entity_type = "service" - for line in lines[:30]: - m = re.match(r'\*\*risk_class:\*\*\s*(\w+)', line, re.IGNORECASE) - if m: - risk_class = m.group(1) - m = re.match(r'\*\*applies_to:\*\*\s*(\w[\w-]*)', line, re.IGNORECASE) - if m: - entity_type = m.group(1) - - name = skill_dir.name - runbooks.append({ - 'slug': name, - 'name': title or name, - 'risk_class': risk_class, - 'entity_type': entity_type, - 'procedure': {}, # SKILL.md is narrative, not structured yet - 'content': text, - 'tags': ['skill', 'runbook'], - }) - print(f" runbook: {name}") - - # Build seed YAML - seed = { - 'version': 1, - 'documents': [{ - 'slug': d['slug'], - 'title': d['title'], - 'content': d['content'], - 'entity_slug': d['entity_slug'], - 'tags': d['tags'], - 'at_glance': d['at_glance'], - 'changelog': d['changelog'], - } for d in documents], - 'investigations': [{ - 'slug': i['slug'], - 'title': i['title'], - 'date': i['date'], - 'status': i['status'], - 'duration': i['duration'], - 'content': i['content'], - 'about_slugs': i['about_slugs'], - 'tags': i['tags'], - } for i in investigations], - 'runbooks': [{ - 'slug': r['slug'], - 'name': r['name'], - 'risk_class': r['risk_class'], - 'entity_type': r['entity_type'], - 'procedure': r['procedure'], - 'content': r['content'], - 'tags': r['tags'], - } for r in runbooks], - } - - out_path = REPO / "seeds" / "knowledge.yaml" - out_path.write_text(yaml.dump(seed, allow_unicode=True, width=120, sort_keys=False)) - print(f"\nWrote {out_path}") - print(f" {len(documents)} documents") - print(f" {len(investigations)} investigations") - print(f" {len(runbooks)} runbooks") - - -if __name__ == "__main__": - main() \ No newline at end of file diff --git a/archive/mcp/build_host_files.py b/archive/mcp/build_host_files.py deleted file mode 100755 index 9630ebb..0000000 --- a/archive/mcp/build_host_files.py +++ /dev/null @@ -1,162 +0,0 @@ -#!/usr/bin/env python3 -""" -Generate hosts/.yaml from inventory.yaml. - -Run from the repo root: - python3 mcp/build_host_files.py # writes files, exits non-zero on diff - python3 mcp/build_host_files.py --check # exits non-zero if any output differs - -Designed to be wired into a pre-commit hook or Gitea Action so generated -hosts/*.yaml never drift from inventory.yaml. -""" - -from __future__ import annotations - -import argparse -import difflib -import os -import sys -from pathlib import Path - -try: - import yaml -except ImportError: # pragma: no cover - print("PyYAML is required: pip install pyyaml", file=sys.stderr) - sys.exit(2) - - -REPO = Path(__file__).resolve().parent.parent -INVENTORY = REPO / "inventory.yaml" -HOSTS_DIR = REPO / "hosts" - -GENERATED_BANNER = ( - "# Generated by mcp/build_host_files.py from inventory.yaml.\n" - "# Do NOT edit by hand — your changes will be overwritten.\n" - "# Source of truth: ../inventory.yaml\n" -) - - -def narrative_page(name: str, kind: str, pve_id: int | None) -> str | None: - """Best-guess path to the human-authored narrative page for this host.""" - if kind == "proxmox-host": - candidate = REPO / "hosts" / f"{name}.md" - elif kind == "lxc": - candidate = REPO / "containers" / f"{pve_id}-{name}.md" - elif kind == "vm": - candidate = REPO / "vms" / f"{pve_id}-{name}.md" - else: - return None - if candidate.exists(): - return str(candidate.relative_to(REPO)) - return None - - -def build_one(name: str, entry: dict, inventory: dict) -> dict: - """Project the entry for a single host into a per-host yaml record.""" - services = inventory.get("services", {}) - mesh = inventory.get("mesh", {}) - pve_id = entry.get("pve_id") - - # Services this host runs: scan inventory.services for matching backend. - runs_services = sorted( - svc for svc, sentry in services.items() - if isinstance(sentry, dict) and sentry.get("backend") == name - ) - - record = { - "name": name, - "kind": entry.get("kind"), - "os": entry.get("os"), - "role": entry.get("role"), - # Oikos lifecycle (oikos/ontology.yaml); absent in inventory = active - "state": entry.get("state", "active"), - "host": entry.get("host"), - "pve_id": pve_id, - "storage": entry.get("storage"), - "depends_on": entry.get("depends_on", []), - "lan_ip": entry.get("lan_ip"), - "mesh": entry.get("mesh", {}), - "mesh_globals": { - "primary": mesh.get("primary"), - "accepted": mesh.get("accepted"), - }, - "peers": entry.get("peers", []), - "mounts": entry.get("mounts", []), - "public_host": entry.get("public_host"), - "public_hosts": entry.get("public_hosts", []), - "ssh": entry.get("ssh", {}), - "runs": entry.get("runs", []) + runs_services, - "services_hosted": [ - {"name": svc, **services[svc]} for svc in runs_services - ], - "notes": entry.get("notes", []), - "age_pubkey": entry.get("age_pubkey", ""), - "see_also": [ - page for page in [narrative_page(name, entry.get("kind", ""), pve_id)] - if page - ], - "mcp_endpoint": services.get("homelab_mcp", {}).get("endpoint"), - "secrets_issuance_endpoint": ( - services.get("secrets_issuance", {}).get("endpoint") - ), - } - # Strip None and empty containers so the file stays readable. - return {k: v for k, v in record.items() if v not in (None, {}, [], "")} - - -def serialize(record: dict) -> str: - return GENERATED_BANNER + yaml.safe_dump( - record, sort_keys=False, default_flow_style=False, width=100 - ) - - -def main() -> int: - parser = argparse.ArgumentParser() - parser.add_argument("--check", action="store_true", - help="exit 1 if any output would change (don't write)") - args = parser.parse_args() - - inventory = yaml.safe_load(INVENTORY.read_text()) - hosts = inventory.get("hosts", {}) - - HOSTS_DIR.mkdir(exist_ok=True) - desired: dict[Path, str] = {} - for name, entry in hosts.items(): - desired[HOSTS_DIR / f"{name}.yaml"] = serialize(build_one(name, entry, inventory)) - - diff_count = 0 - for path, content in desired.items(): - existing = path.read_text() if path.exists() else "" - if existing != content: - diff_count += 1 - if args.check: - diff = difflib.unified_diff( - existing.splitlines(keepends=True), - content.splitlines(keepends=True), - fromfile=str(path), - tofile=str(path) + " (generated)", - ) - sys.stdout.writelines(diff) - else: - path.write_text(content) - print(f"wrote {path.relative_to(REPO)}") - - # Clean up orphans (file exists but host removed from inventory). - for existing_path in HOSTS_DIR.glob("*.yaml"): - if existing_path not in desired: - diff_count += 1 - if args.check: - print(f"orphan: {existing_path.relative_to(REPO)} (would delete)") - else: - existing_path.unlink() - print(f"deleted orphan {existing_path.relative_to(REPO)}") - - if args.check and diff_count > 0: - print(f"\n{diff_count} file(s) would change. Run without --check to write.", - file=sys.stderr) - return 1 - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/plans/2026-07-07-client-lifecycle-in-go.md b/plans/2026-07-07-client-lifecycle-in-go.md new file mode 100644 index 0000000..4b8e62f --- /dev/null +++ b/plans/2026-07-07-client-lifecycle-in-go.md @@ -0,0 +1,451 @@ +# Plan: Client lifecycle — enrollment through deprecation in Oikos Go + +**Status:** Planned (2026-07-07) + +## Goal + +Define and implement the complete lifecycle of a homelab client in the Oikos Go +runtime: how a new machine is provisioned, enrolled, given secrets, synced, +operated, and eventually deprecated (or decommissioned or destroyed). Every +state transition feeds the Postgres DB as the authoritative source of truth. +No step depends on the archived Python `secrets-issuance` server or the +non-existent `bin/homelab` CLI. + +## Current state — what exists vs. what runs + +| Component | Exists? | Runs? | Notes | +|-----------|---------|-------|-------| +| `bootstrap.sh` (684 lines) | ✅ repo | ⚠️ references dead endpoints | Calls `https://secrets.hubris.network/issue` (Python server, stopped per Phase 6). Symlinks `bin/homelab` (file absent). References `tools/*.setup.sh` (files absent). | +| `archive/secrets-issuance/server.py` | ✅ archive | ❌ stopped (Phase 6) | Issued age keys, validated mesh IP. No Go replacement. | +| `archive/mcp/` (build_host_files.py, deleted) | ❌ deleted | ❌ | Legacy host file builder. | +| `inventory.yaml` (root) | ✅ | ⚠️ edited manually | Flat `hosts:` + `services:` layout. Diverges from `seeds/inventory.yaml` entity-relationship format. | +| `seeds/inventory.yaml` | ✅ | ✅ ingested into DB | Entity-relationship format with slugs (`host:hubris`, `ws:mac-mini`). No translation path from root format. | +| Go entity API (`POST/GET/PATCH /entities`) | ✅ | ✅ | Generic CRUD. No client-specific validation, no key issuance, no lifecycle gating. | +| Go MCP server | ✅ | ✅ | 15 tools. Missing `whoami`, `explain`, `preflight`, `get_change_history`, `get_state_snapshot`. | +| `oikos secret` CLI (Infisical/SOPS) | ✅ | ✅ | Secrets read/migrate/export. No client-key provisioning. | +| Sync timer (post-pull.sh) | ✅ | ⚠️ partially broken | References `tools/*.setup.sh` (glob returns zero files). `setup-caveman.sh` and `setup-hermes-soul.sh` documented but absent. | + +**Takeaway**: Enrollment today runs on shell scripts calling archived Python +services. The Go runtime has zero awareness of client lifecycle. This plan +closes that gap — the DB becomes the sole engine for client identity, +secrets, state, and lifecycle transitions. + +## Target architecture + +``` +┌──────────────────────────────────────────────────────────────┐ +│ NEW CLIENT (bare machine) │ +│ │ +│ 1. curl bootstrap.sh | sudo bash │ +│ → clones repo, installs sync timer │ +│ → calls POST /api/v1/clients/enroll (new endpoint) │ +│ → receives age keypair from Oikos API │ +│ → writes /etc/age/key.txt │ +│ → sync timer starts pulling every 5 min │ +└──────────────────────────┬───────────────────────────────────┘ + │ POST /api/v1/clients/enroll + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ OIKOS API (Go, :8090) │ +│ │ +│ POST /api/v1/clients/enroll — issue age key, set state │ +│ POST /api/v1/clients/{slug}/activate — provisioning→active │ +│ POST /api/v1/clients/{slug}/deprecate — active→deprecated │ +│ POST /api/v1/clients/{slug}/destroy — deprecated→destroyed │ +│ GET /api/v1/clients/{slug}/secrets — client's accessible │ +│ secrets (Infisical lookup by machine identity) │ +│ MCP whoami(hostname) — client self-introspection │ +│ MCP explain(service) — compact context card │ +│ MCP preflight(service) — risk classification │ +│ MCP get_change_history(entity) — ledger entries │ +│ MCP get_state_snapshot() — last scheduler pass │ +└──────────────────────────┬───────────────────────────────────┘ + │ writes + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ POSTGRES (TimescaleDB) │ +│ │ +│ entities table: slug, type, name, state, attributes (JSONB) │ +│ entity_status: health, disk, drift count (scheduler) │ +│ audit_log: every state transition, enrollment, revocation │ +│ executions: approved actions, results │ +│ secrets (via Infisical): age keys, API tokens │ +└──────────────────────────────────────────────────────────────┘ +``` + +## Client lifecycle: state machine + +``` + [planned] ──→ provisioning ──→ active ──→ migrating ──→ active + │ │ │ + │ │ ├──→ deprecated ──→ destroyed + │ │ │ + │ └──→ failed └──→ failed + │ + └──→ destroyed (cancelled) +``` + +### State: `planned` + +The operator declares intent. A client entity exists in the DB with state +`planned` but has no host, no keys, no sync. + +**Entry condition**: Operator creates the entity via API or seed file. + +**Required attributes**: +- `slug` — `ws:` for workstations, `host:` for servers +- `type` — `workstation`, `standalone-server`, or `proxmox-host` +- `name` — human-readable name +- `lan_ip` — expected LAN IP (reserved in DHCP) +- `os` — `linux` or `macos` +- `role` — free-text description of what this machine does +- `mesh.expected_type` — `netbird` or `tailscale` (which mesh it will join) +- `ssh.user` — login user (default `root`) + +**Allowed transitions**: `→ provisioning` (operator triggers), `→ destroyed` (cancelled). + +### State: `provisioning` + +The machine has been declared. Operator runs `bootstrap.sh` on the target, +which calls the enrollment API. The API validates identity (mesh IP matches +expected subnet, hostname matches slug), issues an age keypair, and records +the public key. The sync timer starts pulling the repo. + +**Go API**: `POST /api/v1/clients/enroll` +```json +{ + "slug": "ws:new-laptop", + "hostname": "new-laptop", + "mesh_ip": "100.122.x.x" +} +``` + +**What the enrollment endpoint does**: +1. Looks up entity by slug — must exist, must be in state `planned` or `provisioning` +2. Validates mesh IP is in `100.122.0.0/16` (Netbird) or `100.64.0.0/10` (Tailscale) or `192.168.8.0/24` (LAN) +3. Validates hostname has no conflicting mesh IP already recorded +4. Generates an age keypair (`age-keygen`) +5. Stores the private key in Infisical under path `/clients//age-key` +6. Creates an Infisical machine identity for the client (UniversalAuth) +7. Updates entity `attributes` with `age_pubkey`, `mesh_ip`, `enrolled_at` +8. Writes audit log: `client.enrolled` +9. Returns the age private key, Infisical client ID + secret, and machine identity token + +**Response** (to bootstrap.sh, over mesh — TLS + mesh IP validation): +```json +{ + "age_private_key": "AGE-SECRET-KEY-...", + "age_public_key": "age1...", + "infisical_client_id": "...", + "infisical_client_secret": "...", + "machine_identity_token": "..." +} +``` + +**Bootstrap script changes**: +- Remove call to `https://secrets.hubris.network/issue` +- Replace with `POST /api/v1/clients/enroll` to `https://oikos.hubris.network` +- Remove `--no-secrets` / `--no-mesh` flags (or keep as escape hatches with degraded state) +- Remove symlink to `bin/homelab` (file doesn't exist) +- After receiving keys, bootstrap.sh writes `/etc/age/key.txt` (0600) and `/etc/infisical/identity` (0600) + +**Pre-built bootstrap**: The bootstrap.sh is served from the Gitea repo raw URL +(already the case). After this plan, it calls Oikos API instead of the dead +Python service. + +**Allowed transition**: `→ active` (when `age-key-enrolled`, `mesh-joined`, +`doc-page-complete` checks pass). + +### State: `active` + +Normal operation. The client pulls the repo every 5 minutes, uses its age key +to decrypt SOPS secrets (fallback), and authenticates to Infisical via its +machine identity (primary). The MCP `whoami(hostname)` tool returns its +entity record, peer list, accessible secrets, and current health. + +**Go enforcement of transition checks** (`provisioning → active`): +- `age-key-enrolled-if-needed`: entity.attributes.age_pubkey is non-empty +- `mesh-joined-if-needed`: entity.attributes.mesh_ip is non-empty +- `ingress-live-if-public`: skipped for workstations (no public ingress) +- `health-check-answering`: scheduler probe passes for this entity +- `doc-page-complete`: entity has at least one `documents` edge +- `inventory-in-db`: entity exists in DB with all required attributes + +**API**: `POST /api/v1/clients/{slug}/activate` +- Validates all `provisioning → active` transition checks +- Sets state to `active` +- Writes audit log: `client.activated` + +**MCP tools active clients get**: +- `whoami(hostname)` — returns entity record, peers, secrets list, health +- `list_my_secrets(caller_pubkey?)` — secrets this client can decrypt + +**Allowed transitions**: `→ migrating`, `→ deprecated`, `→ failed`. + +### State: `migrating` + +Client is being moved — OS reinstall, hardware swap, role change. Inbound edges +still exist; no deprovisioning has started. + +**Allowed transition**: `→ active` (migration complete, post-verify passes). + +**API**: `POST /api/v1/clients/{slug}/migrate` (sets state, links migration plan). + +### State: `deprecated` + +Client is being phased out. Services moved off, mesh disconnected, secrets +rotation started. The deprecation gate (`no-inbound-edges`) blocks `→ destroyed` +until all `depends-on`, `hosts`, `provides`, and `mounts` edges are gone. + +**API**: `POST /api/v1/clients/{slug}/deprecate` +- Validates `replacement-live-or-role-retired`: operator confirms replacement exists or role is no longer needed +- Sets state to `deprecated` +- Writes audit log: `client.deprecated` + +**Allowed transitions**: `→ active` (un-deprecate), `→ destroyed`. + +### State: `destroyed` + +Client is gone. All edges removed, secrets revoked, archaeology entry written. + +**API**: `POST /api/v1/clients/{slug}/destroy` +- Validates all `deprecated → destroyed` transition checks: + - `backups-verified`: any data on this client was backed up + - `secrets-revoked-and-rekeyed`: age key removed from Infisical, SOPS recipients updated, machine identity deleted + - `ingress-and-dns-removed`: no remaining DNS records or Caddy backends + - `no-inbound-edges`: zero `depends-on`, `hosts`, `provides`, `mounts` edges pointing to this entity + - `archaeology-entry`: writes a record explaining why and when +- Sets state to `destroyed` +- Revokes Infisical machine identity +- Removes age public key from `.sops.yaml` +- Writes audit log: `client.destroyed` + +### State: `failed` + +Something went wrong during provisioning or operation. Requires operator +intervention. Treated as informational — no automatic recovery. + +**API**: `POST /api/v1/clients/{slug}/fail` +- Sets state to `failed` +- Requires `reason` field explaining what broke +- Writes audit log: `client.failed` + +## Secrets integration + +### Age key lifecycle + +``` +planned ────────────→ no key exists +provisioning ───────→ keypair generated, pubkey stored in entity attributes, + private key delivered to client via enroll response, + private key stored in Infisical under /clients//age-key +active ─────────────→ key used for SOPS decryption fallback, authenticated + to Infisical via machine identity for primary secrets +deprecated ─────────→ key still valid, but rotation initiated +destroyed ──────────→ key revoked from Infisical, removed from .sops.yaml, + machine identity deleted +``` + +### Infisical machine identity + +Each client gets an Infisical machine identity during enrollment. This is the +primary secrets path — the age key is fallback for SOPS-encrypted DR files. + +- **Client ID + Secret** returned in enroll response +- **Scoped to paths**: `/clients//*`, `/shared/*` +- **Revoked on destroy**: identity deleted, access gone + +### SOPS fallback + +The age public key is added to `.sops.yaml` recipients during enrollment. +On destroy, it is removed via `oikos secret export-sops` regeneration. + +### bootstrap.sh changes + +```diff +- # calls https://secrets.hubris.network/issue (Python, dead) +- AGE_KEY=$(curl -s -X POST "$ISSUANCE_URL" ...) +- ++ # calls Oikos API enrollment endpoint ++ ENROLL_RESP=$(curl -s -X POST "$OIKOS_URL/api/v1/clients/enroll" \ ++ -H "Content-Type: application/json" \ ++ -d "{\"slug\":\"ws:$HNAME\",\"hostname\":\"$HNAME\",\"mesh_ip\":\"$MESH_IP\"}") ++ AGE_PRIVKEY=$(echo "$ENROLL_RESP" | jq -r '.age_private_key') +``` + +## DB integration + +### New migration + +`012_client_enrollment.up.sql`: +```sql +-- No new tables needed — entities table already holds clients. +-- Add enrollment-specific attributes validation via check constraints +-- or application-level validation. + +-- Enforce slug format for machine entities +-- ws: for workstations, host: for servers +-- (application-level validation in Go, not a DB constraint) + +-- Add index for slug-based client lookups +CREATE INDEX IF NOT EXISTS idx_entities_slug_type + ON entities (slug, type) + WHERE type IN ('workstation', 'standalone-server', 'proxmox-host'); +``` + +### Entity attributes schema (for `machine` types) + +```json +{ + "cpu_arch": "arm64", + "ram_gb": 16, + "os": "macos", + "lan_ip": "192.168.8.175", + "mesh": { + "netbird": {"ip": "100.122.x.x", "fqdn": "hostname.netbird.selfhosted"} + }, + "ssh": {"user": "dtoro"}, + "age_pubkey": "age1...", + "enrolled_at": "2026-07-07T12:00:00Z", + "enrolled_by": "ws:mac-mini", + "infisical_identity_id": "identity_abc123" +} +``` + +All attributes are stored in the `attributes` JSONB column on the `entities` +table. Validation happens at the application layer (Go) using the schema +defined in `seeds/ontology.yaml`. + +## MCP tools to add + +These are documented in AGENTS.md section 3 but not implemented in the Go MCP +server. Implementation: register in `internal/mcp/server.go`. + +| Tool | Input | Output | Implementation | +|------|-------|--------|----------------| +| `whoami` | `hostname` | Entity record, peers, accessible secrets, health | DB lookup by slug derived from hostname | +| `list_my_secrets` | `caller_pubkey?` | Secrets this client can decrypt | Infisical list + SOPS `.sops.yaml` match | +| `explain` | `service_slug` | Compact context card: type, state, health, relations, last change | DB join: entity + entity_status + audit_log | +| `preflight` | `service_slug` | Risk class, approval requirement, verification command | Policy classifier on the entity's type | +| `get_change_history` | `entity_slug`, `limit` | Last N audit_log entries for entity | DB query on audit_log table | +| `get_state_snapshot` | none | Last scheduler Observe pass: health, disk, drift count | DB query on entity_status + signals | + +## API endpoints to add + +Add to `api/openapi.yaml`, regenerate with `make generate`, implement in +`internal/httpapi/impl.go`. + +| Method | Path | Scope | Purpose | +|--------|------|-------|---------| +| `POST` | `/api/v1/clients/enroll` | agent | Issue age key, validate mesh, set state → provisioning | +| `POST` | `/api/v1/clients/{slug}/activate` | operator | Run transition checks, state → active | +| `POST` | `/api/v1/clients/{slug}/deprecate` | operator | State → deprecated | +| `POST` | `/api/v1/clients/{slug}/destroy` | operator | Run destroy checks, revoke secrets, state → destroyed | +| `POST` | `/api/v1/clients/{slug}/fail` | operator | State → failed with reason | +| `GET` | `/api/v1/clients/{slug}/secrets` | agent | List secrets this client can access | + +## Files changed + +| File | Change | +|------|--------| +| `bootstrap.sh` | Replace `secrets.hubris.network/issue` call with `POST /api/v1/clients/enroll`. Remove dead symlinks. | +| `api/openapi.yaml` | Add client enrollment, lifecycle, and secret endpoints | +| `internal/httpapi/impl.go` | Implement client lifecycle handlers | +| `internal/db/queries/clients.sql` | Add client-specific sqlc queries | +| `internal/mcp/server.go` | Register whoami, explain, preflight, get_change_history, get_state_snapshot, list_my_secrets | +| `internal/secrets/infisical.go` | Add `CreateMachineIdentity`, `DeleteMachineIdentity`, `StoreClientKey` | +| `internal/ontology/validate.go` | Implement lifecycle transition checks for infrastructure lifecycle | +| `seeds/ontology.yaml` | Add client-specific attributes schema for machine types | +| `migrations/012_client_enrollment.up.sql` | Index for slug+type lookups | +| `AGENTS.md` | Update MCP tool list to match actual implementation | +| `CLIENTS.md` | Update enrollment flow to reference Oikos API, not Python issuance | +| `CONTRIBUTING.md` | Add client lifecycle as a documented extension point | + +## Files deleted or deprecated + +| File | Disposition | +|------|-------------| +| `archive/secrets-issuance/` | Already archived. Add deprecation notice referencing this plan. | +| `archive/secrets-sops-backup/` | Keep for DR. Add note that new clients use Infisical, SOPS is fallback. | +| Any reference to `bin/homelab` | Delete or comment out in bootstrap.sh; CLI doesn't exist. | +| `tools/*.setup.sh` references | Either create the files or remove the auto-setup convention from post-pull.sh. | + +## Phased implementation + +### Phase 1 — API + DB (P0, this week) + +1. Write `migrations/012_client_enrollment.up.sql` +2. Add client endpoints to `api/openapi.yaml` +3. Run `make generate` +4. Implement enrollment handler (`POST /api/v1/clients/enroll`): + - Age key generation + - Infisical machine identity creation + - Entity attribute update + - Audit log write +5. Implement lifecycle transition handlers (activate, deprecate, destroy, fail) +6. Implement `GET /api/v1/clients/{slug}/secrets` +7. Update `seeds/ontology.yaml` with client attribute schemas +8. Add sqlc queries in `internal/db/queries/clients.sql` + +### Phase 2 — MCP tools (P1, next week) + +1. Register `whoami(hostname)` in `internal/mcp/server.go` +2. Register `explain(service)` — compact context card from DB +3. Register `preflight(service)` — risk classification +4. Register `get_change_history(entity, limit)` +5. Register `get_state_snapshot()` +6. Register `list_my_secrets(caller_pubkey?)` + +### Phase 3 — Bootstrap script cleanup (P1, next week) + +1. Replace secrets issuance URL with Oikos API endpoint +2. Remove `--no-secrets` / `--no-mesh` or rewire them to degraded modes +3. Remove `bin/homelab` symlink +4. Update Infisical identity file creation +5. Test full enrollment on a fresh machine + +### Phase 4 — Transition check enforcement (P2, within 2 weeks) + +1. Implement all `provisioning → active` checks in `internal/ontology/validate.go` +2. Implement all `deprecated → destroyed` checks +3. Wire checks into lifecycle transition handlers +4. Test that `POST /activate` fails when checks don't pass +5. Test that `POST /destroy` fails when inbound edges exist + +### Phase 5 — Cleanup (P2, within 2 weeks) + +1. Delete or comment-out dead code in bootstrap.sh +2. Recreate `tools/setup-caveman.sh` and `tools/setup-hermes-soul.sh` (or remove references) +3. Update AGENTS.md MCP tool list +4. Update CLIENTS.md enrollment flow +5. Archive Python secrets-issuance with final deprecation note +6. Run `make generate-check` and full test suite + +## Verification + +- Fresh machine with no prior state: `curl bootstrap.sh | sudo bash` → machine + shows up in DB as `provisioning` with age pubkey, Infisical identity, and sync + timer running +- `POST /api/v1/clients/ws:test-machine/activate` → state → `active`, all checks pass +- `POST /api/v1/clients/ws:test-machine/deprecate` → state → `deprecated` +- `POST /api/v1/clients/ws:test-machine/destroy` → fails if edges exist; succeeds + after edges removed, secrets revoked +- MCP `whoami(ws:test-machine)` returns client record with peers and health +- MCP `explain(service:caddy)` returns context card with relations and risk class +- `GET /api/v1/clients/ws:test-machine/secrets` returns secrets list scoped to client +- Existing clients continue working through the sync timer (no regression) +- `make test test-db generate-check` passes + +## Related + +- [2026-07-07-migrate-bin-homelab-to-go.md](2026-07-07-migrate-bin-homelab-to-go.md) — MCP tool completion plan (whoami, explain, preflight) +- [seeds/ontology.yaml](../seeds/ontology.yaml) — lifecycle definitions, entity type hierarchy +- [seeds/policy.yaml](../seeds/policy.yaml) — risk classes, approval rules +- [CLIENTS.md](../CLIENTS.md) — client onboarding guide (update after this plan) +- [bootstrap.sh](../bootstrap.sh) — current enrollment script (rewrite in Phase 3) + +## Changelog + +- 2026-07-07 — initial plan. Replaces Python secrets-issuance, defines full + lifecycle in Go, adds client API endpoints, MCP tools, and Infisical + machine identity integration. \ No newline at end of file diff --git a/plans/index.md b/plans/index.md index af8f154..a0e6b09 100644 --- a/plans/index.md +++ b/plans/index.md @@ -10,10 +10,10 @@ went sideways, open an investigation. | ---- | ----- | ------ | | 2026-07-05 | [Oikos Prometheus LXC](2026-07-05-oikos-prometheus-lxc.md) | Planned | | 2026-07-06 | [Consolidate Oikos control plane onto mac-mini](2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md) | In Progress (Phase 1-6 implemented, pending cutover) | +| 2026-07-07 | [Client lifecycle in Go — enrollment through deprecation](2026-07-07-client-lifecycle-in-go.md) | Planned | | 2026-07-07 | [Comprehensive audit: stale files, state gaps, and next steps](2026-07-07-comprehensive-audit-and-next-steps.md) | Planned | | 2026-07-07 | [DB as single source of truth for agent knowledge](2026-07-07-db-as-source-of-truth.md) | Proposed | | 2026-07-07 | [Migrate bin/homelab CLI to Go oikos homelab](2026-07-07-migrate-bin-homelab-to-go.md) | Planned | -| 2026-07-07 | [Migrate bin/homelab CLI to Go oikos homelab](2026-07-07-migrate-bin-homelab-to-go.md) | Planned | ## Done