dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

docs: add client lifecycle plan, cleanup stale files, document repo for 3 audiences

Browse Source 
Problem: Repo had no developer guide, no client onboarding doc, no agent
dev instructions. Stale files (675KB SQL dump, one-off convert script,
legacy MCP builder) cluttered the tree. Client enrollment was a documented
intention with no Go implementation.

Changes:
- New docs: CONTRIBUTING.md (dev setup), CLIENTS.md (client onboarding),
  .agents/dev/CONTRIBUTING.md (agent codebase map)
- New plan: plans/2026-07-07-client-lifecycle-in-go.md — full client
  lifecycle (planned→provisioning→active→deprecated→destroyed) in Go,
  replacing archived Python secrets-issuance, adding client API endpoints
  and 6 missing MCP tools
- Cleanup: deleted archive/convert-wiki.py (one-off), archive/mcp/
  build_host_files.py (legacy), backups/pre-deploy-7f7d039.sql (local)
- Fixes: plans/index.md duplicate row removed, README.md repo layout
  updated for current state, AGENTS.md header points to new guides

Risk: low. Docs only + stale file deletion. No code changes. New plan is
proposal, not implementation.
Verification: git diff reviewed, all changes are prose/docs/plans.
This commit is contained in:
authentik Default Admin
2026-07-07 23:45:32 +02:00
parent 85b541a1cc
commit 638e313c66
9 changed files with 921 additions and 563 deletions
Download Patch File Download Diff File Expand all files Collapse all files

179
.agents/dev/CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,179 @@
# Agent developer guide
Instructions for AI agents working on the Oikos codebase. Read this after
[AGENTS.md](../../AGENTS.md) and [OIKOS.md](../OIKOS.md). Human developers:
see [CONTRIBUTING.md](../../CONTRIBUTING.md) for a human-friendly version.
## Codebase map
```
cmd/oikos/main.go Entry point. Subcommands: api, scheduler, notifier, migrate,
seed, export, secret, all
cmd/hermes/main.go Hermes MCP client gateway (standalone binary)
internal/httpapi/ REST + MCP server. Chi router. OpenAPI-generated types from
internal/httpapi/gen/api.gen.go. Strict server in impl.go.
internal/mcp/ MCP tool implementations (get_entity, search_knowledge, etc.)
internal/db/ Connection pool (pool.go), seed ingestion (seed.go), DB→YAML
export (export.go), type hierarchy (typetree.go)
internal/db/queries/ SQL query files → sqlc generates internal/db/sqlcgen/
internal/scheduler/ Observe loop: probes, signals, check_defs
internal/actuator/ SSH execution with circuit breaker + retry
internal/learning/ Pattern extraction, anomaly detection
internal/notifier/ Matrix notification + approval token generation
internal/policy/ Risk classifier (read policy.yaml → classify action)
internal/secrets/ Backend abstraction: Infisical (primary) + SOPS (fallback)
internal/domain/ Core types: entities, approvals, executions, signals, patterns
internal/ontology/ Type hierarchy validation, relationship checks
internal/knowledge/ Knowledge YAML seed ingestion
internal/config/ Config loading from env vars
api/openapi.yaml REST API contract. Source of truth for endpoints.
api/codegen.yaml oapi-codegen config → generates internal/httpapi/gen/
migrations/ Forward-only SQL. Format: NNN_name.up.sql. No down migrations.
seeds/ Bootstrap YAML. ontology.yaml, inventory.yaml, policy.yaml,
knowledge.yaml. Regenerated from DB via oikos export.
compose/ Dockerfiles. oikos/ (multi-stage), hermes/ (distroless).
Caddy config at compose/caddy/Caddyfile.oikos.
scripts/ Deploy, rollback, watchdog, verification, cutover checklist.
hermes/ Hermes config.yaml, SOUL.md, skills.
.agents/ Agent instruction files, domains, shared conventions, skills.
plans/ Design documents. active/ + done/.
docs/adr/ Architecture decision records. Numbered, prefix-sorted.
```
## Development loop
```bash
# Start dependencies
make dev
# Generate code after API/SQL changes
make generate
# Build
make build
# Run tests
make test # all unit tests
make test-db # integration tests (needs compose Postgres)
# Lint
make lint
# CI drift guard (run before commit)
make generate-check
```
## Adding a feature or phase
Oikos features follow a phase model (read [OIKOS.md](../OIKOS.md) for the
current phase status). To add a new capability:
1. **ADR first.** Write an architecture decision record in `docs/adr/` with
the next sequence number. Document the decision, context, alternatives
considered, and consequences.
2. **Plan.** If the change is non-trivial, create a plan in `plans/` following
the template in [page-templates.md](../shared/page-templates.md).
3. **Schema.** If the feature needs new DB tables, write a forward-only
migration in `migrations/`. Use `IF NOT EXISTS` for idempotency.
4. **API.** If the feature exposes endpoints, define them in
`api/openapi.yaml` first, then run `make generate`, then implement.
5. **Domain.** Add types to `internal/domain/` before adding logic.
6. **Tests.** Write tests alongside implementation. Integration tests go in
`*_test.go` in the relevant package, using the compose Postgres.
7. **Policy.** If the feature introduces new mutation types, update
`seeds/policy.yaml` and the classifier in `internal/policy/`.
8. **Run `make generate-check`** before commit to ensure generated code is
current.
## SQL conventions
- Queries live in `internal/db/queries/*.sql` with `-- name: FuncName :exec`
annotations for sqlc
- Use `pgx/v5` driver. UUIDs use `pgtype.UUID`, timestamps use `time.Time`
- CTEs for graph traversals (blast radius, dependency chains)
- CAGGs and retention policies for TimescaleDB hypertables
- FTS via `tsvector` + `tsquery` for knowledge search (migration 011)
## OpenAPI codegen
- Config: `api/codegen.yaml`. Uses `oapi-codegen/v2` with Chi server template
- Generated output: `internal/httpapi/gen/api.gen.go` — never hand-edit
- Strict server interface: `api.gen.go` generates the `StrictServerInterface`;
implement it in `internal/httpapi/impl.go`
- Problem+JSON errors via `internal/httpapi/problem.go` — RFC 9457 format
- Cursor pagination, If-Match/ETag, idempotency keys, SSE streaming
## Testing philosophy
- **Race detector always on.** `make test` runs `go test -race -cover ./...`
- **Integration tests** use the compose Postgres. Run with `make test-db`.
Each test creates + tears down its own schema namespace.
- **Coverage gates** in CI: policy + learning ≥ 80%, others ≥ 60%
- Tests use `testing.T` directly, no assertion library
- Table-driven tests for validation and classification logic
## Migration rules
- Forward-only. No down migrations (ADR 0008)
- Idempotent: use `IF NOT EXISTS`, `DO $$ BEGIN ... END $$` blocks
- Sequence numbers are sequential integers (001, 002, ...)
- Each migration file is `NNN_name.up.sql`
- Migrations are embedded in the binary via `migrations/embed.go`
## Seed files
- `seeds/ontology.yaml` — entity types, relationship types, lifecycles
(validated against schema in `internal/ontology/`)
- `seeds/inventory.yaml` — hosts, services, entities (the topology)
- `seeds/policy.yaml` — risk classes, approval rules, autonomy settings
- `seeds/knowledge.yaml` — documents, investigations, runbooks (DB is source
of truth; this file is the DR export)
- After DB changes via the API, run `make export` to regenerate seeds
## Secrets handling
- No secrets in code, config, or commits
- Dev secrets in `.env` (gitignored)
- Primary: Infisical (`internal/secrets/infisical.go`)
- Fallback: SOPS + age (`internal/secrets/sops.go`)
- Backend interface: `internal/secrets/backend.go`
- Machine identities via Infisical UniversalAuth
- In-memory cache with TTL for performance
## Staging and deployment
- CI pipeline: `.gitea/workflows/ci.yml` — lint, vet, vulncheck, test, docker build
- Deploy: `scripts/deploy.sh` — git pull → docker build → compose up → health check
- Watchdog: `scripts/watchdog.sh` — 2-minute cron, Matrix alert on failure
- Rollback: `scripts/rollback.sh` — checkout SHA + pg_restore
- Cutover checklist: `scripts/cutover-checklist.md`
## Writing conventions
Apply [writing-style.md](../shared/writing-style.md) for all committed prose.
Terse, reference-style, no marketing vocabulary. Code comments explain intent
and trade-offs, not mechanics.
Apply [caveman.md](../shared/caveman.md) for agent communication. The caveman
standard applies to agent *chat responses*, not committed documentation.
## Skills
Agent skills live under `.agents/skills/<name>/SKILL.md`. Each skill has a
frontmatter description that tools match against tasks. To add a skill:
1. Create `.agents/skills/<name>/SKILL.md`
2. Include frontmatter with description field
3. Document the procedure following the runbook template
4. Reference relevant files, commands, and policy classes
Skills that require code (e.g. linting) may include companion scripts in the
same directory.
## When in doubt
- Query MCP tools first (search_knowledge, get_entity)
- Read the relevant ADR in `docs/adr/`
- Grep the codebase: `rg <symbol> internal/`
- Check `plans/` for in-progress work that may conflict
- Classify any new mutation against `seeds/policy.yaml` before suggesting it

3
AGENTS.md
View File

@@ -4,6 +4,9 @@ You are running on a machine that is part of the **hubris** homelab. The full
context is in this checkout at `/opt/homelab-context/`. This file is the entry context is in this checkout at `/opt/homelab-context/`. This file is the entry
point. Read it once at start, then keep working. point. Read it once at start, then keep working.
- **New client?** Read [CLIENTS.md](CLIENTS.md) first.
- **Developing on this repo?** Also read [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
The operating model — OODA loop, risk classes, approval rules, the ontology, The operating model — OODA loop, risk classes, approval rules, the ontology,
and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation, and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation,
classify the action against `seeds/policy.yaml`; when the class requires classify the action against `seeds/policy.yaml`; when the class requires

117
CLIENTS.md Normal file
View File

@@ -0,0 +1,117 @@
# Oikos — client guide
If you are a homelab machine, this is what Oikos is and what it gives you.
If you are an AI agent running on this machine, also read [AGENTS.md](AGENTS.md).
## What is Oikos?
Oikos is the agentic operating system for the **hubris** Proxmox homelab. It
observes state, classifies actions against policy, executes approved
procedures, learns from outcomes, and notifies the operator. It runs as a
Docker stack on mac-mini and exposes an MCP server + REST API.
## What Oikos provides
| Capability | How you access it |
|------------|-------------------|
| Entity query (topology, blast radius) | MCP `get_entity`, `get_blast_radius` |
| Full-text knowledge search | MCP `search_knowledge` |
| Service status + logs | MCP `get_service_status`, `tail_log` |
| LXC inventory + state | MCP `list_lxcs`, `get_lxc_state` |
| Context cards | MCP `explain` |
| Pre-flight risk classification | MCP `preflight` |
| Change history | MCP `get_change_history` |
| State snapshot (health, disk, drift) | MCP `get_state_snapshot` |
| Secrets (Infisical) | REST API + `oikos secret` CLI |
| Approval tokens | Matrix via notifier |
All MCP tools are read-only. Mutations use the `homelab` CLI with operator
approval.
## Enrollment
Enrolled clients have a checkout at `/opt/homelab-context/`. If this
directory does not exist, the client is not enrolled.
To enroll:
```bash
# Run from an existing enrolled client
homelab client add <hostname>
```
This runs `bootstrap.sh` on the target, which:
1. Clones the repo to `/opt/homelab-context/`
2. Configures the auto-sync timer
3. Provisions agent persona from `hermes/SOUL.md` (on Hermes agents)
4. Installs Caveman tooling for terse communication
## After enrollment
### What changes on your machine
- `/opt/homelab-context/` — the repo checkout, your source of truth
- `/opt/homelab-context/inventory.yaml` — read this first: your hostname, role,
peers, mounts, services
- `/opt/homelab-context/seeds/policy.yaml` — rules for what actions you can
take autonomously vs. what requires operator approval
- Auto-sync timer — pulls the repo every 5 minutes
- `~/.hermes/SOUL.md` — agent persona (on Hermes agents)
- `~/bin/caveman_wrapper.sh` — terse communication tooling
### Your identity
Your identity in the homelab is defined in `inventory.yaml`. Run `hostname`
(Linux) or `scutil --get LocalHostName` (macOS), then look up your entry.
It tells you your role, what services you host, what's mounted, and your
mesh address.
### Source of truth hierarchy
1. **Postgres database** (runtime) — authoritative for entities, knowledge,
signals, ledger. Query via MCP or REST API.
2. **Repo at `/opt/homelab-context/`** — bootstrap seeds + documentation.
On disk, available offline.
3. **Never guess.** If data is missing, query MCP. If MCP is down, grep the
clone.
## The sync timer
Every 5 minutes, `systemd` (Linux) or `launchd` (macOS) runs:
1. `git pull` via `tools/post-pull.sh`
2. Any `tools/*.setup.sh` scripts that need to run
This keeps your checkout current and applies tooling updates automatically.
To trigger sync manually: `sudo homelab sync`.
## Making changes
- **Read state**: use MCP tools or the repo checkout
- **Mutate state** (restart, edit config, deploy): classify the action against
`seeds/policy.yaml`:
- `read_only` / `reversible_low` — execute directly
- `config_mutation` / `destructive` — request operator approval via the
`homelab` CLI
- **Secrets**: use Infisical (primary) or SOPS (fallback). Never hardcode.
- **Knowledge**: if you observe a discrepancy between docs and live state,
update the DB via the API in the same session. Run `oikos export` to
regenerate seeds.
## MCP endpoint
```
https://mcp.hubris.network/mcp
```
Available tools are listed in [AGENTS.md](AGENTS.md#3-the-mcp-server).
## Communication mode
Apply [.agents/shared/caveman.md](.agents/shared/caveman.md) — terse,
fragment-heavy communication. Drop filler, keep substance.
## Related
- [AGENTS.md](AGENTS.md) — full agent orientation (read this first)
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, risk classes
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide for the Oikos codebase

155
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,155 @@
# Contributing to Oikos
Developer guide for the Oikos codebase. If you are a homelab client consuming
Oikos, see [CLIENTS.md](CLIENTS.md). If you are an AI agent working on the
repo, see [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
## Dev setup
- **Go 1.26+** (see `go.mod` for pinned version)
- **PostgreSQL with TimescaleDB** — the compose stack includes `timescale/timescaledb:2.17.2-pg16`
- **Docker** for the full dev stack
```bash
# Start dependencies (Postgres + Redis)
docker compose --profile dev up -d
# Run all tests
make test
# Run integration tests (needs compose Postgres)
make test-db
# Build the binary
make build
```
## Project structure
```
cmd/oikos/ Single-binary entry point
cmd/hermes/ Hermes MCP client gateway
internal/ All Go packages
httpapi/ REST + MCP server (OpenAPI-generated)
mcp/ MCP tool implementations
db/ Connection pool, migrations, seeds, sqlc queries
scheduler/ Observe loop, probes, signals
actuator/ SSH execution
learning/ Pattern recognition, anomaly detection
notifier/ Matrix notifications, approval tokens
policy/ Risk classifier
secrets/ Infisical + SOPS backend
domain/ Core types: entities, approvals, signals, patterns
ontology/ Type hierarchy, relationship validation
knowledge/ Knowledge YAML seed ingestion
api/openapi.yaml API contract — the source of truth for endpoints
migrations/ Forward-only SQL migrations (TimescaleDB)
seeds/ Bootstrap YAML: ontology, inventory, policy, knowledge
compose/ Dockerfiles + Caddy config
scripts/ Deploy, watchdog, rollback
hermes/ Hermes config, persona, skills
.agents/ Agent instruction files + skills
plans/ Design documents
docs/adr/ Architecture decision records
```
## Commands
| Command | Purpose |
|---------|---------|
| `make build` | Build `oikos` binary |
| `make test` | Run all tests with race detection |
| `make test-db` | Run integration tests against compose Postgres |
| `make lint` | `go vet` + `golangci-lint` |
| `make generate` | Regenerate OpenAPI + sqlc code |
| `make generate-check` | CI drift guard — fail if generated code is stale |
| `make migrate` | Apply DB migrations |
| `make seed` | Ingest seeds into DB |
| `make export` | Export DB state to YAML seeds |
| `make dev` | Start compose dev stack |
| `make clean` | Remove binary + test cache |
## Conventions
### APIs are OpenAPI-first
The REST API is defined in `api/openapi.yaml`. Server code is generated with
`oapi-codegen` into `internal/httpapi/gen/`. To add an endpoint:
1. Add the path + schema to `api/openapi.yaml`
2. Run `make generate`
3. Implement the handler in `internal/httpapi/impl.go`
4. Add tests in `internal/httpapi/api_test.go`
Never hand-edit `internal/httpapi/gen/api.gen.go`.
### Database access is sqlc-first
SQL queries live in `internal/db/queries/*.sql`. Go code is generated with
`sqlc` into `internal/db/sqlcgen/`. Config in `sqlc.yaml`.
- Queries target pgx/v5 with UUID + timestamptz overrides
- Never hand-edit generated sqlc code
### Migrations are forward-only
SQL migrations live in `migrations/` as `NNN_name.up.sql`. There are no down
migrations (see [ADR 0008](docs/adr/0008-forward-only-migrations.md)).
Migrations are idempotent where possible (`IF NOT EXISTS`, `DO $$` blocks).
To add a migration:
1. Create `migrations/NNN_name.up.sql` with the next sequence number
2. Write the DDL
3. Run `make migrate` to apply
### Seeds are DB-generated
`seeds/*.yaml` are the bootstrap files used by `oikos seed`. After making
changes via the API, run `make export` to regenerate the seed files. These
files are version-controlled and serve as DR fallback.
### Writing style
Follow [.agents/shared/writing-style.md](.agents/shared/writing-style.md).
Documentation is reference prose, not marketing. Banned vocabulary includes
"robust", "seamless", "leverage", "utilize", "delve", "cutting-edge".
### Risk classification
Every mutation is classified against `seeds/policy.yaml` before execution.
Four risk classes: `read_only`, `reversible_low`, `config_mutation`,
`destructive`. The classifier can only lower autonomy relative to policy,
never raise it. When in doubt, escalate.
## CI
Gitea Actions runs on push to `main` and pull requests (`ci.yml`):
1. `go vet` + `golangci-lint` + `govulncheck`
2. Generated code drift check (`make generate-check`)
3. Build (`go build ./...`)
4. Test with race detector + coverage
5. Docker build verification (no push)
Coverage gates: policy + learning packages ≥ 80%, others ≥ 60%.
## PR workflow
1. Create a branch from `main`
2. Make changes, write tests
3. Run `make lint test generate-check`
4. Commit with a message following: problem → change → risk → verification
5. Push to Gitea; CI gates PRs on green
## Secrets
Secrets are managed by Infisical (primary) with SOPS as DR fallback. Never
hardcode secrets. Use environment variables from `.env` for local dev.
The `.env` and `.infisical-credentials` files are gitignored.
## Related
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
- [CLIENTS.md](CLIENTS.md) — for homelab clients consuming Oikos
- [docs/adr/](docs/adr/) — architecture decision records

23
README.md
View File

@@ -7,6 +7,8 @@ state, classifies actions against policy, executes approved procedures over SSH,
learns from outcomes, and escalates when uncertain. learns from outcomes, and escalates when uncertain.
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md). **For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md).
**For client machines:** see [CLIENTS.md](CLIENTS.md).
**For developers:** see [CONTRIBUTING.md](CONTRIBUTING.md).
## Quick start ## Quick start
@@ -101,29 +103,34 @@ oikos secret migrate # SOPS → Infisical
cmd/oikos/ Go entry point — single binary cmd/oikos/ Go entry point — single binary
cmd/hermes/ Hermes MCP client gateway cmd/hermes/ Hermes MCP client gateway
internal/ Go packages (httpapi, mcp, scheduler, actuator, learning, internal/ Go packages (httpapi, mcp, scheduler, actuator, learning,
notifier, policy, secrets, db, config, ontology, domain) notifier, policy, secrets, db, config, ontology, domain,
knowledge)
api/openapi.yaml API contract (OpenAPI 3.1) api/openapi.yaml API contract (OpenAPI 3.1)
migrations/ Forward-only SQL migrations (TimescaleDB) migrations/ Forward-only SQL migrations (TimescaleDB)
seeds/ Bootstrap YAML (ontology, inventory, policy) seeds/ Bootstrap YAML (ontology, inventory, policy, knowledge)
compose/ Dockerfiles + Caddy config compose/ Dockerfiles + Caddy config
scripts/ Deploy, watchdog, verification, rollback scripts/ Deploy, watchdog, verification, rollback
hermes/ Hermes config, persona, skills hermes/ Hermes config, persona, skills
archive/knowledge/ Narrative documentation (containers, hosts, infrastructure) .agents/ Agent instruction files, shared conventions, skills
.agents/ Agent instruction files + skills archive/ Historical reference (legacy wiki, plans, SOPS backups)
plans/ Design documents plans/ Design documents (active + done)
docs/adr/ Architecture decision records
``` ```
## For agents ## For agents
See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference: See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference:
- **Source of truth:** DB (runtime) then repo (bootstrap + docs) - **Source of truth:** DB (runtime) then seeds (bootstrap). Old wiki is
- **Mutations:** classify against policy, request approval for `destructive`/`config_mutation` archived at `archive/knowledge/` — use MCP `search_knowledge` instead.
- **Wiki:** files under `archive/knowledge/`, changelog at bottom of each page - **Mutations:** classify against policy, request approval for
`destructive`/`config_mutation`
- **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode - **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode
## Related ## Related
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology - [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
- [CLIENTS.md](CLIENTS.md) — client onboarding guide
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide
- [plans/](plans/) — design documents and cutover checklist - [plans/](plans/) — design documents and cutover checklist
- [docs/adr/](docs/adr/) — architecture decision records - [docs/adr/](docs/adr/) — architecture decision records

392
archive/convert-wiki.py
View File

@@ -1,392 +0,0 @@
#!/usr/bin/env python3
"""One-shot: convert knowledge/wiki/ to seeds/knowledge.yaml."""
import os, re, yaml
from pathlib import Path
from hashlib import sha256
REPO = Path("/Users/dtoro/Projects/oikos")
WIKI = REPO / "archive" / "knowledge"
SOURCES = REPO / "archive" / "knowledge"
GLOSSARY = REPO / "archive" / "knowledge" / "GLOSSARY.md"
# Maps wiki path components to entity slugs
# Format: (path_pattern, entity_slug)
PATH_TO_ENTITY = {
# Containers
"containers/101-jellyfin": "lxc:jellyfin",
"containers/102-nfs-export": "lxc:nfs-export",
"containers/103-paperless": "lxc:paperless",
"containers/104-gitea": "lxc:gitea",
"containers/105-apps": "lxc:apps",
"containers/106-auth-outpost": "lxc:auth-outpost",
"containers/107-dns": "lxc:dns",
"containers/114-nextcloud": "lxc:nextcloud",
"containers/118-elementsynapse": "lxc:elementsynapse",
"containers/119-sophia": "lxc:sophia",
"containers/120-mule-images": "lxc:mule-images",
"containers/121-caddy": "lxc:caddy",
"containers/122-arriman": "lxc:arriman",
"containers/128-trmnl": "lxc:trmnl",
"containers/129-house": "lxc:house",
"containers/130-grimmory": "lxc:grimmory",
"containers/131-teddycloud": "lxc:teddycloud",
"containers/132-rclone": "lxc:rclone",
"containers/133-seanime": "lxc:seanime",
"containers/134-romm": "lxc:romm",
# Hosts
"hosts/hubris": "host:hubris",
"hosts/strong": "host:strong",
# VMs
"vms/100-zimaos": "vm:zimaos",
"vms/108-haos": "vm:haos",
# Infrastructure → services
"infrastructure/auto-deploy": None,
"infrastructure/backups": None,
"infrastructure/dns": "service:dns",
"infrastructure/homelab-context": "service:homelab-mcp",
"infrastructure/ingress": "service:caddy",
"infrastructure/media-permissions": "service:jellyfin",
"infrastructure/mesh": None,
"infrastructure/monitoring": None,
"infrastructure/network": None,
"infrastructure/ssh-access": None,
"infrastructure/topology": None,
"infrastructure/vps-hardening": "host:netbird-vps",
}
def parse_page(path):
"""Parse a wiki page into structured sections."""
if not path.exists():
return None
text = path.read_text()
lines = text.split('\n')
# Title is first H1
title = ""
for line in lines:
if line.startswith('# ') and not line.startswith('## '):
title = line[2:].strip()
break
# Find sections by H2 headings
sections = {}
current_heading = "_preamble"
current_content = []
for line in lines:
if line.startswith('## ') and not line.startswith('### '):
if current_content:
sections[current_heading] = '\n'.join(current_content).strip()
current_heading = line[3:].strip().lower()
current_content = []
else:
current_content.append(line)
if current_content:
sections[current_heading] = '\n'.join(current_content).strip()
# Parse at-a-glance
at_glance = {}
ag_text = sections.get('at a glance', '')
for line in ag_text.split('\n'):
line = line.strip()
# Strip leading bullet
line = re.sub(r'^[-*]\s+', '', line)
# Match **Key:** value or **Key Word:** value
m = re.match(r'\*\*([^*]+?):?\*\*\s+(.+)', line)
if not m:
m = re.match(r'([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*):\s+(.+)', line)
if m:
key = m.group(1).lower().strip().replace(' ', '_').replace('/', '_')
val = m.group(2).strip()
# Strip trailing parenthetical notes
val = re.sub(r'\s*\([^)]*\)$', '', val)
# Strip markdown formatting from value
val = re.sub(r'\*\*([^*]+)\*\*', r'\1', val)
val = re.sub(r'`([^`]+)`', r'\1', val)
# Simplify link text
val = re.sub(r'\[([^\]]+)\]\([^)]+\)', r'\1', val)
val = re.sub(r'↔', '', val).strip()
# Normalize keys
key_map = {
'cores': 'cores', 'core': 'cores',
'ram': 'ram', 'memory': 'ram',
'mounts': 'mounts', 'mount': 'mounts',
'host': 'host', 'ip': 'ip',
'public_host': 'public_host', 'public_hostname': 'public_host',
'lan_ip': 'lan_ip',
'os': 'os', 'kind': 'kind',
'runtime': 'runtime', 'role': 'role',
'pve_id': 'pve_id', 'privilege': 'privileged',
'resources': 'resources', 'gpu': 'gpu',
'swap': 'swap', 'rootfs': 'rootfs',
'version': 'version', 'hardware': 'hardware',
}
key = key_map.get(key, key)
at_glance[key] = val
# Parse changelog
changelog = []
cl_text = sections.get('changelog', '')
current_entry = None
for line in cl_text.split('\n'):
m = re.match(r'###\s+(\d{4}-\d{2}-\d{2})\s+[—–-]\s+(.+)', line)
if m:
if current_entry:
changelog.append(current_entry)
current_entry = {'date': m.group(1), 'title': m.group(2).strip(), 'body': ''}
elif current_entry is not None:
stripped = line.strip()
if stripped and not stripped.startswith('#'):
if current_entry['body']:
current_entry['body'] += ' '
current_entry['body'] += stripped
if current_entry:
changelog.append(current_entry)
# Tags from path
parts = path.relative_to(REPO).parts
tags = []
if 'containers' in parts:
tags.append('container')
elif 'hosts' in parts:
tags.append('host')
elif 'vms' in parts:
tags.append('vm')
elif 'infrastructure' in parts:
tags.append('infrastructure')
# Determine slug from relative path
rel = str(path.relative_to(WIKI))
slug = rel.replace('.md', '')
# Entity mapping
entity_slug = PATH_TO_ENTITY.get(slug, None)
return {
'slug': slug,
'title': title,
'content': text,
'entity_slug': entity_slug,
'tags': tags,
'at_glance': at_glance,
'changelog': changelog,
'is_investigation': 'investigations' in rel,
}
def parse_investigation(path):
"""Parse an investigation page."""
if not path.exists():
return None
text = path.read_text()
lines = text.split('\n')
title = ""
for line in lines:
if line.startswith('# '):
title = line[2:].strip()
break
# Extract date from title or filename
date = ""
status = "resolved"
duration = ""
for line in lines[:30]:
m = re.search(r'(\d{4}-\d{2}-\d{2})', line)
if m:
date = m.group(1)
break
for line in lines:
if '**Status:**' in line:
status = line.split('**Status:**')[-1].strip().lower()
if '**Duration:**' in line:
duration = line.split('**Duration:**')[-1].strip()
# Extract entity references for about_slugs
about_slugs = []
entity_patterns = [
(r'\bcaddy\b', 'service:caddy'),
(r'\bauthentik\b', 'service:authentik'),
(r'\bdns\b', 'service:dns'),
(r'\bgitea\b', 'service:gitea'),
(r'\bjellyfin\b', 'service:jellyfin'),
(r'\bmatrix\b', 'service:matrix'),
(r'\bpaperless\b', 'service:paperless'),
(r'\bnextcloud\b', 'service:nextcloud'),
(r'\bartifacto\b', 'service:artifacto'),
(r'\barriman\b', 'lxc:arriman'),
(r'\btrmnl\b', 'service:trmnl'),
(r'\bmac-mini\b', 'ws:mac-mini'),
(r'\bhubris\b', 'host:hubris'),
(r'\bstrong\b', 'host:strong'),
]
for pattern, slug in entity_patterns:
if re.search(pattern, text, re.IGNORECASE):
about_slugs.append(slug)
rel = str(path.relative_to(WIKI))
slug = rel.replace('.md', '')
return {
'slug': slug,
'title': title,
'date': date,
'status': status,
'duration': duration,
'content': text,
'about_slugs': about_slugs,
'tags': ['investigation'],
}
def main():
documents = []
investigations = []
runbooks = []
# Container pages
containers_dir = WIKI / "containers"
for f in sorted(containers_dir.glob("*.md")):
if 'index' in f.name:
continue
if f.parent.name == 'archive':
continue
result = parse_page(f)
if result and result['title']:
documents.append(result)
print(f" document: {result['slug']} → {result['entity_slug']}")
# Host pages
hosts_dir = WIKI / "hosts"
for f in sorted(hosts_dir.glob("*.md")):
if 'index' in f.name:
continue
result = parse_page(f)
if result and result['title']:
documents.append(result)
print(f" document: {result['slug']} → {result['entity_slug']}")
# VM pages
vms_dir = WIKI / "vms"
for f in sorted(vms_dir.glob("*.md")):
if 'index' in f.name:
continue
result = parse_page(f)
if result and result['title']:
documents.append(result)
print(f" document: {result['slug']} → {result['entity_slug']}")
# Infrastructure pages
infra_dir = WIKI / "infrastructure"
for f in sorted(infra_dir.glob("*.md")):
if 'index' in f.name:
continue
result = parse_page(f)
if result and result['title']:
documents.append(result)
print(f" document: {result['slug']} → {result['entity_slug']}")
# Investigation pages
inv_dir = SOURCES / "investigations"
for f in sorted(inv_dir.glob("*.md")):
if 'index' in f.name:
continue
result = parse_investigation(f)
if result and result['title']:
investigations.append(result)
print(f" investigation: {result['slug']} → {result['about_slugs']}")
# Archive investigations too
inv_archive = inv_dir / "archive"
if inv_archive.exists():
for f in sorted(inv_archive.glob("*.md")):
result = parse_investigation(f)
if result and result['title']:
investigations.append(result)
print(f" investigation: {result['slug']} → {result['about_slugs']}")
# Runbooks from .agents/skills/
skills_dir = REPO / ".agents" / "skills"
for skill_dir in sorted(skills_dir.iterdir()):
if not skill_dir.is_dir():
continue
skill_file = skill_dir / "SKILL.md"
if not skill_file.exists():
continue
text = skill_file.read_text()
lines = text.split('\n')
title = ""
for line in lines:
if line.startswith('# '):
title = line[2:].strip()
break
# Extract risk_class and entity_type from frontmatter
risk_class = "read_only"
entity_type = "service"
for line in lines[:30]:
m = re.match(r'\*\*risk_class:\*\*\s*(\w+)', line, re.IGNORECASE)
if m:
risk_class = m.group(1)
m = re.match(r'\*\*applies_to:\*\*\s*(\w[\w-]*)', line, re.IGNORECASE)
if m:
entity_type = m.group(1)
name = skill_dir.name
runbooks.append({
'slug': name,
'name': title or name,
'risk_class': risk_class,
'entity_type': entity_type,
'procedure': {}, # SKILL.md is narrative, not structured yet
'content': text,
'tags': ['skill', 'runbook'],
})
print(f" runbook: {name}")
# Build seed YAML
seed = {
'version': 1,
'documents': [{
'slug': d['slug'],
'title': d['title'],
'content': d['content'],
'entity_slug': d['entity_slug'],
'tags': d['tags'],
'at_glance': d['at_glance'],
'changelog': d['changelog'],
} for d in documents],
'investigations': [{
'slug': i['slug'],
'title': i['title'],
'date': i['date'],
'status': i['status'],
'duration': i['duration'],
'content': i['content'],
'about_slugs': i['about_slugs'],
'tags': i['tags'],
} for i in investigations],
'runbooks': [{
'slug': r['slug'],
'name': r['name'],
'risk_class': r['risk_class'],
'entity_type': r['entity_type'],
'procedure': r['procedure'],
'content': r['content'],
'tags': r['tags'],
} for r in runbooks],
}
out_path = REPO / "seeds" / "knowledge.yaml"
out_path.write_text(yaml.dump(seed, allow_unicode=True, width=120, sort_keys=False))
print(f"\nWrote {out_path}")
print(f" {len(documents)} documents")
print(f" {len(investigations)} investigations")
print(f" {len(runbooks)} runbooks")
if __name__ == "__main__":
main()

162
archive/mcp/build_host_files.py
View File

@@ -1,162 +0,0 @@
#!/usr/bin/env python3
"""
Generate hosts/<name>.yaml from inventory.yaml.
Run from the repo root:
python3 mcp/build_host_files.py # writes files, exits non-zero on diff
python3 mcp/build_host_files.py --check # exits non-zero if any output differs
Designed to be wired into a pre-commit hook or Gitea Action so generated
hosts/*.yaml never drift from inventory.yaml.
"""
from __future__ import annotations
import argparse
import difflib
import os
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("PyYAML is required: pip install pyyaml", file=sys.stderr)
sys.exit(2)
REPO = Path(__file__).resolve().parent.parent
INVENTORY = REPO / "inventory.yaml"
HOSTS_DIR = REPO / "hosts"
GENERATED_BANNER = (
"# Generated by mcp/build_host_files.py from inventory.yaml.\n"
"# Do NOT edit by hand — your changes will be overwritten.\n"
"# Source of truth: ../inventory.yaml\n"
)
def narrative_page(name: str, kind: str, pve_id: int | None) -> str | None:
"""Best-guess path to the human-authored narrative page for this host."""
if kind == "proxmox-host":
candidate = REPO / "hosts" / f"{name}.md"
elif kind == "lxc":
candidate = REPO / "containers" / f"{pve_id}-{name}.md"
elif kind == "vm":
candidate = REPO / "vms" / f"{pve_id}-{name}.md"
else:
return None
if candidate.exists():
return str(candidate.relative_to(REPO))
return None
def build_one(name: str, entry: dict, inventory: dict) -> dict:
"""Project the entry for a single host into a per-host yaml record."""
services = inventory.get("services", {})
mesh = inventory.get("mesh", {})
pve_id = entry.get("pve_id")
# Services this host runs: scan inventory.services for matching backend.
runs_services = sorted(
svc for svc, sentry in services.items()
if isinstance(sentry, dict) and sentry.get("backend") == name
)
record = {
"name": name,
"kind": entry.get("kind"),
"os": entry.get("os"),
"role": entry.get("role"),
# Oikos lifecycle (oikos/ontology.yaml); absent in inventory = active
"state": entry.get("state", "active"),
"host": entry.get("host"),
"pve_id": pve_id,
"storage": entry.get("storage"),
"depends_on": entry.get("depends_on", []),
"lan_ip": entry.get("lan_ip"),
"mesh": entry.get("mesh", {}),
"mesh_globals": {
"primary": mesh.get("primary"),
"accepted": mesh.get("accepted"),
},
"peers": entry.get("peers", []),
"mounts": entry.get("mounts", []),
"public_host": entry.get("public_host"),
"public_hosts": entry.get("public_hosts", []),
"ssh": entry.get("ssh", {}),
"runs": entry.get("runs", []) + runs_services,
"services_hosted": [
{"name": svc, **services[svc]} for svc in runs_services
],
"notes": entry.get("notes", []),
"age_pubkey": entry.get("age_pubkey", ""),
"see_also": [
page for page in [narrative_page(name, entry.get("kind", ""), pve_id)]
if page
],
"mcp_endpoint": services.get("homelab_mcp", {}).get("endpoint"),
"secrets_issuance_endpoint": (
services.get("secrets_issuance", {}).get("endpoint")
),
}
# Strip None and empty containers so the file stays readable.
return {k: v for k, v in record.items() if v not in (None, {}, [], "")}
def serialize(record: dict) -> str:
return GENERATED_BANNER + yaml.safe_dump(
record, sort_keys=False, default_flow_style=False, width=100
)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--check", action="store_true",
help="exit 1 if any output would change (don't write)")
args = parser.parse_args()
inventory = yaml.safe_load(INVENTORY.read_text())
hosts = inventory.get("hosts", {})
HOSTS_DIR.mkdir(exist_ok=True)
desired: dict[Path, str] = {}
for name, entry in hosts.items():
desired[HOSTS_DIR / f"{name}.yaml"] = serialize(build_one(name, entry, inventory))
diff_count = 0
for path, content in desired.items():
existing = path.read_text() if path.exists() else ""
if existing != content:
diff_count += 1
if args.check:
diff = difflib.unified_diff(
existing.splitlines(keepends=True),
content.splitlines(keepends=True),
fromfile=str(path),
tofile=str(path) + " (generated)",
)
sys.stdout.writelines(diff)
else:
path.write_text(content)
print(f"wrote {path.relative_to(REPO)}")
# Clean up orphans (file exists but host removed from inventory).
for existing_path in HOSTS_DIR.glob("*.yaml"):
if existing_path not in desired:
diff_count += 1
if args.check:
print(f"orphan: {existing_path.relative_to(REPO)} (would delete)")
else:
existing_path.unlink()
print(f"deleted orphan {existing_path.relative_to(REPO)}")
if args.check and diff_count > 0:
print(f"\n{diff_count} file(s) would change. Run without --check to write.",
file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())

451
plans/2026-07-07-client-lifecycle-in-go.md Normal file
View File

@@ -0,0 +1,451 @@
# Plan: Client lifecycle — enrollment through deprecation in Oikos Go
**Status:** Planned (2026-07-07)
## Goal
Define and implement the complete lifecycle of a homelab client in the Oikos Go
runtime: how a new machine is provisioned, enrolled, given secrets, synced,
operated, and eventually deprecated (or decommissioned or destroyed). Every
state transition feeds the Postgres DB as the authoritative source of truth.
No step depends on the archived Python `secrets-issuance` server or the
non-existent `bin/homelab` CLI.
## Current state — what exists vs. what runs
| Component | Exists? | Runs? | Notes |
|-----------|---------|-------|-------|
| `bootstrap.sh` (684 lines) | ✅ repo | ⚠️ references dead endpoints | Calls `https://secrets.hubris.network/issue` (Python server, stopped per Phase 6). Symlinks `bin/homelab` (file absent). References `tools/*.setup.sh` (files absent). |
| `archive/secrets-issuance/server.py` | ✅ archive | ❌ stopped (Phase 6) | Issued age keys, validated mesh IP. No Go replacement. |
| `archive/mcp/` (build_host_files.py, deleted) | ❌ deleted | ❌ | Legacy host file builder. |
| `inventory.yaml` (root) | ✅ | ⚠️ edited manually | Flat `hosts:` + `services:` layout. Diverges from `seeds/inventory.yaml` entity-relationship format. |
| `seeds/inventory.yaml` | ✅ | ✅ ingested into DB | Entity-relationship format with slugs (`host:hubris`, `ws:mac-mini`). No translation path from root format. |
| Go entity API (`POST/GET/PATCH /entities`) | ✅ | ✅ | Generic CRUD. No client-specific validation, no key issuance, no lifecycle gating. |
| Go MCP server | ✅ | ✅ | 15 tools. Missing `whoami`, `explain`, `preflight`, `get_change_history`, `get_state_snapshot`. |
| `oikos secret` CLI (Infisical/SOPS) | ✅ | ✅ | Secrets read/migrate/export. No client-key provisioning. |
| Sync timer (post-pull.sh) | ✅ | ⚠️ partially broken | References `tools/*.setup.sh` (glob returns zero files). `setup-caveman.sh` and `setup-hermes-soul.sh` documented but absent. |
**Takeaway**: Enrollment today runs on shell scripts calling archived Python
services. The Go runtime has zero awareness of client lifecycle. This plan
closes that gap — the DB becomes the sole engine for client identity,
secrets, state, and lifecycle transitions.
## Target architecture
```
┌──────────────────────────────────────────────────────────────┐
│ NEW CLIENT (bare machine) │
│ │
│ 1. curl bootstrap.sh | sudo bash │
│ → clones repo, installs sync timer │
│ → calls POST /api/v1/clients/enroll (new endpoint) │
│ → receives age keypair from Oikos API │
│ → writes /etc/age/key.txt │
│ → sync timer starts pulling every 5 min │
└──────────────────────────┬───────────────────────────────────┘
│ POST /api/v1/clients/enroll
▼
┌──────────────────────────────────────────────────────────────┐
│ OIKOS API (Go, :8090) │
│ │
│ POST /api/v1/clients/enroll — issue age key, set state │
│ POST /api/v1/clients/{slug}/activate — provisioning→active │
│ POST /api/v1/clients/{slug}/deprecate — active→deprecated │
│ POST /api/v1/clients/{slug}/destroy — deprecated→destroyed │
│ GET /api/v1/clients/{slug}/secrets — client's accessible │
│ secrets (Infisical lookup by machine identity) │
│ MCP whoami(hostname) — client self-introspection │
│ MCP explain(service) — compact context card │
│ MCP preflight(service) — risk classification │
│ MCP get_change_history(entity) — ledger entries │
│ MCP get_state_snapshot() — last scheduler pass │
└──────────────────────────┬───────────────────────────────────┘
│ writes
▼
┌──────────────────────────────────────────────────────────────┐
│ POSTGRES (TimescaleDB) │
│ │
│ entities table: slug, type, name, state, attributes (JSONB) │
│ entity_status: health, disk, drift count (scheduler) │
│ audit_log: every state transition, enrollment, revocation │
│ executions: approved actions, results │
│ secrets (via Infisical): age keys, API tokens │
└──────────────────────────────────────────────────────────────┘
```
## Client lifecycle: state machine
```
[planned] ──→ provisioning ──→ active ──→ migrating ──→ active
│ │ │
│ │ ├──→ deprecated ──→ destroyed
│ │ │
│ └──→ failed └──→ failed
│
└──→ destroyed (cancelled)
```
### State: `planned`
The operator declares intent. A client entity exists in the DB with state
`planned` but has no host, no keys, no sync.
**Entry condition**: Operator creates the entity via API or seed file.
**Required attributes**:
- `slug` — `ws:<hostname>` for workstations, `host:<hostname>` for servers
- `type` — `workstation`, `standalone-server`, or `proxmox-host`
- `name` — human-readable name
- `lan_ip` — expected LAN IP (reserved in DHCP)
- `os` — `linux` or `macos`
- `role` — free-text description of what this machine does
- `mesh.expected_type` — `netbird` or `tailscale` (which mesh it will join)
- `ssh.user` — login user (default `root`)
**Allowed transitions**: `→ provisioning` (operator triggers), `→ destroyed` (cancelled).
### State: `provisioning`
The machine has been declared. Operator runs `bootstrap.sh` on the target,
which calls the enrollment API. The API validates identity (mesh IP matches
expected subnet, hostname matches slug), issues an age keypair, and records
the public key. The sync timer starts pulling the repo.
**Go API**: `POST /api/v1/clients/enroll`
```json
{
"slug": "ws:new-laptop",
"hostname": "new-laptop",
"mesh_ip": "100.122.x.x"
}
```
**What the enrollment endpoint does**:
1. Looks up entity by slug — must exist, must be in state `planned` or `provisioning`
2. Validates mesh IP is in `100.122.0.0/16` (Netbird) or `100.64.0.0/10` (Tailscale) or `192.168.8.0/24` (LAN)
3. Validates hostname has no conflicting mesh IP already recorded
4. Generates an age keypair (`age-keygen`)
5. Stores the private key in Infisical under path `/clients/<slug>/age-key`
6. Creates an Infisical machine identity for the client (UniversalAuth)
7. Updates entity `attributes` with `age_pubkey`, `mesh_ip`, `enrolled_at`
8. Writes audit log: `client.enrolled`
9. Returns the age private key, Infisical client ID + secret, and machine identity token
**Response** (to bootstrap.sh, over mesh — TLS + mesh IP validation):
```json
{
"age_private_key": "AGE-SECRET-KEY-...",
"age_public_key": "age1...",
"infisical_client_id": "...",
"infisical_client_secret": "...",
"machine_identity_token": "..."
}
```
**Bootstrap script changes**:
- Remove call to `https://secrets.hubris.network/issue`
- Replace with `POST /api/v1/clients/enroll` to `https://oikos.hubris.network`
- Remove `--no-secrets` / `--no-mesh` flags (or keep as escape hatches with degraded state)
- Remove symlink to `bin/homelab` (file doesn't exist)
- After receiving keys, bootstrap.sh writes `/etc/age/key.txt` (0600) and `/etc/infisical/identity` (0600)
**Pre-built bootstrap**: The bootstrap.sh is served from the Gitea repo raw URL
(already the case). After this plan, it calls Oikos API instead of the dead
Python service.
**Allowed transition**: `→ active` (when `age-key-enrolled`, `mesh-joined`,
`doc-page-complete` checks pass).
### State: `active`
Normal operation. The client pulls the repo every 5 minutes, uses its age key
to decrypt SOPS secrets (fallback), and authenticates to Infisical via its
machine identity (primary). The MCP `whoami(hostname)` tool returns its
entity record, peer list, accessible secrets, and current health.
**Go enforcement of transition checks** (`provisioning → active`):
- `age-key-enrolled-if-needed`: entity.attributes.age_pubkey is non-empty
- `mesh-joined-if-needed`: entity.attributes.mesh_ip is non-empty
- `ingress-live-if-public`: skipped for workstations (no public ingress)
- `health-check-answering`: scheduler probe passes for this entity
- `doc-page-complete`: entity has at least one `documents` edge
- `inventory-in-db`: entity exists in DB with all required attributes
**API**: `POST /api/v1/clients/{slug}/activate`
- Validates all `provisioning → active` transition checks
- Sets state to `active`
- Writes audit log: `client.activated`
**MCP tools active clients get**:
- `whoami(hostname)` — returns entity record, peers, secrets list, health
- `list_my_secrets(caller_pubkey?)` — secrets this client can decrypt
**Allowed transitions**: `→ migrating`, `→ deprecated`, `→ failed`.
### State: `migrating`
Client is being moved — OS reinstall, hardware swap, role change. Inbound edges
still exist; no deprovisioning has started.
**Allowed transition**: `→ active` (migration complete, post-verify passes).
**API**: `POST /api/v1/clients/{slug}/migrate` (sets state, links migration plan).
### State: `deprecated`
Client is being phased out. Services moved off, mesh disconnected, secrets
rotation started. The deprecation gate (`no-inbound-edges`) blocks `→ destroyed`
until all `depends-on`, `hosts`, `provides`, and `mounts` edges are gone.
**API**: `POST /api/v1/clients/{slug}/deprecate`
- Validates `replacement-live-or-role-retired`: operator confirms replacement exists or role is no longer needed
- Sets state to `deprecated`
- Writes audit log: `client.deprecated`
**Allowed transitions**: `→ active` (un-deprecate), `→ destroyed`.
### State: `destroyed`
Client is gone. All edges removed, secrets revoked, archaeology entry written.
**API**: `POST /api/v1/clients/{slug}/destroy`
- Validates all `deprecated → destroyed` transition checks:
- `backups-verified`: any data on this client was backed up
- `secrets-revoked-and-rekeyed`: age key removed from Infisical, SOPS recipients updated, machine identity deleted
- `ingress-and-dns-removed`: no remaining DNS records or Caddy backends
- `no-inbound-edges`: zero `depends-on`, `hosts`, `provides`, `mounts` edges pointing to this entity
- `archaeology-entry`: writes a record explaining why and when
- Sets state to `destroyed`
- Revokes Infisical machine identity
- Removes age public key from `.sops.yaml`
- Writes audit log: `client.destroyed`
### State: `failed`
Something went wrong during provisioning or operation. Requires operator
intervention. Treated as informational — no automatic recovery.
**API**: `POST /api/v1/clients/{slug}/fail`
- Sets state to `failed`
- Requires `reason` field explaining what broke
- Writes audit log: `client.failed`
## Secrets integration
### Age key lifecycle
```
planned ────────────→ no key exists
provisioning ───────→ keypair generated, pubkey stored in entity attributes,
private key delivered to client via enroll response,
private key stored in Infisical under /clients/<slug>/age-key
active ─────────────→ key used for SOPS decryption fallback, authenticated
to Infisical via machine identity for primary secrets
deprecated ─────────→ key still valid, but rotation initiated
destroyed ──────────→ key revoked from Infisical, removed from .sops.yaml,
machine identity deleted
```
### Infisical machine identity
Each client gets an Infisical machine identity during enrollment. This is the
primary secrets path — the age key is fallback for SOPS-encrypted DR files.
- **Client ID + Secret** returned in enroll response
- **Scoped to paths**: `/clients/<slug>/*`, `/shared/*`
- **Revoked on destroy**: identity deleted, access gone
### SOPS fallback
The age public key is added to `.sops.yaml` recipients during enrollment.
On destroy, it is removed via `oikos secret export-sops` regeneration.
### bootstrap.sh changes
```diff
- # calls https://secrets.hubris.network/issue (Python, dead)
- AGE_KEY=$(curl -s -X POST "$ISSUANCE_URL" ...)
-
+ # calls Oikos API enrollment endpoint
+ ENROLL_RESP=$(curl -s -X POST "$OIKOS_URL/api/v1/clients/enroll" \
+ -H "Content-Type: application/json" \
+ -d "{\"slug\":\"ws:$HNAME\",\"hostname\":\"$HNAME\",\"mesh_ip\":\"$MESH_IP\"}")
+ AGE_PRIVKEY=$(echo "$ENROLL_RESP" | jq -r '.age_private_key')
```
## DB integration
### New migration
`012_client_enrollment.up.sql`:
```sql
-- No new tables needed — entities table already holds clients.
-- Add enrollment-specific attributes validation via check constraints
-- or application-level validation.
-- Enforce slug format for machine entities
-- ws:<hostname> for workstations, host:<hostname> for servers
-- (application-level validation in Go, not a DB constraint)
-- Add index for slug-based client lookups
CREATE INDEX IF NOT EXISTS idx_entities_slug_type
ON entities (slug, type)
WHERE type IN ('workstation', 'standalone-server', 'proxmox-host');
```
### Entity attributes schema (for `machine` types)
```json
{
"cpu_arch": "arm64",
"ram_gb": 16,
"os": "macos",
"lan_ip": "192.168.8.175",
"mesh": {
"netbird": {"ip": "100.122.x.x", "fqdn": "hostname.netbird.selfhosted"}
},
"ssh": {"user": "dtoro"},
"age_pubkey": "age1...",
"enrolled_at": "2026-07-07T12:00:00Z",
"enrolled_by": "ws:mac-mini",
"infisical_identity_id": "identity_abc123"
}
```
All attributes are stored in the `attributes` JSONB column on the `entities`
table. Validation happens at the application layer (Go) using the schema
defined in `seeds/ontology.yaml`.
## MCP tools to add
These are documented in AGENTS.md section 3 but not implemented in the Go MCP
server. Implementation: register in `internal/mcp/server.go`.
| Tool | Input | Output | Implementation |
|------|-------|--------|----------------|
| `whoami` | `hostname` | Entity record, peers, accessible secrets, health | DB lookup by slug derived from hostname |
| `list_my_secrets` | `caller_pubkey?` | Secrets this client can decrypt | Infisical list + SOPS `.sops.yaml` match |
| `explain` | `service_slug` | Compact context card: type, state, health, relations, last change | DB join: entity + entity_status + audit_log |
| `preflight` | `service_slug` | Risk class, approval requirement, verification command | Policy classifier on the entity's type |
| `get_change_history` | `entity_slug`, `limit` | Last N audit_log entries for entity | DB query on audit_log table |
| `get_state_snapshot` | none | Last scheduler Observe pass: health, disk, drift count | DB query on entity_status + signals |
## API endpoints to add
Add to `api/openapi.yaml`, regenerate with `make generate`, implement in
`internal/httpapi/impl.go`.
| Method | Path | Scope | Purpose |
|--------|------|-------|---------|
| `POST` | `/api/v1/clients/enroll` | agent | Issue age key, validate mesh, set state → provisioning |
| `POST` | `/api/v1/clients/{slug}/activate` | operator | Run transition checks, state → active |
| `POST` | `/api/v1/clients/{slug}/deprecate` | operator | State → deprecated |
| `POST` | `/api/v1/clients/{slug}/destroy` | operator | Run destroy checks, revoke secrets, state → destroyed |
| `POST` | `/api/v1/clients/{slug}/fail` | operator | State → failed with reason |
| `GET` | `/api/v1/clients/{slug}/secrets` | agent | List secrets this client can access |
## Files changed
| File | Change |
|------|--------|
| `bootstrap.sh` | Replace `secrets.hubris.network/issue` call with `POST /api/v1/clients/enroll`. Remove dead symlinks. |
| `api/openapi.yaml` | Add client enrollment, lifecycle, and secret endpoints |
| `internal/httpapi/impl.go` | Implement client lifecycle handlers |
| `internal/db/queries/clients.sql` | Add client-specific sqlc queries |
| `internal/mcp/server.go` | Register whoami, explain, preflight, get_change_history, get_state_snapshot, list_my_secrets |
| `internal/secrets/infisical.go` | Add `CreateMachineIdentity`, `DeleteMachineIdentity`, `StoreClientKey` |
| `internal/ontology/validate.go` | Implement lifecycle transition checks for infrastructure lifecycle |
| `seeds/ontology.yaml` | Add client-specific attributes schema for machine types |
| `migrations/012_client_enrollment.up.sql` | Index for slug+type lookups |
| `AGENTS.md` | Update MCP tool list to match actual implementation |
| `CLIENTS.md` | Update enrollment flow to reference Oikos API, not Python issuance |
| `CONTRIBUTING.md` | Add client lifecycle as a documented extension point |
## Files deleted or deprecated
| File | Disposition |
|------|-------------|
| `archive/secrets-issuance/` | Already archived. Add deprecation notice referencing this plan. |
| `archive/secrets-sops-backup/` | Keep for DR. Add note that new clients use Infisical, SOPS is fallback. |
| Any reference to `bin/homelab` | Delete or comment out in bootstrap.sh; CLI doesn't exist. |
| `tools/*.setup.sh` references | Either create the files or remove the auto-setup convention from post-pull.sh. |
## Phased implementation
### Phase 1 — API + DB (P0, this week)
1. Write `migrations/012_client_enrollment.up.sql`
2. Add client endpoints to `api/openapi.yaml`
3. Run `make generate`
4. Implement enrollment handler (`POST /api/v1/clients/enroll`):
- Age key generation
- Infisical machine identity creation
- Entity attribute update
- Audit log write
5. Implement lifecycle transition handlers (activate, deprecate, destroy, fail)
6. Implement `GET /api/v1/clients/{slug}/secrets`
7. Update `seeds/ontology.yaml` with client attribute schemas
8. Add sqlc queries in `internal/db/queries/clients.sql`
### Phase 2 — MCP tools (P1, next week)
1. Register `whoami(hostname)` in `internal/mcp/server.go`
2. Register `explain(service)` — compact context card from DB
3. Register `preflight(service)` — risk classification
4. Register `get_change_history(entity, limit)`
5. Register `get_state_snapshot()`
6. Register `list_my_secrets(caller_pubkey?)`
### Phase 3 — Bootstrap script cleanup (P1, next week)
1. Replace secrets issuance URL with Oikos API endpoint
2. Remove `--no-secrets` / `--no-mesh` or rewire them to degraded modes
3. Remove `bin/homelab` symlink
4. Update Infisical identity file creation
5. Test full enrollment on a fresh machine
### Phase 4 — Transition check enforcement (P2, within 2 weeks)
1. Implement all `provisioning → active` checks in `internal/ontology/validate.go`
2. Implement all `deprecated → destroyed` checks
3. Wire checks into lifecycle transition handlers
4. Test that `POST /activate` fails when checks don't pass
5. Test that `POST /destroy` fails when inbound edges exist
### Phase 5 — Cleanup (P2, within 2 weeks)
1. Delete or comment-out dead code in bootstrap.sh
2. Recreate `tools/setup-caveman.sh` and `tools/setup-hermes-soul.sh` (or remove references)
3. Update AGENTS.md MCP tool list
4. Update CLIENTS.md enrollment flow
5. Archive Python secrets-issuance with final deprecation note
6. Run `make generate-check` and full test suite
## Verification
- Fresh machine with no prior state: `curl bootstrap.sh | sudo bash` → machine
shows up in DB as `provisioning` with age pubkey, Infisical identity, and sync
timer running
- `POST /api/v1/clients/ws:test-machine/activate` → state → `active`, all checks pass
- `POST /api/v1/clients/ws:test-machine/deprecate` → state → `deprecated`
- `POST /api/v1/clients/ws:test-machine/destroy` → fails if edges exist; succeeds
after edges removed, secrets revoked
- MCP `whoami(ws:test-machine)` returns client record with peers and health
- MCP `explain(service:caddy)` returns context card with relations and risk class
- `GET /api/v1/clients/ws:test-machine/secrets` returns secrets list scoped to client
- Existing clients continue working through the sync timer (no regression)
- `make test test-db generate-check` passes
## Related
- [2026-07-07-migrate-bin-homelab-to-go.md](2026-07-07-migrate-bin-homelab-to-go.md) — MCP tool completion plan (whoami, explain, preflight)
- [seeds/ontology.yaml](../seeds/ontology.yaml) — lifecycle definitions, entity type hierarchy
- [seeds/policy.yaml](../seeds/policy.yaml) — risk classes, approval rules
- [CLIENTS.md](../CLIENTS.md) — client onboarding guide (update after this plan)
- [bootstrap.sh](../bootstrap.sh) — current enrollment script (rewrite in Phase 3)
## Changelog
- 2026-07-07 — initial plan. Replaces Python secrets-issuance, defines full
lifecycle in Go, adds client API endpoints, MCP tools, and Infisical
machine identity integration.

2
plans/index.md
View File

@@ -10,10 +10,10 @@ went sideways, open an investigation.
| ---- | ----- | ------ | | ---- | ----- | ------ |
| 2026-07-05 | [Oikos Prometheus LXC](2026-07-05-oikos-prometheus-lxc.md) | Planned | | 2026-07-05 | [Oikos Prometheus LXC](2026-07-05-oikos-prometheus-lxc.md) | Planned |
| 2026-07-06 | [Consolidate Oikos control plane onto mac-mini](2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md) | In Progress (Phase 1-6 implemented, pending cutover) | | 2026-07-06 | [Consolidate Oikos control plane onto mac-mini](2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md) | In Progress (Phase 1-6 implemented, pending cutover) |
| 2026-07-07 | [Client lifecycle in Go — enrollment through deprecation](2026-07-07-client-lifecycle-in-go.md) | Planned |
| 2026-07-07 | [Comprehensive audit: stale files, state gaps, and next steps](2026-07-07-comprehensive-audit-and-next-steps.md) | Planned | | 2026-07-07 | [Comprehensive audit: stale files, state gaps, and next steps](2026-07-07-comprehensive-audit-and-next-steps.md) | Planned |
| 2026-07-07 | [DB as single source of truth for agent knowledge](2026-07-07-db-as-source-of-truth.md) | Proposed | | 2026-07-07 | [DB as single source of truth for agent knowledge](2026-07-07-db-as-source-of-truth.md) | Proposed |
| 2026-07-07 | [Migrate bin/homelab CLI to Go oikos homelab](2026-07-07-migrate-bin-homelab-to-go.md) | Planned | | 2026-07-07 | [Migrate bin/homelab CLI to Go oikos homelab](2026-07-07-migrate-bin-homelab-to-go.md) | Planned |
| 2026-07-07 | [Migrate bin/homelab CLI to Go oikos homelab](2026-07-07-migrate-bin-homelab-to-go.md) | Planned |
## Done ## Done