move 5 completed plans to plans/done/
- Consolidate Oikos on mac-mini (2026-07-06) - Client lifecycle in Go (2026-07-07) - Comprehensive audit & next steps (2026-07-07) - DB as source of truth (2026-07-07) - MCP tool completion (2026-07-07) Paths fixed in index.md to reflect planes/done/ locations. Active plans remaining: Prometheus LXC (Planned), implementation audit (Active).
This commit is contained in:
File diff suppressed because it is too large
Load Diff
741
plans/done/2026-07-07-client-lifecycle-in-go.md
Normal file
741
plans/done/2026-07-07-client-lifecycle-in-go.md
Normal file
@@ -0,0 +1,741 @@
|
||||
# Plan: Client lifecycle — enrollment through deprecation in Oikos Go
|
||||
|
||||
**Status:** Done (2026-07-08) — 12/12 verified. Full API + preconditions + thin-client distribution.
|
||||
|
||||
## Goal
|
||||
|
||||
Define and implement the complete lifecycle of every compute entity in the Oikos
|
||||
Go runtime — workstations, Proxmox hosts, LXCs, VMs, and containers. Two
|
||||
onboarding paths share the same lifecycle state machine:
|
||||
|
||||
1. **Workstation self-enrollment**: the machine calls the API to enroll itself
|
||||
2. **Compute entity provisioning**: Oikos creates the entity (LXC/VM/container)
|
||||
on a host via the actuator, and the DB is the sole source of truth from the
|
||||
first API call
|
||||
|
||||
Every state transition feeds the Postgres DB. No step depends on the archived
|
||||
Python `secrets-issuance` server or the non-existent `bin/homelab` CLI. The
|
||||
full repo clone + git-sync timer is replaced with a thin API-distribution model
|
||||
for workstations.
|
||||
|
||||
## Current state — what exists vs. what runs
|
||||
|
||||
| Component | Exists? | Runs? | Notes |
|
||||
|-----------|---------|-------|-------|
|
||||
| `bootstrap.sh` (684 lines) | ✅ repo | ⚠️ references dead endpoints | Calls `https://secrets.hubris.network/issue` (Python server, stopped per Phase 6). Symlinks `bin/homelab` (file absent). References `tools/*.setup.sh` (files absent). |
|
||||
| `archive/secrets-issuance/server.py` | ✅ archive | ❌ stopped (Phase 6) | Issued age keys, validated mesh IP. No Go replacement. |
|
||||
| `archive/mcp/` (build_host_files.py, deleted) | ❌ deleted | ❌ | Legacy host file builder. |
|
||||
| `inventory.yaml` (root) | ✅ | ⚠️ edited manually | Flat `hosts:` + `services:` layout. Diverges from `seeds/inventory.yaml` entity-relationship format. |
|
||||
| `seeds/inventory.yaml` | ✅ | ✅ ingested into DB | Entity-relationship format with slugs (`host:hubris`, `ws:mac-mini`). No translation path from root format. |
|
||||
| Go entity API (`POST/GET/PATCH /entities`) | ✅ | ✅ | Generic CRUD. No client-specific validation, no key issuance, no lifecycle gating. |
|
||||
| Go MCP server | ✅ | ✅ | 15 tools. Missing `whoami`, `explain`, `preflight`, `get_change_history`, `get_state_snapshot`. |
|
||||
| `oikos secret` CLI (Infisical/SOPS) | ✅ | ✅ | Secrets read/migrate/export. No client-key provisioning. |
|
||||
| Sync timer (post-pull.sh) | ✅ | ⚠️ partially broken | References `tools/*.setup.sh` (glob returns zero files). `setup-caveman.sh` and `setup-hermes-soul.sh` documented but absent. |
|
||||
|
||||
**Takeaway**: Enrollment today runs on shell scripts calling archived Python
|
||||
services. The Go runtime has zero awareness of client lifecycle. This plan
|
||||
closes that gap — the DB becomes the sole engine for client identity,
|
||||
secrets, state, and lifecycle transitions.
|
||||
|
||||
## Target architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ WORKSTATION (bare machine) │
|
||||
│ │
|
||||
│ 1. curl bootstrap.sh | sudo bash (from raw Gitea URL) │
|
||||
│ → fetches CLIENTS.md, AGENTS.md, OIKOS.md, tools/ │
|
||||
│ → calls POST /api/v1/clients/enroll │
|
||||
│ → receives age keypair + Infisical identity │
|
||||
│ → writes /etc/age/key.txt, /etc/infisical/identity │
|
||||
│ → polls GET /api/v1/clients/{slug}/context for updates │
|
||||
│ (no git clone, no sync timer) │
|
||||
└──────────────────────────┬───────────────────────────────────┘
|
||||
│ POST /api/v1/clients/enroll
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ OIKOS API (Go, :8090) │
|
||||
│ │
|
||||
│ Workstation endpoints: │
|
||||
│ POST /api/v1/clients/enroll — issue age key, set state │
|
||||
│ POST /api/v1/clients/{slug}/activate — provisioning→active │
|
||||
│ POST /api/v1/clients/{slug}/deprecate — active→deprecated │
|
||||
│ POST /api/v1/clients/{slug}/destroy — deprecated→destroyed │
|
||||
│ GET /api/v1/clients/{slug}/secrets — accessible secrets │
|
||||
│ GET /api/v1/clients/{slug}/context — agent context delta │
|
||||
│ │
|
||||
│ Compute entity endpoints: │
|
||||
│ POST /api/v1/entities/provision — create LXC/VM/container │
|
||||
│ GET /api/v1/entities/{slug}/provision/status — progress │
|
||||
│ │
|
||||
│ MCP tools: │
|
||||
│ whoami, explain, preflight, get_change_history, │
|
||||
│ get_state_snapshot, list_my_secrets │
|
||||
└──────────────────────────┬───────────────────────────────────┘
|
||||
│ writes
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ POSTGRES (TimescaleDB) │
|
||||
│ │
|
||||
│ entities table: slug, type, name, state, attributes (JSONB) │
|
||||
│ entity_status: health, disk, drift count (scheduler) │
|
||||
│ audit_log: every state transition, enrollment, revocation │
|
||||
│ executions: approved actions, results │
|
||||
│ secrets (via Infisical): age keys, API tokens │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Thin client model — why the clone + sync timer goes away
|
||||
|
||||
The repo was the source of truth when Oikos was Python + YAML files. Now the DB
|
||||
is the source of truth. A full clone of the repo onto every workstation is
|
||||
unnecessary — the only things a workstation actually needs on disk are:
|
||||
|
||||
| Artifact | Why local? | How delivered |
|
||||
|----------|-----------|---------------|
|
||||
| `CLIENTS.md` + `AGENTS.md` + `OIKOS.md` | Agent reads these at startup; can't query MCP before knowing MCP exists | Fetched once by bootstrap.sh, cached to `/opt/homelab/` |
|
||||
| `.sops.yaml` | Recipient rules for age decryption (DR fallback if Infisical is down) | Served by `GET /api/v1/clients/{slug}/context` |
|
||||
| SOPS-encrypted secrets | DR cold recovery (Infisical is primary) | Served by `GET /api/v1/secrets/sops-backups` or fetched on enrollment |
|
||||
| Age private key | Decrypt SOPS files | Delivered once in enroll response, written to `/etc/age/key.txt` |
|
||||
| Infisical identity | Authenticate to Infisical for primary secrets | Delivered once in enroll response, written to `/etc/infisical/identity` |
|
||||
| `tools/` (caveman, hermes-soul) | Auto-setup after pull | Fetched by bootstrap.sh once; API context endpoint serves updates |
|
||||
|
||||
Everything else — topology, knowledge, policy, runbooks, ledger — is queried
|
||||
live from the DB via MCP.
|
||||
|
||||
### What replaces the sync timer
|
||||
|
||||
The 5-minute `git pull` is replaced by a lightweight API poll:
|
||||
|
||||
```
|
||||
GET /api/v1/clients/{slug}/context?since=2026-07-07T12:00:00Z
|
||||
```
|
||||
|
||||
Returns a JSON delta:
|
||||
```json
|
||||
{
|
||||
"agent_files_changed": ["CLIENTS.md", "AGENTS.md", "OIKOS.md"],
|
||||
"sops_config_changed": true,
|
||||
"tools_changed": ["setup-caveman.sh"],
|
||||
"since": "2026-07-07T12:05:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
The client fetches only changed files. Poll interval: 5 minutes (same as old
|
||||
timer, but HTTP instead of git). On delta, the client writes updated files to
|
||||
`/opt/homelab/` and re-runs any changed `tools/*.setup.sh` scripts.
|
||||
|
||||
### What stays a full clone
|
||||
|
||||
The **control plane host** (mac-mini, running the Oikos Docker stack) keeps the
|
||||
full clone at `/opt/homelab-context/`. It is the deployment target, the seed
|
||||
source, and the operator's working copy. All other workstations are thin clients.
|
||||
|
||||
### bootstrap.sh changes (thin client)
|
||||
|
||||
```bash
|
||||
# Old: full clone
|
||||
git clone "$REPO_HTTPS" "$CLONE_DIR"
|
||||
|
||||
# New: fetch only what the agent needs
|
||||
mkdir -p /opt/homelab/.agents/shared /opt/homelab/.agents/skills
|
||||
curl -s "$RAW_URL/CLIENTS.md" -o /opt/homelab/CLIENTS.md
|
||||
curl -s "$RAW_URL/AGENTS.md" -o /opt/homelab/AGENTS.md
|
||||
curl -s "$RAW_URL/.agents/OIKOS.md" -o /opt/homelab/OIKOS.md
|
||||
curl -s "$RAW_URL/.agents/shared/caveman.md" -o /opt/homelab/caveman.md
|
||||
|
||||
# Then enroll via API (gets age key + Infisical identity)
|
||||
# Then start the context poller
|
||||
```
|
||||
|
||||
No `git` dependency on workstations. No `post-pull.sh`. No sync timer unit
|
||||
files. Just a cron/launchd job that hits `GET /context` every 5 minutes.
|
||||
|
||||
### Transition for existing enrolled clients
|
||||
|
||||
Existing clients with full clones continue working. The context poller is
|
||||
additive — it writes updated files to `/opt/homelab/` alongside the clone.
|
||||
Once the poller is proven, the clone can be removed and the git-based timer
|
||||
disabled. Backward compatible, no flag day.
|
||||
|
||||
## Compute entity lifecycle — LXC, VM, container provisioning
|
||||
|
||||
Workstations self-enroll. Compute entities (LXCs, VMs, Docker containers) are
|
||||
**created by Oikos** — an operator (or Hermes) issues an intent, and Oikos
|
||||
provisions the entity on a Proxmox host via the actuator. The DB is the source
|
||||
of truth from the first API call; there is no bootstrap.sh, no mesh join, no
|
||||
age key — the entity exists because Oikos put it there.
|
||||
|
||||
### Provisioning flow
|
||||
|
||||
```
|
||||
Operator (or Hermes via MCP):
|
||||
"create LXC 130 on hubris running jellyfin, 2 cores, 2GB RAM, 20GB disk,
|
||||
mount /mnt/library, IP 192.168.8.130, privileged"
|
||||
|
||||
│
|
||||
▼
|
||||
POST /api/v1/entities/provision
|
||||
{
|
||||
"slug": "lxc:jellyfin",
|
||||
"type": "lxc",
|
||||
"name": "Jellyfin",
|
||||
"host": "host:hubris",
|
||||
"attributes": {
|
||||
"vmid": 130,
|
||||
"cores": 2,
|
||||
"ram_mb": 2048,
|
||||
"disk_gb": 20,
|
||||
"ip": "192.168.8.130",
|
||||
"privileged": true,
|
||||
"mounts": [{"source": "/mnt/library", "target": "/mnt/library"}],
|
||||
"template": "debian-12-standard",
|
||||
"services": ["jellyfin"]
|
||||
}
|
||||
}
|
||||
|
||||
│ Oikos API:
|
||||
│ 1. Creates entity in DB: lxc:jellyfin, state = planned
|
||||
│ 2. Validates: IP not in use, VMID not taken, host has capacity
|
||||
│ 3. Policy classifier: config_mutation → requests operator approval
|
||||
│ 4. On approval, transitions to provisioning
|
||||
│ 5. Creates execution record
|
||||
│ 6. Actuator SSHs to host:hubris:
|
||||
│ pct create 130 /var/lib/vz/template/cache/debian-12-standard.tar.zst \
|
||||
│ --cores 2 --memory 2048 --rootfs local-lvm:20 \
|
||||
│ --net0 name=eth0,bridge=vmbr0,ip=192.168.8.130/24,gw=192.168.8.2 \
|
||||
│ --unprivileged 0
|
||||
│ pct start 130
|
||||
│ pct exec 130 -- apt update && apt install -y jellyfin-server
|
||||
│ # mount, service enable, firewall rules
|
||||
│ 7. Polls health check until passing
|
||||
│ 8. Transitions to active
|
||||
│ 9. Creates relationship edges: host:hubris hosts lxc:jellyfin,
|
||||
│ lxc:jellyfin provides service:jellyfin, etc.
|
||||
│ 10. Writes audit log
|
||||
|
||||
▼
|
||||
Entity in DB: lxc:jellyfin, state = active
|
||||
Relationships: host:hubris → hosts → lxc:jellyfin
|
||||
lxc:jellyfin → provides → service:jellyfin
|
||||
lxc:jellyfin → mounts → storage:library
|
||||
```
|
||||
|
||||
### State machine (shared by workstations and compute entities)
|
||||
|
||||
```
|
||||
┌─────────────────────────────┐
|
||||
│ WORKSTATION │
|
||||
│ Operator runs bootstrap.sh │
|
||||
│ → self-enrolls via API │
|
||||
│ → gets age key + Infisical │
|
||||
│ → polls /context for deltas │
|
||||
└──────────────┬──────────────┘
|
||||
│
|
||||
[planned] ──→ provisioning ──→ active ──→ migrating ──→ active
|
||||
▲ ▲ │ │
|
||||
│ │ ├──→ deprecated ──→ destroyed
|
||||
│ │ │
|
||||
│ │ └──→ failed
|
||||
│ │
|
||||
│ └──→ failed ┌─────────────────────────────┐
|
||||
│ │ COMPUTE ENTITY │
|
||||
│ │ POST /entities/provision │
|
||||
└──→ destroyed (cancelled) │ → Oikos actuator creates │
|
||||
│ LXC/VM/container on host │
|
||||
│ → no self-enrollment │
|
||||
│ → no mesh join │
|
||||
│ → no age key │
|
||||
└─────────────────────────────┘
|
||||
```
|
||||
|
||||
### Provisioning differences by type
|
||||
|
||||
| Aspect | Workstation | LXC | VM | Docker container |
|
||||
|--------|------------|-----|----|-----------------|
|
||||
| Who creates the entity | Operator via API/seed | Operator/Hermes via `POST /entities/provision` | Same as LXC | Same as LXC |
|
||||
| How it reaches `provisioning` | Self-enrolls via `POST /clients/enroll` | API transitions automatically after operator approval | Same as LXC | Same as LXC |
|
||||
| Age keypair | Generated, delivered to client | None — no SOPS access needed | None | None |
|
||||
| Infisical identity | Created, scoped to `/clients/<slug>/*` | Only if the LXC hosts services that need secrets | Same as LXC | Same as LXC |
|
||||
| Mesh join | Netbird/Tailscale (if remote access needed) | No — reachable via host LAN IP | Optional | No |
|
||||
| Health check | Scheduler probes HTTP/TCP/SSH | Scheduler probes service port on host LAN IP | Same as LXC | Scheduler probes container port |
|
||||
| `provisioning → active` checks | `age-key-enrolled`, `mesh-joined`, `health-check-answering` | `health-check-answering`, `service-running`, `mounts-verified` | Same as LXC + `vm-agent-responding` | `health-check-answering`, `container-running` |
|
||||
| `deprecated → destroyed` checks | `secrets-revoked`, `no-inbound-edges`, `ingress-dns-removed` | `backups-verified`, `no-inbound-edges`, `ingress-dns-removed` | Same as LXC | `no-inbound-edges` |
|
||||
| Secrets to revoke on destroy | Age key, Infisical identity | Infisical identity (if any), service tokens | Same as LXC | Infisical identity (if any) |
|
||||
|
||||
### Compute entity attributes schema
|
||||
|
||||
```json
|
||||
{
|
||||
"vmid": 130,
|
||||
"cores": 2,
|
||||
"ram_mb": 2048,
|
||||
"disk_gb": 20,
|
||||
"ip": "192.168.8.130",
|
||||
"privileged": true,
|
||||
"os": "debian-12",
|
||||
"template": "debian-12-standard",
|
||||
"mounts": [
|
||||
{"source": "/mnt/library", "target": "/mnt/library", "options": "ro"}
|
||||
],
|
||||
"services": ["jellyfin"],
|
||||
"provisioned_by": "ws:mac-mini",
|
||||
"provisioned_at": "2026-07-07T14:00:00Z",
|
||||
"host": "host:hubris"
|
||||
}
|
||||
```
|
||||
|
||||
### API endpoints for compute entity provisioning
|
||||
|
||||
| Method | Path | Scope | Purpose |
|
||||
|--------|------|-------|---------|
|
||||
| `POST` | `/api/v1/entities/provision` | operator | Declare intent to create an LXC/VM/container. Creates entity in `planned`, validates constraints, queues execution for operator approval. |
|
||||
| `GET` | `/api/v1/entities/{slug}/provision/status` | operator | Poll provisioning progress: steps completed, current step, errors |
|
||||
| `POST` | `/api/v1/entities/{slug}/provision/retry` | operator | Retry a failed provisioning step |
|
||||
|
||||
### What `POST /entities/provision` validates
|
||||
|
||||
1. **Slug is available**: no existing entity with same slug
|
||||
2. **VMID not in use**: query entities where `attributes->>'vmid'` = requested VMID
|
||||
3. **IP not in use**: query entities where `attributes->>'ip'` = requested IP
|
||||
4. **Host exists and is active**: `host:hubris` must be an entity with state `active`
|
||||
5. **Host has capacity**: check entity_status for CPU/RAM/disk headroom on host
|
||||
6. **Template exists**: Proxmox template must be available on the host
|
||||
7. **Mount source exists**: storage entity must exist for each mount
|
||||
8. **Policy classification**: `config_mutation` → operator approval required (auto-act only if Hermes has autonomy for this specific action type)
|
||||
|
||||
### Relationship edges created on provision
|
||||
|
||||
When an LXC is provisioned, these edges are written to the `relationships` table:
|
||||
|
||||
| Source | Edge type | Target | Why |
|
||||
|--------|-----------|--------|-----|
|
||||
| `host:hubris` | `hosts` | `lxc:jellyfin` | Physical colocation |
|
||||
| `lxc:jellyfin` | `provides` | `service:jellyfin` | Service mapping |
|
||||
| `lxc:jellyfin` | `mounts` | `storage:library` | Storage dependency |
|
||||
| `lxc:jellyfin` | `depends-on` | `host:hubris` | Blast radius: hubris down → jellyfin down |
|
||||
| `service:jellyfin` | `depends-on` | `lxc:jellyfin` | Service availability ties to container |
|
||||
| `lxc:jellyfin` | `depends-on` | `service:dns` | DNS resolution |
|
||||
| `lxc:jellyfin` | `depends-on` | `service:caddy` | If ingress-exposed |
|
||||
|
||||
All edges are created atomically during provisioning. The `blast_radius()` CTE
|
||||
walks these edges to answer "what breaks if hubris goes down?"
|
||||
|
||||
### State: `planned`
|
||||
|
||||
The operator declares intent. A client entity exists in the DB with state
|
||||
`planned` but has no host, no keys, no sync.
|
||||
|
||||
**Entry condition**: Operator creates the entity via API or seed file.
|
||||
|
||||
**Required attributes**:
|
||||
- `slug` — `ws:<hostname>` for workstations, `host:<hostname>` for servers
|
||||
- `type` — `workstation`, `standalone-server`, or `proxmox-host`
|
||||
- `name` — human-readable name
|
||||
- `lan_ip` — expected LAN IP (reserved in DHCP)
|
||||
- `os` — `linux` or `macos`
|
||||
- `role` — free-text description of what this machine does
|
||||
- `mesh.expected_type` — `netbird` or `tailscale` (which mesh it will join)
|
||||
- `ssh.user` — login user (default `root`)
|
||||
|
||||
**Allowed transitions**: `→ provisioning` (operator triggers), `→ destroyed` (cancelled).
|
||||
|
||||
### State: `provisioning`
|
||||
|
||||
The machine has been declared. Operator runs `bootstrap.sh` on the target,
|
||||
which calls the enrollment API. The API validates identity (mesh IP matches
|
||||
expected subnet, hostname matches slug), issues an age keypair, and records
|
||||
the public key. The sync timer starts pulling the repo.
|
||||
|
||||
**Go API**: `POST /api/v1/clients/enroll`
|
||||
```json
|
||||
{
|
||||
"slug": "ws:new-laptop",
|
||||
"hostname": "new-laptop",
|
||||
"mesh_ip": "100.122.x.x"
|
||||
}
|
||||
```
|
||||
|
||||
**What the enrollment endpoint does**:
|
||||
1. Looks up entity by slug — must exist, must be in state `planned` or `provisioning`
|
||||
2. Validates mesh IP is in `100.122.0.0/16` (Netbird) or `100.64.0.0/10` (Tailscale) or `192.168.8.0/24` (LAN)
|
||||
3. Validates hostname has no conflicting mesh IP already recorded
|
||||
4. Generates an age keypair (`age-keygen`)
|
||||
5. Stores the private key in Infisical under path `/clients/<slug>/age-key`
|
||||
6. Creates an Infisical machine identity for the client (UniversalAuth)
|
||||
7. Updates entity `attributes` with `age_pubkey`, `mesh_ip`, `enrolled_at`
|
||||
8. Writes audit log: `client.enrolled`
|
||||
9. Returns the age private key, Infisical client ID + secret, and machine identity token
|
||||
|
||||
**Response** (to bootstrap.sh, over mesh — TLS + mesh IP validation):
|
||||
```json
|
||||
{
|
||||
"age_private_key": "AGE-SECRET-KEY-...",
|
||||
"age_public_key": "age1...",
|
||||
"infisical_client_id": "...",
|
||||
"infisical_client_secret": "...",
|
||||
"machine_identity_token": "..."
|
||||
}
|
||||
```
|
||||
|
||||
**Bootstrap script changes**:
|
||||
- Remove call to `https://secrets.hubris.network/issue`
|
||||
- Replace with `POST /api/v1/clients/enroll` to `https://oikos.hubris.network`
|
||||
- Remove `--no-secrets` / `--no-mesh` flags (or keep as escape hatches with degraded state)
|
||||
- Remove symlink to `bin/homelab` (file doesn't exist)
|
||||
- After receiving keys, bootstrap.sh writes `/etc/age/key.txt` (0600) and `/etc/infisical/identity` (0600)
|
||||
|
||||
**Pre-built bootstrap**: The bootstrap.sh is served from the Gitea repo raw URL
|
||||
(already the case). After this plan, it calls Oikos API instead of the dead
|
||||
Python service.
|
||||
|
||||
**Allowed transition**: `→ active` (when `age-key-enrolled`, `mesh-joined`,
|
||||
`doc-page-complete` checks pass).
|
||||
|
||||
### State: `active`
|
||||
|
||||
Normal operation. The client pulls the repo every 5 minutes, uses its age key
|
||||
to decrypt SOPS secrets (fallback), and authenticates to Infisical via its
|
||||
machine identity (primary). The MCP `whoami(hostname)` tool returns its
|
||||
entity record, peer list, accessible secrets, and current health.
|
||||
|
||||
**Go enforcement of transition checks** (`provisioning → active`):
|
||||
- `age-key-enrolled-if-needed`: entity.attributes.age_pubkey is non-empty
|
||||
- `mesh-joined-if-needed`: entity.attributes.mesh_ip is non-empty
|
||||
- `ingress-live-if-public`: skipped for workstations (no public ingress)
|
||||
- `health-check-answering`: scheduler probe passes for this entity
|
||||
- `doc-page-complete`: entity has at least one `documents` edge
|
||||
- `inventory-in-db`: entity exists in DB with all required attributes
|
||||
|
||||
**API**: `POST /api/v1/clients/{slug}/activate`
|
||||
- Validates all `provisioning → active` transition checks
|
||||
- Sets state to `active`
|
||||
- Writes audit log: `client.activated`
|
||||
|
||||
**MCP tools active clients get**:
|
||||
- `whoami(hostname)` — returns entity record, peers, secrets list, health
|
||||
- `list_my_secrets(caller_pubkey?)` — secrets this client can decrypt
|
||||
|
||||
**Allowed transitions**: `→ migrating`, `→ deprecated`, `→ failed`.
|
||||
|
||||
### State: `migrating`
|
||||
|
||||
Client is being moved — OS reinstall, hardware swap, role change. Inbound edges
|
||||
still exist; no deprovisioning has started.
|
||||
|
||||
**Allowed transition**: `→ active` (migration complete, post-verify passes).
|
||||
|
||||
**API**: `POST /api/v1/clients/{slug}/migrate` (sets state, links migration plan).
|
||||
|
||||
### State: `deprecated`
|
||||
|
||||
Client is being phased out. Services moved off, mesh disconnected, secrets
|
||||
rotation started. The deprecation gate (`no-inbound-edges`) blocks `→ destroyed`
|
||||
until all `depends-on`, `hosts`, `provides`, and `mounts` edges are gone.
|
||||
|
||||
**API**: `POST /api/v1/clients/{slug}/deprecate`
|
||||
- Validates `replacement-live-or-role-retired`: operator confirms replacement exists or role is no longer needed
|
||||
- Sets state to `deprecated`
|
||||
- Writes audit log: `client.deprecated`
|
||||
|
||||
**Allowed transitions**: `→ active` (un-deprecate), `→ destroyed`.
|
||||
|
||||
### State: `destroyed`
|
||||
|
||||
Client is gone. All edges removed, secrets revoked, archaeology entry written.
|
||||
|
||||
**API**: `POST /api/v1/clients/{slug}/destroy`
|
||||
- Validates all `deprecated → destroyed` transition checks:
|
||||
- `backups-verified`: any data on this client was backed up
|
||||
- `secrets-revoked-and-rekeyed`: age key removed from Infisical, SOPS recipients updated, machine identity deleted
|
||||
- `ingress-and-dns-removed`: no remaining DNS records or Caddy backends
|
||||
- `no-inbound-edges`: zero `depends-on`, `hosts`, `provides`, `mounts` edges pointing to this entity
|
||||
- `archaeology-entry`: writes a record explaining why and when
|
||||
- Sets state to `destroyed`
|
||||
- Revokes Infisical machine identity
|
||||
- Removes age public key from `.sops.yaml`
|
||||
- Writes audit log: `client.destroyed`
|
||||
|
||||
### State: `failed`
|
||||
|
||||
Something went wrong during provisioning or operation. Requires operator
|
||||
intervention. Treated as informational — no automatic recovery.
|
||||
|
||||
**API**: `POST /api/v1/clients/{slug}/fail`
|
||||
- Sets state to `failed`
|
||||
- Requires `reason` field explaining what broke
|
||||
- Writes audit log: `client.failed`
|
||||
|
||||
## Secrets integration
|
||||
|
||||
### Age key lifecycle
|
||||
|
||||
```
|
||||
planned ────────────→ no key exists
|
||||
provisioning ───────→ keypair generated, pubkey stored in entity attributes,
|
||||
private key delivered to client via enroll response,
|
||||
private key stored in Infisical under /clients/<slug>/age-key
|
||||
active ─────────────→ key used for SOPS decryption fallback, authenticated
|
||||
to Infisical via machine identity for primary secrets
|
||||
deprecated ─────────→ key still valid, but rotation initiated
|
||||
destroyed ──────────→ key revoked from Infisical, removed from .sops.yaml,
|
||||
machine identity deleted
|
||||
```
|
||||
|
||||
### Infisical machine identity
|
||||
|
||||
Each client gets an Infisical machine identity during enrollment. This is the
|
||||
primary secrets path — the age key is fallback for SOPS-encrypted DR files.
|
||||
|
||||
- **Client ID + Secret** returned in enroll response
|
||||
- **Scoped to paths**: `/clients/<slug>/*`, `/shared/*`
|
||||
- **Revoked on destroy**: identity deleted, access gone
|
||||
|
||||
### SOPS fallback
|
||||
|
||||
The age public key is added to `.sops.yaml` recipients during enrollment.
|
||||
On destroy, it is removed via `oikos secret export-sops` regeneration.
|
||||
|
||||
### bootstrap.sh changes
|
||||
|
||||
```diff
|
||||
- # calls https://secrets.hubris.network/issue (Python, dead)
|
||||
- AGE_KEY=$(curl -s -X POST "$ISSUANCE_URL" ...)
|
||||
-
|
||||
+ # calls Oikos API enrollment endpoint
|
||||
+ ENROLL_RESP=$(curl -s -X POST "$OIKOS_URL/api/v1/clients/enroll" \
|
||||
+ -H "Content-Type: application/json" \
|
||||
+ -d "{\"slug\":\"ws:$HNAME\",\"hostname\":\"$HNAME\",\"mesh_ip\":\"$MESH_IP\"}")
|
||||
+ AGE_PRIVKEY=$(echo "$ENROLL_RESP" | jq -r '.age_private_key')
|
||||
```
|
||||
|
||||
## DB integration
|
||||
|
||||
### New migration
|
||||
|
||||
`012_client_enrollment.up.sql`:
|
||||
```sql
|
||||
-- No new tables needed — entities table already holds clients.
|
||||
-- Add enrollment-specific attributes validation via check constraints
|
||||
-- or application-level validation.
|
||||
|
||||
-- Enforce slug format for machine entities
|
||||
-- ws:<hostname> for workstations, host:<hostname> for servers
|
||||
-- (application-level validation in Go, not a DB constraint)
|
||||
|
||||
-- Add index for slug-based client lookups
|
||||
CREATE INDEX IF NOT EXISTS idx_entities_slug_type
|
||||
ON entities (slug, type)
|
||||
WHERE type IN ('workstation', 'standalone-server', 'proxmox-host');
|
||||
```
|
||||
|
||||
### Entity attributes schema (for `machine` types)
|
||||
|
||||
```json
|
||||
{
|
||||
"cpu_arch": "arm64",
|
||||
"ram_gb": 16,
|
||||
"os": "macos",
|
||||
"lan_ip": "192.168.8.175",
|
||||
"mesh": {
|
||||
"netbird": {"ip": "100.122.x.x", "fqdn": "hostname.netbird.selfhosted"}
|
||||
},
|
||||
"ssh": {"user": "dtoro"},
|
||||
"age_pubkey": "age1...",
|
||||
"enrolled_at": "2026-07-07T12:00:00Z",
|
||||
"enrolled_by": "ws:mac-mini",
|
||||
"infisical_identity_id": "identity_abc123"
|
||||
}
|
||||
```
|
||||
|
||||
All attributes are stored in the `attributes` JSONB column on the `entities`
|
||||
table. Validation happens at the application layer (Go) using the schema
|
||||
defined in `seeds/ontology.yaml`.
|
||||
|
||||
## MCP tools to add
|
||||
|
||||
These are documented in AGENTS.md section 3 but not implemented in the Go MCP
|
||||
server. Implementation: register in `internal/mcp/server.go`.
|
||||
|
||||
| Tool | Input | Output | Implementation |
|
||||
|------|-------|--------|----------------|
|
||||
| `whoami` | `hostname` | Entity record, peers, accessible secrets, health | DB lookup by slug derived from hostname |
|
||||
| `list_my_secrets` | `caller_pubkey?` | Secrets this client can decrypt | Infisical list + SOPS `.sops.yaml` match |
|
||||
| `explain` | `service_slug` | Compact context card: type, state, health, relations, last change | DB join: entity + entity_status + audit_log |
|
||||
| `preflight` | `service_slug` | Risk class, approval requirement, verification command | Policy classifier on the entity's type |
|
||||
| `get_change_history` | `entity_slug`, `limit` | Last N audit_log entries for entity | DB query on audit_log table |
|
||||
| `get_state_snapshot` | none | Last scheduler Observe pass: health, disk, drift count | DB query on entity_status + signals |
|
||||
|
||||
## API endpoints to add
|
||||
|
||||
Add to `api/openapi.yaml`, regenerate with `make generate`, implement in
|
||||
`internal/httpapi/impl.go`.
|
||||
|
||||
| Method | Path | Scope | Purpose |
|
||||
|--------|------|-------|---------|
|
||||
| `POST` | `/api/v1/clients/enroll` | agent | Issue age key, validate mesh, set state → provisioning |
|
||||
| `POST` | `/api/v1/clients/{slug}/activate` | operator | Run transition checks, state → active |
|
||||
| `POST` | `/api/v1/clients/{slug}/deprecate` | operator | State → deprecated |
|
||||
| `POST` | `/api/v1/clients/{slug}/destroy` | operator | Run destroy checks, revoke secrets, state → destroyed |
|
||||
| `POST` | `/api/v1/clients/{slug}/fail` | operator | State → failed with reason |
|
||||
| `GET` | `/api/v1/clients/{slug}/secrets` | agent | List secrets this client can access |
|
||||
|
||||
## Files changed
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `bootstrap.sh` | Replace git clone with thin fetch of AGENTS.md + OIKOS.md + tools/. Replace `secrets.hubris.network/issue` with `POST /api/v1/clients/enroll`. Replace sync timer with context poller. Remove dead symlinks. |
|
||||
| `api/openapi.yaml` | Add workstation enrollment, lifecycle, secret, and context endpoints. Add compute entity provision, status, retry endpoints. |
|
||||
| `internal/httpapi/impl.go` | Implement workstation lifecycle handlers. Implement compute entity provisioning handlers. |
|
||||
| `internal/db/queries/clients.sql` | Add client-specific sqlc queries (lookup by slug+type, capacity checks) |
|
||||
| `internal/db/queries/entities.sql` | Add provision validation queries (VMID collision, IP collision, template availability) |
|
||||
| `internal/mcp/server.go` | Register whoami, explain, preflight, get_change_history, get_state_snapshot, list_my_secrets |
|
||||
| `internal/secrets/infisical.go` | Add `CreateMachineIdentity`, `DeleteMachineIdentity`, `StoreClientKey` |
|
||||
| `internal/actuator/actuator.go` | Add `ProvisionLXC`, `ProvisionVM` methods (pct create, start, exec, health poll) |
|
||||
| `internal/ontology/validate.go` | Implement lifecycle transition checks for infrastructure lifecycle (both workstation and compute variants) |
|
||||
| `seeds/ontology.yaml` | Add workstation-specific and compute-entity attribute schemas. Add provision-entity skill definition. |
|
||||
| `migrations/012_client_enrollment.up.sql` | Index for slug+type lookups. Add `provisioning_steps` tracking table. |
|
||||
| `tools/context-poller.sh` | New: lightweight daemon that polls `GET /context` and applies deltas |
|
||||
| `AGENTS.md` | Update MCP tool list. Document thin client model. |
|
||||
| `CLIENTS.md` | Update enrollment flow: thin client, API-based, no git clone. |
|
||||
| `CONTRIBUTING.md` | Add compute entity provisioning as an extension point. |
|
||||
|
||||
## Files deleted or deprecated
|
||||
|
||||
| File | Disposition |
|
||||
|------|-------------|
|
||||
| `archive/secrets-issuance/` | Already archived. Add deprecation notice referencing this plan. |
|
||||
| `archive/secrets-sops-backup/` | Keep for DR. SOPS backups served via `GET /api/v1/secrets/sops-backups`. |
|
||||
| `scripts/sync/` (install.sh, linux/*, macos/*) | Deprecated. Replaced by `tools/context-poller.sh`. |
|
||||
| `tools/post-pull.sh` | Keep on control plane host only. Remove from thin client distribution. |
|
||||
| Any reference to `bin/homelab` | Delete or comment out in bootstrap.sh; CLI doesn't exist. |
|
||||
| `tools/*.setup.sh` references | Either create the files or remove the auto-setup convention. Context poller can trigger setup scripts on delta. |
|
||||
| Root `inventory.yaml` | Eventually deprecated. `seeds/inventory.yaml` + DB are authoritative. Keep during transition. |
|
||||
|
||||
## Phased implementation
|
||||
|
||||
### Phase 1 — API + DB (P0, this week)
|
||||
|
||||
1. Write `migrations/012_client_enrollment.up.sql` (add `provisioning_steps` table)
|
||||
2. Add workstation + compute entity endpoints to `api/openapi.yaml`
|
||||
3. Run `make generate`
|
||||
4. Implement enrollment handler (`POST /api/v1/clients/enroll`):
|
||||
- Age key generation
|
||||
- Infisical machine identity creation
|
||||
- Entity attribute update
|
||||
- Audit log write
|
||||
5. Implement workstation lifecycle transition handlers (activate, deprecate, destroy, fail)
|
||||
6. Implement `GET /api/v1/clients/{slug}/secrets`
|
||||
7. Implement `GET /api/v1/clients/{slug}/context` (agent file deltas)
|
||||
8. Update `seeds/ontology.yaml` with workstation and compute attribute schemas
|
||||
9. Add sqlc queries in `internal/db/queries/clients.sql`
|
||||
|
||||
### Phase 2 — Compute entity provisioning (P0, this week)
|
||||
|
||||
1. Implement `POST /api/v1/entities/provision`:
|
||||
- Entity creation in `planned` state
|
||||
- Constraint validation (VMID, IP, capacity, template, mounts)
|
||||
- Policy classification + operator approval flow
|
||||
- Transition to `provisioning` on approval
|
||||
2. Implement actuator provisioning methods:
|
||||
- `ProvisionLXC(slug, attributes)` → `pct create`, `pct start`, package install, mount, service enable
|
||||
- `ProvisionVM(slug, attributes)` → `qm create`, `qm start`
|
||||
3. Implement `GET /api/v1/entities/{slug}/provision/status`
|
||||
4. Implement `POST /api/v1/entities/{slug}/provision/retry`
|
||||
5. Write relationship edges atomically on provision completion
|
||||
6. Add validation queries for collision detection
|
||||
|
||||
### Phase 3 — MCP tools (P1, next week)
|
||||
|
||||
1. Register `whoami(hostname)` in `internal/mcp/server.go`
|
||||
2. Register `explain(service)` — compact context card from DB
|
||||
3. Register `preflight(service)` — risk classification
|
||||
4. Register `get_change_history(entity, limit)`
|
||||
5. Register `get_state_snapshot()`
|
||||
6. Register `list_my_secrets(caller_pubkey?)`
|
||||
|
||||
### Phase 4 — Thin client distribution (P1, next week)
|
||||
|
||||
1. Write `tools/context-poller.sh` — polls `GET /context`, writes deltas, triggers setup scripts
|
||||
2. Rewrite `bootstrap.sh`:
|
||||
- Fetch AGENTS.md, OIKOS.md, tools/ from raw Gitea URL (not full clone)
|
||||
- Call `POST /api/v1/clients/enroll`
|
||||
- Install context poller (cron/launchd)
|
||||
- Remove git dependency for workstations
|
||||
3. Create missing `tools/setup-caveman.sh` and `tools/setup-hermes-soul.sh` (or serve from context endpoint)
|
||||
4. Test full enrollment on a fresh machine
|
||||
|
||||
### Phase 5 — Transition check enforcement (P2, within 2 weeks)
|
||||
|
||||
1. Implement all `provisioning → active` checks (workstation and compute variants)
|
||||
2. Implement all `deprecated → destroyed` checks (workstation and compute variants)
|
||||
3. Wire checks into lifecycle transition handlers
|
||||
4. Test that `POST /activate` fails when checks don't pass
|
||||
5. Test that `POST /destroy` fails when inbound edges exist
|
||||
6. Test that provision fails when VMID/IP collision detected
|
||||
|
||||
### Phase 6 — Cleanup + existing client migration (P2, within 2 weeks)
|
||||
|
||||
1. Delete or comment-out dead code in bootstrap.sh
|
||||
2. Deprecate `scripts/sync/` (install.sh, systemd timer, launchd plist)
|
||||
3. Keep `tools/post-pull.sh` for control plane host only
|
||||
4. Update AGENTS.md MCP tool list + thin client model
|
||||
5. Update CLIENTS.md enrollment flow
|
||||
6. Archive Python secrets-issuance with final deprecation note
|
||||
7. Add backward-compat: existing clients with full clones continue working;
|
||||
context poller runs alongside until operator removes the clone
|
||||
8. Run `make generate-check` and full test suite
|
||||
|
||||
## Verification
|
||||
|
||||
### Workstation enrollment
|
||||
- Fresh machine with no prior state: `curl bootstrap.sh | sudo bash` → fetches
|
||||
agent files only (no git clone), enrolls via API, machine shows up in DB as
|
||||
`provisioning` with age pubkey and Infisical identity
|
||||
- Context poller running: `GET /context?since=...` returns file deltas
|
||||
- `POST /api/v1/clients/ws:test-machine/activate` → state → `active`, all checks pass
|
||||
- `POST /api/v1/clients/ws:test-machine/deprecate` → state → `deprecated`
|
||||
- `POST /api/v1/clients/ws:test-machine/destroy` → fails if edges exist; succeeds
|
||||
after edges removed, secrets revoked
|
||||
- MCP `whoami(ws:test-machine)` returns client record with peers and health
|
||||
|
||||
### Compute entity provisioning
|
||||
- `POST /api/v1/entities/provision` with valid LXC spec → entity created in
|
||||
`planned`, approval requested, on approval → `provisioning`, actuator creates
|
||||
LXC on host, health check passes → `active`
|
||||
- `POST /api/v1/entities/provision` with conflicting VMID → 409, error message
|
||||
with conflicting entity slug
|
||||
- `POST /api/v1/entities/provision` with IP already in use → 409
|
||||
- `POST /api/v1/entities/provision` with unknown host → 400
|
||||
- Relationship edges created: `hosts`, `provides`, `mounts`, `depends-on`
|
||||
all present after provisioning
|
||||
- `GET /entities/lxc:test-lxc/blast-radius` → returns host, dependent services
|
||||
- `POST /destroy` on a provisioned LXC → validates backups + edges, deletes
|
||||
relationships, transitions to `destroyed`
|
||||
|
||||
### Integrated
|
||||
- MCP `explain(service:caddy)` returns context card with relations and risk class
|
||||
- `GET /api/v1/clients/ws:test-machine/secrets` returns secrets list scoped to client
|
||||
- Thin client survives Infisical outage: SOPS fallback works with local age key + `.sops.yaml`
|
||||
served via context endpoint
|
||||
- Existing clients with full clones continue working (no regression)
|
||||
- `make test test-db generate-check` passes
|
||||
|
||||
## Related
|
||||
|
||||
- [2026-07-07-migrate-bin-homelab-to-go.md](2026-07-07-migrate-bin-homelab-to-go.md) — MCP tool completion plan (whoami, explain, preflight)
|
||||
- [seeds/ontology.yaml](../seeds/ontology.yaml) — lifecycle definitions, entity type hierarchy
|
||||
- [seeds/policy.yaml](../seeds/policy.yaml) — risk classes, approval rules
|
||||
- [CLIENTS.md](../CLIENTS.md) — client onboarding guide (update after this plan)
|
||||
- [bootstrap.sh](../bootstrap.sh) — current enrollment script (rewrite in Phase 4)
|
||||
|
||||
## Changelog
|
||||
|
||||
- 2026-07-07 rev 2 — added thin-client API distribution model (no git clone,
|
||||
no sync timer; context poller replaces 5-minute pull). Added compute entity
|
||||
provisioning flow (LXC/VM/container created by Oikos actuator, not
|
||||
self-enrolled). Extended state machine to cover both onboarding paths with
|
||||
type-specific transition checks, attribute schemas, and relationship edges.
|
||||
- 2026-07-07 rev 1 — initial plan. Replaces Python secrets-issuance, defines
|
||||
full workstation lifecycle in Go, adds client API endpoints, MCP tools,
|
||||
and Infisical machine identity integration.
|
||||
364
plans/done/2026-07-07-comprehensive-audit-and-next-steps.md
Normal file
364
plans/done/2026-07-07-comprehensive-audit-and-next-steps.md
Normal file
@@ -0,0 +1,364 @@
|
||||
# 2026-07-07 — Comprehensive audit: stale files, state gaps, and next steps
|
||||
|
||||
**Status:** Done (2026-07-08) — all actionable cleanup items resolved. Remaining items are operational (cutover) or cross-plan (covered by consolidation + Prometheus plans).
|
||||
|
||||
## Executive summary
|
||||
|
||||
Oikos is in a stable intermediate state: Go rewrite (Phases 1–6) is implemented and
|
||||
running in Docker on mac-mini, the narrative wiki has been restructured into
|
||||
`knowledge/wiki/`, and the strong migration (Tier 1 + 2) is complete. But the
|
||||
transition left behind stale artifacts, half-updated plans, and several documentation
|
||||
gaps that create confusion for agents and operators.
|
||||
|
||||
This plan catalogs every stale file and gap found, assigns ownership, and proposes a
|
||||
prioritized work queue for the next 2–4 weeks. **No action is taken by this plan
|
||||
itself** — it is the map from which concrete work items are drawn.
|
||||
|
||||
---
|
||||
|
||||
## 1. Stale files — catalog
|
||||
|
||||
### 1.1 Python kernel (`oikos/*.py`) — 11 files
|
||||
|
||||
| File | Risk | Verdict |
|
||||
|------|------|---------|
|
||||
| `oikos/__init__.py` | Low | Package init. Harmless but unused. |
|
||||
| `oikos/approve.py` | Low | Superseded by `internal/domain/approval.go`. Remove. |
|
||||
| `oikos/decide.py` | Low | Superseded by `internal/policy/classify.go`. Remove. |
|
||||
| `oikos/drift.py` | **Medium** | Superseded by `internal/scheduler/scheduler.go`. References old paths (`containers/121-caddy.md` at L218). Remove. |
|
||||
| `oikos/ledger.py` | **Medium** | Superseded by DB view + `internal/notifier/notifier.go`. Remove. |
|
||||
| `oikos/policy.py` | Low | Superseded by `internal/policy/classify.go`. Remove. |
|
||||
| `oikos/relations.py` | Low | Superseded by `blast_radius()` SQL function + `internal/ontology/`. Remove. |
|
||||
| `oikos/scheduler.py` | **Medium** | Superseded by `internal/scheduler/`. Referenced by Prometheus plan. Remove. |
|
||||
| `oikos/signal.py` | Low | Superseded by signals table (migration 003) + `internal/domain/signal.go`. Remove. |
|
||||
| `oikos/gen-topology.py` | **High** | Still active — writes `knowledge/wiki/infrastructure/topology.md`, consumed by `homelab` CLI. **Keep until ported to Go.** |
|
||||
| `oikos/gen_topology_lib.py` | **High** | Imported by `gen-topology.py`. **Keep until ported.** |
|
||||
|
||||
**Action:** Remove the 9 superseded `.py` files. Keep `gen-topology.py` +
|
||||
`gen_topology_lib.py` until a Go equivalent exists. Update `.gitignore` and any
|
||||
cross-references (Prometheus plan, bin/homelab docs).
|
||||
|
||||
### 1.2 `.hermes/plans/` — 7 files
|
||||
|
||||
These plans live in `.hermes/plans/` but the wiki-hq convention (`plans/index.md` L125,
|
||||
`.agents/domains/operations/schema.md`) mandates all plans go in `plans/`. The strong
|
||||
migration assessment (`.hermes/plans/2026-07-05_strong-migration-assessment.md`) is
|
||||
particularly valuable and has already been executed.
|
||||
|
||||
| File | Status | Action |
|
||||
|------|--------|--------|
|
||||
| `2026-06-03_110000-library-ssd-migration-to-ludo-mini.md` | Superseded by strong migration | Move to `plans/done/` |
|
||||
| `2026-06-03_150000-homelab-structure-revision.md` | Executed (is this repo structure) | Move to `plans/done/` |
|
||||
| `2026-06-03_223218-dhcp-pool-exclude-static-ips.md` | Executed | Move to `plans/done/` |
|
||||
| `2026-06-05_170000-prevent-dhcp-ip-drift.md` | Executed | Move to `plans/done/` |
|
||||
| `2026-06-06_232200-authentik-frequent-login-fix.md` | Executed | Move to `plans/done/` |
|
||||
| `2026-06-06_234500-caddyfile-truncation-permanent-fix.md` | Executed | Move to `plans/done/` |
|
||||
| `2026-07-05_strong-migration-assessment.md` | Executed (Phases 1–2d done) | Move to `plans/done/`, add Phase 3 status note |
|
||||
|
||||
### 1.3 `oikos/cards/` — 45 context cards
|
||||
|
||||
These `.md` files are consumed by the Python MCP server's `explain` tool
|
||||
(`mcp/server.py:38` referenced `CARD_DIR`). The Go MCP server (`internal/mcp/`) now
|
||||
serves entity context from the database via `get_entity` / `list_entities`.
|
||||
|
||||
| Risk | Verdict |
|
||||
|------|---------|
|
||||
| **Medium** | Cards are redundant with DB entities but may still be read by the legacy `bin/homelab` Python CLI for `homelab explain`. If `bin/homelab` is ported to Go (via generated OpenAPI client), these become dead weight. |
|
||||
|
||||
**Action:** Verify whether `bin/homelab explain` reads `oikos/cards/` or queries the
|
||||
API. If it reads cards: mark them as "keep until homelab CLI ported." If it queries the
|
||||
API: remove cards and mention in the next Go release.
|
||||
|
||||
### 1.4 `secrets-issuance/` — identity issuance service
|
||||
|
||||
Still running on apps/105 (LXC) as `secrets-issuance.service`. The Docker stack
|
||||
hasn't taken over this function yet. The cutover checklist (`scripts/cutover-checklist.md:5`)
|
||||
notes "Disable apps/105 services" as ticked, but secrets-issuance was specifically stopped,
|
||||
not replaced.
|
||||
|
||||
| Risk | Verdict |
|
||||
|------|---------|
|
||||
| **Medium** | If apps/105 is destroyed (cutover cleanup), secrets issuance must be replaced or explicitly decommissioned. |
|
||||
|
||||
**Action:** Decide: (a) port secrets issuance to Docker stack, or (b) decommission it
|
||||
(age keys can be generated ad-hoc on each client). Document the decision.
|
||||
|
||||
### 1.5 Traefik references
|
||||
|
||||
`knowledge/sources/references/cert-sync-and-traefik-config.md` and
|
||||
`inventory.yaml:245` (`vps/management.json.tmpl` references traefik) refer to a reverse
|
||||
proxy that was replaced by Caddy for internal routing. Traefik **still runs on the VPS**
|
||||
for public termination, so the references may be valid — but the doc is unclear.
|
||||
|
||||
**Action:** Audit traefik references. If VPS traefik is still active, document its
|
||||
scope (public termination only, not internal). If fully replaced, remove the references.
|
||||
|
||||
---
|
||||
|
||||
## 2. Plan status drift
|
||||
|
||||
### 2.1 TRMNL (128) — marked "Planned" but done
|
||||
|
||||
`plans/2026-06-24-trmnl-plugins-lxc.md` shows status "Planned" but:
|
||||
- LXC 128 exists in `inventory.yaml` with IP `192.168.8.211`
|
||||
- Wiki page `knowledge/wiki/containers/128-trmnl.md` exists
|
||||
- `hosts/trmnl.yaml` exists
|
||||
- Caddy route `trmnl.hubris.network` exists
|
||||
- Service `services.trmnl` exists in `inventory.yaml`
|
||||
|
||||
**Action:** Mark TRMNL plan as "Done", move to `plans/done/`, add changelog entries
|
||||
on affected pages (as specified in the plan's "Post-migration" section).
|
||||
|
||||
### 2.2 Consolidation plan — cutover incomplete
|
||||
|
||||
`plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md` status says
|
||||
"Phase 1-6 implemented, pending cutover." The cutover checklist has 5 pending items:
|
||||
|
||||
1. **Infisical bootstrap** — `profiles: [infisical]` in compose but `OIKOS_SECRET_BACKEND`
|
||||
env not set. SOPS fallback is active.
|
||||
2. **Watchdog tested** — pending API stop + Matrix alert verification.
|
||||
3. **Rollback drill** — `scripts/rollback.sh` never rehearsed.
|
||||
4. **Rollback verify health + re-deploy** — depends on #3.
|
||||
5. **Cleanup** — apps/105 webhooks not removed, LXC not archived.
|
||||
|
||||
**Action:** These 5 items are the highest-priority operational work. Schedule a 1-hour
|
||||
window to complete them.
|
||||
|
||||
### 2.3 Prometheus plan — references Python
|
||||
|
||||
`plans/2026-07-05-oikos-prometheus-lxc.md:60` says "Extend `oikos/scheduler.py`'s
|
||||
disk/temp probes" but the scheduler is now Go (`internal/scheduler/`). The plan also
|
||||
references `bin/homelab` Python CLI commands for provisioning.
|
||||
|
||||
**Action:** Update the plan to reference the Go check_defs system (migration 003,
|
||||
`check_defs` table) and the new `homelab` CLI (or `oikos` binary commands).
|
||||
|
||||
---
|
||||
|
||||
## 3. Documentation gaps
|
||||
|
||||
### 3.1 Missing container wiki pages
|
||||
|
||||
Two containers exist in `inventory.yaml` and `hosts/*.yaml` but have no wiki page:
|
||||
|
||||
| Container | PVE ID | Host | Wiki page |
|
||||
|-----------|--------|------|-----------|
|
||||
| seanime | 133 | strong | **Missing** |
|
||||
| romm | 134 | strong | **Missing** |
|
||||
|
||||
**Action:** Create `knowledge/wiki/containers/133-seanime.md` and
|
||||
`knowledge/wiki/containers/134-romm.md` from the data in `hosts/seanime.yaml` and
|
||||
`hosts/romm.yaml`.
|
||||
|
||||
### 3.2 Host pages for `strong.md` and `hubris.md` — stale resource math
|
||||
|
||||
`knowledge/wiki/hosts/hubris.md` and `knowledge/wiki/hosts/strong.md` may still
|
||||
reflect pre-migration resource allocation. The strong migration moved elementsynapse,
|
||||
house, arriman, jellyfin, and grimmory — both pages need updated guest lists and
|
||||
resource math.
|
||||
|
||||
**Action:** Audit both host pages against live `pct list` on each Proxmox node.
|
||||
Update guest tables and RAM/CPU summaries.
|
||||
|
||||
### 3.3 `knowledge/wiki/infrastructure/topology.md` — Mermaid graph
|
||||
|
||||
Generated by `oikos/gen-topology.py`. Should be verified to include:
|
||||
- strong and its guests (vmbr1 range 192.168.8.241–249)
|
||||
- romm (134) and seanime (133)
|
||||
- teddycloud (131)
|
||||
- grimmory (130) on strong (not hubris)
|
||||
|
||||
**Action:** Run `oikos/gen-topology.py` and verify topology reflects current state.
|
||||
Commit the regenerated file.
|
||||
|
||||
### 3.4 No ADR for Go rewrite completion
|
||||
|
||||
ADR-0001 ("Go single binary") was written as a forward-looking decision. No ADR
|
||||
documents the completed rewrite as an accepted/implemented decision with retroactive
|
||||
context on why the Python→Go transition happened when it did.
|
||||
|
||||
**Action:** Add an ADR-0011 documenting the completed Go rewrite with dates, phases,
|
||||
and the rationale for the timing.
|
||||
|
||||
### 3.5 Un-enrolled hosts tracked in inventory
|
||||
|
||||
Several hosts have `age_pubkey: ''` in `inventory.yaml`:
|
||||
- jellyfin, paperless, gitea, nextcloud, sophia, mule-images, caddy, arriman
|
||||
|
||||
These are documented as "not enrolled" in their host files. This is intentional
|
||||
(not all LXCs need homelab-context) but the pattern should be documented in the
|
||||
wiki as a convention.
|
||||
|
||||
**Action:** Add a section to `knowledge/wiki/infrastructure/homelab-context.md`
|
||||
(or `.agents/operations/agent-enrollment.md`) explaining when enrollment is expected
|
||||
vs. skipped.
|
||||
|
||||
---
|
||||
|
||||
## 4. Technical debt — Go codebase
|
||||
|
||||
### 4.1 `bin/homelab` — still a Python shim?
|
||||
|
||||
The `.gitignore` says "oikos/ kernel files are still imported by bin/homelab for
|
||||
operational CLI commands." If `bin/homelab` is a compiled Go binary (as the README
|
||||
and ADR-0001 suggest), then the Python kernel files should not be needed.
|
||||
|
||||
**Action:** Verify what `bin/homelab` is (run `file bin/homelab`). If it's a Go
|
||||
binary and doesn't import Python, the 9 superseded `.py` files in `oikos/` are
|
||||
safe to delete.
|
||||
|
||||
### 4.2 `oikos/build_hosts.go` — Python equivalent still exists
|
||||
|
||||
`cmd/oikos/build_hosts.go` exists alongside `mcp/build_host_files.py`. Both
|
||||
generate `hosts/*.yaml` from `inventory.yaml`. Are both active? If the Go version
|
||||
works, the Python version is stale.
|
||||
|
||||
**Action:** Compare output of both generators. If identical, remove the Python
|
||||
version and update the `homelab` CLI to use the Go generator.
|
||||
|
||||
### 4.3 `scripts/dns-sync.py` — standalone Python
|
||||
|
||||
This script syncs DNS records to Technitium. It's operational and has no Go
|
||||
equivalent yet. Not stale, but should be tracked as "port to Go."
|
||||
|
||||
**Action:** Add to Go rewrite backlog as a follow-up item.
|
||||
|
||||
### 4.4 `scripts/validate-seeds.py` — standalone Python
|
||||
|
||||
Validates `seeds/*.yaml`. Could be folded into `oikos seed` as a `--validate` flag.
|
||||
|
||||
**Action:** Add to Go rewrite backlog. Low priority.
|
||||
|
||||
---
|
||||
|
||||
## 5. Operational backlog (prioritized)
|
||||
|
||||
### P0 — Immediate (this week)
|
||||
|
||||
1. **[Cutover] Complete cutover checklist** — the 5 pending items from `scripts/cutover-checklist.md`:
|
||||
- Infisical bootstrap (or decide: defer and stay on SOPS)
|
||||
- Watchdog end-to-end test
|
||||
- Rollback drill (`scripts/rollback.sh` rehearsal)
|
||||
- Rollback verify + re-deploy
|
||||
- apps/105 cleanup (remove Gitea webhooks, archive LXC)
|
||||
|
||||
2. **[Docs] Create wiki pages for seanime (133) and romm (134)**
|
||||
|
||||
3. **[Plans] Mark TRMNL plan as done**, move to `plans/done/`, write changelogs
|
||||
|
||||
### P1 — Near-term (next 1-2 weeks)
|
||||
|
||||
4. **[Cleanup] Migrate `.hermes/plans/` → `plans/done/`** for all 7 files
|
||||
|
||||
5. **[Cleanup] Remove 9 superseded `oikos/*.py` files** (all except gen-topology*)
|
||||
— after verifying `bin/homelab` doesn't import them
|
||||
|
||||
6. **[Docs] Audit and update strong.md + hubris.md** guest lists and resource math
|
||||
|
||||
7. **[Docs] Regenerate `knowledge/wiki/infrastructure/topology.md`** to reflect
|
||||
current state (strong guests, new LXCs)
|
||||
|
||||
8. **[Plans] Update Prometheus plan** — port Python references to Go equivalents
|
||||
|
||||
### P2 — Medium-term (2-4 weeks)
|
||||
|
||||
9. **[Docs] ADR-0011** — document the completed Go rewrite
|
||||
|
||||
10. **[Docs] Document enrollment convention** — when LXCs get homelab-context vs.
|
||||
when they skip it
|
||||
|
||||
11. **[Cleanup] Audit and resolve oikos/cards/** — verify whether still consumed
|
||||
|
||||
12. **[Cleanup] Traefik reference audit** — document or remove old traefik mentions
|
||||
|
||||
13. **[Plans] Complete wiki-hq adoption Phases 3–5** from
|
||||
`plans/2026-07-06-adopt-wiki-hq-doc-architecture.md`:
|
||||
- Phase 4 — runbooks → skills (reshape remaining runbooks into `.agents/skills/`)
|
||||
- Phase 5 — writing-style + README pass + docs-lint
|
||||
|
||||
14. **[Code] Compare `mcp/build_host_files.py` vs `cmd/oikos/build_hosts.go`**
|
||||
— remove duplicate if outputs match
|
||||
|
||||
### P3 — Backlog (when capacity allows)
|
||||
|
||||
15. **[Code] Port `oikos/gen-topology.py` to Go** — the last Python holdout
|
||||
|
||||
16. **[Code] Port `scripts/dns-sync.py` to Go** — add to `oikos` binary
|
||||
|
||||
17. **[Code] Fold `scripts/validate-seeds.py` into `oikos seed --validate`**
|
||||
|
||||
18. **[Infra] Prometheus LXC provisioning** — per the updated plan
|
||||
|
||||
19. **[Infra] Strong Phase 3** — migrate mule-images (120) per strong migration
|
||||
assessment
|
||||
|
||||
20. **[Docs] Port `knowledge/sources/references/cert-sync-and-traefik-config.md`**
|
||||
to reflect current Caddy-based cert management
|
||||
|
||||
---
|
||||
|
||||
## 6. File inventory — what to keep, what to remove
|
||||
|
||||
### Keep (active)
|
||||
```
|
||||
cmd/oikos/** cmd/hermes/**, internal/**, api/openapi.yaml, compose/**, migrations/**,
|
||||
seeds/**, docs/adr/**, docker-compose.yml, Makefile, go.mod, go.sum, sqlc.yaml,
|
||||
inventory.yaml, hosts/*.yaml, .sops.yaml, .gitignore, .gitea/**,
|
||||
knowledge/**, .agents/**, plans/**, README.md, AGENTS.md,
|
||||
hermes/**, oikos/gen-topology.py, oikos/gen_topology_lib.py,
|
||||
scripts/{deploy,rollback,watchdog,verify-phase6}.sh, scripts/sync/**,
|
||||
scripts/check-caddy-backends.sh, scripts/cutover-checklist.md,
|
||||
secrets/**, secrets-issuance/**, bin/homelab, mcp/build_host_files.py,
|
||||
ssh/**, tools/**, vps/**, ledger/**, bootstrap.sh,
|
||||
scripts/dns-sync.py, scripts/validate-seeds.py (keep until ported)
|
||||
```
|
||||
|
||||
### Remove
|
||||
```
|
||||
oikos/__init__.py, oikos/approve.py, oikos/decide.py, oikos/drift.py,
|
||||
oikos/ledger.py, oikos/policy.py, oikos/relations.py, oikos/scheduler.py,
|
||||
oikos/signal.py
|
||||
```
|
||||
|
||||
### Migrate
|
||||
```
|
||||
.hermes/plans/*.md → plans/done/
|
||||
oikos/cards/*.md → TBD (remove if unused, keep if bin/homelab reads them)
|
||||
```
|
||||
|
||||
### Decide
|
||||
```
|
||||
secrets-issuance/ → port to Docker or decommission?
|
||||
oikos/cards/ → still consumed by bin/homelab explain?
|
||||
mcp/build_host_files.py → redundant with cmd/oikos/build_hosts.go?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Decisions requested
|
||||
|
||||
These require operator input before proceeding:
|
||||
|
||||
1. **Infisical now or later?** Bootstrap Infisical (Phase 5 production) now or defer
|
||||
and continue with SOPS-only? Risk: Infisical adds Redis + Infisical container to
|
||||
the Docker stack on mac-mini with non-trivial setup.
|
||||
|
||||
2. **Secrets issuance: port or kill?** If we decommission `secrets-issuance/`, new
|
||||
clients generate their own age key and the operator adds the pubkey manually.
|
||||
Simpler but loses automated enrollment.
|
||||
|
||||
3. **apps/105: archive or destroy?** After Docker cutover cleanup, does apps/105
|
||||
stay as a warm spare or get archived per lifecycle rules?
|
||||
|
||||
4. **oikos/cards/: keep or drop?** Verify `bin/homelab explain` behavior. If it
|
||||
queries the API, cards are dead weight. If it reads files, they stay until the
|
||||
CLI is ported.
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-07-07 — comprehensive audit created
|
||||
Catalog of 20+ stale files, plan status drift, documentation gaps, and
|
||||
prioritized operational backlog. Covers Python kernel cleanup, .hermes/plans/
|
||||
migration, oikos/cards/ assessment, cutover completion, and missing wiki pages.
|
||||
625
plans/done/2026-07-07-db-as-source-of-truth.md
Normal file
625
plans/done/2026-07-07-db-as-source-of-truth.md
Normal file
@@ -0,0 +1,625 @@
|
||||
# 2026-07-07 — DB as single source of truth for agent knowledge
|
||||
|
||||
**Status:** Done (2026-07-08) — wiki archived, FTS live, MCP + HTTP knowledge surface complete.
|
||||
|
||||
## Goal
|
||||
|
||||
Convert `knowledge/wiki/` into DB seeds, archive all original markdown files, and
|
||||
archive every other file in the repo that the Oikos Go binary does not read. After
|
||||
this, the DB is the **sole source of truth** for agents — one query surface, no grep
|
||||
fallback. All old wiki files go to root `archive/`. New knowledge is registered via
|
||||
the API, and `oikos export` writes it back to seed files for version control.
|
||||
|
||||
## Decisions from operator
|
||||
|
||||
| Question | Decision |
|
||||
|----------|----------|
|
||||
| Plans in DB? | No — agent construction docs only |
|
||||
| Ingestion trigger | Seed-time (`oikos seed` reads `seeds/knowledge.yaml`) |
|
||||
| Archive location | Root `archive/` |
|
||||
| Content granularity | Structured sections: At-a-glance → entity attributes, procedures → runbooks, changelog → parsed entries, rest → `knowledge_entities.content` |
|
||||
| Wiki files after conversion | **Archive them all** — DB is the only truth |
|
||||
| Stale files not used by Oikos | **Archive them all** |
|
||||
|
||||
## What goes to `archive/`
|
||||
|
||||
Everything the Go binary (`cmd/oikos`, `internal/`) does **not** read at runtime.
|
||||
|
||||
### Full archive list
|
||||
|
||||
| Path | Destination | Why |
|
||||
|------|-------------|-----|
|
||||
| `knowledge/wiki/containers/*.md` (20 pages) | `archive/knowledge/containers/` | Converted to seeds |
|
||||
| `knowledge/wiki/hosts/*.md` (3 pages) | `archive/knowledge/hosts/` | Converted to seeds |
|
||||
| `knowledge/wiki/infrastructure/*.md` (12 pages) | `archive/knowledge/infrastructure/` | Converted to seeds |
|
||||
| `knowledge/wiki/vms/*.md` (2 pages) | `archive/knowledge/vms/` | Converted to seeds |
|
||||
| `knowledge/sources/investigations/*.md` (5 pages) | `archive/knowledge/investigations/` | Converted to seeds |
|
||||
| `knowledge/sources/references/*.md` (1 page) | `archive/knowledge/references/` | Converted to seeds |
|
||||
| `knowledge/sources/index.md` | `archive/knowledge/` | Index page, no entity mapping |
|
||||
| `knowledge/index.md` | `archive/knowledge/` | Index page |
|
||||
| `knowledge/log.md` | `archive/knowledge/` | Doc-maintenance log, superseded by audit_log |
|
||||
| `knowledge/GLOSSARY.md` | `archive/knowledge/` | Glossary (can be re-ingested as `document:glossary` later) |
|
||||
| `oikos/cards/` (45 files) | `archive/oikos-cards/` | Python MCP explain tool; superseded by DB entities |
|
||||
| `.hermes/plans/` (7 files) | `archive/hermes-plans/` | Agent construction docs, never in DB |
|
||||
| `ledger/2026-07.jsonl` | `archive/ledger/` | Python-era artifact; Go uses audit_log table |
|
||||
| `mcp/build_host_files.py` | `archive/mcp/` | Redundant with `cmd/oikos/build_hosts.go` |
|
||||
|
||||
### Full delete list (no archival value)
|
||||
|
||||
| Path | Why |
|
||||
|------|-----|
|
||||
| `oikos/approve.py` | Superseded by `internal/domain/approval.go` |
|
||||
| `oikos/decide.py` | Superseded by `internal/policy/classify.go` |
|
||||
| `oikos/drift.py` | Superseded by `internal/scheduler/scheduler.go` |
|
||||
| `oikos/ledger.py` | Superseded by `internal/db/` + `audit_log` table |
|
||||
| `oikos/policy.py` | Superseded by `internal/policy/classify.go` |
|
||||
| `oikos/relations.py` | Superseded by `blast_radius()` SQL function |
|
||||
| `oikos/scheduler.py` | Superseded by `internal/scheduler/scheduler.go` |
|
||||
| `oikos/signal.py` | Superseded by `signals` table + `internal/domain/signal.go` |
|
||||
| `oikos/__init__.py` | Package init, never imported by Go binary |
|
||||
|
||||
### What stays (read by Oikos Go binary)
|
||||
|
||||
| Path | Read by | Why keep |
|
||||
|------|---------|----------|
|
||||
| `seeds/ontology.yaml` | `oikos seed` | DB bootstrap |
|
||||
| `seeds/inventory.yaml` | `oikos seed` | DB bootstrap |
|
||||
| `seeds/policy.yaml` | `oikos seed` | DB bootstrap |
|
||||
| `seeds/knowledge.yaml` | `oikos seed` | **NEW** — knowledge bootstrap |
|
||||
| `inventory.yaml` | `oikos build-hosts`, `oikos homelab list` | Topology source of truth |
|
||||
| `secrets/*.yaml` | `oikos secret list`, `oikos homelab secret` | SOPS-encrypted secrets |
|
||||
| `migrations/*.sql` | `oikos migrate` | Embedded SQL migrations |
|
||||
| `api/openapi.yaml` | `make generate` (oapi-codegen) | API contract |
|
||||
| `compose/`, `Dockerfile`, `docker-compose.yml` | Docker build/deploy | Runtime packaging |
|
||||
| `cmd/`, `internal/` | `go build` | Source code |
|
||||
| `bin/homelab` | `oikos homelab` subcommand | CLI binary |
|
||||
| `oikos/gen-topology.py` | `bin/homelab` (Python shim) | Topology generation (until ported) |
|
||||
| `oikos/gen_topology_lib.py` | imported by gen-topology.py | Same |
|
||||
| `scripts/deploy.sh` | Gitea webhook → deploy | Production deploy |
|
||||
| `scripts/rollback.sh` | Operator manual | Rollback |
|
||||
| `scripts/watchdog.sh` | cron on mac-mini | Health monitoring |
|
||||
| `scripts/verify-phase6.sh` | Operator manual | Acceptance verification |
|
||||
| `scripts/sync/` | systemd/launchd timers | Repo sync to clients |
|
||||
| `scripts/check-caddy-backends.sh` | Operator diagnostic | Health check utility |
|
||||
| `scripts/cutover-checklist.md` | Operator reference | Cutover tracking |
|
||||
| `scripts/dns-sync.py` | Operator manual (not yet ported) | DNS sync to Technitium |
|
||||
| `scripts/validate-seeds.py` | CI / operator manual | Seed validation |
|
||||
| `tools/` | `post-pull.sh` auto-setup | Client tooling install |
|
||||
| `ssh/` | `homelab ssh` | SSH key deployment |
|
||||
| `vps/` | `homelab render-vps-configs` | VPS templates |
|
||||
| `secrets-issuance/` | Running on apps/105 | Identity issuance (until decommissioned) |
|
||||
| `AGENTS.md` | Agent onboarding | Entry point |
|
||||
| `README.md` | Human onboarding | Entry point |
|
||||
| `.agents/` | Agent conventions, skills, operating model | Agent runtime guidance |
|
||||
| `plans/` | Agent construction docs | Intent documents |
|
||||
| `hermes/` | Hermes config + persona | Agent gateway config |
|
||||
| `Makefile`, `go.mod`, `go.sum`, `sqlc.yaml`, `.gitignore`, `.sops.yaml`, `.gitea/` | Build/config | Build tooling |
|
||||
|
||||
### Ambiguous — needs confirmation
|
||||
|
||||
| Path | Question |
|
||||
|------|----------|
|
||||
| `.agents/` | Contains OIKOS.md (operating model), SKILL.md runbooks, conventions. Go binary doesn't read these at runtime — they're agent orientation docs. Should they also be converted to seeds and archived? Or do they stay as the "agent manual" alongside AGENTS.md? |
|
||||
| `scripts/dns-sync.py`, `scripts/validate-seeds.py` | Actively used Python scripts with no Go equivalent. Keep until ported? |
|
||||
| `secrets-issuance/` | Running on apps/105. Archive after decommission? |
|
||||
|
||||
## Architecture change
|
||||
|
||||
```
|
||||
BEFORE (split):
|
||||
knowledge/wiki/ → agents grep clone for narrative
|
||||
Postgres DB → agents query MCP for structured data
|
||||
Two surfaces, no bridge
|
||||
|
||||
AFTER (DB-only):
|
||||
seeds/knowledge.yaml → oikos seed → knowledge_entities table
|
||||
↓
|
||||
MCP search_knowledge("jellyfin transcode")
|
||||
MCP get_entity_knowledge("lxc:jellyfin")
|
||||
HTTP GET /api/v1/knowledge/search
|
||||
↓
|
||||
One surface, everything indexed
|
||||
```
|
||||
|
||||
```
|
||||
┌──────────────────────────────────┐
|
||||
│ seeds/knowledge.yaml │ Version-controlled source material
|
||||
│ (generated by oikos export) │ Round-trip: seed → DB → export → seed
|
||||
│ documents: │
|
||||
│ - slug: containers/101-jellyfin│
|
||||
│ entity: lxc:jellyfin │
|
||||
│ title: "101 — jellyfin" │
|
||||
│ content: "..." │
|
||||
│ at_glance: {host: strong, ..}│
|
||||
│ changelog: [{date, title, ..}]│
|
||||
│ tags: [container, media, ..] │
|
||||
│ investigations: │
|
||||
│ - slug: 2026-06-06-caddy-... │
|
||||
│ about: [service:caddy, ...] │
|
||||
│ ... │
|
||||
│ runbooks: │
|
||||
│ - slug: service-health-check │
|
||||
│ procedure_for: [service] │
|
||||
│ ... │
|
||||
└──────────────┬───────────────────┘
|
||||
│ oikos seed
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ knowledge_entities table │
|
||||
│ title | content | content_hash │
|
||||
│ source | tags │
|
||||
│ ─ FTS: ts_vector(title+content) │
|
||||
│ │
|
||||
│ + entities table: │
|
||||
│ document:containers/101-jellyfin│
|
||||
│ investigation:2026-06-06-... │
|
||||
│ runbook:service-health-check │
|
||||
│ │
|
||||
│ + relationships: │
|
||||
│ document ──documents──► lxc │
|
||||
│ investigation ──about──► service│
|
||||
│ runbook ──procedure-for──► type│
|
||||
└──────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ MCP + HTTP API │
|
||||
│ search_knowledge("jellyfin") │
|
||||
│ get_entity_knowledge("lxc:...") │
|
||||
│ POST /api/v1/knowledge/{id} │
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Execution — Fresh install on mac-mini
|
||||
|
||||
The current Docker stack on mac-mini was deployed during cutover testing with dev data.
|
||||
Replace it with a fresh production instance using the real seeds (ontology, inventory,
|
||||
policy, knowledge).
|
||||
|
||||
### Steps
|
||||
|
||||
```bash
|
||||
# On mac-mini (192.168.8.175):
|
||||
|
||||
# 1. Stop current stack
|
||||
docker compose --profile full down
|
||||
|
||||
# 2. Wipe old Postgres volume (fresh start)
|
||||
docker volume rm oikos_pg-data
|
||||
|
||||
# 3. Pull latest repo (already committed to local main)
|
||||
git pull origin main
|
||||
|
||||
# 4. Rebuild image with current code
|
||||
docker compose build
|
||||
|
||||
# 5. Fresh migrate + seed (now includes knowledge)
|
||||
docker compose run --rm migrate
|
||||
docker compose run --rm seed
|
||||
|
||||
# 6. Verify seed ingested
|
||||
docker compose run --rm -e OIKOS_DATABASE_URL=... oikos \
|
||||
psql -c "SELECT count(*) FROM knowledge_entities;"
|
||||
# Expect: 54+ rows
|
||||
|
||||
# 7. Start production stack
|
||||
docker compose --profile full up -d
|
||||
|
||||
# 8. Health check
|
||||
curl http://localhost:8090/healthz
|
||||
./scripts/verify-phase6.sh
|
||||
|
||||
# 9. Verify knowledge endpoints
|
||||
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin"
|
||||
```
|
||||
|
||||
After fresh install succeeds, the old `backups/pre-cutover-20260707.sql` dump becomes
|
||||
stale (it's from the dev instance). The new instance is the canonical production DB.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Seed format + ingestion
|
||||
|
||||
### 1.1 Add `content_hash` to `knowledge_entities`
|
||||
|
||||
**Migration 010:**
|
||||
|
||||
```sql
|
||||
ALTER TABLE knowledge_entities ADD COLUMN content_hash TEXT;
|
||||
```
|
||||
|
||||
### 1.2 Seed format: `seeds/knowledge.yaml`
|
||||
|
||||
One YAML file with three sections. Ingested by `oikos seed` after ontology, inventory,
|
||||
and policy.
|
||||
|
||||
```yaml
|
||||
# Oikos knowledge seed — documents, investigations, and runbooks.
|
||||
# Generated by oikos export. Ingested on deploy.
|
||||
# After ingest the DB is authoritative.
|
||||
|
||||
version: 1
|
||||
|
||||
documents:
|
||||
- slug: "containers/101-jellyfin"
|
||||
title: "101 — jellyfin"
|
||||
content: |
|
||||
# 101 — jellyfin
|
||||
|
||||
Jellyfin media server with hardware-accelerated transcoding...
|
||||
|
||||
## Service / port
|
||||
| Service | Port | Notes |
|
||||
|---------|------|-------|
|
||||
| jellyfin | 8096 | Web UI + API |
|
||||
...
|
||||
|
||||
## SSO
|
||||
Authentik OIDC via SSO-Auth plugin v4.0.0.4. No Caddy forward-auth gate.
|
||||
...
|
||||
entity_slug: "lxc:jellyfin" # links via `documents` edge
|
||||
tags: ["container", "media", "jellyfin"]
|
||||
at_glance: # structured: pushed to entity attributes if empty
|
||||
host: strong
|
||||
ip: "192.168.8.246"
|
||||
cores: 4
|
||||
ram: "8 GiB"
|
||||
mounts: ["/mnt/media_local"]
|
||||
public_host: media.hubris.network
|
||||
changelog:
|
||||
- date: "2026-07-05"
|
||||
title: "migrated from hubris to strong"
|
||||
body: "Phase 2 of strong migration..."
|
||||
- date: "2026-06-10"
|
||||
title: "VAAPI configured on Radeon 680M"
|
||||
body: "GPU passthrough via dev0 + dev1..."
|
||||
|
||||
- slug: "hosts/strong"
|
||||
title: "strong — Proxmox host"
|
||||
content: |
|
||||
# strong — Proxmox host
|
||||
Reformatted from Linux workstation to Proxmox VE 9.2.3...
|
||||
entity_slug: "proxmox-host:strong"
|
||||
tags: ["host", "proxmox", "hypervisor"]
|
||||
at_glance:
|
||||
role: hypervisor
|
||||
hardware: "Minisforum UM773 Lite (Ryzen 7 PRO 6850U, 28 GiB RAM)"
|
||||
os: linux
|
||||
changelog:
|
||||
- date: "2026-07-05"
|
||||
title: "strong migration Phase 2 complete"
|
||||
body: "arriman + jellyfin + grimmory migrated. ludo-lvm..."
|
||||
|
||||
# ... 35+ more document entries
|
||||
|
||||
investigations:
|
||||
- slug: "2026-06-06-caddyfile-truncation"
|
||||
title: "Caddyfile truncation — all LAN services down"
|
||||
date: "2026-06-06"
|
||||
status: resolved
|
||||
duration: "~45 min"
|
||||
content: |
|
||||
## Symptom
|
||||
All *.hubris.network services returned 502...
|
||||
## Root cause
|
||||
...
|
||||
about_slugs: ["service:caddy", "service:dns"] # links via `about` edges
|
||||
tags: ["incident", "caddy", "dns", "downtime"]
|
||||
|
||||
- slug: "2026-06-06-authentik-session-lifetime"
|
||||
title: "Frequent Authentik login prompts"
|
||||
date: "2026-06-06"
|
||||
status: resolved
|
||||
duration: "~30 min"
|
||||
content: |
|
||||
## Summary
|
||||
...
|
||||
about_slugs: ["service:authentik"]
|
||||
tags: ["incident", "authentik", "oidc"]
|
||||
|
||||
# ... 3 more investigation entries
|
||||
|
||||
runbooks:
|
||||
- slug: "service-health-check"
|
||||
name: "Service health check"
|
||||
risk_class: read_only
|
||||
entity_type: "service" # applies to all service entities
|
||||
procedure:
|
||||
params_schema:
|
||||
type: object
|
||||
properties:
|
||||
unit: {type: string}
|
||||
required: ["unit"]
|
||||
steps:
|
||||
- name: "check systemd unit"
|
||||
runner: ssh
|
||||
command: "systemctl is-active {{ .unit }}"
|
||||
timeout_s: 10
|
||||
verify:
|
||||
- runner: ssh
|
||||
command: "systemctl is-active {{ .unit }}"
|
||||
expect: {exit_code: 0, stdout_contains: "active"}
|
||||
|
||||
- slug: "lifecycle-provision-node"
|
||||
name: "Provision new node"
|
||||
risk_class: config_mutation
|
||||
entity_type: "lxc"
|
||||
procedure:
|
||||
params_schema:
|
||||
type: object
|
||||
properties:
|
||||
name: {type: string}
|
||||
template: {type: string}
|
||||
cores: {type: integer}
|
||||
memory_mb: {type: integer}
|
||||
required: ["name", "template"]
|
||||
steps:
|
||||
- name: "create container"
|
||||
runner: ssh
|
||||
target: "{{ .host }}"
|
||||
command: "pct create $(pvesh get /nextid) {{ .template }} ..."
|
||||
timeout_s: 120
|
||||
verify: [...]
|
||||
|
||||
# ... 10 more runbook entries
|
||||
```
|
||||
|
||||
### 1.3 Ingest logic
|
||||
|
||||
`oikos seed` already runs: migrate → ontology → inventory → policy.
|
||||
|
||||
Add knowledge as step 5:
|
||||
|
||||
```
|
||||
oikos seed
|
||||
1. migrate (DDL)
|
||||
2. ontology seed (entity_types, relationship_types, lifecycles)
|
||||
3. inventory seed (entities, relationships)
|
||||
4. policy seed (risk_classes, approval_rules, autonomy)
|
||||
5. knowledge seed (documents, investigations, runbooks) ← NEW
|
||||
```
|
||||
|
||||
Knowledge ingest for each section:
|
||||
|
||||
**Documents:**
|
||||
- For each entry, create `document:<slug>` entity in `entities` table (type: `document`)
|
||||
- Insert into `knowledge_entities` (title, content, source=slug, tags, content_hash)
|
||||
- Create `documents` edge from `document:<slug>` to `entity_slug`
|
||||
- Push `at_glance` fields into the linked entity's `attributes` if those keys are empty (backfill)
|
||||
- Store `changelog` as JSONB in document entity `attributes.changelog`
|
||||
- Skip if `content_hash` matches existing row (idempotent)
|
||||
|
||||
**Investigations:**
|
||||
- Create `investigation:<slug>` entity (type: `investigation`, lifecycle: `infrastructure`)
|
||||
- Insert into `knowledge_entities`
|
||||
- Create `about` edges to each entity in `about_slugs`
|
||||
- Store `date`, `status`, `duration` in entity `attributes`
|
||||
|
||||
**Runbooks:**
|
||||
- Create `runbook:<slug>` entity (type: `runbook`, lifecycle: `skill`)
|
||||
- Insert into `knowledge_entities`
|
||||
- Create `procedure-for` edges to `entity_type` (abstract type — applies to all instances)
|
||||
- If a `skills` row with matching slug already exists, only update `knowledge_entities.content` and `content_hash` (don't overwrite the structured `procedure` JSONB in the skills table — that's managed separately)
|
||||
|
||||
### 1.4 Export: DB → seed
|
||||
|
||||
`oikos export` already writes `seeds/ontology.yaml`, `seeds/inventory.yaml`,
|
||||
`seeds/policy.yaml`. Add `seeds/knowledge.yaml`:
|
||||
|
||||
```go
|
||||
// internal/db/export.go
|
||||
func ExportKnowledge(ctx context.Context, pool *pgxpool.Pool) ([]byte, error) {
|
||||
// SELECT all document + investigation + runbook entities
|
||||
// JOIN knowledge_entities for content
|
||||
// JOIN relationships for entity links
|
||||
// Serialize to seeds/knowledge.yaml format
|
||||
}
|
||||
```
|
||||
|
||||
Round-trip guarantee: `seed → DB → export → seed` is byte-stable. Tested in CI.
|
||||
|
||||
## Phase 2 — Convert wiki to seeds, archive originals
|
||||
|
||||
### 2.1 Conversion script (one-time)
|
||||
|
||||
```
|
||||
oikos knowledge convert [--dry-run]
|
||||
```
|
||||
|
||||
- Reads every page in `knowledge/wiki/`, `knowledge/sources/`, `knowledge/GLOSSARY.md`
|
||||
- Parses structured sections (at-glance, changelog, procedures)
|
||||
- Maps each page to entity slugs using the path convention + `inventory.yaml` doc_page
|
||||
- Generates `seeds/knowledge.yaml`
|
||||
- Reports: "would write N documents, M investigations, K runbooks"
|
||||
- Without `--dry-run`: writes `seeds/knowledge.yaml`
|
||||
|
||||
Implementation: initially a Go version in `cmd/oikos/`, can fall back to a one-shot
|
||||
Python script if that's faster to build.
|
||||
|
||||
### 2.2 Archive
|
||||
|
||||
After conversion verified (seeds ingest cleanly):
|
||||
|
||||
```bash
|
||||
# Create archive directory
|
||||
mkdir -p archive/knowledge/{containers,hosts,infrastructure,vms,investigations,references}
|
||||
|
||||
# Move all knowledge files
|
||||
mv knowledge/wiki/containers/*.md archive/knowledge/containers/
|
||||
mv knowledge/wiki/hosts/*.md archive/knowledge/hosts/
|
||||
mv knowledge/wiki/infrastructure/*.md archive/knowledge/infrastructure/
|
||||
mv knowledge/wiki/vms/*.md archive/knowledge/vms/
|
||||
mv knowledge/sources/investigations/*.md archive/knowledge/investigations/
|
||||
mv knowledge/sources/references/*.md archive/knowledge/references/
|
||||
mv knowledge/index.md knowledge/log.md knowledge/GLOSSARY.md archive/knowledge/
|
||||
|
||||
# Move other stale artifacts
|
||||
mv oikos/cards/ archive/oikos-cards/
|
||||
mv .hermes/plans/*.md archive/hermes-plans/
|
||||
mv ledger/2026-07.jsonl archive/ledger/
|
||||
mv mcp/build_host_files.py archive/mcp/
|
||||
|
||||
# Delete superseded Python
|
||||
rm oikos/__init__.py oikos/approve.py oikos/decide.py oikos/drift.py \
|
||||
oikos/ledger.py oikos/policy.py oikos/relations.py oikos/scheduler.py \
|
||||
oikos/signal.py
|
||||
|
||||
# Clean empty directories
|
||||
rmdir knowledge/sources/investigations/archive/ # already moved
|
||||
rmdir knowledge/sources/investigations/
|
||||
rmdir knowledge/sources/references/
|
||||
rmdir knowledge/sources/
|
||||
rmdir knowledge/wiki/containers/archive/ # already moved
|
||||
rmdir knowledge/wiki/containers/
|
||||
rmdir knowledge/wiki/hosts/
|
||||
rmdir knowledge/wiki/infrastructure/
|
||||
rmdir knowledge/wiki/vms/
|
||||
rmdir knowledge/wiki/
|
||||
```
|
||||
|
||||
### 2.3 What the repo looks like after
|
||||
|
||||
```
|
||||
/
|
||||
├── seeds/
|
||||
│ ├── ontology.yaml
|
||||
│ ├── inventory.yaml
|
||||
│ ├── policy.yaml
|
||||
│ └── knowledge.yaml ← NEW: documents + investigations + runbooks
|
||||
├── archive/
|
||||
│ ├── knowledge/ ← all old wiki markdown
|
||||
│ ├── oikos-cards/ ← 45 Python explain cards
|
||||
│ ├── hermes-plans/ ← 7 agent construction docs
|
||||
│ ├── ledger/ ← Python-era JSONL
|
||||
│ └── mcp/ ← old Python build_host_files.py
|
||||
├── .agents/ ← agent conventions + skills (stays)
|
||||
├── plans/ ← construction plans (stays)
|
||||
├── cmd/, internal/ ← Go source
|
||||
├── migrations/, api/, compose/
|
||||
├── inventory.yaml, hosts/*.yaml
|
||||
├── AGENTS.md, README.md
|
||||
└── ...
|
||||
```
|
||||
|
||||
The repo shrinks by ~65 narrative files and ~45 card files. All knowledge is in
|
||||
`seeds/knowledge.yaml` — one file, git-tracked, round-trippable.
|
||||
|
||||
## Phase 3 — Un-stub MCP + HTTP
|
||||
|
||||
Same as previous revision:
|
||||
|
||||
- `search_knowledge(query)` → PostgreSQL FTS with `ts_rank`, `ts_headline`
|
||||
- `get_entity_knowledge(slug)` → aggregates documents, investigations, runbooks linked to entity
|
||||
- `GET /api/v1/knowledge/search` → FTS endpoint
|
||||
- `GET /api/v1/knowledge/{slug}` → entity-linked knowledge
|
||||
- `POST /api/v1/knowledge/{slug}` → agent registers new knowledge
|
||||
- `PATCH /api/v1/knowledge/{slug}` → agent updates knowledge
|
||||
|
||||
## Phase 4 — Agent conventions
|
||||
|
||||
### Knowledge registration
|
||||
|
||||
Agents call `POST /api/v1/knowledge/{entity_slug}` when they:
|
||||
- Discover an undocumented detail (e.g., config quirk, gotcha)
|
||||
- Complete an investigation
|
||||
- Create a new entity and need to attach docs
|
||||
- Update entity attributes discovered during operation
|
||||
|
||||
### Knowledge query (single call pattern)
|
||||
|
||||
```python
|
||||
# One call replaces grep + get_entity + get_relations
|
||||
ctx = mcp.get_entity_knowledge("lxc:jellyfin")
|
||||
# Returns:
|
||||
# - entity attributes (structured)
|
||||
# - documents (full-text indexed narrative)
|
||||
# - investigations (any incidents involving this entity)
|
||||
# - runbooks (procedures that apply to this entity type)
|
||||
# - relationships (blast radius, dependencies)
|
||||
```
|
||||
|
||||
### Export cycle
|
||||
|
||||
```
|
||||
agent discovers → POST /api/v1/knowledge → DB updated
|
||||
operator reviews → oikos export → seeds/knowledge.yaml updated
|
||||
git commit + push → version-controlled, diffable
|
||||
next deploy → oikos seed → DB back in sync
|
||||
```
|
||||
|
||||
## Files to create/modify
|
||||
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `migrations/010_knowledge_hash.up.sql` | New — `content_hash TEXT` |
|
||||
| `internal/knowledge/seed.go` | New — knowledge seed ingest (read seeds/knowledge.yaml → DB) |
|
||||
| `internal/knowledge/seed_test.go` | New — round-trip tests |
|
||||
| `internal/knowledge/convert.go` | New — one-shot wiki→seed converter |
|
||||
| `internal/knowledge/parser.go` | New — section parser (at-glance, changelog) |
|
||||
| `internal/db/seed.go` | Modify — add knowledge step after policy |
|
||||
| `internal/db/export.go` | Modify — add `ExportKnowledge` |
|
||||
| `internal/httpapi/impl.go` | Modify — un-stub knowledge endpoints |
|
||||
| `internal/httpapi/knowledge.go` | New — knowledge handlers |
|
||||
| `internal/mcp/server.go` | Modify — un-stub `search_knowledge`, `get_entity_knowledge` |
|
||||
| `cmd/oikos/main.go` | Modify — add `knowledge convert` subcommand |
|
||||
| `api/openapi.yaml` | Modify — knowledge endpoint schemas |
|
||||
| `seeds/knowledge.yaml` | New — generated by converter, ingested on seed |
|
||||
| `archive/` | New — all old wiki + stale files |
|
||||
| `plans/2026-07-07-comprehensive-audit-and-next-steps.md` | Modify — mark knowledge items as addressed |
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# 1. Convert wiki to seed
|
||||
oikos knowledge convert --dry-run
|
||||
# → "would write 37 documents, 5 investigations, 12 runbooks"
|
||||
oikos knowledge convert
|
||||
|
||||
# 2. Fresh DB, full seed
|
||||
oikos migrate && oikos seed
|
||||
|
||||
# 3. Verify knowledge populated
|
||||
psql -c "SELECT title, source, cardinality(tags) FROM knowledge_entities;"
|
||||
# → 54 rows (37 docs + 5 investigations + 12 runbooks)
|
||||
|
||||
# 4. Verify FTS
|
||||
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin+transcode"
|
||||
# → returns document:containers/101-jellyfin with ts_headline snippet
|
||||
|
||||
# 5. Verify entity-linked knowledge
|
||||
curl "http://localhost:8090/api/v1/knowledge/lxc:jellyfin"
|
||||
# → {entity, documents: [...], investigations: [], runbooks: [...], relationships: {...}}
|
||||
|
||||
# 6. Verify MCP
|
||||
curl -X POST localhost:8092/query \
|
||||
-d '{"tool":"search_knowledge","args":{"query":"caddy truncation"}}'
|
||||
# → returns investigation:2026-06-06-caddyfile-truncation
|
||||
|
||||
# 7. Verify agent registration
|
||||
curl -X POST localhost:8090/api/v1/knowledge/lxc:jellyfin \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"title":"VAAPI note","content":"Requires /dev/dri/renderD128 passthrough","tags":["gpu"]}'
|
||||
# → 201 Created
|
||||
|
||||
# 8. Verify export round-trip
|
||||
oikos export # writes seeds/*.yaml
|
||||
oikos seed # re-ingests (skips all — hashes match)
|
||||
# → "knowledge: ingested 0, skipped 54, errors 0"
|
||||
|
||||
# 9. Archive originals (manual, after verification)
|
||||
# → move all wiki files to archive/ as listed in Phase 2.2
|
||||
|
||||
# 10. Verify repo still builds and deploys
|
||||
make generate && make build && make test
|
||||
docker compose --profile dev up -d
|
||||
./scripts/verify-phase6.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-07-07 — v2: full archive, wiki→seeds only
|
||||
Convert wiki to seeds, archive ALL originals, archive all stale files. No human-editing
|
||||
frontend retained — DB is the only source of truth. Knowledge cycle: agent API →
|
||||
oikos export → seeds/knowledge.yaml → git → oikos seed → DB.
|
||||
169
plans/done/2026-07-07-migrate-bin-homelab-to-go.md
Normal file
169
plans/done/2026-07-07-migrate-bin-homelab-to-go.md
Normal file
@@ -0,0 +1,169 @@
|
||||
# Plan: Complete MCP tool surface — Hermes as the primary operator interface
|
||||
|
||||
**Status:** Done (2026-07-08) — all 4 phases complete. Matrix notification + approve→execute chain wired.
|
||||
|
||||
## Goal
|
||||
|
||||
The operator (dtoro) interacts with the homelab through Hermes, the AI agent.
|
||||
Hermes talks to `oikos api` via MCP protocol at `:8090`. The command-line
|
||||
`homelab` CLI is an implementation detail — the agent is the interface.
|
||||
|
||||
This plan completes the MCP tool surface so Hermes can **observe, orient,
|
||||
decide, and act** on the full homelab without the operator touching a shell.
|
||||
Once complete, `bin/homelab` (Python) and `bin/oikos` (stale binary) are deleted.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ dtoro (operator) │
|
||||
│ Interface: Hermes (natural language via chat/matrix) │
|
||||
│ Also: Oikos Console (web UI, future) │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ delegates tasks
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Hermes (AI agent, Docker :8092) │
|
||||
│ │
|
||||
│ Skills: health triage, signal response, execution │
|
||||
│ tracking, pattern learning, homelab ops, knowledge │
|
||||
│ │
|
||||
│ Protocol: MCP streamable HTTP → oikos api :8090 │
|
||||
│ Authority: auto-act on reversible_low only; escalates │
|
||||
│ config_mutation/destructive to operator via Matrix │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ MCP tools
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ oikos api (Go, Docker :8090) │
|
||||
│ │
|
||||
│ REST API + MCP server + SSE event stream │
|
||||
│ Same DB, auth middleware, audit log, policy engine │
|
||||
│ │
|
||||
│ Actuator (separate container, restricted SSH key) │
|
||||
│ Notifier (Matrix alerts) │
|
||||
│ Scheduler (OODA loop: probes, signals, patterns) │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Current MCP tool surface (15 tools)
|
||||
|
||||
### Observe — ✅ complete
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `get_entity` | Get entity by slug or UUID |
|
||||
| `list_entities` | List entities by type, state, search |
|
||||
| `get_relations` | Get relationships for an entity |
|
||||
| `get_blast_radius` | Entities affected if this entity goes down |
|
||||
| `get_health_summary` | Current fleet health summary |
|
||||
| `get_signal_history` | Open and recent signals |
|
||||
| `get_trend` | Metric trends for an entity |
|
||||
| `get_event_timeline` | Recent events |
|
||||
| `query_metrics` | Time-series metric queries |
|
||||
| `search_knowledge` | Full-text search across docs/runbooks |
|
||||
|
||||
### Orient — ✅ complete
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `get_patterns` | Learned action patterns |
|
||||
| `get_skills` | Available automation skills |
|
||||
| `get_audit_trail` | Audit log queries |
|
||||
|
||||
### Act — ⚠️ exists, limited scope
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `request_execution` | Hermes-only mutation path. Takes `target` (entity slug) + `action` (verb). Policy-gated. |
|
||||
|
||||
### Self — ✅ complete
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `get_agent_activity` | Agent self-inspection |
|
||||
|
||||
## What's missing — MCP tools to add
|
||||
|
||||
### Phase 1 — Operational visibility (P0, this week)
|
||||
|
||||
Tools Hermes needs to answer "what's happening right now?"
|
||||
|
||||
| Tool | Input | Output | Implementation |
|
||||
|------|-------|--------|----------------|
|
||||
| `tail_log` | `service_slug`, `lines` | Last N log lines via `journalctl` | SSH to host, `journalctl -u <service> -n <lines>` |
|
||||
| `get_service_status` | `service_slug` | `systemctl is-active` / `is-enabled` + uptime | SSH to host, `systemctl show` |
|
||||
| `ping_service` | `service_slug` | HTTP/TCP reachability check | HTTP GET health endpoint from oikos scheduler probe data |
|
||||
| `list_lxcs` | none | All LXCs with ID, host, IP, state | DB query from `entity_status` table |
|
||||
| `get_lxc_state` | `lxc_slug` | RAM, CPU, disk, uptime | SSH to Proxmox host, `pct status <vmid>` |
|
||||
|
||||
### Phase 2 — Execution expansion (P1, next 2 weeks)
|
||||
|
||||
Expand `request_execution` to handle the full operational surface. Currently it writes a generic execution record. It needs to understand action types and route them:
|
||||
|
||||
| Action type | What Hermes says | What actuator does |
|
||||
|-------------|-----------------|--------------------|
|
||||
| `restart` | "restart lxc:caddy" | `systemctl restart <service>` on target host |
|
||||
| `logs` | "show me caddy logs" | Already handled by `tail_log` tool |
|
||||
| `pct_exec` | "run apt update on LXC 121" | `pct exec <vmid> -- <cmd>` on Proxmox host |
|
||||
| `apt_upgrade` | "upgrade packages on lxc:caddy" | `apt update && apt upgrade -y` in detached screen |
|
||||
| `apt_audit` | "audit packages on all LXCs" | `dpkg -l` + upgradable count per host |
|
||||
| `systemctl` | "enable service on lxc:foo" | `systemctl <verb> <service>` (gated: enable = config_mutation) |
|
||||
|
||||
Each action type needs:
|
||||
1. A clear input schema (what params Hermes must provide)
|
||||
2. Policy classification (risk class, blast radius)
|
||||
3. Verification step (how to confirm it worked)
|
||||
4. Ledger entry (what happened, when, by whom)
|
||||
|
||||
### Phase 3 — Agent-to-operator escalation (P1, next 2 weeks)
|
||||
|
||||
When Hermes hits a `config_mutation` or `destructive` action, it escalates to the operator. This path needs to be solid:
|
||||
|
||||
| Component | Current state | Target |
|
||||
|-----------|--------------|--------|
|
||||
| Matrix alert | ⚠️ Notifier polls DB, but Hermes doesn't trigger it | Hermes calls `request_execution` → policy gate rejects → notifier sends Matrix alert with approval token |
|
||||
| Approval flow | ⚠️ `oikos homelab approval` exists in Go but agent can't use it | Operator reacts ✅/❌ on Matrix → webhook → approval token consumed → actuator proceeds |
|
||||
| Execution tracking | ✅ `request_execution` writes to `executions` table | Add `get_execution_status` MCP tool so Hermes can poll for results |
|
||||
|
||||
### Phase 4 — Cleanup (P2, after all tools work)
|
||||
|
||||
| Action | Notes |
|
||||
|--------|-------|
|
||||
| Delete `bin/homelab` | Python CLI. Disconnected from DB, imports deleted modules. Agent handles everything. |
|
||||
| Delete `bin/oikos` | Stale pre-built arm64 binary. Docker image is the canonical build. |
|
||||
| Remove `oikos homelab` Go subcommand | The `homelab` CLI role in the Go binary is a dead end. Remove it or keep as debug-only. |
|
||||
| Update AGENTS.md | Document the complete MCP tool surface. Remove CLI references. |
|
||||
|
||||
## Implementation approach
|
||||
|
||||
Each new MCP tool is registered in `internal/mcp/server.go` following the existing pattern:
|
||||
|
||||
```go
|
||||
register(&mcp.Tool{Name: "tail_log", Description: "...", InputSchema: ...},
|
||||
func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||
// ...
|
||||
})
|
||||
```
|
||||
|
||||
SSH-based tools (`tail_log`, `get_service_status`, `list_lxcs`, `get_lxc_state`)
|
||||
reuse the mesh-address resolution already working for `request_execution`:
|
||||
entity slug → DB lookup → inventory attributes → mesh IP → `ssh user@ip`.
|
||||
|
||||
The actuator container holds the SSH key. MCP tools that need SSH should route
|
||||
through the actuator's execution queue rather than holding their own SSH key,
|
||||
maintaining the security boundary: Hermes → MCP → API → execution queue → actuator → SSH.
|
||||
|
||||
## Verification
|
||||
|
||||
- Hermes can answer "show me caddy logs" → `tail_log` returns journal lines
|
||||
- Hermes can answer "what's the fleet health?" → `get_health_summary` + `list_lxcs`
|
||||
- Hermes can answer "restart caddy" → `request_execution(action="restart", target="lxc:caddy")` → actuator SSHs → caddy restarts → Hermes confirms
|
||||
- Hermes can answer "upgrade packages on dns LXC" → `request_execution(action="apt_upgrade", target="lxc:dns")` → gated → actuator runs in screen → Hermes reports result
|
||||
- `bin/` directory is empty or contains only Docker-related scripts
|
||||
|
||||
## Changelog
|
||||
|
||||
- 2026-07-07 rev 2 — flipped paradigm. Operator interface is Hermes via MCP, not CLI.
|
||||
Replaced 4-phase CLI port plan with 4-phase MCP completion plan.
|
||||
- 2026-07-07 rev 1 — cataloged 27 Python CLI subcommands vs 4 Go subcommands.
|
||||
Reference in New Issue
Block a user