plans: add oikos gaps review and control-room webui plans

Documents a full-project review (confirmed bugs, security gaps, user- and
agent-perspective gaps) and a realtime control-room web UI plan, per prior
codebase exploration on this branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

plans/2026-07-08-control-room-webui.md

@@ -0,0 +1,184 @@
+# 2026-07-08 — Control room web UI
+
+**Status:** Planned
+
+## Goal
+
+A realtime "control room" web UI for inspecting the state of Oikos and all its
+entities — graphs, tables, and lists — that updates live as an agent (Hermes)
+interacts with the system: executions appearing, approvals firing, health
+changing, the entity graph mutating. Inspect **and act**: approve/deny pending
+approvals and ack/resolve/mute signals directly from the UI.
+
+---
+
+## Stack (decided)
+
+**Svelte 5 + Vite + TypeScript SPA**, compiled to static assets in
+`web/dist`, embedded into the existing `oikos` binary via
+`go:embed all:web/dist` in a new `internal/httpapi/ui.go`, served at `/ui/`
+(redirect `/` → `/ui/`, SPA fallback to `index.html`).
+
+Why: the hard requirements (force-directed graph, time-series charts, one SSE
+stream patching many widgets) are client-side-JS problems; Svelte's reactive
+stores map 1:1 onto "SSE event mutates shared state, every widget reacts";
+the compiled runtime keeps the embed small; and `openapi-typescript` generates
+frontend types from `api/openapi.yaml` — the frontend twin of the repo's
+oapi-codegen contract-first discipline (ADR-0004). Embedding preserves the
+single-binary story: no new container, no CORS, no Caddy changes.
+
+Dependencies kept minimal:
+- `d3-force` — graph physics only; render SVG/canvas by hand
+- `uPlot` — ~45 KB canvas time-series, ideal for `/metrics` rollups
+- `openapi-typescript` — dev-only, generates `api-types.d.ts`
+- No SvelteKit (no SSR wanted — the Go binary is the server), no component
+  framework; hash router; hand-rolled dark-theme CSS.
+
+Build integration: commit a placeholder `web/dist/index.html` so backend-only
+`go build` never breaks; `make ui` runs the Vite build; add a node stage to
+`compose/oikos/Dockerfile` (3-stage: node → go → runtime). Local dev:
+`vite dev` proxying `/api` → `:8090`.
+
+---
+
+## Realtime
+
+Reuse the existing pipeline: migration 008's `pg_notify('oikos_events')`
+trigger → `internal/httpapi/sse.go` broker → `GET /api/v1/events/stream`
+(Last-Event-ID replay, 15s heartbeat). One shared `EventSource` in a Svelte
+store; pages subscribe by event type and either patch state from `event.data`
+or trigger a targeted refetch. Scheduler and actuator are separate processes
+but share Postgres, so their events reach the api role's LISTEN automatically.
+
+### Event emission gap-fill (required)
+
+Today only 5 event types are emitted (`entity.created`, `entity.updated`,
+`client.enrolled`, `entity.provisioned`, `execution.requested`). Most of what
+the control room must show live is silent. Add `observability.Event(...)`
+calls at:
+
+| Event | Site |
+|-------|------|
+| `signal.raised` / `signal.resolved` | `internal/scheduler/scheduler.go` (~115 / ~145) |
+| `health.changed` (on transition only, to avoid flooding) | scheduler `entity_status` writes (~104/135/156) |
+| `signal.acked` / `signal.muted` / `signal.resolved` (API side) | `internal/httpapi/impl.go:545/582/619` |
+| `approval.created` | `internal/mcp/server.go` `createApproval` — depends on bug A1 in [gaps plan](2026-07-08-oikos-gaps-and-improvements.md) |
+| `approval.decided` | `internal/httpapi/phase3.go:871` |
+| `execution.started/completed/failed/cancelled` | `internal/actuator/actuator.go`, `phase3.go:746` |
+| `relationship.created` / `relationship.ended` | `phase3.go:1560/1621` |
+
+Agent activity: poll `GET /agent-activity` every ~5s rather than duplicating
+every tool call into `events`.
+
+---
+
+## API surface
+
+Existing endpoints already cover nearly everything the UI needs:
+`/graph?root=&depth=&rel_type=` (impl.go:268), `/health` fleet rollup
+(impl.go:481), `/metrics?rollup=raw|1h|1d|auto&from=&to=`, `/trends/{id}`,
+`/events` (filterable) + `/events/stream` (SSE), `/signals` + ack/resolve/
+mute, `/executions`, `/approvals` + `POST /approvals/{id}/decision`,
+`/agent-activity`, `/knowledge/search`, `/audit`, entities CRUD, `/ontology`.
+
+New endpoints (spec-first in `api/openapi.yaml`, regen, then implement):
+
+1. `GET /api/v1/dashboard/summary` (new `internal/httpapi/dashboard.go`) —
+   one round-trip for the overview: entity counts by type/state, health
+   rollup from `entity_status`, open signals by severity, pending approvals,
+   executions by state (24h), event-rate sparkline (count/5min from the
+   `events` hypertable).
+2. `GET /graph?include=status` — join `entity_status` so graph nodes can be
+   colored by health (preferred over a separate bulk-status endpoint).
+
+---
+
+## Auth & deployment
+
+- Serve `/ui/*` assets without `combinedAuth` — Caddy/Authentik already gates
+  the `oikos.hubris.network` vhost, and the assets are public JS/CSS.
+- **Trusted-proxy header auth** for API calls from the browser: extend
+  `combinedAuth` (`internal/httpapi/server.go:157`) with a config flag
+  (`OIKOS_TRUSTED_PROXY_AUTH`) — when no Bearer token is present but Authentik
+  forward-auth headers (`X-Authentik-Username`/`-Email`) are, resolve an
+  operator actor. Caddy must inject these via `forward_auth` and strip inbound
+  `X-Authentik-*` from clients. Result: the UI needs no token handling, and —
+  critically — `EventSource` works unmodified (it cannot set Authorization
+  headers).
+- Docker: node build stage in `compose/oikos/Dockerfile`; no compose or Caddy
+  routing changes.
+
+---
+
+## Pages
+
+Persistent nav + live top status strip (health dots, open-signal badge,
+pending-approval badge):
+
+1. **Overview** — summary cards from `/dashboard/summary`, event-rate
+   sparkline, live event ticker, top degraded entities.
+2. **Graph explorer** — d3-force over `/graph`; node color = health; filters
+   by entity type / rel_type / root+depth; click → side panel with attributes
+   + blast radius; live mutation from `entity.*` / `relationship.*` /
+   `health.changed` events.
+3. **Entity detail** — attributes, relations mini-graph, blast radius, uPlot
+   charts (`/metrics`, `/trends`), entity-scoped events, knowledge, open
+   signals, executions.
+4. **Entities table** — filter/sort by type/state/health, live badges.
+5. **Operations ledger** — executions + approvals panes with **approve/deny
+   buttons** (`POST /approvals/{id}/decision`); live via `execution.*` /
+   `approval.*` events; grouped by `correlation_id`. The "watch Hermes work"
+   page.
+6. **Signals** — tabs by state, severity filter, **ack/resolve/mute** actions.
+7. **Live event feed** — full stream tail with filters, pause,
+   correlation-id clustering (one agent action = one cluster), scroll-back via
+   `GET /events`.
+8. **Agent activity** — polled `/agent-activity` timeline joined to
+   executions/approvals by correlation_id.
+9. **Knowledge** — FTS search over `/knowledge/search`. **Audit** — low
+   priority.
+
+---
+
+## Milestones
+
+- **M1 (MVP):** scaffold + embed + proxy auth; Overview, Entities table, Live
+  event feed over SSE. Proves the pipeline end-to-end (agent creates an
+  entity → it appears live in the browser).
+- **M2:** event gap-fill (all sites above); Operations ledger with live
+  approvals + decision buttons; Signals page. The payoff milestone. *Depends
+  on gaps-plan bug A1 for approvals to exist at all.*
+- **M3:** graph explorer with live mutation; entity detail with charts;
+  `/dashboard/summary`; `include=status`.
+- **M4:** agent activity, knowledge, audit, correlation grouping, polish.
+
+## File layout
+
+```
+web/
+  package.json  vite.config.ts
+  src/
+    main.ts  App.svelte  router.ts
+    lib/api.ts  lib/api-types.d.ts
+    lib/stores/{events.ts, summary.ts}
+    pages/{Overview,Graph,Entity,Entities,Ops,Signals,Events,Agent,Knowledge}.svelte
+  dist/index.html                     # committed placeholder
+internal/httpapi/ui.go                # go:embed + SPA fallback at /ui
+internal/httpapi/dashboard.go         # GET /dashboard/summary
+internal/httpapi/server.go            # mount /ui; trusted-proxy auth
+api/openapi.yaml                      # new paths
+internal/{scheduler/scheduler.go, mcp/server.go, actuator/actuator.go,
+  httpapi/{impl,phase3}.go}           # missing event emissions
+Makefile  compose/oikos/Dockerfile    # ui build targets, node stage
+```
+
+## Verification
+
+- M1: run `oikos api` locally, open `/ui/`, confirm Overview + Entities render
+  from live API; `POST /api/v1/entities` from curl and watch it appear in the
+  event feed without refresh.
+- M2: drive `request_execution` through Hermes and watch the approval appear
+  on the Operations ledger, decide it from the UI, and see the execution state
+  advance live.
+- Backend-only `go build ./...` succeeds without a node toolchain (placeholder
+  dist).

plans/2026-07-08-oikos-gaps-and-improvements.md

@@ -0,0 +1,202 @@
+# 2026-07-08 — Oikos gaps, broken things, and improvements
+
+**Status:** Planned
+
+## Goal
+
+Full-project review of Oikos from two vantage points — a user interacting
+through Hermes, and an agent working through the MCP tool surface — with every
+finding verified against source (file:line), plus a prioritized fix order.
+This plan is the map; each numbered fix is small enough to land independently.
+
+---
+
+## A. Confirmed bugs (verified in source)
+
+### A1. Approvals are never created — FK violation, errors swallowed (CRITICAL)
+
+`createApproval` at `internal/mcp/server.go:970` inserts a fresh
+`uuid.NewV7()` as `approvals.entity_id`, but
+`migrations/003_operations.up.sql:41` declares
+`entity_id UUID PRIMARY KEY REFERENCES entities(id)`. The INSERT always
+violates the FK, and both `pool.Exec` errors are discarded.
+
+Net effect: `request_execution` for `systemctl enable/disable` or
+`apt_upgrade` marks the execution `pending_approval`
+(`internal/mcp/server.go:311`, `:366`) and tells the agent it's queued, but no
+approval row exists → the notifier never sends a Matrix alert → the execution
+is orphaned forever. From the Hermes user's perspective, config mutations
+silently dead-end while appearing accepted.
+
+Contrast: `request_execution` (server.go:286) correctly creates a companion
+`entities` row for the execution first — approvals just never got the same
+treatment.
+
+**Fix:**
+- Preferred: migrate `approvals` to its own `id UUID PRIMARY KEY` (not FK'd to
+  `entities`), keeping `subject_entity_id` as the entity link. Update notifier
+  and `DecideApproval` queries accordingly.
+- Alternative (no migration): create a companion `entities` row like
+  executions do.
+- Either way: check and log every `Exec` error in `createApproval`, and verify
+  the `UPDATE executions SET approval_id = $2 WHERE entity_id = $1` column
+  semantics (`executions` is also keyed by `entity_id`).
+
+### A2. Matrix message-flooding vectors (`internal/notifier/notifier.go`)
+
+- Initial alert is guarded by `alert_sent_at` (notifier.go:103), but the guard
+  is written *after* the Matrix send (notifier.go:111). If the UPDATE fails
+  after a successful send, the ticker re-sends every cycle.
+- No dedup of approvals by `(subject_entity_id, action, payload)`. Once A1 is
+  fixed, every retried `request_execution` mints a new approval → one Matrix
+  message each.
+- `pollReactions` (notifier.go:118) issues one Matrix relations GET per
+  pending approval per 30s poll, uncapped — ignored approvals accumulate for
+  their 1h lifetime and multiply API calls. It also re-dispatches decisions
+  for approvals stuck `pending` (no "already acted" guard if the
+  DecideApproval call errors without flipping status).
+
+**Fix:** mark-then-send (or transactional outbox) for `alert_sent_at`; upsert/
+dedup open approvals on `(subject_entity_id, action, payload)`; cap + backoff
+on reaction polling; guard against re-dispatching a decision already in
+flight.
+
+### A3. Hermes "help" is broken + dead code
+
+`cmd/hermes/main.go:165` handles "what can you do"/"help" by calling
+`client.callTool("tools/list", nil)` — a `tools/call` for a tool literally
+named `tools/list`, which doesn't exist. The correct `listTools()` helper
+(main.go:318) is dead code, never called. **Fix:** wire `listTools()` in.
+
+### A4. `resolveHost` never returns a per-entity SSH user
+
+`internal/mcp/server.go:943` — the named return `sshUser` is always `""`; the
+per-entity user branch is dead and everything relies on `sshExec`'s global
+default fallback. **Fix:** read the SSH user from entity attributes or delete
+the dead return to make the behavior honest.
+
+### A5. `queryRows` stringifies every column
+
+`internal/mcp/server.go:861` renders all values via `fmt.Sprintf("%v", ...)`,
+so numbers, bools, timestamps, and JSON all reach agents as strings.
+**Fix:** type-preserving serialization (pass through pgx-native values into
+`json.Marshal`) — improves every read tool at once.
+
+### A6. `get_state_snapshot` description is stale
+
+`internal/mcp/server.go:689` still advertises "disk, drift count" — columns
+removed in commit 3ea43ad. **Fix:** update the description.
+
+---
+
+## B. Security gaps
+
+### B1. Enrollment is unauthenticated, with a false comment
+
+`internal/httpapi/server.go:97` says "unauthenticated (IP-gated in handler)"
+but `EnrollClient` (`internal/httpapi/impl.go:1099`) performs no IP check at
+all — the only gate is the target entity being in state
+`planned`/`provisioning`. Caddy's `@enroll` matcher bypasses Authentik.
+Anyone reaching `oikos.hubris.network` who knows (or guesses) a planned slug
+receives that node's **age private key** in the HTTP response body.
+
+**Fix:** enforce a real gate (mesh-CIDR check, one-time enrollment token
+minted when the entity is created, or both), and stop returning the age
+private key in the response — have the client fetch it from the secret store.
+
+### B2. Fake Infisical credentials returned to enrollees
+
+`internal/httpapi/impl.go:1191-1192` returns `"inf_client_"+uuid` /
+`"inf_secret_"+uuid` — random strings wired to nothing. Enrolled clients hold
+credentials that authenticate against nothing.
+**Fix:** implement `CreateMachineIdentity` in `internal/secrets/infisical.go`,
+or return no credentials and document the manual step.
+
+### B3. Hermes `/query` has no auth
+
+`hermes/config.yaml:9` sets `mesh_only: true` but `cmd/hermes/main.go` never
+reads or enforces it — it serves any caller on :8092, who can invoke
+`request_execution`. **Fix:** enforce mesh-CIDR (or bearer token) in the
+handler; fail closed.
+
+### B4. SSH host keys not verified
+
+`ssh.InsecureIgnoreHostKey()` at `internal/mcp/server.go:920`.
+**Fix:** known_hosts pinning (keys are already inventory-managed per node).
+
+### B5. `list_my_secrets` enumerates all node pubkeys
+
+Without `caller_pubkey`, `internal/mcp/server.go:709-720` returns every entity
+that has an `age_pubkey`; nothing ties the caller to what it may list.
+**Fix:** require `caller_pubkey` and scope results to the caller's
+entitlements.
+
+---
+
+## C. User perspective (interacting via Hermes)
+
+- `routeQuery` NLU is hardcoded `strings.Contains`; `extractEntity`
+  (`cmd/hermes/main.go:173`) recognizes only 5 services (`authentik, caddy,
+  vaultwarden, gitea, immich`) plus `mac-mini`/`hubris`. Any other entity →
+  "no entity found", and *any* unmatched query silently falls back to
+  `get_health_summary` — wrong answers that look like answers.
+- No conversation/session context; no follow-up capability.
+- Config mutations appear accepted but silently dead-end (A1).
+
+**Recommendation:** either make Hermes a real LLM-backed agent loop (Claude
+API driving the 28 MCP tools) or explicitly scope it as a structured-tool
+gateway: remove the toy NLU, make the fallback say "I don't understand this
+query; here are the tools" (via the fixed `listTools()`), and document that
+natural language belongs to the calling agent, not the gateway.
+
+---
+
+## D. Agent perspective (MCP tooling gaps)
+
+28 tools are registered in `internal/mcp/server.go` (README says 15, AGENTS.md
+says 21 — both stale). Missing capabilities:
+
+1. **No knowledge write.** AGENTS.md tells agents to register knowledge via
+   `POST /api/v1/knowledge/{slug}`, but there is no MCP tool — MCP-only agents
+   cannot write back what they learn. Add `upsert_knowledge`.
+2. **No entity/signal mutation.** Create/patch entity, state transitions, and
+   signal ack/resolve/mute all exist in REST (`internal/httpapi/impl.go`,
+   `phase3.go`) but not in MCP. Add at least signal ack/resolve/mute and a
+   policy-gated entity attribute patch.
+3. **No approval visibility.** After `request_execution` returns
+   `pending_approval`, an agent has no way to check or reference the approval.
+   Add `get_approval_status` / `list_pending_approvals`.
+4. Execution actions limited to `restart | systemctl | pct_exec |
+   apt_upgrade` — no deploy/rollback/config-edit path.
+5. Architecture/doc mismatch: `hermes/SOUL.md` claims "no SSH access; all
+   mutations flow through the actuator", but the MCP server runs
+   `restart`/`pct_exec` synchronously over SSH from inside the api process
+   (`sshExec`, server.go:902). Align docs or move execution to the actuator.
+
+---
+
+## E. Doc drift / housekeeping
+
+- Tool counts: README 15 / AGENTS.md 21 / actual 28 — regenerate from
+  `internal/mcp/server.go` (consider a doc-gen make target).
+- `compose/caddy/Caddyfile.oikos` retains literal `<mac-mini-mesh-ip>`
+  placeholders in all three vhosts.
+- `.agents/HERMES.md` lists "`inventory.yaml`, `inventory.yaml`" (duplicate).
+- `plans/index.md` drift: fix-MCP-tools row sat in Active with a broken link
+  after the file moved to `done/` (fixed alongside this plan); TRMNL listed
+  active though in `done/`; Grimmory header says `in-progress` though in
+  `done/`; `.hermes/plans/` (7 executed plans) missing from disk.
+- `plans/2026-07-05-oikos-prometheus-lxc.md` (~0% done) references deleted
+  `oikos/scheduler.py` and `bin/homelab`; LXC 131 collision unresolved.
+
+---
+
+## F. Prioritized fix order
+
+1. **A1** approval FK + error handling — unblocks the entire
+   approval → Matrix → execution path.
+2. **A2** notifier flooding guards — this branch's namesake.
+3. **B1/B2** enrollment security + **B3** Hermes auth.
+4. **D** MCP tool additions — approval status first, then knowledge write,
+   then signal ops.
+5. **C** Hermes routing honesty + **A3-A6**, **B4/B5**, **E** drift cleanup.

plans/index.md

@@ -10,7 +10,8 @@ went sideways, open an investigation.
 | ---- | ----- | ------ |
 | 2026-07-05 | [Oikos Prometheus LXC](2026-07-05-oikos-prometheus-lxc.md) | Planned |
 | 2026-07-08 | [Plan vs implementation cross-reference](2026-07-08-plan-implementation-audit.md) | Planned |
-| 2026-07-08 | [Fix MCP analysis tools](2026-07-08-fix-mcp-analysis-tools.md) | Done |
+| 2026-07-08 | [Oikos gaps, broken things, and improvements](2026-07-08-oikos-gaps-and-improvements.md) | Planned |
+| 2026-07-08 | [Control room web UI](2026-07-08-control-room-webui.md) | Planned |
 
 ## Done
 
@@ -28,6 +29,7 @@ See [`done/`](done/) for executed plans:
 | 2026-07-07 | [DB as single source of truth for agent knowledge](done/2026-07-07-db-as-source-of-truth.md) |
 | 2026-07-07 | [Comprehensive audit: stale files, state gaps, and next steps](done/2026-07-07-comprehensive-audit-and-next-steps.md) |
 | 2026-07-07 | [Client lifecycle in Go — enrollment through deprecation](done/2026-07-07-client-lifecycle-in-go.md) |
+| 2026-07-08 | [Fix MCP analysis tools](done/2026-07-08-fix-mcp-analysis-tools.md) |
 | 2026-07-06 | [Consolidate Oikos control plane onto mac-mini](done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md) |
 
 ## Conventions