oikos/plans/2026-08-05-backend-evaluation-improvements.md

# 2026-08-05 — Backend evaluation: architecture, security, and reliability improvements

Status: **In Progress** — Phase 0 code changes complete (B1, B2, B4, B5, B6, B7); B3
is a post-deploy operational step.

Scope: full evaluation of the oikos backend (Go binaries `oikos`, `nomos`, `webhook`,
Postgres/TimescaleDB, Docker deployment, MCP server) excluding frontend clients
(`web/` SPA and `desktop/` Wails app). Research-only pass — no code changes.

Method: four parallel research passes (Go backend structure, database schema,
deployment/infrastructure, API/MCP design) plus Infisical secrets audit and
dependency analysis of `go.mod`.

---

## A. Summary of findings

The backend is well-architected with strong fundamentals: contract-first API
(oapi-codegen), type-safe SQL (sqlc + pgx), TimescaleDB observability, policy-
governed autonomy, and a sophisticated ontology-driven data model. The main gaps
are secret management (Infisical is wired but barely used — 3 of 4 binaries
read secrets from env/plaintext), operational maturity (CI, image tagging, backup
reliability), security hardening (SSH host keys, unauthenticated endpoints), and
code hygiene (monolithic files, mixed SQL access patterns).

| Category | Grade | Notes |
|----------|-------|-------|
| Tech stack | A | Go + pgx + sqlc + TimescaleDB + chi + slog — all correct choices |
| Data model | A- | Dual-entity pattern, temporal relationships, partial indexes, 6 state machines |
| API design | B+ | Contract-first with ~50 MCP tools, RFC 9457 errors; no rate limiting |
| Secret management | **D** | Infisical SDK wired but only in API/MCP tools path; nomos, webhook, scheduler, notifier all read plaintext env vars. 7 production secrets in `.env`, HMAC in world-readable plist. SOPS fallback is dead code. |
| Security | C | OIDC+static token auth is good, but SSH host keys disabled, unauthenticated nomos endpoint, HMAC secret in world-readable plist |
| Deployment | C+ | Multi-stage builds, pre-deploy backups, but no CI, no versioned images, silent backup failures |
| Code quality | B | Good error handling, panic safety, doc; but 3 files over 1100 lines, mixed raw/sqlc SQL |
| Performance | B | Appropriate for scale; SSH check storm risk, no query caching |
| Observability | B- | TimescaleDB hypertables + SSE + slog; no Prometheus/Grafana, no OTel tracing |
| Testing | C | `make test` exists but many core packages (scheduler, domain, actuator, policy) have 0% coverage |

## B. Infisical consolidation (critical)

Infisical is deployed (Redis + Infisical service in compose, Go SDK in go.mod,
`internal/secrets/infisical.go` fully implemented) but severely underutilized.
Only the API server's MCP tools path creates an Infisical backend. Every other
binary reads secrets from env vars or plaintext files.

### Current wiring map

| Binary / role | Uses Infisical? | Secrets read from env/plaintext |
|---------------|----------------|-------------------------------|
| oikos `api` role | Yes (MCP tools only) | `OIKOS_DATABASE_URL`, `OIKOS_MCP_BEARER_TOKEN`, `OIKOS_API_TOKEN`, `OIKOS_OIDC_CLIENT_SECRET` |
| oikos `scheduler` role | **No** | `OIKOS_SSH_KEY_PATH`, `OIKOS_DATABASE_URL` |
| oikos `notifier` role | **No** | `OIKOS_MATRIX_TOKEN`, `OIKOS_APPROVAL_HMAC_SECRET`, `OIKOS_DATABASE_URL` |
| nomos | **No** | `OPENROUTER_API_KEY`, `OIKOS_MCP_BEARER_TOKEN`, `DATABASE_URL` |
| webhook | **No** | `WEBHOOK_HMAC_SECRET` (also hardcoded in plist) |

### B1. Wire Infisical into all binaries at startup

- **Goal**: Every binary fetches its secrets from Infisical at startup instead of
  relying on env vars. Bootstrap-only env vars (`INFISICAL_CLIENT_ID`,
  `INFISICAL_CLIENT_SECRET`, `INFISICAL_SITE_URL`, `INFISICAL_PROJECT_ID`,
  `OIKOS_DATABASE_URL`) remain as env vars (chicken-egg).
- **Approach**: Add a `secrets.InitFromEnv(ctx)` call to each binary's `main()` that
  creates a `secrets.Manager` (primary Infisical + SOPS fallback). Store the
  manager in a package-level var or pass it through the initialization chain.
- **Files to change**:
  - `cmd/oikos/main.go` — create Manager in `runWithPool`, pass to scheduler and
    notifier runners alongside cfg and pool
  - `cmd/nomos/main.go` — create Manager at startup, fetch `OPENROUTER_API_KEY`
    and `OIKOS_MCP_BEARER_TOKEN` from Infisical before creating the agent
  - `cmd/webhook/main.go` — create Manager at startup, fetch `WEBHOOK_HMAC_SECRET`
    from Infisical
- **Secrets to migrate into Infisical** (move from env vars / `.env` / plist):

  | Secret key (in Infisical) | Current source | Used by |
  |---------------------------|---------------|---------|
  | `matrix/token` | `OIKOS_MATRIX_TOKEN` env | oikos notifier |
  | `approval/hmac-secret` | `OIKOS_APPROVAL_HMAC_SECRET` env, plist | oikos notifier, webhook |
  | `mcp/bearer-token` | `OIKOS_MCP_BEARER_TOKEN` env | oikos api, nomos |
  | `api/token` | `OIKOS_API_TOKEN` env | oikos api |
  | `oidc/client-secret` | `OIKOS_OIDC_CLIENT_SECRET` env | oikos api |
  | `openrouter/api-key` | `OPENROUTER_API_KEY` env | nomos |
  | `webhook/hmac-secret` | `WEBHOOK_HMAC_SECRET` env + plist | webhook |
- **Risk class**: config_mutation
- **Prerequisite**: Populate Infisical with these secrets via `oikos secret set` before
  deploying the code change. Existing `.env` values serve as the source of truth
  for the initial migration.

### B2. Activate the SOPS fallback path

- **Status**: Done (commit pending)
- **Current**: `secrets.NewManager(infisical, sops)` is only used in tests.
  `httpapi/server.go` creates `InfisicalBackend` directly — if Infisical is down,
  there is no fallback.
- **Fix**: Use `secrets.NewManager()` in production everywhere so the SOPS DR
  fallback actually works when Infisical is unreachable. The Manager's cache
  (5min TTL) already masks transient Infisical blips.
- **What changed**: `httpapi/server.go` now creates `secrets.NewManager(infisical,
  sopsFallback)` instead of bare `secrets.NewInfisicalBackend`. SOPS backend
  is created from `cfg.SecretsDir` when set.
- **Risk class**: config_mutation

### B3. Remove `.env` plaintext secrets after migration

- **Current**: `.env` contains 7 production secrets in plaintext on mac-mini disk.
- **Fix**: After B1 is deployed and all binaries read from Infisical, strip
  secrets from `.env` leaving only non-secret config (`OIKOS_API_LISTEN`,
  `OIKOS_SCHEDULER_INTERVAL`, etc.). Bootstrap env vars
  (`OIKOS_DATABASE_URL`, `OIKOS_INFISICAL_*`) stay — they're the trust anchor.
- **Risk class**: config_mutation

### B4. Remove HMAC secret from plist

- **Status**: Done (code change; plist cleanup is post-deploy)
- **Current**: `scripts/deploy/network.hubris.oikos-deploy-webhook.plist` line 14
  has `WEBHOOK_HMAC_SECRET` hardcoded in plaintext. World-readable.
- **Fix**: After B1 (webhook reads from Infisical), remove the `EnvironmentVariables`
  `WEBHOOK_HMAC_SECRET` entry from the plist entirely. The webhook binary will
  fetch it from Infisical at startup.
- **Risk class**: config_mutation
- **Supersedes**: Original plan item B3 (HMAC secret in plist) — same issue,
  now resolved via Infisical instead of file permissions workarounds.

### B5. Store SSH host keys in Infisical

- **Status**: Done (commit pending)
- **Current**: `ssh.InsecureIgnoreHostKey()` in 3 code paths
  (`internal/actuator/ssh.go:160`, `internal/actuator/actuator.go:427`,
  `internal/mcp/server.go`).
- **Fix**: Store Proxmox host public keys in Infisical under
  `ssh/host-keys/{hostname}`. Actuator reads them at connection init and builds
  a `knownhosts` callback. For dynamic targets, implement TOFU (trust-on-first-
  use) writing back to Infisical.
- **What changed**:
  - `internal/actuator/hostkeys.go` — `HostKeyCallback()` returns an
    `ssh.HostKeyCallback` that verifies against cached keys (MITM detection)
    and accepts unknown hosts via TOFU, persisting new keys to Infisical.
  - `internal/actuator/hostkeys_infisical.go` — `InfisicalHostKeySource`
    implements `HostKeySource` over `secrets.Backend`; `ResolveSSHHosts()`
    queries DB for active proxmox-host/standalone-server slugs.
  - `internal/actuator/ssh.go:160` — replaced `InsecureIgnoreHostKey()` with
    `HostKeyCallback()`.
  - `internal/actuator/actuator.go:427` — replaced `InsecureIgnoreHostKey()` with
    `HostKeyCallback()`.
  - `internal/mcp/server.go:494` — replaced `InsecureIgnoreHostKey()` with
    `actuator.HostKeyCallback()`.
  - `internal/httpapi/server.go` — pre-loads SSH host keys from Infisical at
    startup (queries active hosts, loads their keys from Infisical).
- **Post-deploy step**: On first deploy, TOFU will accept all current host
  keys and store them in Infisical under `ssh/host-keys/{slug}`. Verify the
  stored keys are correct by checking `oikos secret list`. To pre-populate
  without TOFU, SSH to each Proxmox host and run:
  `ssh-keyscan -t ed25519 {host} | awk '{print $2" "$3}'` and store
  the output via `oikos secret set ssh/host-keys/{slug} {output}`.
- **Risk class**: config_mutation (initial pin) / destructive (if keys change)

### B6. Fix env var naming inconsistency

- **Status**: Done (commit pending)
- **Current**: Config uses `OIKOS_INFISICAL_*` prefix in docker-compose but
  `internal/secrets/infisical.go` lines 58–62 falls back to bare `INFISICAL_*`
  (without OIKOS prefix). Two naming conventions for the same bootstrap vars.
- **Fix**: Standardize on `OIKOS_INFISICAL_*` everywhere. Remove the bare
  `INFISICAL_*` fallback in infisical.go.
- **What changed**: Removed the `os.Getenv("INFISICAL_CLIENT_ID")` and
  `os.Getenv("INFISICAL_CLIENT_SECRET")` fallbacks in `infisical.go connect()`.
  Removed unused `os` import. Error message updated to reference
  `OIKOS_INFISICAL_*` names.
- **Risk class**: config_mutation

### B7. Align `secretsBackend` interface with `secrets.Backend`

- **Status**: Done (commit pending)
- **Current**: `internal/httpapi/server.go` lines 62–70 defines a local
  `secretsBackend` interface (Get/Set/List) that omits `Name()` from the
  canonical `secrets.Backend`.
- **Fix**: Use `secrets.Backend` directly in httpapi, or embed it in the local
  interface.
- **What changed**:
  - Removed `secretsBackend` interface from `httpapi/server.go`; `Server.secretsManager`
    now uses `secrets.Backend` directly.
  - Removed `secretBackend` interface from `mcp/server.go`; `NewHandler` and
    `newServer` now accept `secrets.Backend`.
  - Updated `mcp/tools.go` `allTools()` signature to accept `secrets.Backend`.
  - Updated `mcp/secrets_tools_test.go` mock to implement `secrets.Backend`
    (added `Name()` method, uses `secrets.ErrNotFound` instead of custom error).
- **Risk class**: read_only

## C. Security fixes (critical, non-Infisical)

### C1. nomos.hubris.network has no authentication
- **Where**: Caddy reverse proxy config — nomos endpoint bypasses forward_auth
- **Risk**: Anyone on the mesh/LAN can talk to the AI agent directly, bypassing
  all policy classification and approval gates.
- **Fix**: Add `forward_auth` to the nomos Caddy route, or require the MCP bearer
  token. At minimum, add a shared secret via Caddy `basicauth`.
- **Risk class**: config_mutation

### C2. pg_dump failure is silently ignored
- **Where**: `scripts/deploy.sh` — `pg_dump ... || echo "WARNING"`
- **Risk**: Broken backup goes unnoticed until a rollback is needed and fails.
- **Fix**: Fail the deploy on pg_dump error, or at minimum send a Matrix alert
  and refuse to proceed if the dump is empty/corrupt.
- **Risk class**: config_mutation

### C3. CORS defaults to `*`
- **Where**: `internal/httpapi/server.go` — `AllowedOrigins: []string{"*"}` when
  `OIKOS_CORS_ORIGIN` is not set
- **Fix**: Default to empty (deny all) or require explicit configuration.
- **Risk class**: config_mutation

## D. Operational improvements (high)

### D1. Add CI pipeline
- **Current**: No automated build/test on push. `make lint test generate-check`
  exists but is manual.
- **Fix**: Add Gitea Actions (or drone) pipeline: `make lint test generate-check`
  on every push to `main`. Block deploy if pipeline fails.
- **Risk class**: config_mutation

### D2. Version Docker images
- **Current**: All images built as `:latest`. Rollback requires full rebuild.
- **Fix**: Tag images with `v$VERSION` from the VERSION file in deploy.sh. Keep
  last 3 versions. Enable `docker compose up` to pin a version tag.
- **Risk class**: config_mutation

### D3. Add rate limiting
- **Current**: No throttling on HTTP API or MCP endpoints. An agent in a loop
  could hammer the API or exhaust DB connections.
- **Fix**: Add `golang.org/x/time/rate` middleware to chi router. Per-IP or
  per-token rate limit with burst allowance. Separate limits for API vs MCP.
- **Risk class**: config_mutation

### D4. Add container resource limits
- **Current**: No `mem_limit`, `cpus`, or `ulimits` on any compose service.
- **Fix**: Add memory and CPU limits to all services in docker-compose.yml.
  Suggested: API 512MB, scheduler 256MB, notifier 128MB, nomos 512MB.
- **Risk class**: config_mutation

### D5. Add healthchecks to all compose services
- **Current**: Only postgres, api, and redis have healthchecks.
- **Fix**: Add `healthcheck` to scheduler, notifier, and nomos. Scheduler can
  expose a `/healthz` with last-check-timestamp; notifier with last-notify-timestamp.

## E. Code quality (medium)

### E1. Split monolithic files
| File | Lines | Split target |
|------|-------|-------------|
| `internal/mcp/tools.go` | 1774 | `entity_tools.go`, `ops_tools.go`, `knowledge_tools.go`, `analysis_tools.go` |
| `internal/httpapi/impl.go` | 1533 | Handler group files by domain (entities, executions, approvals, metrics, etc.) |
| `cmd/nomos/main.go` | 1127 | `server.go`, `workers.go`, `mcp.go` (partially done — `agent.go` and `store.go` exist) |

### E2. Migrate raw pool.Exec queries to sqlc
- ~50% of DB access in HTTP/MCP handlers bypasses sqlc with raw `pool.Exec`/`pool.QueryRow`.
- Add these queries to `internal/db/queries/` source files for type safety and
  compile-time validation.

### E3. Unify SSH implementations
- Scheduler uses `os/exec ssh` (system binary), MCP/actuator uses `crypto/ssh`.
- Unify on `crypto/ssh` throughout for consistency, testability, and connection
  multiplexing (single TCP connection, multiple sessions).
- Consider a shared SSH pool in `internal/actuator/` used by both scheduler and MCP.

### E4. Fix lifecycle attribute check
- `internal/db/lifecycle.go`: `checkPrecondition` uses `strings.Contains(attrs, want)`
  on raw JSONB text, bypassing the GIN index.
- Parse attributes properly and use `@>` or `?` JSONB operators.

### E5. Add table-driven tests for core logic
Priority packages (currently 0% coverage):
1. `internal/policy` — risk classification rules (table-driven with seed policy.yaml cases)
2. `internal/domain` — lifecycle state machine transitions
3. `internal/scheduler` — check dispatch and signal resolution
4. `internal/ontology` — monitoring resolution and type tree traversal
5. `internal/checkdefaults` — check derivation from monitoring specs

## F. Performance (medium)

### F1. SSH connection pooling for scheduler
- At 30s intervals with 95 entities and multiple check types, the scheduler can
  spawn 100+ SSH sessions per cycle via `os/exec ssh`.
- Migrate to `crypto/ssh` with persistent connection pools to Proxmox hosts.
  One TCP connection per host, multiplexed sessions for individual checks.

### F2. Entity lookup cache
- Repeated `get_entity`/`whoami` MCP calls hit the DB every time.
- Add an in-memory TTL cache (hashicorp/golang-lru, already in go.mod) with
  60s TTL for entity lookups. Invalidate on write.

### F3. Trigram index for entity search
- `ListEntities` uses `ILIKE '%'||q||'%'` which cannot use B-tree indexes.
- Add GIN trigram indexes on `entities.slug` and `entities.name`.
- Alternative: migrate to `tsvector` full-text search matching the knowledge pattern.

### F4. Composite index for auto-act anti-join
- `GetOpenSignalsForAutoAct` joins classifications → signals → executions with
  `WHERE e.entity_id IS NULL`. No composite index on `(classification_id, entity_id)`.
- Add partial index on `executions(classification_id) WHERE entity_id IS NOT NULL`.

## G. Observability (low)

### G1. Add OpenTelemetry tracing
- OTel SDK is already in go.mod as indirect dependency.
- Instrument HTTP handlers, MCP tools, and DB queries with spans.
- Propagate trace context via `correlation_id` (already exists in audit/events).

### G2. Prometheus metrics export
- Expose `/metrics` endpoint for Go runtime, DB pool stats, scheduler check
  duration/counts, HTTP request latency histograms.
- Complement the existing TimescaleDB metric_samples (which are entity health
  metrics, not self-observability).

### G3. Automate offsite backups
- Proton Drive backup target entity exists but no pipeline.
- Add `rclone cron` to `pg_dump | zstd | rclone sync` to Proton Drive.
- Weekly backup verification (restore to test DB, run `make test-db`).

## H. Infrastructure (low)

### H1. Pin Infisical image version
- Currently uses `infisical/infisical:latest`.
- Pin to a specific version tag.

### H2. Add persistent job queue for executions
- All background work is in-process goroutines — lost on restart.
- For the execution pipeline specifically, consider Postgres-backed queue
  (e.g., `river` or custom `pending_executions` poll with advisory lock).
- Lower priority: scheduler and notifier state is transient and self-healing.

### H3. Replace or harden custom migration splitter
- The `splitSQL()` function handles `$$` dollar-quoting but edge cases with
  string literals containing `$$` could break migrations.
- Add test cases for nested quoting, or adopt golang-migrate.

### H4. Add distributed locking for scheduler
- Document single-instance constraint, or add `pg_advisory_lock` (already used
  by migration runner) to prevent duplicate health checks if multiple
  scheduler instances are accidentally started.

---

## Execution order

1. **Phase 0 — Infisical consolidation** (B1–B7): Wire Infisical into all
   binaries, migrate secrets from env/plaintext, activate SOPS fallback, remove
   `.env` secrets and plist HMAC. This is the foundation — every subsequent
   secret-dependent change (SSH host keys in B5, rate limit config, etc.) goes
   through Infisical. **Do this first.**
2. **Phase 1 — Security** (C1–C3, B5): Nomos auth, pg_dump failure, CORS default,
   SSH host keys (now stored in Infisical per B5).
3. **Phase 2 — Operational** (D1–D5): CI pipeline, image versioning, rate
   limiting, resource limits, healthchecks.
4. **Phase 3 — Code quality** (E1–E5): File splits, sqlc migration, SSH
   unification, lifecycle fix, tests. E1–E3 are large refactors — do one
   file/area per commit.
5. **Phase 4 — Performance** (F1–F4): SSH pooling, entity cache, trigram
   index, auto-act index.
6. **Phase 5 — Observability** (G1–G3): OTel tracing, Prometheus, offsite backups.
7. **Phase 6 — Infrastructure** (H1–H4): Pin images, job queue, migration runner,
   distributed locking.

Phase 0 is the gate. Once all secrets flow through Infisical, phases 1–2 can
proceed. Phases 3–4 should wait for CI (D1) so refactors are validated.
Phases 5–6 are backlog.

---

## Phase 0 post-deploy checklist

Run these on mac-mini after deploying the Phase 0 code changes.

### Step 1: Populate Infisical with secrets

For each secret, read the current value from `.env` and store it in Infisical:

```bash
# Values from .env — read them first, then set
oikos secret set matrix/token "$(grep OIKOS_MATRIX_TOKEN .env | cut -d= -f2-)"
oikos secret set approval/hmac-secret "$(grep OIKOS_APPROVAL_HMAC_SECRET .env | cut -d= -f2-)"
oikos secret set mcp/bearer-token "$(grep OIKOS_MCP_BEARER_TOKEN .env | cut -d= -f2-)"
oikos secret set api/token "$(grep OIKOS_API_TOKEN .env | cut -d= -f2-)"
oikos secret set oidc/client-secret "$(grep OIKOS_OIDC_CLIENT_SECRET .env | cut -d= -f2-)"
oikos secret set openrouter/api-key "$(grep OPENROUTER_API_KEY .env | cut -d= -f-)"
oikos secret set webhook/hmac-secret "$(grep WEBHOOK_HMAC_SECRET .env | cut -d= -f2-)"
```

Verify: `oikos secret list` should show all 8 keys.

### Step 2: Pre-populate SSH host keys (optional, skip if TOFU is acceptable)

```bash
# For each Proxmox host, scan and store the public key
for host in pve1 pve2; do
  key=$(ssh-keyscan -t ed25519 $host 2>/dev/null | awk '{print $2" "$3}')
  oikos secret set "ssh/host-keys/$host" "ssh-ed25519 $key"
done
```

Alternatively, skip this step — the first deployment will TOFU-accept all current
host keys and persist them to Infisical automatically.

### Step 3: Remove HMAC secret from webhook plist

On mac-mini:
```bash
sudo launchctl unload ~/Library/LaunchAgents/network.hubris.oikos-deploy-webhook.plist
# Edit the plist: remove the <key>WEBHOOK_HMAC_SECRET</key> block
sudo launchctl load ~/Library/LaunchAgents/network.hubris.oikos-deploy-webhook.plist
```

### Step 4: Strip secrets from .env

Edit `.env` to remove the 7 migrated secrets, keeping only bootstrap and
non-secret config:

```bash
# Remove these lines:
#   INFISICAL_ENCRYPTION_KEY=...
#   OIKOS_MATRIX_TOKEN=...
#   OIKOS_INFISICAL_CLIENT_ID=...
#   OIKOS_INFISICAL_CLIENT_SECRET=...
#   OIKOS_INFISICAL_PROJECT_ID=...
#   OPENROUTER_API_KEY=...
#   OIKOS_MCP_BEARER_TOKEN=...

# Keep these (bootstrap / non-secret):
#   OIKOS_DATABASE_URL=...
#   OIKOS_API_LISTEN=...
#   OIKOS_INFISICAL_SITE_URL=...
#   OIKOS_INFISICAL_ENV=...
```

### Step 5: Verify

1. `oikos secret list` — 8 keys + SSH host keys
2. `docker compose logs api | grep "secrets resolved"` — should show count=5
3. Trigger a test deploy — webhook should still validate HMAC signatures
4. `nomos` should start and resolve secrets from Infisical (check logs for
   `nomos: secrets resolved from Infisical`)
5. Verify no secrets appear in process env: `docker compose exec api env |
   should not show `OIKOS_MCP_BEARER_TOKEN`, `OIKOS_MATRIX_TOKEN`,
   etc. (they come from Infisical at startup, not env)