oikos/plans/2026-08-05-backend-evaluation-improvements.md

2026-08-05 — Backend evaluation: architecture, security, and reliability improvements

Status: In Progress — Phase 0 (B1, B2, B4, B5, B6, B7) and Phase 2 (D1–D5) complete; D1–D5 were hardened across two /review passes (deploy lock, TOCTOU guard, token hygiene, XFF rightmost-hop, ctx-driven sweep). B3 is a post-deploy operational step. Remaining: Phase 1 security (C1–C3), Phase 3 code quality (E1–E5), and Phase 4–6 backlog.

Scope: full evaluation of the oikos backend (Go binaries oikos, nomos, webhook, Postgres/TimescaleDB, Docker deployment, MCP server) excluding frontend clients (web/ SPA and desktop/ Wails app). Began as a research-only pass; Phase 0 (B) and Phase 2 (D) have since been implemented as code changes — see each item's "Status".

Method: four parallel research passes (Go backend structure, database schema, deployment/infrastructure, API/MCP design) plus Infisical secrets audit and dependency analysis of go.mod.

---

A. Summary of findings

The backend is well-architected with strong fundamentals: contract-first API (oapi-codegen), type-safe SQL (sqlc + pgx), TimescaleDB observability, policy- governed autonomy, and a sophisticated ontology-driven data model. The main gaps are secret management (Infisical is wired but barely used — 3 of 4 binaries read secrets from env/plaintext), operational maturity (CI, image tagging, backup reliability), security hardening (SSH host keys, unauthenticated endpoints), and code hygiene (monolithic files, mixed SQL access patterns).

Category	Grade	Notes
Tech stack	A	Go + pgx + sqlc + TimescaleDB + chi + slog — all correct choices
Data model	A-	Dual-entity pattern, temporal relationships, partial indexes, 6 state machines
API design	B+	Contract-first with ~50 MCP tools, RFC 9457 errors; no rate limiting
Secret management	D	Infisical SDK wired but only in API/MCP tools path; nomos, webhook, scheduler, notifier all read plaintext env vars. 7 production secrets in .env, HMAC in world-readable plist. SOPS fallback is dead code.
Security	C	OIDC+static token auth is good, but SSH host keys disabled, unauthenticated nomos endpoint, HMAC secret in world-readable plist
Deployment	C+	Multi-stage builds, pre-deploy backups, but no CI, no versioned images, silent backup failures
Code quality	B	Good error handling, panic safety, doc; but 3 files over 1100 lines, mixed raw/sqlc SQL
Performance	B	Appropriate for scale; SSH check storm risk, no query caching
Observability	B-	TimescaleDB hypertables + SSE + slog; no Prometheus/Grafana, no OTel tracing
Testing	C	make test exists but many core packages (scheduler, domain, actuator, policy) have 0% coverage

B. Infisical consolidation (critical)

Infisical is deployed (Redis + Infisical service in compose, Go SDK in go.mod, internal/secrets/infisical.go fully implemented) but severely underutilized. Only the API server's MCP tools path creates an Infisical backend. Every other binary reads secrets from env vars or plaintext files.

Current wiring map

Binary / role	Uses Infisical?	Secrets read from env/plaintext
oikos api role	Yes (MCP tools only)	OIKOS_DATABASE_URL, OIKOS_MCP_BEARER_TOKEN, OIKOS_API_TOKEN, OIKOS_OIDC_CLIENT_SECRET
oikos scheduler role	No	OIKOS_SSH_KEY_PATH, OIKOS_DATABASE_URL
oikos notifier role	No	OIKOS_MATRIX_TOKEN, OIKOS_APPROVAL_HMAC_SECRET, OIKOS_DATABASE_URL
nomos	No	OPENROUTER_API_KEY, OIKOS_MCP_BEARER_TOKEN, DATABASE_URL
webhook	No	WEBHOOK_HMAC_SECRET (also hardcoded in plist)

B1. Wire Infisical into all binaries at startup

• Goal: Every binary fetches its secrets from Infisical at startup instead of relying on env vars. Bootstrap-only env vars (INFISICAL_CLIENT_ID, INFISICAL_CLIENT_SECRET, INFISICAL_SITE_URL, INFISICAL_PROJECT_ID, OIKOS_DATABASE_URL) remain as env vars (chicken-egg).
• Approach: Add a secrets.InitFromEnv(ctx) call to each binary's main() that creates a secrets.Manager (primary Infisical + SOPS fallback). Store the manager in a package-level var or pass it through the initialization chain.
• Files to change:
  • cmd/oikos/main.go — create Manager in runWithPool, pass to scheduler and notifier runners alongside cfg and pool
  • cmd/nomos/main.go — create Manager at startup, fetch OPENROUTER_API_KEY and OIKOS_MCP_BEARER_TOKEN from Infisical before creating the agent
  • cmd/webhook/main.go — create Manager at startup, fetch WEBHOOK_HMAC_SECRET from Infisical
• Secrets to migrate into Infisical (move from env vars / .env / plist):

  Secret key (in Infisical)	Current source	Used by
  matrix/token	OIKOS_MATRIX_TOKEN env	oikos notifier
  approval/hmac-secret	OIKOS_APPROVAL_HMAC_SECRET env, plist	oikos notifier, webhook
  mcp/bearer-token	OIKOS_MCP_BEARER_TOKEN env	oikos api, nomos
  api/token	OIKOS_API_TOKEN env	oikos api
  oidc/client-secret	OIKOS_OIDC_CLIENT_SECRET env	oikos api
  openrouter/api-key	OPENROUTER_API_KEY env	nomos
  webhook/hmac-secret	WEBHOOK_HMAC_SECRET env + plist	webhook

• Risk class: config_mutation
• Prerequisite: Populate Infisical with these secrets via oikos secret set before deploying the code change. Existing .env values serve as the source of truth for the initial migration.

B2. Activate the SOPS fallback path

• Status: Done (commit pending)
• Current: secrets.NewManager(infisical, sops) is only used in tests. httpapi/server.go creates InfisicalBackend directly — if Infisical is down, there is no fallback.
• Fix: Use secrets.NewManager() in production everywhere so the SOPS DR fallback actually works when Infisical is unreachable. The Manager's cache (5min TTL) already masks transient Infisical blips.
• What changed: httpapi/server.go now creates secrets.NewManager(infisical, sopsFallback) instead of bare secrets.NewInfisicalBackend. SOPS backend is created from cfg.SecretsDir when set.
• Risk class: config_mutation

B3. Remove .env plaintext secrets after migration

• Current: .env contains 7 production secrets in plaintext on mac-mini disk.
• Fix: After B1 is deployed and all binaries read from Infisical, strip secrets from .env leaving only non-secret config (OIKOS_API_LISTEN, OIKOS_SCHEDULER_INTERVAL, etc.). Bootstrap env vars (OIKOS_DATABASE_URL, OIKOS_INFISICAL_*) stay — they're the trust anchor.
• Risk class: config_mutation

B4. Remove HMAC secret from plist

• Status: Done (code change; plist cleanup is post-deploy)
• Current: scripts/deploy/network.hubris.oikos-deploy-webhook.plist line 14 has WEBHOOK_HMAC_SECRET hardcoded in plaintext. World-readable.
• Fix: After B1 (webhook reads from Infisical), remove the EnvironmentVariables WEBHOOK_HMAC_SECRET entry from the plist entirely. The webhook binary will fetch it from Infisical at startup.
• Risk class: config_mutation
• Supersedes: Original plan item B3 (HMAC secret in plist) — same issue, now resolved via Infisical instead of file permissions workarounds.

B5. Store SSH host keys in Infisical

• Status: Done (commit pending)
• Current: ssh.InsecureIgnoreHostKey() in 3 code paths (internal/actuator/ssh.go:160, internal/actuator/actuator.go:427, internal/mcp/server.go).
• Fix: Store Proxmox host public keys in Infisical under ssh/host-keys/{hostname}. Actuator reads them at connection init and builds a knownhosts callback. For dynamic targets, implement TOFU (trust-on-first- use) writing back to Infisical.
• What changed:
  • internal/actuator/hostkeys.go — HostKeyCallback() returns an ssh.HostKeyCallback that verifies against cached keys (MITM detection) and accepts unknown hosts via TOFU, persisting new keys to Infisical.
  • internal/actuator/hostkeys_infisical.go — InfisicalHostKeySource implements HostKeySource over secrets.Backend; ResolveSSHHosts() queries DB for active proxmox-host/standalone-server slugs.
  • internal/actuator/ssh.go:160 — replaced InsecureIgnoreHostKey() with HostKeyCallback().
  • internal/actuator/actuator.go:427 — replaced InsecureIgnoreHostKey() with HostKeyCallback().
  • internal/mcp/server.go:494 — replaced InsecureIgnoreHostKey() with actuator.HostKeyCallback().
  • internal/httpapi/server.go — pre-loads SSH host keys from Infisical at startup (queries active hosts, loads their keys from Infisical).
• Post-deploy step: On first deploy, TOFU will accept all current host keys and store them in Infisical under ssh/host-keys/{slug}. Verify the stored keys are correct by checking oikos secret list. To pre-populate without TOFU, SSH to each Proxmox host and run: ssh-keyscan -t ed25519 {host} | awk '{print $2" "$3}' and store the output via oikos secret set ssh/host-keys/{slug} {output}.
• Risk class: config_mutation (initial pin) / destructive (if keys change)

B6. Fix env var naming inconsistency

• Status: Done (commit pending)
• Current: Config uses OIKOS_INFISICAL_* prefix in docker-compose but internal/secrets/infisical.go lines 58–62 falls back to bare INFISICAL_* (without OIKOS prefix). Two naming conventions for the same bootstrap vars.
• Fix: Standardize on OIKOS_INFISICAL_* everywhere. Remove the bare INFISICAL_* fallback in infisical.go.
• What changed: Removed the os.Getenv("INFISICAL_CLIENT_ID") and os.Getenv("INFISICAL_CLIENT_SECRET") fallbacks in infisical.go connect(). Removed unused os import. Error message updated to reference OIKOS_INFISICAL_* names.
• Risk class: config_mutation

B7. Align secretsBackend interface with secrets.Backend

• Status: Done (commit pending)
• Current: internal/httpapi/server.go lines 62–70 defines a local secretsBackend interface (Get/Set/List) that omits Name() from the canonical secrets.Backend.
• Fix: Use secrets.Backend directly in httpapi, or embed it in the local interface.
• What changed:
  • Removed secretsBackend interface from httpapi/server.go; Server.secretsManager now uses secrets.Backend directly.
  • Removed secretBackend interface from mcp/server.go; NewHandler and newServer now accept secrets.Backend.
  • Updated mcp/tools.go allTools() signature to accept secrets.Backend.
  • Updated mcp/secrets_tools_test.go mock to implement secrets.Backend (added Name() method, uses secrets.ErrNotFound instead of custom error).
• Risk class: read_only

C. Security fixes (critical, non-Infisical)

C1. nomos.hubris.network has no authentication

• Where: Caddy reverse proxy config — nomos endpoint bypasses forward_auth
• Risk: Anyone on the mesh/LAN can talk to the AI agent directly, bypassing all policy classification and approval gates.
• Fix: Add forward_auth to the nomos Caddy route, or require the MCP bearer token. At minimum, add a shared secret via Caddy basicauth.
• Risk class: config_mutation

C2. pg_dump failure is silently ignored

• Where: scripts/deploy.sh — pg_dump ... || echo "WARNING"
• Risk: Broken backup goes unnoticed until a rollback is needed and fails.
• Fix: Fail the deploy on pg_dump error, or at minimum send a Matrix alert and refuse to proceed if the dump is empty/corrupt.
• Risk class: config_mutation

C3. CORS defaults to *

• Where: internal/httpapi/server.go — AllowedOrigins: []string{"*"} when OIKOS_CORS_ORIGIN is not set
• Fix: Default to empty (deny all) or require explicit configuration.
• Risk class: config_mutation

D. Operational improvements (high)

D1. Add CI pipeline

• Status: Done (hardened after review)
• Current: No automated build/test on push. make lint test generate-check exists but is manual.
• Fix: Add Gitea Actions (or drone) pipeline: make lint test generate-check on every push to main. Block deploy if pipeline fails.
• What changed: The Gitea Actions pipeline already exists (.gitea/workflows/ci.yml). Added the missing deploy gate as step [1/8] in scripts/deploy.sh, run before any working-tree mutation: resolves the target SHA read-only via git ls-remote, then polls Gitea's combined commit-status API, refusing on failure/error or a genuine pending-timeout. Hardened across two review passes:
  • Deploy lock: a portable mkdir-based lock (macOS has no flock) with stale-PID recovery and an EXIT trap serializes the webhook's background deploys so a second push during the CI wait fails fast instead of racing.
  • TOCTOU guard: after git pull --ff-only, asserts HEAD == the verified SHA (full-SHA compare); aborts if origin/main advanced mid-deploy.
  • Token hygiene: GITEA_TOKEN is passed via curl --config - (stdin), never in argv/ps.
  • Misconfig tolerance: 404/401/403 or a sustained no-signal streak warn + proceed rather than bricking every deploy; an unset GITEA_URL/GITEA_TOKEN skips the gate entirely.
• Risk class: config_mutation

D2. Version Docker images

• Status: Done
• Current: All images built as :latest. Rollback requires full rebuild.
• Fix: Tag images with v$VERSION from the VERSION file in deploy.sh. Keep last 3 versions. Enable docker compose up to pin a version tag.
• What changed: Every built compose service now carries an image: oikos-<svc>:${OIKOS_VERSION:-latest} tag. deploy.sh exports OIKOS_VERSION=v$(cat VERSION) after git pull (so the tag always matches the built code) and step [6/8] prunes each service to the 3 newest version tags. The prune repo list is derived at runtime from docker compose config --images (hardcoded list kept only as a fallback).
• Risk class: config_mutation

D3. Add rate limiting

• Status: Done
• Current: No throttling on HTTP API or MCP endpoints. An agent in a loop could hammer the API or exhaust DB connections.
• Fix: Add golang.org/x/time/rate middleware to chi router. Per-IP or per-token rate limit with burst allowance. Separate limits for API vs MCP.
• What changed: New internal/httpapi/ratelimit.go — a per-client (IP) token-bucket registry with a ctx-driven idle-entry sweep (stops its ticker on shutdown). Wired into NewHandler before CORS/auth; /healthz is exempt. Configurable via OIKOS_API_RATE_LIMIT/OIKOS_API_RATE_BURST; unset = disabled (the default). Returns RFC 9457 429 + Retry-After. x/time promoted to a direct dependency. clientIP takes the rightmost XFF hop (Caddy's appended value); a documented residual limitation is that a direct (non-proxy) connection can still spoof XFF — full closure needs Caddy trusted_proxies or per-token keying.
• Risk class: config_mutation

D4. Add container resource limits

• Status: Done
• Current: No mem_limit, cpus, or ulimits on any compose service.
• Fix: Add memory and CPU limits to all services in docker-compose.yml. Suggested: API 512MB, scheduler 256MB, notifier 128MB, nomos 512MB.
• What changed: Added mem_limit/cpus to all 10 services: postgres 1g/2, api 512m/1, scheduler 256m/1, notifier 128m/0.5, nomos 512m/1, web 64m/0.25, redis 128m/0.5, infisical 512m/1, migrate/seed 512m/1.
• Risk class: config_mutation

D5. Add healthchecks to all compose services

• Status: Done
• Current: Only postgres, api, and redis have healthchecks.
• Fix: Add healthcheck to scheduler, notifier, and nomos. Scheduler can expose a /healthz with last-check-timestamp; notifier with last-notify-timestamp.
• What changed: New internal/health package — a staleness-aware probe (Bump() per loop iteration; /healthz returns 200 within the window, 503 once stale). Wired into scheduler.Run (:8093, 3× interval) and notifier.Run (:8094, 2 min); nomos already served :8092/healthz. Added compose healthchecks for scheduler, notifier, and nomos. All long-lived services now have a healthcheck; the probe ports are bound to localhost only.
• Risk class: config_mutation

E. Code quality (medium)

E1. Split monolithic files

File	Lines	Split target
internal/mcp/tools.go	1774	entity_tools.go, ops_tools.go, knowledge_tools.go, analysis_tools.go
internal/httpapi/impl.go	1533	Handler group files by domain (entities, executions, approvals, metrics, etc.)
cmd/nomos/main.go	1127	server.go, workers.go, mcp.go (partially done — agent.go and store.go exist)

E2. Migrate raw pool.Exec queries to sqlc

• ~50% of DB access in HTTP/MCP handlers bypasses sqlc with raw pool.Exec/pool.QueryRow.
• Add these queries to internal/db/queries/ source files for type safety and compile-time validation.

E3. Unify SSH implementations

• Scheduler uses os/exec ssh (system binary), MCP/actuator uses crypto/ssh.
• Unify on crypto/ssh throughout for consistency, testability, and connection multiplexing (single TCP connection, multiple sessions).
• Consider a shared SSH pool in internal/actuator/ used by both scheduler and MCP.

E4. Fix lifecycle attribute check

• internal/db/lifecycle.go: checkPrecondition uses strings.Contains(attrs, want) on raw JSONB text, bypassing the GIN index.
• Parse attributes properly and use @> or ? JSONB operators.

E5. Add table-driven tests for core logic

Priority packages (currently 0% coverage):

1. internal/policy — risk classification rules (table-driven with seed policy.yaml cases)
2. internal/domain — lifecycle state machine transitions
3. internal/scheduler — check dispatch and signal resolution
4. internal/ontology — monitoring resolution and type tree traversal
5. internal/checkdefaults — check derivation from monitoring specs

F. Performance (medium)

F1. SSH connection pooling for scheduler

• At 30s intervals with 95 entities and multiple check types, the scheduler can spawn 100+ SSH sessions per cycle via os/exec ssh.
• Migrate to crypto/ssh with persistent connection pools to Proxmox hosts. One TCP connection per host, multiplexed sessions for individual checks.

F2. Entity lookup cache

• Repeated get_entity/whoami MCP calls hit the DB every time.
• Add an in-memory TTL cache (hashicorp/golang-lru, already in go.mod) with 60s TTL for entity lookups. Invalidate on write.

F3. Trigram index for entity search

• ListEntities uses ILIKE '%'||q||'%' which cannot use B-tree indexes.
• Add GIN trigram indexes on entities.slug and entities.name.
• Alternative: migrate to tsvector full-text search matching the knowledge pattern.

F4. Composite index for auto-act anti-join

• GetOpenSignalsForAutoAct joins classifications → signals → executions with WHERE e.entity_id IS NULL. No composite index on (classification_id, entity_id).
• Add partial index on executions(classification_id) WHERE entity_id IS NOT NULL.

G. Observability (low)

G1. Add OpenTelemetry tracing

• OTel SDK is already in go.mod as indirect dependency.
• Instrument HTTP handlers, MCP tools, and DB queries with spans.
• Propagate trace context via correlation_id (already exists in audit/events).

G2. Prometheus metrics export

• Expose /metrics endpoint for Go runtime, DB pool stats, scheduler check duration/counts, HTTP request latency histograms.
• Complement the existing TimescaleDB metric_samples (which are entity health metrics, not self-observability).

G3. Automate offsite backups

• Proton Drive backup target entity exists but no pipeline.
• Add rclone cron to pg_dump | zstd | rclone sync to Proton Drive.
• Weekly backup verification (restore to test DB, run make test-db).

H. Infrastructure (low)

H1. Pin Infisical image version

• Currently uses infisical/infisical:latest.
• Pin to a specific version tag.

H2. Add persistent job queue for executions

• All background work is in-process goroutines — lost on restart.
• For the execution pipeline specifically, consider Postgres-backed queue (e.g., river or custom pending_executions poll with advisory lock).
• Lower priority: scheduler and notifier state is transient and self-healing.

H3. Replace or harden custom migration splitter

• The splitSQL() function handles $$ dollar-quoting but edge cases with string literals containing $$ could break migrations.
• Add test cases for nested quoting, or adopt golang-migrate.

H4. Add distributed locking for scheduler

• Document single-instance constraint, or add pg_advisory_lock (already used by migration runner) to prevent duplicate health checks if multiple scheduler instances are accidentally started.

---

Execution order

1. Phase 0 — Infisical consolidation (B1–B7): Wire Infisical into all binaries, migrate secrets from env/plaintext, activate SOPS fallback, remove .env secrets and plist HMAC. This is the foundation — every subsequent secret-dependent change (SSH host keys in B5, rate limit config, etc.) goes through Infisical. Do this first.
2. Phase 1 — Security (C1–C3, B5): Nomos auth, pg_dump failure, CORS default, SSH host keys (now stored in Infisical per B5).
3. Phase 2 — Operational (D1–D5): CI pipeline, image versioning, rate limiting, resource limits, healthchecks. Done.
4. Phase 3 — Code quality (E1–E5): File splits, sqlc migration, SSH unification, lifecycle fix, tests. E1–E3 are large refactors — do one file/area per commit.
5. Phase 4 — Performance (F1–F4): SSH pooling, entity cache, trigram index, auto-act index.
6. Phase 5 — Observability (G1–G3): OTel tracing, Prometheus, offsite backups.
7. Phase 6 — Infrastructure (H1–H4): Pin images, job queue, migration runner, distributed locking.

Phase 0 is the gate. Once all secrets flow through Infisical, phases 1–2 can proceed. Phases 3–4 should wait for CI (D1) so refactors are validated. Phases 5–6 are backlog.

---

Phase 0 post-deploy checklist

Run these on mac-mini after deploying the Phase 0 code changes.

Step 1: Populate Infisical with secrets

For each secret, read the current value from .env and store it in Infisical:

# Values from .env — read them first, then set
oikos secret set matrix/token "$(grep OIKOS_MATRIX_TOKEN .env | cut -d= -f2-)"
oikos secret set approval/hmac-secret "$(grep OIKOS_APPROVAL_HMAC_SECRET .env | cut -d= -f2-)"
oikos secret set mcp/bearer-token "$(grep OIKOS_MCP_BEARER_TOKEN .env | cut -d= -f2-)"
oikos secret set api/token "$(grep OIKOS_API_TOKEN .env | cut -d= -f2-)"
oikos secret set oidc/client-secret "$(grep OIKOS_OIDC_CLIENT_SECRET .env | cut -d= -f2-)"
oikos secret set openrouter/api-key "$(grep OPENROUTER_API_KEY .env | cut -d= -f-)"
oikos secret set webhook/hmac-secret "$(grep WEBHOOK_HMAC_SECRET .env | cut -d= -f2-)"

Verify: oikos secret list should show all 8 keys.

Step 2: Pre-populate SSH host keys (optional, skip if TOFU is acceptable)

# For each Proxmox host, scan and store the public key
for host in pve1 pve2; do
  key=$(ssh-keyscan -t ed25519 $host 2>/dev/null | awk '{print $2" "$3}')
  oikos secret set "ssh/host-keys/$host" "ssh-ed25519 $key"
done

Alternatively, skip this step — the first deployment will TOFU-accept all current host keys and persist them to Infisical automatically.

Step 3: Remove HMAC secret from webhook plist

On mac-mini:

sudo launchctl unload ~/Library/LaunchAgents/network.hubris.oikos-deploy-webhook.plist
# Edit the plist: remove the <key>WEBHOOK_HMAC_SECRET</key> block
sudo launchctl load ~/Library/LaunchAgents/network.hubris.oikos-deploy-webhook.plist

Step 4: Strip secrets from .env

Edit .env to remove the 7 migrated secrets, keeping only bootstrap and non-secret config:

# Remove these lines:
#   INFISICAL_ENCRYPTION_KEY=...
#   OIKOS_MATRIX_TOKEN=...
#   OIKOS_INFISICAL_CLIENT_ID=...
#   OIKOS_INFISICAL_CLIENT_SECRET=...
#   OIKOS_INFISICAL_PROJECT_ID=...
#   OPENROUTER_API_KEY=...
#   OIKOS_MCP_BEARER_TOKEN=...

# Keep these (bootstrap / non-secret):
#   OIKOS_DATABASE_URL=...
#   OIKOS_API_LISTEN=...
#   OIKOS_INFISICAL_SITE_URL=...
#   OIKOS_INFISICAL_ENV=...

Step 5: Verify

1. oikos secret list — 8 keys + SSH host keys
2. docker compose logs api | grep "secrets resolved" — should show count=5
3. Trigger a test deploy — webhook should still validate HMAC signatures
4. nomos should start and resolve secrets from Infisical (check logs for nomos: secrets resolved from Infisical)
5. Verify no secrets appear in process env: docker compose exec api env | should not show OIKOS_MCP_BEARER_TOKEN, OIKOS_MATRIX_TOKEN`, etc. (they come from Infisical at startup, not env)