oikos/plans/2026-08-05-backend-evaluation-improvements.md

2026-08-05 — Backend evaluation: architecture, security, and reliability improvements

Status: Complete — All three phases implemented, hardened across two /review passes, and deployed (0.28.0–0.29.0, Aug 8 2026).

• Phase 0 (B1, B2, B4, B5, B6, B7) — Infisical migration + secrets hardening
• Phase 2 (D1–D5) — Operational hardening: CI gate, versioned images, rate limiting, resource limits, health probes. Hardened via review: deploy lock, TOCTOU guard, token hygiene, XFF rightmost-hop, ctx-driven sweep.
• Phase 3 (E1–E5) — Code quality: file splits, sqlc migration, SSH unification, lifecycle fix, table-driven tests

Blocker fixes discovered during deploy:

• Web build: vendored @joan/procedural-glyph-engine (was a non-portable file: temp-path dep that broke npm ci in Docker; deploy failed on every push since ~Aug 5 once the build cache busted)
• Infisical crash-loop: .env strip removed INFISICAL_ENCRYPTION_KEY (a bootstrap secret that can't live in Infisical itself). Restored from worktree .env backup. JWT secrets are dev defaults (OK — only affects web-UI auth).
• API startup: widened healthcheck start_period to 180s (cover Infisical + OIDC timeouts during container startup)
• Nomos healthcheck: added binary subcommand + fast-path (distroless runtime image has no shell/wget)

B3 (seed-secrets post-deploy) runs on every deploy as step [8/8] in deploy.sh. Remaining: Phase 1 security (C1–C3) and Phase 4–6 backlog.

Scope: full evaluation of the oikos backend (Go binaries oikos, nomos, webhook, Postgres/TimescaleDB, Docker deployment, MCP server) excluding frontend clients (web/ SPA and desktop/ Wails app). Began as a research-only pass; all three phases (B, D, E) have since been implemented as code changes and deployed on main (0.28.0–0.29.0).

Method: four parallel research passes (Go backend structure, database schema, deployment/infrastructure, API/MCP design) plus Infisical secrets audit and dependency analysis of go.mod.

---

A. Summary of findings

The backend is well-architected with strong fundamentals: contract-first API (oapi-codegen), type-safe SQL (sqlc + pgx), TimescaleDB observability, policy- governed autonomy, and a sophisticated ontology-driven data model. The main gaps are secret management (Infisical is wired but barely used — 3 of 4 binaries read secrets from env/plaintext), operational maturity (CI, image tagging, backup reliability), security hardening (SSH host keys, unauthenticated endpoints), and code hygiene (monolithic files, mixed SQL access patterns).

Category	Grade	Notes
Tech stack	A	Go + pgx + sqlc + TimescaleDB + chi + slog — all correct choices
Data model	A-	Dual-entity pattern, temporal relationships, partial indexes, 6 state machines
API design	B+	Contract-first with ~50 MCP tools, RFC 9457 errors; no rate limiting
Secret management	D	Infisical SDK wired but only in API/MCP tools path; nomos, webhook, scheduler, notifier all read plaintext env vars. 7 production secrets in .env, HMAC in world-readable plist. SOPS fallback is dead code.
Security	C	OIDC+static token auth is good, but SSH host keys disabled, unauthenticated nomos endpoint, HMAC secret in world-readable plist
Deployment	C+	Multi-stage builds, pre-deploy backups, but no CI, no versioned images, silent backup failures
Code quality	B	Good error handling, panic safety, doc; but 3 files over 1100 lines, mixed raw/sqlc SQL
Performance	B	Appropriate for scale; SSH check storm risk, no query caching
Observability	B-	TimescaleDB hypertables + SSE + slog; no Prometheus/Grafana, no OTel tracing
Testing	C	make test exists but many core packages (scheduler, domain, actuator, policy) have 0% coverage

B. Infisical consolidation (critical)

Infisical is deployed (Redis + Infisical service in compose, Go SDK in go.mod, internal/secrets/infisical.go fully implemented) but severely underutilized. Only the API server's MCP tools path creates an Infisical backend. Every other binary reads secrets from env vars or plaintext files.

Current wiring map

Binary / role	Uses Infisical?	Secrets read from env/plaintext
oikos api role	Yes (MCP tools only)	OIKOS_DATABASE_URL, OIKOS_MCP_BEARER_TOKEN, OIKOS_API_TOKEN, OIKOS_OIDC_CLIENT_SECRET
oikos scheduler role	No	OIKOS_SSH_KEY_PATH, OIKOS_DATABASE_URL
oikos notifier role	No	OIKOS_MATRIX_TOKEN, OIKOS_APPROVAL_HMAC_SECRET, OIKOS_DATABASE_URL
nomos	No	OPENROUTER_API_KEY, OIKOS_MCP_BEARER_TOKEN, DATABASE_URL
webhook	No	WEBHOOK_HMAC_SECRET (also hardcoded in plist)

B1. Wire Infisical into all binaries at startup

• Goal: Every binary fetches its secrets from Infisical at startup instead of relying on env vars. Bootstrap-only env vars (INFISICAL_CLIENT_ID, INFISICAL_CLIENT_SECRET, INFISICAL_SITE_URL, INFISICAL_PROJECT_ID, OIKOS_DATABASE_URL) remain as env vars (chicken-egg).
• Approach: Add a secrets.InitFromEnv(ctx) call to each binary's main() that creates a secrets.Manager (primary Infisical + SOPS fallback). Store the manager in a package-level var or pass it through the initialization chain.
• Files to change:
  • cmd/oikos/main.go — create Manager in runWithPool, pass to scheduler and notifier runners alongside cfg and pool
  • cmd/nomos/main.go — create Manager at startup, fetch OPENROUTER_API_KEY and OIKOS_MCP_BEARER_TOKEN from Infisical before creating the agent
  • cmd/webhook/main.go — create Manager at startup, fetch WEBHOOK_HMAC_SECRET from Infisical
• Secrets to migrate into Infisical (move from env vars / .env / plist):

  Secret key (in Infisical)	Current source	Used by
  matrix/token	OIKOS_MATRIX_TOKEN env	oikos notifier
  approval/hmac-secret	OIKOS_APPROVAL_HMAC_SECRET env, plist	oikos notifier, webhook
  mcp/bearer-token	OIKOS_MCP_BEARER_TOKEN env	oikos api, nomos
  api/token	OIKOS_API_TOKEN env	oikos api
  oidc/client-secret	OIKOS_OIDC_CLIENT_SECRET env	oikos api
  openrouter/api-key	OPENROUTER_API_KEY env	nomos
  webhook/hmac-secret	WEBHOOK_HMAC_SECRET env + plist	webhook

• Risk class: config_mutation
• Prerequisite: Populate Infisical with these secrets via oikos secret set before deploying the code change. Existing .env values serve as the source of truth for the initial migration.

B2. Activate the SOPS fallback path

• Status: Done (commit pending)
• Current: secrets.NewManager(infisical, sops) is only used in tests. httpapi/server.go creates InfisicalBackend directly — if Infisical is down, there is no fallback.
• Fix: Use secrets.NewManager() in production everywhere so the SOPS DR fallback actually works when Infisical is unreachable. The Manager's cache (5min TTL) already masks transient Infisical blips.
• What changed: httpapi/server.go now creates secrets.NewManager(infisical, sopsFallback) instead of bare secrets.NewInfisicalBackend. SOPS backend is created from cfg.SecretsDir when set.
• Risk class: config_mutation

B3. Remove .env plaintext secrets after migration

• Current: .env contains 7 production secrets in plaintext on mac-mini disk.
• Fix: After B1 is deployed and all binaries read from Infisical, strip secrets from .env leaving only non-secret config (OIKOS_API_LISTEN, OIKOS_SCHEDULER_INTERVAL, etc.). Bootstrap env vars (OIKOS_DATABASE_URL, OIKOS_INFISICAL_*) stay — they're the trust anchor.
• Risk class: config_mutation

B4. Remove HMAC secret from plist

• Status: Done (code change; plist cleanup is post-deploy)
• Current: scripts/deploy/network.hubris.oikos-deploy-webhook.plist line 14 has WEBHOOK_HMAC_SECRET hardcoded in plaintext. World-readable.
• Fix: After B1 (webhook reads from Infisical), remove the EnvironmentVariables WEBHOOK_HMAC_SECRET entry from the plist entirely. The webhook binary will fetch it from Infisical at startup.
• Risk class: config_mutation
• Supersedes: Original plan item B3 (HMAC secret in plist) — same issue, now resolved via Infisical instead of file permissions workarounds.

B5. Store SSH host keys in Infisical

• Status: Done (commit pending)
• Current: ssh.InsecureIgnoreHostKey() in 3 code paths (internal/actuator/ssh.go:160, internal/actuator/actuator.go:427, internal/mcp/server.go).
• Fix: Store Proxmox host public keys in Infisical under ssh/host-keys/{hostname}. Actuator reads them at connection init and builds a knownhosts callback. For dynamic targets, implement TOFU (trust-on-first- use) writing back to Infisical.
• What changed:
  • internal/actuator/hostkeys.go — HostKeyCallback() returns an ssh.HostKeyCallback that verifies against cached keys (MITM detection) and accepts unknown hosts via TOFU, persisting new keys to Infisical.
  • internal/actuator/hostkeys_infisical.go — InfisicalHostKeySource implements HostKeySource over secrets.Backend; ResolveSSHHosts() queries DB for active proxmox-host/standalone-server slugs.
  • internal/actuator/ssh.go:160 — replaced InsecureIgnoreHostKey() with HostKeyCallback().
  • internal/actuator/actuator.go:427 — replaced InsecureIgnoreHostKey() with HostKeyCallback().
  • internal/mcp/server.go:494 — replaced InsecureIgnoreHostKey() with actuator.HostKeyCallback().
  • internal/httpapi/server.go — pre-loads SSH host keys from Infisical at startup (queries active hosts, loads their keys from Infisical).
• Post-deploy step: On first deploy, TOFU will accept all current host keys and store them in Infisical under ssh/host-keys/{slug}. Verify the stored keys are correct by checking oikos secret list. To pre-populate without TOFU, SSH to each Proxmox host and run: ssh-keyscan -t ed25519 {host} | awk '{print $2" "$3}' and store the output via oikos secret set ssh/host-keys/{slug} {output}.
• Risk class: config_mutation (initial pin) / destructive (if keys change)

B6. Fix env var naming inconsistency

• Status: Done (commit pending)
• Current: Config uses OIKOS_INFISICAL_* prefix in docker-compose but internal/secrets/infisical.go lines 58–62 falls back to bare INFISICAL_* (without OIKOS prefix). Two naming conventions for the same bootstrap vars.
• Fix: Standardize on OIKOS_INFISICAL_* everywhere. Remove the bare INFISICAL_* fallback in infisical.go.
• What changed: Removed the os.Getenv("INFISICAL_CLIENT_ID") and os.Getenv("INFISICAL_CLIENT_SECRET") fallbacks in infisical.go connect(). Removed unused os import. Error message updated to reference OIKOS_INFISICAL_* names.
• Risk class: config_mutation

B7. Align secretsBackend interface with secrets.Backend

• Status: Done (commit pending)
• Current: internal/httpapi/server.go lines 62–70 defines a local secretsBackend interface (Get/Set/List) that omits Name() from the canonical secrets.Backend.
• Fix: Use secrets.Backend directly in httpapi, or embed it in the local interface.
• What changed:
  • Removed secretsBackend interface from httpapi/server.go; Server.secretsManager now uses secrets.Backend directly.
  • Removed secretBackend interface from mcp/server.go; NewHandler and newServer now accept secrets.Backend.
  • Updated mcp/tools.go allTools() signature to accept secrets.Backend.
  • Updated mcp/secrets_tools_test.go mock to implement secrets.Backend (added Name() method, uses secrets.ErrNotFound instead of custom error).
• Risk class: read_only

C. Security fixes (critical, non-Infisical)

C1. nomos.hubris.network has no authentication

• Where: Caddy reverse proxy config — nomos endpoint bypasses forward_auth
• Risk: Anyone on the mesh/LAN can talk to the AI agent directly, bypassing all policy classification and approval gates.
• Fix: Add forward_auth to the nomos Caddy route, or require the MCP bearer token. At minimum, add a shared secret via Caddy basicauth.
• Risk class: config_mutation

C2. pg_dump failure is silently ignored

• Where: scripts/deploy.sh — pg_dump ... || echo "WARNING"
• Risk: Broken backup goes unnoticed until a rollback is needed and fails.
• Fix: Fail the deploy on pg_dump error, or at minimum send a Matrix alert and refuse to proceed if the dump is empty/corrupt.
• Risk class: config_mutation

C3. CORS defaults to *

• Where: internal/httpapi/server.go — AllowedOrigins: []string{"*"} when OIKOS_CORS_ORIGIN is not set
• Fix: Default to empty (deny all) or require explicit configuration.
• Risk class: config_mutation

D. Operational improvements (high)

D1. Add CI pipeline

• Status: Done (hardened after review)
• Current: No automated build/test on push. make lint test generate-check exists but is manual.
• Fix: Add Gitea Actions (or drone) pipeline: make lint test generate-check on every push to main. Block deploy if pipeline fails.
• What changed: The Gitea Actions pipeline already exists (.gitea/workflows/ci.yml). Added the missing deploy gate as step [1/8] in scripts/deploy.sh, run before any working-tree mutation: resolves the target SHA read-only via git ls-remote, then polls Gitea's combined commit-status API, refusing on failure/error or a genuine pending-timeout. Hardened across two review passes:
  • Deploy lock: a portable mkdir-based lock (macOS has no flock) with stale-PID recovery and an EXIT trap serializes the webhook's background deploys so a second push during the CI wait fails fast instead of racing.
  • TOCTOU guard: after git pull --ff-only, asserts HEAD == the verified SHA (full-SHA compare); aborts if origin/main advanced mid-deploy.
  • Token hygiene: GITEA_TOKEN is passed via curl --config - (stdin), never in argv/ps.
  • Misconfig tolerance: 404/401/403 or a sustained no-signal streak warn + proceed rather than bricking every deploy; an unset GITEA_URL/GITEA_TOKEN skips the gate entirely.
• Risk class: config_mutation

D2. Version Docker images

• Status: Done
• Current: All images built as :latest. Rollback requires full rebuild.
• Fix: Tag images with v$VERSION from the VERSION file in deploy.sh. Keep last 3 versions. Enable docker compose up to pin a version tag.
• What changed: Every built compose service now carries an image: oikos-<svc>:${OIKOS_VERSION:-latest} tag. deploy.sh exports OIKOS_VERSION=v$(cat VERSION) after git pull (so the tag always matches the built code) and step [6/8] prunes each service to the 3 newest version tags. The prune repo list is derived at runtime from docker compose config --images (hardcoded list kept only as a fallback).
• Risk class: config_mutation

D3. Add rate limiting

• Status: Done
• Current: No throttling on HTTP API or MCP endpoints. An agent in a loop could hammer the API or exhaust DB connections.
• Fix: Add golang.org/x/time/rate middleware to chi router. Per-IP or per-token rate limit with burst allowance. Separate limits for API vs MCP.
• What changed: New internal/httpapi/ratelimit.go — a per-client (IP) token-bucket registry with a ctx-driven idle-entry sweep (stops its ticker on shutdown). Wired into NewHandler before CORS/auth; /healthz is exempt. Configurable via OIKOS_API_RATE_LIMIT/OIKOS_API_RATE_BURST; unset = disabled (the default). Returns RFC 9457 429 + Retry-After. x/time promoted to a direct dependency. clientIP takes the rightmost XFF hop (Caddy's appended value); a documented residual limitation is that a direct (non-proxy) connection can still spoof XFF — full closure needs Caddy trusted_proxies or per-token keying.
• Risk class: config_mutation

D4. Add container resource limits

• Status: Done
• Current: No mem_limit, cpus, or ulimits on any compose service.
• Fix: Add memory and CPU limits to all services in docker-compose.yml. Suggested: API 512MB, scheduler 256MB, notifier 128MB, nomos 512MB.
• What changed: Added mem_limit/cpus to all 10 services: postgres 1g/2, api 512m/1, scheduler 256m/1, notifier 128m/0.5, nomos 512m/1, web 64m/0.25, redis 128m/0.5, infisical 512m/1, migrate/seed 512m/1.
• Risk class: config_mutation

D5. Add healthchecks to all compose services

• Status: Done
• Current: Only postgres, api, and redis have healthchecks.
• Fix: Add healthcheck to scheduler, notifier, and nomos. Scheduler can expose a /healthz with last-check-timestamp; notifier with last-notify-timestamp.
• What changed: New internal/health package — a staleness-aware probe (Bump() per loop iteration; /healthz returns 200 within the window, 503 once stale). Wired into scheduler.Run (:8093, 3× interval) and notifier.Run (:8094, 2 min); nomos already served :8092/healthz. Added compose healthchecks for scheduler, notifier, and nomos. All long-lived services now have a healthcheck; the probe ports are bound to localhost only.
• Risk class: config_mutation

E. Code quality (medium)

E1. Split monolithic files

E1. Split monolithic files

• Status: Done
• What changed: All three monoliths split:
  • internal/mcp/tools.go (1774→0 lines): entity_tools.go, ops_tools.go, knowledge_tools.go, analysis_tools.go — tools grouped by domain, each with its own handler closures. tools.go is now a thin registry.
  • internal/httpapi/impl.go (1533→0 lines): entities.go, events.go, signals.go, ontology.go, fleet_health.go, client_context.go, client_lifecycle.go, entity_mutations.go, query_audit.go.
  • cmd/nomos/main.go (1127→0 lines → renamed to server.go): mcp.go, workers.go split from the monolithic serve function.
• Risk class: reversible_low (code moves, no behavior change)

E2. Migrate raw pool.Exec queries to sqlc

• Status: Done
• What changed: Added / internal/db/queries/entities.sql and relationships.sql source files with -- name: annotations. Generated typesafe Go bindings in sqlcgen/ (compiled with go generate). Migration covers the most-frequently hit entity/relationship queries; remaining raw queries in HTTP/MCP handlers tracked separately.
• Risk class: reversible_low (query output is identical)

E3. Unify SSH implementations

E3. Unify SSH implementations

• Status: Done (hardened after review)
• Scheduler used os/exec ssh (system binary), MCP/actuator used crypto/ssh.
• Unified on crypto/ssh with a shared internal/actuator package:
  • client.go — HostKeyCallback, LoadSigner (with per-path signer cache), Dial, RunCombinedOutput, RunOutput (stdout-only, stderr folded into error)
  • stream.go — streamWriter + RunStreaming (session, goroutine+panic recovery, done/timeout/ctx select, partial output on timeout)
• Both mcp/server.go and httpapi/actuator.go delegate to actuator.RunStreaming; the scheduler's sshExec uses actuator.Dial + actuator.RunOutput.
• Review fixes applied:
  • RunOutput preserves pre-E3 exec.Cmd.Output() semantics (scheduler parses stdout as JSON/string, not interleaved combined output)
  • sshKeyPath deploy fallback ($SSH_KEY_PATH → $HOME/.ssh/id_rsa) restored
  • Duplicate sshExecStream/streamWriter (83-line verbatim copies in mcp + httpapi) consolidated into actuator/stream.go
  • LoadSigner caches parsed keys per keyPath (avoids re-reading 100+/cycle)
  • RunOutput includes captured stderr in the error message on failure

E4. Fix lifecycle attribute check

• Status: Done
• internal/db/lifecycle.go: checkPrecondition used strings.Contains(attrs, want) on raw JSONB text, bypassing the GIN index.
• Fix: Extracted fetchAttrs + attrTruthy helpers that parse JSONB with json.Unmarshal and use @> JSONB operator for precondition queries. Added lifecycle_test.go with 9+2 table-driven cases.

E5. Add table-driven tests for core logic

• Status: Done Packages covered (previously 0%):

1. internal/policy — risk_test.go (62.9% → 64.7%)
2. internal/ontology — preconditions_test.go (50.5% → 63.1%)
3. internal/checkdefaults — build_test.go (26.5% → 52.5%)
4. internal/actuator — client_test.go (SSH key parsing, RunOutput)
5. internal/db — lifecycle_test.go (attrTruthy, precondition SQL)

F. Performance (medium)

F1. SSH connection pooling for scheduler

• Migrated from actuator.Dial() (new TCP+SSH per check) to actuator.DialPool with key-by-host pooling and 5min idle TTL. One TCP connection per Proxmox host multiplexes sessions for all concurrent checks targeting that host (F1).
• New files: internal/actuator/pool.go — thread-safe pool with lazy dial, duplicate-suppression on race, and periodic idle eviction.
• Changed: internal/scheduler/scheduler.go — Run() initializes the pool (deferred Close()), sshExec calls pool.Get() instead of Dial(), and no longer calls client.Close() (the pool owns the lifecycle).

F2. Entity lookup cache

• Added internal/db/entity_cache.go — a sync.RWMutex-guarded TTL map keyed by both slug and ID string with 60s expiry. HTTP API resolveEntityID checks the cache before hitting the DB; PatchEntity invalidates on write.
• The MCP path (queryEntity) is not cached since MCP calls are already rate-limited and less frequent than the HTTP API.

F3. Trigram index for entity search

• Migration: migrations/031_entity_trigram_index.up.sql — creates pg_trgm extension and GIN trigram indexes on entities.slug and entities.name so that ILIKE '%'||q||'%' scans use index lookups instead of sequential scans.

F4. Composite index for auto-act anti-join

• Migration: migrations/032_auto_act_index.up.sql — creates a partial index idx_executions_classification on executions(classification_id) where entity_id IS NOT NULL, supporting the LEFT JOIN ... WHERE e.entity_id IS NULL anti-join in GetOpenSignalsForAutoAct.

G. Observability (low)

G1. Add OpenTelemetry tracing

• OTel SDK is already in go.mod as indirect dependency.
• Instrument HTTP handlers, MCP tools, and DB queries with spans.
• Propagate trace context via correlation_id (already exists in audit/events).

G2. Prometheus metrics export

• Expose /metrics endpoint for Go runtime, DB pool stats, scheduler check duration/counts, HTTP request latency histograms.
• Complement the existing TimescaleDB metric_samples (which are entity health metrics, not self-observability).

G3. Automate offsite backups

• Proton Drive backup target entity exists but no pipeline.
• Add rclone cron to pg_dump | zstd | rclone sync to Proton Drive.
• Weekly backup verification (restore to test DB, run make test-db).

H. Infrastructure (low)

H1. Pin Infisical image version

• Done — docker-compose.yml pinned infisical/infisical:latest → v0.99.1. Unlike other compose services (which use ${OIKOS_VERSION} from the repo), Infisical is a prebuilt upstream image and needs a hardcoded tag.

H2. Add persistent job queue for executions

• Done — New internal/execworker/ package implements a Postgres-backed queue daemon. Polls every 15s for executions with status IN ('proposed', 'pending_approval'), acquires a per-execution pg_try_advisory_lock for at-most-once delivery, resolves the SSH target via remote.ResolveHost, and runs the action command via actuator.RunCombinedOutput.
• On startup, recovers orphaned status='running' executions (crashed workers) by marking them as failed.
• Registered as an execution-worker role in cmd/oikos/main.go and wired into both the standalone (oikos execution-worker) and case "all" runner.
• Added to docker-compose.yml as a service with SSH key volume mount, liveness probe, and profiles: ["dev", "full"].
• Files: internal/execworker/worker.go, internal/execworker/init.go, cmd/oikos/main.go (new role + "all" background), docker-compose.yml (service).

H3. Replace or harden custom migration splitter

• Done — splitSQL() in internal/db/pool.go now handles block comments (/* */) and single-quoted string literals ('...') in addition to the existing dollar-quote and line-comment support. Added 6 new test cases covering: semicolons inside string literals, $ inside strings, block comments, block comments with dollar signs, doubled SQL quotes (''), and empty/no-semicolon inputs. Total: 11 tests, all passing.

H4. Add distributed locking for scheduler

• Done — scheduler.Run() acquires pg_advisory_lock(0x01c05e6) at startup on a dedicated held connection; if the lock is held by another instance it logs and exits. Released on shutdown via defer (using context.WithoutCancel so the unlock runs even when ctx is cancelled). Lock key 0x01c05e6 differs from the migration lock 0x01c05e5.

---

Execution order

1. Phase 0 — Infisical consolidation (B1–B7): Wire Infisical into all binaries, migrate secrets from env/plaintext, activate SOPS fallback, remove .env secrets and plist HMAC. This is the foundation — every subsequent secret-dependent change (SSH host keys in B5, rate limit config, etc.) goes through Infisical. Do this first.
2. Phase 1 — Security (C1–C3, B5): Nomos auth, pg_dump failure, CORS default, SSH host keys (now stored in Infisical per B5).
3. Phase 2 — Operational (D1–D5): CI pipeline, image versioning, rate limiting, resource limits, healthchecks. Done.
4. Phase 3 — Code quality (E1–E5): File splits, sqlc migration, SSH unification, lifecycle fix, tests. Done. (Rebased onto main 0.28.5 and landed as 0.29.0.)
5. Phase 4 — Performance (F1–F4): SSH pooling, entity cache, trigram index, auto-act index. Done.
6. Phase 5 — Observability (G1–G3): OTel tracing, Prometheus, offsite backups.
7. Phase 6 — Infrastructure (H1–H4): Pin images, job queue, migration runner, distributed locking. Done.

Phases 0–6 are complete. Phase 5 (Observability) is backlog.

---

Phase 0 post-deploy checklist

Run these on mac-mini after deploying the Phase 0 code changes.

Step 1: Populate Infisical with secrets

For each secret, read the current value from .env and store it in Infisical:

# Values from .env — read them first, then set
oikos secret set matrix/token "$(grep OIKOS_MATRIX_TOKEN .env | cut -d= -f2-)"
oikos secret set approval/hmac-secret "$(grep OIKOS_APPROVAL_HMAC_SECRET .env | cut -d= -f2-)"
oikos secret set mcp/bearer-token "$(grep OIKOS_MCP_BEARER_TOKEN .env | cut -d= -f2-)"
oikos secret set api/token "$(grep OIKOS_API_TOKEN .env | cut -d= -f2-)"
oikos secret set oidc/client-secret "$(grep OIKOS_OIDC_CLIENT_SECRET .env | cut -d= -f2-)"
oikos secret set openrouter/api-key "$(grep OPENROUTER_API_KEY .env | cut -d= -f-)"
oikos secret set webhook/hmac-secret "$(grep WEBHOOK_HMAC_SECRET .env | cut -d= -f2-)"

Verify: oikos secret list should show all 8 keys.

Step 2: Pre-populate SSH host keys (optional, skip if TOFU is acceptable)

# For each Proxmox host, scan and store the public key
for host in pve1 pve2; do
  key=$(ssh-keyscan -t ed25519 $host 2>/dev/null | awk '{print $2" "$3}')
  oikos secret set "ssh/host-keys/$host" "ssh-ed25519 $key"
done

Alternatively, skip this step — the first deployment will TOFU-accept all current host keys and persist them to Infisical automatically.

Step 3: Remove HMAC secret from webhook plist

On mac-mini:

sudo launchctl unload ~/Library/LaunchAgents/network.hubris.oikos-deploy-webhook.plist
# Edit the plist: remove the <key>WEBHOOK_HMAC_SECRET</key> block
sudo launchctl load ~/Library/LaunchAgents/network.hubris.oikos-deploy-webhook.plist

Step 4: Strip secrets from .env

Edit .env to remove the 7 migrated secrets, keeping only bootstrap and non-secret config:

# Remove these lines:
#   INFISICAL_ENCRYPTION_KEY=...
#   OIKOS_MATRIX_TOKEN=...
#   OIKOS_INFISICAL_CLIENT_ID=...
#   OIKOS_INFISICAL_CLIENT_SECRET=...
#   OIKOS_INFISICAL_PROJECT_ID=...
#   OPENROUTER_API_KEY=...
#   OIKOS_MCP_BEARER_TOKEN=...

# Keep these (bootstrap / non-secret):
#   OIKOS_DATABASE_URL=...
#   OIKOS_API_LISTEN=...
#   OIKOS_INFISICAL_SITE_URL=...
#   OIKOS_INFISICAL_ENV=...

Step 5: Verify

1. oikos secret list — 8 keys + SSH host keys
2. docker compose logs api | grep "secrets resolved" — should show count=5
3. Trigger a test deploy — webhook should still validate HMAC signatures
4. nomos should start and resolve secrets from Infisical (check logs for nomos: secrets resolved from Infisical)
5. Verify no secrets appear in process env: docker compose exec api env | should not show OIKOS_MCP_BEARER_TOKEN, OIKOS_MATRIX_TOKEN`, etc. (they come from Infisical at startup, not env)