plans: review and archive all plans to done/

Architecture has changed drastically (hexagonal refactor, web client extraction). Every active plan has been reviewed, annotated with 'Completed' or 'Won't do' status, and moved to plans/done/. Completed (7): gaps-and-improvements, liveness-drift, gated-execution, nomos-code-review, codebase-cleanup, mascot-physics, backend-eval Won't do (7): prometheus-lxc, control-room-webui, activity-gaps, activity-timeline, frontend-os-apps, haos-capability-gaps, arr-audit
2026-08-16 11:18:47 +02:00
parent 7a7d390718
commit 104593de0c
16 changed files with 47 additions and 18 deletions
--- a/plans/done/2026-08-05-backend-evaluation-improvements.md
+++ b/plans/done/2026-08-05-backend-evaluation-improvements.md
@@ -0,0 +1,564 @@
+# 2026-08-05 — Backend evaluation: architecture, security, and reliability improvements
+
+**Reviewed 2026-08-16 — Status: Completed** — Phases 0, 2, 3, 4, 6 implemented and deployed (0.28.0–0.29.0); Phase 1 security and Phase 5 observability remain backlog.
+
+Status: **Complete** — All three phases implemented, hardened across two `/review`
+passes, and deployed (0.28.0–0.29.0, Aug 8 2026).
+
+- **Phase 0** (B1, B2, B4, B5, B6, B7) — Infisical migration + secrets hardening
+- **Phase 2** (D1–D5) — Operational hardening: CI gate, versioned images, rate
+  limiting, resource limits, health probes. Hardened via review: deploy lock,
+  TOCTOU guard, token hygiene, XFF rightmost-hop, ctx-driven sweep.
+- **Phase 3** (E1–E5) — Code quality: file splits, sqlc migration, SSH
+  unification, lifecycle fix, table-driven tests
+
+**Blocker fixes discovered during deploy:**
+- Web build: vendored `@joan/procedural-glyph-engine` (was a non-portable `file:`
+  temp-path dep that broke `npm ci` in Docker; deploy failed on every push since
+  ~Aug 5 once the build cache busted)
+- Infisical crash-loop: `.env` strip removed `INFISICAL_ENCRYPTION_KEY` (a
+  bootstrap secret that can't live in Infisical itself). Restored from worktree
+  `.env` backup. JWT secrets are dev defaults (OK — only affects web-UI auth).
+- API startup: widened healthcheck `start_period` to 180s (cover Infisical +
+  OIDC timeouts during container startup)
+- Nomos healthcheck: added binary subcommand + fast-path (distroless runtime
+  image has no shell/wget)
+
+B3 (seed-secrets post-deploy) runs on every deploy as step [8/8] in deploy.sh.
+Remaining: Phase 1 security (C1–C3) and Phase 4–6 backlog.
+
+Scope: full evaluation of the oikos backend (Go binaries `oikos`, `nomos`, `webhook`,
+Postgres/TimescaleDB, Docker deployment, MCP server) excluding frontend clients
+(`web/` SPA and `desktop/` Wails app). Began as a research-only pass; all three
+phases (B, D, E) have since been implemented as code changes and deployed on main
+(0.28.0–0.29.0).
+
+Method: four parallel research passes (Go backend structure, database schema,
+deployment/infrastructure, API/MCP design) plus Infisical secrets audit and
+dependency analysis of `go.mod`.
+
+---
+
+## A. Summary of findings
+
+The backend is well-architected with strong fundamentals: contract-first API
+(oapi-codegen), type-safe SQL (sqlc + pgx), TimescaleDB observability, policy-
+governed autonomy, and a sophisticated ontology-driven data model. The main gaps
+are secret management (Infisical is wired but barely used — 3 of 4 binaries
+read secrets from env/plaintext), operational maturity (CI, image tagging, backup
+reliability), security hardening (SSH host keys, unauthenticated endpoints), and
+code hygiene (monolithic files, mixed SQL access patterns).
+
+| Category | Grade | Notes |
+|----------|-------|-------|
+| Tech stack | A | Go + pgx + sqlc + TimescaleDB + chi + slog — all correct choices |
+| Data model | A- | Dual-entity pattern, temporal relationships, partial indexes, 6 state machines |
+| API design | B+ | Contract-first with ~50 MCP tools, RFC 9457 errors; no rate limiting |
+| Secret management | **D** | Infisical SDK wired but only in API/MCP tools path; nomos, webhook, scheduler, notifier all read plaintext env vars. 7 production secrets in `.env`, HMAC in world-readable plist. SOPS fallback is dead code. |
+| Security | C | OIDC+static token auth is good, but SSH host keys disabled, unauthenticated nomos endpoint, HMAC secret in world-readable plist |
+| Deployment | C+ | Multi-stage builds, pre-deploy backups, but no CI, no versioned images, silent backup failures |
+| Code quality | B | Good error handling, panic safety, doc; but 3 files over 1100 lines, mixed raw/sqlc SQL |
+| Performance | B | Appropriate for scale; SSH check storm risk, no query caching |
+| Observability | B- | TimescaleDB hypertables + SSE + slog; no Prometheus/Grafana, no OTel tracing |
+| Testing | C | `make test` exists but many core packages (scheduler, domain, actuator, policy) have 0% coverage |
+
+## B. Infisical consolidation (critical)
+
+Infisical is deployed (Redis + Infisical service in compose, Go SDK in go.mod,
+`internal/secrets/infisical.go` fully implemented) but severely underutilized.
+Only the API server's MCP tools path creates an Infisical backend. Every other
+binary reads secrets from env vars or plaintext files.
+
+### Current wiring map
+
+| Binary / role | Uses Infisical? | Secrets read from env/plaintext |
+|---------------|----------------|-------------------------------|
+| oikos `api` role | Yes (MCP tools only) | `OIKOS_DATABASE_URL`, `OIKOS_MCP_BEARER_TOKEN`, `OIKOS_API_TOKEN`, `OIKOS_OIDC_CLIENT_SECRET` |
+| oikos `scheduler` role | **No** | `OIKOS_SSH_KEY_PATH`, `OIKOS_DATABASE_URL` |
+| oikos `notifier` role | **No** | `OIKOS_MATRIX_TOKEN`, `OIKOS_APPROVAL_HMAC_SECRET`, `OIKOS_DATABASE_URL` |
+| nomos | **No** | `OPENROUTER_API_KEY`, `OIKOS_MCP_BEARER_TOKEN`, `DATABASE_URL` |
+| webhook | **No** | `WEBHOOK_HMAC_SECRET` (also hardcoded in plist) |
+
+### B1. Wire Infisical into all binaries at startup
+
+- **Goal**: Every binary fetches its secrets from Infisical at startup instead of
+  relying on env vars. Bootstrap-only env vars (`INFISICAL_CLIENT_ID`,
+  `INFISICAL_CLIENT_SECRET`, `INFISICAL_SITE_URL`, `INFISICAL_PROJECT_ID`,
+  `OIKOS_DATABASE_URL`) remain as env vars (chicken-egg).
+- **Approach**: Add a `secrets.InitFromEnv(ctx)` call to each binary's `main()` that
+  creates a `secrets.Manager` (primary Infisical + SOPS fallback). Store the
+  manager in a package-level var or pass it through the initialization chain.
+- **Files to change**:
+  - `cmd/oikos/main.go` — create Manager in `runWithPool`, pass to scheduler and
+    notifier runners alongside cfg and pool
+  - `cmd/nomos/main.go` — create Manager at startup, fetch `OPENROUTER_API_KEY`
+    and `OIKOS_MCP_BEARER_TOKEN` from Infisical before creating the agent
+  - `cmd/webhook/main.go` — create Manager at startup, fetch `WEBHOOK_HMAC_SECRET`
+    from Infisical
+- **Secrets to migrate into Infisical** (move from env vars / `.env` / plist):
+
+  | Secret key (in Infisical) | Current source | Used by |
+  |---------------------------|---------------|---------|
+  | `matrix/token` | `OIKOS_MATRIX_TOKEN` env | oikos notifier |
+  | `approval/hmac-secret` | `OIKOS_APPROVAL_HMAC_SECRET` env, plist | oikos notifier, webhook |
+  | `mcp/bearer-token` | `OIKOS_MCP_BEARER_TOKEN` env | oikos api, nomos |
+  | `api/token` | `OIKOS_API_TOKEN` env | oikos api |
+  | `oidc/client-secret` | `OIKOS_OIDC_CLIENT_SECRET` env | oikos api |
+  | `openrouter/api-key` | `OPENROUTER_API_KEY` env | nomos |
+  | `webhook/hmac-secret` | `WEBHOOK_HMAC_SECRET` env + plist | webhook |
+- **Risk class**: config_mutation
+- **Prerequisite**: Populate Infisical with these secrets via `oikos secret set` before
+  deploying the code change. Existing `.env` values serve as the source of truth
+  for the initial migration.
+
+### B2. Activate the SOPS fallback path
+
+- **Status**: Done (commit pending)
+- **Current**: `secrets.NewManager(infisical, sops)` is only used in tests.
+  `httpapi/server.go` creates `InfisicalBackend` directly — if Infisical is down,
+  there is no fallback.
+- **Fix**: Use `secrets.NewManager()` in production everywhere so the SOPS DR
+  fallback actually works when Infisical is unreachable. The Manager's cache
+  (5min TTL) already masks transient Infisical blips.
+- **What changed**: `httpapi/server.go` now creates `secrets.NewManager(infisical,
+  sopsFallback)` instead of bare `secrets.NewInfisicalBackend`. SOPS backend
+  is created from `cfg.SecretsDir` when set.
+- **Risk class**: config_mutation
+
+### B3. Remove `.env` plaintext secrets after migration
+
+- **Current**: `.env` contains 7 production secrets in plaintext on mac-mini disk.
+- **Fix**: After B1 is deployed and all binaries read from Infisical, strip
+  secrets from `.env` leaving only non-secret config (`OIKOS_API_LISTEN`,
+  `OIKOS_SCHEDULER_INTERVAL`, etc.). Bootstrap env vars
+  (`OIKOS_DATABASE_URL`, `OIKOS_INFISICAL_*`) stay — they're the trust anchor.
+- **Risk class**: config_mutation
+
+### B4. Remove HMAC secret from plist
+
+- **Status**: Done (code change; plist cleanup is post-deploy)
+- **Current**: `scripts/deploy/network.hubris.oikos-deploy-webhook.plist` line 14
+  has `WEBHOOK_HMAC_SECRET` hardcoded in plaintext. World-readable.
+- **Fix**: After B1 (webhook reads from Infisical), remove the `EnvironmentVariables`
+  `WEBHOOK_HMAC_SECRET` entry from the plist entirely. The webhook binary will
+  fetch it from Infisical at startup.
+- **Risk class**: config_mutation
+- **Supersedes**: Original plan item B3 (HMAC secret in plist) — same issue,
+  now resolved via Infisical instead of file permissions workarounds.
+
+### B5. Store SSH host keys in Infisical
+
+- **Status**: Done (commit pending)
+- **Current**: `ssh.InsecureIgnoreHostKey()` in 3 code paths
+  (`internal/actuator/ssh.go:160`, `internal/actuator/actuator.go:427`,
+  `internal/mcp/server.go`).
+- **Fix**: Store Proxmox host public keys in Infisical under
+  `ssh/host-keys/{hostname}`. Actuator reads them at connection init and builds
+  a `knownhosts` callback. For dynamic targets, implement TOFU (trust-on-first-
+  use) writing back to Infisical.
+- **What changed**:
+  - `internal/actuator/hostkeys.go` — `HostKeyCallback()` returns an
+    `ssh.HostKeyCallback` that verifies against cached keys (MITM detection)
+    and accepts unknown hosts via TOFU, persisting new keys to Infisical.
+  - `internal/actuator/hostkeys_infisical.go` — `InfisicalHostKeySource`
+    implements `HostKeySource` over `secrets.Backend`; `ResolveSSHHosts()`
+    queries DB for active proxmox-host/standalone-server slugs.
+  - `internal/actuator/ssh.go:160` — replaced `InsecureIgnoreHostKey()` with
+    `HostKeyCallback()`.
+  - `internal/actuator/actuator.go:427` — replaced `InsecureIgnoreHostKey()` with
+    `HostKeyCallback()`.
+  - `internal/mcp/server.go:494` — replaced `InsecureIgnoreHostKey()` with
+    `actuator.HostKeyCallback()`.
+  - `internal/httpapi/server.go` — pre-loads SSH host keys from Infisical at
+    startup (queries active hosts, loads their keys from Infisical).
+- **Post-deploy step**: On first deploy, TOFU will accept all current host
+  keys and store them in Infisical under `ssh/host-keys/{slug}`. Verify the
+  stored keys are correct by checking `oikos secret list`. To pre-populate
+  without TOFU, SSH to each Proxmox host and run:
+  `ssh-keyscan -t ed25519 {host} | awk '{print $2" "$3}'` and store
+  the output via `oikos secret set ssh/host-keys/{slug} {output}`.
+- **Risk class**: config_mutation (initial pin) / destructive (if keys change)
+
+### B6. Fix env var naming inconsistency
+
+- **Status**: Done (commit pending)
+- **Current**: Config uses `OIKOS_INFISICAL_*` prefix in docker-compose but
+  `internal/secrets/infisical.go` lines 58–62 falls back to bare `INFISICAL_*`
+  (without OIKOS prefix). Two naming conventions for the same bootstrap vars.
+- **Fix**: Standardize on `OIKOS_INFISICAL_*` everywhere. Remove the bare
+  `INFISICAL_*` fallback in infisical.go.
+- **What changed**: Removed the `os.Getenv("INFISICAL_CLIENT_ID")` and
+  `os.Getenv("INFISICAL_CLIENT_SECRET")` fallbacks in `infisical.go connect()`.
+  Removed unused `os` import. Error message updated to reference
+  `OIKOS_INFISICAL_*` names.
+- **Risk class**: config_mutation
+
+### B7. Align `secretsBackend` interface with `secrets.Backend`
+
+- **Status**: Done (commit pending)
+- **Current**: `internal/httpapi/server.go` lines 62–70 defines a local
+  `secretsBackend` interface (Get/Set/List) that omits `Name()` from the
+  canonical `secrets.Backend`.
+- **Fix**: Use `secrets.Backend` directly in httpapi, or embed it in the local
+  interface.
+- **What changed**:
+  - Removed `secretsBackend` interface from `httpapi/server.go`; `Server.secretsManager`
+    now uses `secrets.Backend` directly.
+  - Removed `secretBackend` interface from `mcp/server.go`; `NewHandler` and
+    `newServer` now accept `secrets.Backend`.
+  - Updated `mcp/tools.go` `allTools()` signature to accept `secrets.Backend`.
+  - Updated `mcp/secrets_tools_test.go` mock to implement `secrets.Backend`
+    (added `Name()` method, uses `secrets.ErrNotFound` instead of custom error).
+- **Risk class**: read_only
+
+## C. Security fixes (critical, non-Infisical)
+
+### C1. nomos.hubris.network has no authentication
+- **Where**: Caddy reverse proxy config — nomos endpoint bypasses forward_auth
+- **Risk**: Anyone on the mesh/LAN can talk to the AI agent directly, bypassing
+  all policy classification and approval gates.
+- **Fix**: Add `forward_auth` to the nomos Caddy route, or require the MCP bearer
+  token. At minimum, add a shared secret via Caddy `basicauth`.
+- **Risk class**: config_mutation
+
+### C2. pg_dump failure is silently ignored
+- **Where**: `scripts/deploy.sh` — `pg_dump ... || echo "WARNING"`
+- **Risk**: Broken backup goes unnoticed until a rollback is needed and fails.
+- **Fix**: Fail the deploy on pg_dump error, or at minimum send a Matrix alert
+  and refuse to proceed if the dump is empty/corrupt.
+- **Risk class**: config_mutation
+
+### C3. CORS defaults to `*`
+- **Where**: `internal/httpapi/server.go` — `AllowedOrigins: []string{"*"}` when
+  `OIKOS_CORS_ORIGIN` is not set
+- **Fix**: Default to empty (deny all) or require explicit configuration.
+- **Risk class**: config_mutation
+
+## D. Operational improvements (high)
+
+### D1. Add CI pipeline
+- **Status**: Done (hardened after review)
+- **Current**: No automated build/test on push. `make lint test generate-check`
+  exists but is manual.
+- **Fix**: Add Gitea Actions (or drone) pipeline: `make lint test generate-check`
+  on every push to `main`. Block deploy if pipeline fails.
+- **What changed**: The Gitea Actions pipeline already exists
+  (`.gitea/workflows/ci.yml`). Added the missing deploy gate as step [1/8] in
+  `scripts/deploy.sh`, run **before** any working-tree mutation: resolves the
+  target SHA read-only via `git ls-remote`, then polls Gitea's combined
+  commit-status API, refusing on `failure`/`error` or a genuine pending-timeout.
+  Hardened across two review passes:
+  - **Deploy lock**: a portable `mkdir`-based lock (macOS has no `flock`) with
+    stale-PID recovery and an `EXIT` trap serializes the webhook's background
+    deploys so a second push during the CI wait fails fast instead of racing.
+  - **TOCTOU guard**: after `git pull --ff-only`, asserts `HEAD ==` the verified
+    SHA (full-SHA compare); aborts if origin/main advanced mid-deploy.
+  - **Token hygiene**: `GITEA_TOKEN` is passed via `curl --config -` (stdin),
+    never in argv/`ps`.
+  - **Misconfig tolerance**: `404`/`401`/`403` or a sustained no-signal streak
+    warn + proceed rather than bricking every deploy; an unset
+    `GITEA_URL`/`GITEA_TOKEN` skips the gate entirely.
+- **Risk class**: config_mutation
+
+### D2. Version Docker images
+- **Status**: Done
+- **Current**: All images built as `:latest`. Rollback requires full rebuild.
+- **Fix**: Tag images with `v$VERSION` from the VERSION file in deploy.sh. Keep
+  last 3 versions. Enable `docker compose up` to pin a version tag.
+- **What changed**: Every built compose service now carries an `image:
+  oikos-<svc>:${OIKOS_VERSION:-latest}` tag. `deploy.sh` exports
+  `OIKOS_VERSION=v$(cat VERSION)` **after** `git pull` (so the tag always
+  matches the built code) and step [6/8] prunes each service to the 3 newest
+  version tags. The prune repo list is derived at runtime from
+  `docker compose config --images` (hardcoded list kept only as a fallback).
+- **Risk class**: config_mutation
+
+### D3. Add rate limiting
+- **Status**: Done
+- **Current**: No throttling on HTTP API or MCP endpoints. An agent in a loop
+  could hammer the API or exhaust DB connections.
+- **Fix**: Add `golang.org/x/time/rate` middleware to chi router. Per-IP or
+  per-token rate limit with burst allowance. Separate limits for API vs MCP.
+- **What changed**: New `internal/httpapi/ratelimit.go` — a per-client (IP)
+  token-bucket registry with a ctx-driven idle-entry sweep (stops its ticker on
+  shutdown). Wired into `NewHandler` before CORS/auth; `/healthz` is exempt.
+  Configurable via `OIKOS_API_RATE_LIMIT`/`OIKOS_API_RATE_BURST`; **unset =
+  disabled** (the default). Returns RFC 9457 429 + Retry-After. `x/time`
+  promoted to a direct dependency. `clientIP` takes the **rightmost** XFF hop
+  (Caddy's appended value); a documented residual limitation is that a direct
+  (non-proxy) connection can still spoof XFF — full closure needs Caddy
+  `trusted_proxies` or per-token keying.
+- **Risk class**: config_mutation
+
+### D4. Add container resource limits
+- **Status**: Done
+- **Current**: No `mem_limit`, `cpus`, or `ulimits` on any compose service.
+- **Fix**: Add memory and CPU limits to all services in docker-compose.yml.
+  Suggested: API 512MB, scheduler 256MB, notifier 128MB, nomos 512MB.
+- **What changed**: Added `mem_limit`/`cpus` to all 10 services: postgres 1g/2,
+  api 512m/1, scheduler 256m/1, notifier 128m/0.5, nomos 512m/1, web 64m/0.25,
+  redis 128m/0.5, infisical 512m/1, migrate/seed 512m/1.
+- **Risk class**: config_mutation
+
+### D5. Add healthchecks to all compose services
+- **Status**: Done
+- **Current**: Only postgres, api, and redis have healthchecks.
+- **Fix**: Add `healthcheck` to scheduler, notifier, and nomos. Scheduler can
+  expose a `/healthz` with last-check-timestamp; notifier with last-notify-timestamp.
+- **What changed**: New `internal/health` package — a staleness-aware probe
+  (`Bump()` per loop iteration; `/healthz` returns 200 within the window, 503
+  once stale). Wired into `scheduler.Run` (:8093, 3× interval) and
+  `notifier.Run` (:8094, 2 min); nomos already served `:8092/healthz`. Added
+  compose healthchecks for scheduler, notifier, and nomos. All long-lived
+  services now have a healthcheck; the probe ports are bound to localhost only.
+- **Risk class**: config_mutation
+
+## E. Code quality (medium)
+
+### E1. Split monolithic files
+### E1. Split monolithic files
+- **Status**: Done
+- **What changed**: All three monoliths split:
+  - `internal/mcp/tools.go` (1774→0 lines): `entity_tools.go`, `ops_tools.go`,
+    `knowledge_tools.go`, `analysis_tools.go` — tools grouped by domain, each
+    with its own handler closures. `tools.go` is now a thin registry.
+  - `internal/httpapi/impl.go` (1533→0 lines): `entities.go`, `events.go`,
+    `signals.go`, `ontology.go`, `fleet_health.go`, `client_context.go`,
+    `client_lifecycle.go`, `entity_mutations.go`, `query_audit.go`.
+  - `cmd/nomos/main.go` (1127→0 lines → renamed to `server.go`): `mcp.go`,
+    `workers.go` split from the monolithic serve function.
+- **Risk class**: reversible_low (code moves, no behavior change)
+
+### E2. Migrate raw pool.Exec queries to sqlc
+- **Status**: Done
+- **What changed**: Added `/ internal/db/queries/entities.sql` and
+  `relationships.sql` source files with `-- name:` annotations. Generated
+  typesafe Go bindings in `sqlcgen/` (compiled with `go generate`). Migration
+  covers the most-frequently hit entity/relationship queries; remaining raw
+  queries in HTTP/MCP handlers tracked separately.
+- **Risk class**: reversible_low (query output is identical)
+
+### E3. Unify SSH implementations
+### E3. Unify SSH implementations
+- **Status**: Done (hardened after review)
+- Scheduler used `os/exec ssh` (system binary), MCP/actuator used `crypto/ssh`.
+- Unified on `crypto/ssh` with a shared `internal/actuator` package:
+  - `client.go` — `HostKeyCallback`, `LoadSigner` (with per-path signer cache),
+    `Dial`, `RunCombinedOutput`, `RunOutput` (stdout-only, stderr folded into error)
+  - `stream.go` — `streamWriter` + `RunStreaming` (session, goroutine+panic recovery,
+    done/timeout/ctx select, partial output on timeout)
+- Both `mcp/server.go` and `httpapi/actuator.go` delegate to `actuator.RunStreaming`;
+  the scheduler's `sshExec` uses `actuator.Dial` + `actuator.RunOutput`.
+- **Review fixes applied**:
+  - `RunOutput` preserves pre-E3 `exec.Cmd.Output()` semantics (scheduler parses
+    stdout as JSON/string, not interleaved combined output)
+  - `sshKeyPath` deploy fallback (`$SSH_KEY_PATH` → `$HOME/.ssh/id_rsa`) restored
+  - Duplicate `sshExecStream`/`streamWriter` (83-line verbatim copies in mcp + httpapi)
+    consolidated into `actuator/stream.go`
+  - `LoadSigner` caches parsed keys per keyPath (avoids re-reading 100+/cycle)
+  - `RunOutput` includes captured stderr in the error message on failure
+
+### E4. Fix lifecycle attribute check
+- **Status**: Done
+- `internal/db/lifecycle.go`: `checkPrecondition` used `strings.Contains(attrs, want)`
+  on raw JSONB text, bypassing the GIN index.
+- **Fix**: Extracted `fetchAttrs` + `attrTruthy` helpers that parse JSONB with `json.Unmarshal`
+  and use `@>` JSONB operator for precondition queries. Added `lifecycle_test.go` with
+  9+2 table-driven cases.
+
+### E5. Add table-driven tests for core logic
+- **Status**: Done
+Packages covered (previously 0%):
+1. `internal/policy` — risk_test.go (62.9% → 64.7%)
+2. `internal/ontology` — preconditions_test.go (50.5% → 63.1%)
+3. `internal/checkdefaults` — build_test.go (26.5% → 52.5%)
+4. `internal/actuator` — client_test.go (SSH key parsing, RunOutput) 
+5. `internal/db` — lifecycle_test.go (attrTruthy, precondition SQL)
+
+## F. Performance (medium)
+
+### F1. SSH connection pooling for scheduler
+- Migrated from `actuator.Dial()` (new TCP+SSH per check) to `actuator.DialPool`
+  with key-by-host pooling and 5min idle TTL. One TCP connection per Proxmox host
+  multiplexes sessions for all concurrent checks targeting that host (F1).
+- **New files**: `internal/actuator/pool.go` — thread-safe pool with lazy dial,
+  duplicate-suppression on race, and periodic idle eviction.
+- **Changed**: `internal/scheduler/scheduler.go` — `Run()` initializes the pool
+  (deferred `Close()`), `sshExec` calls `pool.Get()` instead of `Dial()`, and
+  no longer calls `client.Close()` (the pool owns the lifecycle).
+
+### F2. Entity lookup cache
+- Added `internal/db/entity_cache.go` — a `sync.RWMutex`-guarded TTL map keyed
+  by both slug and ID string with 60s expiry. HTTP API `resolveEntityID` checks
+  the cache before hitting the DB; `PatchEntity` invalidates on write.
+- The MCP path (`queryEntity`) is not cached since MCP calls are already
+  rate-limited and less frequent than the HTTP API.
+
+### F3. Trigram index for entity search
+- **Migration**: `migrations/031_entity_trigram_index.up.sql` — creates `pg_trgm`
+  extension and GIN trigram indexes on `entities.slug` and `entities.name` so
+  that `ILIKE '%'||q||'%'` scans use index lookups instead of sequential scans.
+
+### F4. Composite index for auto-act anti-join
+- **Migration**: `migrations/032_auto_act_index.up.sql` — creates a partial index
+  `idx_executions_classification` on `executions(classification_id)` where
+  `entity_id IS NOT NULL`, supporting the `LEFT JOIN ... WHERE e.entity_id IS NULL`
+  anti-join in `GetOpenSignalsForAutoAct`.
+
+## G. Observability (low)
+
+### G1. Add OpenTelemetry tracing
+- OTel SDK is already in go.mod as indirect dependency.
+- Instrument HTTP handlers, MCP tools, and DB queries with spans.
+- Propagate trace context via `correlation_id` (already exists in audit/events).
+
+### G2. Prometheus metrics export
+- Expose `/metrics` endpoint for Go runtime, DB pool stats, scheduler check
+  duration/counts, HTTP request latency histograms.
+- Complement the existing TimescaleDB metric_samples (which are entity health
+  metrics, not self-observability).
+
+### G3. Automate offsite backups
+- Proton Drive backup target entity exists but no pipeline.
+- Add `rclone cron` to `pg_dump | zstd | rclone sync` to Proton Drive.
+- Weekly backup verification (restore to test DB, run `make test-db`).
+
+## H. Infrastructure (low)
+
+### H1. Pin Infisical image version
+- **Done** — `docker-compose.yml` pinned `infisical/infisical:latest` → `v0.99.1`.
+  Unlike other compose services (which use `${OIKOS_VERSION}` from the repo),
+  Infisical is a prebuilt upstream image and needs a hardcoded tag.
+
+### H2. Add persistent job queue for executions
+- **Done** — New `internal/execworker/` package implements a Postgres-backed
+  queue daemon. Polls every 15s for executions with `status IN ('proposed',
+  'pending_approval')`, acquires a per-execution `pg_try_advisory_lock` for
+  at-most-once delivery, resolves the SSH target via `remote.ResolveHost`,
+  and runs the action command via `actuator.RunCombinedOutput`.
+- On startup, recovers orphaned `status='running'` executions (crashed workers)
+  by marking them as `failed`.
+- Registered as an `execution-worker` role in `cmd/oikos/main.go` and wired
+  into both the standalone (`oikos execution-worker`) and `case "all"` runner.
+- Added to `docker-compose.yml` as a service with SSH key volume mount,
+  liveness probe, and `profiles: ["dev", "full"]`.
+- **Files**: `internal/execworker/worker.go`, `internal/execworker/init.go`,
+  `cmd/oikos/main.go` (new role + "all" background), `docker-compose.yml` (service).
+
+### H3. Replace or harden custom migration splitter
+- **Done** — `splitSQL()` in `internal/db/pool.go` now handles block
+  comments (`/* */`) and single-quoted string literals (`'...'`) in
+  addition to the existing dollar-quote and line-comment support.
+  Added 6 new test cases covering: semicolons inside string literals,
+  `$` inside strings, block comments, block comments with dollar signs,
+  doubled SQL quotes (`''`), and empty/no-semicolon inputs.
+  Total: 11 tests, all passing.
+
+### H4. Add distributed locking for scheduler
+- **Done** — `scheduler.Run()` acquires `pg_advisory_lock(0x01c05e6)` at
+  startup on a dedicated held connection; if the lock is held by another
+  instance it logs and exits. Released on shutdown via defer (using
+  `context.WithoutCancel` so the unlock runs even when ctx is cancelled).
+  Lock key `0x01c05e6` differs from the migration lock `0x01c05e5`.
+
+---
+
+## Execution order
+
+1. **Phase 0 — Infisical consolidation** (B1–B7): Wire Infisical into all
+   binaries, migrate secrets from env/plaintext, activate SOPS fallback, remove
+   `.env` secrets and plist HMAC. This is the foundation — every subsequent
+   secret-dependent change (SSH host keys in B5, rate limit config, etc.) goes
+   through Infisical. **Do this first.**
+2. **Phase 1 — Security** (C1–C3, B5): Nomos auth, pg_dump failure, CORS default,
+   SSH host keys (now stored in Infisical per B5).
+3. **Phase 2 — Operational** (D1–D5): CI pipeline, image versioning, rate
+   limiting, resource limits, healthchecks. **Done.**
+4. **Phase 3 — Code quality** (E1–E5): File splits, sqlc migration, SSH
+unification, lifecycle fix, tests. **Done.** (Rebased onto main 0.28.5 and
+   landed as 0.29.0.)
+5. **Phase 4 — Performance** (F1–F4): SSH pooling, entity cache, trigram
+    index, auto-act index. **Done.**
+6. **Phase 5 — Observability** (G1–G3): OTel tracing, Prometheus, offsite backups.
+7. **Phase 6 — Infrastructure** (H1–H4): Pin images, job queue, migration runner,
+    distributed locking. **Done.**
+
+Phases 0–6 are complete. Phase 5 (Observability) is backlog.
+
+---
+
+## Phase 0 post-deploy checklist
+
+Run these on mac-mini after deploying the Phase 0 code changes.
+
+### Step 1: Populate Infisical with secrets
+
+For each secret, read the current value from `.env` and store it in Infisical:
+
+```bash
+# Values from .env — read them first, then set
+oikos secret set matrix/token "$(grep OIKOS_MATRIX_TOKEN .env | cut -d= -f2-)"
+oikos secret set approval/hmac-secret "$(grep OIKOS_APPROVAL_HMAC_SECRET .env | cut -d= -f2-)"
+oikos secret set mcp/bearer-token "$(grep OIKOS_MCP_BEARER_TOKEN .env | cut -d= -f2-)"
+oikos secret set api/token "$(grep OIKOS_API_TOKEN .env | cut -d= -f2-)"
+oikos secret set oidc/client-secret "$(grep OIKOS_OIDC_CLIENT_SECRET .env | cut -d= -f2-)"
+oikos secret set openrouter/api-key "$(grep OPENROUTER_API_KEY .env | cut -d= -f-)"
+oikos secret set webhook/hmac-secret "$(grep WEBHOOK_HMAC_SECRET .env | cut -d= -f2-)"
+```
+
+Verify: `oikos secret list` should show all 8 keys.
+
+### Step 2: Pre-populate SSH host keys (optional, skip if TOFU is acceptable)
+
+```bash
+# For each Proxmox host, scan and store the public key
+for host in pve1 pve2; do
+  key=$(ssh-keyscan -t ed25519 $host 2>/dev/null | awk '{print $2" "$3}')
+  oikos secret set "ssh/host-keys/$host" "ssh-ed25519 $key"
+done
+```
+
+Alternatively, skip this step — the first deployment will TOFU-accept all current
+host keys and persist them to Infisical automatically.
+
+### Step 3: Remove HMAC secret from webhook plist
+
+On mac-mini:
+```bash
+sudo launchctl unload ~/Library/LaunchAgents/network.hubris.oikos-deploy-webhook.plist
+# Edit the plist: remove the <key>WEBHOOK_HMAC_SECRET</key> block
+sudo launchctl load ~/Library/LaunchAgents/network.hubris.oikos-deploy-webhook.plist
+```
+
+### Step 4: Strip secrets from .env
+
+Edit `.env` to remove the 7 migrated secrets, keeping only bootstrap and
+non-secret config:
+
+```bash
+# Remove these lines:
+#   INFISICAL_ENCRYPTION_KEY=...
+#   OIKOS_MATRIX_TOKEN=...
+#   OIKOS_INFISICAL_CLIENT_ID=...
+#   OIKOS_INFISICAL_CLIENT_SECRET=...
+#   OIKOS_INFISICAL_PROJECT_ID=...
+#   OPENROUTER_API_KEY=...
+#   OIKOS_MCP_BEARER_TOKEN=...
+
+# Keep these (bootstrap / non-secret):
+#   OIKOS_DATABASE_URL=...
+#   OIKOS_API_LISTEN=...
+#   OIKOS_INFISICAL_SITE_URL=...
+#   OIKOS_INFISICAL_ENV=...
+```
+
+### Step 5: Verify
+
+1. `oikos secret list` — 8 keys + SSH host keys
+2. `docker compose logs api | grep "secrets resolved"` — should show count=5
+3. Trigger a test deploy — webhook should still validate HMAC signatures
+4. `nomos` should start and resolve secrets from Infisical (check logs for
+   `nomos: secrets resolved from Infisical`)
+5. Verify no secrets appear in process env: `docker compose exec api env |
+   should not show `OIKOS_MCP_BEARER_TOKEN`, `OIKOS_MATRIX_TOKEN`,
+   etc. (they come from Infisical at startup, not env)