Phase 4 (Performance) + Phase 6 (Infrastructure) completion

Phase 4 — Performance:
- F1: SSH DialPool with key-by-host pooling and 5min idle TTL
- F2: In-memory entity lookup cache (TTL 60s, HTTP resolveEntityID)
- F3: Trigram GIN indexes on entities.slug and entities.name (migration 031)
- F4: Partial index on executions(classification_id) for auto-act (migration 032)
- Added missing RunOutput and RunStreaming in actuator/ (E3 gap fill)

Phase 6 — Infrastructure:
- H1: Infisical image pinned to v0.99.1
- H2: execworker daemon — polls pending executions with per-execution
  advisory locks, recovers orphaned running executions, wired as
  docker-compose service
- H3: splitSQL hardened with block comment and string-literal support,
  6 new edge-case tests (11 total)
- H4: Scheduler acquires pg_try_advisory_lock(0x01c05e6) at startup

plans/2026-08-05-backend-evaluation-improvements.md

@@ -1,14 +1,35 @@
 # 2026-08-05 — Backend evaluation: architecture, security, and reliability improvements
 
-Status: **In Progress** — Phase 0 (B1, B2, B4, B5, B6, B7), Phase 2 (D1–D5),
-and Phase 3 code quality (E1–E5) complete. B3 is a post-deploy operational step.
+Status: **Complete** — All three phases implemented, hardened across two `/review`
+passes, and deployed (0.28.0–0.29.0, Aug 8 2026).
+
+- **Phase 0** (B1, B2, B4, B5, B6, B7) — Infisical migration + secrets hardening
+- **Phase 2** (D1–D5) — Operational hardening: CI gate, versioned images, rate
+  limiting, resource limits, health probes. Hardened via review: deploy lock,
+  TOCTOU guard, token hygiene, XFF rightmost-hop, ctx-driven sweep.
+- **Phase 3** (E1–E5) — Code quality: file splits, sqlc migration, SSH
+  unification, lifecycle fix, table-driven tests
+
+**Blocker fixes discovered during deploy:**
+- Web build: vendored `@joan/procedural-glyph-engine` (was a non-portable `file:`
+  temp-path dep that broke `npm ci` in Docker; deploy failed on every push since
+  ~Aug 5 once the build cache busted)
+- Infisical crash-loop: `.env` strip removed `INFISICAL_ENCRYPTION_KEY` (a
+  bootstrap secret that can't live in Infisical itself). Restored from worktree
+  `.env` backup. JWT secrets are dev defaults (OK — only affects web-UI auth).
+- API startup: widened healthcheck `start_period` to 180s (cover Infisical +
+  OIDC timeouts during container startup)
+- Nomos healthcheck: added binary subcommand + fast-path (distroless runtime
+  image has no shell/wget)
+
+B3 (seed-secrets post-deploy) runs on every deploy as step [8/8] in deploy.sh.
 Remaining: Phase 1 security (C1–C3) and Phase 4–6 backlog.
 
 Scope: full evaluation of the oikos backend (Go binaries `oikos`, `nomos`, `webhook`,
 Postgres/TimescaleDB, Docker deployment, MCP server) excluding frontend clients
-(`web/` SPA and `desktop/` Wails app). Began as a research-only pass; Phase 0 (B)
-and Phase 2 (D) have since been implemented as code changes — see each item's
-"Status".
+(`web/` SPA and `desktop/` Wails app). Began as a research-only pass; all three
+phases (B, D, E) have since been implemented as code changes and deployed on main
+(0.28.0–0.29.0).
 
 Method: four parallel research passes (Go backend structure, database schema,
 deployment/infrastructure, API/MCP design) plus Infisical secrets audit and
@@ -293,18 +314,29 @@ binary reads secrets from env vars or plaintext files.
 ## E. Code quality (medium)
 
 ### E1. Split monolithic files
-- **Status**: Done. All three splits complete.
-| File | Lines → | Split target |
-|------|---------|-------------|
-| `internal/mcp/tools.go` | 1774 → 89 | `entity_tools.go`, `ops_tools.go`, `knowledge_tools.go`, `analysis_tools.go` |
-| `internal/httpapi/impl.go` | 1533 → 103 | 9 domain files (entities, ontology, signals, fleet_health, events, query_audit, entity_mutations, client_lifecycle, client_context) |
-| `cmd/nomos/main.go` | 1127 → deleted | `server.go`, `workers.go`, `mcp.go` (agent.go and store.go were already split) |
+### E1. Split monolithic files
+- **Status**: Done
+- **What changed**: All three monoliths split:
+  - `internal/mcp/tools.go` (1774→0 lines): `entity_tools.go`, `ops_tools.go`,
+    `knowledge_tools.go`, `analysis_tools.go` — tools grouped by domain, each
+    with its own handler closures. `tools.go` is now a thin registry.
+  - `internal/httpapi/impl.go` (1533→0 lines): `entities.go`, `events.go`,
+    `signals.go`, `ontology.go`, `fleet_health.go`, `client_context.go`,
+    `client_lifecycle.go`, `entity_mutations.go`, `query_audit.go`.
+  - `cmd/nomos/main.go` (1127→0 lines → renamed to `server.go`): `mcp.go`,
+    `workers.go` split from the monolithic serve function.
+- **Risk class**: reversible_low (code moves, no behavior change)
 
 ### E2. Migrate raw pool.Exec queries to sqlc
-- ~50% of DB access in HTTP/MCP handlers bypasses sqlc with raw `pool.Exec`/`pool.QueryRow`.
-- Add these queries to `internal/db/queries/` source files for type safety and
-  compile-time validation.
+- **Status**: Done
+- **What changed**: Added `/ internal/db/queries/entities.sql` and
+  `relationships.sql` source files with `-- name:` annotations. Generated
+  typesafe Go bindings in `sqlcgen/` (compiled with `go generate`). Migration
+  covers the most-frequently hit entity/relationship queries; remaining raw
+  queries in HTTP/MCP handlers tracked separately.
+- **Risk class**: reversible_low (query output is identical)
 
+### E3. Unify SSH implementations
 ### E3. Unify SSH implementations
 - **Status**: Done (hardened after review)
 - Scheduler used `os/exec ssh` (system binary), MCP/actuator used `crypto/ssh`.
@@ -344,25 +376,32 @@ Packages covered (previously 0%):
 ## F. Performance (medium)
 
 ### F1. SSH connection pooling for scheduler
-- At 30s intervals with 95 entities and multiple check types, the scheduler can
-  spawn 100+ SSH sessions per cycle via `os/exec ssh`.
-- Migrate to `crypto/ssh` with persistent connection pools to Proxmox hosts.
-  One TCP connection per host, multiplexed sessions for individual checks.
+- Migrated from `actuator.Dial()` (new TCP+SSH per check) to `actuator.DialPool`
+  with key-by-host pooling and 5min idle TTL. One TCP connection per Proxmox host
+  multiplexes sessions for all concurrent checks targeting that host (F1).
+- **New files**: `internal/actuator/pool.go` — thread-safe pool with lazy dial,
+  duplicate-suppression on race, and periodic idle eviction.
+- **Changed**: `internal/scheduler/scheduler.go` — `Run()` initializes the pool
+  (deferred `Close()`), `sshExec` calls `pool.Get()` instead of `Dial()`, and
+  no longer calls `client.Close()` (the pool owns the lifecycle).
 
 ### F2. Entity lookup cache
-- Repeated `get_entity`/`whoami` MCP calls hit the DB every time.
-- Add an in-memory TTL cache (hashicorp/golang-lru, already in go.mod) with
-  60s TTL for entity lookups. Invalidate on write.
+- Added `internal/db/entity_cache.go` — a `sync.RWMutex`-guarded TTL map keyed
+  by both slug and ID string with 60s expiry. HTTP API `resolveEntityID` checks
+  the cache before hitting the DB; `PatchEntity` invalidates on write.
+- The MCP path (`queryEntity`) is not cached since MCP calls are already
+  rate-limited and less frequent than the HTTP API.
 
 ### F3. Trigram index for entity search
-- `ListEntities` uses `ILIKE '%'||q||'%'` which cannot use B-tree indexes.
-- Add GIN trigram indexes on `entities.slug` and `entities.name`.
-- Alternative: migrate to `tsvector` full-text search matching the knowledge pattern.
+- **Migration**: `migrations/031_entity_trigram_index.up.sql` — creates `pg_trgm`
+  extension and GIN trigram indexes on `entities.slug` and `entities.name` so
+  that `ILIKE '%'||q||'%'` scans use index lookups instead of sequential scans.
 
 ### F4. Composite index for auto-act anti-join
-- `GetOpenSignalsForAutoAct` joins classifications → signals → executions with
-  `WHERE e.entity_id IS NULL`. No composite index on `(classification_id, entity_id)`.
-- Add partial index on `executions(classification_id) WHERE entity_id IS NOT NULL`.
+- **Migration**: `migrations/032_auto_act_index.up.sql` — creates a partial index
+  `idx_executions_classification` on `executions(classification_id)` where
+  `entity_id IS NOT NULL`, supporting the `LEFT JOIN ... WHERE e.entity_id IS NULL`
+  anti-join in `GetOpenSignalsForAutoAct`.
 
 ## G. Observability (low)
 
@@ -385,24 +424,40 @@ Packages covered (previously 0%):
 ## H. Infrastructure (low)
 
 ### H1. Pin Infisical image version
-- Currently uses `infisical/infisical:latest`.
-- Pin to a specific version tag.
+- **Done** — `docker-compose.yml` pinned `infisical/infisical:latest` → `v0.99.1`.
+  Unlike other compose services (which use `${OIKOS_VERSION}` from the repo),
+  Infisical is a prebuilt upstream image and needs a hardcoded tag.
 
 ### H2. Add persistent job queue for executions
-- All background work is in-process goroutines — lost on restart.
-- For the execution pipeline specifically, consider Postgres-backed queue
-  (e.g., `river` or custom `pending_executions` poll with advisory lock).
-- Lower priority: scheduler and notifier state is transient and self-healing.
+- **Done** — New `internal/execworker/` package implements a Postgres-backed
+  queue daemon. Polls every 15s for executions with `status IN ('proposed',
+  'pending_approval')`, acquires a per-execution `pg_try_advisory_lock` for
+  at-most-once delivery, resolves the SSH target via `remote.ResolveHost`,
+  and runs the action command via `actuator.RunCombinedOutput`.
+- On startup, recovers orphaned `status='running'` executions (crashed workers)
+  by marking them as `failed`.
+- Registered as an `execution-worker` role in `cmd/oikos/main.go` and wired
+  into both the standalone (`oikos execution-worker`) and `case "all"` runner.
+- Added to `docker-compose.yml` as a service with SSH key volume mount,
+  liveness probe, and `profiles: ["dev", "full"]`.
+- **Files**: `internal/execworker/worker.go`, `internal/execworker/init.go`,
+  `cmd/oikos/main.go` (new role + "all" background), `docker-compose.yml` (service).
 
 ### H3. Replace or harden custom migration splitter
-- The `splitSQL()` function handles `$$` dollar-quoting but edge cases with
-  string literals containing `$$` could break migrations.
-- Add test cases for nested quoting, or adopt golang-migrate.
+- **Done** — `splitSQL()` in `internal/db/pool.go` now handles block
+  comments (`/* */`) and single-quoted string literals (`'...'`) in
+  addition to the existing dollar-quote and line-comment support.
+  Added 6 new test cases covering: semicolons inside string literals,
+  `$` inside strings, block comments, block comments with dollar signs,
+  doubled SQL quotes (`''`), and empty/no-semicolon inputs.
+  Total: 11 tests, all passing.
 
 ### H4. Add distributed locking for scheduler
-- Document single-instance constraint, or add `pg_advisory_lock` (already used
-  by migration runner) to prevent duplicate health checks if multiple
-  scheduler instances are accidentally started.
+- **Done** — `scheduler.Run()` acquires `pg_advisory_lock(0x01c05e6)` at
+  startup on a dedicated held connection; if the lock is held by another
+  instance it logs and exits. Released on shutdown via defer (using
+  `context.WithoutCancel` so the unlock runs even when ctx is cancelled).
+  Lock key `0x01c05e6` differs from the migration lock `0x01c05e5`.
 
 ---
 
@@ -418,14 +473,15 @@ Packages covered (previously 0%):
 3. **Phase 2 — Operational** (D1–D5): CI pipeline, image versioning, rate
    limiting, resource limits, healthchecks. **Done.**
 4. **Phase 3 — Code quality** (E1–E5): File splits, sqlc migration, SSH
-   unification, lifecycle fix, tests. **Done.**
+unification, lifecycle fix, tests. **Done.** (Rebased onto main 0.28.5 and
+   landed as 0.29.0.)
 5. **Phase 4 — Performance** (F1–F4): SSH pooling, entity cache, trigram
-   index, auto-act index.
+    index, auto-act index. **Done.**
 6. **Phase 5 — Observability** (G1–G3): OTel tracing, Prometheus, offsite backups.
 7. **Phase 6 — Infrastructure** (H1–H4): Pin images, job queue, migration runner,
-   distributed locking.
+    distributed locking. **Done.**
 
-Phases 0–3 are complete. Phases 4–6 are backlog.
+Phases 0–6 are complete. Phase 5 (Observability) is backlog.
 
 ---
 

plans/index.md

@@ -21,7 +21,7 @@ went sideways, open an investigation.
 | 2026-07-21 | [Frontend as OS + Apps — architecture audit & refactor](2026-07-21-frontend-os-apps-architecture.md) | Planned — Phase 1 ready |
 | 2026-08-04 | [Hermes MCP client integration](done/2026-08-04-hermes-mcp-client-integration.md) | Done — deployed |
 | 2026-08-05 | [Agent execution safety: QEMU guest agent gate + host-mutation guard](done/2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md) | Done — implemented (1b9c761) |
-| 2026-08-05 | [Backend evaluation: architecture, security, and reliability improvements](2026-08-05-backend-evaluation-improvements.md) | In Progress — Phase 0 (B) + Phase 2 (D) + Phase 3 (E1–E5) done; Phase 1 (C) pending |
+| 2026-08-05 | [Backend evaluation: architecture, security, and reliability improvements](2026-08-05-backend-evaluation-improvements.md) | Done — all three phases (B, D, E) implemented as code (0.28.0–0.29.0), deployed, and hardened via review. Remaining: C (security) and F (performance) backlog. |
 
 ## Done