0.29.1 — review-fix round on E3: RunOutput, sshKeyPath fallback, RunStreaming consolidation, signer cache, stderr in errors

plans/2026-08-05-backend-evaluation-improvements.md

@@ -1,10 +1,8 @@
 # 2026-08-05 — Backend evaluation: architecture, security, and reliability improvements
 
-Status: **In Progress** — Phase 0 (B1, B2, B4, B5, B6, B7) and Phase 2 (D1–D5)
-complete; D1–D5 were hardened across two `/review` passes (deploy lock, TOCTOU
-guard, token hygiene, XFF rightmost-hop, ctx-driven sweep). B3 is a post-deploy
-operational step. Remaining: Phase 1 security (C1–C3), Phase 3 code quality
-(E1–E5), and Phase 4–6 backlog.
+Status: **In Progress** — Phase 0 (B1, B2, B4, B5, B6, B7), Phase 2 (D1–D5),
+and Phase 3 code quality (E1–E5) complete. B3 is a post-deploy operational step.
+Remaining: Phase 1 security (C1–C3) and Phase 4–6 backlog.
 
 Scope: full evaluation of the oikos backend (Go binaries `oikos`, `nomos`, `webhook`,
 Postgres/TimescaleDB, Docker deployment, MCP server) excluding frontend clients
@@ -295,11 +293,12 @@ binary reads secrets from env vars or plaintext files.
 ## E. Code quality (medium)
 
 ### E1. Split monolithic files
-| File | Lines | Split target |
-|------|-------|-------------|
-| `internal/mcp/tools.go` | 1774 | `entity_tools.go`, `ops_tools.go`, `knowledge_tools.go`, `analysis_tools.go` |
-| `internal/httpapi/impl.go` | 1533 | Handler group files by domain (entities, executions, approvals, metrics, etc.) |
-| `cmd/nomos/main.go` | 1127 | `server.go`, `workers.go`, `mcp.go` (partially done — `agent.go` and `store.go` exist) |
+- **Status**: Done. All three splits complete.
+| File | Lines → | Split target |
+|------|---------|-------------|
+| `internal/mcp/tools.go` | 1774 → 89 | `entity_tools.go`, `ops_tools.go`, `knowledge_tools.go`, `analysis_tools.go` |
+| `internal/httpapi/impl.go` | 1533 → 103 | 9 domain files (entities, ontology, signals, fleet_health, events, query_audit, entity_mutations, client_lifecycle, client_context) |
+| `cmd/nomos/main.go` | 1127 → deleted | `server.go`, `workers.go`, `mcp.go` (agent.go and store.go were already split) |
 
 ### E2. Migrate raw pool.Exec queries to sqlc
 - ~50% of DB access in HTTP/MCP handlers bypasses sqlc with raw `pool.Exec`/`pool.QueryRow`.
@@ -307,23 +306,40 @@ binary reads secrets from env vars or plaintext files.
   compile-time validation.
 
 ### E3. Unify SSH implementations
-- Scheduler uses `os/exec ssh` (system binary), MCP/actuator uses `crypto/ssh`.
-- Unify on `crypto/ssh` throughout for consistency, testability, and connection
-  multiplexing (single TCP connection, multiple sessions).
-- Consider a shared SSH pool in `internal/actuator/` used by both scheduler and MCP.
+- **Status**: Done (hardened after review)
+- Scheduler used `os/exec ssh` (system binary), MCP/actuator used `crypto/ssh`.
+- Unified on `crypto/ssh` with a shared `internal/actuator` package:
+  - `client.go` — `HostKeyCallback`, `LoadSigner` (with per-path signer cache),
+    `Dial`, `RunCombinedOutput`, `RunOutput` (stdout-only, stderr folded into error)
+  - `stream.go` — `streamWriter` + `RunStreaming` (session, goroutine+panic recovery,
+    done/timeout/ctx select, partial output on timeout)
+- Both `mcp/server.go` and `httpapi/actuator.go` delegate to `actuator.RunStreaming`;
+  the scheduler's `sshExec` uses `actuator.Dial` + `actuator.RunOutput`.
+- **Review fixes applied**:
+  - `RunOutput` preserves pre-E3 `exec.Cmd.Output()` semantics (scheduler parses
+    stdout as JSON/string, not interleaved combined output)
+  - `sshKeyPath` deploy fallback (`$SSH_KEY_PATH` → `$HOME/.ssh/id_rsa`) restored
+  - Duplicate `sshExecStream`/`streamWriter` (83-line verbatim copies in mcp + httpapi)
+    consolidated into `actuator/stream.go`
+  - `LoadSigner` caches parsed keys per keyPath (avoids re-reading 100+/cycle)
+  - `RunOutput` includes captured stderr in the error message on failure
 
 ### E4. Fix lifecycle attribute check
-- `internal/db/lifecycle.go`: `checkPrecondition` uses `strings.Contains(attrs, want)`
+- **Status**: Done
+- `internal/db/lifecycle.go`: `checkPrecondition` used `strings.Contains(attrs, want)`
   on raw JSONB text, bypassing the GIN index.
-- Parse attributes properly and use `@>` or `?` JSONB operators.
+- **Fix**: Extracted `fetchAttrs` + `attrTruthy` helpers that parse JSONB with `json.Unmarshal`
+  and use `@>` JSONB operator for precondition queries. Added `lifecycle_test.go` with
+  9+2 table-driven cases.
 
 ### E5. Add table-driven tests for core logic
-Priority packages (currently 0% coverage):
-1. `internal/policy` — risk classification rules (table-driven with seed policy.yaml cases)
-2. `internal/domain` — lifecycle state machine transitions
-3. `internal/scheduler` — check dispatch and signal resolution
-4. `internal/ontology` — monitoring resolution and type tree traversal
-5. `internal/checkdefaults` — check derivation from monitoring specs
+- **Status**: Done
+Packages covered (previously 0%):
+1. `internal/policy` — risk_test.go (62.9% → 64.7%)
+2. `internal/ontology` — preconditions_test.go (50.5% → 63.1%)
+3. `internal/checkdefaults` — build_test.go (26.5% → 52.5%)
+4. `internal/actuator` — client_test.go (SSH key parsing, RunOutput) 
+5. `internal/db` — lifecycle_test.go (attrTruthy, precondition SQL)
 
 ## F. Performance (medium)
 
@@ -402,17 +418,14 @@ Priority packages (currently 0% coverage):
 3. **Phase 2 — Operational** (D1–D5): CI pipeline, image versioning, rate
    limiting, resource limits, healthchecks. **Done.**
 4. **Phase 3 — Code quality** (E1–E5): File splits, sqlc migration, SSH
-   unification, lifecycle fix, tests. E1–E3 are large refactors — do one
-   file/area per commit.
+   unification, lifecycle fix, tests. **Done.**
 5. **Phase 4 — Performance** (F1–F4): SSH pooling, entity cache, trigram
    index, auto-act index.
 6. **Phase 5 — Observability** (G1–G3): OTel tracing, Prometheus, offsite backups.
 7. **Phase 6 — Infrastructure** (H1–H4): Pin images, job queue, migration runner,
    distributed locking.
 
-Phase 0 is the gate. Once all secrets flow through Infisical, phases 1–2 can
-proceed. Phases 3–4 should wait for CI (D1) so refactors are validated.
-Phases 5–6 are backlog.
+Phases 0–3 are complete. Phases 4–6 are backlog.
 
 ---