diff --git a/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md b/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md index 854f8cf..14c8c0b 100644 --- a/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md +++ b/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md @@ -1785,7 +1785,579 @@ should be addressed during the phase they belong to.** - Multi-node deployment (designed for, not implemented) - Vector embeddings / semantic search (structured graph only for now) - SSH-key-signed approval requests -- Prometheus / trend signals -- Actuator gateway pattern (Phase 2 of hybrid — start with mounted SSH) +- Actuator gateway pattern (Phase 2 of hybrid — start with restricted SSH key, + build gateway in Phase 3 alongside the actuator) - Automated skill extraction via LLM (patterns extracted statistically for now; LLM-assisted skill refinement is a future enhancement) + +## Audit remediation — HIGH priority + architect/developer review + +This section addresses all HIGH-severity audit findings and the 18 new findings +from the systems architect + senior Go developer review. Each fix references the +finding ID. + +### Schema fixes (CRITICAL/HIGH) + +**SA1 + SG1 — Cognition entities + broken FKs + hypertable PKs:** + +The core issue: the BDD models cognition objects (Signal, Execution, Feedback, +Pattern, Skill, Classification) as graph entities with typed edges, but the schema +only makes Signal a dual entity. `executions.skill_id REFERENCES entities(id)` is a +broken FK (skills live in `skills(id)`, not `entities(id)`). Additionally, hypertable +PKs (`id BIGSERIAL PRIMARY KEY`) don't include the time column — TimescaleDB rejects +this. + +**Resolution — dual entity pattern for all cognition objects:** + +Every cognition object gets an `entities` row (type = `signal`, `execution`, etc.) +AND a typed table for indexed querying. The typed table's PK references +`entities(id)`. Graph relationships (triggers, produces, contributes-to) live in the +`relationships` table, making the cognition loop traversable as a graph. + +```sql +-- Fix: hypertable PKs must include the time column +CREATE TABLE audit_log ( + id BIGSERIAL, + ts TIMESTAMPTZ NOT NULL DEFAULT now(), + -- ... columns ... + PRIMARY KEY (id, ts) -- was: id BIGSERIAL PRIMARY KEY +); +-- Same fix for events and agent_activity: PRIMARY KEY (id, ts) + +-- Fix: executions.skill_id → references skills, not entities +-- (or if using dual-entity pattern, reference entities(id) where type='skill') +CREATE TABLE executions ( + id SERIAL PRIMARY KEY, + -- ... existing columns ... + skill_id TEXT, -- entity ID of skill used + skill_version INTEGER, -- SG9: snapshot version at exec time + -- ... +); + +-- Fix: skills versioning preserves history (SG9) +-- Change PK to composite so old versions remain queryable +CREATE TABLE skills ( + id TEXT NOT NULL, -- 'skill-restart-service' + version INTEGER NOT NULL DEFAULT 1, + name TEXT NOT NULL, + ts TIMESTAMPTZ DEFAULT now(), + procedure TEXT NOT NULL, + applies_to TEXT REFERENCES entity_types(name), + pattern_ids TEXT[], + status TEXT DEFAULT 'drafted', + success_rate REAL, + last_used_at TIMESTAMPTZ, + PRIMARY KEY (id, version) -- was: id TEXT PRIMARY KEY +); +CREATE TABLE skill_versions_audit ( -- SG9: track procedure changes + id SERIAL PRIMARY KEY, + skill_id TEXT NOT NULL, + version INTEGER NOT NULL, + changed_at TIMESTAMPTZ DEFAULT now(), + changed_by TEXT, -- agent or operator entity + diff TEXT, -- diff of procedure field + reason TEXT +); +``` + +**SA5 — Classification entity must be persisted:** + +Add a `classifications` table. Every classifier decision is recorded with the full +reasoning — this is the audit trail for autonomous decisions. + +```sql +CREATE TABLE classifications ( + id SERIAL PRIMARY KEY, + ts TIMESTAMPTZ DEFAULT now(), + signal_entity_id TEXT REFERENCES entities(id), + entity_id TEXT REFERENCES entities(id), + action TEXT NOT NULL, + risk_class TEXT NOT NULL, + route TEXT NOT NULL, -- 'auto-act' or 'escalate' + blast_radius TEXT[], + pattern_confidence REAL, + skill_match TEXT, -- skill entity ID if matched + autonomy_check TEXT, -- 'allowed' or 'blocked: ' + reasoning JSONB NOT NULL, -- full decision explanation + correlation_id TEXT +); +CREATE INDEX idx_class_signal ON classifications(signal_entity_id); +CREATE INDEX idx_class_entity ON classifications(entity_id); +``` + +**SG2 — CAGG array_agg unsupported:** + +Drop `representative_tags` from the 1h continuous aggregate. Query tags from raw +data when needed. + +```sql +CREATE MATERIALIZED VIEW metric_rollups_1h +WITH (timescaledb.continuous) AS + SELECT + time_bucket('1 hour', ts) AS bucket, + entity_id, + metric, + avg(value) AS avg_value, + min(value) AS min_value, + max(value) AS max_value, + count(*) AS sample_count + -- removed: (array_agg(tags))[1] AS representative_tags + FROM metric_samples + GROUP BY bucket, entity_id, metric; +``` + +**SG3 — TimescaleDB functions not idempotent:** + +Use `if_not_exists => TRUE` and exception guards: + +```sql +SELECT create_hypertable('metric_samples', 'ts', + chunk_time_interval => INTERVAL '7 days', + if_not_exists => TRUE); + +DO $$ BEGIN + PERFORM add_retention_policy('metric_samples', INTERVAL '90 days'); +EXCEPTION WHEN OTHERS THEN NULL; +END $$; +``` + +### Ontology fixes (HIGH) + +**SA2 — Layer boundary "cognition creates governance" arrow:** + +Remove the unsupported arrow. The learning engine does NOT write to governance +(policy/autonomy) tables. Instead, validated patterns *propose* autonomy changes as +approval-request entities — the operator must accept. This keeps governance +authoritative and prevents the learning-poisoning vector (S4). + +Also rename the cognition subgraph node from "Governance" to "Approvals" to avoid +the layer-name collision. + +**SA3 — Missing entity types (Person, Agent, IdentityProvider):** + +Add BDD definitions and entity_type seeds: + +``` +Person: matrix_id, oidc_sub +Agent: provider, model, gateway_port +IdentityProvider: issuer, client_id, auth_mode (oidc/forward-auth) +``` + +These are first-class governance-layer entities. `audit_log.actor_id` now resolves +to a real entity. + +**SA4 — Lifecycle dead-ends and missing transitions:** + +Updated infrastructure lifecycle (add `failed` state): +``` +planned → provisioning → active → migrating → deprecated → destroyed + ↘ failed ↗ ↘ failed ↗ +failed → active (recovery) | failed → deprecated (write-off) +planned → destroyed (cancel) +deprecated → active (un-deprecate if replacement fails) +``` + +Updated signal lifecycle (add terminal states): +``` +raised → acknowledged → acting → resolved | failed +acknowledged → resolved (manual resolve without acting) +acknowledged → muted +acting → failed (permanent failure, terminal — needs operator) +failed → acknowledged (operator retries) +``` + +Updated execution lifecycle (add cancellation + rollback failure): +``` +approved → expired (TTL ran out) +executing → cancelled (operator abort) +failed → rolled_back | rollback_failed +timed_out → verifying (check if the command actually completed despite timeout) +``` + +Updated pattern lifecycle: +``` +hypothesized → validated → active → deprecated +hypothesized → invalidated (disproven, terminal) +active → invalidated (new evidence contradicts) +``` + +Updated skill lifecycle: +``` +drafted → tested → active → refined → active (new version) +drafted → deprecated (abandoned) +tested → failed → drafted (back to drawing board) +active → deprecated (superseded or unsafe) +``` + +Approval lifecycle (new diagram — was missing): +``` +pending → approved | denied | expired +approved → revoked (operator changes mind before execution) +``` + +**SA6 — recommended_action data source:** + +Move `recommended_action` from `signals` to `classifications`. The probe raises a +signal (kind, severity, evidence). The classifier populates the classification with +the recommended action based on signal kind + skill lookup. The actuator reads the +classification, not the signal, for the action to take. + +**SA8 — Missing entities (Cluster, ComposeStack, ManagedHost):** + +Add to ontology: +- `Cluster` entity (software domain) — `ProxmoxHost` `member-of` `Cluster` +- `ComposeStack` entity (software domain) — `DockerContainer` `part-of` `ComposeStack` +- Add `provider`, `control_level` (`full`/`partial`/`none`) to `StandaloneServer` + +### Security remediation (HIGH) + +**S1 — SSH keys in containers:** + +Dual approach (operator decision): +1. **Phase 1-2 (immediate):** Restricted SSH key — dedicated key pair with + `command="..."` and `from="..."` restrictions in `authorized_keys` on hubris/ + strong. The key can only run specific commands (pct, qm, df, systemctl status), + not arbitrary shells. Mounted read-only into the actuator container only (not + Hermes). +2. **Phase 3 (actuator build):** Full actuator gateway — the API's `/exec` endpoint + brokers all SSH. The actuator holds the keys, executes per-action, logs every + command. Hermes never touches SSH. + +**S2 — MCP auth:** +- Shared secret between API and Hermes (HMAC-signed requests) +- MCP bound to a dedicated Docker network (not the default bridge) +- Never exposed via Caddy without auth + +**S3 — Policy DB mutability:** +- Policy mutations (`risk_classes`, `approval_rules`, `autonomy_settings`) require a + meta-approval: the operator must approve the policy change itself (dual-control) +- Immutable audit log of all policy changes with before/after hash +- Startup self-check: compute policy hash, alert if differs from last-known-good + +**S4 — Learning model poisoning:** +- Pattern transitions to `active` require operator confirmation (`PATCH /api/v1/ + patterns/{id}` with `status=active` — policy-gated as `config_mutation`) +- Confidence capped by sample size: `confidence = min(raw_confidence, N/5)` where N + = evidence_count (requires N≥5 for confidence > 0.2) +- Anomaly detection: if >10 identical-outcome feedback entries arrive within 1 hour + for the same (entity_type, action), quarantine the pattern for review +- Skills can never auto-promote to destructive risk class — always escalate + +**S5 — confirmation_phrase replacement:** +- Replace with single-use signed approval tokens +- Token = HMAC(approval_id + entity_id + action + risk_class + nonce, shared_secret) +- Stored hashed in `approvals` table +- Transmitted via Matrix as the approval ID + decision; token verified server-side + +**SA10 — Gateway + Caddy trust boundary:** +- Port 8092 (Hermes gateway): mTLS or token auth. Mesh membership is the network + boundary; gateway auth is the application boundary. +- Caddy is an explicit trust root. API validates OIDC JWTs in middleware (not just + trusting Caddy headers). Documented: compromising Caddy ≠ compromising the API. + +### Operational remediation (HIGH) + +**A3 + O3 + O4 — Backup, restore, DR:** + +Backup strategy: +- **Daily `pg_dump`** (compressed, custom format) + WAL archiving for PITR +- **Off-host storage:** Proton Drive (cloud object storage, encrypted at rest) +- **Push via rclone** from the scheduler container (already have rclone LXC in the + fleet — reuse credentials) +- **Retention:** 30 daily + 12 monthly snapshots +- **Infisical backup:** Infisical has its own backup mechanism; also export secrets + to an encrypted SOPS file as a fallback (chicken-and-egg: keep one age key for + this purpose) +- **Hermes volume:** backed up with `pg_dump` of agent_activity + session data + +Restore procedure (`docs/operations/backup-restore.md`): +1. Restore Postgres: `pg_restore -d oikos < dump.psql` +2. Verify seed ingest matches (run `GET /api/v1/export` and diff against seed YAML) +3. Restore Infisical from its backup +4. `docker compose up -d` +5. Monthly restore drill (scheduled, automated, alert if restore fails) + +DR plan: +- **RTO:** 4 hours (fresh machine → Docker → restore → running) +- **RPO:** 24 hours (last daily backup) +- **Cold-start runbook:** install Docker → clone repo → restore Infisical → + restore DB → `docker compose up -d` → verify health +- **Off-host backup target:** Proton Drive (encrypted, offsite) + +**O1 — Rollback strategy:** +- **Forward-only migrations** (no `down.sql` beyond development). Compensating + migrations for production rollbacks. +- **Pre-deploy DB backup:** the deploy script runs `pg_dump` before `docker compose + up -d` +- **Migration compatibility:** new code must tolerate old schema for one deploy + window (additive migrations only — new columns nullable, new tables optional) +- **Rollback runbook:** revert git commit → `pg_restore` from pre-deploy backup → + `docker compose up -d` with old image + +**O2 — External watchdog:** +- Cron job on apps/105 (outside the Docker stack): `curl -sf + http://mac-mini:8090/healthz || curl -X POST matrix-webhook ...` +- Runs every 5 minutes +- Alerts operator directly via Matrix if the API is unreachable +- Also checks: Docker daemon running (`docker info`), Postgres accepting + connections (`pg_isready`) + +**M1 — CI/CD:** +- Gitea Actions (or simple webhook + script): + - `go vet ./...` + - `golangci-lint run` + - `go test ./... -race -cover` + - `docker build` (no push — just verify it builds) +- Webhook deploy gated on green CI +- Deploy script runs `go test` as a final safety check before `docker compose up` + +### Architecture remediation (HIGH) + +**A1 — Testing strategy:** + +Add testing workstream with specific tests: + +| Package | Test type | What to test | +|---|---|---| +| `internal/policy/` | Unit | Classifier scoring: risk × blast × confidence. Table-driven: every (risk_class, blast_radius, confidence) combination. Edge: unknown action, ambiguous entity. | +| `internal/policy/` | Unit | Approval lifecycle: token issue, verify, single-use enforcement, TTL expiry. | +| `internal/learning/` | Unit | Pattern confidence calculation. Skill versioning. Feedback → pattern extraction. | +| `internal/ontology/` | Unit | Lifecycle transition validation: every legal transition succeeds, every illegal one fails. Graph traversal (mock relationships). | +| `internal/db/` | Integration | testcontainers Postgres: seed ingest idempotency, blast_radius CTE (with cycles), hypertable insert + query, continuous aggregate refresh. | +| `internal/api/` | Integration | testcontainers: REST routes return correct status codes, MCP tools return expected shapes, audit middleware records entries, pagination works. | +| `internal/actuator/` | Integration | Mock SSH: execution → verification → feedback → pattern update. Loop-guard prevents retry storms. Circuit breaker trips after N failures. | +| `migrations/` | Property | Every migration is forward-only. `blast_radius` returns correct results on cyclic graphs. Hypertable retention doesn't drop data younger than threshold. | +| `internal/observability/` | Unit | Correlation ID propagation through context. Event emitter transactional with state change. Metric recording. | + +Coverage gate: ≥80% on `internal/policy/` and `internal/learning/` (the autonomy- +granting code). ≥60% on everything else. + +**A2 — Observability:** Already addressed (Migration 6, Workstream 14). Update audit +status to resolved. + +**SG4 — Graceful shutdown:** + +Every `cmd/*/main.go` implements: +1. `signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)` +2. Context propagated to all long-running loops and HTTP server +3. Shutdown sequence: stop accepting new work → wait for in-flight (30s deadline) → + for actuator: if execution in-flight, mark `failed` with "shutdown interrupted" + + emit feedback → close DB pool +4. `docker-compose.yml`: `stop_grace_period: 30s` on actuator, `stop_signal: SIGTERM` + on all services + +**SG5 — Entity-level concurrency:** + +```sql +-- Per-entity advisory lock during execution (prevents concurrent actions +-- on the same target entity, e.g., restart + deploy on the same service) +SELECT pg_advisory_xact_lock(hashtext($1)); -- $1 = target_entity_id +-- ... execute, verify, feedback ... +-- lock released on transaction commit/rollback +``` + +**SG6 — Domain layer:** + +Add `internal/domain/` package: +``` +internal/domain/ +├── entity.go # Entity, EntityType, Relationship domain types +├── signal.go # Signal domain type + lifecycle transition logic +├── execution.go # Execution domain type + state machine +├── classification.go # Classification domain type +├── pattern.go # Pattern domain type + confidence calculation +├── skill.go # Skill domain type + versioning +├── approval.go # Approval domain type + token verification +└── errors.go # Sentinel errors: ErrNotFound, ErrInvalidTransition, + # ErrApprovalRequired, ErrAutonomyBlocked, ErrConflict +``` + +DB ↔ domain mapping in `internal/db/` (repository pattern). API handlers accept/ +return domain types. sqlc models never escape `internal/db/`. + +**SG11 — Error handling:** + +```go +// internal/domain/errors.go +var ( + ErrNotFound = errors.New("entity not found") + ErrInvalidTransition = errors.New("invalid lifecycle transition") + ErrApprovalRequired = errors.New("operator approval required") + ErrAutonomyBlocked = errors.New("autonomy policy blocks this action") + ErrConflict = errors.New("concurrent modification conflict") + ErrCircuitOpen = errors.New("circuit breaker open for target") +) +``` + +HTTP mapping middleware: `ErrNotFound → 404`, `ErrInvalidTransition → 409`, +`ErrApprovalRequired → 403`, `ErrAutonomyBlocked → 403`, `ErrConflict → 409`, +`ErrCircuitOpen → 503`. + +SSH error classification in `internal/actuator/execute.go`: +- Network unreachable → retryable, circuit breaker +- Auth failure → fatal, alert operator +- Command exit non-zero → execution failed, feedback +- Command timeout → timed_out, feedback + +DB retry: serialization failures (SQLSTATE 40001, 40P01) → retry with exponential +backoff (max 3 retries). + +### Go implementation fixes (MEDIUM) + +**SA9 — TimescaleDB Docker image + migration runner:** +- Image: `timescale/timescaledb:2.x-pg16` (not `postgres:16`) +- Migrations run in a one-shot init container (`compose/migrate/Dockerfile`) with a + dedicated DB user that has DDL but no runtime data privileges +- API's DB user gets DML only (least privilege) +- `golang-migrate` Go API with `//go:embed migrations/*.up.sql` + +**SG7 — Pattern/skill management endpoints:** +- `PATCH /api/v1/patterns/{id}` — state transition (validate/invalidate/deprecate), + policy-gated as `config_mutation`, audit-logged +- `PATCH /api/v1/skills/{id}` — state transition + version pin, same gating +- This is the operator's manual safety valve for learning-model issues (S4) + +**SG8 — WebSocket push mechanism:** +- In-process event bus (Go channel pub/sub) for events written by the API itself + (zero-latency push to WebSocket subscribers) +- Postgres `LISTEN/NOTIFY` for events written by other services (scheduler, + actuator) — trigger on `events` table fires NOTIFY after commit +- Both feed the WebSocket handler + +**SG10 — Transactional event emission:** +- Event + audit entries written in the same DB transaction as the state change +- If transaction rolls back, events are discarded (never emitted) +- `LISTEN/NOTIFY` fires after commit — subscribers only see committed events + +**SG13 — Context-aware SSH:** +```go +func runSSH(ctx context.Context, client *ssh.Client, cmd string) ([]byte, error) { + session, err := client.NewSession() + if err != nil { return nil, err } + defer session.Close() + type result struct { out []byte; err error } + ch := make(chan result, 1) + go func() { out, err := session.CombinedOutput(cmd); ch <- result{out, err} }() + select { + case r := <-ch: return r.out, r.err + case <-ctx.Done(): + session.Close() // unblocks CombinedOutput + client.Close() + return nil, ctx.Err() + } +} +``` + +**SG14 — Connection pool sizing:** +- API: 15 connections, scheduler: 5, actuator: 5, learning: 3 = 28 total +- Postgres `max_connections` set to 80 +- Monitor `db_connections_active`, alert if >80% of pool + +**SG15 — RESTful exec endpoint:** +- `POST /api/v1/executions` (was `POST /api/v1/exec`) — creates an execution + resource. Handler classifies, checks approval, creates execution in `proposed` + state. +- `GET /api/v1/executions/{id}` — status +- `POST /api/v1/executions/{id}/cancel` — cancellation + +**SG16 — Pagination:** +- All hypertable-backed endpoints: cursor-based (`?cursor=&limit=50`) +- `entities` and other small tables: keyset pagination (`?after=&limit=50`) +- Default limit: 50, max: 200 +- MCP tools support `limit` parameter + +**SG17 — Go tooling:** +- `sqlc.yaml` added to repo layout +- Module path: `github.com/dtoro/oikos` +- `CGO_ENABLED=0` in Dockerfiles, `pgx` (pure Go), `gcr.io/distroless/static` + runtime image +- Migrations embedded with `//go:embed` + +**SG18 — Health/metrics bypass auth:** +- `/healthz` and `/metrics` on a separate Gin router group, no auth, no audit +- `/healthz`: `SELECT 1` against DB +- `/metrics`: internal-only (not exposed via Caddy), or token-protected + +**SA7 — Notifier decoupling:** +- API writes `approvals` row (status=pending) + emits `approval.requested` event +- Notifier polls pending approvals, sends to Matrix, writes decision directly to + `approvals` table (has DB access, not API access) +- API polls `approvals.status` +- No service-to-service calls in either direction — DB is the rendezvous point +- Pending approvals survive Notifier restart + +**SA9 + SA10 — Threat model documentation:** +- Caddy is an explicit trust root (compromise = API compromise, mitigated by JWT + validation in API middleware) +- Mesh membership is the network boundary for Hermes gateway +- Docker network is the trust boundary for internal services (mTLS between API and + Hermes, TLS to Postgres) +- The actuator is the only container with SSH egress (not Hermes, not the API) + +### Updated phasing (incorporating remediation) + +**Phase 0 — Ontology design (no code):** +- Finalize entity types (including Person, Agent, IdentityProvider, Cluster, + ComposeStack) + relationship types + lifecycles (all with terminal states) +- Write seed manifests +- Review diagrams + lifecycle completeness with operator + +**Phase 1 — Foundation (Go + DB):** +- Go module setup (`github.com/dtoro/oikos`), project structure with domain layer +- PostgreSQL + TimescaleDB (`timescale/timescaledb:2.x-pg16`) +- Migrations 1-6 (with fixed hypertable PKs, idempotent TimescaleDB calls, dual + entity pattern for cognition objects, classifications table) +- Migration runner as init container (DDL-only DB user) +- Seed ingest pipeline (transactional, init container) +- sqlc + domain layer (repository pattern) +- Structured logging (slog) + error sentinels + HTTP error mapping +- **Testing foundation:** testcontainers setup, unit test framework, coverage gates +- **Backup setup:** daily `pg_dump` + rclone push to Proton Drive, WAL archiving +- **External watchdog:** cron on apps/105 + +**Phase 2 — API (Go):** +- Gin server with REST routes (resource-oriented, paginated, `/executions` not + `/exec`) +- MCP protocol adapter (shared-secret auth, dedicated network) +- Policy enforcement middleware (OIDC JWT validation, not just Caddy headers) +- Audit middleware (transactional with state changes) +- Event emitter (transactional, in-process bus + LISTEN/NOTIFY) +- Observability routes (metrics, trends, audit, events, health, agent-activity) +- WebSocket (in-process bus + LISTEN/NOTIFY, bounded channels, backpressure) +- Pattern/skill management endpoints (operator override for learning model) +- Domain layer fully fleshed out +- **CI:** Gitea Actions (go vet, golangci-lint, go test -race -cover, docker build) + +**Phase 3 — Control loop (Go):** +- Scheduler (Observe) — probes, signals, state snapshots, metric recording +- Actuator (Act) — classify, execute, verify, correlation ID propagation + - **Restricted SSH key** (command= in authorized_keys, actuator-only) + - **Entity-level advisory locks** (pg_advisory_xact_lock) + - **Context-aware SSH** (context cancellation, hard timeout) + - **Circuit breaker** per target host + - **Graceful shutdown** (SIGTERM, in-flight protection, stop_grace_period: 30s) +- Learning engine — feedback, patterns, skills, learning metrics + - **Pattern activation requires operator confirmation** (PATCH endpoint) + - **Confidence capped by sample size** (N≥5) + - **Anomaly detection** for feedback bursts +- Approval tokens (single-use HMAC, not confirmation phrases) +- Policy meta-approval (dual-control for policy mutations) +- Notifier (DB rendezvous, no service-to-service calls, Matrix impl) + +**Phase 4 — Agent (Hermes container):** +- Hermes Docker image, gateway config (mTLS on port 8092) +- Homelab skills +- Agent activity logging +- Connect from workstation, verify MCP (shared-secret auth) + SSH via actuator + +**Phase 5 — Secrets (Infisical):** +- Stand up Infisical, migrate SOPS, wire services +- Infisical backup + SOPS fallback (one age key kept for DR) + +**Phase 6 — Deploy + cutover:** +- Docker Compose (timescale image, init containers, pool sizing, stop_grace_period) +- Gitea webhook (HMAC auth, non-root deploy user, CI-gated) +- Caddy re-point + JWT validation in API +- End-to-end verification (14 checks) +- Stop apps/105, clean up mac-mini +- **Restore drill** (monthly, automated)