plans: remediate all HIGH audit items + architect/developer review
Addresses 15 original HIGH audit findings + 18 new findings from systems architect + senior Go developer review (572 lines added). CRITICAL fixes: - SA1: Cognition objects (execution/feedback/pattern/skill) get dual entity pattern — entities row + typed table, graph-traversable - SG1: Hypertable PKs fixed — PRIMARY KEY (id, ts) for audit_log, events, agent_activity (was id-only, would fail create_hypertable) HIGH fixes: - SA2: Remove 'cognition creates governance' arrow (unsupported, was learning-poisoning vector). Patterns propose, operator accepts. - SA3: Add Person, Agent, IdentityProvider to ontology (were used in BDD but never defined) - SA4: Fix all lifecycle dead-ends — add 'failed' state to infra, terminal 'failed'/'invalidated' to signals/patterns/skills, add approval lifecycle diagram, add cancellation/rollback-failure to executions - SA5: Add classifications table — persist classifier reasoning (was modeled in BDD but never stored) - SA6: Move recommended_action from signals to classifications - SA8: Add Cluster, ComposeStack, ManagedHost to ontology - SG2: Drop array_agg from CAGG (unsupported by TimescaleDB) - SG3: Idempotent TimescaleDB calls (if_not_exists, exception guards) - SG4: Graceful shutdown (SIGTERM, in-flight protection, 30s grace) - SG5: Entity-level advisory locks (pg_advisory_xact_lock per target) - SG6: Domain layer (internal/domain/) — sqlc models never escape db/ Security: - S1: Restricted SSH key (command=) now + actuator gateway in Phase 3 - S2: MCP shared-secret auth + dedicated Docker network - S3: Policy mutations require meta-approval (dual-control) - S4: Pattern activation needs operator confirmation + confidence capped by sample size (N>=5) + anomaly detection - S5: Single-use HMAC approval tokens replace confirmation_phrase - SA10: Gateway mTLS + Caddy as documented trust root + JWT validation Operational: - A3/O3/O4: Backup to Proton Drive (daily pg_dump + WAL), restore runbook, DR plan (RTO 4h, RPO 24h), monthly restore drill - O1: Forward-only migrations + pre-deploy backup + rollback runbook - O2: External watchdog cron on apps/105 - M1: CI/CD via Gitea Actions (go vet, lint, test -race, docker build) Architecture: - A1: Testing strategy with specific tests per package + coverage gates - SA7: Notifier decoupled via DB rendezvous (no service-to-service calls) - SA9: TimescaleDB Docker image specified + init container for migrations - SG7: Pattern/skill management endpoints (operator override) - SG8: WebSocket push via in-process bus + LISTEN/NOTIFY - SG10: Transactional event emission (same tx as state change) - SG11: Error handling — sentinel errors + HTTP mapping + SSH taxonomy - SG13: Context-aware SSH (x/crypto/ssh doesn't honor context) - SG14: Connection pool sizing (28 total, max_connections=80) - SG15: RESTful /executions (was /exec) - SG16: Pagination on all list endpoints - SG17: Go tooling (sqlc.yaml, module path, CGO_ENABLED=0, distroless) - SG18: /healthz and /metrics bypass auth + audit Updated phasing incorporates all remediation.
This commit is contained in:
@@ -1785,7 +1785,579 @@ should be addressed during the phase they belong to.**
|
||||
- Multi-node deployment (designed for, not implemented)
|
||||
- Vector embeddings / semantic search (structured graph only for now)
|
||||
- SSH-key-signed approval requests
|
||||
- Prometheus / trend signals
|
||||
- Actuator gateway pattern (Phase 2 of hybrid — start with mounted SSH)
|
||||
- Actuator gateway pattern (Phase 2 of hybrid — start with restricted SSH key,
|
||||
build gateway in Phase 3 alongside the actuator)
|
||||
- Automated skill extraction via LLM (patterns extracted statistically for now;
|
||||
LLM-assisted skill refinement is a future enhancement)
|
||||
|
||||
## Audit remediation — HIGH priority + architect/developer review
|
||||
|
||||
This section addresses all HIGH-severity audit findings and the 18 new findings
|
||||
from the systems architect + senior Go developer review. Each fix references the
|
||||
finding ID.
|
||||
|
||||
### Schema fixes (CRITICAL/HIGH)
|
||||
|
||||
**SA1 + SG1 — Cognition entities + broken FKs + hypertable PKs:**
|
||||
|
||||
The core issue: the BDD models cognition objects (Signal, Execution, Feedback,
|
||||
Pattern, Skill, Classification) as graph entities with typed edges, but the schema
|
||||
only makes Signal a dual entity. `executions.skill_id REFERENCES entities(id)` is a
|
||||
broken FK (skills live in `skills(id)`, not `entities(id)`). Additionally, hypertable
|
||||
PKs (`id BIGSERIAL PRIMARY KEY`) don't include the time column — TimescaleDB rejects
|
||||
this.
|
||||
|
||||
**Resolution — dual entity pattern for all cognition objects:**
|
||||
|
||||
Every cognition object gets an `entities` row (type = `signal`, `execution`, etc.)
|
||||
AND a typed table for indexed querying. The typed table's PK references
|
||||
`entities(id)`. Graph relationships (triggers, produces, contributes-to) live in the
|
||||
`relationships` table, making the cognition loop traversable as a graph.
|
||||
|
||||
```sql
|
||||
-- Fix: hypertable PKs must include the time column
|
||||
CREATE TABLE audit_log (
|
||||
id BIGSERIAL,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- ... columns ...
|
||||
PRIMARY KEY (id, ts) -- was: id BIGSERIAL PRIMARY KEY
|
||||
);
|
||||
-- Same fix for events and agent_activity: PRIMARY KEY (id, ts)
|
||||
|
||||
-- Fix: executions.skill_id → references skills, not entities
|
||||
-- (or if using dual-entity pattern, reference entities(id) where type='skill')
|
||||
CREATE TABLE executions (
|
||||
id SERIAL PRIMARY KEY,
|
||||
-- ... existing columns ...
|
||||
skill_id TEXT, -- entity ID of skill used
|
||||
skill_version INTEGER, -- SG9: snapshot version at exec time
|
||||
-- ...
|
||||
);
|
||||
|
||||
-- Fix: skills versioning preserves history (SG9)
|
||||
-- Change PK to composite so old versions remain queryable
|
||||
CREATE TABLE skills (
|
||||
id TEXT NOT NULL, -- 'skill-restart-service'
|
||||
version INTEGER NOT NULL DEFAULT 1,
|
||||
name TEXT NOT NULL,
|
||||
ts TIMESTAMPTZ DEFAULT now(),
|
||||
procedure TEXT NOT NULL,
|
||||
applies_to TEXT REFERENCES entity_types(name),
|
||||
pattern_ids TEXT[],
|
||||
status TEXT DEFAULT 'drafted',
|
||||
success_rate REAL,
|
||||
last_used_at TIMESTAMPTZ,
|
||||
PRIMARY KEY (id, version) -- was: id TEXT PRIMARY KEY
|
||||
);
|
||||
CREATE TABLE skill_versions_audit ( -- SG9: track procedure changes
|
||||
id SERIAL PRIMARY KEY,
|
||||
skill_id TEXT NOT NULL,
|
||||
version INTEGER NOT NULL,
|
||||
changed_at TIMESTAMPTZ DEFAULT now(),
|
||||
changed_by TEXT, -- agent or operator entity
|
||||
diff TEXT, -- diff of procedure field
|
||||
reason TEXT
|
||||
);
|
||||
```
|
||||
|
||||
**SA5 — Classification entity must be persisted:**
|
||||
|
||||
Add a `classifications` table. Every classifier decision is recorded with the full
|
||||
reasoning — this is the audit trail for autonomous decisions.
|
||||
|
||||
```sql
|
||||
CREATE TABLE classifications (
|
||||
id SERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ DEFAULT now(),
|
||||
signal_entity_id TEXT REFERENCES entities(id),
|
||||
entity_id TEXT REFERENCES entities(id),
|
||||
action TEXT NOT NULL,
|
||||
risk_class TEXT NOT NULL,
|
||||
route TEXT NOT NULL, -- 'auto-act' or 'escalate'
|
||||
blast_radius TEXT[],
|
||||
pattern_confidence REAL,
|
||||
skill_match TEXT, -- skill entity ID if matched
|
||||
autonomy_check TEXT, -- 'allowed' or 'blocked: <reason>'
|
||||
reasoning JSONB NOT NULL, -- full decision explanation
|
||||
correlation_id TEXT
|
||||
);
|
||||
CREATE INDEX idx_class_signal ON classifications(signal_entity_id);
|
||||
CREATE INDEX idx_class_entity ON classifications(entity_id);
|
||||
```
|
||||
|
||||
**SG2 — CAGG array_agg unsupported:**
|
||||
|
||||
Drop `representative_tags` from the 1h continuous aggregate. Query tags from raw
|
||||
data when needed.
|
||||
|
||||
```sql
|
||||
CREATE MATERIALIZED VIEW metric_rollups_1h
|
||||
WITH (timescaledb.continuous) AS
|
||||
SELECT
|
||||
time_bucket('1 hour', ts) AS bucket,
|
||||
entity_id,
|
||||
metric,
|
||||
avg(value) AS avg_value,
|
||||
min(value) AS min_value,
|
||||
max(value) AS max_value,
|
||||
count(*) AS sample_count
|
||||
-- removed: (array_agg(tags))[1] AS representative_tags
|
||||
FROM metric_samples
|
||||
GROUP BY bucket, entity_id, metric;
|
||||
```
|
||||
|
||||
**SG3 — TimescaleDB functions not idempotent:**
|
||||
|
||||
Use `if_not_exists => TRUE` and exception guards:
|
||||
|
||||
```sql
|
||||
SELECT create_hypertable('metric_samples', 'ts',
|
||||
chunk_time_interval => INTERVAL '7 days',
|
||||
if_not_exists => TRUE);
|
||||
|
||||
DO $$ BEGIN
|
||||
PERFORM add_retention_policy('metric_samples', INTERVAL '90 days');
|
||||
EXCEPTION WHEN OTHERS THEN NULL;
|
||||
END $$;
|
||||
```
|
||||
|
||||
### Ontology fixes (HIGH)
|
||||
|
||||
**SA2 — Layer boundary "cognition creates governance" arrow:**
|
||||
|
||||
Remove the unsupported arrow. The learning engine does NOT write to governance
|
||||
(policy/autonomy) tables. Instead, validated patterns *propose* autonomy changes as
|
||||
approval-request entities — the operator must accept. This keeps governance
|
||||
authoritative and prevents the learning-poisoning vector (S4).
|
||||
|
||||
Also rename the cognition subgraph node from "Governance" to "Approvals" to avoid
|
||||
the layer-name collision.
|
||||
|
||||
**SA3 — Missing entity types (Person, Agent, IdentityProvider):**
|
||||
|
||||
Add BDD definitions and entity_type seeds:
|
||||
|
||||
```
|
||||
Person: matrix_id, oidc_sub
|
||||
Agent: provider, model, gateway_port
|
||||
IdentityProvider: issuer, client_id, auth_mode (oidc/forward-auth)
|
||||
```
|
||||
|
||||
These are first-class governance-layer entities. `audit_log.actor_id` now resolves
|
||||
to a real entity.
|
||||
|
||||
**SA4 — Lifecycle dead-ends and missing transitions:**
|
||||
|
||||
Updated infrastructure lifecycle (add `failed` state):
|
||||
```
|
||||
planned → provisioning → active → migrating → deprecated → destroyed
|
||||
↘ failed ↗ ↘ failed ↗
|
||||
failed → active (recovery) | failed → deprecated (write-off)
|
||||
planned → destroyed (cancel)
|
||||
deprecated → active (un-deprecate if replacement fails)
|
||||
```
|
||||
|
||||
Updated signal lifecycle (add terminal states):
|
||||
```
|
||||
raised → acknowledged → acting → resolved | failed
|
||||
acknowledged → resolved (manual resolve without acting)
|
||||
acknowledged → muted
|
||||
acting → failed (permanent failure, terminal — needs operator)
|
||||
failed → acknowledged (operator retries)
|
||||
```
|
||||
|
||||
Updated execution lifecycle (add cancellation + rollback failure):
|
||||
```
|
||||
approved → expired (TTL ran out)
|
||||
executing → cancelled (operator abort)
|
||||
failed → rolled_back | rollback_failed
|
||||
timed_out → verifying (check if the command actually completed despite timeout)
|
||||
```
|
||||
|
||||
Updated pattern lifecycle:
|
||||
```
|
||||
hypothesized → validated → active → deprecated
|
||||
hypothesized → invalidated (disproven, terminal)
|
||||
active → invalidated (new evidence contradicts)
|
||||
```
|
||||
|
||||
Updated skill lifecycle:
|
||||
```
|
||||
drafted → tested → active → refined → active (new version)
|
||||
drafted → deprecated (abandoned)
|
||||
tested → failed → drafted (back to drawing board)
|
||||
active → deprecated (superseded or unsafe)
|
||||
```
|
||||
|
||||
Approval lifecycle (new diagram — was missing):
|
||||
```
|
||||
pending → approved | denied | expired
|
||||
approved → revoked (operator changes mind before execution)
|
||||
```
|
||||
|
||||
**SA6 — recommended_action data source:**
|
||||
|
||||
Move `recommended_action` from `signals` to `classifications`. The probe raises a
|
||||
signal (kind, severity, evidence). The classifier populates the classification with
|
||||
the recommended action based on signal kind + skill lookup. The actuator reads the
|
||||
classification, not the signal, for the action to take.
|
||||
|
||||
**SA8 — Missing entities (Cluster, ComposeStack, ManagedHost):**
|
||||
|
||||
Add to ontology:
|
||||
- `Cluster` entity (software domain) — `ProxmoxHost` `member-of` `Cluster`
|
||||
- `ComposeStack` entity (software domain) — `DockerContainer` `part-of` `ComposeStack`
|
||||
- Add `provider`, `control_level` (`full`/`partial`/`none`) to `StandaloneServer`
|
||||
|
||||
### Security remediation (HIGH)
|
||||
|
||||
**S1 — SSH keys in containers:**
|
||||
|
||||
Dual approach (operator decision):
|
||||
1. **Phase 1-2 (immediate):** Restricted SSH key — dedicated key pair with
|
||||
`command="..."` and `from="..."` restrictions in `authorized_keys` on hubris/
|
||||
strong. The key can only run specific commands (pct, qm, df, systemctl status),
|
||||
not arbitrary shells. Mounted read-only into the actuator container only (not
|
||||
Hermes).
|
||||
2. **Phase 3 (actuator build):** Full actuator gateway — the API's `/exec` endpoint
|
||||
brokers all SSH. The actuator holds the keys, executes per-action, logs every
|
||||
command. Hermes never touches SSH.
|
||||
|
||||
**S2 — MCP auth:**
|
||||
- Shared secret between API and Hermes (HMAC-signed requests)
|
||||
- MCP bound to a dedicated Docker network (not the default bridge)
|
||||
- Never exposed via Caddy without auth
|
||||
|
||||
**S3 — Policy DB mutability:**
|
||||
- Policy mutations (`risk_classes`, `approval_rules`, `autonomy_settings`) require a
|
||||
meta-approval: the operator must approve the policy change itself (dual-control)
|
||||
- Immutable audit log of all policy changes with before/after hash
|
||||
- Startup self-check: compute policy hash, alert if differs from last-known-good
|
||||
|
||||
**S4 — Learning model poisoning:**
|
||||
- Pattern transitions to `active` require operator confirmation (`PATCH /api/v1/
|
||||
patterns/{id}` with `status=active` — policy-gated as `config_mutation`)
|
||||
- Confidence capped by sample size: `confidence = min(raw_confidence, N/5)` where N
|
||||
= evidence_count (requires N≥5 for confidence > 0.2)
|
||||
- Anomaly detection: if >10 identical-outcome feedback entries arrive within 1 hour
|
||||
for the same (entity_type, action), quarantine the pattern for review
|
||||
- Skills can never auto-promote to destructive risk class — always escalate
|
||||
|
||||
**S5 — confirmation_phrase replacement:**
|
||||
- Replace with single-use signed approval tokens
|
||||
- Token = HMAC(approval_id + entity_id + action + risk_class + nonce, shared_secret)
|
||||
- Stored hashed in `approvals` table
|
||||
- Transmitted via Matrix as the approval ID + decision; token verified server-side
|
||||
|
||||
**SA10 — Gateway + Caddy trust boundary:**
|
||||
- Port 8092 (Hermes gateway): mTLS or token auth. Mesh membership is the network
|
||||
boundary; gateway auth is the application boundary.
|
||||
- Caddy is an explicit trust root. API validates OIDC JWTs in middleware (not just
|
||||
trusting Caddy headers). Documented: compromising Caddy ≠ compromising the API.
|
||||
|
||||
### Operational remediation (HIGH)
|
||||
|
||||
**A3 + O3 + O4 — Backup, restore, DR:**
|
||||
|
||||
Backup strategy:
|
||||
- **Daily `pg_dump`** (compressed, custom format) + WAL archiving for PITR
|
||||
- **Off-host storage:** Proton Drive (cloud object storage, encrypted at rest)
|
||||
- **Push via rclone** from the scheduler container (already have rclone LXC in the
|
||||
fleet — reuse credentials)
|
||||
- **Retention:** 30 daily + 12 monthly snapshots
|
||||
- **Infisical backup:** Infisical has its own backup mechanism; also export secrets
|
||||
to an encrypted SOPS file as a fallback (chicken-and-egg: keep one age key for
|
||||
this purpose)
|
||||
- **Hermes volume:** backed up with `pg_dump` of agent_activity + session data
|
||||
|
||||
Restore procedure (`docs/operations/backup-restore.md`):
|
||||
1. Restore Postgres: `pg_restore -d oikos < dump.psql`
|
||||
2. Verify seed ingest matches (run `GET /api/v1/export` and diff against seed YAML)
|
||||
3. Restore Infisical from its backup
|
||||
4. `docker compose up -d`
|
||||
5. Monthly restore drill (scheduled, automated, alert if restore fails)
|
||||
|
||||
DR plan:
|
||||
- **RTO:** 4 hours (fresh machine → Docker → restore → running)
|
||||
- **RPO:** 24 hours (last daily backup)
|
||||
- **Cold-start runbook:** install Docker → clone repo → restore Infisical →
|
||||
restore DB → `docker compose up -d` → verify health
|
||||
- **Off-host backup target:** Proton Drive (encrypted, offsite)
|
||||
|
||||
**O1 — Rollback strategy:**
|
||||
- **Forward-only migrations** (no `down.sql` beyond development). Compensating
|
||||
migrations for production rollbacks.
|
||||
- **Pre-deploy DB backup:** the deploy script runs `pg_dump` before `docker compose
|
||||
up -d`
|
||||
- **Migration compatibility:** new code must tolerate old schema for one deploy
|
||||
window (additive migrations only — new columns nullable, new tables optional)
|
||||
- **Rollback runbook:** revert git commit → `pg_restore` from pre-deploy backup →
|
||||
`docker compose up -d` with old image
|
||||
|
||||
**O2 — External watchdog:**
|
||||
- Cron job on apps/105 (outside the Docker stack): `curl -sf
|
||||
http://mac-mini:8090/healthz || curl -X POST matrix-webhook ...`
|
||||
- Runs every 5 minutes
|
||||
- Alerts operator directly via Matrix if the API is unreachable
|
||||
- Also checks: Docker daemon running (`docker info`), Postgres accepting
|
||||
connections (`pg_isready`)
|
||||
|
||||
**M1 — CI/CD:**
|
||||
- Gitea Actions (or simple webhook + script):
|
||||
- `go vet ./...`
|
||||
- `golangci-lint run`
|
||||
- `go test ./... -race -cover`
|
||||
- `docker build` (no push — just verify it builds)
|
||||
- Webhook deploy gated on green CI
|
||||
- Deploy script runs `go test` as a final safety check before `docker compose up`
|
||||
|
||||
### Architecture remediation (HIGH)
|
||||
|
||||
**A1 — Testing strategy:**
|
||||
|
||||
Add testing workstream with specific tests:
|
||||
|
||||
| Package | Test type | What to test |
|
||||
|---|---|---|
|
||||
| `internal/policy/` | Unit | Classifier scoring: risk × blast × confidence. Table-driven: every (risk_class, blast_radius, confidence) combination. Edge: unknown action, ambiguous entity. |
|
||||
| `internal/policy/` | Unit | Approval lifecycle: token issue, verify, single-use enforcement, TTL expiry. |
|
||||
| `internal/learning/` | Unit | Pattern confidence calculation. Skill versioning. Feedback → pattern extraction. |
|
||||
| `internal/ontology/` | Unit | Lifecycle transition validation: every legal transition succeeds, every illegal one fails. Graph traversal (mock relationships). |
|
||||
| `internal/db/` | Integration | testcontainers Postgres: seed ingest idempotency, blast_radius CTE (with cycles), hypertable insert + query, continuous aggregate refresh. |
|
||||
| `internal/api/` | Integration | testcontainers: REST routes return correct status codes, MCP tools return expected shapes, audit middleware records entries, pagination works. |
|
||||
| `internal/actuator/` | Integration | Mock SSH: execution → verification → feedback → pattern update. Loop-guard prevents retry storms. Circuit breaker trips after N failures. |
|
||||
| `migrations/` | Property | Every migration is forward-only. `blast_radius` returns correct results on cyclic graphs. Hypertable retention doesn't drop data younger than threshold. |
|
||||
| `internal/observability/` | Unit | Correlation ID propagation through context. Event emitter transactional with state change. Metric recording. |
|
||||
|
||||
Coverage gate: ≥80% on `internal/policy/` and `internal/learning/` (the autonomy-
|
||||
granting code). ≥60% on everything else.
|
||||
|
||||
**A2 — Observability:** Already addressed (Migration 6, Workstream 14). Update audit
|
||||
status to resolved.
|
||||
|
||||
**SG4 — Graceful shutdown:**
|
||||
|
||||
Every `cmd/*/main.go` implements:
|
||||
1. `signal.NotifyContext(ctx, syscall.SIGTERM, syscall.SIGINT)`
|
||||
2. Context propagated to all long-running loops and HTTP server
|
||||
3. Shutdown sequence: stop accepting new work → wait for in-flight (30s deadline) →
|
||||
for actuator: if execution in-flight, mark `failed` with "shutdown interrupted" +
|
||||
emit feedback → close DB pool
|
||||
4. `docker-compose.yml`: `stop_grace_period: 30s` on actuator, `stop_signal: SIGTERM`
|
||||
on all services
|
||||
|
||||
**SG5 — Entity-level concurrency:**
|
||||
|
||||
```sql
|
||||
-- Per-entity advisory lock during execution (prevents concurrent actions
|
||||
-- on the same target entity, e.g., restart + deploy on the same service)
|
||||
SELECT pg_advisory_xact_lock(hashtext($1)); -- $1 = target_entity_id
|
||||
-- ... execute, verify, feedback ...
|
||||
-- lock released on transaction commit/rollback
|
||||
```
|
||||
|
||||
**SG6 — Domain layer:**
|
||||
|
||||
Add `internal/domain/` package:
|
||||
```
|
||||
internal/domain/
|
||||
├── entity.go # Entity, EntityType, Relationship domain types
|
||||
├── signal.go # Signal domain type + lifecycle transition logic
|
||||
├── execution.go # Execution domain type + state machine
|
||||
├── classification.go # Classification domain type
|
||||
├── pattern.go # Pattern domain type + confidence calculation
|
||||
├── skill.go # Skill domain type + versioning
|
||||
├── approval.go # Approval domain type + token verification
|
||||
└── errors.go # Sentinel errors: ErrNotFound, ErrInvalidTransition,
|
||||
# ErrApprovalRequired, ErrAutonomyBlocked, ErrConflict
|
||||
```
|
||||
|
||||
DB ↔ domain mapping in `internal/db/` (repository pattern). API handlers accept/
|
||||
return domain types. sqlc models never escape `internal/db/`.
|
||||
|
||||
**SG11 — Error handling:**
|
||||
|
||||
```go
|
||||
// internal/domain/errors.go
|
||||
var (
|
||||
ErrNotFound = errors.New("entity not found")
|
||||
ErrInvalidTransition = errors.New("invalid lifecycle transition")
|
||||
ErrApprovalRequired = errors.New("operator approval required")
|
||||
ErrAutonomyBlocked = errors.New("autonomy policy blocks this action")
|
||||
ErrConflict = errors.New("concurrent modification conflict")
|
||||
ErrCircuitOpen = errors.New("circuit breaker open for target")
|
||||
)
|
||||
```
|
||||
|
||||
HTTP mapping middleware: `ErrNotFound → 404`, `ErrInvalidTransition → 409`,
|
||||
`ErrApprovalRequired → 403`, `ErrAutonomyBlocked → 403`, `ErrConflict → 409`,
|
||||
`ErrCircuitOpen → 503`.
|
||||
|
||||
SSH error classification in `internal/actuator/execute.go`:
|
||||
- Network unreachable → retryable, circuit breaker
|
||||
- Auth failure → fatal, alert operator
|
||||
- Command exit non-zero → execution failed, feedback
|
||||
- Command timeout → timed_out, feedback
|
||||
|
||||
DB retry: serialization failures (SQLSTATE 40001, 40P01) → retry with exponential
|
||||
backoff (max 3 retries).
|
||||
|
||||
### Go implementation fixes (MEDIUM)
|
||||
|
||||
**SA9 — TimescaleDB Docker image + migration runner:**
|
||||
- Image: `timescale/timescaledb:2.x-pg16` (not `postgres:16`)
|
||||
- Migrations run in a one-shot init container (`compose/migrate/Dockerfile`) with a
|
||||
dedicated DB user that has DDL but no runtime data privileges
|
||||
- API's DB user gets DML only (least privilege)
|
||||
- `golang-migrate` Go API with `//go:embed migrations/*.up.sql`
|
||||
|
||||
**SG7 — Pattern/skill management endpoints:**
|
||||
- `PATCH /api/v1/patterns/{id}` — state transition (validate/invalidate/deprecate),
|
||||
policy-gated as `config_mutation`, audit-logged
|
||||
- `PATCH /api/v1/skills/{id}` — state transition + version pin, same gating
|
||||
- This is the operator's manual safety valve for learning-model issues (S4)
|
||||
|
||||
**SG8 — WebSocket push mechanism:**
|
||||
- In-process event bus (Go channel pub/sub) for events written by the API itself
|
||||
(zero-latency push to WebSocket subscribers)
|
||||
- Postgres `LISTEN/NOTIFY` for events written by other services (scheduler,
|
||||
actuator) — trigger on `events` table fires NOTIFY after commit
|
||||
- Both feed the WebSocket handler
|
||||
|
||||
**SG10 — Transactional event emission:**
|
||||
- Event + audit entries written in the same DB transaction as the state change
|
||||
- If transaction rolls back, events are discarded (never emitted)
|
||||
- `LISTEN/NOTIFY` fires after commit — subscribers only see committed events
|
||||
|
||||
**SG13 — Context-aware SSH:**
|
||||
```go
|
||||
func runSSH(ctx context.Context, client *ssh.Client, cmd string) ([]byte, error) {
|
||||
session, err := client.NewSession()
|
||||
if err != nil { return nil, err }
|
||||
defer session.Close()
|
||||
type result struct { out []byte; err error }
|
||||
ch := make(chan result, 1)
|
||||
go func() { out, err := session.CombinedOutput(cmd); ch <- result{out, err} }()
|
||||
select {
|
||||
case r := <-ch: return r.out, r.err
|
||||
case <-ctx.Done():
|
||||
session.Close() // unblocks CombinedOutput
|
||||
client.Close()
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**SG14 — Connection pool sizing:**
|
||||
- API: 15 connections, scheduler: 5, actuator: 5, learning: 3 = 28 total
|
||||
- Postgres `max_connections` set to 80
|
||||
- Monitor `db_connections_active`, alert if >80% of pool
|
||||
|
||||
**SG15 — RESTful exec endpoint:**
|
||||
- `POST /api/v1/executions` (was `POST /api/v1/exec`) — creates an execution
|
||||
resource. Handler classifies, checks approval, creates execution in `proposed`
|
||||
state.
|
||||
- `GET /api/v1/executions/{id}` — status
|
||||
- `POST /api/v1/executions/{id}/cancel` — cancellation
|
||||
|
||||
**SG16 — Pagination:**
|
||||
- All hypertable-backed endpoints: cursor-based (`?cursor=<ts>&limit=50`)
|
||||
- `entities` and other small tables: keyset pagination (`?after=<id>&limit=50`)
|
||||
- Default limit: 50, max: 200
|
||||
- MCP tools support `limit` parameter
|
||||
|
||||
**SG17 — Go tooling:**
|
||||
- `sqlc.yaml` added to repo layout
|
||||
- Module path: `github.com/dtoro/oikos`
|
||||
- `CGO_ENABLED=0` in Dockerfiles, `pgx` (pure Go), `gcr.io/distroless/static`
|
||||
runtime image
|
||||
- Migrations embedded with `//go:embed`
|
||||
|
||||
**SG18 — Health/metrics bypass auth:**
|
||||
- `/healthz` and `/metrics` on a separate Gin router group, no auth, no audit
|
||||
- `/healthz`: `SELECT 1` against DB
|
||||
- `/metrics`: internal-only (not exposed via Caddy), or token-protected
|
||||
|
||||
**SA7 — Notifier decoupling:**
|
||||
- API writes `approvals` row (status=pending) + emits `approval.requested` event
|
||||
- Notifier polls pending approvals, sends to Matrix, writes decision directly to
|
||||
`approvals` table (has DB access, not API access)
|
||||
- API polls `approvals.status`
|
||||
- No service-to-service calls in either direction — DB is the rendezvous point
|
||||
- Pending approvals survive Notifier restart
|
||||
|
||||
**SA9 + SA10 — Threat model documentation:**
|
||||
- Caddy is an explicit trust root (compromise = API compromise, mitigated by JWT
|
||||
validation in API middleware)
|
||||
- Mesh membership is the network boundary for Hermes gateway
|
||||
- Docker network is the trust boundary for internal services (mTLS between API and
|
||||
Hermes, TLS to Postgres)
|
||||
- The actuator is the only container with SSH egress (not Hermes, not the API)
|
||||
|
||||
### Updated phasing (incorporating remediation)
|
||||
|
||||
**Phase 0 — Ontology design (no code):**
|
||||
- Finalize entity types (including Person, Agent, IdentityProvider, Cluster,
|
||||
ComposeStack) + relationship types + lifecycles (all with terminal states)
|
||||
- Write seed manifests
|
||||
- Review diagrams + lifecycle completeness with operator
|
||||
|
||||
**Phase 1 — Foundation (Go + DB):**
|
||||
- Go module setup (`github.com/dtoro/oikos`), project structure with domain layer
|
||||
- PostgreSQL + TimescaleDB (`timescale/timescaledb:2.x-pg16`)
|
||||
- Migrations 1-6 (with fixed hypertable PKs, idempotent TimescaleDB calls, dual
|
||||
entity pattern for cognition objects, classifications table)
|
||||
- Migration runner as init container (DDL-only DB user)
|
||||
- Seed ingest pipeline (transactional, init container)
|
||||
- sqlc + domain layer (repository pattern)
|
||||
- Structured logging (slog) + error sentinels + HTTP error mapping
|
||||
- **Testing foundation:** testcontainers setup, unit test framework, coverage gates
|
||||
- **Backup setup:** daily `pg_dump` + rclone push to Proton Drive, WAL archiving
|
||||
- **External watchdog:** cron on apps/105
|
||||
|
||||
**Phase 2 — API (Go):**
|
||||
- Gin server with REST routes (resource-oriented, paginated, `/executions` not
|
||||
`/exec`)
|
||||
- MCP protocol adapter (shared-secret auth, dedicated network)
|
||||
- Policy enforcement middleware (OIDC JWT validation, not just Caddy headers)
|
||||
- Audit middleware (transactional with state changes)
|
||||
- Event emitter (transactional, in-process bus + LISTEN/NOTIFY)
|
||||
- Observability routes (metrics, trends, audit, events, health, agent-activity)
|
||||
- WebSocket (in-process bus + LISTEN/NOTIFY, bounded channels, backpressure)
|
||||
- Pattern/skill management endpoints (operator override for learning model)
|
||||
- Domain layer fully fleshed out
|
||||
- **CI:** Gitea Actions (go vet, golangci-lint, go test -race -cover, docker build)
|
||||
|
||||
**Phase 3 — Control loop (Go):**
|
||||
- Scheduler (Observe) — probes, signals, state snapshots, metric recording
|
||||
- Actuator (Act) — classify, execute, verify, correlation ID propagation
|
||||
- **Restricted SSH key** (command= in authorized_keys, actuator-only)
|
||||
- **Entity-level advisory locks** (pg_advisory_xact_lock)
|
||||
- **Context-aware SSH** (context cancellation, hard timeout)
|
||||
- **Circuit breaker** per target host
|
||||
- **Graceful shutdown** (SIGTERM, in-flight protection, stop_grace_period: 30s)
|
||||
- Learning engine — feedback, patterns, skills, learning metrics
|
||||
- **Pattern activation requires operator confirmation** (PATCH endpoint)
|
||||
- **Confidence capped by sample size** (N≥5)
|
||||
- **Anomaly detection** for feedback bursts
|
||||
- Approval tokens (single-use HMAC, not confirmation phrases)
|
||||
- Policy meta-approval (dual-control for policy mutations)
|
||||
- Notifier (DB rendezvous, no service-to-service calls, Matrix impl)
|
||||
|
||||
**Phase 4 — Agent (Hermes container):**
|
||||
- Hermes Docker image, gateway config (mTLS on port 8092)
|
||||
- Homelab skills
|
||||
- Agent activity logging
|
||||
- Connect from workstation, verify MCP (shared-secret auth) + SSH via actuator
|
||||
|
||||
**Phase 5 — Secrets (Infisical):**
|
||||
- Stand up Infisical, migrate SOPS, wire services
|
||||
- Infisical backup + SOPS fallback (one age key kept for DR)
|
||||
|
||||
**Phase 6 — Deploy + cutover:**
|
||||
- Docker Compose (timescale image, init containers, pool sizing, stop_grace_period)
|
||||
- Gitea webhook (HMAC auth, non-root deploy user, CI-gated)
|
||||
- Caddy re-point + JWT validation in API
|
||||
- End-to-end verification (14 checks)
|
||||
- Stop apps/105, clean up mac-mini
|
||||
- **Restore drill** (monthly, automated)
|
||||
|
||||
Reference in New Issue
Block a user