dtoro
d80a394b7f
Documentation and repo-hygiene pass following the client/server split:
Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.
Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).
Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).
Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.
Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.
Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
GetClientContext handler) since the mechanism's introduction on
2026-06-02 — never matched any real filename, so no client has ever
picked up an auto-setup script via git-pull or the context-poller sync.
Fixed all three; the Go server-side fix is the one that actually matters
since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
(parsed, never consumed) left over from an earlier clone-based model.
Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
253 lines
12 KiB
Markdown
253 lines
12 KiB
Markdown
# 2026-07-08 — Oikos gaps, broken things, and improvements
|
|
|
|
**Status:** In Progress — audited 2026-07-11, re-audited 2026-07-12 for
|
|
drift from the `cmd/hermes`→`cmd/nomos` rename and later fixes. Done: A1
|
|
(approval FK bug), A3 (Hermes→Nomos help text), **Section C** (toy NLU /
|
|
silent-wrong-answer fallback — nomos now calls real `listTools()` and
|
|
routes unmatched queries to `/chat` instead of guessing, per
|
|
`cmd/nomos/main.go:444-465`), **D.5** (SOUL.md/actuator architecture
|
|
mismatch — `nomos/SOUL.md:21-22,40` now accurately documents SSH via the
|
|
`run` tool), D1 (`upsert_knowledge`), D4-partial (general `run` tool).
|
|
Still open: A2 (notifier flooding/dedup), A4 (`resolveHost` dead code), A5
|
|
(`queryRows` stringly-typed columns), A6 (stale `get_state_snapshot`
|
|
description), B1-B5 (enrollment auth, fake Infisical creds, `/query`
|
|
mesh-only auth unenforced, insecure host key checking, optional
|
|
`caller_pubkey`), D2/D3 (no `get_approval_status`/`list_pending_approvals`/
|
|
signal ack-resolve-mute tools), E-partial (Caddyfile placeholders still
|
|
present; tool count now 33, documented in AGENTS.md as of 2026-07-12).
|
|
2026-07-12 re-audit also refreshed every `cmd/hermes`→`cmd/nomos` and
|
|
`internal/mcp/server.go` line-number citation below (the file grew from 28
|
|
to 33 registered tools since 2026-07-11) — content/status of each finding
|
|
unchanged, only citations moved.
|
|
|
|
## Goal
|
|
|
|
Full-project review of Oikos from two vantage points — a user interacting
|
|
through Hermes, and an agent working through the MCP tool surface — with every
|
|
finding verified against source (file:line), plus a prioritized fix order.
|
|
This plan is the map; each numbered fix is small enough to land independently.
|
|
|
|
---
|
|
|
|
## A. Confirmed bugs (verified in source)
|
|
|
|
### A1. Approvals are never created — FK violation, errors swallowed (CRITICAL)
|
|
|
|
`createApproval` at `internal/mcp/server.go:970` inserts a fresh
|
|
`uuid.NewV7()` as `approvals.entity_id`, but
|
|
`migrations/003_operations.up.sql:41` declares
|
|
`entity_id UUID PRIMARY KEY REFERENCES entities(id)`. The INSERT always
|
|
violates the FK, and both `pool.Exec` errors are discarded.
|
|
|
|
Net effect: `request_execution` for `systemctl enable/disable` or
|
|
`apt_upgrade` marks the execution `pending_approval`
|
|
(`internal/mcp/server.go:311`, `:366`) and tells the agent it's queued, but no
|
|
approval row exists → the notifier never sends a Matrix alert → the execution
|
|
is orphaned forever. From the Hermes user's perspective, config mutations
|
|
silently dead-end while appearing accepted.
|
|
|
|
Contrast: `request_execution` (server.go:286) correctly creates a companion
|
|
`entities` row for the execution first — approvals just never got the same
|
|
treatment.
|
|
|
|
**Fix:**
|
|
- Preferred: migrate `approvals` to its own `id UUID PRIMARY KEY` (not FK'd to
|
|
`entities`), keeping `subject_entity_id` as the entity link. Update notifier
|
|
and `DecideApproval` queries accordingly.
|
|
- Alternative (no migration): create a companion `entities` row like
|
|
executions do.
|
|
- Either way: check and log every `Exec` error in `createApproval`, and verify
|
|
the `UPDATE executions SET approval_id = $2 WHERE entity_id = $1` column
|
|
semantics (`executions` is also keyed by `entity_id`).
|
|
|
|
### A2. Matrix message-flooding vectors (`internal/notifier/notifier.go`)
|
|
|
|
- Initial alert is guarded by `alert_sent_at` (notifier.go:103), but the guard
|
|
is written *after* the Matrix send (notifier.go:111). If the UPDATE fails
|
|
after a successful send, the ticker re-sends every cycle.
|
|
- No dedup of approvals by `(subject_entity_id, action, payload)`. Once A1 is
|
|
fixed, every retried `request_execution` mints a new approval → one Matrix
|
|
message each.
|
|
- `pollReactions` (notifier.go:118) issues one Matrix relations GET per
|
|
pending approval per 30s poll, uncapped — ignored approvals accumulate for
|
|
their 1h lifetime and multiply API calls. It also re-dispatches decisions
|
|
for approvals stuck `pending` (no "already acted" guard if the
|
|
DecideApproval call errors without flipping status).
|
|
|
|
**Fix:** mark-then-send (or transactional outbox) for `alert_sent_at`; upsert/
|
|
dedup open approvals on `(subject_entity_id, action, payload)`; cap + backoff
|
|
on reaction polling; guard against re-dispatching a decision already in
|
|
flight.
|
|
|
|
### A3. Hermes "help" is broken + dead code
|
|
|
|
`cmd/hermes/main.go:165` handles "what can you do"/"help" by calling
|
|
`client.callTool("tools/list", nil)` — a `tools/call` for a tool literally
|
|
named `tools/list`, which doesn't exist. The correct `listTools()` helper
|
|
(main.go:318) is dead code, never called. **Fix:** wire `listTools()` in.
|
|
|
|
### A4. `resolveHost` never returns a per-entity SSH user
|
|
|
|
`internal/mcp/server.go:1222` (was :943 — line moved) — the named return
|
|
`sshUser` is always `""`; the per-entity user branch is dead and everything
|
|
relies on `sshExec`'s global default fallback. **Fix:** read the SSH user
|
|
from entity attributes or delete the dead return to make the behavior
|
|
honest.
|
|
|
|
### A5. `queryRows` stringifies every column
|
|
|
|
`internal/mcp/server.go:1090` (was :861 — line moved) renders all values via
|
|
`fmt.Sprintf("%v", ...)`, so numbers, bools, timestamps, and JSON all reach
|
|
agents as strings. **Fix:** type-preserving serialization (pass through
|
|
pgx-native values into `json.Marshal`) — improves every read tool at once.
|
|
|
|
### A6. `get_state_snapshot` description is stale
|
|
|
|
`internal/mcp/server.go:863` (was :689 — line moved) still advertises "disk,
|
|
drift count" — columns removed in commit 3ea43ad. **Fix:** update the
|
|
description.
|
|
|
|
---
|
|
|
|
## B. Security gaps
|
|
|
|
### B1. Enrollment is unauthenticated, with a false comment
|
|
|
|
`internal/httpapi/server.go:111` (was :97) says "unauthenticated (IP-gated
|
|
in handler)" but `EnrollClient` (`internal/httpapi/impl.go:1166`, was
|
|
:1099) performs no IP check at all — the only gate is the target entity
|
|
being in state `planned`/`provisioning`. Caddy's `@enroll` matcher bypasses
|
|
Authentik. Anyone reaching `oikos.hubris.network` who knows (or guesses) a
|
|
planned slug receives that node's **age private key** in the HTTP response
|
|
body. Still open — line numbers only, substance unchanged.
|
|
|
|
**Fix:** enforce a real gate (mesh-CIDR check, one-time enrollment token
|
|
minted when the entity is created, or both), and stop returning the age
|
|
private key in the response — have the client fetch it from the secret store.
|
|
|
|
### B2. Fake Infisical credentials returned to enrollees
|
|
|
|
`internal/httpapi/impl.go:1260-1261` (was :1191-1192) returns
|
|
`"inf_client_"+uuid` / `"inf_secret_"+uuid` — random strings wired to
|
|
nothing. Enrolled clients hold credentials that authenticate against
|
|
nothing. Still open — line numbers only, substance unchanged.
|
|
**Fix:** implement `CreateMachineIdentity` in `internal/secrets/infisical.go`,
|
|
or return no credentials and document the manual step.
|
|
|
|
### B3. Nomos's `/query` has no auth
|
|
|
|
`nomos/config.yaml:9` (was `hermes/config.yaml:9`) sets `mesh_only: true`
|
|
but `cmd/nomos/main.go` (was `cmd/hermes/main.go`) never reads or enforces
|
|
it — it serves any caller on :8092, who can invoke `request_execution`.
|
|
Still open, now also tracked as C1 in
|
|
[2026-07-11-nomos-agent-code-review.md](2026-07-11-nomos-agent-code-review.md),
|
|
deferred by the operator. **Fix:** enforce mesh-CIDR (or bearer token) in
|
|
the handler; fail closed.
|
|
|
|
### B4. SSH host keys not verified
|
|
|
|
`ssh.InsecureIgnoreHostKey()` at `internal/mcp/server.go:1155` (was :920).
|
|
Still open — line number only, substance unchanged.
|
|
**Fix:** known_hosts pinning (keys are already inventory-managed per node).
|
|
|
|
### B5. `list_my_secrets` enumerates all node pubkeys
|
|
|
|
Without `caller_pubkey`, `internal/mcp/server.go:879-883` (was :709-720)
|
|
returns every entity that has an `age_pubkey`; nothing ties the caller to
|
|
what it may list. Still open — line numbers only, substance unchanged.
|
|
**Fix:** require `caller_pubkey` and scope results to the caller's
|
|
entitlements.
|
|
|
|
---
|
|
|
|
## C. User perspective (interacting via Hermes) — RESOLVED
|
|
|
|
**Resolved as of the Hermes→Nomos rewrite (verified 2026-07-12).** This
|
|
entire section described `cmd/hermes`, which no longer exists — Hermes was
|
|
renamed and rebuilt as `cmd/nomos`, a real LLM-backed agent loop, which is
|
|
exactly the recommendation below. `cmd/nomos/main.go:444-465` now calls the
|
|
real `listTools()` for "help"/"what can you do", and routes unmatched
|
|
queries to "natural language queries belong to `/chat`..." instead of
|
|
silently falling back to `get_health_summary`. Kept below for history —
|
|
original text unchanged.
|
|
|
|
- `routeQuery` NLU is hardcoded `strings.Contains`; `extractEntity`
|
|
(`cmd/hermes/main.go:173`) recognizes only 5 services (`authentik, caddy,
|
|
vaultwarden, gitea, immich`) plus `mac-mini`/`hubris`. Any other entity →
|
|
"no entity found", and *any* unmatched query silently falls back to
|
|
`get_health_summary` — wrong answers that look like answers.
|
|
- No conversation/session context; no follow-up capability.
|
|
- Config mutations appear accepted but silently dead-end (A1).
|
|
|
|
**Recommendation:** either make Hermes a real LLM-backed agent loop (Claude
|
|
API driving the 28 MCP tools) or explicitly scope it as a structured-tool
|
|
gateway: remove the toy NLU, make the fallback say "I don't understand this
|
|
query; here are the tools" (via the fixed `listTools()`), and document that
|
|
natural language belongs to the calling agent, not the gateway.
|
|
|
|
---
|
|
|
|
## D. Agent perspective (MCP tooling gaps)
|
|
|
|
28 tools are registered in `internal/mcp/server.go` (README says 15, AGENTS.md
|
|
says 21 — both stale). Missing capabilities:
|
|
|
|
1. **No knowledge write.** AGENTS.md tells agents to register knowledge via
|
|
`POST /api/v1/knowledge/{slug}`, but there is no MCP tool — MCP-only agents
|
|
cannot write back what they learn. Add `upsert_knowledge`.
|
|
2. **No entity/signal mutation.** Create/patch entity, state transitions, and
|
|
signal ack/resolve/mute all exist in REST (`internal/httpapi/impl.go`,
|
|
`phase3.go`) but not in MCP. Add at least signal ack/resolve/mute and a
|
|
policy-gated entity attribute patch.
|
|
3. **No approval visibility.** After `request_execution` returns
|
|
`pending_approval`, an agent has no way to check or reference the approval.
|
|
Add `get_approval_status` / `list_pending_approvals`.
|
|
4. Execution actions limited to `restart | systemctl | pct_exec |
|
|
apt_upgrade` — no deploy/rollback/config-edit path. Partially
|
|
superseded: the general `run` MCP tool (D4-partial, done) covers
|
|
arbitrary commands now; `request_execution`'s fixed enum is still there
|
|
for the specific actions it names (see
|
|
[2026-07-10-general-gated-execution.md](2026-07-10-general-gated-execution.md)).
|
|
5. **RESOLVED (verified 2026-07-12).** Architecture/doc mismatch:
|
|
`hermes/SOUL.md` claimed "no SSH access; all mutations flow through the
|
|
actuator", but the MCP server ran `restart`/`pct_exec` synchronously over
|
|
SSH from inside the api process. `nomos/SOUL.md:21-22,40` now accurately
|
|
documents SSH access via the policy-gated `run` tool — matches the
|
|
architecture the general-gated-execution plan built. No longer a
|
|
mismatch.
|
|
|
|
---
|
|
|
|
## E. Doc drift / housekeeping
|
|
|
|
- **RESOLVED (verified 2026-07-12):** Tool counts. README 15 / AGENTS.md 21
|
|
/ actual 28 was already stale by 2026-07-11 (registered tools grew to
|
|
33) — AGENTS.md now documents all 33 with the full catalog (2026-07-12).
|
|
- **Still open:** `compose/caddy/Caddyfile.oikos` retains literal
|
|
`<mac-mini-mesh-ip>` placeholders (this repo's copy is a reference only —
|
|
see [2026-07-12-wails-desktop-app.md](2026-07-12-wails-desktop-app.md)'s
|
|
"Plan review" — the real config lives in `dtoro/caddy-conf`).
|
|
- **RESOLVED:** `.agents/HERMES.md` renamed to `.agents/NOMOS.md`; the
|
|
duplicate-line bug itself is still present at `.agents/NOMOS.md:11` —
|
|
only the file citation was stale, the underlying nit is still open.
|
|
- **RESOLVED (verified 2026-07-12):** `plans/index.md` drift — the broken
|
|
link, TRMNL/Grimmory Active/Done mismatch, and missing `.hermes/plans/`
|
|
entries described here are no longer present in the current
|
|
`plans/index.md`; already fixed sometime after this plan was written.
|
|
- **RESOLVED (verified 2026-07-12):** `plans/2026-07-05-oikos-prometheus-lxc.md`
|
|
already self-corrected both the deleted-file references and the LXC 131
|
|
collision in its own 2026-07-08 changelog — this bullet describes a
|
|
pre-fix state.
|
|
|
|
---
|
|
|
|
## F. Prioritized fix order
|
|
|
|
1. **A1** approval FK + error handling — unblocks the entire
|
|
approval → Matrix → execution path.
|
|
2. **A2** notifier flooding guards — this branch's namesake.
|
|
3. **B1/B2** enrollment security + **B3** Hermes auth.
|
|
4. **D** MCP tool additions — approval status first, then knowledge write,
|
|
then signal ops.
|
|
5. **C** Hermes routing honesty + **A3-A6**, **B4/B5**, **E** drift cleanup.
|