docs: fix plan/repo drift, retire dead Goose+Nomos and Caveman tooling
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled

Documentation and repo-hygiene pass following the client/server split:

Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
  described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
  refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
  deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
  to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.

Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).

Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).

Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.

Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.

Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
  places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
  GetClientContext handler) since the mechanism's introduction on
  2026-06-02 — never matched any real filename, so no client has ever
  picked up an auto-setup script via git-pull or the context-poller sync.
  Fixed all three; the Go server-side fix is the one that actually matters
  since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
  (parsed, never consumed) left over from an earlier clone-based model.

Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-12 18:19:41 +02:00
parent 0c0f35a3a9
commit d80a394b7f
43 changed files with 609 additions and 1430 deletions

View File

@@ -1,8 +1,11 @@
# 2026-07-08 — Liveness, drift, and UX cohesion
**Status:** In Progress — Phases 14 code complete; not yet deployed. Phase 5 deferred.
(Audited 2026-07-11 — still accurate; prompt caching within Phase 4 also
confirmed not implemented.)
**Status:** In Progress — Phases 14 code complete and now deployed
(re-verified 2026-07-12: mac-mini was redeployed from `main` that day for
unrelated auth work — plans/2026-07-12-wails-desktop-app.md — which carried
every commit up to that point, including this plan's, so "not yet deployed"
below is stale). Phase 5 deferred. (Audited 2026-07-11 — still accurate;
prompt caching within Phase 4 also confirmed not implemented.)
- **Phase 1 (drift/staleness):** done. Health/metrics/events misattribution
fix, staleness sweep, `/entities` health+freshness, dashboard/fleet-health
@@ -26,20 +29,19 @@ confirmed not implemented.)
- **Phase 4 (agent efficiency):** core piece done — prior turns' tool
calls/results are now replayed into the conversation (previously dropped
entirely), and a compact live fleet-health snapshot is injected into the
system prompt each turn so the agent starts oriented. Prompt caching and
reconsidering the default model are **not done** (lower priority, no
measured regression without them).
system prompt each turn so the agent starts oriented. Prompt caching is
**not done** (lower priority, no measured regression without it).
Reconsidering the default model — done, but not by this plan: switched to
`deepseek/deepseek-v4-pro` on 2026-07-10 (`cmd/nomos/agent.go:70`) for
reliability, per that commit's own comment ("the flash tier over-narrates,
occasionally emits canned refusals, and is unreliable at multi-step tool
use").
- **Phase 5 (CRUD):** `PatchEntity` and a full `/checks` CRUD API
(list/create/patch, including enable/disable) already existed server-side;
the new Monitoring card's toggle uses `PatchCheck`. **Not done**: a
"run check now" endpoint (no scheduler on-demand entrypoint exists yet),
relationship editing, and an entity attribute editor UI.
**Not yet deployed** — the live `oikos-api`/`oikos-scheduler`/nomos
containers still run the pre-fix binaries; rebuilding and restarting them
needs an explicit go-ahead since it touches the running homelab control
plane.
Addresses five felt problems with the current system: (1) the agent reports
stale machine state as if it were fresh, (2) sessions can't be opened and feel
disconnected from chat, (3) the Nomos agent re-derives state every turn and

View File

@@ -1,14 +1,24 @@
# 2026-07-08 — Oikos gaps, broken things, and improvements
**Status:** In Progress — audited 2026-07-11. Done: A1 (approval FK bug),
A3 (Hermes→Nomos help text), D1 (`upsert_knowledge`), D4-partial (general
`run` tool). Still open: A2 (notifier flooding/dedup), A4 (`resolveHost`
dead code), A5 (`queryRows` stringly-typed columns), A6 (stale
`get_state_snapshot` description), B1-B5 (enrollment auth, fake Infisical
creds, `/query` mesh-only auth unenforced, insecure host key checking,
optional `caller_pubkey`), D2/D3 (no `get_approval_status`/
`list_pending_approvals`/signal ack-resolve-mute tools), E (README tool
count, Caddyfile placeholders, NOMOS.md duplicate line).
**Status:** In Progress — audited 2026-07-11, re-audited 2026-07-12 for
drift from the `cmd/hermes``cmd/nomos` rename and later fixes. Done: A1
(approval FK bug), A3 (Hermes→Nomos help text), **Section C** (toy NLU /
silent-wrong-answer fallback — nomos now calls real `listTools()` and
routes unmatched queries to `/chat` instead of guessing, per
`cmd/nomos/main.go:444-465`), **D.5** (SOUL.md/actuator architecture
mismatch — `nomos/SOUL.md:21-22,40` now accurately documents SSH via the
`run` tool), D1 (`upsert_knowledge`), D4-partial (general `run` tool).
Still open: A2 (notifier flooding/dedup), A4 (`resolveHost` dead code), A5
(`queryRows` stringly-typed columns), A6 (stale `get_state_snapshot`
description), B1-B5 (enrollment auth, fake Infisical creds, `/query`
mesh-only auth unenforced, insecure host key checking, optional
`caller_pubkey`), D2/D3 (no `get_approval_status`/`list_pending_approvals`/
signal ack-resolve-mute tools), E-partial (Caddyfile placeholders still
present; tool count now 33, documented in AGENTS.md as of 2026-07-12).
2026-07-12 re-audit also refreshed every `cmd/hermes``cmd/nomos` and
`internal/mcp/server.go` line-number citation below (the file grew from 28
to 33 registered tools since 2026-07-11) — content/status of each finding
unchanged, only citations moved.
## Goal
@@ -78,22 +88,24 @@ named `tools/list`, which doesn't exist. The correct `listTools()` helper
### A4. `resolveHost` never returns a per-entity SSH user
`internal/mcp/server.go:943` — the named return `sshUser` is always `""`; the
per-entity user branch is dead and everything relies on `sshExec`'s global
default fallback. **Fix:** read the SSH user from entity attributes or delete
the dead return to make the behavior honest.
`internal/mcp/server.go:1222` (was :943 — line moved) — the named return
`sshUser` is always `""`; the per-entity user branch is dead and everything
relies on `sshExec`'s global default fallback. **Fix:** read the SSH user
from entity attributes or delete the dead return to make the behavior
honest.
### A5. `queryRows` stringifies every column
`internal/mcp/server.go:861` renders all values via `fmt.Sprintf("%v", ...)`,
so numbers, bools, timestamps, and JSON all reach agents as strings.
**Fix:** type-preserving serialization (pass through pgx-native values into
`json.Marshal`) — improves every read tool at once.
`internal/mcp/server.go:1090` (was :861 — line moved) renders all values via
`fmt.Sprintf("%v", ...)`, so numbers, bools, timestamps, and JSON all reach
agents as strings. **Fix:** type-preserving serialization (pass through
pgx-native values into `json.Marshal`) — improves every read tool at once.
### A6. `get_state_snapshot` description is stale
`internal/mcp/server.go:689` still advertises "disk, drift count" — columns
removed in commit 3ea43ad. **Fix:** update the description.
`internal/mcp/server.go:863` (was :689 — line moved) still advertises "disk,
drift count" — columns removed in commit 3ea43ad. **Fix:** update the
description.
---
@@ -101,12 +113,13 @@ removed in commit 3ea43ad. **Fix:** update the description.
### B1. Enrollment is unauthenticated, with a false comment
`internal/httpapi/server.go:97` says "unauthenticated (IP-gated in handler)"
but `EnrollClient` (`internal/httpapi/impl.go:1099`) performs no IP check at
all — the only gate is the target entity being in state
`planned`/`provisioning`. Caddy's `@enroll` matcher bypasses Authentik.
Anyone reaching `oikos.hubris.network` who knows (or guesses) a planned slug
receives that node's **age private key** in the HTTP response body.
`internal/httpapi/server.go:111` (was :97) says "unauthenticated (IP-gated
in handler)" but `EnrollClient` (`internal/httpapi/impl.go:1166`, was
:1099) performs no IP check at all — the only gate is the target entity
being in state `planned`/`provisioning`. Caddy's `@enroll` matcher bypasses
Authentik. Anyone reaching `oikos.hubris.network` who knows (or guesses) a
planned slug receives that node's **age private key** in the HTTP response
body. Still open — line numbers only, substance unchanged.
**Fix:** enforce a real gate (mesh-CIDR check, one-time enrollment token
minted when the entity is created, or both), and stop returning the age
@@ -114,34 +127,49 @@ private key in the response — have the client fetch it from the secret store.
### B2. Fake Infisical credentials returned to enrollees
`internal/httpapi/impl.go:1191-1192` returns `"inf_client_"+uuid` /
`"inf_secret_"+uuid` — random strings wired to nothing. Enrolled clients hold
credentials that authenticate against nothing.
`internal/httpapi/impl.go:1260-1261` (was :1191-1192) returns
`"inf_client_"+uuid` / `"inf_secret_"+uuid` — random strings wired to
nothing. Enrolled clients hold credentials that authenticate against
nothing. Still open — line numbers only, substance unchanged.
**Fix:** implement `CreateMachineIdentity` in `internal/secrets/infisical.go`,
or return no credentials and document the manual step.
### B3. Hermes `/query` has no auth
### B3. Nomos's `/query` has no auth
`hermes/config.yaml:9` sets `mesh_only: true` but `cmd/hermes/main.go` never
reads or enforces it — it serves any caller on :8092, who can invoke
`request_execution`. **Fix:** enforce mesh-CIDR (or bearer token) in the
handler; fail closed.
`nomos/config.yaml:9` (was `hermes/config.yaml:9`) sets `mesh_only: true`
but `cmd/nomos/main.go` (was `cmd/hermes/main.go`) never reads or enforces
it — it serves any caller on :8092, who can invoke `request_execution`.
Still open, now also tracked as C1 in
[2026-07-11-nomos-agent-code-review.md](2026-07-11-nomos-agent-code-review.md),
deferred by the operator. **Fix:** enforce mesh-CIDR (or bearer token) in
the handler; fail closed.
### B4. SSH host keys not verified
`ssh.InsecureIgnoreHostKey()` at `internal/mcp/server.go:920`.
`ssh.InsecureIgnoreHostKey()` at `internal/mcp/server.go:1155` (was :920).
Still open — line number only, substance unchanged.
**Fix:** known_hosts pinning (keys are already inventory-managed per node).
### B5. `list_my_secrets` enumerates all node pubkeys
Without `caller_pubkey`, `internal/mcp/server.go:709-720` returns every entity
that has an `age_pubkey`; nothing ties the caller to what it may list.
Without `caller_pubkey`, `internal/mcp/server.go:879-883` (was :709-720)
returns every entity that has an `age_pubkey`; nothing ties the caller to
what it may list. Still open — line numbers only, substance unchanged.
**Fix:** require `caller_pubkey` and scope results to the caller's
entitlements.
---
## C. User perspective (interacting via Hermes)
## C. User perspective (interacting via Hermes) — RESOLVED
**Resolved as of the Hermes→Nomos rewrite (verified 2026-07-12).** This
entire section described `cmd/hermes`, which no longer exists — Hermes was
renamed and rebuilt as `cmd/nomos`, a real LLM-backed agent loop, which is
exactly the recommendation below. `cmd/nomos/main.go:444-465` now calls the
real `listTools()` for "help"/"what can you do", and routes unmatched
queries to "natural language queries belong to `/chat`..." instead of
silently falling back to `get_health_summary`. Kept below for history —
original text unchanged.
- `routeQuery` NLU is hardcoded `strings.Contains`; `extractEntity`
(`cmd/hermes/main.go:173`) recognizes only 5 services (`authentik, caddy,
@@ -175,27 +203,41 @@ says 21 — both stale). Missing capabilities:
`pending_approval`, an agent has no way to check or reference the approval.
Add `get_approval_status` / `list_pending_approvals`.
4. Execution actions limited to `restart | systemctl | pct_exec |
apt_upgrade` — no deploy/rollback/config-edit path.
5. Architecture/doc mismatch: `hermes/SOUL.md` claims "no SSH access; all
mutations flow through the actuator", but the MCP server runs
`restart`/`pct_exec` synchronously over SSH from inside the api process
(`sshExec`, server.go:902). Align docs or move execution to the actuator.
apt_upgrade` — no deploy/rollback/config-edit path. Partially
superseded: the general `run` MCP tool (D4-partial, done) covers
arbitrary commands now; `request_execution`'s fixed enum is still there
for the specific actions it names (see
[2026-07-10-general-gated-execution.md](2026-07-10-general-gated-execution.md)).
5. **RESOLVED (verified 2026-07-12).** Architecture/doc mismatch:
`hermes/SOUL.md` claimed "no SSH access; all mutations flow through the
actuator", but the MCP server ran `restart`/`pct_exec` synchronously over
SSH from inside the api process. `nomos/SOUL.md:21-22,40` now accurately
documents SSH access via the policy-gated `run` tool — matches the
architecture the general-gated-execution plan built. No longer a
mismatch.
---
## E. Doc drift / housekeeping
- Tool counts: README 15 / AGENTS.md 21 / actual 28 — regenerate from
`internal/mcp/server.go` (consider a doc-gen make target).
- `compose/caddy/Caddyfile.oikos` retains literal `<mac-mini-mesh-ip>`
placeholders in all three vhosts.
- `.agents/HERMES.md` lists "`inventory.yaml`, `inventory.yaml`" (duplicate).
- `plans/index.md` drift: fix-MCP-tools row sat in Active with a broken link
after the file moved to `done/` (fixed alongside this plan); TRMNL listed
active though in `done/`; Grimmory header says `in-progress` though in
`done/`; `.hermes/plans/` (7 executed plans) missing from disk.
- `plans/2026-07-05-oikos-prometheus-lxc.md` (~0% done) references deleted
`oikos/scheduler.py` and `bin/homelab`; LXC 131 collision unresolved.
- **RESOLVED (verified 2026-07-12):** Tool counts. README 15 / AGENTS.md 21
/ actual 28 was already stale by 2026-07-11 (registered tools grew to
33) — AGENTS.md now documents all 33 with the full catalog (2026-07-12).
- **Still open:** `compose/caddy/Caddyfile.oikos` retains literal
`<mac-mini-mesh-ip>` placeholders (this repo's copy is a reference only —
see [2026-07-12-wails-desktop-app.md](2026-07-12-wails-desktop-app.md)'s
"Plan review" — the real config lives in `dtoro/caddy-conf`).
- **RESOLVED:** `.agents/HERMES.md` renamed to `.agents/NOMOS.md`; the
duplicate-line bug itself is still present at `.agents/NOMOS.md:11` —
only the file citation was stale, the underlying nit is still open.
- **RESOLVED (verified 2026-07-12):** `plans/index.md` drift — the broken
link, TRMNL/Grimmory Active/Done mismatch, and missing `.hermes/plans/`
entries described here are no longer present in the current
`plans/index.md`; already fixed sometime after this plan was written.
- **RESOLVED (verified 2026-07-12):** `plans/2026-07-05-oikos-prometheus-lxc.md`
already self-corrected both the deleted-file references and the LXC 131
collision in its own 2026-07-08 changelog — this bullet describes a
pre-fix state.
---

View File

@@ -182,16 +182,27 @@ board reflects reality instead of showing a task that looks perpetually
### C1. Nomos's own HTTP gateway has zero authentication
[docker-compose.yml:133](../docker-compose.yml) publishes port 8092 directly
[docker-compose.yml:144](../docker-compose.yml) publishes port 8092 directly
(`"8092:8092"`, comment: *"mesh-published"*) and
[Caddyfile.oikos](../compose/caddy/Caddyfile.oikos:19,34) reverse-proxies to
it from two routes. `grep -n "Authorization\|Bearer\|auth" cmd/nomos/main.go`
returns **nothing**`/chat`, `/sessions`, `/sessions/{id}` (including
`DELETE`), and `/query` have no credential check of any kind. Anyone who can
reach the LAN or mesh network can converse with Nomos directly: start tasks,
read/delete any session, answer pending questions, and — via chat-assent —
approve gated executions by typing "yes" or "I confirm" to whatever the agent
proposes, with no authentication at all. This is the same class of gap
[Caddyfile.oikos:52-54](../compose/caddy/Caddyfile.oikos) reverse-proxies to
it — as of the client/server split
([2026-07-12-wails-desktop-app.md](2026-07-12-wails-desktop-app.md)), only
from `nomos.hubris.network` now, not two routes: `/agent/*` on
`oikos.hubris.network` was repointed to go through `api`'s own authenticated
proxy mount instead of straight to nomos:8092, but that's `combinedAuth`
authenticating the *hop into api*, not anything nomos itself checks — this
finding is unaffected by that change, still fully open. `grep -n
"Authorization\|Bearer\|auth" cmd/nomos/main.go` still returns **nothing**
for nomos's inbound routes (nomos did gain outbound auth as *part of* the
client/server split — it now sends `Authorization: Bearer
$OIKOS_MCP_BEARER_TOKEN` on its own calls to `api` — but that's the opposite
direction from this finding) — `/chat`, `/sessions`, `/sessions/{id}`
(including `DELETE`), and `/query` have no credential check of any kind.
Anyone who can reach the LAN or mesh network can converse with Nomos
directly: start tasks, read/delete any session, answer pending questions,
and — via chat-assent — approve gated executions by typing "yes" or "I
confirm" to whatever the agent proposes, with no authentication at all. This
is the same class of gap
[oikos-gaps-and-improvements](2026-07-08-oikos-gaps-and-improvements.md)
flagged for the `api`/MCP surface (items B1-B5), but specifically for nomos's
*own* port, which doesn't sit behind `combinedAuth` the way `api`'s routes do.

View File

@@ -1,10 +1,33 @@
# 2026-07-12 — Wails desktop application
**Status:** In Progress — Phase 0 (0.1-0.4, 0.6) done and verified live
(browser: cross-origin static SPA + API on different ports, CORS, bearer
auth, SSE query-token auth, localStorage persistence across reload — see
"Plan review" for the gaps found and fixed along the way). Phase 1 (Wails
shell) not started.
**Status:** In Progress — Phase 0 (0.1-0.4, 0.6) done, verified live in a
local browser test, and **deployed to production** (mac-mini, commit
`0c0f35a`, 2026-07-12). Phase 1 (Wails shell) not started.
**Production deploy (2026-07-12):** merged to `main`, picked up by the
2-minute deploy poller (`scripts/deploy.sh`: pg_dump backup → rebuild →
rolling restart → health check), `healthy after 1s`. Verified post-deploy:
unauthenticated `/api/v1/*` now 401s (the dev-open bypass was live in
production before this — `OIKOS_ENV=dev` with no token set — so this closed
a real, currently-exploitable hole, not just future prep); `/healthz` stayed
open; nomos reconnected its MCP session with the new
`OIKOS_MCP_BEARER_TOKEN` and a real tool call round-tripped end to end
(`get_health_summary` via `/query`). A real random token was generated and
added to mac-mini's `.env` (not committed — gitignored) before deploy, so
the `${OIKOS_MCP_BEARER_TOKEN:-dev-token}` fallback in `docker-compose.yml`
never activated with the weak literal default.
**Deliberately not done as part of this deploy** (out of scope — a different
host/repo than "mac-mini", not touched): the Caddy LXC (121) and
`dtoro/caddy-conf`. Checked the real production Caddyfile directly — there is
**no `oikos.hubris.network` site block at all yet**, so the Authentik-bypass
risk (gap 1 below) doesn't apply yet; there's no public UI exposed to break.
`mcp.hubris.network` exists but still reverse-proxies to the old
pre-consolidation service on LXC 105 (`192.168.8.205:9810`), unrelated to
this stack — stale, but pre-existing and out of scope here. Exposing
`oikos.hubris.network` publicly (with the `@api` bypass this plan's
Caddyfile.oikos reference copy already has) is unstarted follow-up work, not
a regression from this deploy.
## Plan review — gaps found before starting Phase 0

View File

@@ -10,11 +10,11 @@ went sideways, open an investigation.
| ---- | ----- | ------ |
| 2026-07-05 | [Oikos Prometheus LXC](2026-07-05-oikos-prometheus-lxc.md) | Planned — not started |
| 2026-07-08 | [Oikos gaps, broken things, and improvements](2026-07-08-oikos-gaps-and-improvements.md) | In Progress — security items (B1-B5) and doc drift (E) still open |
| 2026-07-08 | [Control room web UI](2026-07-08-control-room-webui.md) | In Progress |
| 2026-07-08 | [Control room web UI](2026-07-08-control-room-webui.md) | In Progress — packaging/auth sections superseded by the Wails plan's Phase 0 (client/server split); M4 still open |
| 2026-07-08 | [Liveness, drift, and UX cohesion](2026-07-08-liveness-drift-and-ux-cohesion.md) | In Progress — Phase 5 deferred |
| 2026-07-10 | [General gated execution: unlimited actions, gated by risk](2026-07-10-general-gated-execution.md) | In Progress — enum retirement + auto-act revival still open |
| 2026-07-11 | [Nomos agent code review: gaps and improvement plan](2026-07-11-nomos-agent-code-review.md) | In Progress — only C1 (unauthenticated nomos gateway) still open, deferred |
| 2026-07-12 | [Wails desktop application](2026-07-12-wails-desktop-app.md) | In Progress — Phase 0 done, Phase 1 not started |
| 2026-07-12 | [Wails desktop application](2026-07-12-wails-desktop-app.md) | In Progress — Phase 0 done and deployed to production (2026-07-12), Phase 1 not started |
## Done