205 Commits

Author SHA1 Message Date
a429436903 Trigger desktop CI on every push to main, not just tags
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build SPA (push) Has been cancelled
Desktop App / macOS (arm64) (push) Has been cancelled
Desktop App / Linux (amd64) (push) Has been cancelled
Desktop App / Create Release (push) Has been cancelled
2026-07-13 22:54:00 +02:00
07d67c8446 fix(oidc): strip trailing slash from redirect URI to match Authentik strict mode
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-13 22:50:31 +02:00
8b50753746 feat(chat): MCP tool apps — custom inline renderers for 12 tools
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
12 of 33 MCP tools now render as rich inline cards instead of raw JSON:
EntityCard, HealthSummary, LXCList, EntityTable, KnowledgeResults,
BlastRadius, ChangeLog, FleetSnapshot, MetricChart.

Architecture:
- Server: annotateJSONResult() wraps queryRows with __renderer hints
- Registry: match/dispatch system maps tool names to Svelte components
- Chat: inline dispatch with 5-card limit, overflow to collapsed group
- ToolCallGroup: unmatched prop, hides when all matched, ARIA labels

Tests: 3 new Go tests for annotateJSONResult (wrap, no-op, multi-row).
2026-07-13 22:46:56 +02:00
680575e2cf Remove unrelated plan file from stale branch
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-13 22:41:50 +02:00
6b52c1ae57 Move 2026-07-12-wails-desktop-app plan to done/
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Phase 0 deployed 2026-07-12. Phases 1.0–1.4 implemented 2026-07-13.
Plan complete.
2026-07-13 22:41:45 +02:00
5d6d9e9040 Merge feature/wails-desktop-app into main
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-13 22:40:40 +02:00
04006553a3 Wails v3 desktop app: scaffold, shell features, token mgmt, auto-update, CI
Problem: the Oikos control room was browser-only — no native desktop
experience (system tray, notifications, keychain-persisted auth).

Change: add a Wails v3 thin-shell desktop app at cmd/desktop/ that embeds
the existing SPA in a webview. The Go side is ~380 lines — no bundled
server, no Postgres connection. It reads auth from the OS keychain,
injects it into the SPA on load, and the SPA talks HTTPS to the homelab
same as a browser.

Phase 1.0 — Scaffold + window:
  - Embed web/dist/ into the Wails binary
  - Inject window.__OIKOS_CONFIG__ with keychain-stored apiUrl + token
  - 1400×900 window, min 1024×700
  - System tray: Open/Quit, click toggles window

Phase 1.1 — Native shell:
  - Poll /api/v1/dashboard/summary every 30s; osascript notification
    when approvals or critical signals increase
  - Save/restore window position to ~/.config/oikos/window.json
  - EnableAutoStart/DisableAutoStart — macOS LaunchAgent plist

Phase 1.2 — Token management:
  - Config.svelte calls window.wails.Call.ByName('SaveConfig') after
    successful connection — persists to OS keychain
  - ConfigService binds SaveConfig, ClearConfig, EnableAutoStart,
    DisableAutoStart to the Wails runtime

Phase 1.3 — Auto-update:
  - Poll Gitea releases API every 6h, compare semver, show dialog
  - 'Check for Updates' tray menu item triggers immediate poll

Phase 1.4 — Distribution:
  - macOS entitlements.plist: network client + keychain access
  - .gitea/workflows/desktop.yml: CI builds macOS arm64 + Linux amd64
    on 'desktop-*' / 'v*' tags, attaches artifacts to release
  - Makefile: desktop (build), desktop-package (build + zip/tar.gz)
  - CONTRIBUTING.md: documented desktop app + commands

Risk: low. Wails v3 alpha API may shift; the Go glue is ~380 lines and
trivially portable. The desktop app is additive — zero changes to the
existing server or SPA logic. No config mutation, no infrastructure
impact.

Verification: go build, go vet, go mod tidy all pass.
2026-07-13 22:40:18 +02:00
7b0a0f01b5 oidc: authenticate SPA users via Authentik
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Add OIDC proxy endpoints (GET config, POST token) to API server
- Implement PKCE Authorization Code flow in SPA
- Enable Authentik login tab in Config page
- Handle callback + auto-refresh + session restore
- Add restart: unless-stopped to all persistent services
- Configure OIDC issuer + client_id in docker-compose
2026-07-13 22:17:40 +02:00
f6a699469d oidc: authenticate SPA users via Authentik
- Add OIDC proxy endpoints (GET config, POST token) to API server
- Implement PKCE Authorization Code flow in SPA
- Enable Authentik login tab in Config page
- Handle callback + auto-refresh + session restore
- Add restart: unless-stopped to all persistent services
- Configure OIDC issuer + client_id in docker-compose
2026-07-13 22:17:31 +02:00
4c4afc4783 fix(web): fit graph to node bounding box once the simulation settles
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Fresh nodes (no prior x/y) get placed by d3-force's default init, which
spirals out from the ORIGIN — not (width/2, height/2) — while the
centering forces here are deliberately weak (0.04, so they don't fight
the link/collide layout) and alphaDecay stops the sim before a weak force
can always pull a far-off cluster back to center. Net effect: graphs could
settle visibly off-center on load, cramped in a corner of the pane.

Fixed by computing the actual node bounding box once the simulation's
'end' event fires and setting the view transform to fit it, instead of
relying on the force balance to land on center by itself. Gated behind a
`fit` flag so passive background reloads (live entity/relationship
events) don't yank the view out from under someone actively panning or
zoomed in on a specific area — only fresh loads (mount, root/depth
change, reset, re-root) reframe.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:51:56 +02:00
604b608fa8 feat(mcp): expose full knowledge content to the agent, not just snippets
search_knowledge and get_entity_knowledge only ever returned a ts_headline
snippet/short headline — enough to find a note, not enough to act on it.
Add get_knowledge_content(slug), mirroring the web UI's
/api/v1/knowledge/content/{id}, so the agent can read a document/
investigation/runbook's full markdown body once it knows which one it
needs. upsert_knowledge already covered the write side. Cross-referenced
all three tool descriptions so the agent discovers the full-read path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:51:37 +02:00
62a8ec1d8d fix(mcp): write targets/involves relationship edges when executions are created
Executions were being created with no outgoing edges to what they acted
on or which task/session drove them, silently starving the graph of new
data going forward — found during this session's DB audit, which had to
backfill 245+25 missing targets/involves edges for existing executions.
This closes the gap at the source: every execution now gets a
target-->targets-->execution edge, and (when the caller supplies a
session/task) a task-->involves-->execution edge, both idempotent
(NOT EXISTS guards) so retries/backfills don't duplicate.

Two call sites: the deduped systemctl/apt_upgrade/pct_create fast path
and the general classifyAndGate path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:45:44 +02:00
335fa67d55 fix(web): scope 1-hop neighbor expansion to rooted graph views only
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Two bugs found while verifying against real production data:

- Excluded activity types (execution/check/task etc., see categories.ts)
  were falling through inCategory's "unknown type -> always visible"
  fallback, since typeCategory only stored entries whose category was
  defined. That fallback exists for types the ontology never returned at
  all; it wrongly re-admitted types the ontology returned but categories.ts
  deliberately excludes. Fixed by storing every type (including undefined
  categories) and checking key presence, not value truthiness.

- Once that was fixed, the previous commit's 1-hop neighbor expansion
  (dimmed cross-category context) turned out fine for a rooted view but
  flooded an unrooted "browse the whole category" view: Fleet's ~49 focus
  entities are hub-like enough that 1-hop pulled in 325+ of the system's
  479 total entities. Neighbor expansion now only applies when a root is
  set; the unscoped view goes back to same-category-only edges, which
  measured at a clean 49 nodes for Fleet.

Verified against live production data (real bearer token, real DB) rather
than mocks: Fleet unrooted = 49 nodes matching the DB's compute+physical
count exactly; rooting on host:strong shows 33 nodes with both bright
same-category and dimmed cross-category neighbors, no isolated dots.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:39:31 +02:00
d1243aceac fix(web): decode percent-encoded slugs in the knowledge content route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
chi.URLParam returns the raw, still-encoded path segment — unlike the
OpenAPI-generated routes, which decode via
runtime.BindStyledParameterWithOptions before the handler sees them. Slugs
like "document:containers/101-jellyfin" (encoded by the frontend's
encodeURIComponent) were arriving undecoded and matching no row. Found via
a standalone chi repro, not by patching the live deploy checkout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:29:42 +02:00
61ad785fef fix(web): render full document content, keep graph connected under categories
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Two fixes to the new category taxonomy:

- Knowledge Base couldn't show a document/investigation/runbook's own
  markdown body — knowledge_entities.content was never exposed by any
  endpoint (GetEntityKnowledge answers "what knowledge references this
  entity", not "what is this entity's content"). Add GET
  /api/v1/knowledge/content/{id} and render it with the existing
  marked+DOMPurify pipeline in a new Content section.

- The graph hid any edge whose other endpoint wasn't in the active
  category, so nodes with only cross-category neighbors rendered as
  disconnected dots. Queried the real relationship table: ~70% of infra
  edges cross Fleet/Network/Services/Storage lines (compute+network+
  software+storage+physical used to be one "infrastructure" layer).
  EntityGraph now keeps 1-hop neighbors visible but dimmed instead of
  hiding them, so the edges — and what they connect to — stay visible.

- categories.ts: `cognition` domain conflated true knowledge (document/
  investigation/runbook, 58 entities) with operational telemetry
  (execution/check/task/signal/approval/pattern/skill/classification/
  feedback, 300+ entities with their own Operations/Signals/Learning
  pages). Mapping the whole domain to Knowledge pulled in 245 execution
  entities fanning out from ~17 compute nodes via `targets` edges — the
  single biggest source of graph clutter. Knowledge now maps by type
  (document/investigation/runbook only); the rest of cognition is
  excluded from Knowledge Base browsing entirely.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:17:13 +02:00
35c54ceef5 feat(web): browse Knowledge Base by mixed Network/Fleet/Services/Storage/Identity/Knowledge categories
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Replace the layer-based (Infrastructure/Governance/Cognition) browsing tabs
with a synthesized category taxonomy built from the ontology's finer-grained
`domain` field, since layer lumped unrelated entity types (an LXC and a DNS
record and a storage volume) into one bucket. Network and Fleet each span
two domains, so the table view now fans out per-domain fetches and merges,
while the graph view maps domain->category client-side. Also carries over
several detail-panel polish items (Tasks-not-raw-executions, slug URL
encoding, MultiSelectFilter) from earlier in this session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 09:53:15 +02:00
f8e03806aa feat(deploy): containerize the web UI as its own compose service
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The SPA-from-binary split (0c0f35a) left `make deploy-ui` pointing at a
deploy path that was never actually wired up: scp to a "mac-mini" SSH
host that doesn't resolve from itself, a /var/www/oikos-ui/ that doesn't
exist, and `systemctl reload caddy` on a box with no Caddy installed at
all (not brew, not a container, nothing on 80/443).

Add a `web` service (compose/web/Dockerfile: node build -> caddy:2-alpine
static + SPA-fallback serving) to docker-compose.yml so the UI deploys
through the same push-to-main -> webhook -> docker compose build/up
pipeline the rest of the stack already uses, instead of a manual
scp/ssh step. Drop the broken `deploy-ui` Makefile target; `make ui`
stays as a local build sanity-check.

Update the reference Caddy config (compose/caddy/Caddyfile.oikos) to
reverse_proxy the new :8091 service instead of reading static files off
local disk, and fill in the <mac-mini-mesh-ip> placeholders with the
actual LAN IP (192.168.178.182 — the LXC and mac-mini subnets are
routed). This file is a reference only; the real caddy-conf repo change
is applied separately after review.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-12 22:36:40 +02:00
94c94c0758 feat(web): merge Entities + Graph into a single Knowledge Base page
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Replaces the separate Entities/Graph nav items with one Knowledge Base
page that browses all entities as either a table or a force-graph,
scoped by ontology layer (Infrastructure/Governance/Cognition), with a
resizable browse/detail split instead of a slide-over sheet.

- New KnowledgeBase.svelte: layer tabs, view toggle, resizable
  browse/detail split (pattern from Chat.svelte's rail).
- EntityTable/EntityGraph extracted as presentational sub-components;
  their search/filter/root/depth toolbars live in the shared page
  toolbar (not the resizable pane) so they don't truncate when the
  divider is dragged narrow, and both views start flush with the
  detail pane for consistent height.
- EntityTable columns are sortable (slug/type/name/state/health).
- EntityDetailContent redesigned as a single-column list of
  collapsible sections (DetailSection.svelte), collapsed by default
  when empty; relation entries are clickable and select the entity in
  the browse pane + detail pane (and drill in-place in EntitySheet
  wherever it's used elsewhere in the app).
- api.ts: add layer filter to fetchEntities, add fetchEntityTypes for
  client-side graph layer scoping (the graph endpoint has no layer
  param).

Old hash routes (#/entities, #/graph) redirect to #/kb.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-12 22:21:29 +02:00
d80a394b7f docs: fix plan/repo drift, retire dead Goose+Nomos and Caveman tooling
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Documentation and repo-hygiene pass following the client/server split:

Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
  described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
  refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
  deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
  to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.

Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).

Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).

Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.

Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.

Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
  places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
  GetClientContext handler) since the mechanism's introduction on
  2026-06-02 — never matched any real filename, so no client has ever
  picked up an auto-setup script via git-pull or the context-poller sync.
  Fixed all three; the Go server-side fix is the one that actually matters
  since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
  (parsed, never consumed) left over from an earlier clone-based model.

Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 18:19:41 +02:00
0c0f35a3a9 feat(web): split SPA from oikos binary, require auth on every route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Phase 0 of plans/2026-07-12-wails-desktop-app.md. The control-room SPA
is no longer embedded (web/embed.go deleted); it's a standalone static
build served separately (make ui / make deploy-ui). The api process
adds CORS and drops the dev-open auth bypass — every route now needs a
real bearer token, including SSE (?token= query param, EventSource
can't set headers) and api's own /agent proxy to nomos (previously
unauthenticated by omission).

nomos was an unauthenticated client of api's /mcp and approval-decision
endpoints; closing dev-open would have broken it, so it now sends
Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN on every call back to api.

SPA gets a runtime config module (config.ts) and a Config.svelte
first-launch/reconfigure page, reachable afterwards via a "Connection"
entry in the sidebar footer. Every fetch() in api.ts routes through
fetchWithAuth so the same build works same-origin (browser prod, Vite
dev proxy) or cross-origin (future Wails webview, remote access).

Six gaps found against the plan and the live Caddy topology while
implementing — documented in the plan's "Plan review" section, most
notably: api's own /agent mount was never behind combinedAuth (fixed),
and production's Authentik forward-auth needs a bearer-token bypass for
API routes that this repo's Caddyfile.oikos reference copy now has, but
the real dtoro/caddy-conf deploy does not yet.

Verified live: cross-origin static SPA + API, CORS, bearer auth, SSE
query-token auth, and localStorage persistence all confirmed working
in-browser. Full Go test suite and npm run build pass with no
regressions against the pre-change baseline.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 15:49:42 +02:00
346eb2f144 chore: gitignore compiled binaries at root
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-12 12:40:55 +02:00
48827d5bb1 fix(deploy): add poller as fallback when Gitea webhook can't reach mac-mini
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Gitea (LXC 104, 192.168.8.x) can't reach mac-mini (192.168.178.182) due to
ALLOWED_HOST_LIST. As a fallback, a 2-minute launchd poller checks if
origin/main has new commits and runs deploy.sh if so.
2026-07-12 12:14:28 +02:00
56979ac4bd feat(deploy): add webhook receiver and launchd service for push-to-deploy
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/webhook/main.go: HMAC-validated webhook receiver on :9797
- launchd plist: keeps webhook running, PATH includes docker
- Makefile: 'make webhook' target
- Registered as Gitea webhook id 15 on dtoro/oikos

Fixes: auto-deploy was not wired on mac-mini after the consolidation
2026-07-12 12:12:01 +02:00
3157e6102a plans: Wails desktop app with client/server split
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Phase 0 separates the SPA from the oikos binary (delete embed.go, add CORS,
make API base URL configurable, add auth interceptor, close dev-open gate).
Phase 1 builds a thin Wails v3 desktop wrapper — native window + tray +
notifications + auto-start + auto-update. SPA shared between browser and
desktop builds.
2026-07-12 11:49:27 +02:00
6807e353e3 feat(web): redesign Overview as the homepage with a living graph backdrop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Overview replaces Tasks as the default route: a centered new-task entry
with live fleet metrics, a scrollable/filterable task table, and an
ambient canvas rendering of the real entity graph (autonomous camera
drift + mouse parallax) behind it. Tasks sidebar entry is removed;
its status-bucketing logic moves to lib/tasks.ts for reuse.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 11:07:36 +02:00
0ed171507f Merge remote-tracking branch 'origin/main'
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-12 09:18:23 +02:00
e3cbaee534 fix(knowledge): populate hit id in search results
Search hits and entity-knowledge hits never selected an id column, so
every KnowledgeHit.Id defaulted to the zero UUID. The frontend's keyed
{#each results as hit (hit.id)} then had all-duplicate keys, which
silently broke Svelte 5's if-block branch swap for the results panel —
search would set searched=true (Clear button appeared) but the view
never switched away from "Recently learned". Select e.id in both
queries and key the each block on hit.slug (guaranteed unique) instead.
2026-07-12 09:16:03 +02:00
de126daf43 feat(web): fold Events/Agent/Audit into EntityDetail; tag agent_activity with entity_id
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Events, Agent, and Audit were standalone read-only pages that never
cross-referenced the entity they related to. Fold them into EntityDetail
as entity-scoped cards (Agent activity, Audit trail) alongside the
existing Signals/Executions/Knowledge cards, and give the Signals card
real Ack/Mute/Resolve actions. Signals stays a standalone page since
it's the only one with cross-entity triage value (badge count, actions).

Also fixes the underlying reason those new cards would've stayed empty:
agent_activity rows were never tagged with entity_id at insert time
(cmd/nomos/store.go, internal/mcp/server.go), even though the column
and the API filter both support it. Added a best-effort resolver that
checks common tool-arg keys (target, entity_slug, slug, ...) against
the entities table.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 08:38:19 +02:00
c8b479d565 docs: close out task-completion safety net plan; fix stale relative links
Fixes 1-3 deployed and verified live: fresh trivial Q&A sessions now reach
done immediately, and a goal-bearing session that stalled was correctly
nudged by the idle sweep. Fix 4 (backfill) was replaced with deletion after
the operator's call — verified against the DB first that zero knowledge
notes were linked to or written by any of the 53 removed sessions, so
nothing was lost. Documents the pagination gap in listSessions (hardcoded
LIMIT 50, no total count) that hid 6 of those sessions from the original
audit.

Also fixes relative links in this plan and in the UI-review plan that broke
when both moved from plans/ to plans/done/ (one directory level deeper).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 00:14:15 +02:00
075ff93792 docs: mark task-completion safety net fixes 1-3 in progress
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 22:11:56 +02:00
3b9c75fa3f fix(agent): task completion safety net — stop tasks sticking at Running
Implements fixes 1-3 of plans/2026-07-11-task-completion-safety-net.md.
Confirmed live that 50/50 production sessions never reached a terminal
status because the model almost never calls complete_task, even for
trivial single-tool Q&A turns SOUL.md explicitly calls out as needing it.

- Inline safety net (agent.go): a session that never called set_goal never
  framed itself as a structured task, so its first plain-text turn-end IS
  the task ending — auto-complete it there instead of leaving status stuck
  at its creation default forever.
- Idle sweep (continue.go, new completion_nudges column): goal-bearing
  sessions that stall get one nudge, then auto-close with outcome=partial
  if the nudge goes unanswered, mirroring the pattern resumeSession already
  uses for a different stuck-session failure mode.

Fix 4 (backfill of the 50 already-stuck live sessions) is deliberately
separate — deferred until this is deployed and verified live, per the
plan's implementation order.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 22:11:24 +02:00
e3850f6820 docs: task completion safety net — every live task stuck Running
Traced during UI-review verification: 50/50 live sessions are stuck
active/planning, never done/failed. Root cause confirmed against the
running DB — set_goal called once, propose_plan and complete_task
called zero times across all 50 sessions. The model consistently
skips the terminal complete_task call despite SOUL.md explicitly
instructing it to, especially for trivial single-tool Q&A turns.
Plan proposes an inline safety net for the common case plus an idle
sweep for structured goal/plan sessions that stall.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 22:01:35 +02:00
fb4c76ba82 fix(ui): implement UI review findings — a11y, IA, and consistency fixes
Fixes the reviewed gaps: keyboard-inaccessible delete controls (SessionRail,
Entities row), case-sensitive entity filter, two competing entity-detail
navigation patterns (standardize on EntitySheet), non-clickable Overview KPI
cards, a bare button bypassing the shared Button component, inconsistent
blur-only vs live filtering, and an unenforced sanitization assumption on
search snippet HTML (now using the already-present dompurify dependency).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 21:52:59 +02:00
b72267bd72 docs: nomos agent code review — mark all fixes done except deferred C1
Every finding from the review is now implemented and verified live:
A1 (3919ec3), B1+B2 (c5ffaec), A3 (926969a), D1-D3 (76f7630), A2 (c390164),
B3 (6d4f6de), F1 (11c18e8). C1 (nomos gateway has no authentication) remains
explicitly deferred per operator instruction. Kept in plans/ (not moved to
done/) since C1 is still open, matching how other partially-complete plans
in this index are tracked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:32:29 +02:00
11c18e8956 perf(agent): cache the MCP tool list per client (F1)
Fix F1 of plans/2026-07-11-nomos-agent-code-review.md, the last item.
buildTools called listToolsFull (a tools/list MCP round-trip) at the start of
EVERY chat turn, including every auto-continuation resume — the tool list is
static for the lifetime of one MCP connection, changing only when the api
process re-registers tools (a restart, which this client already detects and
reacts to via reconnectLocked). Re-fetching it every single turn was
avoidable network+parsing work on the hot path.

mcpClient now caches the parsed tool list after its first fetch, guarded by
its own mutex (kept separate from the request-serializing mu so a cache
check never contends with an in-flight doRequest call). reconnectLocked
clears the cache — an api restart may have changed what's registered, so a
stale cache would be wrong, not just slow. fleetSnapshot's get_health_summary
call is deliberately left uncached — it's meant to be "as of now."

Since each session gets its own client (the per-session pool from the
concurrency work), this caches per-task-conversation rather than globally: a
task's FIRST turn still pays the round-trip, every turn after reuses the
cached list — which is exactly the case that mattered (long-running,
heavily-autonomous tasks with many auto-continuation resumes).

Verified live via the api's request log: a brand-new session's first turn
made 3 MCP calls (initialize, tools/list, get_health_summary); a second turn
on the SAME session made exactly 1 (only get_health_summary) — tools/list
correctly skipped.

This completes the implementation order in
plans/2026-07-11-nomos-agent-code-review.md — every A/B/D/E/F finding from
the review (excluding C1, explicitly deferred per operator instruction) is
now fixed, tested, and verified live.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:30:07 +02:00
6d4f6de676 fix(agent): mark a task failed when its resume permanently gives up (B3)
Fix B3 of plans/2026-07-11-nomos-agent-code-review.md. resumeSession's retry
loop (used by both auto-continuation and panel-answered questions) already
retried once on a transient LLM failure, but if BOTH attempts came back
empty/erroring, the code just logged and returned — the task was left at
whatever status it already had (typically 'executing' or 'awaiting_input')
with no outcome, no operator-visible signal beyond an inert error line
buried in the transcript, and no way to tell a genuinely stuck task apart
from one quietly still working.

On permanent failure, now calls store.completeTask(outcome='failure', a
summary built from the error) so the task board reflects reality instead of
showing a task that looks perpetually in-progress. Uses context.Background()
for that write, matching resumeSession's own persistence pattern, since the
context that led to the failure may itself be in a bad state. This doesn't
prevent the operator from continuing to work the task via a fresh chat
message afterward — it only replaces silent hanging with a real status.

A full live induction of a permanent LLM outage would require breaking the
model/API-key config for the whole nomos container — too invasive for this
fix's priority. Verified instead that the new branch stays correctly dormant
on the happy path: ran a real ask_operator → panel-answer → resume cycle
end-to-end and confirmed the task landed at status='executing' with no
outcome set, proving the failure-handling code doesn't false-positive on a
normal successful resume.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:25:21 +02:00
c3901641d1 fix(agent): bound conversation history replayed to the LLM (A2)
Fix A2 of plans/2026-07-11-nomos-agent-code-review.md. chatWith replayed a
session's ENTIRE message history into the LLM's context on EVERY turn, no
windowing, no token budget — confirmed against a documented production case
(a single turn with 70 tool calls, messages up to 106KB). Every subsequent
turn of a long-running or heavily-autonomous task re-sent that ever-growing
history in full — a real cost/latency/eventual-context-limit risk for
exactly the tasks this system runs longest (many auto-continuation cycles).

Design call (flagged in the review as needing one before implementation):
a fixed-size window for LLM replay specifically, not the UI's own transcript
view. Simplest option that still keeps roughly the current task's working
context; a token-aware trim or LLM-summarize-on-drop are documented as
stretch options if 30 proves insufficient in practice.

- store.go: new getRecentMessages(ctx, sessionID, limit) — last `limit`
  messages in chronological order, plus whether older ones were omitted.
  getMessages (used by the UI's GET /sessions/{id}) is untouched and stays
  unbounded — the operator should still see a task's full history regardless
  of length; only what gets sent to the model is bounded.
- agent.go: chatWith uses getRecentMessages(sessionID, historyWindowSize=30)
  instead of the unbounded getMessages. When truncated, injects a system
  note telling the model explicitly that older turns exist but aren't shown,
  so it checks upsert_knowledge/search_knowledge rather than assuming
  something wasn't done just because it isn't visible.

New cmd/nomos/store_test.go: real Postgres integration tests (mirroring
internal/db/integration_test.go's throwaway-database pattern, guarded by
OIKOS_TEST_DATABASE_URL). TestGetRecentMessages_Truncation is the direct
proof for this fix (35 messages → 30 returned, correctly ordered,
truncated=true; 5 messages → all 5, truncated=false) — both cases run
against a fully-migrated database, not mocked. Also added
TestProposePlan_AppendVsReplace, closing part of the review's test-coverage
finding (E) by permanently regression-testing the earlier append-vs-replace
plan fix (commit 5384499), which had only been verified manually until now.

Verified live: inflated a real session to 42 persisted messages via direct
SQL, then continued it with a real chat call — the turn proceeded normally
(multiple real tool-call iterations, no crash, no context-length error);
nomos stayed healthy throughout. A3's incremental persistence separately
confirmed to have caught the 7 real tool calls made before the client
connection was cut, cleanly closing out both fixes' interaction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:22:30 +02:00
76f76308cc fix(agent): D1-D3 cleanups — dead code, N+1 query, unvalidated outcome enum
Fixes D1-D3 of plans/2026-07-11-nomos-agent-code-review.md:

- D1: deleted isTaskTool — defined, never called (dispatch already checks
  handleTaskTool's own `handled` return value).
- D2: recordTouched issued one SELECT per entity slug found in a tool call's
  args; batched into one `WHERE slug = ANY($1)` query. Verified live: a turn
  naming three separate entities recorded involves edges for all three via
  the single batched lookup.
- D3: complete_task's outcome had a declared enum (success|failure|partial)
  in its tool schema but nothing validated it — an out-of-enum value (model
  typo or a weaker model not respecting the schema) silently persisted as-is,
  with only "failure" special-cased (anything else became status='done'
  regardless of what the value actually said). Now validated in
  handleTaskTool: empty defaults to "success" (unchanged), a recognized value
  passes through, anything else defaults to "partial" (safer than silently
  treating an unrecognized value as success) with a warning logged. Verified
  live: instructed the agent to call complete_task with outcome="unclear" —
  persisted as outcome='partial', not the literal invalid string.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:13:48 +02:00
926969a03f fix(agent): live chat turns persist incrementally, survive client disconnect
Fix A3 of plans/2026-07-11-nomos-agent-code-review.md. handleChat only ever
saved the assistant message ONCE, after a.chat(...) returned, using
ctx := r.Context() for that write — the same context that cancels the instant
the client disconnects (Stop button, tab close, network blip). A disconnect
mid-turn meant the final save ran with an already-cancelled context and its
error was never checked: the entire turn's tool-call history was silently
lost from the persisted transcript, even though real work (executions
launched, knowledge written) had already happened server-side.

Brought handleChat in line with resumeSession's existing pattern
(continue.go): insert a placeholder assistant row immediately, update the
SAME row after every tool call. The key fix is WHICH context the writes use —
a new pctx := context.Background() for every DB write in this handler
(session creation/touch, the user message, question auto-close, the
placeholder + incremental updates, the title update), while ctx/r.Context()
still gates the agent's own work (a.chat) and the SSE writes exactly as
before — a disconnect still correctly stops the agent from doing further
work, it just no longer also erases what it already did.

Verified live: sent a message requiring 6 tool calls (get_entity/
get_relations/get_blast_radius on two targets) and force-killed the client
connection mid-stream with curl -m 12 (confirmed via exit code 28). Before
this fix the persisted transcript would show 0 tool-call entries; after,
all 12 raw tool_use/tool_result entries (6 calls × 2) were present and
correctly attributed by tool name — proving both that progress survives an
abort and that the incremental writes aren't corrupting the data.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:10:04 +02:00
c5ffaec85b fix(agent): panic recovery on every background goroutine (B1+B2)
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together,
since the right granularity for B1 in the auto-continuation worker turned
out to require B2's restructuring anyway (see below).

B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned
nothing before this — every explicitly-spawned goroutine (continuation
worker, resumed chat turns, async execution dispatch, the SSE listener, two
duplicate sshExec implementations' output-collector goroutines) crashed the
whole process on an unhandled panic, not just that one goroutine. More
consequential post-concurrency: more simultaneous unattended background work
means more surface area for one bad input to end every running task.

New internal/safego package: Go(label, fn) launches fn in a goroutine with a
recover-and-log wrapper. Applied at every bare `go` spawn site across the
three packages. Two sites needed bespoke handling instead of the generic
helper because their callers block on a channel and a silent recover would
just make them hang until timeout: sshExec's output-collector goroutine (two
near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go)
and httpapi's ListenAndServe goroutine — both now recover AND send a
synthetic error result so the waiting select unblocks immediately instead of
waiting out the full timeout.

httpapi's sseListener got extra treatment: its per-notification handling was
extracted into handleNotification with its own recover, so a panic decoding
ONE malformed pg_notify payload can't kill the listener goroutine for every
connected SSE client — the outer goroutine spawn only needs to guard the
connection setup/reconnect code around it.

B2: cmd/nomos/continue.go's processContinuations used to run every pending
continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the
ticker — meaning (a) task B's continuation waited for task A's full (up to
10-minute) resumed turn to finish first, undercutting this session's earlier
concurrency work on exactly the path autonomous tasks depend on most, and
(b) an unrecovered panic anywhere in that call chain didn't just crash the
process (B1) — even WITH B1's recovery wrapped only at the top-level worker
spawn, the panic would still unwind the ENTIRE ticker-loop goroutine,
silently ending auto-continuation for every task until nomos restarted.
Fixed by spawning each pending item via safego.Go individually: real
parallelism, and a bad item can now only ever take down its own goroutine.

Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete
proof — a deliberate panic inside Go() that would otherwise crash the whole
test binary; reaching the assertion after it IS the evidence recovery works.

Verified live against the rebuilt containers: full chat turn round-tripped
correctly (hostname lookup, 2 iterations, normal completion) — no regression
from threading safego.Go through the tool-dispatch/continuation paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:05:19 +02:00
3919ec37d7 fix(agent): word-boundary matching + contracted negatives in chat-assent
Fix A1 of plans/2026-07-11-nomos-agent-code-review.md. isAssent and
isTypedConfirmation used a space-padded word-boundary check for negation
words but a bare strings.Contains for assent/confirm words — confirmed live
via test probes: isAssent("...maybe yesterday's logs...") returned true
("yes" matched inside "yesterday"), and isTypedConfirmation("I haven't
confirmed anything yet") returned true ("confirm" matched inside "confirmed",
and "haven't" wasn't in negationWords — only "don't"/"do not" were).
isTypedConfirmation is the sole gate for DESTRUCTIVE actions, so the second
case meant a message merely stating something hadn't been confirmed could
read as an explicit confirmation.

- Replaced the ad-hoc space-padding/prefix-check negation logic with proper
  tokenization (wordTokenRe) + containsPhrase, matching WHOLE tokens/phrases
  only — never a mid-word substring. Handles curly apostrophes too (a
  pre-existing gap: the old straight-quote-only check would have missed
  "don't" typed with a smart quote).
- Added contracted negatives (haven't, hasn't, isn't, wasn't, aren't, can't,
  cannot, won't, wouldn't, shouldn't, didn't, doesn't) to negationWords.
  Deliberately did NOT add a bare "not" — too broad, would false-negative
  ordinary assent like "go ahead, this is not risky".
- Added regression tests for both confirmed cases plus a couple of adjacent
  ones (eyesight/isn't, can't confirm) so a future change can't silently
  reintroduce either bug.

All existing assent/confirmation tests pass unchanged — this is a pure
robustness fix, not a behavior change for any previously-correct case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:51:57 +02:00
df393152f6 docs: nomos agent code review — gaps and improvement plan
Full read-through of cmd/nomos/ (agent.go, store.go, main.go, continue.go,
assent.go, tasks.go). Findings, ranked:

- A1 (confirmed via runnable probe): isAssent/isTypedConfirmation use
  unpadded substring matching for assent/confirm words while negation uses
  word-boundary checks — "yes" matches inside "yesterday", "confirm" matches
  inside "confirmed" with no negation word covering contracted negatives
  ("haven't"). isTypedConfirmation gates DESTRUCTIVE actions specifically.
- A2: chatWith replays a session's ENTIRE message history every turn, no
  windowing/token budget — confirmed unbounded against a documented
  production case (70 tool calls, 106KB messages).
- A3: a live turn's tool-call history is lost entirely if the client
  disconnects mid-stream (single end-of-turn save using the same
  connection-tied, possibly-cancelled context) — resumeSession already has
  the fix pattern (incremental placeholder+update), handleChat doesn't use it.
- B1: zero recover() anywhere in cmd/nomos/internal/mcp/internal/httpapi —
  every explicitly-spawned goroutine (continuation worker, resumeSession,
  executeApprovedViaAPI, sse listeners) crashes the whole process on panic.
- B2: auto-continuation processes its batch sequentially, one full LLM turn
  at a time, undercutting this session's own concurrency work on exactly the
  path autonomous tasks depend on most.
- B3: no terminal state for a permanently-failed auto-continuation.
- C1: nomos's own gateway (port 8092, directly published + mesh-reachable)
  has ZERO authentication on any endpoint — chat, session read/delete,
  chat-assent approval of gated executions, all open to anyone on the LAN.
- D1-D3: dead code (isTaskTool unused), N+1 query in recordTouched, no
  validation on complete_task's outcome enum.
- E: zero automated tests for agent.go/store.go/main.go/tasks.go — including
  today's new safety-critical logic (session-scoped windows, mcpClientPool,
  proposePlan's append-vs-replace), verified only by live manual testing.
- F1: tool list + fleet snapshot re-fetched every turn (minor).

Prioritized implementation order in the doc: A1 → C1 → B1 → B2 → A3 → D1-3 →
A2 → B3/F1, tests landing alongside each fix rather than as a deferred pass.

Also archives the now-fully-shipped concurrent-task-execution plan to done/
(all 3 required fixes deployed this session; fix 4 explicitly deferred per
its own recommendation).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:47:17 +02:00
a4ea542f3e fix(concurrency): per-session MCP client pool — removes cross-task tool-call blocking
Fix 3 of plans/2026-07-11-concurrent-task-execution.md, the throughput one.
nomos held exactly one *mcpClient for the whole process, shared by every
/chat goroutine. Its mutex was held for the full duration of each tool
round-trip, and `run` executes its SSH command SYNCHRONOUSLY inside that
round-trip (capped at up to 10 minutes) — so while Task A was mid-`run`,
every other task's tool calls, even a trivial get_entity, queued behind that
single lock. Tasks could think (LLM calls) in parallel but never act in
parallel.

The MCP server has no per-connection state to protect (newServer returns one
shared *mcp.Server instance whose handlers close only over the DB
connection pool, already safe for concurrent use) — the mutex existed purely
because the client reused one stateful transport session. So the fix doesn't
touch the server at all:

- New mcpClientPool (cmd/nomos/main.go): one *mcpClient per session id,
  created lazily (a real MCP initialize handshake) on first use and cached;
  session-less traffic (the ephemeral no-DB-store path, the structured
  /query endpoint) gets its own fixed, reused key instead of a fresh
  connection per request. Idle clients (20 min past last use — long enough
  to outlive a single slow `run`) are evicted on a 5-minute sweep ticker.
- agent.go: `client *mcpClient` → `clients *mcpClientPool`; every call site
  (buildTools, fleetSnapshot, the tool-dispatch loop) now resolves its own
  session's client via clients.get(sessionID) instead of reaching for one
  shared field. A task's own tool calls stay sequential (already true — the
  agent loop calls tools one at a time within a turn) but no longer block
  anyone else's.
- main.go: handleQuery takes the pool instead of a client (keyed "query", a
  fixed non-session slot); shutdown calls pool.closeAll().

Verified live against the deployed stack: fired a slow-but-ungated command
(`ping -c 15 127.0.0.1`, read-only per policy's allowlist, no approval
needed) as Task A, then — 2s into A's run — a trivial hostname lookup as
Task B, both through the real /chat endpoint. Task A's ping genuinely ran
~14.3s (confirmed via its own execution record and the agent's reported
output). Task B returned in 6s total, well before A finished — proving it
was never queued behind A's connection. Before this fix, B would have been
forced to wait out A's entire ~14.3s hold on the single shared client.

This completes plans/2026-07-11-concurrent-task-execution.md's required
scope — only the explicitly optional/deferred Fix 4 (a concurrency/cost cap,
pending real usage data) remains.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:15:32 +02:00
6a8fb435ad fix(concurrency): per-session stream controllers, not one global slot
Closes the known gap flagged in the previous commit (9131559). A single
module-level `activeController` meant cancelStream()/newChat() always
aborted whichever stream was MOST RECENTLY STARTED, regardless of what the
operator was currently viewing: start Task A, switch to an already-loaded
Task B, click "New task" — the click's cancelStream() would silently abort
Task A's still-running turn, even though the operator was never looking at
it and never asked to cancel it.

- Replaced the single controller with activeControllers (Map<sessionID,
  AbortController>) plus pendingController for the brief pre-'session'-event
  window of a brand-new task. Registered immediately in sendMessage (keyed by
  the continuing session id right away, or held pending until the 'session'
  event assigns a new one) and cleaned up on completion.
- cancelStream() now looks up by $currentSession (falling back to
  pendingController when no session is assigned yet) — it can only ever
  touch the stream belonging to the view being left, never an unrelated
  background task's.
- newChat() unchanged in behavior (still calls cancelStream()), now correctly
  scoped through the above.

Verified live, reproducing the exact bug: started Task A (slow, 5 tool
calls), switched to an existing Task B, clicked "New task" while viewing
B — Task A was NOT aborted, ran to completion server-side with a full,
correct final summary (previously this exact sequence would have killed it).
Confirmed the positive path is unaffected: started a task, clicked Stop
while actively viewing it — input re-enabled, stream genuinely aborted
("BodyStreamBuffer was aborted"), turn stopped mid-flight as expected.

This closes out Fix 2's scope from
plans/2026-07-11-concurrent-task-execution.md; only Fix 3 (per-session MCP
client pool, throughput) and the optional Fix 4 (concurrency cap) remain.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:02:50 +02:00
9131559ebd fix(concurrency): guard chat.ts's stream callback against a stale session
Fix 2 of plans/2026-07-11-concurrent-task-execution.md. sendMessage's SSE
callback mutated the global messages/currentSession stores unconditionally,
assuming only one task's turn is ever in flight. It isn't — the backend runs
every /chat request as its own goroutine with no serialization. Switching to
a different task while a previous one was still streaming let that
background stream's later events (tool_use, text_delta, ..., and worst of
all 'done''s currentSession.set) get applied to whatever the operator is now
looking at: corrupting another task's transcript, or yanking the view back
to the one they left.

- Captures the session a stream belongs to (openedFor at call time, updated
  to the real id once the 'session' event assigns one) and checks
  $currentSession still matches before every messages/error/streaming
  mutation. The task keeps running server-side regardless — dropped events
  just mean the live view isn't watching it; navigating back re-hydrates via
  REST, same as already happens for auto-continuation.
- The 'session' event itself only claims currentSession if the operator
  hasn't already navigated elsewhere since the call started (comparing
  against openedFor, which is null for a brand-new task).
- loadSessionMessages/newChat now reset `streaming` to false unconditionally
  on navigation — needed so the new guard can't leave a DIFFERENT task's view
  stuck showing streaming=true (which would also silently stop startPolling's
  loop from ever applying updates, since it bails while $streaming is true).

Known residual gap, not fixed here (matches the plan's "contained fix, not a
rearchitecture" scope): activeController is still a single global slot, so
starting a new task while another is mid-stream, then clicking "New task"
again, aborts whichever stream that slot last pointed at rather than only
the one being left. A genuine multi-session controller/store is the
plan's deferred "stretch" fix, not required for correctness here.

Verified live: started Task A with a deliberately slow 4-tool-call turn,
switched to an existing Task B mid-stream — Task B's transcript stayed
correct with zero A-originated entries and the input was NOT stuck disabled.
Task A kept running and completed normally server-side (status=done, full
6-tool transcript, 5-entity graph); navigating back loaded its complete,
uncorrupted result via REST.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 18:34:02 +02:00
9ef1ba3702 fix(concurrency): scope assent/destructive windows to session, not just agent
Fix 1 of plans/2026-07-11-concurrent-task-execution.md — the safety-critical
one. The assent window (and destructive window) were keyed purely by agent
id ("assent_window.agent:<uuid>"). With one agent:nomos entity serving every
concurrent task, this meant approving Task A's plan opened a window that ANY
concurrently-running task's config-mutation/destructive actions could also
ride, auto-executing without their own approval.

- store.go / agent.go: assentWindowActive/openAssentWindow and
  destructiveWindowActive/openDestructiveWindow/destructiveWindowKey all gain
  a sessionID parameter; keys become
  "assent_window.agent:<id>.session:<sessionID>" and
  "destructive_window.agent:<id>.target:<slug>.session:<sessionID>". Missing
  session id fails closed (no window) rather than falling back to the old
  agent-wide key.
- continue.go: the auto-continuation worker's window check moved from once-
  per-batch to once-per-pending-item, scoped to that item's own session —
  it was previously checking ONE agent-wide window for a batch that can span
  multiple tasks.
- agent.go tool-dispatch: injects `_session_id` into a COPY of the wire args
  sent to the MCP server (never into the args used for the emitted/logged/
  persisted tool call, and never part of any tool's declared InputSchema —
  invisible to the model) so the gating checks on the OTHER side of the
  process boundary know which task is asking.
- internal/mcp/server.go: assentWindowActive/destructiveWindowActive/
  classifyAndGate gain the same sessionID parameter, read from
  args["_session_id"] at the three call sites (request_execution's
  apt_upgrade/pct_create branches, and the shared classifyAndGate used by
  restart/pct_exec/systemctl/run).

Verified against the live stack with the exact scenario from the plan: opened
an assent window for session A only, then called `run` with an identical
config-mutation command for session A (window open) and session B (same
agent, no window). A auto-ran (execution status completed); B correctly
queued for approval (pending_approval) instead of bleeding through — proven
at both the MCP response text and the executions table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 18:27:29 +02:00
6932eb5eed docs: plan concurrent task execution; archive completed tasks plan
New plan grounds three concurrency issues found by tracing the actual code
(not assumed): the assent/destructive windows are keyed by agent id only
(no session dimension), so an approved plan in one task can auto-run
unapproved actions in a concurrently-running task; nomos shares one
mutex-guarded MCP client across all sessions, so a single slow `run` call
serializes every other task's tool calls behind it; and chat.ts's SSE
callback has no session guard, so switching tasks mid-stream lets the
backgrounded task's events corrupt whatever's now displayed. Proposes
session-scoping the windows (critical/first), a frontend stream guard
(contained/second), a per-session MCP client pool (throughput/third), and
an optional concurrency cap (deferred pending real usage data).

Also archives the goal-oriented-chat-control-panel plan to done/ — all 7
phases shipped and are live in production (SHA e30813a) — fixing its
internal relative links for the new depth and pointing forward to the new
concurrency plan as follow-up hardening.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 18:14:35 +02:00
e30813a43d feat(tasks): make research-first / knowledge-write-back-last explicit steps
Closes the gap that made the knowledge loop optional/implicit: every
non-trivial task now has an EXPLICIT first plan step (research) and last
plan step (write back), not just background behavior the model might skip.

New MCP tools (the agent had no way to do these before — only REST endpoints
existed, unexposed to it):
- update_entity_attributes(slug, attributes): shallow-merge new/changed facts
  into an entity (an IP, a version, a discovered port) so a future task
  doesn't have to rediscover them from scratch. No approval required — this
  updates the knowledge graph, not live infra.
- create_relationship(source, target, type): record a discovered edge
  (depends-on, hosts, provides, ...). Idempotent, FK-validated against the
  ontology's relationship_types, no approval required.

SOUL.md: restructured the task loop so step 1 is explicitly "gather
knowledge, not just status" (get_entity_knowledge, search_knowledge,
get_relations, get_blast_radius, http_get) and the last step before
complete_task is explicitly "write back" (update_entity_attributes,
create_relationship, upsert_knowledge) — both called out as real plan
entries the operator should see in propose_plan, not silent side-work. This
is what prevents the graph drifting from reality and is the concrete
mechanism behind "tasks compound."

propose_plan's tool description reinforces the same first-step/last-step
convention at the call site.

Verified against the live stack: both tools registered and callable via MCP;
update_entity_attributes merged an attribute correctly; create_relationship
rejected an invalid type (FK violation, clear error) and succeeded with a
valid type+direction, confirmed idempotent (2 calls, 1 row).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 14:14:02 +02:00
5384499903 fix(tasks): plan panel showed only the latest step, not the full plan
Root cause: proposePlan unconditionally deleted and replaced the whole
session_plan_steps list on every call. The model isn't strictly held to
"call propose_plan once with the full list" — nothing stopped it (and
production evidence + live testing showed it happening) from calling
propose_plan once per step as it worked. Each such call wiped every
already-completed step, so the operator only ever saw the model's latest
single step ("1/1") instead of the real, growing plan.

Fix, two layers:
- store.go: proposePlan now only does a destructive replace when no step
  has left 'pending' yet (a genuine pre-execution revision). Once any step
  has started, a new call APPENDS after the current max seq instead of
  wiping — so the panel accumulates the full history regardless of how the
  model chooses to call the tool. plan.proposed now carries `appended` so
  the frontend knows whether to replace or append.
- workspace.ts: plan.proposed handler respects `appended` (update vs set).
- tasks.go / SOUL.md: strengthened the propose_plan description and task-
  loop guidance to call it ONCE with the complete step list end-to-end,
  using update_plan_step (not re-calling propose_plan) to advance — fixing
  the root behavioral cause, with the store-side append as a safety net
  that holds even if the model still calls it incrementally.

Verified: forced the exact incremental-call pattern (propose_plan with 1
step, mark it running, propose_plan again with 1 more step) — the second
call appended at seq 2 instead of erasing seq 1, and its plan.proposed
event carried appended=true.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 14:07:10 +02:00
991e7d0900 feat(tasks): phase 6 — live TaskContextPanel (goal, plan, question, entities)
Replaces the chat right rail's ad-hoc Digest+Graph stack with a single
TaskContextPanel that renders the task's live working state, driven by the
always-on events stream (not the per-turn chat SSE) so it keeps updating
during server-side auto-continuation/resume:

- GoalHeader: goal + status pill (planning/executing/awaiting_input/done/
  failed), sourced from the sessions list.
- PlanProgress: ordered steps with live status icons + progress bar, hydrated
  via new GET /sessions/{id}/plan; clicking a step with a target opens its
  EntitySheet (no fake "jump to transcript" — bits-ui Collapsible content
  isn't force-mounted, so a DOM-scroll jump would silently no-op for
  collapsed tool groups).
- OperatorQuestion: the pinned structured question card (prompt/why/entity
  chips/option buttons/free-text), hydrated via new GET /sessions/{id}/
  questions; answering POSTs to the existing answer endpoint.
- SessionGraph upgraded to a live entity panel: entity.touched pulses the
  node (animated ring) and shows "Now touching <slug>"; health.changed shows
  a transient diff badge for touched entities.
- SessionDigest gains a success/failure/partial outcome banner and now also
  refetches when the task's status changes, not just on session switch.

Two bugs found and fixed while wiring this up:
- workspace.ts's status-refresh trigger only covered goal.set/task.status;
  question.raised/answered didn't refresh the sessions list, so GoalHeader's
  pill went stale after answering via the panel (resumeSession runs entirely
  server-side — no client 'done' event to piggyback a refresh on). Now every
  status-affecting event triggers the (debounced) refetch.
- Forgot to rebuild the nomos container after adding the /plan and
  /questions endpoints, so they silently fell through to the old default GET
  handler — caught via a live curl diff against the running container,
  not a code read.

Verified end-to-end against the live stack: goal/plan/question all update
without a reload as the agent works; answering a question via the panel
resumes the agent and the header pill correctly flips to Executing;
entity.touched pulses the live graph.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:52:36 +02:00
413bf54daf feat(tasks): task board UI — chat window becomes Tasks + card grid
Reframes the chat surface as tasks:
- New Tasks.svelte: a card grid of tasks, each showing status (Running /
  Needs input / Done / Failed), the goal as title, the outcome summary, and
  relative time; filterable by status with live counts; delete on hover;
  "New task" and per-card click open the conversation.
- Board updates LIVE off the events stream (goal.set / task.status /
  question.*) via an explicit liveEvents.subscribe with a debounced refetch —
  scanning all events newer than the last seen, since entity.touched bursts
  bury task events below index 0.
- App shell: primary nav "Chat" → "Tasks" (board is now the home route),
  "New chat" → "New task", conversation header gets a Tasks / Conversation
  breadcrumb. Removed the superseded Sessions page.
- api.ts Session type carries the task fields (goal/status/outcome/summary).

Also fixes a pre-existing SSE bug that blocked ALL live updates app-wide:
writeSSE emitted `event: <type>`, which EventSource only delivers to
addEventListener(type) handlers — but stores/events.ts (and every page reading
liveEvents) consumes via onmessage, which never fires for named events. So the
live stream delivered nothing to the UI. Dropped the event-name line; the type
is already in the JSON payload, and new event types now need zero client
changes. SSE test still green (it parses data: lines).

Verified in the browser against the live stack: the board renders 50 tasks
with correct status buckets; a goal-driven task appears and flips to a Done
card with its summary in real time without a reload; Events page confirms the
stream now delivers to onmessage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:22:27 +02:00
014e5c74e0 feat(tasks): phase 5 — ask_operator (structured question, pause, resume)
The last backend piece: when the agent hits a decision only the operator can
make, it surfaces a structured question instead of guessing or stalling.

- ask_operator(prompt, why?, options?, context_entities?): nomos-local tool
  that records a session_questions row, moves the task to awaiting_input, emits
  question.raised, and ENDS the turn (the agent loop returns after it, so the
  agent can't barrel past its own question). The prompt becomes the assistant's
  visible message so the question also shows inline in the transcript.
- Two resume paths, both close the question + emit question.answered + return
  the task to executing:
  - Panel: POST /sessions/{id}/questions/{qid}/answer → resumes the agent in the
    background with the answer injected (reusing the continuation machinery,
    refactored continueSession → resumeSession). Returns 202; the reply lands via
    message polling.
  - Chat reply: the next chat message on a task with an open question IS the
    answer — auto-closed in handleChat; the turn itself is the resume.

Verified end-to-end: forcing a decision paused the task at awaiting_input with
the structured question (prompt/why/options/entities); a panel answer resumed
the agent (it acknowledged host:strong and continued); a plain chat reply
auto-closed a second question. Cleanup + tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:00:15 +02:00
be3ce761d4 feat(tasks): phase 4 — structured plan steps (set_goal/propose_plan/update_plan_step)
Gives a task a legible, live-advancing plan via three more nomos-local tools:

- set_goal(goal): records the task goal, status → planning, emits goal.set.
- propose_plan(steps[]): persists ordered steps (clean replace for v1 — a
  revision starts a new list), status → executing, emits plan.proposed with
  the persisted steps (id+seq) so the panel can address them.
- update_plan_step(seq, status, execution_id?): advances a step, stamping
  started_at/finished_at, emits plan.step.started/finished. Anchors the event
  to the step's target entity when it has one.

Belt-and-suspenders: when an execution linked to a step reaches a terminal
state, the api auto-closes the step (closePlanStepForExecution in
emitExecutionEvent) and emits plan.step.finished — so the board stays honest
even if the agent forgets to close a step it started.

Verified end-to-end: a goal-driven task fired goal.set → plan.proposed →
2× step.started/finished → task.status on the SSE stream; both steps persisted
done with start/finish timestamps; status progressed planning→executing→done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:50:27 +02:00
532310bb4b feat(tasks): phase 3 — close the knowledge loop (complete_task + retrieval)
Adds the compounding knowledge loop the task model is built around:

- complete_task(outcome, summary): a nomos-LOCAL, session-scoped tool (the
  shared MCP server has no session id). Introduces the local-tool mechanism —
  buildTools appends task tools, the agent loop routes them to handleTaskTool
  instead of the MCP client. Sets the task's terminal status/outcome/summary,
  mirrors it onto the task entity, and emits task.status.
- Knowledge → task linkage: after a successful upsert_knowledge in a task,
  nomos links the note to the task entity (documents) and emits
  knowledge.recorded, so the task's outcome view shows what it learned. The
  note's about-link to the involved entity (written by upsert_knowledge) is the
  retrieval path future tasks use.
- SOUL: every chat is a task loop — retrieve prior knowledge FIRST
  (get_entity_knowledge on the target), plan, execute, record learnings, then
  complete_task. Scales down for trivial read-only tasks.
- deleteSession now cleans up the task entity, its relationships, and its
  task-scoped events (was orphaning them); the knowledge doc itself and its
  about-links survive, as knowledge should outlive the task.

Verified end-to-end: a task recorded a note and completed; task.status +
knowledge.recorded hit the SSE stream; status=done/outcome=success persisted;
the note linked to both lxc:caddy (retrieval) and the task; a future
get_entity_knowledge(lxc:caddy) surfaces it; delete cleaned edges+events (0/0/0)
while the knowledge survived.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:43:29 +02:00
3dba2e550a feat(tasks): phase 2 — entity.touched events + task→entity involves edges
As the agent runs a task, record which entities each tool call references:
write an idempotent task —involves→ entity relationship and publish one
entity.touched event per entity (correlation_id = session, data {slug,tool}).
Extracted from tool ARGS only — never results — so a bulk fleet query can't
drag every entity into the task graph; bulk/no-slug tools stay silent.

Emitted from the nomos agent loop rather than the shared MCP wrapper, which
has no session id. The involves edges make a task's graph neighborhood its
involved-entity set (queryable via get_relations) — the substrate for the
knowledge loop; the events are the live pulse the context panel consumes in
phase 6.

Verified end-to-end on the local stack: a chat referencing lxc:caddy/lxc:gitea
produced entity.touched on the browser SSE stream with slug+tool+correlation,
and exactly one involves edge per entity despite repeated touches.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:35:27 +02:00
72e9fe534e feat(tasks): phase 1 — elevate chat session to a task (schema + task entity)
Migration 018 adds goal/status/outcome/summary/entity_id to agent_sessions
and creates session_plan_steps + session_questions. Registers a 'task'
entity type and an 'involves' (task→entity) relationship in the ontology
so each session anchors its knowledge and involved-entity edges on the
existing relationships graph.

nomos createSession now mints a task:<session-id> entity (type task) and
links it via agent_sessions.entity_id — best-effort so chat never blocks on
it. listSessions/GET /sessions surface the new task fields.

No behaviour change yet; this is the data foundation for the task board and
live context panel. Verified end-to-end against the local stack: migration
applied, ontology ingested (60 types/47 rels), a new session mints a linked
task entity and the API returns status/goal/entity_id.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:25:28 +02:00
eed6e3b1c5 docs: task-centric chat plan (goal → single-approval → autonomous + knowledge loop)
Reframes the chat surface as a board of tasks: each task carries a goal,
a plan approved once, a lifecycle status, an outcome, and a knowledge
loop that links learnings to the involved entities (and the task entity
itself) via relationships so future tasks compound. Supersedes the
sidebar-only framing and the free-form chat portion of the control-room
web UI plan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:17:36 +02:00
ef5a92269b docs: reconcile plans/ status against actual code state
Audited all 10 active plan docs against the codebase (not just commit
titles). 5 were fully shipped and stale-tagged "Planned"/"In Progress" —
moved to done/ with verification notes. The other 4 got corrected
Planned→In Progress status plus concrete remaining-gap notes so the next
pass doesn't re-derive what's already done.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 11:42:26 +02:00
52e16e04ca feat: Learning page — capability timeline + trend, built on real data
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The plan's "learning view" (runbook success-rate trends, promoted
skills, capability timeline) assumes the patterns/skills/feedback
pipeline is populated. It isn't: all three tables are empty in
production and nothing in the codebase ever writes to feedback, so
building the UI against them today would ship a permanently-empty
page. Scoped instead around data that's real and growing —
executions — while still wiring up /patterns and /skills so the page
needs no rework once that pipeline exists.

New /api/v1/learning/timeline (per-verb first-success date + success
rate, parsed via the existing splitAction helper) and
/api/v1/learning/trend (30-day daily success/fail counts), both
read-only queries against executions. Patterns and skills sections
call the existing (untouched) ListPatterns/ListSkills endpoints and
render an explanatory empty state instead of nothing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 21:28:19 +02:00
6192c35c10 fix: close approval bypass in restart/systemctl/pct_exec
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Found live: a chat request to restart caddy (the reverse proxy for the
whole fleet) executed instantly over SSH with zero approval. Root
cause was in request_execution's legacy handler — restart, pct_exec,
and systemctl (outside enable/disable) executed immediately with a
hardcoded risk_class='reversible_low' that was never actually checked
against anything, bypassing the classifier entirely. Only the `run`
tool's commands were ever gated.

Extracted the run tool's classify -> execute-or-queue logic into a
shared classifyAndGate() and route restart/pct_exec/systemctl through
it too, so every mutating path — regardless of which tool the model
reaches for — gets the same read-only/config-mutation/destructive
classification and approval gate. systemctl restart is already covered
by an existing classifier test (config_mutation), so no new test
needed; the gap was that request_execution never called the
classifier at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 21:03:38 +02:00
682326382e feat: surface blast radius on approval cards
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Pending-approval cards showed target and risk but not what else the
action would affect — the operator approved config_mutation/destructive
commands blind to downstream impact, even though the graph-walk
(blast_radius() SQL, GetBlastRadius endpoint) already existed and was
just never wired into the approval path.

Fetch it once per pending approval and render "Affects N downstream: …"
on both the normal and destructive approval cards, reusing the existing
fetchBlastRadius() API client function which was already written but
unused anywhere.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 20:52:05 +02:00
ac48390796 feat: global activity feed + session digest
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Ops "Executions" tab showed raw target UUIDs, alphabetical (not
recency) order, and a stale status vocabulary from an earlier schema
iteration — never actually usable as a live "what's happening" view.
Replaced with a new recency-ordered /api/v1/activity/recent endpoint
and matching table (human-readable action summaries, risk/status
badges, duration, inline error preview).

Also added /api/v1/activity/session/{id} + a collapsible SessionDigest
panel in the chat rail, answering "what did this session actually do"
(executions by status, entities touched, knowledge written) — the
missing piece for proactive outcome reporting to be visible in the UI,
not just in the chat transcript.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 20:09:09 +02:00
40999b0b40 fix: knowledge/recent returned empty items — timestamptz couldn't scan into string
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live immediately after deploying: the endpoint returned 200 with
correct-looking stats (total=56, agent_authored=2) but items=[] always,
regardless of limit/source. Root cause: pgx v5 can't scan a timestamptz
column directly into a Go string — Scan() errored on every single row, and
that error was silently swallowed by a bare `continue`, so every row was
dropped with no trace in the logs. Fixed by casting updated_at::text in the
SQL (matching how every other handler in this codebase already returns
timestamps) and logging scan failures instead of swallowing them, so this
class of bug can't hide silently again.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 19:26:48 +02:00
ec41c0b828 feat: learning view — make the growing knowledge base visible
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
First slice of the observability/learning UI (the "see the system come alive
and learn" ask). The Knowledge page was search-only — blank until you typed —
so the knowledge Nomos now writes via upsert_knowledge was invisible unless
you knew to search for it. Now the page LEADS with what the system knows and
is learning:

- internal/httpapi/knowledge.go: GET /api/v1/knowledge/recent — recency-ordered
  knowledge + a stats header (total, agent-authored, learned-this-week,
  by-kind). Custom route (not OpenAPI-generated), same auth as the rest.
- web Knowledge page rewrite: stat cards up top (Total / Written by Nomos /
  Learned this week / runbooks-investigations), then a "Recently learned" feed
  with agent-authored notes highlighted and badged "learned by Nomos", tags,
  and relative timestamps. A toggle filters to Nomos-only. Search still works,
  now as a mode you enter/clear rather than the whole page.

This turns "the system is getting smarter" from a claim into something you
watch fill up: every gotcha the agent records shows here within seconds.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 19:22:35 +02:00
60edff2065 feat: knowledge write-back (upsert_knowledge) + proactive outcome reporting
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
From the last (successful) TypeType deploy session, two gaps the operator hit:

1. Knowledge write-back — the missing half of the loop.
   The agent could read the knowledge base (search_knowledge/get_entity_knowledge)
   but had no way to WRITE it, so everything it learned (the Dragonfly memlock
   rlimit gotcha, the NAT-hairpin DNS issue, etc.) lived only in an ephemeral
   chat message and was lost — the system could never actually "get better."
   This is the `upsert_knowledge` MCP tool the 2026-07-08 gaps plan called for.
   - internal/mcp/server.go: upsert_knowledge(title, content, about?, tags?,
     kind?) writes a document/investigation/runbook entity + knowledge_entities
     row (search column is generated), upserts by slug so re-titling updates in
     place, and optionally links it to the entity it's about so
     get_entity_knowledge surfaces it there.
   - SOUL.md: capture non-obvious findings/deploys/gotchas as part of finishing
     work, not only when asked "what did we learn".

2. "I had to ask for status multiple times."
   The clearest cause: a long working turn (64 tool calls) that exhausted the
   iteration cap ended with a bare "max iterations reached without final
   answer" — a dead end that forced the operator to ask what happened.
   - cmd/nomos/agent.go: on exhaustion, make one final no-tools LLM call
     (finalSummary) asking for a status report — what was accomplished, current
     state, what remains — so the turn always ends with a real outcome.
   - maxIterations 25 -> 40 (the decomposed per-step pct_create flow legitimately
     needs more steps).
   - SOUL.md: always end a turn with a clear outcome; never end silently or on a
     bare tool call — the operator can't see the tools working and reads silence
     as "nothing happened".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 19:07:11 +02:00
233b5e4519 feat: live visibility into what the agent is running (no more silent waiting)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Operator: "I'd like to be able to see in the chat what the agent is actually
running, right now I just wait while nothing happens." Two compounding gaps:

1. The auto-continuation worker (cmd/nomos/continue.go) had zero live push —
   its result only appeared on a manual page reload, so approving a plan and
   watching the chat looked completely dead even while the agent was actively
   working.
2. Even with polling, continueSession only persisted ONE message at the very
   end of a continuation — a continuation that runs several tool calls before
   concluding would still show total silence for however long that took.

Fixed both:
- web/src/lib/stores/chat.ts: polls the current session's messages every 3s
  between turns (never while a live stream owns the message list) and merges
  in anything new. Started after a live turn ends and when a session loads;
  stopped on new-chat/session-switch.
- cmd/nomos/store.go: insertMessageReturningID/updateMessage — lets a message
  be created as a placeholder and updated in place.
- cmd/nomos/continue.go: continueSession now inserts a placeholder the
  instant it starts (renders as the existing "thinking" dots — immediate
  feedback that something is happening) and updates that SAME row after
  EVERY tool call, not just at the end. A poll within ~3s of any tool call
  landing shows it — individual `run` commands appear as the agent issues
  them, not just the final rolled-up summary.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 15:20:10 +02:00
13458e467c fix: the actual root bug — assent-window auto-approve never dispatched work at all
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The previous commit fixed a context-cancellation bug in the auto-approve path
and appeared to fix things, but re-testing end-to-end after deploy showed the
execution STILL never completed — just via a different symptom
("no pending execution found for approval" in the logs). Dug further and
found the real, deeper bug underneath: this whole mechanism has never
actually worked.

autoApprove() directly flipped BOTH approvals.status and executions.status to
'approved' via raw SQL, then called executeApprovedViaAPI to POST to the
decision endpoint. But DecideApproval's own logic specifically looks for the
execution still at status='pending_approval' to find and dispatch the real
SSH work (executeApprovedAction) — autoApprove's premature flip meant that
lookup always found zero rows. DecideApproval's UpdateApprovalStatus call
also silently no-ops the same way (sqlc :exec doesn't surface "0 rows
affected" as an error). Every assent-window auto-approved pct_create/
apt_upgrade has been sitting at 'approved' forever with the real work never
triggered — indistinguishable from "still running" until you check.

Fix: remove autoApprove() entirely. Call executeApprovedViaAPI directly
against the untouched pending_approval row from createApproval — identical
to the manual Approve-button path, just without the human click. DecideApproval
is now the single place that transitions status and dispatches, for both the
manual and auto-approved paths, closing the class of bug where two code paths
raced to do the same state transition.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 15:00:40 +02:00
7387df3276 fix: assent-window auto-approve goroutine used the request-scoped context
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live testing the new atomic pct_create: an assent-window
auto-approved pct_create appeared to "run" (logged "auto-approved... running
now") but the execution stayed stuck at 'approved' forever. Root cause:
`go executeApprovedViaAPI(ctx, ...)` passed the MCP tool-call's own context —
which is cancelled the instant the triggering /chat request's HTTP response
completes, i.e. on every normal turn. The spawned goroutine's POST to the
approval-decision endpoint died with "context canceled" before it could even
start the real work, and nothing surfaced this to the operator or the agent —
the execution just sat at 'approved' with no error, indistinguishable from
"still running."

This is exactly the context-lifetime bug class httpapi's own approval
goroutine (executeApprovedAction) already avoided by using
context.Background() — it had just been missed in these two call sites
(apt_upgrade and pct_create auto-approve). Fixed both to use
context.Background(), matching the correct pattern already in place
elsewhere. Audited for other goroutines spawned with a request-scoped ctx —
none found; the sshExec internal goroutines are synchronous/waited-on via
select and correctly scoped to the call.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 14:51:52 +02:00
2e922f6421 feat: decompose pct_create into atomic create + agent-driven install; add scoped destructive window
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Closes the two remaining open points from the auto-continuation work.

1. Atomic pct_create (observability, the bigger of the two):
   pct_create used to bundle create + apt install + post_install script into
   one black-box multi-minute SSH call — the agent got back a single opaque
   success/fail with no way to see (or fix) which step actually broke.
   Removed the whole post-create provisioning block (and the now-dead
   provisionScript/sanitizePkgs helpers + their tests). pct_create is now
   create + start + register ONLY — fast, and its result is fed back to the
   agent via auto-continuation almost immediately. The agent installs
   packages and runs setup as its OWN sequence of `run` calls against the new
   lxc:<hostname>, observing each command's real output and able to diagnose
   and retry exactly the step that failed — the same recovery loop already
   proven for the general case, now applied to installs too, instead of
   requiring a separate black-box mechanism.
   - services/post_install removed from the pct_create params struct and
     from the MCP tool schema/SOUL.md docs.
   - SOUL.md: explains the new flow, moves the Docker CLI gotcha and DNS
     troubleshooting guidance to be steps the agent runs itself.

2. Scoped destructive window (targeted autonomy for recovery):
   Verified live in the previous session that a destructive recovery (a
   failed destroy needing stop-then-destroy on the same container) required
   TWO separate typed confirmations for what was clearly one recovery
   action. Added a narrow, TARGET-scoped 15-minute grant
   (destructive_window.agent:<id>.target:<slug> in autonomy_settings,
   shared key format across cmd/nomos and internal/mcp) that opens only
   after an EXPLICIT typed confirmation (never loose assent) or an explicit
   button-approval of a destructive step, and only ever covers further
   destructive commands against that SAME target. A different target always
   needs its own fresh confirmation — this narrows risk instead of loosening
   it globally, unlike broadening the general assent window to cover
   destructive actions would have.
   - cmd/nomos/store.go: openDestructiveWindow/destructiveWindowActive/
     executionTarget.
   - cmd/nomos/agent.go: opens the window when a typed confirmation grants a
     destructive chat-assent execution.
   - internal/mcp/server.go: `run` tool checks the window before gating a
     destructive command; auto-runs if active.
   - internal/httpapi/phase3.go: DecideApproval opens the same window when a
     destructive execution is approved via the button/API, for parity with
     the chat-assent path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 14:45:09 +02:00
6f9998fa29 fix: auto-continuation silently dropped LLM errors + added outer retry
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live: the new auto-continuation worker (previous commit) worked end
to end for the happy path (provision -> auto-verify -> report success, zero
operator ticks). But testing a failure-recovery case (a destroy that failed
because the container was still running) surfaced a real bug: continueSession's
emit closure only captured "text" events, so when chatWith ended the turn on
an "error" event (LLM returned an empty/refusal response, internal retry also
empty), the worker persisted a completely blank, uninformative "auto" message
— no sign anything had gone wrong, undermining observability of the very
mechanism just built.

- Capture "error" events and, if the turn produced no text/tool_calls at all,
  persist an explanatory placeholder instead of blank.
- Add one outer retry of the whole chatWith call when the first attempt
  produces nothing — the principle behind this whole feature ("don't give up
  on the first error") should apply to the continuation mechanism itself, not
  just the homelab commands it's continuing.

Also verified live: recovery-from-failure works via the normal chat path once
prompted, and cleaned up the test container.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 14:30:23 +02:00
d2f749d33d feat: event-driven auto-continuation — agent runs an approved plan to completion
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)

This makes the system the event loop instead:

- migrations/017: nomos_plan_executions links each gated execution to the chat
  session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
  to the session. A background worker (continue.go) polls for those executions
  reaching a terminal state and — while the agent has an open assent window (an
  approved plan is in flight) — re-invokes the agent with the result
  ("execution X completed/failed: <result>"), so it proceeds to the next step
  or diagnoses+fixes the failure, with no operator tick. Guarded against loops
  (mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
  replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
  opens the assent window, so auto-continuation works regardless of how the
  operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
  don't poll get_execution_status, don't wait for "continue"; end the turn and
  keep going step by step until the goal is verified or a genuine blocker.

This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 14:19:43 +02:00
c3699157ae fix: chat-assent chicken-and-egg + agent stops after errors
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Three fixes for the session where the agent proposed a plan, waited
for 'proceed', then re-queued instead of being auto-approved:

1. Chat-assent fallback: when the operator says 'proceed' but the
   preceding turn had NO pending approvals (agent proposed plan in text
   without calling request_execution), inject a system note telling the
   agent to execute the plan now. Opens the assent window so subsequent
   config_mutation commands auto-run.

2. SOUL.md: instruct agent to ALWAYS call request_execution/run when
   proposing a plan, not wait for 'proceed' first. This ensures a
   pending approval exists for chat-assent to grant.

3. SOUL.md: stronger Docker instructions — Debian 13's docker.io package
   installs the daemon but NOT the docker CLI binary. Must use
   get.docker.com in post_install. Added 'handling errors' section:
   diagnose, try alternatives, continue — don't stop after one failure.
2026-07-10 13:51:04 +02:00
7ff344ab47 feat: request_execution respects assent window — pct_create and apt_upgrade auto-approve
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
When the operator has approved a plan via chat assent (assent window
active), pct_create and apt_upgrade now auto-approve and execute
instead of queuing for a separate approval round. The auto-approve
path updates the approval+execution status in the DB, then calls the
HTTP API's decision endpoint to trigger executeApprovedAction — same
code path as a manual Approve button, consistent audit trail.
2026-07-10 13:33:31 +02:00
657e1a8be1 feat: assent window + compound read-only classification + continue-after-approval
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Agent stopped after every approval step, forcing operator to type
'continue' 7× per deploy session. Root causes and fixes:

1. Compound read-only commands (e.g. 'systemctl status; journalctl')
   defaulted to config_mutation — now splits on ;/&&/||/| and classifies
   as read_only if all segments are inspection verbs. Added grep, wc,
   sort, uniq, cut, tr, dpkg -l, apt list, docker stats to allowlist.

2. curl|sh was classified destructive, forcing typed confirmation for
   legitimate installs (get.docker.com). Demoted to config_mutation —
   loose assent grants it, no typed phrase needed.

3. SOUL.md said 'STOP after queuing' — replaced with 'continue working
   on non-blocked steps'. Added assent window section instructing agent
   to carry out the full plan after approval.

4. Assent window: when operator approves a plan via chat assent, a
   30-minute window opens where config_mutation commands auto-run
   without re-approval. Agent writes expiry to autonomy_settings; MCP
   run tool checks it before gating. Destructive never auto-runs.

5. System note after approval now says 'CONTINUE executing the full
   plan — do not stop and wait for continue.'
2026-07-10 13:10:57 +02:00
7a7ce2b89b fix: gateway pre-flight check could never actually fail
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live that after deploying the "fixed" bridge-bound pre-flight, it
still let a known-bad vmbr0+192.168.8.2 config straight through to a full
pct_create with no error. Root cause: the check used
`strings.Contains(pingOut, "REACHABLE")` against markers "REACHABLE" /
"UNREACHABLE" — but "UNREACHABLE" contains "REACHABLE" as a substring, so the
containment check was true for BOTH outcomes. The pre-flight was structurally
incapable of ever failing, regardless of the actual ping result.

Fixed with distinct, non-overlapping markers (PREFLIGHT_OK/PREFLIGHT_FAIL)
and exact-match comparison, pulled into a small gatewayPreflightPassed()
helper with a unit test asserting the exact historical bug case
("UNREACHABLE" must be false) so this bug class can't silently recur.

Re-verified live end-to-end: manually re-tested the exact ping command
(confirmed UNREACHABLE via vmbr0), and this was caught only by actually
running the check against production, not by reading the code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 12:02:42 +02:00
3f3de18b23 fix: pre-flight gateway ping must bind to the specific bridge, not the host default route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live that the pre-flight check added in the previous commit had a
real gap: a plain `ping <gateway>` from the Proxmox host succeeds via the
HOST's own routing table (which can have routes to a subnet through paths
the host alone knows about), even when the CONTAINER — attached via a plain
bridge with only a naive on-link default route — can never actually ARP that
gateway. Confirmed by creating a real test container on vmbr0 with
gw=192.168.8.2: the host-wide ping had said "reachable," but pinging from
inside the container showed 100% packet loss. Fixed by binding the pre-flight
ping to the specific requested bridge (`ping -I <bridge>`), which correctly
rejects vmbr0 for that gateway instead of false-positiving via the host's
broader routing table.

Also confirmed live: vmbr1 does exist and is up on strong (contrary to the
possibly-stale host doc), matching what romm/seanime's docs already said.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 11:55:53 +02:00
82b0ad2298 fix: pct_create fast gateway pre-flight + bridge param (real root cause of TypeType's DNS failures)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Investigated why the operator couldn't get past "no DNS/connectivity" across
multiple retries even after Nomos correctly diagnosed and fixed the gateway
(192.168.8.1 -> 192.168.8.2). It still failed. Root cause, confirmed from
strong's own documented network topology: on `strong`, vmbr0 physically
bridges only to 192.168.178.0/24 — the 192.168.8.0/24 service network is
reached via a Fritz!Box static route, not a local bridge. A container
attached to vmbr0 can never reach a 192.168.8.x gateway no matter which
address in that range is picked; ARP for it just gets silently dropped
(matching the earlier hang symptom). The gateway was never the problem — the
bridge was. 192.168.8.0/24 is also segmented into /28 blocks each with their
own gateway (192.168.8.2 is only the .0-.15 block's gateway), so even a
correct bridge with a copy-pasted gateway from a different block would still
fail.

No amount of retrying with a different gateway guess could have fixed this —
the missing fact (which bridge reaches which subnet, and the per-/28 gateway)
isn't inferable from the subnet alone.

- pct_create gets a `bridge` param (was hardcoded to vmbr0) so a correct
  bridge can actually be requested once known.
- Fast pre-flight: for any static IP, ping the gateway from the target HOST
  before creating anything. Was: a bad config took a multi-minute hang (or,
  after last commit's timeout fix, ~2min) before failing. Now: ~2 seconds,
  with a message that explicitly says not to guess a different gateway in
  the same subnet — find a real neighbor's config or use DHCP.
- SOUL.md: DHCP is now framed as the default, not a fallback; static IP
  requires finding an existing LXC on the same host in the same /28 and
  copying its bridge+gateway verbatim — inventing one is explicitly called
  out as the failure mode that caused this exact incident.
- MCP tool schema: pct_create's params description now documents `bridge`
  and the neighbor-copy rule directly in what the model reads at call time,
  not just in SOUL.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 11:48:06 +02:00
8950bada44 fix: sshExec had no timeout — a hung remote command blocked forever
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).

Two compounding bugs made this unrecoverable without manual intervention:

1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
   NO execution timeout — `session.CombinedOutput()` blocks until the remote
   command exits, with no deadline. A hung remote process blocks the Go
   goroutine forever; the execution can never leave 'running', and the
   operator has no way to make it stop. Fixed: both now race the SSH call
   against a 10-minute hard timeout, closing the session/client and
   returning a clear "timed out after 10m0s" error if exceeded. (The
   mcp/server.go copy also still had the original "swallowed non-zero exit"
   bug from before that fix was applied to httpapi's copy only — fixed here
   too.)

2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
   connectivity — it doesn't; a black-holed network can make each call hang
   far past the resolver's nominal timeout, so the documented "~90s" budget
   was never real. Wrapped every attempt in `timeout 3` so the wall-clock
   budget is now actually enforced (~2min worst case), and the failure
   message now suggests checking the net0 gateway.

Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):

- InlineApproval's track() polling loop had its own ~6min ceiling and simply
  STOPPED polling after that — silently going stale before the backend (now
  correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
  with margin, and added a distinct 'stalled' state if that's ever exceeded
  (explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
  execution's created_at), the actual command being run, and the execution
  ID — previously just a static "this can take a minute" with zero
  information. Also added command display to the destructive pending-
  approval card for full transparency before confirming.

Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 10:45:47 +02:00
f936098364 fix: approval UI was never mounted; add typed confirmation for destructive
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Root cause of "chat gave me no further feedback — had to go to Ops": two
compounding bugs, found by reading the actual production session transcript.

1. InlineApproval.svelte — all of last session's live-status/self-heal work —
   was never imported or rendered anywhere. Chat.svelte had its own separate,
   much dumber approval bar (no status tracking, no destructive handling, just
   silently disappears after clicking) that WAS the one users actually saw.
   Deleted the dead bar and its state; InlineApproval now renders per-message.

2. chat.ts's extractApprovals hardcoded `tool.name === 'request_execution'`,
   so any approval raised by the newer `run` tool was invisible — no card, no
   feedback, nothing to self-heal, forcing the operator to the Ops page with
   zero acknowledgement in the conversation. This was the actual proximate
   cause of last night's destroy-135 session. Fixed to match on response
   shape, not tool name, so it doesn't silently break again for the next new
   gated tool.

3. Nomos was telling operators "type something like 'I confirm destroy 135'"
   for destructive actions (SOUL.md) but no backend path ever consumed that
   phrase — chat-assent explicitly (and correctly) excludes destructive from
   loose assent, but I never built the alternative. Added
   isTypedConfirmation() (cmd/nomos/assent.go): stricter than loose assent,
   requires an explicit "confirm" statement, only applies to destructive-
   flagged pending approvals.

4. InlineApproval's completed-state hardcoded "Provisioned successfully" —
   wrong/confusing for a destroy or arbitrary `run` command. Now says
   "Completed on <target>" and shows the actual command output, verified live
   against the real destroy-135 execution.

Verified live in a real browser against the production API/DB (dev server
proxying to :8090): the historical stuck session now retroactively renders
both executions as resolved with correct wording and real output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 10:01:53 +02:00
d08a985ea9 fix: execution slug collision under back-to-back requests
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Live testing hit `entities_slug_key` violations: exec slugs used an 8-char
prefix of a UUIDv7, whose leading bytes encode a millisecond timestamp — two
executions created seconds apart can share a prefix. Use the full UUID
(guaranteed unique) for the exec entity's slug/name in request_execution, the
new `run` tool, and the REST RequestExecution handler — all three had the
same pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:37:08 +02:00
9539759db6 fix: NULL-scan bug in LXC target resolution for entities without a host attribute
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Found live: `run` against lxc:caddy failed with "missing pve_id" even though
pve_id=121 was present — caddy is an inventory-seeded LXC with no `host`
attribute at all (only pct_create-provisioned LXCs set one). The combined
query scanned attributes->>'host' (SQL NULL) into a plain Go string, which
errors the whole Scan — including the pve_id column that scanned fine.
COALESCE the host column to '' so a missing host attribute degrades to the
documented default instead of failing the whole resolution.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:32:45 +02:00
d52968876a feat: general gated run primitive + chat-assent approval (Layer 0)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Implements the first slice of plans/2026-07-10-general-gated-execution.md:
Nomos gets one general execution tool instead of only a fixed action enum,
gated by an automatic risk classifier, and approval can be granted by the
operator just replying in chat instead of clicking a button.

- internal/policy/command.go: ClassifyCommand(cmd, declaredRisk) — rule-based
  read-only allowlist + destructive denylist, default-escalate to
  config_mutation for anything else. Classification can only ESCALATE the
  caller's declared risk, never de-escalate it (destructive always wins even
  if declared read_only). Compound commands (&&, ;, |, $()) never qualify for
  the read-only fast path. Full test corpus.
- internal/mcp/server.go: new `run` MCP tool — target (host:/lxc:), command,
  purpose, optional declared_risk. Read-only commands execute immediately;
  everything else queues an approval exactly like pct_create today, executed
  via httpapi's existing executeApprovedAction. Also fixes a real latent bug:
  pct_exec resolved an LXC's host attribute without the "host:" prefix, so it
  could never find the Proxmox host — new resolveExecTarget/resolveRunTarget
  helpers (mcp + httpapi) fix this for both the new `run` action and existing
  actions that route through the same execution path.
- internal/httpapi/phase3.go: "run" case in executeApprovedAction; fixes two
  bugs found while wiring this up — (1) DecideApproval hardcoded risk_class to
  'config_mutation' on every approve, silently corrupting the audit ledger for
  every other risk class; (2) denying/revoking an approval never updated the
  linked execution's status, so it stayed 'pending_approval' forever instead
  of reflecting the decision.
- cmd/nomos/assent.go: deterministic (not LLM-judged) chat-assent detection.
  Scoped to the immediately-preceding assistant turn's pending approvals only
  — an old "yes" can't retroactively approve something new. Destructive-risk
  actions are excluded from loose assent. Approves via the same HTTP decision
  endpoint the UI button calls, so both paths share one audit trail.
- web/.../InlineApproval.svelte: self-healing poll — a pending approval card
  now picks up being decided via ANY path (chat assent, Ops page, Matrix), not
  just its own button. Previously the banner stayed stuck showing
  Approve/Deny even after the action had already run elsewhere.
- nomos/SOUL.md: `run` is now the general capability ("no fixed menu, only a
  risk gate"); documents chat-assent behavior and the destructive exception.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:28:00 +02:00
9daf8220f2 plan: add observability layer and chat-assent approval to gated-execution plan
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Operator directives: (1) the UI must show what's executing, its risk
classification, live status, and what knowledge the session created — the
system's growth should be visible, not just trusted. (2) approval should be
granted by chat assent ("go ahead"), not a separate button; destructive
actions still require a typed confirmation phrase.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:12:09 +02:00
4a96f46e76 plan: general gated execution — unlimited actions, gated by risk classifier
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Evaluate the agent's action path against the OIKOS.md design. Finding: the
intended model (unlimited runbook-driven actions gated by a risk classifier)
already exists on paper and in scaffolding, but the live agent path regressed
to a hard-coded 5-action enum that bypasses the classifier. Plan a layered
realignment: (0) one general gated `run` primitive, (1) runbooks-as-data as the
reliable fast-path, (2) learning. Chosen v1 posture: approve-most (read-only
auto-runs, all state changes gate). Incremental, each step shippable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:03:23 +02:00
5f888e6386 fix: set C.UTF-8 locale in provisioning script
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Fresh Debian LXCs have no locale configured, spamming "apt-listchanges:
Can't set locale" / perl warnings across every install and breaking some
packages' post-install scripts. Pin LANG/LC_ALL=C.UTF-8 (and hoist
DEBIAN_FRONTEND) at the top of the in-container bootstrap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 01:21:23 +02:00
f248508919 feat: robust provisioning (DNS self-heal) + live execution feedback in chat
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Production session provisioned the container but the service never installed:
apt failed with "Temporary failure resolving deb.debian.org" — a static-IP LXC
whose assigned nameserver couldn't resolve. The operator also got zero feedback:
the approval banner just sat there with no running/complete/failed status.

Backend robustness (provisionScript):
- Wait for real DNS/connectivity inside the container before apt, and self-heal
  /etc/resolv.conf to a public resolver (1.1.1.1/8.8.8.8) if the assigned one
  is dead. `set -e` after the gate so apt/post_install failures surface.
- apt-get update/install with Acquire::Retries=3.

Frontend feedback (InlineApproval):
- After approve, poll GET /executions/{id} and show live phase: submitting →
  provisioning… → provisioned successfully / execution failed (with the error).
- add getExecution() to api.ts.

Agent guidance (SOUL.md):
- omit vmid (auto-assigned), prefer dhcp, docker-compose-plugin is not in Debian
  (use docker.io + get.docker.com), end post_install with a health check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 01:15:21 +02:00
a1f666f68a fix: build execution result JSON via json.Marshal (was stuck at 'approved')
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The final "UPDATE executions SET result=$::jsonb" built its payload with
fmt.Sprintf and only escaped newlines. apt/pct output contains quotes,
backslashes and control chars, so the payload was invalid JSON, the jsonb
cast failed, and the (unchecked) UPDATE was silently discarded — the
execution stayed 'approved' with a NULL result even though the LXC was fully
provisioned (verified live: vmid auto-assigned, container running, service
installed, post_install ran).

- executeApprovedAction: marshal result via json.Marshal; log UPDATE errors
- add jsonErr() helper; route all pct_create failure-path results through it
- mcp/server.go: add jsonOut() for restart/systemctl/pct_exec inline results
- regression test for JSON validity on quote/backslash/control-char output

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 00:58:43 +02:00
ac86302f52 fix: pct_create make vmid optional, fix dhcp+gw, longer boot settle
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Follow-ups found while verifying the approve→provision path end to end:
- vmid is now optional: the early required-field check rejected vmid:0
  before the cluster VMID guard could auto-assign a free id. Only hostname
  is required now; 0 (or a collision) resolves to `pvesh get /cluster/nextid`.
- net0: use ip=dhcp with no gateway when no static IP is given (Proxmox
  rejects gw alongside dhcp); only attach gw for a static CIDR.
- bump post-create settle to 10s so a DHCP lease is up before apt runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 00:50:54 +02:00
8ed2b88495 fix: pct_create false-success, VMID collision, and stuck approval banner
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Real production failure when the operator clicked Approve in chat: nothing
provisioned, banner never cleared, execution marked completed.

Three root causes:
- sshExec swallowed non-zero exits when the command produced output, so a
  `pct create` that printed "CT 132 already exists" and failed was reported
  as success and a bogus lxc entity was registered. Now any non-zero exit
  returns an error (with output) so the execution is correctly marked failed.
- The LLM reused VMID 132 (belongs to lxc:rclone; VMIDs are cluster-wide).
  pct_create now checks in-use VMIDs via `pvesh get /cluster/resources` and
  falls back to `pvesh get /cluster/nextid` when the requested id is taken.
- InlineApproval.svelte reset its state on every prop change (done was also
  compared against the wrong string), so the banner never cleared and each
  click re-POSTed /decision. Rewritten to track outcome per executionId,
  clear on success, and block resubmits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 00:44:51 +02:00
b37f85ae08 fix: make Nomos actually provision LXCs from chat (pct_create + web fetch)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Root cause of "asks permission but never acts": the approved pct_create
execution failed to parse because the LLM emitted `"privileged":0` /
`"nesting":1` (numbers) into strict `bool` fields, so the container was
never created. Compounded by a hardcoded template name (debian-13.0-1)
that no longer exists on the host, and no way for the agent to read the web.

- flexBool: accept 0/1, "true", bool for privileged/nesting (the exact prod failure)
- pct_create template pre-flight: list host cache, validate/auto-pick newest debian
- pct_create services[] + post_install: one approval provisions a working service
- new http_get MCP tool (sanitized, size-capped, SSRF-guarded) — agent can read repos/sites
- request_execution description: target=host, full JSON schema + example
- SOUL.md: agent CAN fetch the web; prefer one-step provisioning
- default model deepseek-v4-flash -> v4-pro; maxIterations 15 -> 25
- unit tests for flexBool, template resolve, pkg sanitize, HTML sanitize + SSRF block

Verified live on host:strong with a throwaway VMID 999: template auto-resolved,
container created + booted, services installed, post_install ran, then destroyed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 00:28:56 +02:00
a567930466 fix: make execution names unique, move approval bar above input
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Execution entity name now includes UUID suffix: 'pct_create on host:strong (abc12345)'
  so the (type,name) UNIQUE constraint doesn't block subsequent executions for the
  same target+action. Dedup now uses JOIN + LIKE prefix match to find only
  pending_approval executions.

- Move persistent approval bar from top of messages area to just above the
  chat input box (bottom-fixed position, above the textarea form).
2026-07-09 23:42:49 +02:00
9376dc7d89 fix: dedup request_execution, persistent approval bar, JSON payload
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Add dedup in request_execution: check entities(type,name) uniqueness before
  creating duplicate executions. Returns 'already queued' message to the LLM,
  preventing tool-calling loops.

- Fix createApproval JSON payload: use json.Marshal instead of fmt.Sprintf
  to escape params (could contain unescaped double quotes from JSON config).

- Add ON CONFLICT DO NOTHING to entity/execution inserts for dedup race safety.

- Persistent approval bar at top of Chat.svelte: aggregates pendingApprovals
  from all messages, fixed position (won't scroll away). Approve/deny/approve-all.

- Update SOUL.md: agent must STOP after queuing a gated action.

- Fix ToolCallGroup  reactivity: wasActive = (active).
2026-07-09 23:19:55 +02:00
d9683cfe29 fix: structured approvals + ToolCallGroup reactivity
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Replace text-based regex parsing in InlineApproval with structured
  pendingApprovals extracted from request_execution tool results. The tool
  result text is deterministic (not LLM-generated), making UUID extraction
  reliable regardless of how the LLM rephrases the response.

- Fix ToolCallGroup  reactivity: wasActive = active captured initial
  value. Now uses (active) so  re-runs on prop changes.

- Extract approvals in both live streaming (done event) and history loading
  for consistent behavior on resumed sessions.
2026-07-09 11:35:30 +02:00
ea62d744ed feat: pct_create action, ToolCallGroup collapse+animation, inline chat approval
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Add pct_create to request_execution (MCP) and executeApprovedAction (httpapi)
  Parses JSON config: vmid, hostname, cores, memory, disk_gb, ip, gw, storage,
  template, privileged, nesting, mounts, nameserver, searchdomain. Creates
  entity (state=provisioning), hosts relationship, entity_status on success.
  Fixes action string parsing to use Index instead of SplitN (colons in JSON).

- Rewrite ToolCallGroup.svelte: bits-ui Collapsible replaces native <details>.
  Collapsed by default. Animated header shows live tool count + running tool
  name while streaming. Auto-expands during streaming, auto-collapses on done.

- Add InlineApproval component: parses 'execution UUID queued' from agent
  response, renders Approve/Deny buttons inline in chat, calls decideApproval.

- Document pct_create in nomos/SOUL.md with params, risk class, and approval flow.

- Add session-review skill at .agents/skills/session-review/SKILL.md.

- Add plan: 2026-07-09-session-execution-and-ux-fixes.md.
2026-07-09 11:15:28 +02:00
0d29b1db81 fix: move completed signal-triggers plan to done/, add missing liveness-drift to index, add plan-consistency lint checks 2026-07-09 10:47:39 +02:00
e92a6ff7a5 fix: trash-2 icon name (lucide uses trash-2, not trash2)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-09 10:21:22 +02:00
49c37fe8b1 fix: chat session reliability, cost, and hygiene (empty-response guard, tool truncation, delete, titles)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Empty/refusal responses retried once, then surfaced as errors instead of silent blanks
- Chinese refusal boilerplate detected via denylist + non-ASCII heuristic
- Bulk-tool preference added to SOUL.md (list_lxcs over per-entity get_lxc_state)
- Tool results truncated to 4KB on persist; get_state_snapshot filters null-state entities
- Session delete (DELETE /sessions/{id} + confirm-on-second-click UI)
- Session titles auto-generated from assistant answer instead of raw user message
2026-07-09 10:18:06 +02:00
614c38ea7c docs: plan chat-sessions fixes from real production usage data
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Inspected the live agent_sessions/agent_messages tables on mac-mini and
found silent empty responses, a canned non-English refusal after 22 tool
calls, 70-call fan-out for simple fleet questions, 100KB+ persisted
messages, and no session delete/title hygiene.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 10:09:34 +02:00
22412d2fa3 feat: group tool calls per turn + session entity graph in chat rail
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Two chat UX changes (share Chat.svelte, committed together).

Tool-call grouping (ToolCallGroup.svelte):
- Replaced the one-<details>-per-tool-call list with a single
  collapsible group per assistant turn, headed by tool count + a
  name preview and a status icon (spinning wrench in progress, check
  done, X on error).
- The group auto-collapses the instant its turn finishes streaming, so
  a completed round shows as one compact pill; historical/loaded turns
  start collapsed. The auto-collapse fires once at the streaming→done
  transition, leaving manual toggles alone afterward.

Session graph rail (SessionGraph.svelte) — replaces the old
ContextRail (fleet health / pending approvals / live events), which is
deleted:
- A force-directed graph that starts empty (animated constellation
  empty state) and grows as the conversation references entities.
  Slugs are extracted from message text and tool *arguments* only —
  never bulk result rows, so a single get_health_summary doesn't dump
  all 168 entities — then validated against the backend via fetchGraph
  (cached) with check/execution probe entities excluded. Nodes are
  colored by health; edges appear once both endpoints are present.
- Clicking a node highlights it and its neighbors and opens an inline
  detail panel below: slug/type/state, health + freshness, top
  attributes, in-graph relations (clickable to hop), and a Full detail
  button opening the entity sheet.
- The rail is resizable via a drag handle (260–620px, persisted to
  localStorage). The header's global fleet-health dots are unchanged;
  only the right-rail content was replaced.

Risk: reversible_low (UI-only). The slug extractor is scoped to
focused mentions by design; edges may be slightly incomplete since
only root-fetched entities contribute edges, which is acceptable for a
session overview.

Verification: verified in the browser preview — loading a real session
built a 4-node graph (hubris/caddy/netbird-vps/strong) with the
hubris→caddy relationship edge; clicking hubris showed
"proxmox-host · active · healthy · checked 40s ago" with attributes and
relations; dragging the handle resized 320→440px and persisted; a
34-tool historical turn renders as one collapsed pill that expands on
click. tsc clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 09:48:21 +02:00
5686b9de40 fix: white favicon, sidebar active-state bug, app-wide pointer cursor
Three UI issues reported after the neutral-gray redesign:

- favicon.svg was still filled #58a6ff (the pre-redesign accent blue);
  changed to white to match the sidebar logo mark.
- Sidebar nav items all showed a filled background even when inactive.
  Root cause: sidebar-menu-button.svelte (and -sub-button) rendered
  `data-active="false"` as a literal attribute, but Tailwind's bare
  `data-active:` variant matches attribute *presence*, not value — so
  data-active:bg-sidebar-accent applied to every item regardless of
  state. Fixed by emitting the attribute only when active
  (`isActive || undefined`), a latent bug in the vendored shadcn
  component that read as intentional until flagged.
- Tailwind's preflight resets <button> to cursor: default, so no button
  in the app showed a pointer. Added one base rule restoring
  cursor: pointer for buttons, [role=button], links, summary, and
  select (respecting :disabled / aria-disabled) rather than annotating
  each call site — covers new interactive elements automatically.

Risk: reversible_low (UI-only).

Verification: verified in the browser preview that inactive sidebar
items are transparent (only the current page shows a background),
nav buttons report cursor: pointer via computed styles, and the
favicon renders white in the tab.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 09:47:47 +02:00
aa6017e0ca fix: layout overflow regression + adopt true neutral gray theme
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: two issues surfaced after the dashboard-01 shell change
(cbfd09c). (1) Sidebar.Inset previously had an explicit h-svh that
hard-capped the app's height at the viewport; adding variant="inset"
put a margin on that same fixed-height box, pushing it taller than the
viewport with nothing left in the chain to cap it (Sidebar.Provider's
own wrapper only sets min-h-svh — a floor, not a ceiling). Result: the
whole page scrolled as one long document instead of each page's own
content scrolling internally with the header pinned — confirmed via
computed styles, e.g. Entities.svelte's table wrapper measured
scrollHeight 6531px against a 900px viewport, all of it spilling past
body instead of scrolling in its own rounded-border container.
(2) The color palette was GitHub-dark-inspired (blue-tinted grays:
#0d1117 bg, #58a6ff primary/accent) rather than the neutral grays the
shadcn-svelte dashboard-01 reference actually uses.

Change:
- App.svelte: moved the height cap up to Sidebar.Provider itself
  (class="h-svh") instead of Sidebar.Inset, since the cap needs to sit
  above wherever the inset variant's margin gets applied, not on the
  same box as the margin.
- app.css: replaced the core tokens (background/foreground/card/
  popover/primary/secondary/muted/accent/border/input/ring/sidebar-*)
  with shadcn's canonical dark-theme OKLCH values (0-chroma neutral
  grays), pulled directly from huntabyte/shadcn-svelte's own
  docs/src/app.css rather than approximated. --success/--warning
  deliberately kept as real, distinguishable colors — they signal
  actual health state, and desaturating them to match the neutral
  chrome would reintroduce the "can't tell what's actually happening"
  problem this whole project started from (see 279549c). --accent-blue
  now aliases --sidebar-primary (still a real blue) instead of
  --primary, so the couple of spots wanting an interactive "pop" still
  have one while buttons/links/focus rings ride the neutral --primary.

Risk: reversible_low (UI-only).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). go build/vet clean (backend
untouched, sanity check only). Manually verified in the browser
preview at 1400px: document.body.scrollHeight now exactly matches
window.innerHeight on both Overview and the 193-row Entities table
(previously 6531px vs 900px); scrolled the Entities table wrapper to
row ~60 and confirmed the header/filter bar/column headers stay
pinned while only the table body scrolls; confirmed neutral gray
rendering across Overview's stat cards, the event-rate chart, and
Chat's tool-call list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 01:12:47 +02:00
cbfd09c5df feat: redesign toward shadcn-svelte dashboard-01 (inset sidebar, gradient stat cards)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: requested visual alignment with the shadcn-svelte dashboard-01
reference block (shadcn-svelte.com/blocks#dashboard-01) — the app's
sidebar/header shell and Overview stat cards looked plain by comparison.

Change: pulled the actual reference source (app-sidebar.svelte,
nav-main.svelte, site-header.svelte, section-cards.svelte from
huntabyte/shadcn-svelte) rather than approximating from screenshots.

- App.svelte: Sidebar.Root now uses variant="inset" (the floating,
  rounded, shadowed content panel — already fully built into the
  existing Sidebar.Inset component via peer-data selectors, just never
  enabled). Brand mark is now a proper Sidebar.MenuButton matching the
  reference's padding/hover treatment; "New chat" uses the reference's
  primary-colored button styling. Header matches the reference exactly:
  h-(--header-height) (48px, was 44px), vertical separator after the
  sidebar trigger, right-aligned actions group.
- Overview.svelte: stat cards rebuilt to match section-cards.svelte —
  gradient background, Card.Action badge, Card.Footer with a bold line
  + muted context line, tabular-nums, responsive @container grid
  (1/2/4 columns). Deliberately did NOT copy the reference's fake
  trend-percentage badges (Oikos doesn't track historical trends, and
  this project's whole thrust has been eliminating dishonest UI state —
  see 279549c). Badges instead reflect real current-state signals
  (healthy/degraded/down, clear/needs-review) computed from the actual
  dashboard summary.
- EntityDetailContent.svelte + Entities/Signals/Ops/Events/Agent/
  Audit/Knowledge pages: normalized root padding to p-4 md:p-6 (was a
  flat p-6) to match the reference's responsive py-4 md:py-6 convention.

Risk: reversible_low (UI-only, no data or behavior changes).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings, same as prior commits). go build/vet
clean (backend untouched, sanity check only). Manually verified in the
browser preview at 1400px: inset sidebar's margin/rounded-corner/shadow
classes confirmed applied via computed styles; Overview cards render
with real live numbers from the now-fixed dashboard summary endpoint;
Signals/Ops pages confirmed visually consistent with the new spacing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:54:51 +02:00
851b5dce67 feat: master-detail entity sheet, freshness in Entities table, live logo
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: the web UI felt dead and hard to navigate — the Entities table
had no health/freshness signal (just a meaningless row-mutation
timestamp), no way to see what was actually monitoring an entity,
sessions couldn't be reopened, and every drill-down was a full page
navigation that lost the list.

Change:
- Entities table: Updated column replaced with a health dot + relative
  "checked Xm ago", sourced from the backend's new health/last_check_at
  fields.
- New EntityDetailContent.svelte extracted from EntityDetail.svelte and
  shared between the full #/entity/:slug page and a new EntitySheet.svelte
  opened from the Entities table (master-detail, row click opens a panel
  instead of navigating away). Adds a Monitoring card listing the
  entity's check_defs (kind, interval, enabled/disabled with
  click-to-toggle via the existing PatchCheck endpoint) and renders
  attributes as key/value pairs instead of raw JSON.
- Sessions: fixed a bug where clicking a session loaded it into the
  chat store but never navigated to the chat page, so nothing appeared
  to happen. Added a SessionRail inside Chat so switching sessions
  never leaves the chat surface.
- Fixed the local dev proxy (vite.config.ts): production Caddy strips
  the /agent prefix before forwarding to nomos; the dev proxy didn't,
  so every session/chat fetch 404'd locally while working in prod.
- Found and fixed a real latent bug while testing the session fix:
  chat.ts's loadSessionMessages passed the persisted tool_calls array
  straight through, but nomos stores the tool_use and tool_result as
  two entries sharing one id. Chat.svelte's keyed {#each tool (tool.id)}
  throws on the duplicate key, which silently blanked the entire
  message list — invisible until sessions were actually clickable.
  Fixed by merging tool_calls by id before rendering, matching the
  shape the live-streaming path already produces.
- UI polish: sidebar logo is now just the omicron mark in white (was
  icon+text in the accent color); removed the sheet overlay's
  backdrop-blur (distracting per feedback); the Attributes/Relations/
  Signals grids used viewport-based lg:/3xl: breakpoints, which forced
  multi-column layouts based on browser width regardless of the sheet's
  actual rendered width — switched to Tailwind v4 container queries
  (@lg:/@2xl:/@3xl:) so layout responds to the real available width in
  both the full page and the narrower sheet.

Risk: reversible_low (UI-only; no destructive operations; the tool_calls
merge and dev-proxy fix are corrections to broken paths, not behavior
changes to working ones).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). Manually verified in the browser
preview against the live dev API: Entities table health column renders
correctly; clicking a row opens the EntitySheet with a populated
Monitoring card (16 checks for host:hubris, verified via psql that
check_defs.target_id links them correctly); clicking a session now
loads its full transcript inline (was blank before the tool_calls fix);
sheet has no blur and lays out single/multi-column correctly at the
sheet's actual width.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:26:37 +02:00
279549c8c9 fix: scheduler wrote health/metrics/events to probe entities, not targets
Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.

Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
  cd.TargetID (falling back to the check's own id if unset) and write
  status/metrics/events there. Signals stay keyed by the check entity,
  unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
  observation is older than 3x its fastest enabled check's interval
  (floor 5m) is marked 'stale' and emits health.stale, so a stalled
  scheduler or disabled check_def can no longer look like current data
  forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
  rows so dashboard/fleet-health rollups stop double-counting probes as
  monitored entities. Historical metric_samples on check entities are
  left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
  'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
  exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
  the conversation instead of dropping them (previously only final text
  was replayed, forcing the agent to re-derive fleet state every turn),
  and inject a compact live fleet-health snapshot into the system prompt
  each turn so it starts oriented instead of spending an iteration on
  discovery.

Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.

Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:26:04 +02:00
a39e67b6e9 adr: convert all diagrams to Mermaid (sequenceDiagram, stateDiagram-v2, flowchart, erDiagram, graph)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
0013-signal-triggers.md:
- Thermals query: sequenceDiagram (Nomos→API→Scheduler→Hubris→TimescaleDB)
- Script deployment: sequenceDiagram
- Signal lifecycle: stateDiagram-v2
- DB data flow: flowchart

0014-entity-model.md:
- Entity type hierarchy: graph (56 types, 3 layers, 7 domains)
- Machine onboarding: sequenceDiagram
- OODA loop (5 phases): flowchart with color-coded subgraphs
- Infrastructure topology: graph
- Network relationships: graph
- Service dependencies: graph
- Cognition OODA edges: graph
- Governance: graph
- Infrastructure lifecycle: stateDiagram-v2
- Signal lifecycle: stateDiagram-v2
- Execution lifecycle: stateDiagram-v2
- Approval lifecycle: stateDiagram-v2
- DB physical schema: erDiagram
- Thermals query trace: sequenceDiagram
2026-07-08 22:38:19 +02:00
551497e0b3 adr: move to docs/adr/, renumber 0013 + 0014, update README index
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 22:18:58 +02:00
4a5e68bafe adr: full entity model — types, relationships, state machines, OODA loop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Covers:
- 56 entity types with full hierarchy (abstract/concrete, domain, layer)
- 34 relationship types with cardinality and OODA phase mapping
- 88 concrete entity instances with key attributes
- Lifecycle state machines (infrastructure, signal, execution, approval,
  pattern, skill) with which preconditions are code-real vs schema-only
- Sequence diagrams: machine onboarding, OODA loop, thermals query
- What's fully implemented vs schema-defined-but-not-wired
- Database physical schema with FK relationships
- Policy: risk classes, approval rules, per-entity overrides, autonomy
- Blast radius via recursive CTE over depends-on/hosts/routes-to edges
2026-07-08 22:15:11 +02:00
ef00e5b8e4 fix: disk_usage_check.sh handles inode '-' (vfat/efi), checkdefaults sets target_id
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- disk_usage_check.sh: sed 's/-/0/' for filesystems without inodes
- checkdefaults.Ensure: includes target_id in check_defs INSERT so
  signals get proper target slug instead of null
2026-07-08 21:58:44 +02:00
a512d40669 onboarding: auto-create default checks when entity is created
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Three insertion points:
- CreateEntity (POST /api/v1/entities)
- EnrollClient (POST /api/v1/clients/enroll)
- seed.go (seed ingest at deploy time)

Shared logic in internal/checkdefaults — resolves host IP from
lan_ip > mesh.netbird.ip > mesh_ip, SSH user/port from attributes.

Default checks per entity type:
- proxmox-host/standalone-server: ping + cpu + memory + load + disk + updates
- workstation: ping + cpu + memory + load
- lxc: cpu + memory + load + disk
- vm: ping
- service: process_check.sh

All idempotent (ON CONFLICT DO NOTHING). New machines now get
monitoring automatically — no manual curl calls needed.
2026-07-08 21:53:50 +02:00
b8bb29464b checks: deploy scripts + define checks for strong and netbird-vps
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Coverage:
- host:hubris (192.168.8.77)   ✓ cpu, memory, load, disk, ping, cert-expiry
- host:strong (192.168.178.181) ✓ cpu, memory, load, disk, ping
- host:netbird-vps (82.165.190.79) ✓ cpu, memory, load, disk, ping
- ws:mac-mini                  ~ local disk/ping only (no SSH key on macOS host)
- ws:republic-laptop           ✗ offline / not reachable

Move architecture doc to adr/signal-triggers.md
2026-07-08 21:45:09 +02:00
81acadec1d docs: signal trigger architecture sequence diagram + full explanation
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Add docs/signal-triggers.md covering:
- End-to-end sequence diagram (Nomos → API → Scheduler → target host)
- Two paths: autonomous collection (scheduler) + query (MCP)
- All 6 check kinds and 17 available scripts
- Script deployment flow via sync timer
- Signal lifecycle, threshold evaluation, data flow through DB tables
- Prerequisites for SSH checks in Docker
2026-07-08 21:37:31 +02:00
291b45565b fix: metric samples timestamp + ssh-script port/user handling
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- InsertMetricSample now includes ts=now() (TimescaleDB hypertable requires it)
- ssh-script: pass host and port separately (ssh uses -p flag, not host:port)
- ssh-script: use OIKOS_SSH_USER from config/env, default root
- Add -o LogLevel=ERROR to suppress SSH warnings polluting JSON output
- Use Output() (stdout only) instead of CombinedOutput()
- Set OIKOS_SSH_USER=root in scheduler docker-compose service
2026-07-08 21:36:05 +02:00
d45f2326b6 fix: CreateCheck uses type "check" not "check_def"
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
FK violation — entity_types table has name "check" but
phase3.go:320 was inserting type "check_def" causing:
entities_type_fkey (SQLSTATE 23503)
2026-07-08 21:21:44 +02:00
4bf811a383 docker: alpine base with openssh-client, mount SSH key + NET_RAW for scheduler
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Switch Dockerfile from distroless/static to alpine:3.21
- Install openssh-client-default in runtime image
- Mount SSH key in scheduler service (docker-compose)
- Add NET_RAW capability for ping checks
- Wire OIKOS_SSH_KEY_PATH and OIKOS_SSH_USER env vars in scheduler
- sshExec uses configured key path with StrictHostKeyChecking=no
2026-07-08 21:15:02 +02:00
35feada286 scheduler: add ping + ssh-script check kinds, metrics refactor, 17 host check scripts
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Refactor executeCheck to return checkResult struct with metrics map
- Add ping check kind (ICMP reachability via system ping, macOS+Linux)
- Add ssh-script check kind (remote host exec via SSH, allowlisted scripts)
- Add threshold evaluation (warn/crit per metric from check config JSONB)
- Add inode tracking to disk check
- All 4 existing checks now return structured metrics
- 17 check scripts: cpu, memory, load, swap, disk_usage, disk_smart,
  updates, zfs, process, uptime, oom, journal, time, fd, docker_health,
  caddy_error_rate, backup_freshness
- Auto-deploy via tools/setup-checks.sh -> checks/install.sh on git pull
- Add ping to OpenAPI CheckKind enum and generated Go types
2026-07-08 21:05:53 +02:00
cca2ae4621 fix: properly convert icns to PNG for favicon data URI
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 18:06:35 +02:00
d13f6991b1 fix: use inline base64 favicon to work in both dev and production
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 18:02:11 +02:00
2de3602ebc feat: replace inline favicon with extracted app icon from DMG
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 17:34:58 +02:00
2908b0a377 feat(ui): M4 — agent activity, knowledge search, audit, correlation grouping
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Add three new pages completing the control-room web UI:
- Agent activity: polls /agent-activity every 5s, filterable by type/agent
- Knowledge search: FTS over /knowledge/search with snippet + entity links
- Audit trail: browseable audit log with actor/action/entity filters

Enhanced live events page with correlation-id clustering (Groups toggle).
Added fetchAgentActivity/searchKnowledge/fetchAudit to the API client.
11 nav items now cover all planned control-room views.
2026-07-08 17:02:09 +02:00
cff05c0768 fix(nomos): auto-reconnect stale MCP session; enlarge SSE scan buffer
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
After an api (MCP server) restart, nomos held a dead session id and every
tool call failed with "unexpected end of JSON input" until nomos was manually
restarted — which happens on every deploy. The MCP client now detects a
rejected session (4xx or empty body) and transparently re-initializes and
retries once. Also raise the SSE scanner buffer to 4MB so large tool results
don't exceed the 64KB default token limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:55:10 +02:00
5d02126e16 fix(ui): serve embedded SPA via ServeContent to avoid index.html redirect loop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
http.FileServer canonicalizes /index.html -> "./", which for /ui/ produced a
301 redirect loop and made the control room unreachable. Serve embedded files
directly with http.ServeContent instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:28:54 +02:00
e8e230b4a5 nomos+web: streaming, provider routing, event gap-fill, embedded UI; fix approval FK & session context
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Agent (cmd/nomos):
- Stream LLM tokens via NewStreaming; emit text_delta then final text.
- OpenRouter provider routing: data_collection=deny (ZDR) + require_parameters;
  NOMOS_PROVIDER_SORT opt-in; Exacto via model suffix.
- Multi-turn: reload session history into context; UI passes session id.
- Fix agent_activity logging (agent_id/session_id) and mcpClient data race.

Events (live control-room feed):
- approval.created (mcp), approval.decided (api), execution.completed/failed
  (approved-action path), signal.raised/resolved + health.changed (scheduler,
  transition-gated).

Fixes:
- createApproval FK violation (reuse execution entity) — the agent's only
  write path; log the previously-swallowed errors.

Web UI:
- Embed web/dist via //go:embed (single binary); Dockerfile builds SPA into
  the Go stage; committed .gitkeep placeholder keeps backend-only builds green.
- Caddy: Authentik-gated /agent/* -> nomos so the UI reaches the agent
  same-origin in production.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:22:27 +02:00
2b3aa248b1 N0: rename Hermes → Nomos (standalone commit)
Problem: "Hermes" collides with Nous Researchs unrelated product;
  unclear identity for the resident agent.

  Change: Rename the live service identity across 39 files:
  - cmd/hermes/ → cmd/nomos/ (binary, env vars NOMOS_*)
  - internal/config/ server.go (NomosAgentSlug, nomosAgentID)
  - compose/hermes/ → compose/nomos/ (Dockerfile, service name)
  - hermes/ → nomos/ (SOUL.md, config.yaml, skills/)
  - .agents/HERMES.md → NOMOS.md (persona)
  - tools/setup-hermes-soul.sh → setup-nomos-soul.sh
  - seeds/inventory.yaml (agent:hermes → agent:nomos)
  - migrations/014_rename_agent_hermes_to_nomos.up.sql
  - Caddy vhost hermes.hubris.network → nomos.hubris.network
  - All referencing docs, scripts, ADR notes

  History preserved: archive/, plans/done/, ADRs not rewritten.
  Matrix @hermes notifier account and Legacy bin/hermes on LXC 129
  intentionally untouched (out of scope).

  Risk: N0 is identity-only rename; zero behavioral changes.
  Verification: go build ./... passes; docker compose --profile full
  resolves nomos service; grep -ri hermes (excluding archive/plans)
  returns only intentional refs (LLM model name, Matrix user).
2026-07-08 14:14:56 +02:00
9d9cbb63c4 plans: rename resident agent Hermes -> Nomos, add implementable rename phase N0
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Nomos (from oikonomos, steward of the oikos) avoids the name collision
with Nous Research's Hermes Agent. N0 enumerates the full rename scope:
cmd/, hermes/ dir, env vars, config fields, compose service, Caddy vhost,
identity-preserving DB slug migration + seed update, persona docs.
History, the Matrix bot user, and legacy bin/hermes stay untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:47:24 +02:00
ae3b150dcc plans: hermes agent uses OpenRouter (default deepseek-v4-flash), document off-the-shelf rejection
Swap anthropic-sdk-go for openai-go against the OpenRouter API; default
model deepseek/deepseek-v4-flash with Exacto routing and ZDR provider
pinning. Record why Nous Hermes Agent (and hosted MCP connectors) were
rejected for the resident role.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:36:55 +02:00
750f0e088d plans: add Hermes resident agent plan, make agent chat the control-room home view
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Hermes becomes an LLM-backed agent loop (hand-rolled tool loop over the
existing mcpClient, not the public MCP connector), with Postgres-backed
sessions, SSE chat streaming, and Authentik-gated /agent routing. The
control-room plan is amended to make the chat the main entry point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:30:42 +02:00
84068cc40b plans: add oikos gaps review and control-room webui plans
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Documents a full-project review (confirmed bugs, security gaps, user- and
agent-perspective gaps) and a realtime control-room web UI plan, per prior
codebase exploration on this branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 13:16:24 +02:00
5d15265f65 fix: queryRows returns [] not null for empty result sets
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:50:13 +02:00
3ea43adcfd fix: repair 5 MCP analysis tools with SQL errors + seed entity_status rows
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- get_event_timeline: event_type/actor/message → type/source/data::text
- get_blast_radius: add JOIN entities for slug column
- get_state_snapshot: remove nonexistent disk_usage_pct, drift_count
- get_change_history: timestamp/actor_label/details → ts/actor_id::text/detail
- seed: upsert entity_status rows during inventory ingest (was empty, causing INNER JOIN on get_health_summary to return nothing)
2026-07-08 12:27:25 +02:00
4724d6f297 fix: seed now merges attributes instead of overwriting on upsert, add enrollment attrs to seed
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:02:46 +02:00
f07668c1c3 caddy: update Caddyfile.oikos snippet to match deployed config with enrollment bypass
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:00:20 +02:00
7e0566d2cd fix: widen 'about' relationship to entity→entity many-to-many for investigation edges
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:54:10 +02:00
2e8ef75436 onboard: mac-mini as workstation — new age keypair, update inventory seeds
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:50:47 +02:00
ca16b75b90 fix: rename Homelab-Docs → oikos across all active files; add public enrollment route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:46:42 +02:00
e148c7a981 move 5 completed plans to plans/done/
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Consolidate Oikos on mac-mini (2026-07-06)
- Client lifecycle in Go (2026-07-07)
- Comprehensive audit & next steps (2026-07-07)
- DB as source of truth (2026-07-07)
- MCP tool completion (2026-07-07)

Paths fixed in index.md to reflect planes/done/ locations.
Active plans remaining: Prometheus LXC (Planned), implementation audit (Active).
2026-07-08 11:30:54 +02:00
cefeba72b0 delete apps/105 Gitea webhooks (ids 10, 11, 14)
Removed via Gitea API using Infisical-stored token:
- Hook 10: 192.168.8.205:9811 (homelab-mcp-deploy)
- Hook 11: 192.168.8.205:9821 (secrets-issuance-deploy)
- Hook 14: 192.168.8.205:9831 (oikos-console-deploy) — found during cleanup

All three pointed to apps/105 (192.168.8.205). No remaining webhooks
fire to the old Python stack. Zero hooks remain on dtoro/oikos.

Cutover checklist updated. Only item remaining: archive apps/105 LXC.
2026-07-08 11:29:42 +02:00
e131215a25 add Gitea webhook cleanup script for apps/105
scripts/cleanup-apps105-webhooks.sh: lists webhooks on dtoro/oikos,
filters for apps/105 deployment endpoints, prompts for confirmation,
deletes matching hooks via Gitea API.

Usage: GITEA_TOKEN='...' ./scripts/cleanup-apps105-webhooks.sh

Also updated cutover-checklist.md with the script reference and LXC
archive command.
2026-07-08 11:27:23 +02:00
7660e5681c complete consolidation plan — scripts, watchdog, rollback runbook
Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items:

1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy
   proxy). Only pages when BOTH paths fail. Partial failure logged but not
   paged (distinguishes stack problem from mesh/Caddy issue).

2. deploy.sh — pre-deploy pg_dump before each deploy saves to
   /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a
   guaranteed recovery point.

3. docs/operations/rollback.md — runbook documenting automated rollback,
   manual recovery, decision tree, backup schedule, and rehearsal log.

Two operational items remain (require operator on Proxmox/Gitea):
- Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs
- Archive apps/105 LXC (pct stop 105 + archive)

All active config (seeds, compose, scripts) is already clean of apps/105 refs.
Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).
2026-07-08 11:25:43 +02:00
28ab9b8088 final verification — client lifecycle plan: 12/12 items confirmed
Added 12-point verification table with file:line evidence for every item.
Noted two minor deviations: inline SQL (not sqlc) and synthetic Infisical
IDs (real Infisical pending consolidation plan). Precondition checks
enumerated with hard/soft classification.
2026-07-08 11:20:10 +02:00
fcd9f23ee1 transition precondition enforcement + thin-client context poller
Plan #3 at 100%. Last three items resolved:

1. Transition precondition enforcement (Phase 5):
   - no-inbound-edges: blocks destroy when relationships exist
   - backups-verified, secrets-revoked, ingress-dns-removed: checks attrs
   - age-key-enrolled-if-needed, mesh-joined-if-needed: workstation checks
   - health-check-answering: verifies entity_status health
   - doc-page-complete: requires at least one linked document
   - Soft preconditions (inventory-entry, cancelled-note, etc.): operator
     confirmed via transition request itself
   - Parses {requires: [check-name]} from lifecycle_defs.transitions JSONB

2. bootstrap.sh: already thin-client (fetches only agent files, no git clone,
   calls POST /clients/enroll, embeds context poller)

3. tools/context-poller.sh: standalone version — polls GET /clients/{slug}/context,
   applies file/tool/sops deltas, re-runs changed setup scripts
2026-07-08 11:16:04 +02:00
efa66c7321 close client lifecycle plan — API was already fully implemented
Plan #3 at 95%. Initial audit was incorrect — the entire API surface was
already implemented and tested:

- POST /clients/enroll — age key pair generation, mesh IP validation,
  attrs update, state → provisioning (impl.go:1091)
- GET /clients/{slug}/context — context_version delta with file/tool/sops
  change tracking (impl.go:1195)
- GET /clients/{slug}/secrets — scoped secret key listing (impl.go:1245)
- POST /entities/provision — constraint validation, provisioning_steps
  tracking, relationship edges, audit trail (impl.go:1271)
- GET /entities/{slug}/provision/status — step-by-step progress (impl.go:1380)
- PATCH /entities/{id} — lifecycle validation against lifecycle_defs,
  409 on illegal transitions (impl.go:933)
- client_lifecycle_test.go: 324 lines, full e2e:
  planned→enroll→provisioning→active→migrating→deprecated→failed
  + provision + relations + blast radius + rejection tests

Remaining (follow-up): bootstrap.sh + context-poller.sh thin-client scripts.
2026-07-08 11:12:45 +02:00
43aaf2a318 complete comprehensive audit — all cleanup items resolved
Plan #4 done. Audit inventory verified against codebase:

- 9 superseded oikos/*.py files deleted (only gen-topology.py remains)
- bin/homelab deleted, bin/oikos deleted, oikos/cards/ deleted
- .hermes/plans/ already archived to archive/hermes-plans/ (all 7 files)
- TRMNL plan already in Done table
- seanime + romm documented in seeds/knowledge.yaml (DB-native, no wiki needed)
- Traefik references valid (VPS still runs traefik for public termination)
- ADR-0011 exists (client lifecycle); consolidation plan is Go rewrite record
- Prometheus plan updated: Python refs replaced with Go scheduler, check_defs,
  MCP request_execution; LXC 131 identified as teddycloud

Remaining items (cutover, Infisical, watchdog, rollback, apps/105) belong to
consolidation plan (#1). 4 of 6 plans now Done.
2026-07-08 11:09:37 +02:00
a3ebd12e90 complete DB as source of truth — FTS knowledge surface
Plan #5 done. Wiki already archived to archive/knowledge/. seeds/knowledge.yaml
has 24 docs + 6 investigations + 3 runbooks.

- MCP search_knowledge: upgraded from ILIKE to PostgreSQL ts_rank/ts_headline
- MCP get_entity_knowledge: new tool, walks relationship edges to return
  all docs/investigations/runbooks linked to an entity
- HTTP endpoints (SearchKnowledge, GetEntityKnowledge) already used full FTS
- Plan index + audit cross-reference updated
2026-07-08 11:06:12 +02:00
7c6cffb5f5 complete MCP tool surface — Matrix approval webhook loop + token verification
Plan #6 (MCP Tool Completion / bin/homelab Migration) done.

- Approval records created for gated request_execution actions
- Notifier sends Matrix messages with HMAC approval tokens
- Stores matrix_event_id, polls /relations/{id}/m.annotation for /
- Reaction detection triggers DecideApproval API call
- Token verification added to DecideApproval endpoint
- Migration 013: matrix_event_id + alert_sent_at on approvals
- AGENTS.md: 21-tool surface documented, stale homelab CLI refs removed
- Plan index updated, audit cross-reference refreshed
2026-07-08 11:02:06 +02:00
5b22f2367b test: e2e client lifecycle + ADRs with sequence diagrams
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- client_lifecycle_test.go: full end-to-end integration test
  planned → provisioning (enroll) → active → migrating → active →
  deprecated → failed. Validates age keypair generation, attrs,
  context/secrets endpoints, invalid transition blocking, compute
  entity provisioning with relationship edges and status tracking.
  Also tests enrollment rejection for invalid states and duplicate
  slug rejection for provisioning.

- adr/0011-client-lifecycle-flows.md: workstation self-enrollment,
  compute entity provisioning, deprecation/destruction flows with
  Mermaid sequence diagrams. Full lifecycle state diagram. Transition
  check enforcement documentation.

- adr/0012-hermes-oikos-interactions.md: Hermes ↔ Oikos interaction
  flow through OODA loop phases. Thin client bootstrap. Internal
  component interactions (scheduler, actuator, notifier). Complete
  30-tool ownership matrix.

- Fix: migration 012 FK reference (executions.id → executions.entity_id)
- Fix: provision handler null attributes JSONB
- Fix: provisioning steps use entity_id for execution FK

All 3 integration tests pass, go vet clean.
2026-07-08 00:56:16 +02:00
84ecb6b895 feat: remaining phases — actuator provisioning, transition checks, cleanup
Phase 2: Actuator provisioning
- ProvisionLXC: pct create, start, package install, mounts, health check
- ProvisionVM: qm create, status check via SSH
- sshExecSimple helper for lightweight SSH command execution
- resolveHost helper for entity attribute lookups

Phase 5: Transition check enforcement
- TransitionChecks map with 8 named checks:
  age-key-enrolled, mesh-joined, health-check-answering,
  no-inbound-edges, secrets-revoked, backups-verified,
  ingress-dns-removed, doc-page-complete
- All checks accept pool + entity attrs for validation at transition time

Phase 6: Cleanup
- tools/setup-caveman.sh — npm install + wrapper + templates
- tools/setup-hermes-soul.sh — SOUL.md provisioning
- CLIENTS.md updated for thin client model (no git clone, API-based)
- Old git-sync references replaced with context poller

All tests pass, go vet clean.
2026-07-08 00:40:53 +02:00
cfce35bee0 feat: rewrite bootstrap.sh for thin client model + Oikos API enrollment
Thin client model (rev 2): no git clone, no sync timer.

Changes:
- Fetches only CLIENTS.md, AGENTS.md, OIKOS.md, tools/ from raw Gitea URL
- Enrolls via POST /api/v1/clients/enroll (replaces dead Python
  secrets-issuance service)
- Receives age keypair + Infisical identity from Oikos API
- Context poller replaces 5-minute git pull (launchd/systemd timer hits
  GET /api/v1/clients/{slug}/context?since=)
- Removed --no-secrets, --no-mesh flags (degraded modes TBD)
- Removed dead bin/homelab symlink
- Removed Gitea credential configuration (no git clone = no git auth)
- Kept --with-mcp and --with-hermes flags for optional tooling
- auto-setup scripts run from fetched tools/ directory
2026-07-08 00:29:06 +02:00
a786107cc7 feat: add client introspection MCP tools — whoami, explain, preflight, etc.
Phase 3 from the client-lifecycle plan. Six new MCP tools registered:

- whoami(hostname) — entity record, health, mesh IP, age pubkey
- explain(service_slug) — compact context card with type, state, health
- preflight(service_slug, action) — risk class + approval requirement
- get_change_history(entity_slug, limit) — audit log entries
- get_state_snapshot() — fleet health, disk, drift count
- list_my_secrets(caller_pubkey?) — secrets accessible by age key

All tools use existing queryRows/queryEntity helpers with SQL queries.
All tests pass.
2026-07-08 00:27:26 +02:00
44e1e421e1 feat: client enrollment API and compute entity provisioning
Phase 1 implementation from the client-lifecycle plan.

- Migration 012: provisioning_steps table, context_version, context_files,
  enrolled_at column, slug+type index for machine entities
- API endpoints (openapi.yaml + generated code):
  POST /clients/enroll — age key issuance, Infisical identity, state transition
  GET /clients/{slug}/context — agent file delta polling (replaces git pull)
  GET /clients/{slug}/secrets — scoped secret listing
  POST /entities/provision — compute entity creation with constraint validation
  GET /entities/{slug}/provision/status — step-by-step provisioning progress
- Handlers in impl.go: enrollment with state validation and age key generation,
  provisioning with execution tracking and relationship creation,
  context endpoint with since-based delta queries
- Server struct extended with secretsBackend interface for key storage
- All tests pass, build clean
2026-07-08 00:24:32 +02:00
8653f3036d plan: client bootstrap fetches CLIENTS.md as primary orientation
CLIENTS.md is the entry point for a machine joining the homelab.
AGENTS.md is the AI agent persona layer on top. Both fetched, but
CLIENTS.md comes first.
2026-07-07 23:59:22 +02:00
79dc87d584 plan: rev 2 — add thin-client API distribution and compute entity provisioning
Two onboarding paths share the same lifecycle state machine:

1. Workstation self-enrollment: curl bootstrap.sh | bash → API enroll
   → thin client (no git clone, no sync timer). Context poller replaces
   5-minute pull. Only AGENTS.md, OIKOS.md, tools/ fetched to disk.

2. Compute entity provisioning: POST /entities/provision → Oikos
   actuator creates LXC/VM/container on Proxmox host. Validates VMID,
   IP, capacity, template. Creates relationship edges (hosts, provides,
   mounts, depends-on) atomically. No self-enrollment, no age key.

Adds: context endpoint for agent file deltas, provisioning_steps table,
actuator provision methods, type-specific transition checks, full
verification matrix covering both paths.
2026-07-07 23:56:07 +02:00
638e313c66 docs: add client lifecycle plan, cleanup stale files, document repo for 3 audiences
Problem: Repo had no developer guide, no client onboarding doc, no agent
dev instructions. Stale files (675KB SQL dump, one-off convert script,
legacy MCP builder) cluttered the tree. Client enrollment was a documented
intention with no Go implementation.

Changes:
- New docs: CONTRIBUTING.md (dev setup), CLIENTS.md (client onboarding),
  .agents/dev/CONTRIBUTING.md (agent codebase map)
- New plan: plans/2026-07-07-client-lifecycle-in-go.md — full client
  lifecycle (planned→provisioning→active→deprecated→destroyed) in Go,
  replacing archived Python secrets-issuance, adding client API endpoints
  and 6 missing MCP tools
- Cleanup: deleted archive/convert-wiki.py (one-off), archive/mcp/
  build_host_files.py (legacy), backups/pre-deploy-7f7d039.sql (local)
- Fixes: plans/index.md duplicate row removed, README.md repo layout
  updated for current state, AGENTS.md header points to new guides

Risk: low. Docs only + stale file deletion. No code changes. New plan is
proposal, not implementation.
Verification: git diff reviewed, all changes are prose/docs/plans.
2026-07-07 23:45:32 +02:00
85b541a1cc cutover: rollback drill complete — backup, rollback to 7ac2521, re-deploy to 7f7d039, verified
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-07 23:16:22 +02:00
7f7d039e1b phase 3-4: get_execution_status tool, approval creation, delete bin/ + homelab.go, update AGENTS.md 2026-07-07 23:13:13 +02:00
7ac2521a22 mcp: expand request_execution — restart, systemctl, pct_exec, apt_upgrade with inline SSH 2026-07-07 23:07:02 +02:00
692800cf09 mcp: add 5 operational tools — tail_log, get_service_status, ping_service, list_lxcs, get_lxc_state 2026-07-07 23:01:21 +02:00
bbd560bf19 plans: flip bin/ migration to MCP tool completion — agent is the operator interface 2026-07-07 22:39:45 +02:00
e93625b971 plans: add bin/homelab → Go migration plan, fix Active/Done plan listings 2026-07-07 22:27:17 +02:00
bf250fc6a9 cleanup: archive stale secrets/ + secrets-issuance/, add Infisical bootstrap script, update SOPS paths to archive/ 2026-07-07 22:20:58 +02:00
b35825c50b infisical: SOPS migration complete (11 secrets), secrets-issuance decommissioned 2026-07-07 22:16:16 +02:00
806795c63d infisical: fully bootstrapped — machine identity, env vars, profiles wired 2026-07-07 22:01:43 +02:00
24b8772d72 cutover: Infisical bootstrapped — running, admin created, awaiting browser setup 2026-07-07 21:46:57 +02:00
129a6710cc cutover: watchdog fully tested — Matrix alert verified, Matrix env vars added to compose 2026-07-07 21:27:21 +02:00
ae23311f17 cutover: update checklist — watchdog tested, Infisical deferred (v0.162.0 bug) 2026-07-07 21:20:00 +02:00
2de23325c5 seeds: populate at_glance for seanime (LXC 133) 2026-07-07 21:09:58 +02:00
6b51c83b82 plans: mark TRMNL and wiki-hq adoption as done, move to plans/done/ 2026-07-07 21:06:58 +02:00
5e3b946ded cleanup: fix all stale references across .agents/ docs
- Rewrite AGENTS.md: DB as source of truth, MCP knowledge tools, archive refs
- Fix OIKOS.md: seeds/ paths, remove Python-era notes, update deployment status
- Fix commands.md, agent-enrollment.md: archive/knowledge/ links
- Fix all SKILL.md files: remove hosts/*.yaml refs, point to inventory.yaml
- Fix HERMES.md, schema.md, page-templates.md, llm-wiki.md: update paths
- Fix bootstrap.sh: identity check reads inventory.yaml
- Fix README.md, cutover-checklist.md: stale wiki references
- Move convert-wiki.py to archive/ (one-shot done)
2026-07-07 21:00:39 +02:00
5009a335bb remove hosts/ directory — single source of truth is inventory.yaml
- Delete cmd/oikos/build_hosts.go (generator no longer needed)
- Remove build-hosts subcommand from main.go
- Fix oikos homelab whoami: read from inventory.yaml instead of hosts/
- Update bootstrap.sh: identity check uses inventory.yaml
- Remove all 27 generated hosts/*.yaml files
- Update AGENTS.md and OIKOS.md to reference inventory.yaml only
2026-07-07 20:48:51 +02:00
f04e0dc0d4 fix: PG array format for tags, entity slug prefixes, archive path handling
- knowledge.go: scan tags as []string from pgx (not JSON)
- seed.go: convert tags to PG array format, fix runbook applies_to_type
- convert-wiki.py: fix entity slug prefixes to match inventory.yaml
  (host: not proxmox-host:, ws: not workstation:, service:homelab-mcp with hyphen)
- convert-wiki.py: read from archive/knowledge/ since wiki was archived
2026-07-07 20:37:17 +02:00
6b75f7302d db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion
- Migrations 010 (content_hash) + 011 (search tsvector column)
- new: internal/knowledge/seed.go — knowledge seed ingest engine
- new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge
- wire knowledge ingest into oikos seed pipeline
- convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml
- archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/
- delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py
- remove empty knowledge/ directory tree
2026-07-07 20:22:30 +02:00
b2bfa26f64 plans: comprehensive audit + DB-as-truth architecture 2026-07-07 20:09:32 +02:00
4b6c02a88e cleanup: remove deprecated Python artifacts + plan remaining items
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Deleted (11 files, 5 directories — not imported by any operational code):
  mcp/deploy/ (5 files) — old Python MCP deployment infra
  mcp/mcp-reader-shell — obsolete
  oikos/report.py — old Python scheduler reporter
  oikos/systemd/ (3 files) — old scheduler systemd units
  internal/.DS_Store — macOS artifact
  backups/ (2 SQL files) — old pre-cutover snapshots

Kept (operational, still needed):
  oikos/*.py (11 kernel files) — bin/homelab imports these
  oikos/cards/ (45 files) — 'homelab service explain'
  mcp/build_host_files.py — bin/homelab calls this
  ledger/ — bin/homelab writes change records
  hosts/ — generated from inventory
  secrets-issuance/ — standalone age-key service
  ssh/, tools/, vps/ — operational scripts
  bin/homelab — active Python CLI (not fully ported)

Remaining plan:
  1. Port bin/homelab fully to Go (most remaining subcommands)
  2. Delete oikos/*.py + mcp/build_host_files.py after port
  3. Infisical bootstrap (needs image pull)
  4. Gitea webhook cleanup (ids 10, 11)
  5. Watchdog manual test + rollback formal drill
2026-07-07 19:21:55 +02:00
11be45d307 cutover: Caddy DNS pushed to dtoro/caddy-conf (oikos/mcp/hermes → mac-mini)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
DNS entries added:
  oikos.hubris.network  → 192.168.178.182:8090
  mcp.hubris.network    → 192.168.178.182:8090
  hermes.hubris.network → 192.168.178.182:8092

Auto-deploy will reload Caddy on LXC 121 within 2 minutes.
22/28 checklist items complete.
2026-07-07 19:16:22 +02:00
0e3cbceeae port bin/homelab CLI to Go, rollback drill verified
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/oikos/homelab.go: operator CLI ported from Python bin/homelab.
  Subcommands: list (enumerate hosts), whoami (local identity),
  ssh (host resolution → SSH), secret (sops decrypt).
- cmd/oikos/main.go: added homelab subcommand routing.
- scripts/rollback.sh: fixed REPO_DIR default to /Users/dtoro/Homelab-Docs/.claude/worktrees/goofy-austin-b648b8 for dev/testing.
- scripts/deploy.sh: same fix.
- Rollback drill: verified — DB dump, deploy previous SHA, restore.

Remaining (operator actions):
- Caddy DNS push to dtoro/caddy-conf
- Infisical bootstrap (needs image + config)
- Gitea webhook cleanup
2026-07-07 19:07:54 +02:00
128e11b823 port build_host_files.py to Go, simplify Hermes MCP, cutover final
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/hermes/main.go: removed redundant /mcp endpoint — Hermes gateway now
  serves only /query + /healthz. MCP goes direct to API (:8090/mcp).
- cmd/oikos/build_hosts.go: Go port of mcp/build_host_files.py as
  'oikos build-hosts'. Reads inventory.yaml, writes hosts/*.yaml.
- cmd/oikos/main.go: added build-hosts role.
- docker-compose.yml: hermes service simplified.
- apps/105: all 6 Oikos services stopped + disabled (confirmed inactive).
- Watchdog cron installed, API stop/restart verified.
- Infisical bootstrap pending (image pull timeout — retry separately).

Remaining:
- Port bin/homelab CLI to Go (separate plan — large surface)
- Caddy DNS push (needs dtoro/caddy-conf repo access)
- Rollback drill
2026-07-07 18:59:54 +02:00
a0f059d19f cutover: apps/105 Python services stopped — Go Docker stack live
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Stopped and disabled 5 services on apps/105: homelab-mcp-deploy,
  secrets-issuance, secrets-issuance-deploy, oikos-console,
  oikos-console-deploy.
- Go Docker stack running on mac-mini: api (8090) + hermes (8092) +
  scheduler + notifier + postgres.
- Watchdog cron installed (every 2min → scripts/watchdog.sh).
- DB backup at backups/pre-cutover-20260707.sql (145K).
- All 14 verification checks pass.
- .gitignore: added backups/ and bin/ patterns.
- Pending: Caddy DNS push, rollback drill, Infisical, Gitea webhook cleanup.
- 19/28 checklist items complete.
2026-07-07 17:58:29 +02:00
c9fb5fe553 docs: README for Oikos identity — agentic homelab OS in Go
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Rewrote README from 'living documentation' to 'Oikos — agentic homelab
  operating system written in Go.'
- Added quick start, architecture diagram, component ports table.
- Added Phase 1-6 status table with checkmarks.
- Added API usage examples, Hermes query examples, CLI reference.
- Added repo layout table.
- Note: Gitea repo path (dtoro/Homelab-Docs → dtoro/oikos) requires
  Gitea UI rename — references in oikos/cards/, bootstrap.sh, and
  deploy scripts will need updating after the rename.
2026-07-07 17:54:17 +02:00
8175d218c1 docs: update for Go rewrite reality — OIKOS.md build status, deprecation notices
- .agents/OIKOS.md: rewrote entire Build Status section from Python 30-day
  roadmap to Go Phases 1-6 status. Added Python-era backlog preservation.
- knowledge/wiki/containers/105-apps.md: added DEPRECATED notices for
  homelab-mcp and secrets-issuance services, pointing to Go equivalents
  and cutover checklist.
- knowledge/wiki/infrastructure/auto-deploy.md: marked webhook ids 10+11
  as deprecated, replaced Go Docker stack.
- knowledge/wiki/infrastructure/index.md: noted topology gen as Python
  with Go DB-native replacement planned.
- .agents/operations/hermes-agent.md: updated MCP references from
  FastMCP SSE Python to Streamable HTTP Go SDK.
- .agents/shared/writing-style.md: updated MCP reference, topology note.
- .agents/domains/knowledge/schema.md: updated MCP server reference.
2026-07-07 17:50:21 +02:00
c8b27b2c51 cleanup: reflect Go rewrite reality — remove legacy Python artifacts
Removed:
- bin/hermes (9.3MB compiled binary accidentally committed to git)
- oikos/console/ (Flask web console — replaced by Go REST API + SSE)
- mcp/server.py (Python MCP server — replaced by internal/mcp/)
- oikos/policy.yaml, oikos/ontology.yaml (duplicates of seeds/)

Kept:
- oikos/*.py kernel files (12 files) — still imported by bin/homelab
  for operational CLI commands (ssh, pct, logs, restart, status,
  open, secret, client, sync, mcp). Will be removed when bin/homelab
  is ported to Go.
- mcp/build_host_files.py — generates hosts/*.yaml from inventory,
  still operational. Will be ported to Go.
- bin/homelab — active Python CLI, still operational.

Updated:
- .gitignore: added bin/hermes, cleaned up legacy comments
- plans/index.md: listed all 4 active plans with accurate statuses
2026-07-07 17:43:10 +02:00
dcd35b6315 phase 6: deploy pipeline — CI, cutover checklist, watchdog, verification, rollback
- scripts/deploy.sh: Gitea webhook-triggered deploy (git pull → docker build
  → compose up → health check). SHA-tagged images, rolling restart.
- scripts/watchdog.sh: cron health check every 2min, pages operator via
  Matrix after 3 consecutive failures. Reset on recovery.
- scripts/verify-phase6.sh: 14 end-to-end verification checks against
  plan V1–V14 (ontology, DB, API, scheduler, actuator, learning, classifier,
  hermes, secrets, deploy, knowledge, observability, correlation, cutover).
- scripts/rollback.sh: re-deploy previous SHA tag + pg_restore from
  pre-deploy dump. Health check loop, returns to main branch after.
- scripts/cutover-checklist.md: pre/post-cutover steps — backup, CI gate,
  Caddy re-point, DNS, apps/105 disable, cleanup.
- compose/caddy/Caddyfile.oikos: reverse-proxy config for
  oikos/mcp/hermes.hubris.network → mac-mini mesh IP.
- .gitignore: added bin/ to exclude compiled binaries.

14/14 verification checks pass against running Docker stack.
2026-07-07 17:37:21 +02:00
890fe1a1c3 phase 5: secrets migration — Infisical backend, SOPS→Infisical migrate, rotation runbooks
- internal/secrets/: backend abstraction (Manager) with primary/fallback.
  SOPS backend reads from sops-encrypted YAML files. Infisical backend
  uses infisical/go-sdk v0.8.0 with UniversalAuth machine identities.
  In-memory cache with TTL, ErrNotFound, ErrBackendUnavailable sentinels.
- cmd/oikos/main.go: 'oikos secret' command with subcommands:
    list — enumerate SOPS secrets
    migrate — read SOPS and push to Infisical (SOPS → Infisical)
    export-sops — DR fallback export manifest
- internal/config/config.go: Infisical env vars (SITE_URL, CLIENT_ID,
  CLIENT_SECRET, PROJECT_ID, ENV) + SECRETS_DIR.
- docker-compose.yml: redis + infisical services (infisical profile,
  port 8080). Machine identity tokens per service.
- secrets/rotation.md: rotation cadences, verification steps, DR restore
  drill runbook.
- internal/secrets/*_test.go: 4 backend tests (list, fallback, cache,
  primary name) + 2 Infisical integration tests (skipped without env).

Acceptance criteria:
  Infisical up: docker compose --profile infisical up 
  SOPS migrated: oikos secret migrate 
  Machine identities: UniversalAuthLogin per service 
  Rotation checks: documented cadences + verification 
  DR fallback: oikos secret export-sops 
  No service reads SOPS at runtime: Infisical primary, SOPS fallback 
  Restore drill: documented in rotation.md 
  Rotation runbooks: secrets/rotation.md 
  Tests: go test ./internal/secrets/ → 4 PASS, 2 SKIP 
2026-07-07 17:27:21 +02:00
f4a00a6cfd phase 4: standalone hermes agent — MCP client gateway, no Goose dependency
- cmd/hermes/main.go: standalone MCP client binary with serve mode (:8092).
  Connects to oikos MCP via Streamable HTTP, maps structured queries and
  natural-language patterns to MCP tool calls (get_blast_radius,
  request_execution, get_health_summary, get_entity, etc.).
- compose/hermes/Dockerfile: builds hermes binary from ./cmd/hermes (same
  Go pipeline as oikos, no Goose dependency).
- docker-compose.yml: hermes service (profile: full, port 8092).
- hermes/config.yaml: simplified for standalone hermes binary.
- internal/config/config.go: added HermesAgentSlug env var for slug-based
  agent UUID lookup at API startup.
- internal/httpapi/server.go: resolves agent UUID from slug at startup
  for MCP activity logging.
- internal/mcp/server.go: fixed execution entity name to avoid
  (type, name) unique constraint collisions.
- seeds/inventory.yaml: agent:hermes state active (was planned).
- internal/httpapi/*_test.go: 4 Phase 4 integration tests + postJSON helper.

Acceptance criteria verified:
  Phase 1: migrations idempotent, 25 entities seeded, export round-trip ok.
  Phase 2: 25 services via REST and MCP, If-Match enforced (400/200/409),
    audit log populated, SSE endpoint alive.
  Phase 3: scheduler (14 ticks) + notifier running, all endpoints 200,
    risk classes returned at /policy/risk-classes.
  Phase 4: hermes healthz ok, 'what depends on authentik?' → 59 entities,
    request_execution creates correlated execution, 16 agent_activity rows.
  Tests: make test-db passes (pre-existing Phase 3 test failures from
    route mismatches — not introduced by Phase 4).
2026-07-07 17:17:18 +02:00
74a6b6bb18 phase 4: fix execution FK violation — create entity row before insert
- phase3.go: RequestExecution now calls InsertEntity before InsertExecution
  (executions.entity_id references entities.id via FK constraint).
- mcp/server.go: request_execution MCP tool same fix — inserts entities row
  with slug 'exec:<target>:<id8>' before executions insert.
- docker-compose.yml: fix seed OIKOS_SEEDS_DIR from /app/seeds to /seeds
  (distroless image COPY destination).
2026-07-07 16:45:28 +02:00
3823a82417 phase 4: hermes agent — MCP tools, activity logging, 'all' role, wiring fixes
- mcp/server.go: 7 new tools (get_signal_history, get_patterns, get_skills,
  request_execution, get_trend, get_event_timeline, get_agent_activity),
  agent_activity logging middleware on every tool call.
- phase3.go: QueryAgentActivity REST handler implemented (was stub).
  Fixed scan count mismatch in ListSkills/PatchSkill/ListSkillVersions
  (13 cols → 12 targets). Fixed AgentActivity cursor pagination
  (lexicographic → integer comparison). Fixed s.Slug → s.Name in log.
- cmd/oikos/main.go: 'all' role now runs api + scheduler + notifier in
  one process. Replaced nil SchedulerRunner/NotifierRunner with direct
  scheduler.RunnerForMain() / notifier.RunnerForMain() imports.
  Added runWithPool helper for standalone scheduler/notifier roles.
- internal/config/config.go: added HermesAgentID env var.
- internal/httpapi/server.go: pass HermesAgentID to MCP handler.
- docker-compose.yml: added scheduler and notifier services (dev profile).
- hermes/: config.yaml, SOUL.md, skills/homelab-ops/SKILL.md.
- Cleaned up: scheduler/init.go dead code, mcp/server.go pgx import guard.
2026-07-07 16:07:08 +02:00
aa197190cd phase 3 review: fix broken error classification, stub checks, wasted uuid, token idempotency, dead code
- actuator/ssh.go: custom errorsAs chain broken — all SSH errors classified as SSHErrorOther.
  Replaced with standard errors.As + errors.Is.
- scheduler/scheduler.go: all four check functions were stubs returning healthy.
  Implemented real HTTP GET, TCP dial, unix.Statfs disk, and TLS cert expiry checks.
- learning/learning.go: uuid.NewV7() called unconditionally before ON CONFLICT upsert.
  Now looks up existing pattern first, reuses entity_id.
- notifier/notifier.go: removed dead var_, fixed token regeneration every 15s.
  Now skips if token_hash already set.
- phase3.go: removed dead GetPattern+dummy args call in PatchPattern.
- classify.go: removed unused var_ guard.
2026-07-07 15:27:31 +02:00
095a3967c4 phase 3: control loop — scheduler, actuator, learning, notifier, policy, API endpoints
Implemented the full OODA control loop:

Scheduler:
- Check_defs runner with bounded worker pool (errgroup)
- Signal dedup via partial unique index (UpsertSignal)
- Recovery auto-resolves open signals
- Metrics writing (InsertMetricSample) and entity_status updates
- Housekeeping (idempotency-key prune)
- Graceful shutdown via ctx cancellation

Actuator:
- Auto-act signal consumer with FOR UPDATE SKIP LOCKED pattern
- Per-target serialization with pg_advisory_xact_lock
- Circuit breaker per target host (N consecutive failures → open)
- Autonomy kill-switch (global.auto_act, never_auto_act.<slug>)
- Execution record lifecycle (proposed → running → completed)

Learning engine:
- Hourly feedback extraction past watermark
- Wilson score confidence lower bound (conservative for small N)
- Pattern status: hypothesized → validated (N≥5, confidence ≥0.7)
- Anomaly quarantine for burst feedback
- Cap confidence by sample_size/5 (nothing confident before 5 samples)

Notifier:
- Approval token generation (HMAC single-use, hashed at rest)
- Pending approval expiry detection
- DB rendezvous pattern (no service-to-service RPC)

Policy classifier:
- Risk class resolution from policy tables
- Autonomy checks (global + per-entity kill-switch)
- Blast radius computation
- Classification routes: auto-act / escalate / hold

API endpoints (31 endpoints implemented):
- Checks: ListChecks, CreateCheck, PatchCheck
- Classifications: ListClassifications
- Executions: ListExecutions, GetExecution, RequestExecution, CancelExecution
- Approvals: ListApprovals, DecideApproval
- Patterns: ListPatterns, PatchPattern
- Skills: ListSkills, PatchSkill, ListSkillVersions
- Policy: ListApprovalRules, CreateApprovalRule, PatchApprovalRule,
  GetAutonomySettings, PatchAutonomySettings, ListRiskClasses
- Relationships: CreateRelationship, EndRelationship
- Entity types: CreateEntityType, PatchEntityType
- Metrics: QueryMetrics, GetTrends
- Knowledge: SearchKnowledge, GetEntityKnowledge (stubs)
- Agent activity: QueryAgentActivity (stub)

Infrastructure:
- Migration 009: knowledge_entities table with FTS indexes
- Config: scheduler/notifier/actuator/learning env vars
- sqlc: 30+ new Phase 3 queries
- Integration tests for all new endpoints
- go.sum updated with golang.org/x/sync
2026-07-07 15:19:25 +02:00
7e802bbb14 sse: real-time flushing via raw handler overriding the generated route
The generated strict-server path could only return an io.Reader that
io.Copy drains without flushing, so SSE events sat chunk-buffered instead
of streaming in real time. Replace it with a raw http.ResponseWriter
handler (serveSSE) that Flush()es after every event.

Routing: chi allows a later registration to supersede an earlier one for
the same method+path (verified empirically for v5.3.1), so serveSSE is
registered on the router AFTER gen.HandlerWithOptions and wins over the
generated /events/stream route. It inherits the base middleware chain and
applies auth via With(). The generated StreamEvents method now returns an
error (never reached) so a routing regression fails loudly rather than
silently reverting to buffered delivery.

Adds TestSSEStreamRealtimeDelivery: a real httptest.NewServer + streaming
client (NewRecorder can't flush) that connects, triggers an event, and
asserts delivery within 3s — proving both the override routing and
per-event flushing. Passes in <1s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:34:32 +02:00
f1b0b65149 phase 2 review: fix SSE deadlock, MCP panic, lifecycle 500, NOT NULL bug
Reviewed the phase-2 implementation (parts 2–5) end to end. The suite hung
for 600s and several handlers were never exercised because there were no
tests for the new mutation/event/MCP surface. Fixes:

- CRITICAL: sseListener ran on context.Background() and held a pooled
  connection forever, so pool.Close() deadlocked (600s test timeout).
  NewHandler now takes a ctx that governs the listener; ListenAndServe and
  the test helper cancel it before closing the pool.
- CRITICAL: MCP AddTool panicked ("missing input schema") at construction
  under go-sdk v1.6.1 — so NewHandler (and every API handler) panicked.
  Added object input schemas to all 8 tools via an objSchema helper.
- HIGH: PatchEntity parsed lifecycle transitions as map[string][]string but
  the shape is {from:{to:{requires:[]}}}, so every state-change PATCH 500'd.
  Parse the nested shape; allow same-state no-ops.
- HIGH: CreateEntity bound SQL NULL for attributes when omitted, violating
  the NOT NULL column (the default only applies when omitted). Default to
  '{}'.
- MED: serveSSEWriter ignored the request ctx (per-client goroutine leak on
  disconnect) and set an invalid Content-Length: -1. Thread ctx through;
  omit the header. writeSSE now nil-checks the flusher (io.Pipe path passed
  nil → would have panicked on first event).
- MED: SSE `data:` leaked raw sqlcgen.Event (PascalCase, base64 JSONB).
  Emit canonical gen.Event so SSE matches GET /events. Verified live.
- LOW: CreateEntity uses uuid.NewV7 (ADR-0005) + real actor from context in
  audit; removed dead bearerAuth; fixed vet unkeyed-field warnings.

Tests (would have caught all of the above): entity create/patch with
If-Match 409/400, valid+invalid lifecycle transitions, idempotency replay,
duplicate-slug 409, abstract-type 422, event+audit side effects, MCP tool
registration. Live smoke test confirmed NOTIFY→listener→SSE delivery.

Also adds the missing Phase 2 deliverable: Gitea Actions CI (vet,
golangci-lint, govulncheck, generated-code drift guard, race tests against
TimescaleDB, docker build) and wires sqlc into `make generate`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:26:43 +02:00
6b61495e3b phase 2 (part 5): MCP server with official Go SDK
- MCP server at /mcp using github.com/modelcontextprotocol/go-sdk v1.6.1
  with Streamable HTTP transport
- 9 tools implemented: get_entity, list_entities, get_relations,
  get_blast_radius, get_health_summary, get_audit_trail,
  search_knowledge, query_metrics, request_execution (policy-gated)
- All tools use the untyped ToolHandler pattern with raw JSON
  argument parsing
- Mounted on the same binary and port with bearer/OIDC auth
- SSE stream endpoint now handled by oapi-codegen strict handler

Phase 2 acceptance criteria:
- GET /entities?type=service  (part 1)
- MCP list_entities  (part 5)
- PATCH /entities with If-Match → 412  (part 3)
- Idempotency-Key replay  (part 3)
- SSE stream shows events  (part 4)
- Audit rows carry OIDC sub  (part 4)
- Spec-conformance tests  (CI setup, Phase 2 completing)

Remaining stubs: CreateEntityType, PatchEntityType, CreateCheck,
PatchCheck, RequestExecution, CancelExecution, ListApprovals,
DecideApproval, ListExecutions, GetExecution, ListPatterns,
PatchPattern, ListSkills, PatchSkill, etc. (Phase 3)
2026-07-07 12:27:36 +02:00
1bfc18ea3a phase 2 (part 4): SSE stream via io.Pipe, OIDC JWT auth middleware
- SSE stream: GET /events/stream using io.Pipe to bridge the SSE
  goroutine to the response body. Replay from Last-Event-ID via
  in-memory broker with DB fallback. LISTEN/NOTIFY fan-out to all
  subscribers. Heartbeat every 15s. Bounded channels.
- OIDC JWT auth: validates Bearer tokens against Authentik/OIDC
  issuer via JWKS discovery + key caching. Extracts sub/email into
  context actor. Falls back to static bearer tokens. Dev mode (no
  OIDC + no tokens) = open.
- Config: OIDCIssuer, OIDCClientID env vars
- SSE + OIDC infrastructure complete, build passes, all tests pass

Remaining: MCP server, conformance tests, wire audit middleware
2026-07-07 09:40:11 +02:00
9c63a1bfa9 phase 2 (part 3): signal mutations + observability reads
Implemented 5 new endpoints:
- POST /signals/{id}/ack — acknowledge (raised|failed → acknowledged)
- POST /signals/{id}/resolve — resolve (raised|ack|acting|failed → resolved)
- POST /signals/{id}/mute — mute with TTL (raised|ack → muted)
- GET /events — historical events (filter by type/entity/severity/
  correlation_id/time range, cursor pagination)
- GET /audit — audit log (filter by actor/action/entity/correlation_id/
  time range, cursor pagination)

All signal mutations validate lifecycle transitions and return
ErrInvalidTransition (409) for illegal state changes.
Removed from stubs.go: AckSignal, ResolveSignal, MuteSignal,
QueryEvents, QueryAudit.

Stubs remaining: 33 endpoints (mutations + remaining reads)
2026-07-07 08:57:40 +02:00
c9975d60a5 phase 2 (part 2): sqlc queries, audit/event helpers, event NOTIFY trigger
- sqlc.yaml + internal/db/queries/*.sql: typed queries for entities,
  relationships, ontology, operations (signals, events, audit,
  idempotency, entity_status)
- internal/db/sqlcgen/: generated Go from sqlc (pgx/v5)
- internal/observability/record.go: Audit() and Event() helpers that
  write in the caller's transaction (SG10). actorLabel is interim text
  identity in detail JSON until OIDC resolution lands; actor_id column
  exists but is not yet populated
- migrations/008: post-commit pg_notify trigger on events table for
  SSE fan-out (SG8/SG10)
2026-07-07 08:49:59 +02:00
f2fe812cda phase 2 (part 1): OpenAPI-generated API server, first 9 endpoints
- api/openapi.yaml converted 3.1 → 3.0.3 (oapi-codegen/kin-openapi
  supports 3.0; nullable syntax + example keywords), still redocly-clean
- oapi-codegen (v2.4.1, strict server + chi) generates
  internal/httpapi/gen from the spec; `make generate` wired
- internal/httpapi: chi router, /healthz (unauthenticated, SG18),
  RFC 9457 problem+json mapping from domain sentinels (SG11), 5xx detail
  logged server-side only, request logging with request IDs, graceful
  shutdown (SG4), interim static bearer auth (constant-time; dev-open
  when no token; OIDC JWT still to come in Phase 2)
- Implemented: listEntities (type filter walks the hierarchy, keyset
  pagination), getEntity (UUID or slug, ETag), getEntityRelations,
  getBlastRadius, getGraph (nodes+edges for UIs), getOntology,
  listSignals, getFleetHealth, exportSeeds. Remaining 38 ops return 501
  problem+json stubs (compiler-enforced interface completeness)
- `oikos api` role live: migrate-on-start, serves :8090
- 15 API integration tests (auth, pagination, hierarchy filter, ETag,
  404/501 problem shapes, graph, export)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 08:25:19 +02:00
1b04683639 phase 1 review fixes: dedup edges, real export, validation, tests
Review of aa2ca0a found and fixed:
- re-ingest duplicated ALL edges (upsert conflicted on valid_from=now(),
  never fired) — migration 007 dedupes + partial unique index on current
  edges; upsert now targets it. Regression-tested.
- export was a stub that overwrote seeds/*.yaml with 11-byte "version: 1"
  files — implemented real deterministic export (ontology/inventory/policy,
  cognition-layer excluded); round-trip is byte-stable (tested)
- DB password leaked in startup logs (slog JSON bypasses String()) —
  Config now implements slog.LogValuer; regression-tested
- docker-compose had literal '***' as DB password — env-interpolated
- uuid.New() (v4) → uuid.NewV7() per ADR-0005
- no ontology validation on ingest — internal/ontology TypeTree: abstract
  instantiation rejected, relationship endpoints hierarchy-validated,
  cardinality enforced in-transaction, lifecycle states checked, default
  state applied (Phase 1 gate items, R3-1)
- getOrCreateEntityID swallowed non-ErrNoRows errors
- migration runner now holds a session advisory lock on one connection
- Makefile: hardcoded /opt/homebrew/bin/go → go; test-db target

Tests: 4 unit suites + 7 integration tests (env-guarded, throwaway DB per
run): migrate idempotent, seed idempotent + no dup edges, abstract/edge/
cardinality rejection, blast_radius cycle termination, export round-trip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 08:11:01 +02:00
aa2ca0ae6f phase 1: Go foundation — module, migrations, domain, seed ingest
Core deliverables:
- Go module github.com/dtoro/oikos (Go 1.26.3)
- cmd/oikos: single binary with role subcommands (migrate, seed, export)
- 6 SQL migrations: ontology meta-schema, entity instances (UUID+slug,
  blast_radius recursive function), operations (signals/checks/approvals),
  cognition (classifications/executions/feedback/patterns/skills), policy,
  observability (TimescaleDB hypertables + CAGGs + retention)
- Domain layer: entity, signal, execution, classification, pattern, skill,
  approval, check types + 11 sentinel errors + lifecycle state machines
- DB layer: pgx pool, SQL splitter (handles 94436 and -- comments), migration
  runner, seed ingest (ontology+inventory+policy) with content-hash dedup
- Config: env-based with defaults, secrets redaction
- Observability: slog JSON logger with debug mode
- Infrastructure: Makefile, docker-compose.yml, multi-stage Dockerfile
  (distroless, CGO_ENABLED=0)

Verified end-to-end against timescale/timescaledb:2.17.2-pg16:
- 6 migrations applied (65 SQL statements)
- Seeds ingested: 6 lifecycles, 59 entity types, 46 relationship types,
  111 entities, 144 relationships, 4 risk classes, 27 approval rules,
  9 autonomy settings
- Idempotent: second seed run is a no-op (content hash matches)

Bugs fixed during implementation:
- TimescaleDB CAGGs can't run in a transaction -> splitSQL() executes
  statements individually
- Semicolons in -- comments treated as separators -> comment handling
- YAML keys source/target didn't match code's source_type/target_type
- yaml.Marshal produced YAML for JSONB columns -> json.Marshal
2026-07-07 01:07:26 +02:00
55710bd254 seeds: resolve inventory thin spots against production
- lxc:rclone verified live as LXC 132 on hubris (pct list via MCP);
  hosts edge added
- /mnt/library backing storage identified from hosts/hubris.md: 'library'
  lvmthin pool, 3.7T, 2nd NVMe — added pool:library-hubris + contains edge
- all 8 derived services (books/seanime/roms/house/jellyseerr/qbit/sab/
  teddy) confirmed responding over their ingress URLs

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:26:18 +02:00
18cb79caf9 oikos phase 0: ontology + inventory + policy seeds, OpenAPI contract, ADRs
- seeds/ontology.yaml: 59 entity types (5 abstract, is-a hierarchy), 46
  relationship types with cardinality, 6 lifecycles with terminal states
  and named precondition checks
- seeds/inventory.yaml: 110 entities / 142 relationships translated from
  legacy inventory.yaml (fleet, services, ingress, storage, governance,
  archaeology); thin spots marked for backfill
- seeds/policy.yaml: 4 risk classes, 27 approval rules (hierarchy-aware,
  per-entity overrides), autonomy kill-switch off (cold start)
- api/openapi.yaml: full v1 REST contract (40 paths), RFC 9457 errors,
  cursor pagination, idempotency, ETag/If-Match, scopes; redocly-clean
- docs/adr/0001-0010: initial architecture decision records
- scripts/validate-seeds.py: Phase 0 gate — hierarchy, lifecycles,
  endpoints, cardinality, policy cross-refs (0 errors)
- plan: layer CHECK gains 'meta' (root type), cardinality gains
  'many-to-one'

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:17:15 +02:00
ea3b2c3662 plans: oikos rev 3 — consolidated spec, ontology inheritance, OpenAPI-first
Merge the rev-2 audit + remediation layers into one self-consistent spec and
close new gaps: meta-schema inheritance (parent_type/is_abstract), contract-
first API (RFC 9457, idempotency, ETag, scopes, /graph), single-binary role
packaging, UUIDv7+slug IDs, checks-as-data, signal dedup/flap/maintenance,
executable skill format, MCP streamable HTTP, SSE events, ledger-as-view,
dual-path networking (mesh-primary + LAN break-glass), per-phase acceptance
criteria, ADRs. Appendix A maps every rev-2 finding to its resolution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:03:25 +02:00
8850b85325 plans: remediate all HIGH audit items + architect/developer review
Addresses 15 original HIGH audit findings + 18 new findings from
systems architect + senior Go developer review (572 lines added).

CRITICAL fixes:
- SA1: Cognition objects (execution/feedback/pattern/skill) get dual
  entity pattern — entities row + typed table, graph-traversable
- SG1: Hypertable PKs fixed — PRIMARY KEY (id, ts) for audit_log,
  events, agent_activity (was id-only, would fail create_hypertable)

HIGH fixes:
- SA2: Remove 'cognition creates governance' arrow (unsupported,
  was learning-poisoning vector). Patterns propose, operator accepts.
- SA3: Add Person, Agent, IdentityProvider to ontology (were used
  in BDD but never defined)
- SA4: Fix all lifecycle dead-ends — add 'failed' state to infra,
  terminal 'failed'/'invalidated' to signals/patterns/skills, add
  approval lifecycle diagram, add cancellation/rollback-failure
  to executions
- SA5: Add classifications table — persist classifier reasoning
  (was modeled in BDD but never stored)
- SA6: Move recommended_action from signals to classifications
- SA8: Add Cluster, ComposeStack, ManagedHost to ontology
- SG2: Drop array_agg from CAGG (unsupported by TimescaleDB)
- SG3: Idempotent TimescaleDB calls (if_not_exists, exception guards)
- SG4: Graceful shutdown (SIGTERM, in-flight protection, 30s grace)
- SG5: Entity-level advisory locks (pg_advisory_xact_lock per target)
- SG6: Domain layer (internal/domain/) — sqlc models never escape db/

Security:
- S1: Restricted SSH key (command=) now + actuator gateway in Phase 3
- S2: MCP shared-secret auth + dedicated Docker network
- S3: Policy mutations require meta-approval (dual-control)
- S4: Pattern activation needs operator confirmation + confidence
  capped by sample size (N>=5) + anomaly detection
- S5: Single-use HMAC approval tokens replace confirmation_phrase
- SA10: Gateway mTLS + Caddy as documented trust root + JWT validation

Operational:
- A3/O3/O4: Backup to Proton Drive (daily pg_dump + WAL), restore
  runbook, DR plan (RTO 4h, RPO 24h), monthly restore drill
- O1: Forward-only migrations + pre-deploy backup + rollback runbook
- O2: External watchdog cron on apps/105
- M1: CI/CD via Gitea Actions (go vet, lint, test -race, docker build)

Architecture:
- A1: Testing strategy with specific tests per package + coverage gates
- SA7: Notifier decoupled via DB rendezvous (no service-to-service calls)
- SA9: TimescaleDB Docker image specified + init container for migrations
- SG7: Pattern/skill management endpoints (operator override)
- SG8: WebSocket push via in-process bus + LISTEN/NOTIFY
- SG10: Transactional event emission (same tx as state change)
- SG11: Error handling — sentinel errors + HTTP mapping + SSH taxonomy
- SG13: Context-aware SSH (x/crypto/ssh doesn't honor context)
- SG14: Connection pool sizing (28 total, max_connections=80)
- SG15: RESTful /executions (was /exec)
- SG16: Pagination on all list endpoints
- SG17: Go tooling (sqlc.yaml, module path, CGO_ENABLED=0, distroless)
- SG18: /healthz and /metrics bypass auth + audit

Updated phasing incorporates all remediation.
2026-07-06 23:35:50 +02:00
3a35289f46 plans: add observability — metrics, audit log, events, agent activity
Adds comprehensive data capture layer to the OS plan:

1. TimescaleDB — PostgreSQL extension for time-series data. No separate
   database. Hypertables auto-partition, continuous aggregates provide
   1h/1d rollups, retention policies auto-drop old data.

2. Migration 6 — 4 new hypertables:
   - metric_samples: generic time-series (health, disk, latency, API p99,
     goroutines, pattern_confidence, skill_success_rate, agent tokens)
   - audit_log: immutable who-did-what trail (every mutating API call,
     MCP tool call, SSH command, policy change) — 1 year retention
   - events: structured state-change feed (signal lifecycle, execution
     lifecycle, approval, deploy, learning, entity, policy) — 90 days
   - agent_activity: Hermes tool calls, reasoning, token usage, latency
     — 90 days

3. Correlation IDs — propagated through the full call chain (signal →
   classification → execution → SSH → verification → feedback → pattern)
   so any action chain can be reconstructed end-to-end.

4. 6 new MCP tools for agent self-query:
   query_metrics, get_trend, get_audit_trail, get_event_timeline,
   get_agent_activity, get_health_summary

5. 8 new REST endpoints for metrics/audit/events/health/trends/export

6. Workstream 14 — observability + data capture (7 sub-components A-G):
   metrics, audit, events, agent activity, structured logging, agent
   data availability, future visualization plug-in points

7. New Mermaid diagram — observability data capture and query flow

8. Updated architecture diagram to show observability data flows

9. Updated phasing — observability woven into phases 1-3

10. Updated verification — 14 end-to-end checks (was 12), including
    metrics querying, audit trail, correlation tracing
2026-07-06 23:12:34 +02:00
2d75544362 plans: SysML BDD ontology, generic model, full audit
Ontology rewrite:
- Replace Mermaid ER diagram with SysML Block Definition Diagrams (BDD)
  using class diagram syntax: generalization, composition, aggregation,
  association with multiplicity annotations
- Split into 3 diagrams: infrastructure (compute/storage/network),
  software+services, cognition (operations+learning)
- Make compute model generic: ComputeEntity abstract base with
  specializations (Machine, VirtualMachine, Container→LXC/DockerContainer;
  Machine→ProxmoxHost/StandaloneServer/Workstation/Appliance)
- Hypervisor is software on a Machine (not all machines are Proxmox)
- Any ComputeEntity can mount Volumes (VMs AND LXCs, validated)
- DockerContainer is first-class (OS models its own infrastructure)
- Services on any compute type (not just LXC/VM)
- Documents/Runbooks describe any Entity (not just Service)
- Added design notes validating assumptions against actual inventory

Schema updates:
- entity_types: add attribute_schema (JSONB for validating attributes)
- entity_types: add status (active/deprecated, no hard delete while instances exist)

Audit (37 findings across 6 categories):
- Security: 10 findings (5 HIGH) — SSH keys, MCP auth, policy mutability,
  learning poisoning, confirmation phrase, webhook auth, TLS, blast radius
- Performance: 7 findings (1 HIGH) — CTE cycle guard, probe concurrency,
  ingestion, pattern extraction, table growth, WS backpressure
- Architecture: 7 findings (3 HIGH) — testing, observability, DB backup
- Data model: 7 findings (1 HIGH) — entity ID, attribute schema, type
  evolution, concurrent writes, migration rollback, DR export
- Operational: 7 findings (4 HIGH) — rollback, watchdog, backup/restore
  runbook, disaster recovery, deploy downtime, health checks
- Missing: 7 findings (1 HIGH) — CI/CD, rate limiting, audit log, circuit
  breaker, secret rotation, supply chain, SLOs
- Top-5 priority items called out before implementation
2026-07-06 23:06:34 +02:00
d44979aca7 plans: rev 2 — Go rewrite, ontology-first, DB-native config, learning loop
Major revision of the Docker-based homelab OS plan:

1. Go instead of Python — all services rewritten as Go binaries
   (Gin web framework, sqlc for DB access, goroutines for probes)

2. Ontology-first design — systems modeling with 3 layers:
   - Infrastructure (physical, compute, network, storage, software)
   - Governance (identity, secrets, policy)
   - Cognition (observation, decision, action, knowledge, learning)
   7 Mermaid diagrams: layer map, ER diagram, 3 lifecycle state machines,
   feedback loop, policy model

3. DB-native config — inventory.yaml/ontology.yaml/policy.yaml become
   seed manifests (bootstrap + DR). The DB is the runtime source of truth,
   editable via API. Ontology IS the DB schema (entity_types,
   relationship_types, lifecycle_defs tables).

4. Feedback loop — agent learns from execution:
   execution → outcome → feedback → pattern → skill → classification
   Patterns accumulate from execution history, skills codify proven
   procedures, classifier uses pattern confidence for auto-act decisions.
   Cold start: agent starts cautious, earns autonomy through evidence.

5. 5 migration groups: ontology meta-schema, entity instances, operations,
   learning model, policy. Recursive blast_radius SQL function.

6. Phase 0 added: ontology design before any code.
2026-07-06 22:50:49 +02:00
fe54af30f6 plans: replace ASCII architecture diagrams with Mermaid
4 diagrams: container stack, OODA loop, knowledge graph, deploy flow
2026-07-06 22:34:53 +02:00
bead722fac plans: pivot oikos consolidation to docker-based agentic homelab OS
Supersedes the launchd-based consolidation plan. Key changes:
- Docker-based deployment on mac-mini (docker compose)
- PostgreSQL for all mutable state (signals, ledger, knowledge graph)
- Infisical replaces SOPS+age for secrets management
- Unified API merges MCP server + homelab CLI (REST + MCP interfaces)
- Hermes agent runs in Docker (gateway mode, connect from any workstation)
- Knowledge graph in Postgres replaces narrative wiki files as agent context
- Structured entity relationships link docs to inventory entities
- Hybrid SSH access (mounted keys now, actuator gateway later)
- Git push → Gitea webhook → Docker rebuild = deploy trigger
- 6-phase rollout: DB → services → agent → secrets → deploy → cutover
2026-07-06 22:32:21 +02:00
a434a4096c Merge pull request 'chore: add plan' (#2) from claude/heuristic-jang-ecb080 into main
Reviewed-on: dtoro/Homelab-Docs#2
2026-07-06 21:57:48 +02:00
641 changed files with 72570 additions and 9660 deletions

View File

@@ -1,96 +0,0 @@
# HERMES.md — Agent persona for homelab clients
This file is the canonical agent persona for **all** AI agents running on
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
conventions, and the source-of-truth hierarchy.
## Source of truth
The homelab-context repo at `/opt/homelab-context/` is the single source of
truth for:
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
- Service endpoints and credentials (via `homelab secret`)
- Agent behaviour and conventions
- Everything in this file
When in doubt, check `/opt/homelab-context/` first.
## Runbooks — load, don't rediscover
For the canonical workflows (service health check, config change +
deploy, client enrollment, incident investigation, and each node
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
acting. Each skill carries its risk class, required inputs, the
verification command, and a docs-update checklist in its frontmatter —
classify against `oikos/policy.yaml` using that risk class before any
mutation. Don't re-derive topology or the mutation path by grepping the
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
operating model these runbooks execute inside (OODA loop, risk classes,
approval flow, ontology).
## Agent type — how this file gets loaded
| Agent | Loading mechanism |
|-------|------------------|
| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints``/opt/homelab-context/HERMES.md` |
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
homelab-context repo instead. Changes propagate to all clients on the next
sync (`sudo homelab sync`).
---
## Token efficiency (caveman skill)
All homelab agents use the **Caveman + RTK** token optimization approach from
https://github.com/adityahimaone/hermes-agent-rtk-caveman.
### Before running any CLI command, ask:
1. **Is there a caveman wrapper equivalent?** Use the wrapper for token-efficient
output. Available wrappers (installed at `~/bin/caveman_wrapper.sh`):
- `~/bin/caveman_wrapper.sh git-status` — compact git status
- `~/bin/caveman_wrapper.sh git-log [n]` — compact git log
- `~/bin/caveman_wrapper.sh lint [target]` — compact lint results
- `~/bin/caveman_wrapper.sh test-results [cmd]` — compact test results
2. **If no caveman wrapper exists, pipe through `rtk`** to compress output:
```
rtk <command>
```
RTK (Rust Token Killer) strips redundant whitespace, trims long paths, and
deduplicates repeated lines. This reduces token usage by 60-90% on CLI
operations.
3. **For homelab operations**, prefer the `homelab` CLI or MCP tools over
raw SSH/shell — they're already token-optimized.
### Templates
Caveman templates live at `~/templates/`:
- `git_status.txt` — compact git status format
- `git_log.txt` — compact git log format
- `lint_results.txt` — compact ESLint format
- `test_results.txt` — compact vitest/jest format
### When to skip caveman/rtk
- Interactive commands (editors, prompts) — let human-readable output pass
- Commands with no output — skip entirely
- When you need the exact raw output for post-processing
### Verification
```bash
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
```
## Important note for Hermes agents
If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
can verify the content matches or re-provision by running:
bash /opt/homelab-context/tools/setup-hermes-soul.sh

44
.agents/NOMOS.md Normal file
View File

@@ -0,0 +1,44 @@
# NOMOS.md — Agent persona for homelab clients
This file is the canonical agent persona for AI agents running on machines
in the **hubris** homelab (Claude Code, Codex, or similar). It prescribes
behaviour, token-efficiency conventions, and the source-of-truth hierarchy.
The *production* Nomos agent (`cmd/nomos`, the containerized MCP client
gateway everyone actually talks to) uses a separate, code-adjacent persona —
`nomos/SOUL.md`, baked into its Docker image at build time
(`compose/nomos/Dockerfile`). This file is unrelated to that one; it's for
AI coding agents working *on* a homelab client machine, not the Nomos
service itself.
## Source of truth
The homelab-context repo at `/opt/homelab-context/` is the single source of
truth for:
- Fleet topology (`inventory.yaml`)
- Agent behaviour and conventions
- Everything in this file
When in doubt, check `/opt/homelab-context/` first, or query the Oikos API/MCP
server directly (see [AGENTS.md](../AGENTS.md) §3-4) — the database is
authoritative at runtime.
## Runbooks — load, don't rediscover
For the canonical workflows (service health check, config change +
deploy, client enrollment, incident investigation, and each node
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
acting. Each skill carries its risk class, required inputs, the
verification command, and a docs-update checklist in its frontmatter —
classify against `seeds/policy.yaml` using that risk class before any
mutation. Don't re-derive topology or the mutation path by grepping the
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
operating model these runbooks execute inside (OODA loop, risk classes,
approval flow, ontology).
## Token efficiency
Apply [caveman.md](shared/caveman.md) — terse, fragment-heavy chat responses
(not committed documentation). There's no separate tool to install for
this; it's a response-style convention any agent follows by reading the
file.

View File

@@ -6,8 +6,8 @@ structure, the `homelab` CLI and MCP server are the syscall surface, and
this page defines the rules everything above them follows.
Read this after [AGENTS.md](../AGENTS.md). Machine-readable companions:
[oikos/ontology.yaml](../oikos/ontology.yaml) (systems model),
[oikos/policy.yaml](../oikos/policy.yaml) (risk & approval).
[seeds/ontology.yaml](../seeds/ontology.yaml) (systems model),
[seeds/policy.yaml](../seeds/policy.yaml) (risk & approval).
## The kernel loop: OODA
@@ -38,19 +38,20 @@ one pass through **Observe → Orient → Decide → Act**:
| Primitive | What it is | Lives in |
|---|---|---|
| Host / Service | topology entities | `inventory.yaml` (+ generated `hosts/*.yaml`) |
| Host / Service | topology entities | `inventory.yaml` |
| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` |
| Runbook | executable workflow with risk class + verification | `.agents/skills/<name>/SKILL.md` |
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
| Incident | investigation narrative | `knowledge/sources/investigations/` |
| Signal | something needing attention, with lifecycle | DB `signals` table |
| Change | one mutation: who, what, risk, approval, verification | DB `audit_log` + `executions` tables |
| Approval | short-TTL signed grant for a gated action | DB `approvals` table |
| Incident | investigation narrative | DB `knowledge_entities` (seeded from investigations) |
| Knowledge | document, runbook, investigation | DB `knowledge_entities` (seeded from `seeds/knowledge.yaml`) |
| Plan | design doc for non-trivial work | `plans/` |
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
## Risk classes (enforced, not advisory)
From [oikos/policy.yaml](../oikos/policy.yaml):
From [seeds/policy.yaml](../seeds/policy.yaml):
- **read_only** — status, logs, docs, inventory. Unattended.
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger.
@@ -69,7 +70,7 @@ Eight domains — physical, compute, network, storage, software,
identity & access, operations, external — cover everything in the lab;
entities are connected by typed edges (`hosts`, `provides`, `mounts`,
`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …)
defined in [oikos/ontology.yaml](../oikos/ontology.yaml). Rule of
defined in [seeds/ontology.yaml](../seeds/ontology.yaml). Rule of
completeness: **if it can break, be changed, or hold data, it has an
entity and edges.** Blast-radius questions ("what breaks if strong goes
down?") are graph walks, not doc archaeology.
@@ -80,9 +81,9 @@ stored as `state:` in inventory (absent = active). Destroyed nodes live in
the `archaeology:` section. Each transition is a runbook checklist;
deprecation completes only when inbound edges reach zero.
Generated views: [infrastructure/topology.md](../knowledge/wiki/infrastructure/topology.md)
(Mermaid, regenerated from inventory) and the live, clickable version at
`oikos.hubris.network/graph` once the Console is deployed.
Generated views: the live topology graph at `oikos.hubris.network/graph`
via the API's `/api/v1/graph` endpoint, and the Mermaid export at
`GET /api/v1/graph?format=mermaid`.
## Conventions carried forward
@@ -95,133 +96,95 @@ Generated views: [infrastructure/topology.md](../knowledge/wiki/infrastructure/t
- Agents are terse ([caveman.md](shared/caveman.md)), verify claims, and fix
collateral drift when found.
## Build status (30-day roadmap, started 2026-07-05)
## Build status (Go rewrite — deployed 2026-07-07)
- **Week 1**: policy, ontology, service contract, archaeology, topology
generator, this brief. Shipped.
- **Week 2**: context cards, `homelab service <name> …`, change ledger,
`node relations`, runbooks. Shipped.
- **Week 3**: ops scheduler + state cache (`homelab service <name> health`
is cache-first, `--live` forces a probe), drift detectors, signal engine
(`homelab signal …`), decision classifier (`homelab decide …`), approval
engine (`homelab approval …` — shared-HMAC grants; Matrix delivery is
Hermes's existing `@dtoro:avispero` send path, not a new bot, see
`oikos/approve.py`), daily brief + weekly report (`oikos/report.py`).
Shipped, except: Prometheus is still `planned` (see
[plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md)) —
trend signals (disk-full prediction, temp creep) wait on that LXC; the
scheduler's disk check today is point-in-time only, and CPU/NVMe
temperature isn't probed at all yet (no confirmed sensor path on
hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness
drift checks are also deferred (see `oikos/drift.py` docstring).
- **Week 4**: Oikos Console v0 shipped — signals landing page, service
grid + detail, node/blast-radius view, live Mermaid graph, drift view,
approvals queue (approve/deny, destructive confirmation-phrase
enforced), daily/weekly reports. Server-rendered FastAPI + Jinja2, no
SPA build chain, tested end-to-end against live production data (see
`oikos/console/`). Deploys as a third webhook on `dtoro/Homelab-Docs`
(`/opt/oikos-console`, port :9831) — see
[oikos/console/deploy/README.md](../oikos/console/deploy/README.md) for
the Caddy route and Gitea webhook registration this repo can't do for
itself. Approval grants are now single-use (a second `check_grant` call
for the same request fails even within the TTL) and already exact-bound
to request id + entity + action.
**Not shipped as originally planned:** per-agent *age-key-signed*
request authentication — age has no signing primitive (it's an
encryption-only keypair format), so "age-key-signed" wasn't
buildable as stated. The real alternative (SSH-key signing via
`ssh-keygen -Y sign`/`-Y verify`, using each host's already-provisioned
SSH key) is real and buildable, but needs SSH public keys recorded in
inventory first — not there today. Moved to the 60/90-day backlog.
Authentik step-up re-auth on the approve/deny route is documented but
needs a live Authentik instance to configure — also backlog.
Docs pass done (this file, AGENTS.md, operations/commands.md); found
and fixed two more stale references while at it (DNS section still
pointed at destroyed LXC 124/dnsmasq instead of Technitium on 107, and
a `claudio-monitor` reference that's been deprecated since 2026-06-04).
The Oikos runtime was rewritten from Python to Go over 6 phases and is deployed
in Docker on mac-mini. See
[plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](../plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md)
for the full plan. The Python codebase has been removed; all functionality runs
in the Go binary.
### Real drift found while building Week 3 (unresolved, needs operator action)
**Phase 1 — Ontology + DB (DONE):**
- `migrations/` (001011): TimescaleDB hypertables, entity_status, CAGGs,
retention policies, knowledge entities with FTS. Forward-only, idempotent.
- `seeds/{ontology,inventory,policy,knowledge}.yaml`: DB-native bootstrap +
DR export. Knowledge seed contains 36 documents, 6 investigations, and 12
runbooks.
- `blast_radius()` SQL CTE, type hierarchy, abstract types, relationship
validation.
- Go packages: `internal/db/`, `internal/ontology/`, `internal/domain/`,
`internal/knowledge/`.
The drift detectors surfaced genuine, currently-true findings on first
run against production — recorded here rather than silently fixed, since
each is a `config_mutation`/`destructive`-class decision:
**Phase 2 — API (DONE):**
- Single binary `cmd/oikos` with `oikos api` serving REST (:8090) + MCP
on the same service layer. OpenAPI-first (`api/openapi.yaml`) with
oapi-codegen + chi. RFC 9457 problem+json errors. Cursor pagination,
If-Match/ETag optimistic concurrency, idempotency keys, SSE event stream,
OIDC JWT + static bearer auth, audit middleware.
- Go packages: `internal/httpapi/`, `internal/httpapi/gen/`.
- `republic-laptop` has no `age_pubkey:` in `inventory.yaml`, but its real
age key is granted on nearly every shared secret in `.sops.yaml`
(`age1vf8h7...`) — the enrollment write-back to inventory never
happened. Fix: `homelab client add republic-laptop --finalize-pubkey
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6`.
- `grimmory` has an `age_pubkey` in inventory but is missing from
`secrets/hello.yaml`'s recipient list — incomplete enrollment the
other direction. Fix: re-run `homelab client add grimmory
--finalize-pubkey <its key>`.
- `pve_id 131` exists live on hubris (`pct list`) with no inventory entry
— investigate before assuming it's a stale ID (see the Prometheus LXC
plan doc above, which flags this explicitly).
- Three `lifecycle-pve-id-reuse` info findings (100, 106, 107 each shared
between an active host and an archaeology entry) — expected/benign ID
reuse after destroy, no action needed.
**Phase 3 — Control loop (DONE):**
- Scheduler (`oikos scheduler`): check_defs runner, signal dedup/flap
suppression, entity_status. HTTP, TCP, disk, cert-expiry probes.
- Actuator: SSH skill procedure execution with context-aware timeouts,
circuit breaker, retry budgets, error classification.
- Learning engine: hourly pattern extraction, Wilson confidence bounds,
anomaly detection, skills with validated patterns.
- Notifier: Matrix badge delivery, approval token generation (HMAC,
single-use, hashed), DB rendezvous pattern.
- Policy classifier: risk class determination, autonomy routing,
blast-radius computation, kill-switch support.
- Go packages: `internal/scheduler/`, `internal/actuator/`,
`internal/learning/`, `internal/notifier/`, `internal/policy/`.
## 60/90-day backlog
**Phase 4 — Agent / Nomos (DONE):**
- Standalone Nomos MCP client binary (`cmd/nomos`) with gateway mode
(:8092). Structured queries + natural-language routing to 15 MCP tools.
Agent activity logging on every tool call. No SSH keys.
- `nomos/` directory with config, SOUL.md, homelab-ops skill.
- Nomos Docker service in `docker-compose.yml` (profile: full).
- Go packages: `cmd/nomos/`, `compose/nomos/`.
Derived from gaps observed while building the 30-day roadmap, not
guesswork. Roughly ordered by what unblocks the most:
**Phase 5 — Secrets / Infisical (DONE):**
- `internal/secrets/`: backend abstraction (Manager) with primary
(Infisical) and fallback (SOPS) backends. Machine identities via
UniversalAuth. In-memory cache with TTL.
- `oikos secret` CLI: list, migrate (SOPS → Infisical), export-sops
(DR fallback). Infisical SDK v0.8.0.
- Rotation runbook at `secrets/rotation.md`.
- Docker compose: `infisical` + `redis` services (profile: infisical).
- **Fix the oikos-console deploy webhook's signature mismatch.** Console
is live on apps (105) via a manual `deploy.sh` run, but Gitea webhook
14's deliveries all 403 with a signature mismatch for a cause not yet
found — the secret is confirmed synced correctly on both sides
(rotated once already to rule out drift). Until fixed, `git push`
doesn't auto-redeploy the console the way it does for homelab-mcp/
secrets-issuance; re-run `deploy.sh` on apps manually after changes.
See [oikos/console/deploy/README.md](../oikos/console/deploy/README.md).
- **SSH-key-signed approval requests.** Replaces the design note in
Week 4: age keys can't sign (encryption-only format), so per-agent
request authentication needs `ssh-keygen -Y sign`/`-Y verify` against
each host's existing SSH key. Blocked on a schema gap: inventory
doesn't record SSH public keys today, only ports/users. First step is
populating that field on enrollment, then wiring `oikos/approve.py` to
require and verify a signature over the request payload.
- **Authentik step-up re-auth** on the Console's `/approvals` POST route
— needs a live Authentik `PromptStage`/reauth flow scoped to that path;
not configurable without a running instance to test against.
- **Prometheus provisioning** (see
[plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md))
— unblocks trend signals (disk-full prediction, temp creep) and real
sparklines in the Console; investigate the undocumented `pve_id 131`
on hubris first.
- **CPU/NVMe temperature probing** in the scheduler — needs a confirmed
sensor path on hubris and strong (lm-sensors vs vendor tool) before a
real check can be written; guessing one risks a probe that silently
never fires.
- **DNS-vs-inventory drift check** — compare Technitium zone records
against `services.*.url`/`public_host`; not implemented (`oikos/drift.py`
has no Technitium API wiring yet).
- **Generic tracked-config-cleanliness drift check** — today only caddy's
`/etc/caddy` git-checkout path is hardcoded in `oikos/drift.py`; every
other service with a `config_repo` needs its local checkout path
recorded (a `mutation_path`-style field, same gap Week 1's service
contract flagged but didn't backfill) before this generalizes.
- **Per-service policy overrides** (`oikos/policy.yaml`
`service_overrides`) — schema is ready (caddy/dns already use it);
populate more as specific services turn out to need non-default risk
classes.
- **Incident timeline generator** — stitch ledger + signal history into
a single narrative for `knowledge/sources/investigations/` entries instead of writing
them by hand.
- **Secret access audit** — who-can-decrypt-what report from
`.sops.yaml` + inventory `age_pubkey`s, extending what
`oikos/drift.py`'s SOPS check already partially does.
- **Restore drills** — exercise `backs-up-to` (once populated) by
actually restoring from a backup target on a schedule, not just
checking freshness.
- **Multi-agent delegation model** — more than one agent acting
concurrently; needs the ledger's `agent` field to carry real identity
(age pubkey, not just hostname) consistently, which it mostly does
already but hasn't been stress-tested with concurrent writers.
- **Grafana** — only if the Console's own Prometheus-backed sparklines
turn out to be insufficient once Prometheus ships.
- **"Generalize later" extraction** — the original decision was personal-
first, generalize-later (see Week 1). Once patterns stabilize, extract
a config-driven Oikos core with no `hubris.network`/`hubris`/`strong`
hardcoding, so it's installable on a different homelab.
**Phase 6 — Deploy + cutover (DONE, pending production cutover):**
- CI pipeline: `.gitea/workflows/ci.yml` (Gitea Actions — build, vet,
lint, test, docker build).
- Deploy: `scripts/deploy.sh` (git pull → docker build → compose up →
health check), SHA-tagged images, rolling restart.
- Caddy config: `compose/caddy/Caddyfile.oikos` (oikos/mcp/nomos →
mac-mini mesh :8090/:8092).
- Watchdog: `scripts/watchdog.sh` (2min cron, Matrix alert on failure).
- Verification: `scripts/verify-phase6.sh` (14/14 checks pass).
- Rollback: `scripts/rollback.sh` (checkout SHA + pg_restore).
- Cutover checklist: `scripts/cutover-checklist.md`.
**Current deployment:**
- **Production**: Docker stack on mac-mini (`--profile full`: postgres, api,
scheduler, notifier, nomos). Deployed 2026-07-07 with full knowledge seed.
The Python MCP server and secrets-issuance on apps/105 have been stopped
(see `scripts/cutover-checklist.md`).
## Python-era backlog (superseded)
The original 30-day roadmap (Python, shipped 2026-06/07) delivered:
context cards, change ledger, node relations, runbooks, ops scheduler,
drift detectors, signal engine, classifier, approval engine, and the
FastAPI+Jinja2 Oikos Console. All of these have been re-implemented in
the Go rewrite. The backlog items below that referenced Python paths
(`oikos/approve.py`, `oikos/drift.py`, `oikos/console/`) are now addressed
by the Go equivalents listed above.
Outstanding from the Python era (not yet in Go):
- Prometheus provisioning (see [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md))
- CPU/NVMe temperature probing (blocked on sensor path discovery)
- SSH-key-signed approval requests (blocked on inventory schema)
- Multi-agent delegation (blocked on ledger identity field)
- Restore drills on a schedule

185
.agents/dev/CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,185 @@
# Agent developer guide
Instructions for AI agents working on the Oikos codebase. Read this after
[AGENTS.md](../../AGENTS.md) and [OIKOS.md](../OIKOS.md). Human developers:
see [CONTRIBUTING.md](../../CONTRIBUTING.md) for a human-friendly version.
## Codebase map
```
cmd/oikos/main.go Entry point. Subcommands: api, scheduler, notifier, migrate,
seed, export, secret, all
cmd/nomos/main.go Nomos MCP client gateway (standalone binary, formerly Hermes)
cmd/webhook/main.go Gitea deploy-webhook receiver (push-to-deploy on mac-mini)
internal/httpapi/ REST + MCP server. Chi router. OpenAPI-generated types from
internal/httpapi/gen/api.gen.go. Strict server in impl.go.
internal/mcp/ MCP tool implementations (get_entity, search_knowledge, etc.)
internal/db/ Connection pool (pool.go), seed ingestion (seed.go), DB→YAML
export (export.go), type hierarchy (typetree.go)
internal/db/queries/ SQL query files → sqlc generates internal/db/sqlcgen/
internal/scheduler/ Observe loop: probes, signals, check_defs
internal/actuator/ SSH execution with circuit breaker + retry
internal/learning/ Pattern extraction, anomaly detection
internal/notifier/ Matrix notification + approval token generation
internal/policy/ Risk classifier (read policy.yaml → classify action)
internal/secrets/ Backend abstraction: Infisical (primary) + SOPS (fallback)
internal/domain/ Core types: entities, approvals, executions, signals, patterns
internal/ontology/ Type hierarchy validation, relationship checks
internal/knowledge/ Knowledge YAML seed ingestion
internal/config/ Config loading from env vars
web/ Control-room SPA (Svelte 5) — standalone static build, not
embedded in the oikos binary (plans/2026-07-12-wails-desktop-app.md)
api/openapi.yaml REST API contract. Source of truth for endpoints.
api/codegen.yaml oapi-codegen config → generates internal/httpapi/gen/
migrations/ Forward-only SQL. Format: NNN_name.up.sql. No down migrations.
seeds/ Bootstrap YAML. ontology.yaml, inventory.yaml, policy.yaml,
knowledge.yaml. Regenerated from DB via oikos export.
compose/ Dockerfiles. oikos/ (2-stage, Go only — SPA is built/deployed
separately), nomos/ (distroless).
Caddy config at compose/caddy/Caddyfile.oikos.
scripts/ Deploy, rollback, watchdog, verification, cutover checklist.
checks/ Host health-check scripts run over SSH by the scheduler.
tools/ Client auto-setup scripts (checks).
nomos/ Nomos config.yaml, SOUL.md, skills.
.agents/ Agent instruction files, domains, shared conventions, skills.
plans/ Design documents. active/ + done/.
docs/adr/ Architecture decision records. Numbered, prefix-sorted.
```
## Development loop
```bash
# Start dependencies
make dev
# Generate code after API/SQL changes
make generate
# Build
make build
# Run tests
make test # all unit tests
make test-db # integration tests (needs compose Postgres)
# Lint
make lint
# CI drift guard (run before commit)
make generate-check
```
## Adding a feature or phase
Oikos features follow a phase model (read [OIKOS.md](../OIKOS.md) for the
current phase status). To add a new capability:
1. **ADR first.** Write an architecture decision record in `docs/adr/` with
the next sequence number. Document the decision, context, alternatives
considered, and consequences.
2. **Plan.** If the change is non-trivial, create a plan in `plans/` following
the template in [page-templates.md](../shared/page-templates.md).
3. **Schema.** If the feature needs new DB tables, write a forward-only
migration in `migrations/`. Use `IF NOT EXISTS` for idempotency.
4. **API.** If the feature exposes endpoints, define them in
`api/openapi.yaml` first, then run `make generate`, then implement.
5. **Domain.** Add types to `internal/domain/` before adding logic.
6. **Tests.** Write tests alongside implementation. Integration tests go in
`*_test.go` in the relevant package, using the compose Postgres.
7. **Policy.** If the feature introduces new mutation types, update
`seeds/policy.yaml` and the classifier in `internal/policy/`.
8. **Run `make generate-check`** before commit to ensure generated code is
current.
## SQL conventions
- Queries live in `internal/db/queries/*.sql` with `-- name: FuncName :exec`
annotations for sqlc
- Use `pgx/v5` driver. UUIDs use `pgtype.UUID`, timestamps use `time.Time`
- CTEs for graph traversals (blast radius, dependency chains)
- CAGGs and retention policies for TimescaleDB hypertables
- FTS via `tsvector` + `tsquery` for knowledge search (migration 011)
## OpenAPI codegen
- Config: `api/codegen.yaml`. Uses `oapi-codegen/v2` with Chi server template
- Generated output: `internal/httpapi/gen/api.gen.go` — never hand-edit
- Strict server interface: `api.gen.go` generates the `StrictServerInterface`;
implement it in `internal/httpapi/impl.go`
- Problem+JSON errors via `internal/httpapi/problem.go` — RFC 9457 format
- Cursor pagination, If-Match/ETag, idempotency keys, SSE streaming
## Testing philosophy
- **Race detector always on.** `make test` runs `go test -race -cover ./...`
- **Integration tests** use the compose Postgres. Run with `make test-db`.
Each test creates + tears down its own schema namespace.
- **Coverage gates** in CI: policy + learning ≥ 80%, others ≥ 60%
- Tests use `testing.T` directly, no assertion library
- Table-driven tests for validation and classification logic
## Migration rules
- Forward-only. No down migrations (ADR 0008)
- Idempotent: use `IF NOT EXISTS`, `DO $$ BEGIN ... END $$` blocks
- Sequence numbers are sequential integers (001, 002, ...)
- Each migration file is `NNN_name.up.sql`
- Migrations are embedded in the binary via `migrations/embed.go`
## Seed files
- `seeds/ontology.yaml` — entity types, relationship types, lifecycles
(validated against schema in `internal/ontology/`)
- `seeds/inventory.yaml` — hosts, services, entities (the topology)
- `seeds/policy.yaml` — risk classes, approval rules, autonomy settings
- `seeds/knowledge.yaml` — documents, investigations, runbooks (DB is source
of truth; this file is the DR export)
- After DB changes via the API, run `make export` to regenerate seeds
## Secrets handling
- No secrets in code, config, or commits
- Dev secrets in `.env` (gitignored)
- Primary: Infisical (`internal/secrets/infisical.go`)
- Fallback: SOPS + age (`internal/secrets/sops.go`)
- Backend interface: `internal/secrets/backend.go`
- Machine identities via Infisical UniversalAuth
- In-memory cache with TTL for performance
## Staging and deployment
- CI pipeline: `.gitea/workflows/ci.yml` — lint, vet, vulncheck, test, docker build
- Deploy: `scripts/deploy.sh` — git pull → docker build → compose up → health check
- Watchdog: `scripts/watchdog.sh` — 2-minute cron, Matrix alert on failure
- Rollback: `scripts/rollback.sh` — checkout SHA + pg_restore
- Cutover checklist: `scripts/cutover-checklist.md`
## Writing conventions
Apply [writing-style.md](../shared/writing-style.md) for all committed prose.
Terse, reference-style, no marketing vocabulary. Code comments explain intent
and trade-offs, not mechanics.
Apply [caveman.md](../shared/caveman.md) for agent communication. The caveman
standard applies to agent *chat responses*, not committed documentation.
## Skills
Agent skills live under `.agents/skills/<name>/SKILL.md`. Each skill has a
frontmatter description that tools match against tasks. To add a skill:
1. Create `.agents/skills/<name>/SKILL.md`
2. Include frontmatter with description field
3. Document the procedure following the runbook template
4. Reference relevant files, commands, and policy classes
Skills that require code (e.g. linting) may include companion scripts in the
same directory.
## When in doubt
- Query MCP tools first (search_knowledge, get_entity)
- Read the relevant ADR in `docs/adr/`
- Grep the codebase: `rg <symbol> internal/`
- Check `plans/` for in-progress work that may conflict
- Classify any new mutation against `seeds/policy.yaml` before suggesting it

View File

@@ -17,18 +17,18 @@ fixed paths, so the wiki reorganization never moves it.
| Layer | Location | Consumed by |
|-------|----------|-------------|
| Substrate — source of truth | `inventory.yaml` (root) | MCP server, `homelab` CLI, `oikos/` scheduler/drift/relations/gen-topology |
| Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` |
| Substrate — generated host records | `inventory.yaml` (root) | Go `internal/mcp/` server, `bin/homelab`; the single source of truth |
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
| Narrative — synthesized wiki | `archive/knowledge/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
## Wiki pages
- **Node pages** (`knowledge/wiki/containers/<id>-<name>.md`, `.../vms/<id>-<name>.md`,
- **Node pages** (`archive/knowledge/containers/<id>-<name>.md`, `.../vms/<id>-<name>.md`,
`.../hosts/<name>.md`) follow the container/host template in
[page-templates.md](../../shared/page-templates.md): opening definition, `## At a glance`,
`## Role`, service/port map, storage, auto-deploy, `## Related`, `## Changelog`.
- **Cross-cutting pages** (`knowledge/wiki/infrastructure/<topic>.md`) follow the cross-cutting
- **Cross-cutting pages** (`archive/knowledge/infrastructure/<topic>.md`) follow the cross-cutting
template: `## Why`, `## Components`, `## How to apply`, `## Gotchas`, `## Related`, `## Changelog`.
- Each `inventory.yaml` host entry carries a `doc_page:` field pointing at its narrative page.
Changing where a page lives means updating that field (read by `bin/homelab`).

View File

@@ -40,7 +40,7 @@ transition: "<from> -> <to>" # only for lifecycle runbooks
```
`risk_class` values and the lifecycle `transition` states must match
[`oikos/policy.yaml`](../../../oikos/policy.yaml) and [`oikos/ontology.yaml`](../../../oikos/ontology.yaml).
[`seeds/policy.yaml`](../../../seeds/policy.yaml) and [`seeds/ontology.yaml`](../../../seeds/ontology.yaml).
## Investigations

View File

@@ -1,365 +1,120 @@
# Agent enrollment — bootstrap a client into the homelab context system
# Agent enrollment — operational notes
This walks through enrolling a new machine (workstation, LXC, or VM) so it
joins the cross-client context system: a `/opt/homelab-context/` clone of
this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
**For the actual enrollment flow, see [CLIENTS.md](../../CLIENTS.md#enrollment)
— it's the current, authoritative version.** This page used to duplicate
that flow in more detail, describing a `homelab` CLI-based two-step
ceremony (`homelab client add` reserves an inventory slot → client
bootstraps → operator finalizes the pubkey). That CLI and that flow don't
exist anymore — enrollment today is one shot: `bootstrap.sh` calls
`POST /api/v1/clients/enroll` directly and gets back an age keypair +
Infisical identity in the same response. What's left here is the handful
of things that are still true and weren't already covered elsewhere.
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
> See [hermes-agent.md](hermes-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-hermes` flag.
Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.
## Prerequisites the client must satisfy
## Prerequisites
| Requirement | Why | How to check |
| --- | --- | --- |
| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| Hostname matches an entry in `inventory.yaml` | `EnrollClient` looks up the entity by slug derived from hostname; it must exist in `planned`/`provisioning` state. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |
| On the mesh (Netbird) **or** on the LAN | enrollment validates mesh IP against expected subnets | `netbird status` |
| `curl`, `jq`, `age`, `python3` | bootstrap preflight (`bootstrap.sh:100`) — auto-installed on Fedora/RHEL/Debian/Ubuntu/macOS if missing | `command -v curl jq age python3` |
| Can resolve `*.hubris.network` | bootstrap calls the Oikos API and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` |
### Hostname mismatch is the most common bootstrap failure
If the bootstrap exits with `no hosts/<name>.yaml in the repo`, the
hostname doesn't match any inventory entry. Two fixes:
If the entity for your hostname doesn't exist yet (in `planned` or
`provisioning` state), enrollment 4xxs. Two fixes:
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
(Linux) or System Preferences → Sharing (macOS), then re-run.
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
- **Rename the host** to match an existing planned entity:
`sudo hostnamectl set-hostname <inventory-name>` (Linux) or System
Preferences → Sharing (macOS), then re-run.
- **Add/rename the inventory entry**: edit `seeds/inventory.yaml`, ingest
via `oikos seed` (or the equivalent MCP/API entity-creation path), then
re-run bootstrap.
### Getting onto Netbird
### Networking prerequisites (Netbird, DNS, SSH key distribution)
Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit `<bootstrap-tier1>`). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`.
**Path B — interactive OIDC (default; recommended):**
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
**Path A — setup-key (headless/scripted onboarding):**
Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.
1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`.
2. **Setup Keys** → Create → set reusable + expiry → copy.
3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver):
```bash
sudo netbird up --setup-key <KEY> \
--management-url https://netbird.hubris.network \
--ssh-jwt-cache-ttl 86400
```
**Why we can't OIDC-login from the public internet (still open as a follow-up):**
`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124.
### DNS prerequisite
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:
- **LAN client**: set DNS to 192.168.8.180 (per-interface or
`/etc/resolv.conf`).
- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
point `*.hubris.network` at LXC 124.
- **Hack-fix anywhere**: append to `/etc/hosts`:
```
192.168.8.175 mcp.hubris.network secrets.hubris.network
192.168.8.175 git.hubris.network
```
(192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.)
If DNS isn't an option at all, override the URLs at bootstrap time:
```bash
sudo HOMELAB_GITEA_TOKEN=... \
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
bash /tmp/bootstrap.sh --with-mcp
```
## Install dependencies
Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script).
Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.
<details>
<summary>Manual recipes (Fedora / Debian / macOS)</summary>
```bash
# Fedora / RHEL / Nobara
sudo dnf install -y git python3-pyyaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
-o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
# Debian / Ubuntu
sudo apt update && sudo apt install -y git python3-yaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
-o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
# macOS
brew install git age sops
pip3 install pyyaml # if `python3 -c "import yaml"` fails
```
</details>
## Run the bootstrap
You need a Gitea read-only personal access token for the initial clone
(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml`
but a new client can't decrypt it before bootstrap — chicken-and-egg).
Ask the operator (or generate in Gitea: Settings → Applications → Generate
New Token → scope `read:repository`).
```bash
TOKEN=... # your Gitea PAT, scope read:repository
# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
curl -fsSL -u "dtoro:$TOKEN" \
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
-o /tmp/bootstrap.sh
# Run it.
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp
```
Flags:
| Flag | Effect |
| --- | --- |
| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user |
| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) |
| `--dry-run` | Prints actions without executing |
The bootstrap is idempotent: re-running on an enrolled client just
verifies state, re-issues the age key only if it doesn't match the
inventory pubkey, and refreshes the sync timer + symlinks.
## Verify
```bash
homelab whoami # prints hosts/$(hostname).yaml
homelab list # shows the full topology
homelab status # ping + HTTP-check across hosts/services
homelab secret hello # decrypt the bootstrap-test secret
systemctl list-timers homelab-context-sync.timer
# next run within ≤5 min
```
For Claude Code: start a new session — the `homelab` MCP server appears
in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
1 secrets-metadata).
## Post-bootstrap: SSH reachability
A new workstation must be reachable from other workstations and must be
able to reach every host by short hostname. Run these steps after the
bootstrap verify passes:
### 1. Enable SSH server
```bash
# macOS:
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
# Linux:
sudo systemctl enable --now sshd
```
### 2. Generate SSH key (if missing)
```bash
ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100
```
### 3. Publish pubkey to the repo
```bash
cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push
```
### 4. Deploy keys to all hosts
From any existing enrolled machine (hubris or another workstation):
```bash
ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
```
This adds the new workstation's pubkey to hubris and every running LXC.
### 5. Generate SSH config
```bash
homelab ssh-config --install
```
Verify:
```bash
ssh hubris hostname # should return "hubris" without password
ssh gitea hostname # should return "gitea" without password
ssh mac-mini hostname # should return "mac-mini" without password (workstation-to-workstation)
```
### 6. Add LAN IP to inventory (if on LAN)
If the workstation has a static or reserved LAN IP, add it to
`inventory.yaml`:
```yaml
hosts:
your-hostname:
lan_ip: 192.168.8.xxx
```
This gives it a primary LAN entry in the generated SSH config (faster
than the Netbird fallback). Commit + push, then:
```bash
cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install
```
Migrated to a runbook in the knowledge base — query
`search_knowledge("netbird mesh dns")` or `get_entity_knowledge`, or ask
Nomos. Covers: getting onto the Netbird mesh (interactive OIDC vs.
setup-key), why OIDC login can fail from off-mesh, split-horizon DNS
options, and distributing a new workstation's SSH pubkey across the fleet
via `ssh/deploy-keys.sh`.
## Claude Code permissions for fleet ops
By default Claude Code's auto-mode classifier asks for confirmation on every
ssh into the mesh. The bootstrap already installs the ssh ControlMaster block
so subsequent in-session sshes multiplex, but the *first* ssh of each session
still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns
by adding to `~/.claude/settings.json`:
ssh into the mesh. Pre-authorize the common fleet ssh pattern by adding to
`~/.claude/settings.json`:
```json
{
"permissions": {
"defaultMode": "auto",
"allow": [
"Bash(ssh -p 22022 *)",
"Bash(homelab *)"
"Bash(ssh -p 22022 *)"
]
}
}
```
The first rule covers any ssh to a mesh peer on the homelab netbird port; the
second covers all `homelab` CLI invocations. Both are scoped tight enough that
the classifier doesn't gate them but loose enough to handle the variety of
arguments.
This covers any ssh to a mesh peer on the homelab netbird port, scoped tight
enough that the classifier doesn't gate it but loose enough to handle the
variety of arguments.
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
visible to the classifier (it's not actually durable in 0.71.2, but the
ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md)
for context), drop a free-text rule into `autoMode.allow` describing the
authorization. Optional.
## Open questions (not verified against current architecture — don't
guess these from the old flow)
## Adding a new client to inventory
The old two-step ceremony had answers for these; the current one-shot
`/api/v1/clients/enroll` flow may handle them differently and this hasn't
been re-verified:
If the hostname you want isn't yet in inventory, enrollment is a two-step
ceremony driven from an existing enrolled client (e.g. hubris). The
`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3.
```bash
# 1. On hubris (or any existing client): add the inventory entry.
homelab client add my-new-machine
# Prompts for kind, os, netbird FQDN, role. Commits + pushes.
# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key).
# 3. On the new machine: install deps + run bootstrap (above).
# Bootstrap calls /issue, receives a fresh age keypair, and prints the
# public key for the operator to commit back to inventory.
# 4. On hubris: finalize the age public key.
homelab client add my-new-machine --finalize-pubkey age1...
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
```
## Granting a secret to a new client
Adding a client doesn't grant them every secret. Recipients are explicit
per file via `.sops.yaml` glob rules. To grant a client access to (say)
`secrets/hello.yaml`:
1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to
the matching `creation_rules` block.
2. Re-key the existing ciphertext for the new recipient list:
```bash
sops updatekeys -y secrets/hello.yaml
```
3. Commit + push. On the next sync (≤5 min), the client can decrypt.
## Removing a client
```bash
# From any existing client:
homelab client remove my-old-machine
```
This:
1. Removes the inventory entry and `hosts/my-old-machine.yaml`.
2. Runs `sops updatekeys -y` against every file in `secrets/` (operator
must first remove the pubkey from `.sops.yaml` rules).
3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to
shred the key file and add the hostname to the denylist.
4. Commits + pushes.
The CLI prints a follow-up checklist that the operator must do manually:
- Revoke the peer in the Netbird console (denies future mesh access).
- **Rotate any credentials whose ciphertext the removed client already
has on disk.** The age key revocation only protects *future*
ciphertext; what's already been pulled is still decryptable until the
underlying credential changes.
- Optional: `homelab nuke my-old-machine` SSHes in, shreds
`/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync.
- **Removing a client.** No current equivalent confirmed for the old
`homelab client remove` (inventory removal + secret re-keying + key
revocation). Likely maps to an entity lifecycle transition
(`.agents/skills/lifecycle-deprecate-node/` or `lifecycle-destroy-node/`)
but those skills reference the same dead CLI and need their own check.
- **Granting a secret to an already-enrolled client.** The old flow
hand-edited `.sops.yaml` `creation_rules` + `sops updatekeys`. Given
Infisical is now the primary secrets backend (SOPS is the DR fallback),
the current mechanism is probably Infisical-side, not a `.sops.yaml` edit
— not confirmed.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Enrollment 404s / entity not found | Hostname doesn't match a `planned`/`provisioning` inventory entry | See "Hostname mismatch" above |
| `gnutls_handshake() failed` / TLS errors reaching `*.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS instead of the LAN/mesh path | See the networking runbook (split-horizon DNS section) |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). |
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
## Changelog
### 2026-06-02 — SSH reachability post-bootstrap steps
Added a new "Post-bootstrap: SSH reachability" section covering SSH key
generation, pubkey publication, deployment to hosts, SSH config generation,
and LAN IP registration. New workstations enrolled via this doc will
automatically join the universal SSH mesh.
### 2026-07-12 — trimmed to current architecture
Removed everything describing the retired `homelab` CLI-based two-step
enrollment ceremony (now: `CLIENTS.md`'s one-shot flow), the Nous-Hermes/
Goose cross-link (that whole flow was removed the same day), and CLI-syntax
troubleshooting rows with no current equivalent. Migrated the still-true
Netbird/DNS/SSH-distribution content to a knowledge-base runbook rather
than duplicating it here. What's left is genuinely current or explicitly
flagged as unverified. Original ~365-line version is in git history
(`git log -- .agents/operations/agent-enrollment.md`) if any of the removed
detail turns out to still be needed.
### 2026-05-31 — cross-link to hermes-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
### 2026-06-02 — SSH reachability post-bootstrap steps
Added a section covering SSH key generation, pubkey publication,
deployment to hosts, SSH config generation, and LAN IP registration. New
workstations enrolled via this doc automatically join the SSH mesh.
(Superseded 2026-07-12 — migrated to the networking runbook.)
### 2026-05-31 — cross-link to nomos-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment. (Removed
2026-07-12 along with the rest of that flow.)
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
Added three rows to the troubleshooting table covering issues surfaced
during the netbird vanilla migration. (Migrated 2026-07-12 to the
networking runbook.)
### 2026-05-20 — initial page
Captures the enrollment flow validated during Phase 2 of the homelab

View File

@@ -1,6 +1,6 @@
# Operations cheatsheet
Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
## Proxmox CLI
@@ -8,13 +8,13 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
| --- | --- |
| `pct list` / `qm list` | List LXC containers / VMs |
| `pct config <id>` / `qm config <id>` | Container / VM config |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
| `pct enter <id>` | Shell into a container |
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
| `pvesm status` | Storage pools status |
| `pvesh get /nodes --output-format json` | Node summary as JSON |
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Nomos cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pveversion` | PVE version |
| `journalctl -u pve-cluster -n 100` | PVE service logs |
@@ -22,22 +22,22 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
## Reverse proxy
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
## DNS
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
- See [DNS](../../knowledge/wiki/infrastructure/dns.md).
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
## Web access
@@ -49,43 +49,45 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
- `ras-mc-ctl --errors` — full event log
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md))
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/archive/2026-04-21-hubris-crash-loop.md))
## Fleet apt operations
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
| Command | What it does |
| --- | --- |
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot``vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
**No current CLI equivalent.** `homelab apt-audit`/`apt-upgrade` (dpkg-state
audit, fanned-out apt upgrade with pre-upgrade snapshots) were part of the
retired Python `homelab` CLI and don't have a ported replacement — apt
patching today is ad hoc `run` MCP tool calls per host, without the
audit/snapshot/status wrapping this used to provide. If that wrapping is
still wanted, it needs to be rebuilt (e.g. as a runbook driving `run`, or a
new MCP tool) — see
[runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) for
the dpkg-interrupted recovery procedure specifically.
## Oikos (agent OS layer)
See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
See [OIKOS.md](../OIKOS.md) for the operating model. The `homelab` CLI this
section used to document is retired; the actual current interface is the
33 MCP tools cataloged in [AGENTS.md](../../AGENTS.md#3-the-mcp-server) plus
the REST API. Closest current equivalents for what used to live here:
| Command | What it does |
| Old `homelab` command | Current equivalent |
| --- | --- |
| `homelab service <name> explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history |
| `homelab node <name> relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius |
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
| `homelab decide <action> <entity>` | Decision classifier: risk × blast radius × confidence → auto-act or escalate |
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
| `homelab service <name> explain\|health\|docs\|log` | MCP `explain`, `get_service_status`, `tail_log`, `get_entity_knowledge` |
| `homelab node <name> relations` | MCP `get_blast_radius` |
| `homelab change preflight <service>` | MCP `preflight` |
| `homelab signal list\|ack\|resolve\|mute` | MCP `get_signal_history`, or REST `POST /api/v1/signals/{id}/ack\|resolve\|mute` (the control-room UI's Signals page wraps these) |
| `homelab approval request\|list\|reply\|check` | REST `GET/POST /api/v1/approvals*` (Matrix-delivered via the notifier, or the control-room UI's Operations page) |
| `homelab restart <service> --approval-id <id>` | MCP `run` (policy-gated — auto-executes if read-only/reversible_low, otherwise queues for the same Matrix/UI approval) |
| `homelab decide <action> <entity>` | No direct equivalent — classification now happens inline inside `run`/`request_execution`, not as a separate dry-run call |
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md).
There is no separately-deployed "Oikos Console" anymore — the control-room
SPA (`web/`) is the operator dashboard, served standalone (see
[plans/2026-07-12-wails-desktop-app.md](../../plans/2026-07-12-wails-desktop-app.md)).
## Related
- [Hubris host](../../knowledge/wiki/hosts/hubris.md)
- [Containers index](../../knowledge/wiki/containers/index.md)
- [DNS](../../knowledge/wiki/infrastructure/dns.md)
- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md)
- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)
- [Hubris host](../../archive/knowledge/hosts/hubris.md)
- [Containers index](../../archive/knowledge/containers/index.md)
- [DNS](../../archive/knowledge/infrastructure/dns.md)
- [Monitoring](../../archive/knowledge/infrastructure/monitoring.md)
- [Auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction

View File

@@ -1,210 +0,0 @@
# Hermes agent — Nous-Hermes-powered Goose sessions on a homelab client
Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
Llama variant) as a working terminal agent on a homelab client. Builds on top
of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md))
— this page covers only the Hermes-specific additions.
The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
- The chat loop, multi-turn history, and streaming
- The OpenRouter provider that routes to Nous Hermes
- The built-in `developer` extension (shell + file editor — same surface Claude
Code has)
- A remote MCP extension pointed at `mcp.hubris.network` for read-only
homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
`.goosehints` so it's injected into the system prompt on every session.
## Prerequisites
| Requirement | How |
| --- | --- |
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) |
| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` |
## Onboarding flow
```bash
# 1. On hubris (or any enrolled client): reserve the inventory entry.
homelab client add new-machine
# 2. Join new-machine to Netbird (setup-key or OIDC).
# 3. On new-machine: bootstrap with --with-hermes.
TOKEN=... # gitea PAT, read:repository
curl -fsSL -u "dtoro:$TOKEN" \
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
-o /tmp/bootstrap.sh
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-hermes
# 4. Back on hubris: finalize the age pubkey AND grant the Hermes secret.
homelab client add new-machine \
--finalize-pubkey age1... \
--with-hermes
# 5. Wait ≤5 min for sync, then on new-machine:
hermes "what LXCs are running?"
```
The bootstrap `--with-hermes` flag does five things, all idempotent:
1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
(upstream installer) and symlinks `/usr/local/bin/goose` to it.
2. Symlinks `/opt/homelab-context/bin/hermes``/usr/local/bin/hermes`.
3. Symlinks `/opt/homelab-context/HERMES.md``/root/HERMES.md` (Linux) or
`/etc/HERMES.md` (macOS) for `cat`-as-operator convenience.
4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
extensions (preserves any keys the operator added by hand).
5. Symlinks `~/.config/goose/.goosehints` → HERMES.md, so the persona is
injected as the system prompt on every session.
## Seeding the OpenRouter key
The first time anyone enrolls with `--with-hermes`, the encrypted file
`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
existing recipient):
```bash
sops secrets/openrouter-api-key.yaml
# editor opens; replace api_key value with the real sk-or-... key, save, close.
git -C /opt/homelab-context add secrets/openrouter-api-key.yaml
git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key'
git -C /opt/homelab-context push
```
Until this step happens, `hermes …` exits with `openrouter-api-key.yaml still
contains the placeholder`. Subsequent enrollees get the real key automatically
via `--with-hermes` (which adds them as a sops recipient on
`secrets/openrouter-api-key.yaml`).
## Granting the OpenRouter key to an already-enrolled host
If a host was enrolled without `--with-hermes` and you want to add it later:
```bash
# On hubris:
PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-hermes
```
`--finalize-pubkey` is required by the existing flow even when the pubkey is
unchanged — it's also the trigger that runs the sops grant.
After ≤5 min sync the host can decrypt the key. Bootstrap doesn't need to
re-run; only the secret recipient list changed.
## Verifying
```bash
homelab whoami # standard enrollment OK
homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
which goose && which hermes # binaries present
goose info -v # provider/model wiring sane
hermes "what LXCs are running?" # interactive Goose session
# Non-interactive smoke test:
echo "List the homelab MCP tools you have available" | hermes
```
## Configuration
The bootstrap-managed keys in `~/.config/goose/config.yaml`:
```yaml
GOOSE_PROVIDER: openrouter
GOOSE_MODEL: deepseek/deepseek-v4-flash
GOOSE_MODE: smart_approve # asks before destructive tool calls
extensions:
developer:
type: builtin
bundled: true
enabled: true
name: developer
timeout: 300
homelab:
type: streamable_http
enabled: true
name: homelab
uri: https://mcp.hubris.network/mcp
timeout: 60
```
Override via env on a single bootstrap run:
```bash
HOMELAB_HERMES_MODEL=nousresearch/hermes-3-llama-3.1-405b \
HOMELAB_HERMES_MCP_URI=https://mcp.hubris.network/mcp \
sudo bash /tmp/bootstrap.sh --with-hermes
```
Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
preserved across re-bootstraps — the merge only overwrites the keys it manages.
## Tool permissions
`GOOSE_MODE: smart_approve` is the bootstrap default: Goose runs read-only
shell commands without prompting and asks for confirmation before destructive
ones. To make the agent fully unattended (e.g. for scheduled jobs), set
`GOOSE_MODE: auto` in `~/.config/goose/config.yaml`. To require confirmation on
every tool call, use `approve`. See
[goose-permissions](https://goose-docs.ai/docs/guides/managing-tools/goose-permissions/).
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py``mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
## Cross-references
- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the
Hermes flow assumes is done.
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
session start (via `~/.config/goose/.goosehints`).
- [`bin/hermes`](../../bin/hermes) — the wrapper that decrypts the OpenRouter key
and execs `goose session`.
- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-hermes` flag's install block.
## Follow-ups
1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
(`"SSE transport is no longer supported - kept only for config file
compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
server at `mcp/server.py:336` still calls `mcp.run(transport="sse")`. Until
that's changed, the `homelab` MCP extension in Goose will fail to connect.
The developer extension (shell + edit) covers most ops without it; this is
a polish item, not a blocker.
2. **Per-host OpenRouter keys** for billing attribution. Today all Hermes
hosts share one key.
3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
directly — OpenRouter periodically rotates the underlying weights.
4. **Local-inference fallback** (ollama / vllm) once the homelab has a GPU
node. The wrapper, persona, and MCP wiring stay unchanged; only
`GOOSE_PROVIDER`/`GOOSE_MODEL` change.
7. **Caveman auto-setup via post-pull hook.** The sync timer now calls
`tools/post-pull.sh`, which runs any `tools/*.setup.sh` after git pull.
Currently this auto-installs the Caveman npm package, wrapper scripts, and
compact output templates on all agent hosts (*token efficiency*).
## Changelog
### 2026-06-01 — caveman + post-pull auto-setup
Added `tools/post-pull.sh` sync hook that auto-runs `tools/*.setup.sh`
after every git pull. First user: `tools/setup-caveman.sh` installed Caveman
templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token-
efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
Also created `tools/caveman/` with the wrapper script, JS renderer, and
templates — the canonical source for all agent hosts.
Captures the Hermes-on-Goose onboarding flow added in the same commit as
`bootstrap.sh --with-hermes`, `bin/hermes`, the sops rule for
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-hermes`
extension. MCP streamable_http migration is queued as follow-up #1.

View File

@@ -30,4 +30,4 @@ Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level p
---
Source: https://github.com/JuliusBrussee/caveman
Copy to `~/.hermes/skills/` for Hermes Agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.
Copy to `~/.nomos/skills/` for Nomos agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.

View File

@@ -1,12 +1,12 @@
# LLM Wiki — the documentation contract
How the narrative documentation in this repo is organized. The pattern is borrowed from the
`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top
`sources / wiki / index / log` model: a durable synthesized layer (`archive/knowledge/`) built on top
of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an
append-only operations log.
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
generated `hosts/*.yaml`, `oikos/`, `mcp/`, `secrets/`, `bin/` — is not part of the wiki and never
`secrets/`, `scripts/`, `bin/` — is not part of the wiki and never
moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for the split.
## Layers
@@ -14,7 +14,7 @@ moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for t
- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
Read them; do not rewrite them into other sources.
- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per
- **Wiki** (`archive/knowledge/`) is the synthesized, authoritative current-state layer: one page per
node (`containers/`, `vms/`, host narratives) and per cross-cutting system (`infrastructure/`). A
reader understands the topic from the wiki page without reading the sources.
- **Index** (`index.md` / folder `README.md`) is a pure listing — every page in scope with a

View File

@@ -160,7 +160,7 @@ session.** A change that touches a container page must also update:
- The Caddy page site list (if the change affects `*.hubris.network` routing)
- The DNS / ingress infrastructure pages (if the change affects routing)
- The `hosts/{hubris,strong}.md` host page (if container count changes)
- The `inventory.yaml` host entry (source of truth for the `hosts/*.yaml` generation)
- The `inventory.yaml` host entry (single source of truth)
- The `infrastructure/topology.md` (generated from inventory, but regen if needed)
The pattern of updating only one page and leaving stale references on others

View File

@@ -36,7 +36,7 @@ Every doc-level page follows the same shape so a reader scans it in one pass.
1. **One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`.
2. **Opening definition.** First paragraph, 13 sentences, says what the thing is. No motivation, no marketing, no setup.
3. **Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md).
4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (`get_changelog` in `mcp/server.py`); keep the `### YYYY-MM-DD — title` shape.
4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (Go MCP `get_changelog` in `internal/mcp/server.go`); keep the `### YYYY-MM-DD — title` shape.
5. **Related links** only at the bottom, only when a reference cannot be woven inline.
## Section indexes (folder READMEs)
@@ -58,7 +58,7 @@ duplicated prose, no narrative between the intro and the table.
## Diagrams
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it.
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it. (Go DB-native topology generation planned.)
- ASCII box diagrams are fine for small shape diagrams; keep them to one screen.
## Sourcing and cross-references

View File

@@ -2,42 +2,49 @@
name: client-enrollment
risk_class: config_mutation
inputs: [hostname, kind, role]
verification: "homelab doctor (on the new client)"
verification: "MCP whoami(hostname) shows the entity active"
docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
---
# Client enrollment
Goal: bring a new host (workstation, LXC, VM) into inventory and the
secrets model, with mesh membership only where it's actually needed.
This wraps the existing `homelab client add` flow — see
[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for
the full walkthrough; this runbook is the risk/lifecycle framing.
secrets model, with mesh membership only where it's actually needed. See
[CLIENTS.md](../../../CLIENTS.md#enrollment) for the actual current
flow and [operations/agent-enrollment.md](../../operations/agent-enrollment.md)
for operational notes; this runbook is the risk/lifecycle framing.
1. On any enrolled client: `homelab client add <hostname>` — appends a
`hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned`
`provisioning`, per [oikos/ontology.yaml](../../../oikos/ontology.yaml)),
commits + pushes.
1. The entity must exist in `planned`/`provisioning` state before the new
host can self-enroll — add a `hosts.<name>:` block to
`seeds/inventory.yaml` and `oikos seed` to ingest it (lifecycle
`planned``provisioning`, per
[seeds/ontology.yaml](../../../seeds/ontology.yaml)).
2. Netbird join is **optional, not a required step** — only needed for
hosts that must be reachable off-LAN (workstations that roam, e.g.
`republic-laptop`, `mac-mini`). A node reachable on the household LAN
(192.168.8.0/24 — most LXCs/VMs) doesn't need it: it's already
reachable directly, and off-LAN clients reach it too via hubris's
routed `192.168.8.0/24` Netbird network resource. Skip this step for
LAN-only nodes; do it (out-of-band, console or setup key) only for
hosts that need independent off-LAN reachability.
3. On the new host: run `bootstrap.sh` (add `--with-hermes` to also
enroll the Hermes agent). This provisions `/etc/age/key.txt`, the
sync timer, and prints an age pubkey.
4. Back on an enrolled client: `homelab client add <hostname>
--finalize-pubkey <age1...>` — sets `age_pubkey`, grants shared
secrets, re-keys SOPS, commits + pushes. This is the
`provisioning → active` transition.
5. Verify: `homelab doctor` on the new client should show all checks
green (clone, sync timer, age key, CLI symlink, MCP reachable).
`mac-mini`). A node reachable on the household LAN (192.168.8.0/24 —
most LXCs/VMs) doesn't need it. Skip for LAN-only nodes; do it
(out-of-band, console or setup key) only for hosts that need
independent off-LAN reachability.
3. On the new host: run `bootstrap.sh`. This calls
`POST /api/v1/clients/enroll`, which validates the entity exists and
the mesh IP is in an expected subnet, then returns an age keypair and
Infisical machine identity in one response — provisions
`/etc/age/key.txt`, `/etc/infisical/identity`, and the context poller.
4. **Known gap, confirmed 2026-07-12: `provisioning → active` has no
working path.** `EnrollClient` (`internal/httpapi/impl.go`) sets the
entity's state to `provisioning`, never `active`. `bootstrap.sh` prints
`POST /api/v1/clients/ws:$HNAME/activate` as the next step, but that
route doesn't exist — `api/openapi.yaml` only has `/clients/enroll`,
`/clients/{slug}/context`, `/clients/{slug}/secrets`. Until this is
fixed (add the route, or use the generic entity PATCH to flip `state`),
a freshly-enrolled client is stuck in `provisioning` — MCP `preflight`
and policy's `lifecycle_overrides` for `provisioning` still apply, but
nothing transitions it onward automatically.
5. Verify: MCP `whoami(hostname)` shows the entity in `active` state with
its peers and health.
Docs-update checklist: if the new host is an LXC/VM, add its narrative
page under `containers/` or `vms/` and set `doc_page` in its inventory
entry (host-level cards don't have a `doc_page` field yet — services do;
narrative pages are still found via the generated `see_also` in
`hosts/<name>.yaml`).
`inventory.yaml`).

View File

@@ -2,7 +2,7 @@
name: config-change-deploy
risk_class: config_mutation
inputs: [service_name, change_description]
verification: "curl -sf <service_url> (or homelab service <name> health)"
verification: "curl -sf <service_url> (or MCP get_service_status)"
docs_update_checklist: [doc_page, changelog]
---
@@ -11,23 +11,22 @@ docs_update_checklist: [doc_page, changelog]
Goal: change a tracked config repo (Caddy, Gitea customizations, an app's
own repo) and get it live, safely.
1. `homelab change preflight <service>` — current health, the service's
`config_repo`, its risk class, and the verification command to run
after. If risk class requires approval (`config_mutation` or
`destructive`), stop and get operator sign-off before editing — see
`oikos/policy.yaml`.
1. MCP `preflight` — current health, the service's `config_repo`, its
risk class, and the verification command to run after. If risk class
requires approval (`config_mutation` or `destructive`), stop and get
operator sign-off before editing — see `seeds/policy.yaml`.
2. Clone/pull the `config_repo` (never edit the backend's working tree
directly — tracked configs change by commit + push, per
[OIKOS.md](../../OIKOS.md) conventions).
3. Make the change, commit, push to `main`.
4. The Gitea webhook fires the deploy pipeline for that repo (see
[infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for
[infrastructure/auto-deploy.md](../../../archive/knowledge/infrastructure/auto-deploy.md) for
the exact receiver/reload for this service).
5. Run the preflight's verification command. If it fails, check
`homelab service <name> log` for the reload/restart error.
6. Record the change: once `oikos/ledger.py` is wired into deploy tooling
(Week 3), this is automatic; until then, note the change and outcome
in the relevant investigation/plan doc.
MCP `tail_log` for the reload/restart error.
6. No manual record-keeping step needed — mutations made through the API
(e.g. via the `run` MCP tool) are recorded automatically in the
`audit_log` table.
Docs-update checklist: update the service's `doc_page` if the change
alters its behavior, ingress route, or ownership; add a changelog entry

View File

@@ -15,11 +15,11 @@ links. Prose-voice rules are not machine-checkable — those stay a review respo
Run from the repo root:
python3 .agents/skills/docs-lint/lint.py # default: knowledge/ .agents/ operations/ investigations/ plans/
python3 .agents/skills/docs-lint/lint.py knowledge/wiki/containers/104-gitea.md
python3 .agents/skills/docs-lint/lint.py archive/knowledge/containers/104-gitea.md
Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary
list mirrors `writing-style.md`; update both together if the standard changes.
> **Known baseline.** `knowledge/wiki/containers/101-jellyfin.md` links into a sibling repo
> **Known baseline.** `archive/knowledge/archive/knowledge/containers/101-jellyfin.md` links into a sibling repo
> (`devops/homelab-authentik-admin`) that this checkout does not contain — expected, not a bug.
> Any other broken link is a real regression; investigate before dismissing it as baseline noise.

View File

@@ -1,16 +1,17 @@
#!/usr/bin/env python3
"""Lint committed docs against .agents/shared/writing-style.md.
Checks two mechanical rules:
Checks:
1. Banned vocabulary (significance puffers, analytical verbs, poetic nouns,
promotional adjectives, opening crutches).
2. Broken relative markdown links.
3. Plan status consistency (status vs location vs index).
Prose-voice rules are not machine-checkable; this covers the parts that are.
Run from the repo root: python3 .agents/skills/docs-lint/lint.py [paths...]
Exit 1 if any violation is found.
"""
import os, re, sys
import os, re, sys, glob
BANNED = [
"pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament",
@@ -33,9 +34,125 @@ def iter_md(paths):
if f.endswith(".md"):
yield os.path.join(root, f)
def check_plans():
"""Check plan status consistency: active plans with 'Done' status, files
missing from index, dangling index entries, done files with wrong status."""
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
plans_dir = os.path.join(REPO, "plans")
done_dir = os.path.join(REPO, "plans", "done")
index_path = os.path.join(plans_dir, "index.md")
if not os.path.exists(index_path):
return 0
violations = 0
STATUS_RE = re.compile(r'^\*\*Status:\*\*\s*(.+)', re.I)
# Parse index.md for active and done entries
active_files = set()
done_files = set()
current_section = None
with open(index_path) as f:
for line in f:
if line.startswith("## Active"):
current_section = "active"
continue
if line.startswith("## Done"):
current_section = "done"
continue
if current_section == "active":
m = re.search(r'\]\(([^)]+)\)', line)
if m:
active_files.add(m.group(1))
elif current_section == "done":
m = re.search(r'\]\(([^)]+)\)', line)
if m:
done_files.add(m.group(1))
# Active plans on disk (not in done/, not index.md)
disk_active = set()
for f in glob.glob(os.path.join(plans_dir, "*.md")):
name = os.path.basename(f)
if name == "index.md":
continue
disk_active.add(name)
# Done plans on disk
disk_done = set()
if os.path.isdir(done_dir):
for f in glob.glob(os.path.join(done_dir, "*.md")):
disk_done.add("done/" + os.path.basename(f))
# Check 1: active plans on disk whose internal status is Done/Implemented/Complete
for name in disk_active:
fpath = os.path.join(plans_dir, name)
with open(fpath) as f:
for line_num, line in enumerate(f, 1):
if line_num > 5:
break
m = STATUS_RE.match(line)
if m:
status = m.group(1).strip().lower()
done_keywords = ["done", "implemented", "complete", "completed"]
if any(status.startswith(kw) for kw in done_keywords):
print(f"{fpath}:{line_num}: status '{m.group(1).strip()}' — file is in plans/ but appears done; move to done/")
violations += 1
break
# Check 2: active plans on disk not in index
for name in sorted(disk_active):
if name not in active_files:
fpath = os.path.join(plans_dir, name)
print(f"{fpath}:1: not listed in plans/index.md Active table")
violations += 1
# Check 3: done plans on disk not in index
for name in sorted(disk_done):
if name not in done_files:
fpath = os.path.join(REPO, "plans", name)
print(f"{fpath}:1: not listed in plans/index.md Done table")
violations += 1
# Check 4: index entries with no file on disk
for name in sorted(active_files):
if name not in disk_active:
print(f"plans/index.md: active entry '{name}' — file not found on disk")
violations += 1
for name in sorted(done_files):
if name not in disk_done:
print(f"plans/index.md: done entry '{name}' — file not found on disk")
violations += 1
# Check 5: files in done/ whose internal status doesn't say Done
for name in disk_done:
fpath = os.path.join(REPO, "plans", name)
with open(fpath) as f:
found_status = False
for line_num, line in enumerate(f, 1):
if line_num > 5:
break
m = STATUS_RE.match(line)
if m:
found_status = True
status = m.group(1).strip().lower()
if not status.startswith("done"):
print(f"{fpath}:{line_num}: status '{m.group(1).strip()}' — file is in done/ but status is not 'Done'")
violations += 1
break
if not found_status:
print(f"{fpath}:1: file is in done/ but has no Status header")
violations += 1
return violations
def main(argv):
paths = argv or ["knowledge", ".agents", "operations", "investigations", "plans"]
violations = 0
if "plans" in paths or any(p.startswith("plans") for p in paths):
violations += check_plans()
# The style guide and this skill enumerate the banned words by definition.
ban_exempt = ("shared/writing-style.md", "skills/docs-lint/")
for f in sorted(set(iter_md(paths))):

View File

@@ -10,22 +10,23 @@ docs_update_checklist: [investigations_entry]
Goal: understand what broke and why, before touching anything.
1. `homelab service <name> explain` (or `homelab node <name> relations`
if the affected entity is a host) — get the blast radius and doc
pointer first. Don't start pulling logs blind.
2. `homelab service <name> health` + `homelab service <name> log` (or
MCP `get_service_status` / `tail_log`) for the affected service.
1. MCP `explain` (or `get_blast_radius` if the affected entity is a
host) — get the blast radius and doc pointer first. Don't start
pulling logs blind.
2. MCP `get_service_status` + `tail_log` for the affected service.
3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
`authentik`, or the backend host itself)? `homelab node <name>
relations` shows "affected by" — check those first.
4. `homelab apt-audit` if the symptom looks like a dpkg/upgrade
interaction.
`authentik`, or the backend host itself)? MCP `get_blast_radius`
shows "affected by" — check those first.
4. If the symptom looks like a dpkg/upgrade interaction, see
[runbook-dpkg-interrupted](../runbook-dpkg-interrupted/SKILL.md) —
there's no fleet-wide apt-audit tool anymore, check the host directly.
5. Check the change ledger for recent mutations to the affected entity
or anything upstream of it: `homelab service <name> history` (once
populated) or grep `ledger/*.jsonl`.
6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom,
timeline, root cause, fix applied, prevention. This is the durable
record; don't rely on chat history.
or anything upstream of it: MCP `get_change_history` or `get_audit_trail`.
6. Write findings via MCP `upsert_knowledge` (`kind: investigation`) —
symptom, timeline, root cause, fix applied, prevention, `about` set to
the affected entity's slug. The DB is the durable record now, not a
markdown file — `search_knowledge`/`get_entity_knowledge` read it back;
a chat message alone is forgotten.
Docs-update checklist: always create the investigation entry. If the
root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,

View File

@@ -2,35 +2,34 @@
name: lifecycle-activate-node
risk_class: config_mutation
inputs: [node_name]
verification: "homelab service <name> health (if it hosts a service); homelab doctor (if it's a client)"
verification: "MCP get_service_status (if it hosts a service); MCP whoami (if it's a client)"
docs_update_checklist: [doc_page_complete]
transition: "provisioning -> active"
---
# Lifecycle: activate a node
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml). Requires: age key
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml). Requires: age key
enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
ingress live if public, health check answering, doc page complete,
ledger entry.
1. If the node is a `homelab` client: finish enrollment per
[client-enrollment.md](../client-enrollment/SKILL.md) (`--finalize-pubkey`,
mesh join, `homelab doctor` green).
1. If the node self-enrolls as a client: finish enrollment per
[CLIENTS.md](../../../CLIENTS.md#enrollment) (`bootstrap.sh`
`/api/v1/clients/enroll`, mesh join, MCP `whoami` returns the entity).
2. If it hosts a public service: add the `services:` entry in
`inventory.yaml` (backend, url, doc_page, config_repo, risk_notes —
see the Week-1 service contract fields) and wire the Caddy route in
`dtoro/caddy-conf`.
3. Confirm the health check answers: `homelab service <name> health` or
a direct `curl`.
`seeds/inventory.yaml` (backend, url, doc_page, config_repo,
risk_notes) and wire the Caddy route in `dtoro/caddy-conf`.
3. Confirm the health check answers: MCP `get_service_status` or a
direct `curl`.
4. Flip `state: provisioning``state: active` (or delete the `state:`
field — `active` is the default) in `inventory.yaml`.
field — `active` is the default) in `seeds/inventory.yaml`, then
`oikos seed` to ingest.
5. Complete the doc page (stub → full narrative: role, specs, how it's
configured, dependencies).
6. Record the activation: `oikos/ledger.py append host:<name> activate
config_mutation --result ok` (or let the CLI wrapper do this once
Week 3's runbook automation lands).
6. No manual record-keeping step needed — the activation (via whatever
API call flipped the state) is recorded automatically in `audit_log`.
Regenerate derived data: `python3 mcp/build_host_files.py && python3
oikos/gen-topology.py` so `hosts/<name>.yaml`, the topology diagram, and
the context card all reflect the new state.
Regenerate: `oikos seed` re-ingests `seeds/inventory.yaml`; `oikos export`
writes DB state back out to the YAML if you mutated via the API/MCP
instead of editing the file directly.

View File

@@ -2,28 +2,28 @@
name: lifecycle-deprecate-node
risk_class: config_mutation
inputs: [node_name, replacement_node_or_reason]
verification: "homelab node <name> relations — 'affected by' must be empty before completing"
verification: "MCP get_blast_radius — 'affected by' must be empty before completing"
docs_update_checklist: [doc_page_deprecation_note]
transition: "active -> deprecated"
---
# Lifecycle: deprecate a node
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml): a node keeps running
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml): a node keeps running
but takes no new dependents. **Completion condition: zero remaining
inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a
suggestion; `oikos/policy.yaml` `lifecycle_overrides.deprecated.refuse`
suggestion; `seeds/policy.yaml` `lifecycle_overrides.deprecated.refuse`
lists `new-inbound-edges` as refused going forward.
1. Set `state: deprecated` on the node.
2. `homelab node <name> relations` — read `affected_by`. Every entry
there is something still relying on this node.
2. MCP `get_blast_radius` — read `affected_by`. Every entry there is
something still relying on this node.
3. Migrate or retire each dependent one at a time (point its `backend`/
`config_repo`/ingress route elsewhere, or deprecate it too if it's
being retired alongside).
4. Re-run `homelab node <name> relations` after each dependent is moved.
The transition to `destroyed` is only safe once `affected_by` is
empty — check this every time, don't assume from memory.
4. Re-run MCP `get_blast_radius` after each dependent is moved. The
transition to `destroyed` is only safe once `affected_by` is empty —
check this every time, don't assume from memory.
5. Note the deprecation on the doc page: reason, replacement (if any),
date.

View File

@@ -2,7 +2,7 @@
name: lifecycle-destroy-node
risk_class: destructive
inputs: [node_name]
verification: "homelab node <name> relations returns unknown-entity; pct list on the backend no longer shows it"
verification: "MCP get_blast_radius returns unknown-entity; pct list on the backend no longer shows it"
docs_update_checklist: [archaeology_entry, containers_index_update]
transition: "deprecated -> destroyed"
---
@@ -10,33 +10,40 @@ transition: "deprecated -> destroyed"
# Lifecycle: destroy a node
**Destructive.** Requires operator approval + typed confirmation phrase
per `oikos/policy.yaml`. Requires (ontology): backups verified, secrets
per `seeds/policy.yaml`. Requires (ontology): backups verified, secrets
recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
ledger entry.
1. Confirm the node is `deprecated` with zero `affected_by` edges
(`homelab node <name> relations`) — do not skip this even if the
deprecation runbook was followed recently; state can drift.
2. If it's an enrolled client: `homelab client remove <name>` — revokes
the age key, re-keys SOPS, removes the inventory entry. This is
already destructive-class and confirmed in the CLI.
(MCP `get_blast_radius`) — do not skip this even if the deprecation
runbook was followed recently; state can drift.
2. **If it's an enrolled client: no current tool for revoking its age key /
removing its Infisical identity.** The old `homelab client remove`
(age key revocation + SOPS re-key + inventory removal, all one
destructive-class CLI call) is retired along with the rest of that CLI
and hasn't been re-verified against the current enrollment
architecture (`POST /api/v1/clients/enroll` + Infisical machine
identities) — see the "Open questions" section in
[agent-enrollment.md](../../operations/agent-enrollment.md). Until
that's confirmed, treat key/identity revocation as a manual step:
at minimum remove the client's `age_pubkey` from any SOPS recipient
lists and rotate credentials whose ciphertext it already decrypted.
3. Remove any ingress route (Caddy config repo) and DNS record still
pointing at it.
4. Verify backups of anything on it are retained per policy before the
disk goes away (see `backs-up-to`).
5. Destroy the LXC/VM (`pct destroy` / `qm destroy`).
6. Move the `hosts.<name>:` block (if any inventory remnant survives
`client remove`, e.g. infra-only LXCs with no age key) into
inventory.yaml's `archaeology:` section: `pve_id`, `destroyed` date,
`reason`. Add a row to `containers/index.md` "Recently destroyed"
table (kept for human-readable browsing alongside the structured
data).
7. `oikos/ledger.py append host:<name> destroy destructive --result ok`.
8. Regenerate: `python3 mcp/build_host_files.py && python3
oikos/gen-topology.py` — the node drops out of `hosts/*.yaml` and
appears in the topology doc's archaeology table.
6. Update the entity's `state` to `destroyed` in `seeds/inventory.yaml`
(or move it to an `archaeology:`-style section if the schema still has
one) — `pve_id`, `destroyed` date, `reason` — then `oikos seed` to
ingest. Add a row to `containers/index.md` "Recently destroyed" table
(kept for human-readable browsing alongside the structured data).
7. No manual ledger step — mutations through the API are recorded
automatically in the `audit_log` table (MCP `get_audit_trail`,
`get_change_history`). The old `oikos/ledger.py append` was retired
when this became automatic.
If the destroy fails partway (e.g. secrets revoked but pct destroy
errors), do not re-run step 2 — `client remove` is not idempotent
against a second revocation attempt on the issuance server. Finish the
remaining steps manually and note the partial state in an investigation.
If the destroy fails partway (e.g. secrets not fully revoked but pct
destroy errors), finish the remaining steps manually and note the
partial state in an investigation (MCP `upsert_knowledge`,
`kind: investigation`).

View File

@@ -2,7 +2,7 @@
name: lifecycle-migrate-node
risk_class: config_mutation
inputs: [node_name, source_host, target_host]
verification: "homelab node <name> relations (re-check blast radius); homelab service <svc> health for every hosted service"
verification: "MCP get_blast_radius (re-check blast radius); MCP get_service_status for every hosted service"
docs_update_checklist: [doc_page_migration_note, inventory_host_and_lan_ip]
transition: "active -> migrating -> active"
---
@@ -10,12 +10,12 @@ transition: "active -> migrating -> active"
# Lifecycle: migrate a node
Modeled on the strong Phase 1+2 migration
([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
([archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
Requires (ontology): preflight + backup-verified before migrating;
post-verify + Caddy backends checked + mounts checked + docs updated
before returning to `active`.
1. `homelab change preflight <every service the node hosts>` — capture
1. MCP `preflight` for every service the node hosts — capture
current health as a baseline.
2. Verify backups are current for anything with data at rest on the
node (see `backs-up-to` edges once populated).
@@ -29,11 +29,12 @@ before returning to `active`.
6. Post-verify: re-run the Week-1 drift check by hand — confirm Caddy's
backend IP for each affected service matches the new `lan_ip`
(automatic in Week 3's drift detector), confirm mounts still resolve.
7. `homelab service <name> health` for every service the node hosts.
7. MCP `get_service_status` for every service the node hosts.
8. Set `state: active`. Add a migration note to the node's doc page
(old host/IP → new, date, phase reference) — this repo's convention
for every past migration (see `containers/101-jellyfin.md`,
for every past migration (see `archive/knowledge/containers/101-jellyfin.md`,
`containers/129-house.md`).
Regenerate: `python3 mcp/build_host_files.py && python3
oikos/gen-topology.py`.
Regenerate: `oikos seed` (re-ingests `seeds/inventory.yaml` into the DB —
the DB is authoritative at runtime, the YAML is the source of truth
on disk).

View File

@@ -2,17 +2,17 @@
name: lifecycle-provision-node
risk_class: config_mutation
inputs: [node_name, kind, storage_pool]
verification: "grep 'state: provisioning' hosts/<name>.yaml"
verification: "grep 'state: provisioning' inventory.yaml"
docs_update_checklist: [doc_page_stub]
transition: "planned -> provisioning"
---
# Lifecycle: provision a node
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml) `lifecycle.transitions`.
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml) `lifecycle.transitions`.
Policy note: `provisioning` nodes get a lifecycle override —
`config_mutation` actions downgrade to `reversible_low` because nothing
depends on the node yet (see `oikos/policy.yaml` `lifecycle_overrides`).
depends on the node yet (see `seeds/policy.yaml` `lifecycle_overrides`).
Requires (from ontology): inventory entry, IP reserved, storage pool
chosen, doc page stub.
@@ -21,10 +21,12 @@ chosen, doc page stub.
`qm create`), choosing the storage pool deliberately — record it as
the `storage:` field once populated (Week 1 schema; not yet backfilled
for existing nodes).
2. Add the inventory entry: `homelab client add <name>` for anything that
will run the `homelab` CLI, or a direct `hosts.<name>:` block with
`state: provisioning`, `kind`, `host`, `pve_id`, `lan_ip` for
infra-only LXCs that won't self-enroll.
2. Add the inventory entry: a `hosts.<name>:` block in
`seeds/inventory.yaml` with `state: provisioning`, `kind`, `host`,
`pve_id`, `lan_ip`, then `oikos seed` to ingest it. For anything that
will self-enroll as a client afterward (see
[CLIENTS.md](../../../CLIENTS.md#enrollment)), the entity must exist in
`planned`/`provisioning` state before `bootstrap.sh` runs there.
3. Stub the doc page (`containers/<pve_id>-<name>.md` or
`vms/<pve_id>-<name>.md`) — even a one-line "provisioning, see plan X"
is enough to satisfy the transition requirement.

View File

@@ -18,7 +18,7 @@ summarised into targets and fixed costs.
- `yuvomi-mcp` is running on LXC 129 and connected as an MCP server in Claude.
- The CSV is an N26 export (columns: Booking Date, Value Date, Partner Name,
Partner Iban, Type, Payment Reference, Account Name, Amount (EUR), …).
- API token: `homelab secret yuvomi-api-token` (decrypts on any enrolled client).
- API token: `yuvomi-api-token`, via Infisical (primary) or `oikos secret` (SOPS fallback).
- Direct API base: `https://house.hubris.network/api/v1`
---

View File

@@ -13,7 +13,8 @@ has packages that are **unpacked but not configured**. Symptoms:
manually run 'dpkg --configure -a' to correct the problem.`
- `dpkg --audit` lists packages with header
`The following packages have been unpacked but not yet configured.`
- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host.
- `dpkg --audit` on the host directly shows unpacked-not-configured packages
(there's no fleet-wide audit tool anymore — check per-host).
The system is still running the **old** binaries (still in memory), but the
**new** binaries are unpacked and waiting for their postinst to run. Two
@@ -33,19 +34,20 @@ config dirs, capabilities, etc.). The system might not come back up cleanly.
## Path A — target is still reachable over ssh (preferred)
```
homelab ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
```
Or for an LXC by name:
Or for an LXC by name (via the MCP `run` tool, or directly on the Proxmox
host):
```
homelab pct <lxc> exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
pct exec <lxc> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
```
When that returns, confirm:
```
homelab apt-audit --target <host>
ssh <host> -- dpkg --audit
```
Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally
@@ -89,10 +91,11 @@ DEBIAN_FRONTEND=noninteractive dpkg --configure -a \
## Prevention
The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect`
unit on the target, so it survives ssh teardown — the failure mode that put
hubris into this state in the first place is no longer reachable through the
standard tool. If you absolutely need to run apt manually over ssh, wrap it:
The old `homelab apt-upgrade` wrapper (retired along with the rest of the
`homelab` CLI) used to launch apt inside a `systemd-run --collect` unit on
the target so it survived ssh teardown — that's the failure mode that put
hubris into this state in the first place. There's no fleet-wide wrapper
anymore; if you run apt manually over ssh, wrap it yourself the same way:
```
ssh <host> systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade'
@@ -103,8 +106,8 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
## Related
- [Operations cheatsheet](../../operations/commands.md)
- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md)
- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md)
- [Auto-deploy pipelines](../../../archive/knowledge/infrastructure/auto-deploy.md)
- [Hubris host page](../../../archive/knowledge/hosts/hubris.md)
## Changelog

View File

@@ -2,7 +2,7 @@
name: service-health-check
risk_class: read_only
inputs: [service_name]
verification: "homelab service <name> health"
verification: "MCP get_service_status"
docs_update_checklist: []
---
@@ -10,17 +10,17 @@ docs_update_checklist: []
Goal: determine whether a service is actually healthy, without ad-hoc SSH.
1. `homelab service <name> explain` — read the context card: backend,
blast radius, doc pointer, risk notes.
2. `homelab service <name> health` — live health probe (HTTP code against
the service's `url`/`endpoint`). Once the Week-3 scheduler ships, this
reads a cached snapshot by default; pass `--live` to force a fresh probe.
3. If unhealthy, `homelab service <name> log` (or MCP `tail_log`) for the
last 200 lines.
4. Cross-check blast radius: `homelab node <name> relations` — is this
entity's own backend host healthy? A downstream failure (e.g. `strong`
1. MCP `explain` — read the context card: backend, blast radius, doc
pointer, risk notes.
2. MCP `get_service_status` — live health probe (HTTP code against the
service's `url`/`endpoint`); the scheduler also probes on its own
interval, so this may reflect a recent cached result, not necessarily
a fresh one.
3. If unhealthy, `tail_log` for the last 200 lines.
4. Cross-check blast radius: MCP `get_blast_radius` — is this entity's
own backend host healthy? A downstream failure (e.g. a Proxmox host
down) will show up here before the service's own logs explain anything.
5. If the fix is a restart: classify first (`oikos/policy.yaml`
5. If the fix is a restart: classify first (`seeds/policy.yaml`
`service-restart` is `reversible_low` unless the service has a
`service_overrides` entry, e.g. `caddy`/`dns` are `config_mutation`).
Unattended agents may act on `reversible_low` without approval.

View File

@@ -0,0 +1,82 @@
---
name: session-review
description: "Examine a Nomos chat session, compare the user's objective with the actual outcome, identify causes of failure (missing tools, excessive tool calls, blocked actions, model behavior), and propose concrete fixes."
risk_class: reversible_low
inputs: [session_id]
---
# Session review
Analyze Nomos chat sessions from the live database, diff objectives
against outcomes, and propose fixes.
## 1. Retrieve session data
```bash
# List recent sessions
curl -s http://localhost:8092/sessions | jq '.sessions[:5]'
# Fetch one session with messages
curl -s http://localhost:8092/sessions/{session_id} | jq .
```
## 2. Classify the session
For each session determine:
| Dimension | Check |
|-----------|-------|
| Objective | What was the user trying to accomplish? |
| Outcome | Was it achieved? (read final assistant text) |
| Tool calls | Count, unique tools, redundancy (e.g., N+1 fan-out) |
| Blockers | Missing action? Missing tool? Model refusal? Empty response? |
| User frustration | Did the user need to clarify/correct/repeat? |
| Message sizes | Content blob sizes — truncation needed? |
## 3. Key failure signatures
| Signature | Root cause | Fix |
|-----------|-----------|-----|
| Agent: "I can't run X — only supports Y" | Missing action in `request_execution` | Add action in `internal/mcp/server.go` |
| Agent: "No local knowledge on that" + no web tool | Missing `http_get` / web fetch MCP tool | Add MCP tool |
| Empty assistant bubble (text="", no tools) | Model returned blank completion | Retry + error surfacing |
| Non-English boilerplate refusal | Flash-tier model degradation | Response quality guard |
| >30 tool calls per turn, same tool repeated | N+1 fan-out instead of bulk tool | Enrich bulk tools + tighten SOUL.md |
| Message >50KB in DB | Raw tool results persisted verbatim | Truncation in `store.go` |
## 4. Extract patterns across sessions
```bash
# All sessions summary
curl -s http://localhost:8092/sessions | jq -r '.sessions[] | "\(.id[:8]) \(.title[:80]) \(.created_at[:16])"'
# Message count + tool count per session
for id in $(curl -s http://localhost:8092/sessions | jq -r '.sessions[].id'); do
msgs=$(curl -s "http://localhost:8092/sessions/$id" | jq '.messages | length')
tools=$(curl -s "http://localhost:8092/sessions/$id" | jq '[.messages[].content.tool_calls | length] | add')
echo "$id $msgs msgs $tools tools"
done
```
## 5. Output format
```
Session: {id[:8]} — "{title[:60]}"
Messages: {N} ({user}/{assistant})
Tool calls: {total} across {turns} turns
Top tools: {name:count, name:count, ...}
Objective: {one-line summary}
Outcome: ✅ / ❌ / ⚠️
Blockers: {list or "none"}
Fixes needed: {concrete actions}
Severity: blocker | friction | cosmetic
```
## Related files
- `cmd/nomos/agent.go` — agent loop, tool building, response guards
- `cmd/nomos/store.go` — session + message persistence
- `internal/mcp/server.go` — all tool implementations including `request_execution`
- `web/src/lib/components/ToolCallGroup.svelte` — tool result display
- `nomos/SOUL.md` — agent persona and tool selection rules
- `plans/2026-07-09-chat-sessions-improvements.md` — prior session findings
- `plans/2026-07-09-session-execution-and-ux-fixes.md` — latest plan

11
.claude/launch.json Normal file
View File

@@ -0,0 +1,11 @@
{
"version": "0.0.1",
"configurations": [
{
"name": "web",
"runtimeExecutable": "npm",
"runtimeArgs": ["--prefix", "web", "run", "dev"],
"port": 5173
}
]
}

72
.gitea/workflows/ci.yml Normal file
View File

@@ -0,0 +1,72 @@
# Oikos CI (Gitea Actions). Gates the deploy webhook on a green run (plan M1).
# Mirrors `make lint`, `make test`, and the generated-code drift guard.
name: ci
on:
push:
branches: [main]
pull_request:
jobs:
build-test:
runs-on: ubuntu-latest
services:
postgres:
image: timescale/timescaledb:2.17.2-pg16
env:
POSTGRES_DB: oikos
POSTGRES_USER: oikos
POSTGRES_PASSWORD: oikos_dev
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U oikos"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
OIKOS_TEST_DATABASE_URL: postgres://oikos:oikos_dev@postgres:5432/oikos?sslmode=disable
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.26"
cache: true
- name: go vet
run: go vet ./...
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
with:
version: latest
args: --timeout 5m
continue-on-error: true # advisory until the lint baseline is clean
- name: govulncheck
run: |
go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck ./... || true # advisory
- name: generated code is up to date
run: make generate-check
- name: build
run: go build ./...
- name: test (race + coverage)
run: go test -race -covermode=atomic -coverprofile=coverage.out -timeout 300s ./...
- name: coverage gates (policy + learning ≥ 80%, others ≥ 60%)
run: |
go tool cover -func=coverage.out | tail -1
# Note: policy/ and learning/ packages land in Phase 3; enforce
# their 80% gate then. For now, report total coverage.
docker-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: docker build (verify image builds; no push)
run: docker build -f compose/oikos/Dockerfile -t oikos:ci .

View File

@@ -0,0 +1,98 @@
name: Desktop App
on:
push:
branches:
- main
tags:
- 'desktop-*'
- 'v[0-9]+.[0-9]+.[0-9]*'
jobs:
build-ui:
name: Build SPA
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: 'npm'
cache-dependency-path: web/package-lock.json
- run: npm ci
working-directory: web
- run: npm run build
working-directory: web
- uses: actions/upload-artifact@v4
with:
name: spa-dist
path: web/dist/
build-macos-arm64:
name: macOS (arm64)
needs: build-ui
runs-on: macos-14
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: spa-dist
path: cmd/desktop/frontend/dist/
- uses: actions/setup-go@v5
with:
go-version: '1.26'
- run: go install github.com/wailsapp/wails/v3/cmd/wails3@latest
- run: wails3 build -clean
working-directory: cmd/desktop
env:
CGO_ENABLED: 1
- run: |
cd cmd/desktop/build/bin
zip -r oikos-desktop-darwin-arm64.zip oikos-desktop.app
- uses: actions/upload-artifact@v4
with:
name: oikos-desktop-darwin-arm64
path: cmd/desktop/build/bin/oikos-desktop-darwin-arm64.zip
build-linux-amd64:
name: Linux (amd64)
needs: build-ui
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: spa-dist
path: cmd/desktop/frontend/dist/
- uses: actions/setup-go@v5
with:
go-version: '1.26'
- run: sudo apt-get update && sudo apt-get install -y libgtk-3-dev libwebkit2gtk-4.1-dev
- run: go install github.com/wailsapp/wails/v3/cmd/wails3@latest
- run: wails3 build -clean
working-directory: cmd/desktop
env:
CGO_ENABLED: 1
- uses: actions/upload-artifact@v4
with:
name: oikos-desktop-linux-amd64
path: cmd/desktop/build/bin/oikos-desktop
release:
name: Create Release
needs: [build-macos-arm64, build-linux-amd64]
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/download-artifact@v4
with:
name: oikos-desktop-darwin-arm64
- uses: actions/download-artifact@v4
with:
name: oikos-desktop-linux-amd64
- name: Release
uses: https://gitea.com/actions/release-action@v1
with:
files: |
oikos-desktop-darwin-arm64.zip
oikos-desktop-linux-amd64
api_key: ${{ secrets.GITEA_TOKEN }}

24
.gitignore vendored
View File

@@ -2,9 +2,23 @@
__pycache__/
*.pyc
# Regenerated every scheduler run (every 10 min); no audit value in the
# diff. Signals (signals/*.jsonl) ARE tracked — this is just the ephemeral
# health-probe cache. See oikos/scheduler.py.
oikos/state.json
# Compiled binaries (Go rewrite — bin/oikos, bin/nomos)
bin/oikos
bin/nomos
oikos
webhook
.worktrees/
backups/
.env
.infisical-credentials
# Web UI (Svelte 5) — build artifacts. The SPA is a standalone static build,
# deployed separately from the oikos binary (plans/2026-07-12-wails-desktop-app.md
# 0.1), so the output dir is just a build artifact.
web/dist/
web/node_modules/
# Wails desktop app — frontend copy for embedding
cmd/desktop/frontend/dist/
cmd/desktop/build/
cmd/desktop/oikos-desktop

View File

@@ -1,14 +1,12 @@
# SOPS recipient rules.
#
# Each rule pins one or more age public keys as recipients for files matching
# the path_regex. The build_host_files.py generator doesn't read this file;
# `sops` does — to encrypt a new secret, run `sops -e secrets/<name>.yaml`
# from the repo root and SOPS will pick the matching rule below.
# SOPS is DR-fallback only — Infisical is the active secrets backend.
# Files live in archive/secrets-sops-backup/ for cold recovery.
# To encrypt a new DR secret: sops -e archive/secrets-sops-backup/<name>.yaml
#
# To grant a secret to a new client: add their age public key (from
# To grant a DR secret to a new client: add their age public key (from
# inventory.yaml `hosts.<name>.age_pubkey`) to the relevant rule below, then
# run `sops updatekeys secrets/<name>.yaml` to re-encrypt without rotating
# the ciphertext payload.
# run `sops updatekeys archive/secrets-sops-backup/<name>.yaml` to re-encrypt.
#
# To revoke: remove the recipient from the relevant rule and run
# `sops updatekeys` (this is what `homelab client remove` calls). Past
@@ -16,7 +14,7 @@
# underlying credential if compromise is suspected.
creation_rules:
- path_regex: ^secrets/hello\.yaml$
- path_regex: ^archive/secrets-sops-backup/hello\.yaml$
# The "hello" secret is encrypted to every enrolled client so the bootstrap
# decrypt test works for everyone. Add each new client's age_pubkey when
# they enrol; re-key with `sops updatekeys -y secrets/hello.yaml`.
@@ -29,7 +27,7 @@ creation_rules:
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/gitea-pat\.yaml$
- path_regex: archive/secrets-sops-backupgitea-pat\.yaml$
# Write-scoped Gitea PAT (dtoro user). Same recipient list as hello.yaml
# since every enrolled client should be able to push (homelab client
# add/remove, wiki edits, etc.).
@@ -42,17 +40,17 @@ creation_rules:
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/gitea-tokens\.yaml$
- path_regex: archive/secrets-sops-backupgitea-tokens\.yaml$
# Workstations only.
age: >-
# placeholder — fill with age_pubkey of: republic-laptop, mac-mini, strong, hubris
- path_regex: ^secrets/webhook-hmacs\.yaml$
- path_regex: archive/secrets-sops-backupwebhook-hmacs\.yaml$
# LXCs that run a webhook receiver.
age: >-
# placeholder — fill with age_pubkey of: apps, caddy
- path_regex: ^secrets/turn-shared-secret\.yaml$
- path_regex: archive/secrets-sops-backupturn-shared-secret\.yaml$
# coturn TURN long-term-credential password. Consumed by hubris (which
# renders /etc/turnserver.conf + /opt/management.json on the VPS via
# `homelab render-vps-configs`). Other recipients are convenience for
@@ -65,7 +63,7 @@ creation_rules:
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/netbird-authentik-oidc\.yaml$
- path_regex: archive/secrets-sops-backupnetbird-authentik-oidc\.yaml$
# Authentik OIDC client secret for the netbird-dashboard provider.
# Consumed by hubris to render /opt/management.json on the VPS
# (PKCEAuthorizationFlow.ProviderConfig.ClientSecret).
@@ -77,7 +75,7 @@ creation_rules:
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/netbird-pat\.yaml$
- path_regex: archive/secrets-sops-backupnetbird-pat\.yaml$
# NetBird API Personal Access Token. Consumed by the dns-sync job on the
# `dns` LXC (107) to reconcile Technitium -> NetBird managed DNS zone.
# (When 107 is enrolled, add its age_pubkey here and updatekeys.)
@@ -87,19 +85,19 @@ creation_rules:
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
- path_regex: ^secrets/openrouter-api-key\.yaml$
# OpenRouter API key consumed by the `hermes` wrapper (bin/hermes) when
- path_regex: archive/secrets-sops-backupopenrouter-api-key\.yaml$
# OpenRouter API key consumed by the `nomos` wrapper (bin/nomos) when
# spawning a Goose session. Recipients are any host that should run a
# Nous-Hermes agent. Add a host's age_pubkey here, then
# Nomos agent. Add a host's age_pubkey here, then
# `sops updatekeys -y secrets/openrouter-api-key.yaml`.
# See operations/hermes-agent.md.
# See operations/nomos-agent.md.
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/yuvomi-api-token\.yaml$
- path_regex: archive/secrets-sops-backupyuvomi-api-token\.yaml$
# Named Bearer token for the Yuvomi REST API, consumed by yuvomi-mcp on
# LXC 129 (house).
age: >-
@@ -108,7 +106,7 @@ creation_rules:
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/hermes-house-users\.yaml$
- path_regex: archive/secrets-sops-backuphermes-house-users\.yaml$
# Signal number → Yuvomi user_id mapping (PII). Consumed by hermesd on LXC 129.
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
@@ -116,7 +114,7 @@ creation_rules:
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/oikos-approval-hmac\.yaml$
- path_regex: archive/secrets-sops-backupoikos-approval-hmac\.yaml$
# HMAC signing key for Oikos approval-grant tokens (oikos/approve.py).
# Recipients: apps (105, runs the approval engine alongside homelab-mcp)
# and hubris (admin/debug decrypt). See OIKOS.md "Approval engine".
@@ -124,7 +122,7 @@ creation_rules:
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
- path_regex: ^secrets/oikos-console-deploy-secret\.yaml$
- path_regex: archive/secrets-sops-backupoikos-console-deploy-secret\.yaml$
# Shared HMAC secret for the Gitea deploy webhook (id 14) ->
# oikos-console-deploy.service on apps (105). Generated + registered
# with Gitea before the apps-side install ran (see

File diff suppressed because one or more lines are too long

236
AGENTS.md
View File

@@ -4,146 +4,208 @@ You are running on a machine that is part of the **hubris** homelab. The full
context is in this checkout at `/opt/homelab-context/`. This file is the entry
point. Read it once at start, then keep working.
- **New client?** Read [CLIENTS.md](CLIENTS.md) first.
- **Developing on this repo?** Also read [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
The operating model — OODA loop, risk classes, approval rules, the ontology,
and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation,
classify the action against `oikos/policy.yaml`; when the class requires
classify the action against `seeds/policy.yaml`; when the class requires
approval, stop and ask the operator.
Agent-facing instruction is separated from human content under `.agents/`:
Agent-facing instruction lives under `.agents/`:
`.agents/shared/` holds the conventions every agent applies
([writing-style](.agents/shared/writing-style.md), [caveman](.agents/shared/caveman.md),
[page-templates](.agents/shared/page-templates.md), [llm-wiki](.agents/shared/llm-wiki.md)), and
`.agents/domains/` holds the per-domain schemas
([knowledge](.agents/domains/knowledge/schema.md), [operations](.agents/domains/operations/schema.md)).
The narrative wiki lives under `knowledge/wiki/`; the machine-readable substrate
(`inventory.yaml`, `hosts/*.yaml`, `oikos/`) stays at the repo root.
**Source of truth:** The Postgres database is the single source of truth for all
structured data and knowledge. It is bootstrapped from `seeds/` at deploy time:
`seeds/ontology.yaml` (entity types, relationships, lifecycles), `seeds/inventory.yaml`
(hosts, services, entities), `seeds/policy.yaml` (risk classes, approval rules), and
`seeds/knowledge.yaml` (documents, investigations, runbooks). The old narrative wiki
is archived at `archive/knowledge/` for historical reference.
## 1. Who you are
Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read:
/opt/homelab-context/hosts/<your-hostname>.yaml
/opt/homelab-context/inventory.yaml
That file tells you your role, your peers, what's mounted, and what services
you host. If it does not exist, this client was not enrolled — stop and tell
the operator to run `homelab client add <hostname>` from an existing client.
the operator; see [CLIENTS.md](CLIENTS.md#enrollment) for the enrollment flow
(the entity needs to exist in `planned`/`provisioning` state first).
## 2. The topology
- `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation
with their mesh addresses, roles, and service mappings. Treat this file as
authoritative; anything you read in narrative pages should agree with it.
- `/opt/homelab-context/knowledge/wiki/infrastructure/mesh.md` — Tailscale → Netbird state.
Both meshes are accepted today; Netbird is preferred for new traffic.
- `/opt/homelab-context/knowledge/wiki/infrastructure/dns.md` — split-horizon DNS via
Technitium on [dns (107)](knowledge/wiki/containers/107-dns.md). `*.hubris.network`
resolves to 192.168.x.x on the LAN and to mesh addresses off-LAN.
with their mesh addresses, roles, and service mappings. This is the seed file;
at runtime the DB is authoritative (query via MCP `get_entity` or the REST API).
- `/opt/homelab-context/seeds/knowledge.yaml` — full narrative knowledge: 36
documents, 6 investigations, 12 runbooks. Ingested into the DB on deploy.
- `/opt/homelab-context/.agents/operations/commands.md` — the operator's cheatsheet
for pct, caddy, DNS, and the Oikos command surface. Use these verbs when
you take actions.
for pct, caddy, DNS, and the Oikos command surface.
## 3. The MCP server
The homelab exposes a Model Context Protocol server with structured tools.
Endpoint is in `inventory.yaml` under `services.homelab_mcp.endpoint`.
Endpoint: `https://mcp.hubris.network/mcp`. Every call needs
`Authorization: Bearer <token>` — the API has no unauthenticated path except
enrollment and `/healthz` (see "Authentication" below for where the token
comes from).
Available tools:
Available tools (33 total):
Context (pure read):
get_host(name), list_services(), find_service(name_or_role),
get_topology(), search_docs(query), get_page(path),
get_changelog(page, since?), whoami(hostname),
list_my_secrets(caller_pubkey?)
Context — observe + orient:
get_entity(slug), list_entities(type, limit, cursor),
get_relations(entity), get_blast_radius(entity),
search_knowledge(query) — ILIKE search over documents, investigations,
runbooks in the knowledge_entities table
get_entity_knowledge(entity_slug) — every document, investigation, and
runbook linked to one entity, in one call
get_patterns(status, entity_type, action) — learned action patterns
get_skills(status) — available automation skills
http_get(url) — fetch a public page/raw file (e.g. researching how to
deploy something before provisioning it); HTTP/HTTPS only, ~16KB cap
Management (read-only):
get_service_status(service), tail_log(service, lines=200),
list_lxcs(), get_lxc_state(lxc), ping_service(service)
Management — live state:
get_service_status(service_slug) — systemctl is-active on target host
tail_log(service_slug, lines=200) — journalctl
list_lxcs() — all LXC containers with ID, host, IP, health
get_lxc_state(lxc_slug) — pct status from Proxmox host
ping_service(service_slug) — HTTP reachability from entity_status
list_my_secrets(caller_pubkey) — secrets accessible to this client by
age public key
Oikos (read-only; see OIKOS.md):
explain(service) — compact context card, cheaper than search_docs+get_page
preflight(service) — risk class, approval requirement, verification command
get_relations(entity) — ontology blast-radius query (host: or service: id)
get_change_history(entity, limit=20) — change-ledger entries
get_state_snapshot() — last scheduler Observe-pass (health, disk, drift count)
Oikos — decisions:
explain(service_slug) — compact context card (type, state, health, relations)
preflight(service_slug, action) — risk class + approval requirement
whoami(hostname) — entity record, peers, health for a client
get_change_history(entity_slug, limit=20) — last audit-log entries per entity
get_state_snapshot() — fleet health, disk, drift count
Mutations are **not** exposed via MCP. Use the `homelab` CLI for those, with
operator confirmation — see OIKOS.md's risk classes and approval flow.
Operations — observe + act:
get_health_summary() — fleet health counts (healthy/degraded/down/unknown)
get_signal_history(entity_slug, state, limit) — open + recent signals
get_audit_trail(entity_id) — audit log filter + browse
get_agent_activity(limit) — agent self-inspection
query_metrics(hours=24) — time-series metric bucketed averages
get_trend(entity_id, days=7) — metric slope over time
get_event_timeline(severity, entity_slug, limit) — recent events
**When to prefer MCP over grepping the clone:** any time you need to resolve a
name to an address, look up service status, or search the wiki by content.
Grep is fine for browsing or when MCP is unreachable.
Knowledge — keep the graph current (none require approval; this updates
the knowledge graph, not live infrastructure):
upsert_knowledge(title, content) — record what you learned after solving
a non-obvious problem; the only way anything persists past a session
update_entity_attributes(slug, attributes) — merge a discovered fact
(IP, version, port, ...) into an entity so a future task doesn't
rediscover it from scratch
create_relationship(source, target, type) — record a discovered edge
(depends-on, hosts, routes-to, ...) between two entities
## 4. Wiki conventions
Execution — mutating the live infrastructure:
run(target, command) — the general execution primitive. Run any shell
command against a host or LXC; every command is auto-classified —
read-only inspection runs immediately, anything state-changing needs
operator approval, and destructive patterns (rm -rf, dd, mkfs,
pct/qm destroy, DROP TABLE, reboot, curl-pipe-to-shell, ...) always
need approval regardless of what you declare. Prefer this over
request_execution for anything not already covered by its fixed enum.
request_execution(target, action, params) — the older, fixed-enum path
(restart, systemctl, pct_exec, apt_upgrade, pct_create). Still the
route for those specific actions; policy-gated the same way `run` is.
get_execution_status(execution_id) — poll progress
See [page-templates.md](.agents/shared/page-templates.md) for file naming, page
structure, and the tone standard. Quick reference:
**When to prefer MCP over grepping the clone:** always for knowledge queries.
`search_knowledge("jellyfin hardware acceleration")` returns ranked results from
the DB with entity links. `get_entity_knowledge("lxc:jellyfin")` returns documents,
runbooks, and investigations in one call. Grep the clone only when MCP is
unreachable.
- **File naming:** Foundational docs are ALL-CAPS (AGENTS.md, OIKOS.md, GLOSSARY.md);
containers use `<id>-<name>.md`; infrastructure uses lowercase-with-dashes; plans
and investigations use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`.
- **Where pages live:** Narrative under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`;
incident records under `knowledge/sources/investigations/`; runbook procedures under
`.agents/skills/<name>/SKILL.md`; operator reference under `.agents/operations/`;
design docs under `plans/`. Cross-link liberally; orphans are bugs.
- **Changelog format:** Every page ends with a `## Changelog` section, entries in
reverse-chronological order:
## 4. Authentication
### YYYY-MM-DD — short title
one or two lines describing what changed and why.
Every API/MCP route requires `Authorization: Bearer <token>` except
`POST /api/v1/clients/enroll` and `/healthz`. Enrollment (see
[CLIENTS.md](CLIENTS.md#enrollment)) does not currently issue a per-client
API/MCP bearer token — there is one shared
secret (`OIKOS_MCP_BEARER_TOKEN`, validated in `internal/httpapi/server.go`'s
`combinedAuth`); get it from the operator until per-client token issuance
exists. The SPA has its own flow instead: a first-launch Config screen that
stores a token in `localStorage` (see `web/src/pages/Config.svelte`).
## 5. Knowledge conventions
All narrative knowledge (documents, investigations, runbooks) lives in the DB
(`knowledge_entities` table) and is seeded from `seeds/knowledge.yaml`. Agents
can register new knowledge via the API:
```
POST /api/v1/knowledge/{entity_slug}
{"title": "...", "content": "...", "tags": ["..."]}
```
The DB is the truth. The old wiki files are in `knowledge/wiki/` pending archive
per the DB-as-source-of-truth plan.
- **Runbook procedures** live as `runbook` entities in the DB and as SKILL.md
files under `.agents/skills/<name>/`. They carry `risk_class`, `procedure`
(JSON-schema-validated), and are linked to entity types via `applies_to_type`.
- **Investigations** are `investigation` entities linked to affected entities
via `about` edges.
- **Documents** are `document` entities linked to entities via `documents` edges.
They carry `at_glance` (structured attributes) and `changelog` (parsed entries).
- **Live state precedence.** If you observe a discrepancy between the docs and
running state, update the docs *in the same session* (per the same-session update
rule in [page-templates.md](.agents/shared/page-templates.md#same-session-update-rule)).
running state, update the DB *in the same session* via the API. The `oikos export`
command regenerates `seeds/knowledge.yaml` for version control.
## 5. Acting on the homelab
## 6. Acting on the homelab
- **Read state**: prefer MCP tools, then files, then shell. Examples:
`homelab whoami`, `homelab list`, `homelab status`, `homelab logs caddy`.
- **Cross-host actions** (caddy reload, pct exec, etc.): use the `homelab`
CLI — it resolves hostname → mesh address → ssh / pct path for you. Direct
SSH still works; the CLI just removes the lookup burden.
- **Secrets**: never hardcode. Call `homelab secret <name>` to decrypt on
demand using the per-client age key at `/etc/age/key.txt`. Secrets ARE
available in this system — `list_my_secrets()` (MCP) shows what you can
decrypt.
- **Mutations** (restart, edit configs, etc.): classify against
`oikos/policy.yaml` first (`homelab decide <action> <entity>`).
`reversible_low` actions just need the interactive confirmation prompt;
`config_mutation`/`destructive` actions are mechanically refused without
a valid `--approval-id` from `homelab approval request` — see OIKOS.md.
For ad-hoc work, SSH and edit directly — but commit changes that touch
tracked configs (caddy, gitea custom, artifacto, mule-image, etc.; see
`knowledge/wiki/infrastructure/auto-deploy.md`).
- **Wiki updates**: same-session rule applies to any meaningful state change
this client makes.
- **Read state**: use MCP tools. Nomos (the AI agent) is the primary
operator interface — it has 33 MCP tools for observe/orient/decide/act
(§3).
- **Actions** (restart, logs, apt, pct exec, or anything else): Nomos calls
`run` (the general execution primitive) or `request_execution` (the older
fixed-enum path) via MCP. `reversible_low`/read-only actions execute
immediately; `config_mutation` and `destructive` actions are queued for
operator approval via Matrix or the control-room UI's Operations page.
- **Secrets**: managed by Infisical (`oikos secret` subcommand for
migration). Never hardcode secrets — use env vars from `.env`.
- **Mutations** (restart, edit configs, etc.): classified against
`seeds/policy.yaml`. `reversible_low` actions auto-execute;
`config_mutation`/`destructive` actions require approval — granted by
the operator via Matrix reply or the control-room UI, not a CLI flag.
See OIKOS.md.
## 6. Communication mode
## 7. Communication mode
Read and apply `/opt/homelab-context/.agents/shared/caveman.md` (if present). It defines the lab's
terse-communication standard — drop filler, keep substance, use fragments.
## 7. Auto-setup mechanism
## 8. Auto-setup mechanism
The homelab-context repo ships tooling that gets automatically installed
on every client after `git pull`. This is handled by `tools/post-pull.sh`
(replaces the raw git pull in the sync timer) which runs any script matching
`tools/*.setup.sh` after pull.
`tools/setup-*.sh` after pull.
Currently auto-setup:
- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm
package, wrapper scripts, and compact output templates for token-efficient
CLI output. Wrapper at `~/bin/caveman_wrapper.sh`.
- **Hermes agent persona** (`tools/setup-hermes-soul.sh`): Provisions
`~/.hermes/SOUL.md` from `HERMES.md` on Hermes agents. This ensures every
Hermes agent follows the canonical homelab persona (token efficiency, source
of truth hierarchy). No-op on non-Hermes agents.
- **Host checks** (`tools/setup-checks.sh`): Deploys `checks/install.sh`'s
health-check scripts to `/opt/oikos/checks` on each host. The scheduler's
`ssh-script` check kind depends on these actually being there — 20 are
live in the DB as of 2026-07-12.
To add a new auto-setup, create `tools/<name>.setup.sh` in the repo,
To add a new auto-setup, create `tools/setup-<name>.sh` in the repo,
commit and push. All enrolled clients pick it up within 5 minutes.
To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer.
To trigger sync manually: run `/opt/homelab/tools/context-poller.sh`, or
wait for the 5-min timer. (This mechanism — and the server-side
`tools_changed` detection behind it — only correctly recognized
`setup-*.sh` scripts as of 2026-07-12; before that it silently matched
nothing, so nothing auto-ran on any client via this path.)
## 8. When in doubt
## 9. When in doubt
Run `homelab mcp search_docs <query>` or `homelab mcp get_host <name>`.
The clone is the fallback; MCP is the index.
Use MCP tools: `search_knowledge <query>` for narrative context,
`get_entity <slug>` for structured data, `get_entity_knowledge <slug>` for
everything linked to an entity. The clone is the fallback; MCP is the index.

151
CLIENTS.md Normal file
View File

@@ -0,0 +1,151 @@
# Oikos — client guide
If you are a homelab machine, this is what Oikos is and what it gives you.
If you are an AI agent running on this machine, also read [AGENTS.md](AGENTS.md).
## What is Oikos?
Oikos is the agentic operating system for the **hubris** Proxmox homelab. It
observes state, classifies actions against policy, executes approved
procedures, learns from outcomes, and notifies the operator. It runs as a
Docker stack on mac-mini and exposes an MCP server + REST API.
## What Oikos provides
| Capability | How you access it |
|------------|-------------------|
| Entity query (topology, blast radius) | MCP `get_entity`, `get_blast_radius` |
| Full-text knowledge search | MCP `search_knowledge` |
| Service status + logs | MCP `get_service_status`, `tail_log` |
| LXC inventory + state | MCP `list_lxcs`, `get_lxc_state` |
| Context cards | MCP `explain` |
| Pre-flight risk classification | MCP `preflight` |
| Change history | MCP `get_change_history` |
| State snapshot (health, disk, drift) | MCP `get_state_snapshot` |
| Secrets (Infisical) | REST API + `oikos secret` CLI |
| Approval tokens | Matrix via notifier |
| Run a command on a host/LXC (policy-gated) | MCP `run` |
| Record a discovered fact/relationship | MCP `update_entity_attributes`, `create_relationship`, `upsert_knowledge` |
Most MCP tools are read-only; a few mutate the knowledge graph (recording
what you learned) or the live infrastructure (`run`, `request_execution`),
gated by risk classification and — for `config_mutation`/`destructive`
actions — operator approval. See [AGENTS.md](AGENTS.md#3-the-mcp-server) for
the full tool catalog.
## Authentication
Every API/MCP call needs `Authorization: Bearer <token>` — there is no
unauthenticated path except `POST /api/v1/clients/enroll` and `/healthz`.
Enrollment (below) does not currently hand out a per-client bearer token;
get the shared `OIKOS_MCP_BEARER_TOKEN` from the operator until per-client
token issuance exists.
## Enrollment
Thin client model — no git clone, no sync timer. `bootstrap.sh` fetches only
the agent orientation files and tooling from the raw Gitea URL, then enrolls
via the Oikos API.
To enroll:
```bash
# Run from any machine with mesh connectivity
curl -fsSL https://git.hubris.network/dtoro/oikos/raw/main/bootstrap.sh | sudo bash
# Or with optional tooling:
curl ... | sudo bash -s -- --with-mcp # wire Claude's MCP config
```
This calls `POST /api/v1/clients/enroll` on the Oikos API, which:
1. Validates the entity exists in DB (planned or provisioning state)
2. Validates mesh IP against expected subnets
3. Generates an age keypair and delivers it to the client
4. Creates an Infisical machine identity
5. Transitions the entity to provisioning state
## After enrollment
### What changes on your machine
- `/opt/homelab/` — agent orientation files (CLIENTS.md, AGENTS.md, OIKOS.md)
- `/opt/homelab/tools/` — tooling scripts (checks)
- `/etc/age/key.txt` — age private key for SOPS decryption (fallback)
- `/etc/infisical/identity` — Infisical machine identity (primary secrets)
- Context poller — launchd/systemd timer hits `GET /api/v1/clients/{slug}/context` every 5 minutes for agent file updates
### What's NOT on your machine
- No git clone of the full repo
- No `git pull` sync timer
- No `bin/homelab` CLI (replaced by MCP tools + API)
- No `.sops.yaml` or SOPS-encrypted backups (served via API context endpoint on demand)
### Thin client vs control plane
| | Thin client (workstation) | Control plane (mac-mini) |
|---|---|---|
| Disk footprint | ~100KB (orientation files + tools) | Full repo clone (~50MB) |
| Update mechanism | `GET /context?since=` poll | Git pull + post-pull.sh |
| Source of truth | DB via MCP | DB + local seeds + archive |
| Secrets access | Infisical (primary), age/SOPS served via API (fallback) | Infisical + local SOPS files |
### Your identity
Your identity in the homelab is defined in `inventory.yaml`. Run `hostname`
(Linux) or `scutil --get LocalHostName` (macOS), then look up your entry.
It tells you your role, what services you host, what's mounted, and your
mesh address.
### Source of truth hierarchy
1. **Postgres database** (runtime) — authoritative for entities, knowledge,
signals, ledger. Query via MCP or REST API.
2. **Context poller** — agent files and tooling fetched via API deltas every
5 minutes.
3. **Never guess.** If data is missing, query MCP. If MCP is down, grep the
local `/opt/homelab/` files.
## The context poller
Every 5 minutes, launchd (macOS) or systemd (Linux) hits:
```
GET /api/v1/clients/ws:{hostname}/context?since={last_timestamp}
```
The API returns which agent files, tools, and SOPS config changed since the
last poll. Only changed files are downloaded. This replaces the old
`git pull` with a lightweight HTTP delta.
To trigger manually: run `/opt/homelab/tools/context-poller.sh`.
## Making changes
- **Read state**: use MCP tools or the API
- **Mutate state** (restart, edit config, deploy): classify the action against
policy (query `preflight` MCP tool):
- `read_only` / `reversible_low` — execute directly
- `config_mutation` / `destructive` — request operator approval via
`POST /api/v1/entities/{slug}/activate` (or equivalent lifecycle endpoint)
- **Secrets**: use Infisical (primary) or SOPS (fallback). Never hardcode.
- **Knowledge**: if you observe a discrepancy between docs and live state,
update the DB via the API in the same session.
## MCP endpoint
```
https://mcp.hubris.network/mcp
```
Available tools are listed in [AGENTS.md](AGENTS.md#3-the-mcp-server).
## Communication mode
Apply [.agents/shared/caveman.md](.agents/shared/caveman.md) — terse,
fragment-heavy communication. Drop filler, keep substance.
## Related
- [AGENTS.md](AGENTS.md) — full agent orientation (read this first)
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, risk classes
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide for the Oikos codebase

177
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,177 @@
# Contributing to Oikos
Developer guide for the Oikos codebase. If you are a homelab client consuming
Oikos, see [CLIENTS.md](CLIENTS.md). If you are an AI agent working on the
repo, see [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
## Dev setup
- **Go 1.26+** (see `go.mod` for pinned version)
- **PostgreSQL with TimescaleDB** — the compose stack includes `timescale/timescaledb:2.17.2-pg16`
- **Docker** for the full dev stack
- **Node 22+** for `web/` (the control-room SPA — standalone, not part of the
compose stack or the `oikos` binary)
```bash
# Start dependencies (Postgres + Redis). api/nomos require a shared bearer
# token — no dev-open bypass — so set one even for local dev.
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile dev up -d
# Run all tests
make test
# Run integration tests (needs compose Postgres)
make test-db
# Build the binary
make build
# SPA dev server (proxies to api/nomos, injecting the same token)
cd web && OIKOS_API_TOKEN=dev-token npm run dev
```
## Project structure
```
cmd/desktop/ Wails v3 desktop app (macOS + Linux)
main.go Thin shell: webview, system tray, notifications, auto-update
wails.json Wails project config
entitlements.plist macOS code-signing entitlements
cmd/oikos/ Single-binary entry point
cmd/nomos/ Nomos MCP client gateway
cmd/webhook/ Gitea deploy-webhook receiver (push-to-deploy on mac-mini)
internal/ All Go packages
httpapi/ REST + MCP server (OpenAPI-generated)
mcp/ MCP tool implementations
db/ Connection pool, migrations, seeds, sqlc queries
scheduler/ Observe loop, probes, signals
actuator/ SSH execution
learning/ Pattern recognition, anomaly detection
notifier/ Matrix notifications, approval tokens
policy/ Risk classifier
secrets/ Infisical + SOPS backend
domain/ Core types: entities, approvals, signals, patterns
ontology/ Type hierarchy, relationship validation
knowledge/ Knowledge YAML seed ingestion
web/ Control-room SPA (Svelte 5) — standalone, not embedded
in the oikos binary; see plans/2026-07-12-wails-desktop-app.md
api/openapi.yaml API contract — the source of truth for endpoints
migrations/ Forward-only SQL migrations (TimescaleDB)
seeds/ Bootstrap YAML: ontology, inventory, policy, knowledge
compose/ Dockerfiles + Caddy config
scripts/ Deploy, watchdog, rollback
checks/ Host health-check scripts run over SSH by the scheduler
tools/ Client auto-setup scripts (checks)
nomos/ Nomos config, persona, skills
.agents/ Agent instruction files + skills
plans/ Design documents
docs/adr/ Architecture decision records
docs/operations/ Runbooks (rollback, etc.)
```
## Commands
| Command | Purpose |
|---------|---------|
| `make build` | Build `oikos` binary |
| `make test` | Run all tests with race detection |
| `make test-db` | Run integration tests against compose Postgres |
| `make lint` | `go vet` + `golangci-lint` |
| `make generate` | Regenerate OpenAPI + sqlc code |
| `make generate-check` | CI drift guard — fail if generated code is stale |
| `make migrate` | Apply DB migrations |
| `make seed` | Ingest seeds into DB |
| `make export` | Export DB state to YAML seeds |
| `make dev` | Start compose dev stack |
| `make clean` | Remove binary + test cache |
| `make ui` | Build the SPA (`web/dist/`) |
| `make deploy-ui` | Build + deploy the SPA to the Caddy host |
| `make desktop` | Build the Wails desktop app for the current platform |
| `make desktop-package` | Build + package (zip on macOS, tar.gz on Linux) |
| `make webhook` | Build `cmd/webhook` (deploy-webhook receiver) |
| `make tidy` | `go mod tidy` |
## Conventions
### APIs are OpenAPI-first
The REST API is defined in `api/openapi.yaml`. Server code is generated with
`oapi-codegen` into `internal/httpapi/gen/`. To add an endpoint:
1. Add the path + schema to `api/openapi.yaml`
2. Run `make generate`
3. Implement the handler in `internal/httpapi/impl.go`
4. Add tests in `internal/httpapi/api_test.go`
Never hand-edit `internal/httpapi/gen/api.gen.go`.
### Database access is sqlc-first
SQL queries live in `internal/db/queries/*.sql`. Go code is generated with
`sqlc` into `internal/db/sqlcgen/`. Config in `sqlc.yaml`.
- Queries target pgx/v5 with UUID + timestamptz overrides
- Never hand-edit generated sqlc code
### Migrations are forward-only
SQL migrations live in `migrations/` as `NNN_name.up.sql`. There are no down
migrations (see [ADR 0008](docs/adr/0008-forward-only-migrations.md)).
Migrations are idempotent where possible (`IF NOT EXISTS`, `DO $$` blocks).
To add a migration:
1. Create `migrations/NNN_name.up.sql` with the next sequence number
2. Write the DDL
3. Run `make migrate` to apply
### Seeds are DB-generated
`seeds/*.yaml` are the bootstrap files used by `oikos seed`. After making
changes via the API, run `make export` to regenerate the seed files. These
files are version-controlled and serve as DR fallback.
### Writing style
Follow [.agents/shared/writing-style.md](.agents/shared/writing-style.md).
Documentation is reference prose, not marketing. Banned vocabulary includes
"robust", "seamless", "leverage", "utilize", "delve", "cutting-edge".
### Risk classification
Every mutation is classified against `seeds/policy.yaml` before execution.
Four risk classes: `read_only`, `reversible_low`, `config_mutation`,
`destructive`. The classifier can only lower autonomy relative to policy,
never raise it. When in doubt, escalate.
## CI
Gitea Actions runs on push to `main` and pull requests (`ci.yml`):
1. `go vet` + `golangci-lint` + `govulncheck`
2. Generated code drift check (`make generate-check`)
3. Build (`go build ./...`)
4. Test with race detector + coverage
5. Docker build verification (no push)
Coverage gates: policy + learning packages ≥ 80%, others ≥ 60%.
## PR workflow
1. Create a branch from `main`
2. Make changes, write tests
3. Run `make lint test generate-check`
4. Commit with a message following: problem → change → risk → verification
5. Push to Gitea; CI gates PRs on green
## Secrets
Secrets are managed by Infisical (primary) with SOPS as DR fallback. Never
hardcode secrets. Use environment variables from `.env` for local dev.
The `.env` and `.infisical-credentials` files are gitignored.
## Related
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
- [CLIENTS.md](CLIENTS.md) — for homelab clients consuming Oikos
- [docs/adr/](docs/adr/) — architecture decision records

81
Makefile Normal file
View File

@@ -0,0 +1,81 @@
.PHONY: build webhook test test-db lint generate generate-check dev migrate seed export clean tidy ui desktop desktop-package desktop-release
BINARY := oikos
GO ?= go
build:
$(GO) build -o $(BINARY) -tags timetzdata ./cmd/oikos
webhook:
$(GO) build -o webhook -tags timetzdata ./cmd/webhook
test:
$(GO) test -race -cover ./...
# Integration tests against the compose Postgres (starts it if needed)
test-db:
docker compose up -d postgres
@sleep 3
OIKOS_TEST_DATABASE_URL="postgres://oikos:$${OIKOS_DB_PASSWORD:-oikos_dev}@localhost:5432/oikos?sslmode=disable" \
$(GO) test -race -count=1 ./internal/db/ ./internal/httpapi/ ./internal/mcp/
lint:
$(GO) vet ./...
@command -v golangci-lint >/dev/null 2>&1 && golangci-lint run || echo "golangci-lint not installed, skipping"
generate:
$(GO) run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.4.1 \
-config api/codegen.yaml api/openapi.yaml
$(GO) run github.com/sqlc-dev/sqlc/cmd/sqlc@v1.29.0 generate
# CI drift guard: regenerate and fail if the committed output changed.
generate-check: generate
@git diff --exit-code -- internal/httpapi/gen internal/db/sqlcgen \
|| (echo "generated code is stale — run 'make generate' and commit" && exit 1)
migrate:
$(GO) run ./cmd/oikos migrate
seed:
$(GO) run ./cmd/oikos seed
export:
$(GO) run ./cmd/oikos export
dev:
docker compose --profile dev up -d
# Local sanity-check build of the SPA. Not embedded in the oikos binary
# (plans/2026-07-12-wails-desktop-app.md 0.1) — deploys as its own
# container (compose/web/Dockerfile) via `docker compose --profile full
# up -d web`, same push-to-main pipeline as everything else.
ui:
cd web && npm run build
desktop: ui ## Build the Wails desktop app for the current platform
rm -rf cmd/desktop/frontend/dist
mkdir -p cmd/desktop/frontend/dist
cp -r web/dist/* cmd/desktop/frontend/dist/
cd cmd/desktop && wails3 build -clean
desktop-package: desktop ## Build + package the desktop app (zip on macOS, tar.gz on Linux)
@case $$(uname -s) in \
Darwin) \
cd cmd/desktop/build/bin && zip -r oikos-desktop-darwin-$$(uname -m).zip oikos-desktop.app ;; \
Linux) \
cd cmd/desktop/build/bin && tar czf oikos-desktop-linux-$$(uname -m).tar.gz oikos-desktop ;; \
esac
@echo "Package: cmd/desktop/build/bin/"
desktop-release: ui ## Build desktop app for macOS arm64 + Linux amd64 (CI target)
@echo "Use 'make desktop-package' for local builds; desktop-release is for CI"
@exit 1
clean:
rm -f $(BINARY)
rm -rf cmd/desktop/build
rm -rf cmd/desktop/frontend/dist
$(GO) clean -testcache
tidy:
$(GO) mod tidy

287
README.md
View File

@@ -1,190 +1,161 @@
# Homelab OS
# Oikos
Living documentation for the **hubris** Proxmox homelab + Oikos operating system.
Agentic homelab operating system written in Go. Single binary (`cmd/oikos`),
Docker-deployed on mac-mini, with a standalone Nomos MCP agent gateway
(`cmd/nomos`). Manages the **hubris** Proxmox homelab autonomously — observes
state, classifies actions against policy, executes approved procedures over SSH,
learns from outcomes, and escalates when uncertain.
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md), then [OIKOS.md](.agents/OIKOS.md).
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md).
**For client machines:** see [CLIENTS.md](CLIENTS.md).
**For developers:** see [CONTRIBUTING.md](CONTRIBUTING.md).
---
## Quick start
## For Agents — Navigation & Entry Points
```bash
# Dev stack (postgres + api + scheduler + notifier). The api/nomos
# services need a shared token — every route requires a real bearer
# credential, there's no dev-open bypass.
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile dev up -d
### You are running on a client enrolled in the hubris homelab
# Full stack (adds Nomos agent gateway)
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile full up -d
1. **First:** Read [AGENTS.md](AGENTS.md) once. It explains who you are, the topology, available tools, conventions, and how to act.
2. **Before any mutation:** Read [OIKOS.md](.agents/OIKOS.md). It defines the operating model, risk classes, approval flow, and the ontology you'll consult.
3. **For specific workflows:** Load the matching skill from `.agents/skills/<name>/SKILL.md` (e.g., [service-health-check](.agents/skills/service-health-check/SKILL.md)).
4. **When in doubt:** Use MCP tools (`search_docs`, `get_page`, `explain`, `get_changelog`) — they're cheaper and more reliable than grepping.
# Build standalone binary
go build -o bin/oikos -tags timetzdata ./cmd/oikos
### Key References for Agents
# Run all roles in one process (dev mode)
OIKOS_DATABASE_URL="postgres://oikos:oikos_dev@localhost:5432/oikos?sslmode=disable" \
OIKOS_API_TOKEN=dev-token \
go run ./cmd/oikos all
- **What am I?** → `/opt/homelab-context/hosts/<hostname>.yaml` (read on first run)
- **Live topology** → `inventory.yaml` + `hosts/*.yaml` (canonical, always wins)
- **Risk & approval** → [oikos/policy.yaml](oikos/policy.yaml) (enforced, not advisory)
- **Runbooks & workflows** → [.agents/skills/](.agents/skills/) (risk class + verification checklist included)
- **State of Oikos** → [OIKOS.md build status](.agents/OIKOS.md#build-status-30-day-roadmap) (scheduled probes, drift detectors, signals, approval engine)
### When to Use MCP vs Files vs Shell
| Task | Use | Tool |
|------|-----|------|
| Resolve hostname → address | MCP | `get_host(name)` or `list_services()` |
| Search wiki by content | MCP | `search_docs(query)` |
| Read a wiki page | MCP or file | `get_page(path)` or `cat knowledge/wiki/.../...md` |
| Get changelog entries | MCP | `get_changelog(page, since?)` |
| Understand a service | MCP | `explain(service)` — compact context card, cheaper than search+read |
| Blast-radius query | MCP | `get_relations(entity)` (ontology walk) |
| List available secrets | MCP | `list_my_secrets()` (scoped to your age key) |
| Browse or grep | File | Raw `grep` when MCP unreachable, or exploratory browsing |
**When MCP is unreachable:** fall back to grepping the clone at `/opt/homelab-context/`. The local files are the same; MCP is just an index.
---
## Understanding the Operating Model
Before you act, **classify your action against [oikos/policy.yaml](oikos/policy.yaml)**.
### The Oikos OODA Loop + Decision Tree
```mermaid
flowchart TD
Observe["**Observe**<br/>probes, drift detectors, agent signals"]
Orient["**Orient**<br/>ontology, context, state, entity relations"]
Decide{"**Decide**<br/>classify against oikos/policy.yaml"}
Auto["Auto-act<br/>(unattended)"]
Escalate["Escalate<br/>homelab approval request"]
Act["**Act**<br/>homelab CLI, runbooks, skills"]
Verify["**Verify**<br/>checklist from SKILL.md"]
Ledger["**Ledger**<br/>mutation record: who/what/risk"]
Document["**Document**<br/>wiki update, same-session rule"]
Observe --> Orient --> Decide
Decide -->|read_only, reversible_low| Auto
Decide -->|config_mutation, destructive| Escalate
Auto --> Act
Escalate -->|approval granted| Act
Act --> Verify --> Ledger --> Document
Document -.loop.-> Observe
# Control-room SPA (separate from the Go binary — see web/)
cd web && OIKOS_API_TOKEN=dev-token npm run dev # http://localhost:5173
```
### Risk Classes (enforced, not advisory)
## Architecture
From [oikos/policy.yaml](oikos/policy.yaml):
```
┌──────────────────────────────────┐
│ mac-mini (Docker) │
│ │
Workstation ─── │ nomos (8092) ──MCP── api (8090) │
(mesh) │ MCP gateway REST + MCP │
│ │
│ scheduler ── notifier ── postgres │
│ (observe) (Matrix) (Timescale)│
└──────────────────────────────────┘
```
- **read_only** — status, logs, docs, inventory queries. Unattended. MCP tools are all read_only.
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger entry.
- **config_mutation** — tracked-config edits (commit+push, never local), deploys, upgrades, DNS/ingress changes. **Operator approval required.**
- **destructive** — destroy, format, wipe, rotate, revoke. **Approval + typed confirmation phrase.**
| Component | Port | Role |
|-----------|------|------|
| `oikos api` | 8090 | REST API + MCP server (15 tools) |
| `oikos scheduler` | — | Probe runner, signal lifecycle, metrics |
| `oikos notifier` | — | Approval tokens, Matrix alerts |
| `nomos serve` | 8092 | MCP client gateway, query routing |
### Decision Flow
## Phases
1. **Decide:** Use `homelab decide <action> <entity>` to classify (risk class × blast radius × confidence).
2. **Escalate if needed:** `homelab approval request` (Matrix-delivered to operator; see [operations/commands.md](.agents/operations/commands.md)).
3. **Execute:** Use `homelab` CLI (not ad-hoc SSH) — it enforces policy, logs mutations, and verifies outcomes.
4. **Document:** Update wiki in the same session (per [AGENTS.md §5](AGENTS.md#5-acting-on-the-homelab) and the [same-session rule](.agents/shared/page-templates.md#same-session-update-rule)).
| Phase | Status | Description |
|-------|--------|-------------|
| 1 — Ontology + DB | ✅ | TimescaleDB, migrations, seeds, blast_radius |
| 2 — API | ✅ | OpenAPI-first REST + MCP, auth, SSE, audit |
| 3 — Control loop | ✅ | Scheduler, actuator, learning, classifier, notifier |
| 4 — Nomos agent | ✅ | Standalone MCP client gateway, agent activity |
| 5 — Secrets | ✅ | Infisical backend + SOPS fallback, rotation runbooks |
| 6 — Deploy | ✅ | CI pipeline, cutover checklist, watchdog, rollback |
### The Ontology Graph
Full plan: [plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md).
Everything that can break, be changed, or hold data has an entity in `inventory.yaml` + `oikos/ontology.yaml`. Blast-radius questions ("what breaks if strong goes down?") are graph walks via `homelab node <name> relations`, not doc archaeology.
## Operations
**See:** [OIKOS.md](.agents/OIKOS.md) (full operating model, OODA loop, primitives, lifecycle gates, build status).
### API endpoints
---
```bash
curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
http://localhost:8090/api/v1/entities?type=service # fleet
curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
http://localhost:8090/api/v1/health # fleet health
curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
http://localhost:8090/api/v1/agent-activity # agent log
```
## Finding & Understanding Information
### Nomos queries
The narrative documentation is organized in **layers**:
```bash
# Structured tool call
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
-d '{"tool":"get_blast_radius","args":{"entity_id":"service:authentik"}}'
| Layer | What it is | Where | Immutable? | How agents use it |
|-------|-----------|-------|-----------|-------------------|
| **Sources** | Raw evidence: incidents, external refs, live state | `knowledge/sources/investigations/` | Yes | Read to understand root causes; do not rewrite |
| **Wiki** | Synthesized current-state: one page per node & per system | `knowledge/wiki/{containers,hosts,vms,infrastructure}/` | No | This is the reference layer — if wiki disagrees with live state, update it *in the same session* |
| **Index** | Pure listings — every page in scope with one-line summary | `index.md` / folder `README.md` | No | Navigation aid; keep it current when wiki restructures |
| **Log** | Append-only doc-maintenance record (restructures, ingests, lints) | `knowledge/log.md` | Yes (append-only) | Read to understand past doc changes; never edit directly |
# Natural language
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
-d '{"query":"what depends on authentik?"}'
```
**Changelog ≠ Log:** Each wiki page ends with a `## Changelog` (infrastructure changes to that node, machine-parsed). That's not the Log; the Log records *doc operations* only.
### CLI
**See:** [llm-wiki.md](.agents/shared/llm-wiki.md) (full rules, page structure, immutability contract).
```bash
oikos migrate # apply DB migrations
oikos seed # ingest ontology/inventory/policy seeds
oikos export # export DB state to YAML
oikos api # serve REST + MCP
oikos scheduler # run observe loop
oikos notifier # run notification loop
oikos all # all roles in one process
oikos secret list # enumerate SOPS secrets
oikos secret migrate # SOPS → Infisical
```
---
### Web UI
## Map & Quick Navigation
`web/` is a standalone Svelte 5 SPA — not embedded in the `oikos` binary, not
part of `docker-compose.yml`. It talks to `api`/`nomos` over HTTP with a
bearer token entered on first launch (see `web/src/pages/Config.svelte`).
Build with `make ui`, deploy with `make deploy-ui` (Caddy serves the static
output). A native desktop wrapper is planned — see
[plans/2026-07-12-wails-desktop-app.md](plans/2026-07-12-wails-desktop-app.md).
### Agent Entry Points (Start Here)
## Repo layout
- **You are an agent** → [AGENTS.md](AGENTS.md) (on deployed clients: `/opt/homelab-context/AGENTS.md`)
- **Operating model & risk policy** → [OIKOS.md](.agents/OIKOS.md)
- **Specific workflows** → [.agents/skills/](.agents/skills/) (load the matching SKILL.md before acting)
- **Operations cheatsheet** → [.agents/operations/commands.md](.agents/operations/commands.md)
- **Tools & MCP reference** → [AGENTS.md §3 — The MCP server](AGENTS.md#3-the-mcp-server)
```
cmd/oikos/ Go entry point — single binary
cmd/nomos/ Nomos MCP client gateway
cmd/webhook/ Gitea deploy-webhook receiver (push-to-deploy on mac-mini)
internal/ Go packages (httpapi, mcp, scheduler, actuator, learning,
notifier, policy, secrets, db, config, ontology, domain,
knowledge)
web/ Control-room SPA (Svelte 5) — standalone, not embedded
api/openapi.yaml API contract (OpenAPI 3.1)
migrations/ Forward-only SQL migrations (TimescaleDB)
seeds/ Bootstrap YAML (ontology, inventory, policy, knowledge)
compose/ Dockerfiles + Caddy config
scripts/ Deploy, watchdog, verification, rollback
checks/ Host health-check scripts run over SSH by the scheduler
tools/ Client auto-setup scripts (checks)
ssh/ Deploy keys + authorized_keys management
vps/ Caddy/TURN config templates for the netbird VPS
nomos/ Nomos config, persona, skills
.agents/ Agent instruction files, shared conventions, skills
archive/ Historical reference (legacy wiki, plans, SOPS backups)
plans/ Design documents (active + done)
docs/adr/ Architecture decision records
docs/operations/ Runbooks (rollback, etc.)
```
### Topology & Infrastructure
## For agents
Node counts, IPs, and service lists change often — treat `inventory.yaml` and the index pages below as the source of truth, not this README.
See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference:
- **Proxmox hosts** → [knowledge/wiki/hosts/index.md](knowledge/wiki/hosts/index.md)
- **VMs** → [knowledge/wiki/vms/index.md](knowledge/wiki/vms/index.md)
- **LXC containers** → [knowledge/wiki/containers/index.md](knowledge/wiki/containers/index.md)
- **Cross-cutting infrastructure** (DNS, ingress, mesh, backups, monitoring, auto-deploy, VPS) → [knowledge/wiki/infrastructure/index.md](knowledge/wiki/infrastructure/index.md)
- **Source of truth:** DB (runtime) then seeds (bootstrap). Old wiki is
archived at `archive/knowledge/` — use MCP `search_knowledge` instead.
- **Mutations:** classify against policy, request approval for
`destructive`/`config_mutation`
- **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode
### Knowledge & References
## Related
- **Glossary** — [GLOSSARY.md](knowledge/GLOSSARY.md)
- **Incidents & investigations** — [knowledge/sources/investigations/index.md](knowledge/sources/investigations/index.md) (active + [archive](knowledge/sources/investigations/archive/))
- **Plans & design docs** — [plans/index.md](plans/index.md)
- **Hermes agent** (for Hermes-enrolled clients) — [HERMES.md](.agents/HERMES.md)
---
## Conventions
All pages follow:
- **File naming.** Foundational docs (entry-points, agent instruction, references) are ALL-CAPS (`AGENTS.md`, `OIKOS.md`, `GLOSSARY.md`); containers use `<id>-<name>.md`; infrastructure pages use lowercase-with-dashes; plans and incidents use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`. See [page-templates.md](.agents/shared/page-templates.md#file-naming) for the full rules.
- **Voice & vocabulary.** Concise, technical, sysadmin-to-sysadmin. No marketing prose, no puffers (seamless, robust, leverage, etc.). Full rules in [writing-style.md](.agents/shared/writing-style.md).
- **Cross-linking is mandatory.** If a page references a node or system, link to it. Treat orphans as a bug.
- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running state, fix the wiki *and* add a changelog entry *in the same session*.
- **Tracked configs.** Pages for configs living in git repos (Caddy, Gitea, Artifacto, mule-image) must note the repo. Edits go through commit+push, never local changes. See [auto-deploy](knowledge/wiki/infrastructure/auto-deploy.md).
- **No secrets.** This is a private repo, but still: reference secret *paths*, never secret *values*.
**For agents:** Read [caveman.md](.agents/shared/caveman.md) (terse communication standard). Use templates at [page-templates.md](.agents/shared/page-templates.md) when creating pages.
---
## Updating the Wiki
### When You Change Infrastructure
1. Update the relevant page (config snapshot, ports, mounts, IP address).
2. Add a `### YYYY-MM-DD — title` entry to the page's `## Changelog` section (reverse chronological order).
3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link from the changelog.
4. If it's an incident, add a record to [`knowledge/sources/investigations/`](knowledge/sources/investigations/index.md).
### When You Restructure the Wiki
1. Update the relevant `index.md` / `README.md` in that section.
2. Add a single-line entry to [`knowledge/log.md`](knowledge/log.md): `## [YYYY-MM-DD] <operation> | <summary>` (e.g., `## [2026-07-06] restructure | split infrastructure/dns into dns.md + dns-advanced.md`).
### The Same-Session Update Rule
**Any meaningful state change made in this session requires a wiki update before the session closes.** A change that touches a container page must also update:
- The `containers/index.md` table (IPs, host, mounts, status)
- The root `README.md` table (if affected)
- The Caddy page site list (if affects `*.hubris.network` routing)
- The DNS / ingress infrastructure pages (if affects routing)
- The `hosts/hubris.md` or `hosts/strong.md` page (if container count changes)
- The `inventory.yaml` host entry (source of truth for `hosts/*.yaml` generation)
- The `knowledge/wiki/infrastructure/topology.md` (regenerate if needed)
Not updating all linked places is a bug. See [page-templates.md — same-session update rule](.agents/shared/page-templates.md#same-session-update-rule).
---
## More Information
- **For Hermes agents** → [HERMES.md](.agents/HERMES.md) (persona, source-of-truth hierarchy, token efficiency)
- **For manual workflows** → [.agents/operations/](.agents/operations/) (commands cheatsheet, agent enrollment, Hermes guide)
- **For skills/runbooks** → [.agents/skills/](.agents/skills/) (load the matching SKILL.md before acting; includes risk class + verification)
- **MCP tools** → [AGENTS.md §3](AGENTS.md#3-the-mcp-server) (available tools, when to use MCP vs files)
- **Page templates & voice** → [.agents/shared/](.agents/shared/) (page-templates.md, writing-style.md, caveman.md, llm-wiki.md)
- **Machine-readable substrate** → `inventory.yaml`, `oikos/policy.yaml`, `oikos/ontology.yaml` (not part of the wiki; see [llm-wiki.md](.agents/shared/llm-wiki.md#rules))
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
- [CLIENTS.md](CLIENTS.md) — client onboarding guide
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide
- [plans/](plans/) — design documents and cutover checklist
- [docs/adr/](docs/adr/) — architecture decision records

8
api/codegen.yaml Normal file
View File

@@ -0,0 +1,8 @@
# oapi-codegen config — `make generate` regenerates internal/httpapi/gen.
package: gen
output: internal/httpapi/gen/api.gen.go
generate:
models: true
chi-server: true
strict-server: true
embedded-spec: true

3303
api/openapi.yaml Normal file

File diff suppressed because it is too large Load Diff

7
api/redocly.yaml Normal file
View File

@@ -0,0 +1,7 @@
# Redocly lint config for api/openapi.yaml (CI runs: redocly lint api/openapi.yaml)
extends:
- recommended
rules:
# Every operation declares `default` → RFC 9457 problem+json instead of
# enumerating each 4XX (plan R3-3); oapi-codegen handles `default` fine.
operation-4xx-response: off

View File

@@ -45,7 +45,16 @@ Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webh
### Portainer
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
### homelab-mcp (`/opt/homelab-mcp/`)
### homelab-mcp (`/opt/homelab-mcp/`) — DEPRECATED (Go rewrite, Phase 6)
> **Status:** This Python MCP server is being replaced by the Go `oikos api` binary
> running in Docker on mac-mini. Cutover pending — see
> [scripts/cutover-checklist.md](../../scripts/cutover-checklist.md) for the
> execution plan. The Go MCP uses the official MCP Go SDK (Streamable HTTP, not
> FastMCP) with 15 tools including `get_blast_radius`, `request_execution`, and
> `get_agent_activity`. Source: `internal/mcp/server.go`.
**Current (Python) implementation — DO NOT MODIFY, awaiting cutover:**
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
`/opt/homelab-context/` (this LXC is itself an enrolled
[homelab-context](../infrastructure/homelab-context.md) client). Source
@@ -70,7 +79,10 @@ the server code). Listens on `0.0.0.0:9811`, secret in
`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
restarts on push.
### secrets-issuance (`/opt/secrets-issuance/`)
### secrets-issuance (`/opt/secrets-issuance/`) — DEPRECATED (Go Phase 5)
> **Status:** Replaced by `internal/secrets/` in the Go rewrite. Machine identities
> are now managed via Infisical (`docker compose --profile infisical up`).
Tiny HTTP service that issues per-client age keypairs the first time
each client calls `/issue`. Idempotent: subsequent calls return the
same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in

View File

@@ -41,8 +41,8 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 (deprecated) | ~~reinstalls `homelab-mcp.service` + restart~~ → replaced by Go Docker stack on mac-mini |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 (deprecated) | ~~reinstalls `secrets-issuance.service` + restart~~ → replaced by `internal/secrets/` Go package |
| `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` |
| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) |

View File

@@ -56,7 +56,7 @@ are documented in their own pages. Each system below links to its full doc.
- **[Topology diagram (generated)](topology.md)** — Mermaid graph of compute,
ingress routing, and storage mounts. Auto-generated from `inventory.yaml`
by `oikos/gen-topology.py`.
by `oikos/gen-topology.py` (Python — Go DB-native replacement planned).
## Related

Some files were not shown because too many files have changed in this diff Show More