384 Commits

Author SHA1 Message Date
8657ac5669 feat(web): open tasks/sessions as floating windows with independent live chat
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Clicking a task now opens it as a wmkit floating window (like entity
windows already do) instead of navigating away from wherever you were.
Several task windows can be open and actively streaming at once, each
fully independent — no "which one's on screen" guard needed, since
each window owns its own store bundle:

- chat.ts: chatFor(sessionId)/loadSessionChat/sendSessionMessage give
  each window its own messages/streaming/connectionState, alongside
  the existing singleton path the main Chat page still uses unchanged.
- workspace.ts: same split for plan/questions/touched/health-diffs
  (workspaceFor/startSessionWorkspace), each with its own live-event
  watermark since several windows can watch the same event stream.
- activity.ts: activityLogFor(sessionId) mirrors the global derivation.

SessionGraph.svelte, OperatorQuestion.svelte, and ActivityTimeline.svelte
were converted from store-importing to prop-driven (matching the new
ChatThread.svelte, extracted from Chat.svelte's transcript/input so both
the main page and task windows share one implementation instead of
duplicating markup/styling) so each can render either the global
"current session" or a specific window's session.

Also: minimized-window taskbar chips now cap at a max width with
middle-ellipsis truncation instead of growing unbounded, and the
window header's title/action-button row is fixed to genuinely match
heights (not just share a center point) for more robust alignment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-19 17:01:34 +02:00
e28e0e9ea3 feat(web): unify Knowledge Base filtering into one type multiselect
Replace the Fleet/Network/Identity/Knowledge category tabs (which
scoped entity fetches server-side) with a single "Types" multiselect
shared by both the table and graph views — both now fetch the whole
entity set (paginated via the new fetchAllEntities) and filter
client-side, defaulting to fleet's types. Table and graph also share
one search/highlight field instead of two separately-labeled ones.

Along the way, fixed a real bug the wider entity set exposed: the
treegrid's parent/child grouping fired one fetchGraph call per
candidate root entity, fine for the old ~50-entity fleet scope but an
ERR_INSUFFICIENT_RESOURCES flood once scoped to the full ~1700-entity
set. Replaced with a single whole-graph fetch, deriving parent/child
pairs from its edges client-side.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-19 17:01:34 +02:00
6051fb4845 feat(web): curved edges + unique SVG ids for concurrent graph views
Quadratic-bezier edges instead of straight lines, and drop the
auto-refit-on-load that caused a jarring zoom/pan snap once the force
simulation settled. Also namespace each graph's dot-grid pattern id
with a per-instance uuid — multiple SessionGraph instances can now be
mounted at once (one per open task window), and duplicate SVG ids
silently blanked out every graph's background but the first.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-19 17:01:34 +02:00
544afae77f feat(nomos): retry cap, vm: targets, inspect_path, goal supersession, runbooks
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Session-review implementation for the three sessions audited in
plans/2026-07-18-session-review-three-sessions.md. v0.7.11 → v0.7.12.

P0.1 — retry cap + investigate-before-retry (cmd/nomos/retrycap.go,
agent.go): after 3 identical failing run calls in a single turn, refuse
to dispatch the call again and return a directive to investigate *why*
(ps/strace/lsof) or surface the blocker. Per-turn scope so a fresh turn
after the operator responds can retry once more. Session 1e9c7691's 20+
identical chown retries (knfsd held a kernel lock on the exported NFS
dir) is the direct motivation.

P0.2 + P1.8 + P2.10 — SOUL.md guidance: hung command is not a failed
command (investigate before retry); ask before proposing a multi-step
migration; multi-goal sessions summarize the arc not just the last goal.

P1.3 — two new runbook entities in seeds/knowledge.yaml:
  - nfs-exported-dir-mutation-hang (the knfsd fchownat lock procedure:
    killall → exportfs -u → mutate → exportfs -a → verify)
  - netbird-mgmt-oidc-race-after-upgrade (docker restart netbird-mgmt
    after ~30s for the traefik/authentik OIDC race)

P1.4 — setGoal emits task.superseded event when prior goal is overwritten
by a different goal (store.go, TestSetGoal_SupersededEvent). Session
55927f0a had two set_goal calls with the first silently abandoned.

P1.5 — inspect_path MCP tool: runs mount/df/ls/stat for one path across
up to 8 targets in one parallel call, replacing the 15+ run-call
fact-gathering fan-out sessions 1 and 2 each spent on cross-target path
tracing (tools.go, server.go: inspectPathAcrossTargets, inspectOneTarget).

P1.6 — vm: target support in run via qm guest exec (no more SSH-hop
with nested quoting). Extracted shared resolveProxmoxHostSlug for
LXC + VM, with hosts-relationship fallback when attributes.host is
absent (server.go, tools.go). Session 55927f0a's SSH-hop workarounds
for vm:zimaos are the direct motivation.

Deferred (documented in plan): P1.7 (approval window auto-extend on
timeout) and P2.9 (long-running command PENDING detection) — both
addressed at lower cost by the retry cap. Session 3's poll-after-timeout
pattern already works; the cap protects against the failure mode.
2026-07-19 00:09:39 +02:00
bd44626532 feat(web): floating entity-detail windows (wmkit), replacing sidebar/sheet
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Every place that showed entity detail (Knowledge Base's right sidebar,
the EntitySheet drawer used by Knowledge and the chat session graph,
the standalone /entity/:slug page) now opens the entity in its own
floating, draggable, resizable window instead — several can be open
side by side, and clicking a relation inside one opens another,
building up a stack. Windows are managed by one global wmkit instance
(new $lib/stores/windows.ts + $lib/components/EntityDesktop.svelte,
mounted once in App.svelte), themed with the app's own card/border/ring
tokens rather than wmkit's bundled themes (app.css).

- Delete EntitySheet.svelte (redundant) and the KnowledgeBase resizable
  detail pane; row/graph-node click handlers now call
  openEntityWindow(slug) instead of setting local sidebar state.
- SessionGraph (chat's "Scope" mini-graph): clicking a node opens its
  window directly instead of a click-through mini-detail panel with
  its own resize handle and "Full detail" button — that whole
  subsystem is now dead and removed. Node highlight ring is kept
  (still useful to see what you last opened) and now clears itself via
  an effect watching the shared window-manager store, so closing a
  window drops the highlight instead of leaving it pointing at nothing
  — same fix applied to Knowledge Base's row highlight.
- Compact the entity-detail panel's padding (container + each
  DetailSection) now that it's typically viewed in a small window
  rather than a full-height sidebar.
- Fix KnowledgeBase's browse pane losing its flex-1/min-w-0 (and thus
  full width) when the wrapping single-child div around it was removed
  along with the old detail-pane split.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 10:46:27 +02:00
b0cdf64bbf fix(web): vite-env.d.ts missing vite/client types reference
import.meta.env (used by main.ts's dev-token auto-config) was untyped
since that landed — vite-env.d.ts never referenced Vite's client types.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 10:46:04 +02:00
d6b3d3c88b fix(web): relation rows wrap and break layout on long node names
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The clickable-button variant of relationRow was missing the truncate
class that the read-only span fallback already had — long slugs (task
UUIDs, exec IDs) rendered at their full pre-truncated length inside a
shrink-only flex item, overflowing the narrow detail sidebar and
wrapping to extra lines. Give both sides flex-1 + truncate so they
share the row's width evenly and always stay on one line.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 08:20:40 +02:00
8615f2268f fix(web): entity Relations panel was missing true incoming edges
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
fetchGraph({root, depth:1}) is backed by blast_radius, which only
walks outgoing edges — so it could never surface an edge some other
entity points at this one (e.g. host:hubris —hosts→ lxc:sophia) unless
that other entity happened to also be reachable going forward from
here. The Outgoing/Incoming split was filtering correctly, but
"Incoming" was starved of data by construction.

Switch to GET /entities/{id}/relations?direction=both — a dedicated
endpoint that matches on source_id OR target_id directly — via a new
fetchEntityRelations(). Simplifies the incoming/outgoing derivation
too, since every relation returned is now actually incident to the
entity (no more sibling-edge filtering needed).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18 08:13:19 +02:00
258b14dcbc feat(web): ontology-driven fleet treegrid, relations grouping, dev auto-config
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Knowledge Base / Fleet browsing:
- EntityTable renders as a treegrid (arbitrary depth, expand/collapse,
  ARIA row/level/expanded), grouped by parent-child relationships
  derived entirely from the live ontology graph (cardinality ->
  direction; typeDepth specificity for ties) rather than a hardcoded
  relationship list — see loadFleetGrouping in KnowledgeBase.svelte.
- Fold Services and Storage categories into Fleet (services/pools/
  volumes/datasets now nest under the compute entity or pool that
  provides/contains them instead of having their own browsing tabs).
- Drop `cluster` entities from Fleet browsing so a host's `located-at`
  (site) relationship wins the tree-parent slot without needing a
  hardcoded priority override — member-of simply has no valid target
  left to point at.
- Add a "show destroyed/inactive" Switch (default off) filtering on
  entity.state, replacing an always-on checkbox.

Entity detail panel:
- Split the Relations section into Outgoing/Incoming groups (relative
  to the viewed entity), and scope the section's count to edges
  actually incident to it rather than the whole depth-1 neighborhood.

Dev experience:
- Auto-fill the SPA's token from the dev server's own OIKOS_API_TOKEN
  (vite.config.ts define + main.ts, dev-only, only when unconfigured)
  so the "Connect to Oikos" prompt doesn't reappear on every reload.
- .claude/launch.json: autoPort, since port 5173 is often already
  claimed by another worktree's dev server.

Adds ui/checkbox and ui/switch (bits-ui primitives, following the
existing shadcn-svelte wrapper pattern) and fetchOntology()/
RelationshipTypeDef to api.ts. Also fixes a missing types.ts import
in api.ts (ChatEvent/MessageContent) that predates this branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 23:26:04 +02:00
646373a676 fix(httpapi): GetGraph 500s when rel_type is omitted
req.Params.RelType is *[]string; passing the nil pointer straight
through as a pgx query arg (both in the blast_radius() call and in
ListGraphEdges) panics because pgx can't infer the array element type
from a nil *[]string, only from a concrete (possibly nil) []string.
Dereference once up front instead. Also affected the sqlc-based
ListGraphEdges path added by the R3 refactor, which had the same bug.

Add a regression test for GET /api/v1/graph?root=X&depth=N with no
rel_type.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 23:25:41 +02:00
69964abe2e chore: reconcile on-client path + add golangci-lint config (R13+R14)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
R13 — on-client path reconciliation:
- AGENTS.md: 6 occurrences of /opt/homelab-context/ → /opt/homelab/
  (sections 1, 2, 5, 7)
- .agents/NOMOS.md: 2 occurrences of /opt/homelab-context/ → /opt/homelab/
- CLIENTS.md already used /opt/homelab/ — now consistent across all docs.
  The repo is still named 'homelab-context' (git remote), it just clones
  to /opt/homelab/ on enrolled clients per CLIENTS.md.

R14 — golangci-lint/staticcheck/govulncheck tooling:
- .golangci.yml (new): config enabling govet, staticcheck, ineffassign,
  unused, errcheck, gosimple, typecheck, misspell, revive. Excludes
  generated code (internal/httpapi/gen/, internal/db/sqlcgen/) and
  relaxes errcheck in test files.
- Makefile: split 'lint' target into vet, golangci, govulncheck subtargets.
  Each checks if the tool is installed and prints install instructions
  if not. 'make lint' runs all three.
- CI already had golangci-lint-action + govulncheck (both advisory);
  the action auto-discovers .golangci.yml.
2026-07-17 23:09:47 +02:00
6806fac5fd feat(web): define ChatEvent discriminated union, eliminate all any sites (R9)
Created web/src/lib/types.ts with discriminated unions for SSE event
payloads: ChatEvent (7 variants: session, tool_use, tool_result,
text_delta, text, done, error), ToolCallResult, MessageContent, and
typed data shapes for live events (PlanProposedData, PlanStepEventData,
QuestionRaisedData, QuestionAnsweredData, EntityTouchedData,
HealthChangedData) plus WailsGlobal for the desktop bridge.

Replaced all ~15 `any` sites across 7 files:
- api.ts: Message.content any -> MessageContent | string; removed local
  ChatEvent interface (now imported from types.ts as a discriminated
  union); JSON.parse cast to ChatEvent.
- stores/chat.ts: removed local ToolCallResult interface (imported from
  types.ts, re-exported for backward compat); extractApprovals accesses
  args with typeof guards instead of implicit any access; toChatMessages
  handles string|object Message.content cleanly.
- stores/activity.ts: update_plan_step seq/status extracted via typeof
  guards instead of `as any` casts; toolActivityLabel uses a str() helper
  for safe string extraction from unknown args.
- stores/workspace.ts: applyPlanStepEvent takes PlanStepEventData;
  applyEvent casts data to Record<string, unknown>; switch cases cast to
  typed interfaces (PlanProposedData, QuestionRaisedData, etc.) instead
  of `as any`; applyHealthChanged uses HealthChangedData.
- Config.svelte: (window as any).wails -> typed WailsGlobal cast;
  catch (e: any) -> catch (e: unknown) with instanceof Error check.
- utils.ts: WithoutChild/WithoutChildren `any` -> `unknown`.
- vite.config.ts: authProxy proxy/proxyReq `any` -> ProxyOptions type.

Result: eslint no-explicit-any warnings dropped 12 -> 0. Tests (6/6) and
build pass. VERSION 0.7.10 -> 0.7.11. Plan R9 marked done.
2026-07-17 23:08:35 +02:00
7dc1c1ae39 docs: document the non-OpenAPI routes carve-out (R11)
10 routes are registered manually on the chi router in server.go rather
than generated from openapi.yaml. Added a 'Non-OpenAPI routes' comment
block at the top of NewHandler listing each route with its structural
reason for the carve-out:

  - Auth/infra: /healthz, /api/v1/auth/oidc-*, /oidc-callback — bypass
    auth middleware or aren't JSON API
  - SSE override: /api/v1/events/stream — re-registered for Flush()
  - Ad-hoc aggregations: /knowledge/recent, /knowledge/content/{id},
    /activity/recent, /activity/session/{id}, /learning/timeline,
    /learning/trend — derived shapes with no schema type yet

Updated .agents/dev/CONTRIBUTING.md §OpenAPI codegen with the carve-out
policy: if an ad-hoc route stabilizes, promote it to openapi.yaml with a
proper schema and migrate the serve* function to a strict handler.
2026-07-17 23:02:08 +02:00
8709e01dcb fix(web): eliminate all Svelte 5 runes-mode warnings (R10)
5 warnings → 0:

1. ActivityTimeline.svelte:103 — replaced deprecated <svelte:component
   this={icon}> with direct dynamic component rendering ({@const IconComp
   = icon}<IconComp />). In Svelte 5 runes mode, components are dynamic by
   default; <svelte:component> is unnecessary.

2. DetailSection.svelte:18 — 'let open = (defaultOpen)' captured only
   the initial value. Changed to (false) +  to sync with
   defaultOpen prop changes.

3. EntitySheet.svelte:10 — 'let currentSlug = (slug)' had the same
   issue. Changed to <string|null>(null) +  (the  was
   already there, now the initial value doesn't reference the prop).

4. theme.svelte.ts:23 — 'applyClass(current)' at module level referenced a
    variable, capturing only the initial value. Changed to apply the
   plain storedTheme() result for initialization; setTheme() already calls
   applyClass() on changes.

5. Chat.svelte:326 — unused CSS selector '.prose-chat
   :global(:first-child):is(h1,h2,h3)' replaced with explicit
   :global(> h1:first-child) etc. (the :first-child pseudo wasn't matching
   because the scoped wrapper div is the actual first child).

Build is now warning-free.
2026-07-17 22:58:08 +02:00
c96c795126 test: add unit tests for 6 previously-untested packages (R7)
Added pure unit tests for all packages that had 0% coverage. Where pure
logic was entangled with DB calls, extracted testable helpers first.

internal/domain (0% -> 100%):
- TestIsNil, TestCanTransition (all 30 state transitions), TestSentinelErrors,
  TestSignalTransitionsComplete

internal/learning (0% -> 26.2%):
- Refactored processGroup to extract 4 pure helpers: countOutcomes,
  computeConfidence, shouldValidate, shouldQuarantine
- TestWilsonLowerBound (monotonicity, edge cases, sample-size cap)
- TestCountOutcomes, TestComputeConfidence, TestShouldValidate,
  TestShouldQuarantine (table-driven)
- Remaining gap: extractPatterns/processGroup DB calls need make test-db

internal/policy (39% -> 50%):
- Extracted determineRoute from ClassifySignal (pure route logic)
- TestDetermineRoute (7 cases covering global/entity kill-switches, approval)
- Remaining gap: ClassifySignal/computeBlastRadius need DB mock

internal/knowledge (0% -> 14.2%):
- TestContentHash, TestStr, TestStrSlice, TestMapVal, TestToPGArray
- Documented latent bug: toPGArray doesn't escape " or \\ in tags
- Remaining gap: ingest* functions need make test-db

internal/actuator (0% -> 14.7%):
- TestSSHErrorClassString, TestClassifySSHError (11 cases incl. net.Error mock)
- TestParseProcedure, TestSetDefaultSSHTimeout
- Circuit breaker full state-machine test (open/close/reset/per-target)
- Remaining gap: ExecuteProcedure/ProvisionLXC need SSH+DB fixtures

internal/scheduler (0% -> 7.3%):
- TestParsePingLatency (Linux/macOS formats), TestAllowlistedScript
- TestEvaluateSeverity (threshold logic, crit:0 skip, signalKind fallback)
- Remaining gap: checkHTTP/checkTCP need httptest; runCheckPass needs DB

internal/notifier (0% -> 6.4%):
- TestHashToken, TestGenerateApprovalToken (HMAC re-derivation)
- Remaining gap: checkReaction/sendMatrixAlert need httptest; DB funcs need
  make test-db

All tests pass with -race. domain hits its 60% gate at 100%. The remaining
packages need integration tests (make test-db) and/or httptest-based tests
to reach their coverage gates — tracked as follow-up.
2026-07-17 22:54:44 +02:00
463bdacf5c feat(web): add eslint + prettier + vitest toolchain + web CI job (R8)
Added to web/package.json devDeps: eslint (9, flat config) +
eslint-plugin-svelte + typescript-eslint + globals; prettier +
prettier-plugin-svelte; vitest (jsdom env) + jsdom. New scripts: lint,
lint:fix, format, format:check, test, test:watch.

Configs:
- web/eslint.config.js — flat config, TS + Svelte, browser/node globals,
  no-explicit-any as warn, unused-vars as error (ignores _-prefixed).
- web/.prettierrc.json — single-quote, 100 width, svelte parser override.
- web/.prettierignore — dist/node_modules/build/lockfiles.
- web/vite.config.ts — vitest test block via reference directive, jsdom env,
  globals enabled.

Sample test: web/src/lib/utils.test.ts (6 tests covering relativeTime,
truncateMiddle, debounce — all passing).

CI: new web job in .gitea/workflows/ci.yml (npm ci, check [advisory],
lint [advisory], format:check [advisory], test [gate], build [gate]).
Advisory steps use continue-on-error until the baseline is clean —
matching the existing golangci-lint advisory pattern.

Known baseline surfaced by the new toolchain (pre-existing, not caused
by R8): svelte-check 154 errors (133-file config cascade), eslint 126
errors + 12 warnings (unused vars, @html XSS, unused CSS), prettier 175
unformatted files. Fixing these is a follow-up cleanup.

VERSION 0.7.9 -> 0.7.10. Plan R8 marked done; C.1 updated.
2026-07-17 22:49:27 +02:00
fb39a48bef refactor: split phase3.go + extract MCP tool registry (R4)
internal/httpapi/phase3.go (2627 lines, 12+ resource domains) split into
15 per-resource files:
- actuator.go: SSH execution machinery (initSSH, sshExec, resolveRunTarget,
  executeApprovedAction, jsonErr, gatewayPreflightPassed, resolveTemplate)
- checks.go, classifications.go, executions.go, approvals.go, patterns.go,
  skills.go, approval_rules.go, autonomy.go, risk_classes.go,
  relationships.go, entity_types.go, metrics.go, agent_activity.go,
  helpers.go — one file per resource domain, each with its own imports.

internal/mcp/server.go: newServer (708 lines, 33 inline tool registrations)
refactored to a registry pattern:
- internal/mcp/tools.go (new): toolReg struct + allTools() returning all 33
  tool definitions. Handler logic moved verbatim — no changes to tool names,
  descriptions, schemas, or behavior.
- server.go: newServer is now 9 lines (iterate registry, AddTool each).
  -699 lines.

No function logic, names, or signatures changed. go vet, build, and all
tests pass (httpapi, mcp, db, policy).
2026-07-17 22:41:40 +02:00
a2410cf9c2 docs(R5): rewrite knowledge schema + llm-wiki for DB-native model; deprecate root inventory.yaml
Rewrote .agents/domains/knowledge/schema.md and .agents/shared/llm-wiki.md
which described the deleted Python substrate (bin/homelab, oikos/cards/,
oikos/ledger.py, root inventory.yaml, knowledge/sources/, get_page/
search_docs MCP tools). Now reflect ADR 0003: Postgres DB is the single
source of truth for structured data and narrative knowledge; seeds/*.yaml
are bootstrap+DR manifests (content-hashed via seed_versions); archive/
knowledge/ is the frozen legacy wiki; MCP search_knowledge/get_entity_
knowledge replace get_page/search_docs.

Swept substrate refs in .agents/shared/{writing-style,page-templates}.md
and .agents/domains/operations/schema.md: bare inventory.yaml ->
seeds/inventory.yaml; knowledge/sources/ -> archive/knowledge/sources/
(historical); get_changelog/oikos/ledger.py -> DB audit trail / structured
document changelog field; HERMES -> Nomos.

Root inventory.yaml (618-line Python-era file superseded 2026-07-07 by
seeds/inventory.yaml) replaced with a deprecation stub pointing to the seed
and DB. Kept as a stub rather than deleted because AGENTS.md §1/§2 still
point clients at /opt/homelab-context/inventory.yaml; full on-client path
reconciliation deferred to R13.

Flagged export gap: oikos export regenerates seeds/{ontology,inventory,
policy}.yaml but NOT seeds/knowledge.yaml — API-added knowledge lives only
in the DB until hand-edited into the seed.

VERSION 0.7.7 -> 0.7.8. Plan R5 marked done.
2026-07-17 22:36:41 +02:00
d2950dd09d refactor: sqlc vs raw SQL — hybrid approach (R3)
Deleted 8 genuinely unused sqlc queries (no inline equivalent):
- UpsertCurrentRelationship, ListEntitiesCapped, ListEntityStatus,
  UpdateSignalState, InsertClassification, InsertFeedback, InsertSkill,
  UpsertCurrentRelationship — all had zero call sites.

Migrated 9 inline raw SQL sites to use sqlc queries:
- GetOntology (impl.go): ListEntityTypes, ListRelationshipTypes,
  ListLifecycleDefs — replaces 3 raw pool.Query blocks with typed sqlcgen
  calls, eliminating manual row scanning.
- EndRelationship (phase3.go): EndCurrentRelationship — replaces tx.Exec
  with sqlcgen.New(tx).EndCurrentRelationship.
- checkPrecondition (impl.go): GetEntityStatus — replaces tx.QueryRow +
  manual Scan with sqlcgen.New(tx).GetEntityStatus.
- GetEntityRelations (impl.go): ListEntityRelations — replaces raw pool.Query
  + scanRelationships helper (now deleted).
- GetGraph (impl.go): ListGraphEdges — replaces raw pool.Query +
  scanRelationships.
- resolveEntityID (impl.go): GetEntityBySlug/GetEntityByID — replaces
  raw pool.QueryRow + Scan.
- createApproval (mcp/server.go): InsertApproval — replaces raw pool.Exec
  with sqlcgen.InsertApproval.

Deleted scanRelationships helper (was only used by the two migrated
graph queries above).

Regenerated sqlcgen — also picks up stale model updates (AgentSession,
SessionPlanStep, SessionQuestion, etc. from recent migrations).

Documented the carve-out in .agents/dev/CONTRIBUTING.md §SQL conventions:
sqlc is the default; raw pool.Query/Exec is reserved for LISTEN/NOTIFY,
dynamic WHERE builders, blast_radius(), and COPY.

go vet, build, httpapi/mcp/db tests all pass. -383/+170 lines.
2026-07-17 22:24:23 +02:00
0a3654b08f refactor(web): delete dead code (R2) — 1736 lines removed
Tool-renderer registry (21 files, ~1.5k lines):
- src/lib/tool-renderers.ts — registry + getToolRenderer (exported, never
  imported anywhere)
- src/lib/renderers/index.ts + 10 .ts registrars + 10 .svelte components
- main.ts: removed the requestAnimationFrame(() => import('./lib/renderers'))
  that was the only thing keeping the dead subsystem alive

Dead components (never imported):
- ToolCallGroup, PlanProgress, GoalHeader, InlineApproval, SessionDigest

Dead store exports (written, never read):
- context.ts: pendingApprovals writable (+ Approval type import)
- events.ts: connectionState writable (+ its .set() calls)

Dead API surface:
- api.ts: SessionDigest interface + fetchSessionDigest (only caller was the
  dead SessionDigest.svelte)

Dead npm deps:
- mode-watcher (0 imports; superseded by stores/theme.svelte.ts)
- @internationalized/date (0 imports)

Also: fix stale comments referencing deleted symbols, update plan R1/R2
status. Build clean (4683 modules, down from 4706; one Svelte 5 warning
gone — the dead HealthSummary.svelte was emitting state_referenced_locally).
2026-07-17 22:10:27 +02:00
c3973e7ac9 refactor: delete dead Go code (R1)
- internal/httpapi/stubs.go: delete — 5-line comment-only orphan file with
  no declarations; its own comment said the stubs live in phase3.go.
- internal/notifier/notifier.go: delete VerifyApprovalToken — zero call
  sites; phase3.go:DecideApproval reimplements the check inline (noted as
  dead in docs/mbse). hashToken stays (used by generateApprovalToken).
- internal/checkdefaults/defaults.go: unexport ResolveHost, ForEntityType,
  ShortSlug, DefaultInterval — only called within the package. Ensure stays
  exported (called by internal/db/seed.go).

go vet, go build, and affected tests pass.
2026-07-17 22:06:46 +02:00
e3a0326c78 docs: codebase review + documentation maintenance pass
Full review (plans/2026-07-17-codebase-review-and-cleanup.md) covering Go,
web SPA, and docs. Applied low-risk doc/tooling fixes; code refactors and
dead-code deletions are listed as actionable recommendations pending approval.

Doc fixes:
- AGENTS.md: remove ghost of retired request_execution (contradicted the
  retire notice above it); fix knowledge/wiki/ -> archive/knowledge/;
  replace brittle counts (33 tools, 36 docs, 20 checks) with pointers to
  source; drop point-in-time dates.
- OIKOS.md: fix broken plan link (now in done/); 001-011 -> 001-020;
  15 MCP tools -> pointer; replace hardcoded knowledge counts.
- README.md: 15 tools -> pointer; fix wails plan link (now in done/);
  complete internal/ package list (add checkdefaults, observability, safego);
  add cmd/desktop/ to repo layout.
- commands.md, page-templates.md: fix broken links; HERMES.md -> NOMOS.md.

Plans housekeeping:
- Move 4 done 2026-07-14 plans from plans/ to plans/done/.
- Reconcile plans/index.md: add the 2 missing 2026-07-14 entries and the
  2 missing 2026-07-15 done entries; add this review.
- Fix stale plan path in migrations/020 comment.

New docs:
- docs/index.md and docs/operations/README.md (folder READMEs per
  writing-style.md).

Tooling:
- web/package.json: add check/typecheck/lint scripts + svelte-check devDep.
- Makefile: desktop-package version now reads from VERSION file instead of
  hardcoded 0.1.0.

VERSION 0.7.6 -> 0.7.7 (patch: docs + tooling only).
2026-07-17 22:04:54 +02:00
55781984c7 docs(mbse): add MBSE system model, framework, component and ontology views
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Four cross-linked documents under docs/mbse/, structured after Jon Holt's
Systems Engineering Demystified (2nd ed.): Framework = Ontology + Viewpoints,
producing a Model made of Views.

- framework.md — the Ontology (SE meta-concepts + Oikos's domain ontology)
  and an 11-entry Viewpoint catalog (two repeating: Component, Ontology).
- README.md — the Model's 9 concern-based Views (mission, requirements,
  functional/physical architecture, interfaces, behavior, V&V, risk, roadmap).
- components.md — 8 per-component Views going one layer deeper into each
  running part of the system's own internal structure.
- ontology.md — 4 Views on the domain ontology itself: entity type
  hierarchy (split into 9 digestible per-domain diagrams), full relationship
  catalog, lifecycle state machines with their requires: gates, and concrete
  population.

Grounded in direct verification against source (grep/read), not just
existing docs — every finding is graded verified vs. per-research-pass.
Surfaced several real, previously undocumented findings along the way:
the policy kill-switch (global.auto_act/never_auto_act) is checked only by
dead code and an unstarted actuator package, so it doesn't gate the live
run path; internal/actuator and internal/learning are compiled but never
started by any process; the relationship catalog grew from 34 to 47 types
since ADR-0014; and task has no registered lifecycle_defs entry despite
having a documented, code-enforced state machine.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 21:00:14 +02:00
7012525ad6 chore(web): remove dead 'approval' case in ActivityTimeline icon map
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The approval entry type was removed from ActivityEntry upstream; this
case/import were unreachable leftovers after merging that change in.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-16 08:29:27 +02:00
127b939a95 Merge remote-tracking branch 'origin/main' into claude/frontend-dev-mode-7feb5b 2026-07-16 08:22:38 +02:00
6a8efd22bb feat(web): redesign task rail — Plan/Event log polish, entity graph resize fix
- Plan/Activity panels: humanize step titles, richer icons, empty states
  matching Scope's illustration style, pretty-printed expandable detail
- Activity log renamed to Event log; every step now expandable
- Chat: middle-truncate header title, remove redundant task-list rail and
  header stat cluster (duplicated in the sidebar), simplify markdown styling
- Fix --font-mono actually being a monospace font (was aliased to DM Sans)
- Replace rotating loader-circle spinner with a smoother fading-blade Spinner
- SessionGraph entity detail panel: resizable and self-clamping against its
  live container size (was overflowing into sibling sections), close button
- Dev launch config: fetch bearer token from the running api container so
  `npm run dev` works against the local compose stack without a hardcoded
  secret in a tracked file

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-16 08:22:08 +02:00
876f181068 fix(agent): don't auto-complete sessions with pending approvals
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The auto-complete fired when the agent hit the P5 approval gate — it
queued a config_mutation run for approval, the P5 gate blocked further
runs, the turn ended, and auto-complete closed the session as 'partial'.
The operator's approval would then land on a dead task.

Fix: hasPendingApprovals check — if the session has any executions in
pending_approval state, skip auto-complete. The session stays in
'executing' until the operator approves (or denies).

VERSION 0.7.5 → 0.7.6
2026-07-16 00:17:50 +02:00
df24cae507 fix(agent): fully silent assent — no system notes, no chat_assent events
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The len(pending)>0 path still injected a brief note saying 'execution(s)
are now running' — the model saw this, thought work was being done for
it, and no-op'd (finish_reason=stop, content_len=0). Same confusion as
the len(pending)==0 case, just from the other branch.

Fix: both assent paths are now fully silent. No system note at all. The
model sees 'go ahead' in the replayed history and responds naturally.

Also removed chat_assent tool_use/tool_result emit events. These were
persisted in the transcript and confused the model on replay — it saw
its own 'tool calls' (chat_assent) and thought it had already acted.

VERSION 0.7.4 → 0.7.5
2026-07-16 00:09:31 +02:00
e4e426de7d fix(agent): auto-complete with partial outcome when writeback missing
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The auto-complete safety net required hadEntityWriteback to be true,
which meant sessions where the agent did the work but forgot to call
update_entity_attributes stayed stuck in 'executing' forever.

Relax: auto-complete fires if the agent did discovery (ran run),
regardless of writeback. If writeback happened → success; if not →
partial (honest: work was done but knowledge graph not updated).

VERSION 0.7.3 → 0.7.4
2026-07-15 23:49:50 +02:00
ca2ff56a25 fix(agent): mark chat-assented executions as continued to prevent race
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
When the chat handler approves a pending execution via chat-assent, the
execution completes in ~2s. The continuation worker detects the completed
execution and calls resumeSession — while the chat handler is still
processing 'go ahead'. Two concurrent LLM calls for the same session cause
empty responses (finish_reason=stop) and race conditions.

Fix: mark the execution as continued immediately after chat-assent grants
it, so the continuation worker skips it. The chat handler will drive the
continuation itself (the model sees 'go ahead' and executes the plan).

VERSION 0.7.2 → 0.7.3
2026-07-15 23:18:16 +02:00
d6e180845c fix(agent): silent assent — stop injecting confusing system notes
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The assent pre-processing injected verbose system notes ('the operator
approved... they are now running... you MUST continue...') on top of the
replayed user message ('go ahead'). The model saw both, latched onto
'now running', concluded the work was being done for it, and no-op'd
(finish_reason=stop, content_len=0) — leaving the session stuck in
'executing'.

Root cause: the model already sees 'go ahead' in the replayed history
(the user message is saved to the DB before chat() is called, and
getRecentMessages replays it). The system note was redundant AND
confusing — it told the model work was 'running' when it wasn't.

Fix:
- len(pending)==0 (plan-proposal approval): open assent window silently.
  No system note. The model sees 'go ahead' and responds naturally.
- len(pending)>0 (actual pending executions): brief note naming the
  specific execution IDs that were approved ('don't re-request those').
  No 'continue the plan' directive — the model knows to continue.

VERSION 0.7.1 → 0.7.2
2026-07-15 23:03:34 +02:00
7ef8446825 fix(agent+ui): whatsapp session audit — approvals, stuck indicator, stale execs
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
P1: add docker compose (logs|ps|top|config|images|port|cp) to read-only
allowlist. docker compose logs was classified as config_mutation, causing
individual approval cards for read-only inspection commands.

P2: remove approval entries from activityLog. They were always status=running
and never transitioned to done (the derived store builds from tool-call
text, not execution status), causing AgentIndicator to latch onto a stale
'Approval: ...' entry and never clear — even after the session completed.

P3: remove InlineApproval from Chat.svelte. The green 'Completed in 1s on
lxc:...' boxes were noise in the chat stream. Approval UX belongs in the
Operations page (already has it via Ops.svelte), not inline in the chat.

P4: stale execution cleanup. Startup sweep (mark >1hr non-terminal as
cancelled) + 5-min periodic sweep (mark >10min non-terminal as cancelled).
98 orphaned executions accumulated from eval testing (39 running from
apt_upgrade:audit timeouts, 19 pending_approval, 3 approved).

P5: refuse second config_mutation run when an approval is already pending
for the session. Without this, the agent queues N individual approvals
before the operator can respond — confirmed in session 20757eb9 (two
approval cards for what should have been one plan-level approval).

VERSION 0.7.0 → 0.7.1
2026-07-15 22:19:30 +02:00
a9b3f844b2 fix(eval): raise iteration-followup run cap to 40 (maxIterations)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The agent's run count varies (21-30+) for a real config_mutation task
involving diagnostics. 40 is the natural upper bound (maxIterations).
2026-07-15 14:16:46 +02:00
a3afbb96cf fix(eval): raise iteration-followup run cap to 25
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The agent legitimately runs 20+ diagnostic commands for a config_mutation
task (reset service, re-run backup, verify, check logs). Cap of 8 was
too strict.
2026-07-15 14:05:33 +02:00
d55bae17b9 fix(agent): broaden auto-complete to discovery+writeback path
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The agent often skips update_plan_step bookkeeping (leaving steps
pending/running) but still does the work + writeback. The strict
allPlanStepsTerminal check missed these cases.

Add path (b): if the agent did discovery (ran `run`) AND wrote back
(update_entity_attributes/create_relationship), auto-complete. D.1 already
enforces writeback before completion — if writeback happened, the work
is done.
2026-07-15 13:21:46 +02:00
e3b5fdc358 feat(agent): auto-complete tasks when all plan steps are terminal
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The #1 remaining model reliability gap: the agent does the work (proposes
plan, executes all steps, writes back) but forgets to call complete_task,
leaving the session stuck in 'executing'. The eval showed 3/8 failures
with this pattern.

Fix: autoCompleteIfPlanDone — a structural safety net that fires at both
chat exit paths (normal completion + maxIterations). If the session has a
goal, the agent didn't call complete_task, and ALL plan steps are in a
terminal state (done/failed/replaced/skipped/blocked), auto-complete with
the agent's final text as the summary. Mirrors autoCompleteTrivialTask
but for structured tasks where the work is provably done.

Also: bump maxLLMRetries from 2 to 3 (complex multi-turn flows benefit
from one more retry on empty responses).
2026-07-15 13:06:47 +02:00
3c3b12df5e fix(agent): directive assent notes + retry bump to fix empty responses
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The assent system note said 'Do not re-request or call run again for
these' — the LLM interpreted this as 'don't call run at all' and produced
empty responses (finish_reason=stop, content_len=0) until retries were
exhausted, leaving the session stuck in 'executing'.

Fix: rewrite both assent notes (pending-approval path and pure-plan-approval
path) to be directive about WHAT TO DO NEXT: call update_plan_step(running)
then run for each remaining step. The 'don't re-request' guidance is now
scoped to 'THOSE SPECIFIC' executions, not all run calls.

Also bump maxLLMRetries from 2 to 3 — the empty-response flake on complex
multi-turn flows benefits from one more retry.
2026-07-15 12:50:22 +02:00
3d99282897 fix(agent): move plan step replacement from reopenSession to setGoal
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
reopenSession was replacing plan steps on every follow-up message —
including approvals ('go ahead') — which destroyed the plan the operator
just approved, leaving the agent unable to track step progress and looping
run calls until maxIterations.

Fix: setGoal is the explicit signal for 'new sub-task' (the agent calls
it at the start of each follow-up direction). Step replacement now happens
there, not in reopenSession. An approval ('go ahead') does NOT call
set_goal, so the plan stays intact and the agent can execute + complete
it.
2026-07-15 12:32:02 +02:00
a8f04cc9e3 fix(agent): open assent window on 'go ahead' after propose_plan
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The check len(lastAssistantCalls) == 0 was too restrictive — it only
fired when the assistant had ZERO tool calls. But propose_plan + pre-plan
research are tool calls, so the assent window never opened when the
operator said 'go ahead' after a plan proposal. The agent then tried to
execute config_mutation run calls without the assent window, they queued
for approval, and the turn deadlocked.

Fix: check len(pending) == 0 (no pending APPROVALS) instead of
len(lastAssistantCalls) == 0 (no tool calls at all).
2026-07-15 11:58:57 +02:00
487f9ad358 fix(agent): reopenSession replaces plan steps for executing sessions too
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
A follow-up on an executing session (first turn didn't complete_task) is
still a new direction — the old plan's steps must not block the new one.
Previously reopenSession was a no-op for executing sessions, leaving done
steps that caused errPlanInFlight on the next propose_plan call.
2026-07-15 11:13:23 +02:00
3d7fa99560 fix(eval): preserve plan generations across iterations
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
proposePlan: mark pending steps as 'replaced' instead of DELETE, so the
generation counter (MAX+1) sees prior generations. Without this, a first
plan that was proposed but never executed would be wiped, resetting the
counter — a follow-up's plan would look like generation 1 instead of 2.

plan-always-readonly: raise max_run_calls from 3 to 6 (agent inspects
thoroughly).
2026-07-15 10:48:21 +02:00
844cfe5888 fix(agent): read-only plans execute without approval
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
SOUL.md step 4: all-read-only plans skip the approval wait and execute
immediately. Only config_mutation/destructive steps need operator approval.
set_goal + propose_plan return text updated to match.

Fixes 3/4 eval failures where the agent proposed a plan then waited
for approval on a read-only task.
2026-07-15 10:12:51 +02:00
f1dda7290a fix(eval): fix manifests to require live inspection + approval followup
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
plan-always-readonly: prompt now demands live systemd timer inspection,
not just DB lookup. Added calls_tool: run assertion.
iteration-followup: added 'go ahead' as second followup so the
config_mutation plan gets approved and can execute.
iteration-readonly: replaced nonexistent lxc:prometheus with lxc:dns,
keep it read-only so no approval needed.
2026-07-15 10:02:46 +02:00
462fb4d77b chore(eval): consolidate evals into single evals/ folder
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Move golden.yaml from cmd/nomos/eval/evals/ to the root evals/ folder.
All manifests now live in one place; the -manifest glob points at evals/*.yaml.
2026-07-15 09:50:26 +02:00
e3fa6736c0 feat(agent): plan-first gate, iterative follow-ups, reasoning persistence
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
P1 plan-first: run handler refuses without propose_plan (structural gate,
not SOUL.md prose). Plan window decoupled from set_goal — config_mutation
auto-run only on operator approval (assent window). Closes the approval-free
config_mutation hole confirmed in session d0d562e0.

P2 iteration: reopenSession flips terminal→executing, marks prior plan steps
replaced, clears outcome. proposePlan excludes replaced from in-flight check,
bumps generation. A follow-up on a completed session starts a new sub-task
with a fresh plan — no more errPlanInFlight dead end.

P3 reasoning: accumulate per-iteration text into the persisted row instead
of overwriting with the last text event. Reload shows intermediate thinking,
not just the final summary.

P4 read-only allowlist: add find, tree, locate, systemctl list-timers/
list-unit-files/show, timedatectl, hostnamectl, systemd-analyze, rclone
ls/lsl/md5sum/check/cryptcheck. Fixes the find misclassification from
d0d562e0.

P5 eval harness: new assertion kinds (proposes_plan, plan_before_run,
plan_generations), multi-turn followups, fetch /sessions/{id}/plan. Four
manifests under evals/.

P6 SOUL.md: strip degenerate-case carve-out, add ITERATE step, update
set_goal guidance.

VERSION 0.6.0 → 0.7.0
2026-07-15 09:36:27 +02:00
e8b30cddcf feat: add theme system, fonts, graph styling, rename Overview→Tasks
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Terracotta (light) and Carbon (dark) themes with toggle
- Inknut Antiqua headings, DM Sans body
- Dot grid background on EntityGraph and GraphBackground
- Theme-adaptive graph colors on EntityGraph
- Art Nouveau chat styling (borders, underlines, blockquote quotes)
- Bullet point styles in chat prose
- Task goal in header, rename Overview→Tasks, New Task labels
- Logo uses var(--primary) for theme awareness
2026-07-15 00:14:31 +02:00
49dfaa77e6 fix(api): sort graph nodes by degree instead of alphabetically
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The unrooted graph endpoint caps at 500 entities with ORDER BY e.slug, which fills the cap with exec:* rows and excludes every host/lxc/service/vm entity. Since edges require both endpoints in the node set (ANY/ANY), 99.9% of edges were dropped — 500 nodes but only 1 edge survived.

Fix: select the 500 most-connected entities (by relationship count descending) so the topology is preserved. Result: 500 nodes, 900 edges across all relationship types.
2026-07-14 22:02:58 +02:00
1267c39ab1 docs(plans): mark OIDC token-refresh fix as shipped (3b98097)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The OIDC fix was committed in 3b98097 by a concurrent session. The plan's
status block was stale ('not yet committed/deployed') — updated to reflect
it's done. No remaining open items in this plan.
2026-07-14 21:31:38 +02:00
a5336c02e9 docs(plans): mark post-fix remainders as Done — all 18 fixes shipped, 4/4 evals pass
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Status: In Progress → Done. All 18 fixes (A.1-A.3, B.1-B.6, C.1-C.2, D.1-D.2,
E.1-E.2, F.1-F.3) shipped in commits 337d577 + 3de359b + dd3076a, deployed
to oikos-nomos-1 (v0.5.3). The golden eval harness (cmd/nomos/eval/) passes
4/4 conversations, validating the structural gates + the SOUL.md
consolidation. Also fixed a pre-existing tool-call doubling bug found by
the eval harness.

Only remaining open item: the OIDC token-refresh fix (PM addition, web/src/
lib/{config,oidc,events}.ts) — implemented, not yet committed/deployed.
2026-07-14 21:29:56 +02:00
dd3076a23a feat(agent): close all post-fix remainders + golden eval harness (F.1-F.2, C.1-C.2, B.4-B.6, E.1-E.2)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Ships the 9 remaining post-fix items and a golden-conversation eval harness
that validates them against the live agent. All 4 evals pass.

SOUL.md (F.1, C.2, E.1):
- Consolidated three overlapping task-flow sections (MANDATORY TASK FLOW,
  'Every chat is a task', 'AFTER EVERY TASK: WRITE BACK') into one. ~50
  lines shorter. The operator's 'be more crisp' feedback.
- Added anti-patterns: don't re-execute on UI/sidebar complaints (C.2);
  don't re-run fleet-wide audits when same-day knowledge exists (E.1).
- Updated approval vocabulary in step 4 to match tasks.go (approved/yes/
  go/proceed/continue/ok/go ahead).

Tool-result strings (F.2):
- set_goal: tightened to 'Goal set. NEXT: pre-plan (read-only tools only).
  Then propose_plan. Do not call run.'
- update_plan_step: added '(Advance with update_plan_step + run; do not
  re-propose.)'

C.1 — completeTask rejects re-completion of a terminal session:
- Returns errTaskAlreadyComplete when status is already done/failed.
- The tool result directs: 'Task is already complete. Do not call
  complete_task again. If the operator pointed out a UI/sidebar
  inconsistency, fix it with update_plan_step...'

B.4 — Surface real model error text:
- chatWith's error event now includes finish_reason + refusal text:
  'Nomos returned an empty or unusable response (finish_reason=length).
  Retry or rephrase.' instead of generic 'empty response'.
- The resume-failed note already carried errText (B.3), which now has
  the real context.

B.5 — Back off between resume retries (4s, 8s):
- resumeSession now sleeps before attempts 1 and 2 (exponential backoff).
  A transient provider issue gets time to clear instead of 3 identical
  calls in 3 seconds.

B.6 — Don't persist the empty placeholder as a visible bubble:
- If a chat turn ends with no text and no tool calls (model empty-response'd
  and all retries failed), delete the placeholder row instead of persisting
  an empty bubble. The error was already streamed via done+error=true.

E.2 — list_lxcs last-audited hint:
- The list_lxcs result now includes last_audited_at — the most recent
  knowledge entry (tagged audit/update, or titled audit/update) linked
  via an 'about' edge. The agent can see 'nextcloud — last audited today'
  and skip re-running it.

Tool-call doubling bug fix (found by the eval harness):
- main.go + continue.go: the tool_use and tool_result events were both
  appending separate entries to the persisted tool_calls array, doubling
  every tool call in the transcript. Confirmed pre-existing (d9cdcee1,
  v0.3.x era). Fixed: tool_use creates the entry, tool_result merges the
  result into the same entry (matched by id). One entry per tool call.

Golden eval harness (cmd/nomos/eval/):
- A standalone Go program that loads YAML manifests of golden conversations
  + assertions, sends prompts to the chat endpoint, drains the SSE stream
  (keeping the agent's context alive), and scores structural assertions
  against the persisted transcript.
- 4 golden conversations covering: trivial read-only (degenerate case),
  plan + proceed (the original duplication bug), UI complaint (no re-exec),
  fleet audit (knowledge preferred over re-execution).
- Structural assertions only (tool-call sequences, plan steps, writeback,
  completion) — text quality is model-dependent and not scored.
- Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest
  cmd/nomos/eval/evals/*.yaml  (~$0.10/run in OpenRouter credits).

Eval results (4/4 passed):
  trivial_readonly:              2 tool calls, no plan, no run
  plan_advances_on_proceed:     13 tool calls, propose_plan x1, writes back
  ui_complaint_no_rerun:        12 tool calls, propose_plan x1, writes back
  knowledge_preferred_over_rerun: 7 tool calls, search_knowledge x1, 0 run

Version 0.5.2 -> 0.5.3 (minor: eval harness + structural hardening).
2026-07-14 21:27:57 +02:00
0b5b213b2a docs(plans): mark D.1+D.2 shipped in post-fix remainders (knowledge loop closed)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
D.1 (complete_task refused without writeback) and D.2 (propose_plan
auto-appends writeback step) shipped in 3de359b (v0.5.1), e2e-validated
against the live agent. The knowledge loop is now structurally closed —
no blockers remain. Remaining items (F.1, F.2, C.1, C.2, B.4-B.6, E.1,
E.2) are all friction/cosmetic.
2026-07-14 20:47:35 +02:00
3b98097f58 fix(web): refresh expired OIDC tokens before API calls
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The overview background graph and the Knowledge Base graph both rendered
empty because the SPA's OIDC access token expired (~5 min TTL) and was
never refreshed. fetchWithAuth called getToken() synchronously (no refresh);
ensureToken returned the stale token without refreshing; storeTokens
discarded expires_in; the resulting 401 made fetchGraph return null and
both graphs drew nothing, with no error surfaced.

- oidc.ts: track expiresAt from expires_in; getToken() returns null within
  30s of expiry; ensureToken/initOIDC refresh instead of returning stale
  tokens; isOIDCConfigured no longer claims configured on expired-only state
- config.ts: fetchWithAuth awaits ensureToken (refresh on demand), falls
  back to static token if OIDC can't yield one, flushes OIDC session on 401;
  sseUrl is async + refreshes before constructing the EventSource
- stores/events.ts: connect() awaits the now-async sseUrl
2026-07-14 20:42:07 +02:00
3de359b85f feat(agent): close knowledge loop — refuse complete_task without writeback (D.1+D.2)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
D.1 — complete_task structural gate:
- hadDiscovery(ctx, session) reports whether the session ran `run` successfully
  against a live target (NOT get_entity/list_lxcs — those are DB lookups, not
  new facts). A trivial Q&A that only calls get_entity is a degenerate case
  and must NOT be blocked.
- complete_task with outcome=success is REFUSED when hadDiscovery && !
  hadEntityWriteback. The refusal fires BEFORE completeTask runs, so the
  session stays in 'executing' state and the agent must call
  update_entity_attributes/create_relationship then retry complete_task.
  An explicit failure/partial is allowed through (the agent is acknowledging
  it didn't finish — no reason to force writeback).
- Replaces the prior advisory warning (5.5) which the agent consistently
  ignored. The agent saw the warning and ended the task anyway; this gate
  makes the writeback a hard prerequisite for success.

D.2 — propose_plan auto-append writeback step:
- When the agent proposes a plan whose steps don't mention
  update_entity_attributes or create_relationship, D.2 appends a final
  'Write back: update_entity_attributes + create_relationship +
  upsert_knowledge' step before persisting. The result string tells the
  agent it was appended.
- With the seq-order enforcement (5.6) and D.1's complete_task gate, the
  agent must complete the writeback step (and actually call the tools) to
  finish. Neither relies on the agent reading SOUL.md.
- Removed the old advisory writeback nudge from propose_plan's result
  string — D.2 makes it structural.
- Updated the propose_plan tool description to state both gates crisply.

Verification:
- TestHadDiscoveryAndWriteback: hadDiscovery true only after a successful
  `run`; false after failed run, get_entity, or no calls. hadEntityWriteback
  true only after update_entity_attributes/create_relationship.
- e2e against the live agent (oikos-nomos-1, v0.5.1):
  - D.2: agent proposed 3 steps (no writeback); D.2 auto-appended step 4
    'Write back: update_entity_attributes + ...'. Result string said
    '(appended a writeback step — your plan didn't include one; step 4)'.
  - D.1: agent ran `run` (uptime on lxc:gitea), called complete_task, was
    REFUSED ('Refused: this session ran run against live targets (discovery)
    but did not call update_entity_attributes...'). Agent self-corrected:
    called update_entity_attributes, retried complete_task, succeeded.
    Knowledge loop closed end-to-end.

Version 0.5.0 -> 0.5.1 (patch: structural enforcement of existing intent).
2026-07-14 20:40:49 +02:00
c5bee740ad docs(plans): mark post-fix remainders phases A+B.1-B.3+F.3 as committed+deployed
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Status was 'shipped & e2e-validated'; now reflects the commit (337d577),
push to main, and deploy to oikos-nomos-1 (v0.5.0) that followed the
e2e validation. D.1 (refuse complete_task without writeback) is the next
blocker.
2026-07-14 20:25:47 +02:00
337d577f00 fix(agent): refuse plan re-proposal + emit done on error (close divergence chain)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Operator-reported bug: on 'proceed with the rest' the agent re-proposed the
plan, duplicating it in the sidebar. Root cause was a three-bug chain, not
one bug:

1. Trigger — model empty-response on 'proceed' (approval vocabulary didn't
   list 'proceed', so the agent wasn't sure it was approved and no-op'd).
2. Amplifier — chatWith emitted 'error' without 'done' on empty response
   (agent.go:370). The frontend's onComplete saw !receivedDone and
   misclassified the model failure as a network disconnect, calling
   handleDisconnect -> resumeSession.
3. Divergence — the reconnect note was generic ('report your state'), so
   the agent re-proposed + re-executed instead of advancing the plan.

Fixes (shipped, e2e-validated against the live agent on oikos-nomos-1):

- A.2: proposePlan refuses re-proposal once a step has started (returns
  errPlanInFlight). Drops the append-mode safety net (commit 5384499) that
  was the direct source of the sidebar duplication. The agent must advance
  with update_plan_step + run; the tool result directs it.
- A.1: proposePlan sets the 'generation' column on INSERT (migration 020
  added the column + frontend grouping, but the INSERT never wired it).
- A.3: propose_plan tool description restated as a crisp contract (ONCE,
  STOP and wait, REFUSES once a step started, advance with update_plan_step).
- F.3: approval vocabulary expanded to approved/yes/go/proceed/continue/ok/
  go ahead; propose_plan result string tightened to an imperative.
- B.1: chatWith emits 'done' after 'error' on every terminal path via a new
  emitError helper. The frontend now treats model errors as ended (not
  disconnected), so no auto-reconnect -> resumeSession fires.
- B.2: reconnect/resume note carries the operator's last message + an
  explicit 'advance the plan, do NOT call propose_plan again' directive when
  a plan is in flight. Wired into all 4 resume entry points (reconnect,
  /resume, idle-sweep, question-answer) via enrichResumeNote.
- B.3: resumeSession escalates the recovery note across its 3 attempts (final
  retry: 'pick the lowest-pending step, mark it running, call run — do that
  now') instead of 3 identical notes -> 3 identical empties.

Verification: TestProposePlan_RefuseInFlight replaces TestProposePlan_
AppendVsReplace. e2e conversations against the rebuilt container:
  conv2 ('proceed with the rest') -> 0 propose_plan calls, plan stayed at
    3 steps (was 6+ before), update_plan_step x5 + run x2 + complete_task.
  conv3 (full plan, 'go ahead') -> apt-get update on lxc:dns auto-ran under
    the plan window, update_entity_attributes writeback, clean complete_task.
  nomos logs show zero reconnect/resume entries for the plan-proposing
    sessions (the three-bug chain is closed).

Remaining (not in this commit): D.1 refuse complete_task without writeback
(next blocker), C.1/C.2, F.1/F.2 SOUL.md consolidation, B.4-B.6, E.1/E.2.
See plans/2026-07-14-post-fix-session-remainders.md.

Also: re-audit 2026-07-10-general-gated-execution.md — request_execution enum
retirement (60effcb) closes item 9; only auto-act revival (item 10) remains.

Version 0.4.1 -> 0.5.0 (minor: new structural behavior, not a bugfix).
2026-07-14 15:28:33 +02:00
5f82627fa8 restore entity count in Scope collapsed header
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 13:42:25 +02:00
5caf49bf48 mandatory pre-plan flow: goal → research → plan → APPROVE → execute
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
SOUL.md: mandatory 6-step task flow at TOP of file, unmissable.
Agent MUST: set_goal → pre-plan (research only) → propose_plan → STOP
and wait for approval → execute (auto-run under plan window).

Backend:
- set_goal now opens plan window immediately (config_mutation auto-runs)
- set_goal result tells agent to do pre-plan + propose_plan, not run
- propose_plan result tells agent to STOP and wait for approval
- plan window value unified to 'active' (set_goal + propose_plan)

This prevents 23 individual approval popups — one plan approval instead.
2026-07-14 13:33:54 +02:00
24cc3b1f4e approval lifecycle entries in Activity timeline
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- activityLog now detects 'requires approval' in tool results
- Adds approval entries with shield icon + description + execution ID
- Works for both run and remaining approval paths
2026-07-14 13:07:49 +02:00
b423cf4dea plan-approve-once policy + cooler empty states + remove graph header
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Backend:
- proposePlan sets plan window in autonomy_settings (nomos:plan:<session>)
- run handler checks plan window — auto-executes config_mutation commands
  within plan without per-action approval
- planWindowActive function in server.go
- Plan window cleaned up on completeTask (already covered by LIKE '%:' || )

Frontend:
- Removed 'Session graph' header bar
- Cooler empty states: Plan shows animated dots + 'Awaiting plan…',
  Activity shows pulsing dots + 'Waiting for activity…'
2026-07-14 13:04:51 +02:00
2ed169d239 fix stuck 'Agent is thinking' + flip activity to old-to-new
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- AgentIndicator now shows only during stream or when running tools exist
  (not on session status=executing which never cleared)
- Activity timeline: oldest-first ordering (reads top-to-bottom naturally)
- Removed unused liveStatus derivation and currentTask import from Chat
- Plan: Phase A+B+C for activity gaps + plan-approve-once policy
2026-07-14 12:56:42 +02:00
44720b7b30 group activity entries by plan step + remove inline renderers from chat
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Tool calls in Activity timeline are now tagged with current plan step
- Indented entries show which step they belong to
- Step tracking via update_plan_step(status=running) tool calls
- Removed inline tool renderers from chat (health summary, fleet snapshot, etc.)
  — all tool output now visible only in sidebar Activity timeline
2026-07-14 12:38:02 +02:00
c8c7705046 fix: circular import chat↔workspace — extract activityLog to activity.ts
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 12:28:49 +02:00
54f532f166 sidebar reorganized: Scope, Plan, Activity — collapsible + resizable
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- TaskContextPanel restructured into 3 collapsible sections:
  Scope (graph), Plan (goal + steps + progress), Activity (timeline)
- Collapsed headers show compact live status: 'Graph', 'Step X/N', 'N actions'
- Sections are vertically resizable via drag handles
- Plan section shows goal inline + step list + progress bar
- GoalHeader and PlanProgress no longer rendered separately
- ActivityTimeline header moved to TaskContextPanel
2026-07-14 12:23:26 +02:00
b414722fc7 sidebar activity timeline replaces tool display in chat
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- New ActivityTimeline: unified timeline in sidebar showing all agent actions
  (goal, plan steps, tool calls, knowledge, completion) in reverse chron order
- activityLog derived store merges messages + planSteps + currentTask
- AgentIndicator stays in chat (thinking/working indicator), simplified props
- ToolCallGroup removed from chat — tools visible only in sidebar timeline
- SessionDigest replaced by ActivityTimeline
- PlanProgress restored in sidebar (conceptual steps, separate from timeline)
2026-07-14 12:19:26 +02:00
cc266c238e fix: class:transition-opacity shorthand misparsed by Svelte 5
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 12:04:02 +02:00
9f40f19f25 unified agent indicator at end of conversation
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- New AgentIndicator component: replaces 3 separate indicators
  (loading dots, ToolCallGroup summary, activity bar) with one
- Positioned as last item in message list — scrolls naturally
- Shows current tool action: 'Researching lxc:nfs-export…' etc
- Spinner during work, check on completion, X on error
- Fades out 3s after turn completes
- Activity bar, loading dots, statusLabel removed from Chat
- Continue button moved to sidebar session panel
2026-07-14 11:59:04 +02:00
04677fdf4b tool timeline in sidebar + compact chat tools + scroll fixes
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- SessionDigest now includes live tool timeline, plan steps, knowledge
- ToolCallGroup compact: single-line with collapsible names only (no JSON)
- Activity bar moved to bottom of messages, smart scroll respects user position
- setGoal now sets status=executing (removed stuck planning state)
- PlanProgress merged into SessionDigest, removed from TaskContextPanel
- New toolTimeline derived store in chat.ts
2026-07-14 11:45:36 +02:00
cc6bcdceaa scroll fixes + activity bar position
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Activity bar moved to bottom of message list (before messagesEnd)
- Smart scroll: auto-scroll only during streaming or when near bottom
- Scrolling up pauses auto-scroll until next send
- Removed duplicate $effect block
- Plan: tool timeline in sidebar (plans/2026-07-14-tool-timeline-sidebar.md)
2026-07-14 11:37:52 +02:00
5b403141ea fix: VERSION file resolution in Docker build context
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Dockerfile now copies VERSION into /build/web/VERSION for vite
- vite.config.ts tries ../VERSION (local dev) then ./VERSION (Docker)
2026-07-14 11:13:58 +02:00
7847cdffd6 add version display in UI sidebar + version bump rules
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- VERSION file at repo root (0.3.0)
- vite.config.ts reads VERSION at build time, injects __OIKOS_VERSION__
- App.svelte shows version in sidebar tooltip + subtle text below logo
- AGENTS.md §9: every commit to main MUST bump VERSION
  (patch=bugfix, minor=new features, major=breaking changes)
2026-07-14 11:11:53 +02:00
dce19bd258 fix: PlanProgress template syntax error — multi-statement inline expression
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 11:04:47 +02:00
60effcb2fe session reliability: reconnect, knowledge loop, retire request_execution
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Phase 1 — crash recovery: SSE auto-reconnect + backoff, polling gate
during disconnect, connection banner with retry button, empty-response
retry 3x, non-terminal resume on empty response, persistent error cards.

Phase 2/4 — visibility + continuation: custom ExecutionStatus renderer,
approvals extracted on every tool_result (not just done), activity bar
with status/goal, SessionDigest live polling, Continue button.

Phase 3 — cleanup: complete_task auto-cancels orphaned approvals,
deletes assent/destructive window keys, propose_plan marks pending
steps as replaced, plan step seq-order enforcement.

Phase 5 — knowledge loop: list_lxcs state filter (active/destroyed),
SOUL.md unmissable writeback section, propose_plan validation nudge,
complete_task writeback check, upsert_knowledge about array support,
plan generation grouping in frontend, session approval count badge.

Retire request_execution — all mutations now route through run.
Updated SOUL.md, AGENTS.md, CLIENTS.md, skills, and agent system notes.

Migration 020: plan step generation column, audit_log session_id index,
nomos_plan_executions pending-approval index.
2026-07-14 11:03:23 +02:00
b446909ea5 Tray icon: white logo, 10% smaller, rsvg-convert rendering
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 09:06:42 +02:00
dd27630be3 Tray icons: use rsvg-convert for proper SVG rendering with transparency
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 09:04:53 +02:00
e78a3e9048 Tray icons: strip white bg from qlmanage render, use actual SVG logo
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 09:01:19 +02:00
5e3e4eaf07 Tray icons: extract logo from app .icns, proper transparency
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 08:58:41 +02:00
7087d1ffea Tray icon: use app .icns (matches Dock icon)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 08:57:12 +02:00
6c7631d425 Tray icon: use original .icns file (native macOS icon format)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 08:55:37 +02:00
c4ac0cb935 Tray icon: black for light mode, white for dark mode
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 08:52:24 +02:00
6090ef71d4 Tray: icon only (no label), transparent bg for template icon
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 08:46:47 +02:00
06c6f4eb8c Fix window close: use RegisterHook to hide instead of destroy
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
RegisterHook + e.Cancel() prevents Wails from destroying the WebView
when the window is closed. The app now hides to the system tray. Left-
click on the tray icon correctly restores the window.
2026-07-14 00:34:01 +02:00
7063c90898 Rename app to Oikos (was oikos-desktop)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Binary: oikos-desktop → Oikos
- Bundle: oikos-desktop.app → Oikos.app
- Install path: /Applications/Oikos.app
- Auto-update paths updated
- CI Linux binary renamed
2026-07-14 00:31:05 +02:00
aca6b8bcc2 Update plan status with final iteration details
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 00:28:13 +02:00
eeb78ed3c6 docs: update CONTRIBUTING, remove wails3 CLI dependency
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Makefile desktop target uses go build directly (no wails3 required)
- CI workflow simplified: go build instead of wails3 build
- CONTRIBUTING: add make install, desktop auth docs, auto-update docs
- CONTRIBUTING: add file listing for icon.png, icon.icns, Taskfile, plist
2026-07-14 00:28:00 +02:00
bcef4e6456 Add manual update check to tray menu, /update/check endpoint
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Tray 'Check for Updates' now checks immediately and shows dialog
- Dialog has 'Install' and 'Later' buttons
- /update/check endpoint on local server for SPA to query
2026-07-14 00:25:35 +02:00
23535eac25 Auto-update: download + install + restart
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- CheckForUpdates binding returns version string if newer available
- InstallUpdate binding downloads zip, extracts, replaces app, restarts
- checkUpdates goroutine polls every 6h, shows dialog with version
- make install copies .app to /Applications
- Update script: quit app → sleep → replace .app → relaunch
2026-07-14 00:22:15 +02:00
bcf2b265c5 Pass apiUrl through OIDC redirect so SPA has it on return
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The apiUrl configured on the Config page was lost when the webview
navigated away to localhost and back. Now it's included in the return
URL as ?desktop=1&apiUrl=...&token=...
2026-07-14 00:18:21 +02:00
cea67ccd15 Remove accidentally committed binary
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 00:13:20 +02:00
2bd7de355b Desktop OIDC: full page nav to localhost, meta redirect back
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
SPA navigates to 127.0.0.1:18901/oidc/start, passing ret URL.
Go opens browser, waits for callback, saves token, returns HTML with
<meta refresh> back to Wails app with ?desktop=1&token=TOKEN.
main.ts extracts token from URL on reload.
2026-07-14 00:13:01 +02:00
8b3fe02a10 Desktop OIDC: non-blocking fetch + poll, don't leave webview
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
SPA fetches /oidc/open (returns session ID immediately), then polls
/oidc/result every 500ms. Go server opens browser in a goroutine.
Webview never leaves the Wails origin. Token is saved to keychain and
returned through the poll response.
2026-07-14 00:09:46 +02:00
c8ef3793d7 Keep trailing slash on authorize endpoint URL
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 00:06:56 +02:00
d7197c1952 Desktop OIDC: redirect webview to local server, Go opens browser
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The webview navigates to http://127.0.0.1:18901/oidc/open?apiUrl=...
The Go server opens the system browser to Authentik, waits for callback,
exchanges code for token, saves to keychain, then redirects the webview
back with ?desktop=1&token=TOKEN. main.ts extracts the token from URL.
2026-07-14 00:04:49 +02:00
cac5524402 Detect desktop via URL param not config
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-14 00:02:12 +02:00
56e509d506 Desktop OIDC: open in system browser via window.open, revert to copy-paste
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The local HTTP server approach (fetch to 127.0.0.1) doesn't work in the
Wails webview. Simplify: use window.open() to launch OIDC in the real
browser. After authentication, the callback page at the server shows the
token. User copies and pastes into the Token tab.

Also fix: SetSize before app.Run() crashes with nil pointer — use
WebviewWindowOptions width/height directly from restored state.
2026-07-14 00:00:24 +02:00
68011f9a06 Add OIDC server startup logging
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-13 23:56:33 +02:00
a395771960 Fix doubled /authorize/ in OIDC auth URL
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-13 23:54:19 +02:00
5664a4bf29 OIDC: local HTTP server instead of Wails bindings
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The Wails runtime isn't reliably loading for IPC calls. Replace the
binding-based StartOIDCLogin with a local HTTP server on 127.0.0.1:18901:

- /oidc/login?apiUrl=... — opens system browser, waits for token
- /oidc/callback — Authentik redirect target, exchanges code
- /oidc/config?apiUrl=... — fetches OIDC provider config
- SPA detects desktop via ?desktop=1 URL param
- SPA calls localhost directly via fetch() instead of Wails IPC
2026-07-13 23:52:08 +02:00
ff6608f9ea Use default Wails asset handler + GetStoredConfig binding
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Remove custom asset handler — it broke Wails IPC routing
- Use application.AssetFileServerFS(distFS) so Wails serves its own runtime
- Add GetStoredConfig binding: SPA calls it on startup to retrieve keychain config
- main.ts: loadDesktopConfig() fetches stored creds before mounting
- Remove runtime.js embed (Wails serves it internally)
2026-07-13 23:45:54 +02:00
0271727709 Embed Wails runtime.js, serve it from custom asset handler
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The SPA needs /wails/runtime.js for window.wails to be available.
Since we use a custom AssetOptions.Handler, Wails' internal routing
doesn't serve it. Embed the runtime and serve it explicitly.
2026-07-13 23:40:50 +02:00
c31978042f Desktop OIDC: open system browser, capture callback on localhost
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
ConfigService.StartOIDCLogin():
- Fetches OIDC config from the API
- Generates PKCE params
- Starts local HTTP server on 127.0.0.1:18901
- Opens system browser to Authentik
- Captures callback directly (no copy-paste)
- Exchanges code for token, saves to keychain
- Returns token to SPA → auto-connects

Config.svelte detects Wails environment and calls the binding.
2026-07-13 23:37:17 +02:00
5699a3f758 Shrink logo 30% in app icon and tray icon
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-13 23:32:53 +02:00
44c0145683 App icon: white logo on black rounded-rectangle background
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-13 23:31:27 +02:00
de7eca8b6d Regenerate icon.icns from SVG source with proper transparency
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
The old icon.icns was copied from favicon.png which was actually
a dark-background .icns file. Regenerated from favicon.svg via
qlmanage → sips → iconutil to get white logo on transparent bg.
2026-07-13 23:29:17 +02:00
f6e2079a61 Fix OIDC login in desktop: sync apiUrl before startLogin, add app icon
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Config.svelte: call setConfig() before startLogin() so fetchConfig
  uses the user-entered server URL
- Makefile: copy icon.icns into .app bundle Resources
- Info.plist: add CFBundleIconFile entry
2026-07-13 23:27:12 +02:00
515c9b9174 fix(web): stack auth options vertically in config screen
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-13 23:16:50 +02:00
f1ac82255a Add OIDC desktop callback, app logo, rename to Oikos
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
- Server: /oidc-callback HTML page exchanges Authentik code for token,
  displays it for user to copy into the desktop app's Token tab
- oidc.ts: desktop mode uses apiUrl+/oidc-callback as redirect URI,
  encodes PKCE verifier in state parameter
- Config.svelte: add Server URL field to OIDC tab for desktop UX
- Caddy: add /oidc-callback to enroll bypass (no Authentik gate)
- App: favicon.png as system tray icon, window title 'Oikos'
- web/index.html: title 'Oikos'
2026-07-13 23:14:40 +02:00
62a337f3cc feat(web): redesign config screen with animated particle background and unified auth layout
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-13 23:12:16 +02:00
35ff3f37e1 Fix macOS packaging: create .app bundle manually
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
wails3 build v3 alpha delegates to Taskfile; the go build produces a raw
binary, not a .app. Package step now creates the bundle structure
(Contents/MacOS, Info.plist) and zips it.
2026-07-13 23:03:16 +02:00
8f121cfa1e Drop -clean flag from wails3 build — not supported in v3 alpha
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
2026-07-13 23:00:45 +02:00
1c3a800506 Drop macOS CI job — no macOS runner available. Single Linux build.
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
macOS builds happen locally via 'make desktop-package' on the dev Mac.
2026-07-13 23:00:04 +02:00
a429436903 Trigger desktop CI on every push to main, not just tags
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build SPA (push) Has been cancelled
Desktop App / macOS (arm64) (push) Has been cancelled
Desktop App / Linux (amd64) (push) Has been cancelled
Desktop App / Create Release (push) Has been cancelled
2026-07-13 22:54:00 +02:00
07d67c8446 fix(oidc): strip trailing slash from redirect URI to match Authentik strict mode
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-13 22:50:31 +02:00
8b50753746 feat(chat): MCP tool apps — custom inline renderers for 12 tools
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
12 of 33 MCP tools now render as rich inline cards instead of raw JSON:
EntityCard, HealthSummary, LXCList, EntityTable, KnowledgeResults,
BlastRadius, ChangeLog, FleetSnapshot, MetricChart.

Architecture:
- Server: annotateJSONResult() wraps queryRows with __renderer hints
- Registry: match/dispatch system maps tool names to Svelte components
- Chat: inline dispatch with 5-card limit, overflow to collapsed group
- ToolCallGroup: unmatched prop, hides when all matched, ARIA labels

Tests: 3 new Go tests for annotateJSONResult (wrap, no-op, multi-row).
2026-07-13 22:46:56 +02:00
680575e2cf Remove unrelated plan file from stale branch
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-13 22:41:50 +02:00
6b52c1ae57 Move 2026-07-12-wails-desktop-app plan to done/
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Phase 0 deployed 2026-07-12. Phases 1.0–1.4 implemented 2026-07-13.
Plan complete.
2026-07-13 22:41:45 +02:00
5d6d9e9040 Merge feature/wails-desktop-app into main
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-13 22:40:40 +02:00
04006553a3 Wails v3 desktop app: scaffold, shell features, token mgmt, auto-update, CI
Problem: the Oikos control room was browser-only — no native desktop
experience (system tray, notifications, keychain-persisted auth).

Change: add a Wails v3 thin-shell desktop app at cmd/desktop/ that embeds
the existing SPA in a webview. The Go side is ~380 lines — no bundled
server, no Postgres connection. It reads auth from the OS keychain,
injects it into the SPA on load, and the SPA talks HTTPS to the homelab
same as a browser.

Phase 1.0 — Scaffold + window:
  - Embed web/dist/ into the Wails binary
  - Inject window.__OIKOS_CONFIG__ with keychain-stored apiUrl + token
  - 1400×900 window, min 1024×700
  - System tray: Open/Quit, click toggles window

Phase 1.1 — Native shell:
  - Poll /api/v1/dashboard/summary every 30s; osascript notification
    when approvals or critical signals increase
  - Save/restore window position to ~/.config/oikos/window.json
  - EnableAutoStart/DisableAutoStart — macOS LaunchAgent plist

Phase 1.2 — Token management:
  - Config.svelte calls window.wails.Call.ByName('SaveConfig') after
    successful connection — persists to OS keychain
  - ConfigService binds SaveConfig, ClearConfig, EnableAutoStart,
    DisableAutoStart to the Wails runtime

Phase 1.3 — Auto-update:
  - Poll Gitea releases API every 6h, compare semver, show dialog
  - 'Check for Updates' tray menu item triggers immediate poll

Phase 1.4 — Distribution:
  - macOS entitlements.plist: network client + keychain access
  - .gitea/workflows/desktop.yml: CI builds macOS arm64 + Linux amd64
    on 'desktop-*' / 'v*' tags, attaches artifacts to release
  - Makefile: desktop (build), desktop-package (build + zip/tar.gz)
  - CONTRIBUTING.md: documented desktop app + commands

Risk: low. Wails v3 alpha API may shift; the Go glue is ~380 lines and
trivially portable. The desktop app is additive — zero changes to the
existing server or SPA logic. No config mutation, no infrastructure
impact.

Verification: go build, go vet, go mod tidy all pass.
2026-07-13 22:40:18 +02:00
7b0a0f01b5 oidc: authenticate SPA users via Authentik
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Add OIDC proxy endpoints (GET config, POST token) to API server
- Implement PKCE Authorization Code flow in SPA
- Enable Authentik login tab in Config page
- Handle callback + auto-refresh + session restore
- Add restart: unless-stopped to all persistent services
- Configure OIDC issuer + client_id in docker-compose
2026-07-13 22:17:40 +02:00
f6a699469d oidc: authenticate SPA users via Authentik
- Add OIDC proxy endpoints (GET config, POST token) to API server
- Implement PKCE Authorization Code flow in SPA
- Enable Authentik login tab in Config page
- Handle callback + auto-refresh + session restore
- Add restart: unless-stopped to all persistent services
- Configure OIDC issuer + client_id in docker-compose
2026-07-13 22:17:31 +02:00
4c4afc4783 fix(web): fit graph to node bounding box once the simulation settles
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Fresh nodes (no prior x/y) get placed by d3-force's default init, which
spirals out from the ORIGIN — not (width/2, height/2) — while the
centering forces here are deliberately weak (0.04, so they don't fight
the link/collide layout) and alphaDecay stops the sim before a weak force
can always pull a far-off cluster back to center. Net effect: graphs could
settle visibly off-center on load, cramped in a corner of the pane.

Fixed by computing the actual node bounding box once the simulation's
'end' event fires and setting the view transform to fit it, instead of
relying on the force balance to land on center by itself. Gated behind a
`fit` flag so passive background reloads (live entity/relationship
events) don't yank the view out from under someone actively panning or
zoomed in on a specific area — only fresh loads (mount, root/depth
change, reset, re-root) reframe.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:51:56 +02:00
604b608fa8 feat(mcp): expose full knowledge content to the agent, not just snippets
search_knowledge and get_entity_knowledge only ever returned a ts_headline
snippet/short headline — enough to find a note, not enough to act on it.
Add get_knowledge_content(slug), mirroring the web UI's
/api/v1/knowledge/content/{id}, so the agent can read a document/
investigation/runbook's full markdown body once it knows which one it
needs. upsert_knowledge already covered the write side. Cross-referenced
all three tool descriptions so the agent discovers the full-read path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:51:37 +02:00
62a8ec1d8d fix(mcp): write targets/involves relationship edges when executions are created
Executions were being created with no outgoing edges to what they acted
on or which task/session drove them, silently starving the graph of new
data going forward — found during this session's DB audit, which had to
backfill 245+25 missing targets/involves edges for existing executions.
This closes the gap at the source: every execution now gets a
target-->targets-->execution edge, and (when the caller supplies a
session/task) a task-->involves-->execution edge, both idempotent
(NOT EXISTS guards) so retries/backfills don't duplicate.

Two call sites: the deduped systemctl/apt_upgrade/pct_create fast path
and the general classifyAndGate path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:45:44 +02:00
335fa67d55 fix(web): scope 1-hop neighbor expansion to rooted graph views only
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Two bugs found while verifying against real production data:

- Excluded activity types (execution/check/task etc., see categories.ts)
  were falling through inCategory's "unknown type -> always visible"
  fallback, since typeCategory only stored entries whose category was
  defined. That fallback exists for types the ontology never returned at
  all; it wrongly re-admitted types the ontology returned but categories.ts
  deliberately excludes. Fixed by storing every type (including undefined
  categories) and checking key presence, not value truthiness.

- Once that was fixed, the previous commit's 1-hop neighbor expansion
  (dimmed cross-category context) turned out fine for a rooted view but
  flooded an unrooted "browse the whole category" view: Fleet's ~49 focus
  entities are hub-like enough that 1-hop pulled in 325+ of the system's
  479 total entities. Neighbor expansion now only applies when a root is
  set; the unscoped view goes back to same-category-only edges, which
  measured at a clean 49 nodes for Fleet.

Verified against live production data (real bearer token, real DB) rather
than mocks: Fleet unrooted = 49 nodes matching the DB's compute+physical
count exactly; rooting on host:strong shows 33 nodes with both bright
same-category and dimmed cross-category neighbors, no isolated dots.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:39:31 +02:00
d1243aceac fix(web): decode percent-encoded slugs in the knowledge content route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
chi.URLParam returns the raw, still-encoded path segment — unlike the
OpenAPI-generated routes, which decode via
runtime.BindStyledParameterWithOptions before the handler sees them. Slugs
like "document:containers/101-jellyfin" (encoded by the frontend's
encodeURIComponent) were arriving undecoded and matching no row. Found via
a standalone chi repro, not by patching the live deploy checkout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:29:42 +02:00
61ad785fef fix(web): render full document content, keep graph connected under categories
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Two fixes to the new category taxonomy:

- Knowledge Base couldn't show a document/investigation/runbook's own
  markdown body — knowledge_entities.content was never exposed by any
  endpoint (GetEntityKnowledge answers "what knowledge references this
  entity", not "what is this entity's content"). Add GET
  /api/v1/knowledge/content/{id} and render it with the existing
  marked+DOMPurify pipeline in a new Content section.

- The graph hid any edge whose other endpoint wasn't in the active
  category, so nodes with only cross-category neighbors rendered as
  disconnected dots. Queried the real relationship table: ~70% of infra
  edges cross Fleet/Network/Services/Storage lines (compute+network+
  software+storage+physical used to be one "infrastructure" layer).
  EntityGraph now keeps 1-hop neighbors visible but dimmed instead of
  hiding them, so the edges — and what they connect to — stay visible.

- categories.ts: `cognition` domain conflated true knowledge (document/
  investigation/runbook, 58 entities) with operational telemetry
  (execution/check/task/signal/approval/pattern/skill/classification/
  feedback, 300+ entities with their own Operations/Signals/Learning
  pages). Mapping the whole domain to Knowledge pulled in 245 execution
  entities fanning out from ~17 compute nodes via `targets` edges — the
  single biggest source of graph clutter. Knowledge now maps by type
  (document/investigation/runbook only); the rest of cognition is
  excluded from Knowledge Base browsing entirely.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 10:17:13 +02:00
35c54ceef5 feat(web): browse Knowledge Base by mixed Network/Fleet/Services/Storage/Identity/Knowledge categories
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Replace the layer-based (Infrastructure/Governance/Cognition) browsing tabs
with a synthesized category taxonomy built from the ontology's finer-grained
`domain` field, since layer lumped unrelated entity types (an LXC and a DNS
record and a storage volume) into one bucket. Network and Fleet each span
two domains, so the table view now fans out per-domain fetches and merges,
while the graph view maps domain->category client-side. Also carries over
several detail-panel polish items (Tasks-not-raw-executions, slug URL
encoding, MultiSelectFilter) from earlier in this session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-13 09:53:15 +02:00
f8e03806aa feat(deploy): containerize the web UI as its own compose service
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The SPA-from-binary split (0c0f35a) left `make deploy-ui` pointing at a
deploy path that was never actually wired up: scp to a "mac-mini" SSH
host that doesn't resolve from itself, a /var/www/oikos-ui/ that doesn't
exist, and `systemctl reload caddy` on a box with no Caddy installed at
all (not brew, not a container, nothing on 80/443).

Add a `web` service (compose/web/Dockerfile: node build -> caddy:2-alpine
static + SPA-fallback serving) to docker-compose.yml so the UI deploys
through the same push-to-main -> webhook -> docker compose build/up
pipeline the rest of the stack already uses, instead of a manual
scp/ssh step. Drop the broken `deploy-ui` Makefile target; `make ui`
stays as a local build sanity-check.

Update the reference Caddy config (compose/caddy/Caddyfile.oikos) to
reverse_proxy the new :8091 service instead of reading static files off
local disk, and fill in the <mac-mini-mesh-ip> placeholders with the
actual LAN IP (192.168.178.182 — the LXC and mac-mini subnets are
routed). This file is a reference only; the real caddy-conf repo change
is applied separately after review.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-12 22:36:40 +02:00
94c94c0758 feat(web): merge Entities + Graph into a single Knowledge Base page
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Replaces the separate Entities/Graph nav items with one Knowledge Base
page that browses all entities as either a table or a force-graph,
scoped by ontology layer (Infrastructure/Governance/Cognition), with a
resizable browse/detail split instead of a slide-over sheet.

- New KnowledgeBase.svelte: layer tabs, view toggle, resizable
  browse/detail split (pattern from Chat.svelte's rail).
- EntityTable/EntityGraph extracted as presentational sub-components;
  their search/filter/root/depth toolbars live in the shared page
  toolbar (not the resizable pane) so they don't truncate when the
  divider is dragged narrow, and both views start flush with the
  detail pane for consistent height.
- EntityTable columns are sortable (slug/type/name/state/health).
- EntityDetailContent redesigned as a single-column list of
  collapsible sections (DetailSection.svelte), collapsed by default
  when empty; relation entries are clickable and select the entity in
  the browse pane + detail pane (and drill in-place in EntitySheet
  wherever it's used elsewhere in the app).
- api.ts: add layer filter to fetchEntities, add fetchEntityTypes for
  client-side graph layer scoping (the graph endpoint has no layer
  param).

Old hash routes (#/entities, #/graph) redirect to #/kb.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-12 22:21:29 +02:00
d80a394b7f docs: fix plan/repo drift, retire dead Goose+Nomos and Caveman tooling
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Documentation and repo-hygiene pass following the client/server split:

Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
  described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
  refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
  deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
  to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.

Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).

Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).

Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.

Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.

Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
  places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
  GetClientContext handler) since the mechanism's introduction on
  2026-06-02 — never matched any real filename, so no client has ever
  picked up an auto-setup script via git-pull or the context-poller sync.
  Fixed all three; the Go server-side fix is the one that actually matters
  since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
  (parsed, never consumed) left over from an earlier clone-based model.

Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 18:19:41 +02:00
0c0f35a3a9 feat(web): split SPA from oikos binary, require auth on every route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Phase 0 of plans/2026-07-12-wails-desktop-app.md. The control-room SPA
is no longer embedded (web/embed.go deleted); it's a standalone static
build served separately (make ui / make deploy-ui). The api process
adds CORS and drops the dev-open auth bypass — every route now needs a
real bearer token, including SSE (?token= query param, EventSource
can't set headers) and api's own /agent proxy to nomos (previously
unauthenticated by omission).

nomos was an unauthenticated client of api's /mcp and approval-decision
endpoints; closing dev-open would have broken it, so it now sends
Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN on every call back to api.

SPA gets a runtime config module (config.ts) and a Config.svelte
first-launch/reconfigure page, reachable afterwards via a "Connection"
entry in the sidebar footer. Every fetch() in api.ts routes through
fetchWithAuth so the same build works same-origin (browser prod, Vite
dev proxy) or cross-origin (future Wails webview, remote access).

Six gaps found against the plan and the live Caddy topology while
implementing — documented in the plan's "Plan review" section, most
notably: api's own /agent mount was never behind combinedAuth (fixed),
and production's Authentik forward-auth needs a bearer-token bypass for
API routes that this repo's Caddyfile.oikos reference copy now has, but
the real dtoro/caddy-conf deploy does not yet.

Verified live: cross-origin static SPA + API, CORS, bearer auth, SSE
query-token auth, and localStorage persistence all confirmed working
in-browser. Full Go test suite and npm run build pass with no
regressions against the pre-change baseline.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 15:49:42 +02:00
346eb2f144 chore: gitignore compiled binaries at root
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-12 12:40:55 +02:00
48827d5bb1 fix(deploy): add poller as fallback when Gitea webhook can't reach mac-mini
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Gitea (LXC 104, 192.168.8.x) can't reach mac-mini (192.168.178.182) due to
ALLOWED_HOST_LIST. As a fallback, a 2-minute launchd poller checks if
origin/main has new commits and runs deploy.sh if so.
2026-07-12 12:14:28 +02:00
56979ac4bd feat(deploy): add webhook receiver and launchd service for push-to-deploy
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/webhook/main.go: HMAC-validated webhook receiver on :9797
- launchd plist: keeps webhook running, PATH includes docker
- Makefile: 'make webhook' target
- Registered as Gitea webhook id 15 on dtoro/oikos

Fixes: auto-deploy was not wired on mac-mini after the consolidation
2026-07-12 12:12:01 +02:00
3157e6102a plans: Wails desktop app with client/server split
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Phase 0 separates the SPA from the oikos binary (delete embed.go, add CORS,
make API base URL configurable, add auth interceptor, close dev-open gate).
Phase 1 builds a thin Wails v3 desktop wrapper — native window + tray +
notifications + auto-start + auto-update. SPA shared between browser and
desktop builds.
2026-07-12 11:49:27 +02:00
6807e353e3 feat(web): redesign Overview as the homepage with a living graph backdrop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Overview replaces Tasks as the default route: a centered new-task entry
with live fleet metrics, a scrollable/filterable task table, and an
ambient canvas rendering of the real entity graph (autonomous camera
drift + mouse parallax) behind it. Tasks sidebar entry is removed;
its status-bucketing logic moves to lib/tasks.ts for reuse.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 11:07:36 +02:00
0ed171507f Merge remote-tracking branch 'origin/main'
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-12 09:18:23 +02:00
e3cbaee534 fix(knowledge): populate hit id in search results
Search hits and entity-knowledge hits never selected an id column, so
every KnowledgeHit.Id defaulted to the zero UUID. The frontend's keyed
{#each results as hit (hit.id)} then had all-duplicate keys, which
silently broke Svelte 5's if-block branch swap for the results panel —
search would set searched=true (Clear button appeared) but the view
never switched away from "Recently learned". Select e.id in both
queries and key the each block on hit.slug (guaranteed unique) instead.
2026-07-12 09:16:03 +02:00
de126daf43 feat(web): fold Events/Agent/Audit into EntityDetail; tag agent_activity with entity_id
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Events, Agent, and Audit were standalone read-only pages that never
cross-referenced the entity they related to. Fold them into EntityDetail
as entity-scoped cards (Agent activity, Audit trail) alongside the
existing Signals/Executions/Knowledge cards, and give the Signals card
real Ack/Mute/Resolve actions. Signals stays a standalone page since
it's the only one with cross-entity triage value (badge count, actions).

Also fixes the underlying reason those new cards would've stayed empty:
agent_activity rows were never tagged with entity_id at insert time
(cmd/nomos/store.go, internal/mcp/server.go), even though the column
and the API filter both support it. Added a best-effort resolver that
checks common tool-arg keys (target, entity_slug, slug, ...) against
the entities table.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 08:38:19 +02:00
c8b479d565 docs: close out task-completion safety net plan; fix stale relative links
Fixes 1-3 deployed and verified live: fresh trivial Q&A sessions now reach
done immediately, and a goal-bearing session that stalled was correctly
nudged by the idle sweep. Fix 4 (backfill) was replaced with deletion after
the operator's call — verified against the DB first that zero knowledge
notes were linked to or written by any of the 53 removed sessions, so
nothing was lost. Documents the pagination gap in listSessions (hardcoded
LIMIT 50, no total count) that hid 6 of those sessions from the original
audit.

Also fixes relative links in this plan and in the UI-review plan that broke
when both moved from plans/ to plans/done/ (one directory level deeper).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 00:14:15 +02:00
075ff93792 docs: mark task-completion safety net fixes 1-3 in progress
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 22:11:56 +02:00
3b9c75fa3f fix(agent): task completion safety net — stop tasks sticking at Running
Implements fixes 1-3 of plans/2026-07-11-task-completion-safety-net.md.
Confirmed live that 50/50 production sessions never reached a terminal
status because the model almost never calls complete_task, even for
trivial single-tool Q&A turns SOUL.md explicitly calls out as needing it.

- Inline safety net (agent.go): a session that never called set_goal never
  framed itself as a structured task, so its first plain-text turn-end IS
  the task ending — auto-complete it there instead of leaving status stuck
  at its creation default forever.
- Idle sweep (continue.go, new completion_nudges column): goal-bearing
  sessions that stall get one nudge, then auto-close with outcome=partial
  if the nudge goes unanswered, mirroring the pattern resumeSession already
  uses for a different stuck-session failure mode.

Fix 4 (backfill of the 50 already-stuck live sessions) is deliberately
separate — deferred until this is deployed and verified live, per the
plan's implementation order.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 22:11:24 +02:00
e3850f6820 docs: task completion safety net — every live task stuck Running
Traced during UI-review verification: 50/50 live sessions are stuck
active/planning, never done/failed. Root cause confirmed against the
running DB — set_goal called once, propose_plan and complete_task
called zero times across all 50 sessions. The model consistently
skips the terminal complete_task call despite SOUL.md explicitly
instructing it to, especially for trivial single-tool Q&A turns.
Plan proposes an inline safety net for the common case plus an idle
sweep for structured goal/plan sessions that stall.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 22:01:35 +02:00
fb4c76ba82 fix(ui): implement UI review findings — a11y, IA, and consistency fixes
Fixes the reviewed gaps: keyboard-inaccessible delete controls (SessionRail,
Entities row), case-sensitive entity filter, two competing entity-detail
navigation patterns (standardize on EntitySheet), non-clickable Overview KPI
cards, a bare button bypassing the shared Button component, inconsistent
blur-only vs live filtering, and an unenforced sanitization assumption on
search snippet HTML (now using the already-present dompurify dependency).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 21:52:59 +02:00
b72267bd72 docs: nomos agent code review — mark all fixes done except deferred C1
Every finding from the review is now implemented and verified live:
A1 (3919ec3), B1+B2 (c5ffaec), A3 (926969a), D1-D3 (76f7630), A2 (c390164),
B3 (6d4f6de), F1 (11c18e8). C1 (nomos gateway has no authentication) remains
explicitly deferred per operator instruction. Kept in plans/ (not moved to
done/) since C1 is still open, matching how other partially-complete plans
in this index are tracked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:32:29 +02:00
11c18e8956 perf(agent): cache the MCP tool list per client (F1)
Fix F1 of plans/2026-07-11-nomos-agent-code-review.md, the last item.
buildTools called listToolsFull (a tools/list MCP round-trip) at the start of
EVERY chat turn, including every auto-continuation resume — the tool list is
static for the lifetime of one MCP connection, changing only when the api
process re-registers tools (a restart, which this client already detects and
reacts to via reconnectLocked). Re-fetching it every single turn was
avoidable network+parsing work on the hot path.

mcpClient now caches the parsed tool list after its first fetch, guarded by
its own mutex (kept separate from the request-serializing mu so a cache
check never contends with an in-flight doRequest call). reconnectLocked
clears the cache — an api restart may have changed what's registered, so a
stale cache would be wrong, not just slow. fleetSnapshot's get_health_summary
call is deliberately left uncached — it's meant to be "as of now."

Since each session gets its own client (the per-session pool from the
concurrency work), this caches per-task-conversation rather than globally: a
task's FIRST turn still pays the round-trip, every turn after reuses the
cached list — which is exactly the case that mattered (long-running,
heavily-autonomous tasks with many auto-continuation resumes).

Verified live via the api's request log: a brand-new session's first turn
made 3 MCP calls (initialize, tools/list, get_health_summary); a second turn
on the SAME session made exactly 1 (only get_health_summary) — tools/list
correctly skipped.

This completes the implementation order in
plans/2026-07-11-nomos-agent-code-review.md — every A/B/D/E/F finding from
the review (excluding C1, explicitly deferred per operator instruction) is
now fixed, tested, and verified live.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:30:07 +02:00
6d4f6de676 fix(agent): mark a task failed when its resume permanently gives up (B3)
Fix B3 of plans/2026-07-11-nomos-agent-code-review.md. resumeSession's retry
loop (used by both auto-continuation and panel-answered questions) already
retried once on a transient LLM failure, but if BOTH attempts came back
empty/erroring, the code just logged and returned — the task was left at
whatever status it already had (typically 'executing' or 'awaiting_input')
with no outcome, no operator-visible signal beyond an inert error line
buried in the transcript, and no way to tell a genuinely stuck task apart
from one quietly still working.

On permanent failure, now calls store.completeTask(outcome='failure', a
summary built from the error) so the task board reflects reality instead of
showing a task that looks perpetually in-progress. Uses context.Background()
for that write, matching resumeSession's own persistence pattern, since the
context that led to the failure may itself be in a bad state. This doesn't
prevent the operator from continuing to work the task via a fresh chat
message afterward — it only replaces silent hanging with a real status.

A full live induction of a permanent LLM outage would require breaking the
model/API-key config for the whole nomos container — too invasive for this
fix's priority. Verified instead that the new branch stays correctly dormant
on the happy path: ran a real ask_operator → panel-answer → resume cycle
end-to-end and confirmed the task landed at status='executing' with no
outcome set, proving the failure-handling code doesn't false-positive on a
normal successful resume.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:25:21 +02:00
c3901641d1 fix(agent): bound conversation history replayed to the LLM (A2)
Fix A2 of plans/2026-07-11-nomos-agent-code-review.md. chatWith replayed a
session's ENTIRE message history into the LLM's context on EVERY turn, no
windowing, no token budget — confirmed against a documented production case
(a single turn with 70 tool calls, messages up to 106KB). Every subsequent
turn of a long-running or heavily-autonomous task re-sent that ever-growing
history in full — a real cost/latency/eventual-context-limit risk for
exactly the tasks this system runs longest (many auto-continuation cycles).

Design call (flagged in the review as needing one before implementation):
a fixed-size window for LLM replay specifically, not the UI's own transcript
view. Simplest option that still keeps roughly the current task's working
context; a token-aware trim or LLM-summarize-on-drop are documented as
stretch options if 30 proves insufficient in practice.

- store.go: new getRecentMessages(ctx, sessionID, limit) — last `limit`
  messages in chronological order, plus whether older ones were omitted.
  getMessages (used by the UI's GET /sessions/{id}) is untouched and stays
  unbounded — the operator should still see a task's full history regardless
  of length; only what gets sent to the model is bounded.
- agent.go: chatWith uses getRecentMessages(sessionID, historyWindowSize=30)
  instead of the unbounded getMessages. When truncated, injects a system
  note telling the model explicitly that older turns exist but aren't shown,
  so it checks upsert_knowledge/search_knowledge rather than assuming
  something wasn't done just because it isn't visible.

New cmd/nomos/store_test.go: real Postgres integration tests (mirroring
internal/db/integration_test.go's throwaway-database pattern, guarded by
OIKOS_TEST_DATABASE_URL). TestGetRecentMessages_Truncation is the direct
proof for this fix (35 messages → 30 returned, correctly ordered,
truncated=true; 5 messages → all 5, truncated=false) — both cases run
against a fully-migrated database, not mocked. Also added
TestProposePlan_AppendVsReplace, closing part of the review's test-coverage
finding (E) by permanently regression-testing the earlier append-vs-replace
plan fix (commit 5384499), which had only been verified manually until now.

Verified live: inflated a real session to 42 persisted messages via direct
SQL, then continued it with a real chat call — the turn proceeded normally
(multiple real tool-call iterations, no crash, no context-length error);
nomos stayed healthy throughout. A3's incremental persistence separately
confirmed to have caught the 7 real tool calls made before the client
connection was cut, cleanly closing out both fixes' interaction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:22:30 +02:00
76f76308cc fix(agent): D1-D3 cleanups — dead code, N+1 query, unvalidated outcome enum
Fixes D1-D3 of plans/2026-07-11-nomos-agent-code-review.md:

- D1: deleted isTaskTool — defined, never called (dispatch already checks
  handleTaskTool's own `handled` return value).
- D2: recordTouched issued one SELECT per entity slug found in a tool call's
  args; batched into one `WHERE slug = ANY($1)` query. Verified live: a turn
  naming three separate entities recorded involves edges for all three via
  the single batched lookup.
- D3: complete_task's outcome had a declared enum (success|failure|partial)
  in its tool schema but nothing validated it — an out-of-enum value (model
  typo or a weaker model not respecting the schema) silently persisted as-is,
  with only "failure" special-cased (anything else became status='done'
  regardless of what the value actually said). Now validated in
  handleTaskTool: empty defaults to "success" (unchanged), a recognized value
  passes through, anything else defaults to "partial" (safer than silently
  treating an unrecognized value as success) with a warning logged. Verified
  live: instructed the agent to call complete_task with outcome="unclear" —
  persisted as outcome='partial', not the literal invalid string.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:13:48 +02:00
926969a03f fix(agent): live chat turns persist incrementally, survive client disconnect
Fix A3 of plans/2026-07-11-nomos-agent-code-review.md. handleChat only ever
saved the assistant message ONCE, after a.chat(...) returned, using
ctx := r.Context() for that write — the same context that cancels the instant
the client disconnects (Stop button, tab close, network blip). A disconnect
mid-turn meant the final save ran with an already-cancelled context and its
error was never checked: the entire turn's tool-call history was silently
lost from the persisted transcript, even though real work (executions
launched, knowledge written) had already happened server-side.

Brought handleChat in line with resumeSession's existing pattern
(continue.go): insert a placeholder assistant row immediately, update the
SAME row after every tool call. The key fix is WHICH context the writes use —
a new pctx := context.Background() for every DB write in this handler
(session creation/touch, the user message, question auto-close, the
placeholder + incremental updates, the title update), while ctx/r.Context()
still gates the agent's own work (a.chat) and the SSE writes exactly as
before — a disconnect still correctly stops the agent from doing further
work, it just no longer also erases what it already did.

Verified live: sent a message requiring 6 tool calls (get_entity/
get_relations/get_blast_radius on two targets) and force-killed the client
connection mid-stream with curl -m 12 (confirmed via exit code 28). Before
this fix the persisted transcript would show 0 tool-call entries; after,
all 12 raw tool_use/tool_result entries (6 calls × 2) were present and
correctly attributed by tool name — proving both that progress survives an
abort and that the incremental writes aren't corrupting the data.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:10:04 +02:00
c5ffaec85b fix(agent): panic recovery on every background goroutine (B1+B2)
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together,
since the right granularity for B1 in the auto-continuation worker turned
out to require B2's restructuring anyway (see below).

B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned
nothing before this — every explicitly-spawned goroutine (continuation
worker, resumed chat turns, async execution dispatch, the SSE listener, two
duplicate sshExec implementations' output-collector goroutines) crashed the
whole process on an unhandled panic, not just that one goroutine. More
consequential post-concurrency: more simultaneous unattended background work
means more surface area for one bad input to end every running task.

New internal/safego package: Go(label, fn) launches fn in a goroutine with a
recover-and-log wrapper. Applied at every bare `go` spawn site across the
three packages. Two sites needed bespoke handling instead of the generic
helper because their callers block on a channel and a silent recover would
just make them hang until timeout: sshExec's output-collector goroutine (two
near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go)
and httpapi's ListenAndServe goroutine — both now recover AND send a
synthetic error result so the waiting select unblocks immediately instead of
waiting out the full timeout.

httpapi's sseListener got extra treatment: its per-notification handling was
extracted into handleNotification with its own recover, so a panic decoding
ONE malformed pg_notify payload can't kill the listener goroutine for every
connected SSE client — the outer goroutine spawn only needs to guard the
connection setup/reconnect code around it.

B2: cmd/nomos/continue.go's processContinuations used to run every pending
continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the
ticker — meaning (a) task B's continuation waited for task A's full (up to
10-minute) resumed turn to finish first, undercutting this session's earlier
concurrency work on exactly the path autonomous tasks depend on most, and
(b) an unrecovered panic anywhere in that call chain didn't just crash the
process (B1) — even WITH B1's recovery wrapped only at the top-level worker
spawn, the panic would still unwind the ENTIRE ticker-loop goroutine,
silently ending auto-continuation for every task until nomos restarted.
Fixed by spawning each pending item via safego.Go individually: real
parallelism, and a bad item can now only ever take down its own goroutine.

Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete
proof — a deliberate panic inside Go() that would otherwise crash the whole
test binary; reaching the assertion after it IS the evidence recovery works.

Verified live against the rebuilt containers: full chat turn round-tripped
correctly (hostname lookup, 2 iterations, normal completion) — no regression
from threading safego.Go through the tool-dispatch/continuation paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 20:05:19 +02:00
3919ec37d7 fix(agent): word-boundary matching + contracted negatives in chat-assent
Fix A1 of plans/2026-07-11-nomos-agent-code-review.md. isAssent and
isTypedConfirmation used a space-padded word-boundary check for negation
words but a bare strings.Contains for assent/confirm words — confirmed live
via test probes: isAssent("...maybe yesterday's logs...") returned true
("yes" matched inside "yesterday"), and isTypedConfirmation("I haven't
confirmed anything yet") returned true ("confirm" matched inside "confirmed",
and "haven't" wasn't in negationWords — only "don't"/"do not" were).
isTypedConfirmation is the sole gate for DESTRUCTIVE actions, so the second
case meant a message merely stating something hadn't been confirmed could
read as an explicit confirmation.

- Replaced the ad-hoc space-padding/prefix-check negation logic with proper
  tokenization (wordTokenRe) + containsPhrase, matching WHOLE tokens/phrases
  only — never a mid-word substring. Handles curly apostrophes too (a
  pre-existing gap: the old straight-quote-only check would have missed
  "don't" typed with a smart quote).
- Added contracted negatives (haven't, hasn't, isn't, wasn't, aren't, can't,
  cannot, won't, wouldn't, shouldn't, didn't, doesn't) to negationWords.
  Deliberately did NOT add a bare "not" — too broad, would false-negative
  ordinary assent like "go ahead, this is not risky".
- Added regression tests for both confirmed cases plus a couple of adjacent
  ones (eyesight/isn't, can't confirm) so a future change can't silently
  reintroduce either bug.

All existing assent/confirmation tests pass unchanged — this is a pure
robustness fix, not a behavior change for any previously-correct case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:51:57 +02:00
df393152f6 docs: nomos agent code review — gaps and improvement plan
Full read-through of cmd/nomos/ (agent.go, store.go, main.go, continue.go,
assent.go, tasks.go). Findings, ranked:

- A1 (confirmed via runnable probe): isAssent/isTypedConfirmation use
  unpadded substring matching for assent/confirm words while negation uses
  word-boundary checks — "yes" matches inside "yesterday", "confirm" matches
  inside "confirmed" with no negation word covering contracted negatives
  ("haven't"). isTypedConfirmation gates DESTRUCTIVE actions specifically.
- A2: chatWith replays a session's ENTIRE message history every turn, no
  windowing/token budget — confirmed unbounded against a documented
  production case (70 tool calls, 106KB messages).
- A3: a live turn's tool-call history is lost entirely if the client
  disconnects mid-stream (single end-of-turn save using the same
  connection-tied, possibly-cancelled context) — resumeSession already has
  the fix pattern (incremental placeholder+update), handleChat doesn't use it.
- B1: zero recover() anywhere in cmd/nomos/internal/mcp/internal/httpapi —
  every explicitly-spawned goroutine (continuation worker, resumeSession,
  executeApprovedViaAPI, sse listeners) crashes the whole process on panic.
- B2: auto-continuation processes its batch sequentially, one full LLM turn
  at a time, undercutting this session's own concurrency work on exactly the
  path autonomous tasks depend on most.
- B3: no terminal state for a permanently-failed auto-continuation.
- C1: nomos's own gateway (port 8092, directly published + mesh-reachable)
  has ZERO authentication on any endpoint — chat, session read/delete,
  chat-assent approval of gated executions, all open to anyone on the LAN.
- D1-D3: dead code (isTaskTool unused), N+1 query in recordTouched, no
  validation on complete_task's outcome enum.
- E: zero automated tests for agent.go/store.go/main.go/tasks.go — including
  today's new safety-critical logic (session-scoped windows, mcpClientPool,
  proposePlan's append-vs-replace), verified only by live manual testing.
- F1: tool list + fleet snapshot re-fetched every turn (minor).

Prioritized implementation order in the doc: A1 → C1 → B1 → B2 → A3 → D1-3 →
A2 → B3/F1, tests landing alongside each fix rather than as a deferred pass.

Also archives the now-fully-shipped concurrent-task-execution plan to done/
(all 3 required fixes deployed this session; fix 4 explicitly deferred per
its own recommendation).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:47:17 +02:00
a4ea542f3e fix(concurrency): per-session MCP client pool — removes cross-task tool-call blocking
Fix 3 of plans/2026-07-11-concurrent-task-execution.md, the throughput one.
nomos held exactly one *mcpClient for the whole process, shared by every
/chat goroutine. Its mutex was held for the full duration of each tool
round-trip, and `run` executes its SSH command SYNCHRONOUSLY inside that
round-trip (capped at up to 10 minutes) — so while Task A was mid-`run`,
every other task's tool calls, even a trivial get_entity, queued behind that
single lock. Tasks could think (LLM calls) in parallel but never act in
parallel.

The MCP server has no per-connection state to protect (newServer returns one
shared *mcp.Server instance whose handlers close only over the DB
connection pool, already safe for concurrent use) — the mutex existed purely
because the client reused one stateful transport session. So the fix doesn't
touch the server at all:

- New mcpClientPool (cmd/nomos/main.go): one *mcpClient per session id,
  created lazily (a real MCP initialize handshake) on first use and cached;
  session-less traffic (the ephemeral no-DB-store path, the structured
  /query endpoint) gets its own fixed, reused key instead of a fresh
  connection per request. Idle clients (20 min past last use — long enough
  to outlive a single slow `run`) are evicted on a 5-minute sweep ticker.
- agent.go: `client *mcpClient` → `clients *mcpClientPool`; every call site
  (buildTools, fleetSnapshot, the tool-dispatch loop) now resolves its own
  session's client via clients.get(sessionID) instead of reaching for one
  shared field. A task's own tool calls stay sequential (already true — the
  agent loop calls tools one at a time within a turn) but no longer block
  anyone else's.
- main.go: handleQuery takes the pool instead of a client (keyed "query", a
  fixed non-session slot); shutdown calls pool.closeAll().

Verified live against the deployed stack: fired a slow-but-ungated command
(`ping -c 15 127.0.0.1`, read-only per policy's allowlist, no approval
needed) as Task A, then — 2s into A's run — a trivial hostname lookup as
Task B, both through the real /chat endpoint. Task A's ping genuinely ran
~14.3s (confirmed via its own execution record and the agent's reported
output). Task B returned in 6s total, well before A finished — proving it
was never queued behind A's connection. Before this fix, B would have been
forced to wait out A's entire ~14.3s hold on the single shared client.

This completes plans/2026-07-11-concurrent-task-execution.md's required
scope — only the explicitly optional/deferred Fix 4 (a concurrency/cost cap,
pending real usage data) remains.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:15:32 +02:00
6a8fb435ad fix(concurrency): per-session stream controllers, not one global slot
Closes the known gap flagged in the previous commit (9131559). A single
module-level `activeController` meant cancelStream()/newChat() always
aborted whichever stream was MOST RECENTLY STARTED, regardless of what the
operator was currently viewing: start Task A, switch to an already-loaded
Task B, click "New task" — the click's cancelStream() would silently abort
Task A's still-running turn, even though the operator was never looking at
it and never asked to cancel it.

- Replaced the single controller with activeControllers (Map<sessionID,
  AbortController>) plus pendingController for the brief pre-'session'-event
  window of a brand-new task. Registered immediately in sendMessage (keyed by
  the continuing session id right away, or held pending until the 'session'
  event assigns a new one) and cleaned up on completion.
- cancelStream() now looks up by $currentSession (falling back to
  pendingController when no session is assigned yet) — it can only ever
  touch the stream belonging to the view being left, never an unrelated
  background task's.
- newChat() unchanged in behavior (still calls cancelStream()), now correctly
  scoped through the above.

Verified live, reproducing the exact bug: started Task A (slow, 5 tool
calls), switched to an existing Task B, clicked "New task" while viewing
B — Task A was NOT aborted, ran to completion server-side with a full,
correct final summary (previously this exact sequence would have killed it).
Confirmed the positive path is unaffected: started a task, clicked Stop
while actively viewing it — input re-enabled, stream genuinely aborted
("BodyStreamBuffer was aborted"), turn stopped mid-flight as expected.

This closes out Fix 2's scope from
plans/2026-07-11-concurrent-task-execution.md; only Fix 3 (per-session MCP
client pool, throughput) and the optional Fix 4 (concurrency cap) remain.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 19:02:50 +02:00
9131559ebd fix(concurrency): guard chat.ts's stream callback against a stale session
Fix 2 of plans/2026-07-11-concurrent-task-execution.md. sendMessage's SSE
callback mutated the global messages/currentSession stores unconditionally,
assuming only one task's turn is ever in flight. It isn't — the backend runs
every /chat request as its own goroutine with no serialization. Switching to
a different task while a previous one was still streaming let that
background stream's later events (tool_use, text_delta, ..., and worst of
all 'done''s currentSession.set) get applied to whatever the operator is now
looking at: corrupting another task's transcript, or yanking the view back
to the one they left.

- Captures the session a stream belongs to (openedFor at call time, updated
  to the real id once the 'session' event assigns one) and checks
  $currentSession still matches before every messages/error/streaming
  mutation. The task keeps running server-side regardless — dropped events
  just mean the live view isn't watching it; navigating back re-hydrates via
  REST, same as already happens for auto-continuation.
- The 'session' event itself only claims currentSession if the operator
  hasn't already navigated elsewhere since the call started (comparing
  against openedFor, which is null for a brand-new task).
- loadSessionMessages/newChat now reset `streaming` to false unconditionally
  on navigation — needed so the new guard can't leave a DIFFERENT task's view
  stuck showing streaming=true (which would also silently stop startPolling's
  loop from ever applying updates, since it bails while $streaming is true).

Known residual gap, not fixed here (matches the plan's "contained fix, not a
rearchitecture" scope): activeController is still a single global slot, so
starting a new task while another is mid-stream, then clicking "New task"
again, aborts whichever stream that slot last pointed at rather than only
the one being left. A genuine multi-session controller/store is the
plan's deferred "stretch" fix, not required for correctness here.

Verified live: started Task A with a deliberately slow 4-tool-call turn,
switched to an existing Task B mid-stream — Task B's transcript stayed
correct with zero A-originated entries and the input was NOT stuck disabled.
Task A kept running and completed normally server-side (status=done, full
6-tool transcript, 5-entity graph); navigating back loaded its complete,
uncorrupted result via REST.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 18:34:02 +02:00
9ef1ba3702 fix(concurrency): scope assent/destructive windows to session, not just agent
Fix 1 of plans/2026-07-11-concurrent-task-execution.md — the safety-critical
one. The assent window (and destructive window) were keyed purely by agent
id ("assent_window.agent:<uuid>"). With one agent:nomos entity serving every
concurrent task, this meant approving Task A's plan opened a window that ANY
concurrently-running task's config-mutation/destructive actions could also
ride, auto-executing without their own approval.

- store.go / agent.go: assentWindowActive/openAssentWindow and
  destructiveWindowActive/openDestructiveWindow/destructiveWindowKey all gain
  a sessionID parameter; keys become
  "assent_window.agent:<id>.session:<sessionID>" and
  "destructive_window.agent:<id>.target:<slug>.session:<sessionID>". Missing
  session id fails closed (no window) rather than falling back to the old
  agent-wide key.
- continue.go: the auto-continuation worker's window check moved from once-
  per-batch to once-per-pending-item, scoped to that item's own session —
  it was previously checking ONE agent-wide window for a batch that can span
  multiple tasks.
- agent.go tool-dispatch: injects `_session_id` into a COPY of the wire args
  sent to the MCP server (never into the args used for the emitted/logged/
  persisted tool call, and never part of any tool's declared InputSchema —
  invisible to the model) so the gating checks on the OTHER side of the
  process boundary know which task is asking.
- internal/mcp/server.go: assentWindowActive/destructiveWindowActive/
  classifyAndGate gain the same sessionID parameter, read from
  args["_session_id"] at the three call sites (request_execution's
  apt_upgrade/pct_create branches, and the shared classifyAndGate used by
  restart/pct_exec/systemctl/run).

Verified against the live stack with the exact scenario from the plan: opened
an assent window for session A only, then called `run` with an identical
config-mutation command for session A (window open) and session B (same
agent, no window). A auto-ran (execution status completed); B correctly
queued for approval (pending_approval) instead of bleeding through — proven
at both the MCP response text and the executions table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 18:27:29 +02:00
6932eb5eed docs: plan concurrent task execution; archive completed tasks plan
New plan grounds three concurrency issues found by tracing the actual code
(not assumed): the assent/destructive windows are keyed by agent id only
(no session dimension), so an approved plan in one task can auto-run
unapproved actions in a concurrently-running task; nomos shares one
mutex-guarded MCP client across all sessions, so a single slow `run` call
serializes every other task's tool calls behind it; and chat.ts's SSE
callback has no session guard, so switching tasks mid-stream lets the
backgrounded task's events corrupt whatever's now displayed. Proposes
session-scoping the windows (critical/first), a frontend stream guard
(contained/second), a per-session MCP client pool (throughput/third), and
an optional concurrency cap (deferred pending real usage data).

Also archives the goal-oriented-chat-control-panel plan to done/ — all 7
phases shipped and are live in production (SHA e30813a) — fixing its
internal relative links for the new depth and pointing forward to the new
concurrency plan as follow-up hardening.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 18:14:35 +02:00
e30813a43d feat(tasks): make research-first / knowledge-write-back-last explicit steps
Closes the gap that made the knowledge loop optional/implicit: every
non-trivial task now has an EXPLICIT first plan step (research) and last
plan step (write back), not just background behavior the model might skip.

New MCP tools (the agent had no way to do these before — only REST endpoints
existed, unexposed to it):
- update_entity_attributes(slug, attributes): shallow-merge new/changed facts
  into an entity (an IP, a version, a discovered port) so a future task
  doesn't have to rediscover them from scratch. No approval required — this
  updates the knowledge graph, not live infra.
- create_relationship(source, target, type): record a discovered edge
  (depends-on, hosts, provides, ...). Idempotent, FK-validated against the
  ontology's relationship_types, no approval required.

SOUL.md: restructured the task loop so step 1 is explicitly "gather
knowledge, not just status" (get_entity_knowledge, search_knowledge,
get_relations, get_blast_radius, http_get) and the last step before
complete_task is explicitly "write back" (update_entity_attributes,
create_relationship, upsert_knowledge) — both called out as real plan
entries the operator should see in propose_plan, not silent side-work. This
is what prevents the graph drifting from reality and is the concrete
mechanism behind "tasks compound."

propose_plan's tool description reinforces the same first-step/last-step
convention at the call site.

Verified against the live stack: both tools registered and callable via MCP;
update_entity_attributes merged an attribute correctly; create_relationship
rejected an invalid type (FK violation, clear error) and succeeded with a
valid type+direction, confirmed idempotent (2 calls, 1 row).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 14:14:02 +02:00
5384499903 fix(tasks): plan panel showed only the latest step, not the full plan
Root cause: proposePlan unconditionally deleted and replaced the whole
session_plan_steps list on every call. The model isn't strictly held to
"call propose_plan once with the full list" — nothing stopped it (and
production evidence + live testing showed it happening) from calling
propose_plan once per step as it worked. Each such call wiped every
already-completed step, so the operator only ever saw the model's latest
single step ("1/1") instead of the real, growing plan.

Fix, two layers:
- store.go: proposePlan now only does a destructive replace when no step
  has left 'pending' yet (a genuine pre-execution revision). Once any step
  has started, a new call APPENDS after the current max seq instead of
  wiping — so the panel accumulates the full history regardless of how the
  model chooses to call the tool. plan.proposed now carries `appended` so
  the frontend knows whether to replace or append.
- workspace.ts: plan.proposed handler respects `appended` (update vs set).
- tasks.go / SOUL.md: strengthened the propose_plan description and task-
  loop guidance to call it ONCE with the complete step list end-to-end,
  using update_plan_step (not re-calling propose_plan) to advance — fixing
  the root behavioral cause, with the store-side append as a safety net
  that holds even if the model still calls it incrementally.

Verified: forced the exact incremental-call pattern (propose_plan with 1
step, mark it running, propose_plan again with 1 more step) — the second
call appended at seq 2 instead of erasing seq 1, and its plan.proposed
event carried appended=true.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 14:07:10 +02:00
991e7d0900 feat(tasks): phase 6 — live TaskContextPanel (goal, plan, question, entities)
Replaces the chat right rail's ad-hoc Digest+Graph stack with a single
TaskContextPanel that renders the task's live working state, driven by the
always-on events stream (not the per-turn chat SSE) so it keeps updating
during server-side auto-continuation/resume:

- GoalHeader: goal + status pill (planning/executing/awaiting_input/done/
  failed), sourced from the sessions list.
- PlanProgress: ordered steps with live status icons + progress bar, hydrated
  via new GET /sessions/{id}/plan; clicking a step with a target opens its
  EntitySheet (no fake "jump to transcript" — bits-ui Collapsible content
  isn't force-mounted, so a DOM-scroll jump would silently no-op for
  collapsed tool groups).
- OperatorQuestion: the pinned structured question card (prompt/why/entity
  chips/option buttons/free-text), hydrated via new GET /sessions/{id}/
  questions; answering POSTs to the existing answer endpoint.
- SessionGraph upgraded to a live entity panel: entity.touched pulses the
  node (animated ring) and shows "Now touching <slug>"; health.changed shows
  a transient diff badge for touched entities.
- SessionDigest gains a success/failure/partial outcome banner and now also
  refetches when the task's status changes, not just on session switch.

Two bugs found and fixed while wiring this up:
- workspace.ts's status-refresh trigger only covered goal.set/task.status;
  question.raised/answered didn't refresh the sessions list, so GoalHeader's
  pill went stale after answering via the panel (resumeSession runs entirely
  server-side — no client 'done' event to piggyback a refresh on). Now every
  status-affecting event triggers the (debounced) refetch.
- Forgot to rebuild the nomos container after adding the /plan and
  /questions endpoints, so they silently fell through to the old default GET
  handler — caught via a live curl diff against the running container,
  not a code read.

Verified end-to-end against the live stack: goal/plan/question all update
without a reload as the agent works; answering a question via the panel
resumes the agent and the header pill correctly flips to Executing;
entity.touched pulses the live graph.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:52:36 +02:00
413bf54daf feat(tasks): task board UI — chat window becomes Tasks + card grid
Reframes the chat surface as tasks:
- New Tasks.svelte: a card grid of tasks, each showing status (Running /
  Needs input / Done / Failed), the goal as title, the outcome summary, and
  relative time; filterable by status with live counts; delete on hover;
  "New task" and per-card click open the conversation.
- Board updates LIVE off the events stream (goal.set / task.status /
  question.*) via an explicit liveEvents.subscribe with a debounced refetch —
  scanning all events newer than the last seen, since entity.touched bursts
  bury task events below index 0.
- App shell: primary nav "Chat" → "Tasks" (board is now the home route),
  "New chat" → "New task", conversation header gets a Tasks / Conversation
  breadcrumb. Removed the superseded Sessions page.
- api.ts Session type carries the task fields (goal/status/outcome/summary).

Also fixes a pre-existing SSE bug that blocked ALL live updates app-wide:
writeSSE emitted `event: <type>`, which EventSource only delivers to
addEventListener(type) handlers — but stores/events.ts (and every page reading
liveEvents) consumes via onmessage, which never fires for named events. So the
live stream delivered nothing to the UI. Dropped the event-name line; the type
is already in the JSON payload, and new event types now need zero client
changes. SSE test still green (it parses data: lines).

Verified in the browser against the live stack: the board renders 50 tasks
with correct status buckets; a goal-driven task appears and flips to a Done
card with its summary in real time without a reload; Events page confirms the
stream now delivers to onmessage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:22:27 +02:00
014e5c74e0 feat(tasks): phase 5 — ask_operator (structured question, pause, resume)
The last backend piece: when the agent hits a decision only the operator can
make, it surfaces a structured question instead of guessing or stalling.

- ask_operator(prompt, why?, options?, context_entities?): nomos-local tool
  that records a session_questions row, moves the task to awaiting_input, emits
  question.raised, and ENDS the turn (the agent loop returns after it, so the
  agent can't barrel past its own question). The prompt becomes the assistant's
  visible message so the question also shows inline in the transcript.
- Two resume paths, both close the question + emit question.answered + return
  the task to executing:
  - Panel: POST /sessions/{id}/questions/{qid}/answer → resumes the agent in the
    background with the answer injected (reusing the continuation machinery,
    refactored continueSession → resumeSession). Returns 202; the reply lands via
    message polling.
  - Chat reply: the next chat message on a task with an open question IS the
    answer — auto-closed in handleChat; the turn itself is the resume.

Verified end-to-end: forcing a decision paused the task at awaiting_input with
the structured question (prompt/why/options/entities); a panel answer resumed
the agent (it acknowledged host:strong and continued); a plain chat reply
auto-closed a second question. Cleanup + tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 13:00:15 +02:00
be3ce761d4 feat(tasks): phase 4 — structured plan steps (set_goal/propose_plan/update_plan_step)
Gives a task a legible, live-advancing plan via three more nomos-local tools:

- set_goal(goal): records the task goal, status → planning, emits goal.set.
- propose_plan(steps[]): persists ordered steps (clean replace for v1 — a
  revision starts a new list), status → executing, emits plan.proposed with
  the persisted steps (id+seq) so the panel can address them.
- update_plan_step(seq, status, execution_id?): advances a step, stamping
  started_at/finished_at, emits plan.step.started/finished. Anchors the event
  to the step's target entity when it has one.

Belt-and-suspenders: when an execution linked to a step reaches a terminal
state, the api auto-closes the step (closePlanStepForExecution in
emitExecutionEvent) and emits plan.step.finished — so the board stays honest
even if the agent forgets to close a step it started.

Verified end-to-end: a goal-driven task fired goal.set → plan.proposed →
2× step.started/finished → task.status on the SSE stream; both steps persisted
done with start/finish timestamps; status progressed planning→executing→done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:50:27 +02:00
532310bb4b feat(tasks): phase 3 — close the knowledge loop (complete_task + retrieval)
Adds the compounding knowledge loop the task model is built around:

- complete_task(outcome, summary): a nomos-LOCAL, session-scoped tool (the
  shared MCP server has no session id). Introduces the local-tool mechanism —
  buildTools appends task tools, the agent loop routes them to handleTaskTool
  instead of the MCP client. Sets the task's terminal status/outcome/summary,
  mirrors it onto the task entity, and emits task.status.
- Knowledge → task linkage: after a successful upsert_knowledge in a task,
  nomos links the note to the task entity (documents) and emits
  knowledge.recorded, so the task's outcome view shows what it learned. The
  note's about-link to the involved entity (written by upsert_knowledge) is the
  retrieval path future tasks use.
- SOUL: every chat is a task loop — retrieve prior knowledge FIRST
  (get_entity_knowledge on the target), plan, execute, record learnings, then
  complete_task. Scales down for trivial read-only tasks.
- deleteSession now cleans up the task entity, its relationships, and its
  task-scoped events (was orphaning them); the knowledge doc itself and its
  about-links survive, as knowledge should outlive the task.

Verified end-to-end: a task recorded a note and completed; task.status +
knowledge.recorded hit the SSE stream; status=done/outcome=success persisted;
the note linked to both lxc:caddy (retrieval) and the task; a future
get_entity_knowledge(lxc:caddy) surfaces it; delete cleaned edges+events (0/0/0)
while the knowledge survived.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:43:29 +02:00
3dba2e550a feat(tasks): phase 2 — entity.touched events + task→entity involves edges
As the agent runs a task, record which entities each tool call references:
write an idempotent task —involves→ entity relationship and publish one
entity.touched event per entity (correlation_id = session, data {slug,tool}).
Extracted from tool ARGS only — never results — so a bulk fleet query can't
drag every entity into the task graph; bulk/no-slug tools stay silent.

Emitted from the nomos agent loop rather than the shared MCP wrapper, which
has no session id. The involves edges make a task's graph neighborhood its
involved-entity set (queryable via get_relations) — the substrate for the
knowledge loop; the events are the live pulse the context panel consumes in
phase 6.

Verified end-to-end on the local stack: a chat referencing lxc:caddy/lxc:gitea
produced entity.touched on the browser SSE stream with slug+tool+correlation,
and exactly one involves edge per entity despite repeated touches.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:35:27 +02:00
72e9fe534e feat(tasks): phase 1 — elevate chat session to a task (schema + task entity)
Migration 018 adds goal/status/outcome/summary/entity_id to agent_sessions
and creates session_plan_steps + session_questions. Registers a 'task'
entity type and an 'involves' (task→entity) relationship in the ontology
so each session anchors its knowledge and involved-entity edges on the
existing relationships graph.

nomos createSession now mints a task:<session-id> entity (type task) and
links it via agent_sessions.entity_id — best-effort so chat never blocks on
it. listSessions/GET /sessions surface the new task fields.

No behaviour change yet; this is the data foundation for the task board and
live context panel. Verified end-to-end against the local stack: migration
applied, ontology ingested (60 types/47 rels), a new session mints a linked
task entity and the API returns status/goal/entity_id.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:25:28 +02:00
eed6e3b1c5 docs: task-centric chat plan (goal → single-approval → autonomous + knowledge loop)
Reframes the chat surface as a board of tasks: each task carries a goal,
a plan approved once, a lifecycle status, an outcome, and a knowledge
loop that links learnings to the involved entities (and the task entity
itself) via relationships so future tasks compound. Supersedes the
sidebar-only framing and the free-form chat portion of the control-room
web UI plan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:17:36 +02:00
ef5a92269b docs: reconcile plans/ status against actual code state
Audited all 10 active plan docs against the codebase (not just commit
titles). 5 were fully shipped and stale-tagged "Planned"/"In Progress" —
moved to done/ with verification notes. The other 4 got corrected
Planned→In Progress status plus concrete remaining-gap notes so the next
pass doesn't re-derive what's already done.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 11:42:26 +02:00
52e16e04ca feat: Learning page — capability timeline + trend, built on real data
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The plan's "learning view" (runbook success-rate trends, promoted
skills, capability timeline) assumes the patterns/skills/feedback
pipeline is populated. It isn't: all three tables are empty in
production and nothing in the codebase ever writes to feedback, so
building the UI against them today would ship a permanently-empty
page. Scoped instead around data that's real and growing —
executions — while still wiring up /patterns and /skills so the page
needs no rework once that pipeline exists.

New /api/v1/learning/timeline (per-verb first-success date + success
rate, parsed via the existing splitAction helper) and
/api/v1/learning/trend (30-day daily success/fail counts), both
read-only queries against executions. Patterns and skills sections
call the existing (untouched) ListPatterns/ListSkills endpoints and
render an explanatory empty state instead of nothing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 21:28:19 +02:00
6192c35c10 fix: close approval bypass in restart/systemctl/pct_exec
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Found live: a chat request to restart caddy (the reverse proxy for the
whole fleet) executed instantly over SSH with zero approval. Root
cause was in request_execution's legacy handler — restart, pct_exec,
and systemctl (outside enable/disable) executed immediately with a
hardcoded risk_class='reversible_low' that was never actually checked
against anything, bypassing the classifier entirely. Only the `run`
tool's commands were ever gated.

Extracted the run tool's classify -> execute-or-queue logic into a
shared classifyAndGate() and route restart/pct_exec/systemctl through
it too, so every mutating path — regardless of which tool the model
reaches for — gets the same read-only/config-mutation/destructive
classification and approval gate. systemctl restart is already covered
by an existing classifier test (config_mutation), so no new test
needed; the gap was that request_execution never called the
classifier at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 21:03:38 +02:00
682326382e feat: surface blast radius on approval cards
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Pending-approval cards showed target and risk but not what else the
action would affect — the operator approved config_mutation/destructive
commands blind to downstream impact, even though the graph-walk
(blast_radius() SQL, GetBlastRadius endpoint) already existed and was
just never wired into the approval path.

Fetch it once per pending approval and render "Affects N downstream: …"
on both the normal and destructive approval cards, reusing the existing
fetchBlastRadius() API client function which was already written but
unused anywhere.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 20:52:05 +02:00
ac48390796 feat: global activity feed + session digest
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Ops "Executions" tab showed raw target UUIDs, alphabetical (not
recency) order, and a stale status vocabulary from an earlier schema
iteration — never actually usable as a live "what's happening" view.
Replaced with a new recency-ordered /api/v1/activity/recent endpoint
and matching table (human-readable action summaries, risk/status
badges, duration, inline error preview).

Also added /api/v1/activity/session/{id} + a collapsible SessionDigest
panel in the chat rail, answering "what did this session actually do"
(executions by status, entities touched, knowledge written) — the
missing piece for proactive outcome reporting to be visible in the UI,
not just in the chat transcript.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-10 20:09:09 +02:00
40999b0b40 fix: knowledge/recent returned empty items — timestamptz couldn't scan into string
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live immediately after deploying: the endpoint returned 200 with
correct-looking stats (total=56, agent_authored=2) but items=[] always,
regardless of limit/source. Root cause: pgx v5 can't scan a timestamptz
column directly into a Go string — Scan() errored on every single row, and
that error was silently swallowed by a bare `continue`, so every row was
dropped with no trace in the logs. Fixed by casting updated_at::text in the
SQL (matching how every other handler in this codebase already returns
timestamps) and logging scan failures instead of swallowing them, so this
class of bug can't hide silently again.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 19:26:48 +02:00
ec41c0b828 feat: learning view — make the growing knowledge base visible
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
First slice of the observability/learning UI (the "see the system come alive
and learn" ask). The Knowledge page was search-only — blank until you typed —
so the knowledge Nomos now writes via upsert_knowledge was invisible unless
you knew to search for it. Now the page LEADS with what the system knows and
is learning:

- internal/httpapi/knowledge.go: GET /api/v1/knowledge/recent — recency-ordered
  knowledge + a stats header (total, agent-authored, learned-this-week,
  by-kind). Custom route (not OpenAPI-generated), same auth as the rest.
- web Knowledge page rewrite: stat cards up top (Total / Written by Nomos /
  Learned this week / runbooks-investigations), then a "Recently learned" feed
  with agent-authored notes highlighted and badged "learned by Nomos", tags,
  and relative timestamps. A toggle filters to Nomos-only. Search still works,
  now as a mode you enter/clear rather than the whole page.

This turns "the system is getting smarter" from a claim into something you
watch fill up: every gotcha the agent records shows here within seconds.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 19:22:35 +02:00
60edff2065 feat: knowledge write-back (upsert_knowledge) + proactive outcome reporting
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
From the last (successful) TypeType deploy session, two gaps the operator hit:

1. Knowledge write-back — the missing half of the loop.
   The agent could read the knowledge base (search_knowledge/get_entity_knowledge)
   but had no way to WRITE it, so everything it learned (the Dragonfly memlock
   rlimit gotcha, the NAT-hairpin DNS issue, etc.) lived only in an ephemeral
   chat message and was lost — the system could never actually "get better."
   This is the `upsert_knowledge` MCP tool the 2026-07-08 gaps plan called for.
   - internal/mcp/server.go: upsert_knowledge(title, content, about?, tags?,
     kind?) writes a document/investigation/runbook entity + knowledge_entities
     row (search column is generated), upserts by slug so re-titling updates in
     place, and optionally links it to the entity it's about so
     get_entity_knowledge surfaces it there.
   - SOUL.md: capture non-obvious findings/deploys/gotchas as part of finishing
     work, not only when asked "what did we learn".

2. "I had to ask for status multiple times."
   The clearest cause: a long working turn (64 tool calls) that exhausted the
   iteration cap ended with a bare "max iterations reached without final
   answer" — a dead end that forced the operator to ask what happened.
   - cmd/nomos/agent.go: on exhaustion, make one final no-tools LLM call
     (finalSummary) asking for a status report — what was accomplished, current
     state, what remains — so the turn always ends with a real outcome.
   - maxIterations 25 -> 40 (the decomposed per-step pct_create flow legitimately
     needs more steps).
   - SOUL.md: always end a turn with a clear outcome; never end silently or on a
     bare tool call — the operator can't see the tools working and reads silence
     as "nothing happened".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 19:07:11 +02:00
233b5e4519 feat: live visibility into what the agent is running (no more silent waiting)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Operator: "I'd like to be able to see in the chat what the agent is actually
running, right now I just wait while nothing happens." Two compounding gaps:

1. The auto-continuation worker (cmd/nomos/continue.go) had zero live push —
   its result only appeared on a manual page reload, so approving a plan and
   watching the chat looked completely dead even while the agent was actively
   working.
2. Even with polling, continueSession only persisted ONE message at the very
   end of a continuation — a continuation that runs several tool calls before
   concluding would still show total silence for however long that took.

Fixed both:
- web/src/lib/stores/chat.ts: polls the current session's messages every 3s
  between turns (never while a live stream owns the message list) and merges
  in anything new. Started after a live turn ends and when a session loads;
  stopped on new-chat/session-switch.
- cmd/nomos/store.go: insertMessageReturningID/updateMessage — lets a message
  be created as a placeholder and updated in place.
- cmd/nomos/continue.go: continueSession now inserts a placeholder the
  instant it starts (renders as the existing "thinking" dots — immediate
  feedback that something is happening) and updates that SAME row after
  EVERY tool call, not just at the end. A poll within ~3s of any tool call
  landing shows it — individual `run` commands appear as the agent issues
  them, not just the final rolled-up summary.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 15:20:10 +02:00
13458e467c fix: the actual root bug — assent-window auto-approve never dispatched work at all
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The previous commit fixed a context-cancellation bug in the auto-approve path
and appeared to fix things, but re-testing end-to-end after deploy showed the
execution STILL never completed — just via a different symptom
("no pending execution found for approval" in the logs). Dug further and
found the real, deeper bug underneath: this whole mechanism has never
actually worked.

autoApprove() directly flipped BOTH approvals.status and executions.status to
'approved' via raw SQL, then called executeApprovedViaAPI to POST to the
decision endpoint. But DecideApproval's own logic specifically looks for the
execution still at status='pending_approval' to find and dispatch the real
SSH work (executeApprovedAction) — autoApprove's premature flip meant that
lookup always found zero rows. DecideApproval's UpdateApprovalStatus call
also silently no-ops the same way (sqlc :exec doesn't surface "0 rows
affected" as an error). Every assent-window auto-approved pct_create/
apt_upgrade has been sitting at 'approved' forever with the real work never
triggered — indistinguishable from "still running" until you check.

Fix: remove autoApprove() entirely. Call executeApprovedViaAPI directly
against the untouched pending_approval row from createApproval — identical
to the manual Approve-button path, just without the human click. DecideApproval
is now the single place that transitions status and dispatches, for both the
manual and auto-approved paths, closing the class of bug where two code paths
raced to do the same state transition.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 15:00:40 +02:00
7387df3276 fix: assent-window auto-approve goroutine used the request-scoped context
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live testing the new atomic pct_create: an assent-window
auto-approved pct_create appeared to "run" (logged "auto-approved... running
now") but the execution stayed stuck at 'approved' forever. Root cause:
`go executeApprovedViaAPI(ctx, ...)` passed the MCP tool-call's own context —
which is cancelled the instant the triggering /chat request's HTTP response
completes, i.e. on every normal turn. The spawned goroutine's POST to the
approval-decision endpoint died with "context canceled" before it could even
start the real work, and nothing surfaced this to the operator or the agent —
the execution just sat at 'approved' with no error, indistinguishable from
"still running."

This is exactly the context-lifetime bug class httpapi's own approval
goroutine (executeApprovedAction) already avoided by using
context.Background() — it had just been missed in these two call sites
(apt_upgrade and pct_create auto-approve). Fixed both to use
context.Background(), matching the correct pattern already in place
elsewhere. Audited for other goroutines spawned with a request-scoped ctx —
none found; the sshExec internal goroutines are synchronous/waited-on via
select and correctly scoped to the call.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 14:51:52 +02:00
2e922f6421 feat: decompose pct_create into atomic create + agent-driven install; add scoped destructive window
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Closes the two remaining open points from the auto-continuation work.

1. Atomic pct_create (observability, the bigger of the two):
   pct_create used to bundle create + apt install + post_install script into
   one black-box multi-minute SSH call — the agent got back a single opaque
   success/fail with no way to see (or fix) which step actually broke.
   Removed the whole post-create provisioning block (and the now-dead
   provisionScript/sanitizePkgs helpers + their tests). pct_create is now
   create + start + register ONLY — fast, and its result is fed back to the
   agent via auto-continuation almost immediately. The agent installs
   packages and runs setup as its OWN sequence of `run` calls against the new
   lxc:<hostname>, observing each command's real output and able to diagnose
   and retry exactly the step that failed — the same recovery loop already
   proven for the general case, now applied to installs too, instead of
   requiring a separate black-box mechanism.
   - services/post_install removed from the pct_create params struct and
     from the MCP tool schema/SOUL.md docs.
   - SOUL.md: explains the new flow, moves the Docker CLI gotcha and DNS
     troubleshooting guidance to be steps the agent runs itself.

2. Scoped destructive window (targeted autonomy for recovery):
   Verified live in the previous session that a destructive recovery (a
   failed destroy needing stop-then-destroy on the same container) required
   TWO separate typed confirmations for what was clearly one recovery
   action. Added a narrow, TARGET-scoped 15-minute grant
   (destructive_window.agent:<id>.target:<slug> in autonomy_settings,
   shared key format across cmd/nomos and internal/mcp) that opens only
   after an EXPLICIT typed confirmation (never loose assent) or an explicit
   button-approval of a destructive step, and only ever covers further
   destructive commands against that SAME target. A different target always
   needs its own fresh confirmation — this narrows risk instead of loosening
   it globally, unlike broadening the general assent window to cover
   destructive actions would have.
   - cmd/nomos/store.go: openDestructiveWindow/destructiveWindowActive/
     executionTarget.
   - cmd/nomos/agent.go: opens the window when a typed confirmation grants a
     destructive chat-assent execution.
   - internal/mcp/server.go: `run` tool checks the window before gating a
     destructive command; auto-runs if active.
   - internal/httpapi/phase3.go: DecideApproval opens the same window when a
     destructive execution is approved via the button/API, for parity with
     the chat-assent path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 14:45:09 +02:00
6f9998fa29 fix: auto-continuation silently dropped LLM errors + added outer retry
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live: the new auto-continuation worker (previous commit) worked end
to end for the happy path (provision -> auto-verify -> report success, zero
operator ticks). But testing a failure-recovery case (a destroy that failed
because the container was still running) surfaced a real bug: continueSession's
emit closure only captured "text" events, so when chatWith ended the turn on
an "error" event (LLM returned an empty/refusal response, internal retry also
empty), the worker persisted a completely blank, uninformative "auto" message
— no sign anything had gone wrong, undermining observability of the very
mechanism just built.

- Capture "error" events and, if the turn produced no text/tool_calls at all,
  persist an explanatory placeholder instead of blank.
- Add one outer retry of the whole chatWith call when the first attempt
  produces nothing — the principle behind this whole feature ("don't give up
  on the first error") should apply to the continuation mechanism itself, not
  just the homelab commands it's continuing.

Also verified live: recovery-from-failure works via the normal chat path once
prompted, and cleaned up the test container.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 14:30:23 +02:00
d2f749d33d feat: event-driven auto-continuation — agent runs an approved plan to completion
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)

This makes the system the event loop instead:

- migrations/017: nomos_plan_executions links each gated execution to the chat
  session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
  to the session. A background worker (continue.go) polls for those executions
  reaching a terminal state and — while the agent has an open assent window (an
  approved plan is in flight) — re-invokes the agent with the result
  ("execution X completed/failed: <result>"), so it proceeds to the next step
  or diagnoses+fixes the failure, with no operator tick. Guarded against loops
  (mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
  replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
  opens the assent window, so auto-continuation works regardless of how the
  operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
  don't poll get_execution_status, don't wait for "continue"; end the turn and
  keep going step by step until the goal is verified or a genuine blocker.

This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 14:19:43 +02:00
c3699157ae fix: chat-assent chicken-and-egg + agent stops after errors
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Three fixes for the session where the agent proposed a plan, waited
for 'proceed', then re-queued instead of being auto-approved:

1. Chat-assent fallback: when the operator says 'proceed' but the
   preceding turn had NO pending approvals (agent proposed plan in text
   without calling request_execution), inject a system note telling the
   agent to execute the plan now. Opens the assent window so subsequent
   config_mutation commands auto-run.

2. SOUL.md: instruct agent to ALWAYS call request_execution/run when
   proposing a plan, not wait for 'proceed' first. This ensures a
   pending approval exists for chat-assent to grant.

3. SOUL.md: stronger Docker instructions — Debian 13's docker.io package
   installs the daemon but NOT the docker CLI binary. Must use
   get.docker.com in post_install. Added 'handling errors' section:
   diagnose, try alternatives, continue — don't stop after one failure.
2026-07-10 13:51:04 +02:00
7ff344ab47 feat: request_execution respects assent window — pct_create and apt_upgrade auto-approve
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
When the operator has approved a plan via chat assent (assent window
active), pct_create and apt_upgrade now auto-approve and execute
instead of queuing for a separate approval round. The auto-approve
path updates the approval+execution status in the DB, then calls the
HTTP API's decision endpoint to trigger executeApprovedAction — same
code path as a manual Approve button, consistent audit trail.
2026-07-10 13:33:31 +02:00
657e1a8be1 feat: assent window + compound read-only classification + continue-after-approval
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Agent stopped after every approval step, forcing operator to type
'continue' 7× per deploy session. Root causes and fixes:

1. Compound read-only commands (e.g. 'systemctl status; journalctl')
   defaulted to config_mutation — now splits on ;/&&/||/| and classifies
   as read_only if all segments are inspection verbs. Added grep, wc,
   sort, uniq, cut, tr, dpkg -l, apt list, docker stats to allowlist.

2. curl|sh was classified destructive, forcing typed confirmation for
   legitimate installs (get.docker.com). Demoted to config_mutation —
   loose assent grants it, no typed phrase needed.

3. SOUL.md said 'STOP after queuing' — replaced with 'continue working
   on non-blocked steps'. Added assent window section instructing agent
   to carry out the full plan after approval.

4. Assent window: when operator approves a plan via chat assent, a
   30-minute window opens where config_mutation commands auto-run
   without re-approval. Agent writes expiry to autonomy_settings; MCP
   run tool checks it before gating. Destructive never auto-runs.

5. System note after approval now says 'CONTINUE executing the full
   plan — do not stop and wait for continue.'
2026-07-10 13:10:57 +02:00
7a7ce2b89b fix: gateway pre-flight check could never actually fail
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live that after deploying the "fixed" bridge-bound pre-flight, it
still let a known-bad vmbr0+192.168.8.2 config straight through to a full
pct_create with no error. Root cause: the check used
`strings.Contains(pingOut, "REACHABLE")` against markers "REACHABLE" /
"UNREACHABLE" — but "UNREACHABLE" contains "REACHABLE" as a substring, so the
containment check was true for BOTH outcomes. The pre-flight was structurally
incapable of ever failing, regardless of the actual ping result.

Fixed with distinct, non-overlapping markers (PREFLIGHT_OK/PREFLIGHT_FAIL)
and exact-match comparison, pulled into a small gatewayPreflightPassed()
helper with a unit test asserting the exact historical bug case
("UNREACHABLE" must be false) so this bug class can't silently recur.

Re-verified live end-to-end: manually re-tested the exact ping command
(confirmed UNREACHABLE via vmbr0), and this was caught only by actually
running the check against production, not by reading the code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 12:02:42 +02:00
3f3de18b23 fix: pre-flight gateway ping must bind to the specific bridge, not the host default route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Verified live that the pre-flight check added in the previous commit had a
real gap: a plain `ping <gateway>` from the Proxmox host succeeds via the
HOST's own routing table (which can have routes to a subnet through paths
the host alone knows about), even when the CONTAINER — attached via a plain
bridge with only a naive on-link default route — can never actually ARP that
gateway. Confirmed by creating a real test container on vmbr0 with
gw=192.168.8.2: the host-wide ping had said "reachable," but pinging from
inside the container showed 100% packet loss. Fixed by binding the pre-flight
ping to the specific requested bridge (`ping -I <bridge>`), which correctly
rejects vmbr0 for that gateway instead of false-positiving via the host's
broader routing table.

Also confirmed live: vmbr1 does exist and is up on strong (contrary to the
possibly-stale host doc), matching what romm/seanime's docs already said.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 11:55:53 +02:00
82b0ad2298 fix: pct_create fast gateway pre-flight + bridge param (real root cause of TypeType's DNS failures)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Investigated why the operator couldn't get past "no DNS/connectivity" across
multiple retries even after Nomos correctly diagnosed and fixed the gateway
(192.168.8.1 -> 192.168.8.2). It still failed. Root cause, confirmed from
strong's own documented network topology: on `strong`, vmbr0 physically
bridges only to 192.168.178.0/24 — the 192.168.8.0/24 service network is
reached via a Fritz!Box static route, not a local bridge. A container
attached to vmbr0 can never reach a 192.168.8.x gateway no matter which
address in that range is picked; ARP for it just gets silently dropped
(matching the earlier hang symptom). The gateway was never the problem — the
bridge was. 192.168.8.0/24 is also segmented into /28 blocks each with their
own gateway (192.168.8.2 is only the .0-.15 block's gateway), so even a
correct bridge with a copy-pasted gateway from a different block would still
fail.

No amount of retrying with a different gateway guess could have fixed this —
the missing fact (which bridge reaches which subnet, and the per-/28 gateway)
isn't inferable from the subnet alone.

- pct_create gets a `bridge` param (was hardcoded to vmbr0) so a correct
  bridge can actually be requested once known.
- Fast pre-flight: for any static IP, ping the gateway from the target HOST
  before creating anything. Was: a bad config took a multi-minute hang (or,
  after last commit's timeout fix, ~2min) before failing. Now: ~2 seconds,
  with a message that explicitly says not to guess a different gateway in
  the same subnet — find a real neighbor's config or use DHCP.
- SOUL.md: DHCP is now framed as the default, not a fallback; static IP
  requires finding an existing LXC on the same host in the same /28 and
  copying its bridge+gateway verbatim — inventing one is explicitly called
  out as the failure mode that caused this exact incident.
- MCP tool schema: pct_create's params description now documents `bridge`
  and the neighbor-copy rule directly in what the model reads at call time,
  not just in SOUL.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 11:48:06 +02:00
8950bada44 fix: sshExec had no timeout — a hung remote command blocked forever
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).

Two compounding bugs made this unrecoverable without manual intervention:

1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
   NO execution timeout — `session.CombinedOutput()` blocks until the remote
   command exits, with no deadline. A hung remote process blocks the Go
   goroutine forever; the execution can never leave 'running', and the
   operator has no way to make it stop. Fixed: both now race the SSH call
   against a 10-minute hard timeout, closing the session/client and
   returning a clear "timed out after 10m0s" error if exceeded. (The
   mcp/server.go copy also still had the original "swallowed non-zero exit"
   bug from before that fix was applied to httpapi's copy only — fixed here
   too.)

2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
   connectivity — it doesn't; a black-holed network can make each call hang
   far past the resolver's nominal timeout, so the documented "~90s" budget
   was never real. Wrapped every attempt in `timeout 3` so the wall-clock
   budget is now actually enforced (~2min worst case), and the failure
   message now suggests checking the net0 gateway.

Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):

- InlineApproval's track() polling loop had its own ~6min ceiling and simply
  STOPPED polling after that — silently going stale before the backend (now
  correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
  with margin, and added a distinct 'stalled' state if that's ever exceeded
  (explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
  execution's created_at), the actual command being run, and the execution
  ID — previously just a static "this can take a minute" with zero
  information. Also added command display to the destructive pending-
  approval card for full transparency before confirming.

Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 10:45:47 +02:00
f936098364 fix: approval UI was never mounted; add typed confirmation for destructive
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Root cause of "chat gave me no further feedback — had to go to Ops": two
compounding bugs, found by reading the actual production session transcript.

1. InlineApproval.svelte — all of last session's live-status/self-heal work —
   was never imported or rendered anywhere. Chat.svelte had its own separate,
   much dumber approval bar (no status tracking, no destructive handling, just
   silently disappears after clicking) that WAS the one users actually saw.
   Deleted the dead bar and its state; InlineApproval now renders per-message.

2. chat.ts's extractApprovals hardcoded `tool.name === 'request_execution'`,
   so any approval raised by the newer `run` tool was invisible — no card, no
   feedback, nothing to self-heal, forcing the operator to the Ops page with
   zero acknowledgement in the conversation. This was the actual proximate
   cause of last night's destroy-135 session. Fixed to match on response
   shape, not tool name, so it doesn't silently break again for the next new
   gated tool.

3. Nomos was telling operators "type something like 'I confirm destroy 135'"
   for destructive actions (SOUL.md) but no backend path ever consumed that
   phrase — chat-assent explicitly (and correctly) excludes destructive from
   loose assent, but I never built the alternative. Added
   isTypedConfirmation() (cmd/nomos/assent.go): stricter than loose assent,
   requires an explicit "confirm" statement, only applies to destructive-
   flagged pending approvals.

4. InlineApproval's completed-state hardcoded "Provisioned successfully" —
   wrong/confusing for a destroy or arbitrary `run` command. Now says
   "Completed on <target>" and shows the actual command output, verified live
   against the real destroy-135 execution.

Verified live in a real browser against the production API/DB (dev server
proxying to :8090): the historical stuck session now retroactively renders
both executions as resolved with correct wording and real output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 10:01:53 +02:00
d08a985ea9 fix: execution slug collision under back-to-back requests
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Live testing hit `entities_slug_key` violations: exec slugs used an 8-char
prefix of a UUIDv7, whose leading bytes encode a millisecond timestamp — two
executions created seconds apart can share a prefix. Use the full UUID
(guaranteed unique) for the exec entity's slug/name in request_execution, the
new `run` tool, and the REST RequestExecution handler — all three had the
same pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:37:08 +02:00
9539759db6 fix: NULL-scan bug in LXC target resolution for entities without a host attribute
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Found live: `run` against lxc:caddy failed with "missing pve_id" even though
pve_id=121 was present — caddy is an inventory-seeded LXC with no `host`
attribute at all (only pct_create-provisioned LXCs set one). The combined
query scanned attributes->>'host' (SQL NULL) into a plain Go string, which
errors the whole Scan — including the pve_id column that scanned fine.
COALESCE the host column to '' so a missing host attribute degrades to the
documented default instead of failing the whole resolution.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:32:45 +02:00
d52968876a feat: general gated run primitive + chat-assent approval (Layer 0)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Implements the first slice of plans/2026-07-10-general-gated-execution.md:
Nomos gets one general execution tool instead of only a fixed action enum,
gated by an automatic risk classifier, and approval can be granted by the
operator just replying in chat instead of clicking a button.

- internal/policy/command.go: ClassifyCommand(cmd, declaredRisk) — rule-based
  read-only allowlist + destructive denylist, default-escalate to
  config_mutation for anything else. Classification can only ESCALATE the
  caller's declared risk, never de-escalate it (destructive always wins even
  if declared read_only). Compound commands (&&, ;, |, $()) never qualify for
  the read-only fast path. Full test corpus.
- internal/mcp/server.go: new `run` MCP tool — target (host:/lxc:), command,
  purpose, optional declared_risk. Read-only commands execute immediately;
  everything else queues an approval exactly like pct_create today, executed
  via httpapi's existing executeApprovedAction. Also fixes a real latent bug:
  pct_exec resolved an LXC's host attribute without the "host:" prefix, so it
  could never find the Proxmox host — new resolveExecTarget/resolveRunTarget
  helpers (mcp + httpapi) fix this for both the new `run` action and existing
  actions that route through the same execution path.
- internal/httpapi/phase3.go: "run" case in executeApprovedAction; fixes two
  bugs found while wiring this up — (1) DecideApproval hardcoded risk_class to
  'config_mutation' on every approve, silently corrupting the audit ledger for
  every other risk class; (2) denying/revoking an approval never updated the
  linked execution's status, so it stayed 'pending_approval' forever instead
  of reflecting the decision.
- cmd/nomos/assent.go: deterministic (not LLM-judged) chat-assent detection.
  Scoped to the immediately-preceding assistant turn's pending approvals only
  — an old "yes" can't retroactively approve something new. Destructive-risk
  actions are excluded from loose assent. Approves via the same HTTP decision
  endpoint the UI button calls, so both paths share one audit trail.
- web/.../InlineApproval.svelte: self-healing poll — a pending approval card
  now picks up being decided via ANY path (chat assent, Ops page, Matrix), not
  just its own button. Previously the banner stayed stuck showing
  Approve/Deny even after the action had already run elsewhere.
- nomos/SOUL.md: `run` is now the general capability ("no fixed menu, only a
  risk gate"); documents chat-assent behavior and the destructive exception.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:28:00 +02:00
9daf8220f2 plan: add observability layer and chat-assent approval to gated-execution plan
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Operator directives: (1) the UI must show what's executing, its risk
classification, live status, and what knowledge the session created — the
system's growth should be visible, not just trusted. (2) approval should be
granted by chat assent ("go ahead"), not a separate button; destructive
actions still require a typed confirmation phrase.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:12:09 +02:00
4a96f46e76 plan: general gated execution — unlimited actions, gated by risk classifier
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Evaluate the agent's action path against the OIKOS.md design. Finding: the
intended model (unlimited runbook-driven actions gated by a risk classifier)
already exists on paper and in scaffolding, but the live agent path regressed
to a hard-coded 5-action enum that bypasses the classifier. Plan a layered
realignment: (0) one general gated `run` primitive, (1) runbooks-as-data as the
reliable fast-path, (2) learning. Chosen v1 posture: approve-most (read-only
auto-runs, all state changes gate). Incremental, each step shippable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 09:03:23 +02:00
5f888e6386 fix: set C.UTF-8 locale in provisioning script
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Fresh Debian LXCs have no locale configured, spamming "apt-listchanges:
Can't set locale" / perl warnings across every install and breaking some
packages' post-install scripts. Pin LANG/LC_ALL=C.UTF-8 (and hoist
DEBIAN_FRONTEND) at the top of the in-container bootstrap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 01:21:23 +02:00
f248508919 feat: robust provisioning (DNS self-heal) + live execution feedback in chat
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Production session provisioned the container but the service never installed:
apt failed with "Temporary failure resolving deb.debian.org" — a static-IP LXC
whose assigned nameserver couldn't resolve. The operator also got zero feedback:
the approval banner just sat there with no running/complete/failed status.

Backend robustness (provisionScript):
- Wait for real DNS/connectivity inside the container before apt, and self-heal
  /etc/resolv.conf to a public resolver (1.1.1.1/8.8.8.8) if the assigned one
  is dead. `set -e` after the gate so apt/post_install failures surface.
- apt-get update/install with Acquire::Retries=3.

Frontend feedback (InlineApproval):
- After approve, poll GET /executions/{id} and show live phase: submitting →
  provisioning… → provisioned successfully / execution failed (with the error).
- add getExecution() to api.ts.

Agent guidance (SOUL.md):
- omit vmid (auto-assigned), prefer dhcp, docker-compose-plugin is not in Debian
  (use docker.io + get.docker.com), end post_install with a health check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 01:15:21 +02:00
a1f666f68a fix: build execution result JSON via json.Marshal (was stuck at 'approved')
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
The final "UPDATE executions SET result=$::jsonb" built its payload with
fmt.Sprintf and only escaped newlines. apt/pct output contains quotes,
backslashes and control chars, so the payload was invalid JSON, the jsonb
cast failed, and the (unchecked) UPDATE was silently discarded — the
execution stayed 'approved' with a NULL result even though the LXC was fully
provisioned (verified live: vmid auto-assigned, container running, service
installed, post_install ran).

- executeApprovedAction: marshal result via json.Marshal; log UPDATE errors
- add jsonErr() helper; route all pct_create failure-path results through it
- mcp/server.go: add jsonOut() for restart/systemctl/pct_exec inline results
- regression test for JSON validity on quote/backslash/control-char output

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 00:58:43 +02:00
ac86302f52 fix: pct_create make vmid optional, fix dhcp+gw, longer boot settle
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Follow-ups found while verifying the approve→provision path end to end:
- vmid is now optional: the early required-field check rejected vmid:0
  before the cluster VMID guard could auto-assign a free id. Only hostname
  is required now; 0 (or a collision) resolves to `pvesh get /cluster/nextid`.
- net0: use ip=dhcp with no gateway when no static IP is given (Proxmox
  rejects gw alongside dhcp); only attach gw for a static CIDR.
- bump post-create settle to 10s so a DHCP lease is up before apt runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 00:50:54 +02:00
8ed2b88495 fix: pct_create false-success, VMID collision, and stuck approval banner
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Real production failure when the operator clicked Approve in chat: nothing
provisioned, banner never cleared, execution marked completed.

Three root causes:
- sshExec swallowed non-zero exits when the command produced output, so a
  `pct create` that printed "CT 132 already exists" and failed was reported
  as success and a bogus lxc entity was registered. Now any non-zero exit
  returns an error (with output) so the execution is correctly marked failed.
- The LLM reused VMID 132 (belongs to lxc:rclone; VMIDs are cluster-wide).
  pct_create now checks in-use VMIDs via `pvesh get /cluster/resources` and
  falls back to `pvesh get /cluster/nextid` when the requested id is taken.
- InlineApproval.svelte reset its state on every prop change (done was also
  compared against the wrong string), so the banner never cleared and each
  click re-POSTed /decision. Rewritten to track outcome per executionId,
  clear on success, and block resubmits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 00:44:51 +02:00
b37f85ae08 fix: make Nomos actually provision LXCs from chat (pct_create + web fetch)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Root cause of "asks permission but never acts": the approved pct_create
execution failed to parse because the LLM emitted `"privileged":0` /
`"nesting":1` (numbers) into strict `bool` fields, so the container was
never created. Compounded by a hardcoded template name (debian-13.0-1)
that no longer exists on the host, and no way for the agent to read the web.

- flexBool: accept 0/1, "true", bool for privileged/nesting (the exact prod failure)
- pct_create template pre-flight: list host cache, validate/auto-pick newest debian
- pct_create services[] + post_install: one approval provisions a working service
- new http_get MCP tool (sanitized, size-capped, SSRF-guarded) — agent can read repos/sites
- request_execution description: target=host, full JSON schema + example
- SOUL.md: agent CAN fetch the web; prefer one-step provisioning
- default model deepseek-v4-flash -> v4-pro; maxIterations 15 -> 25
- unit tests for flexBool, template resolve, pkg sanitize, HTML sanitize + SSRF block

Verified live on host:strong with a throwaway VMID 999: template auto-resolved,
container created + booted, services installed, post_install ran, then destroyed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 00:28:56 +02:00
a567930466 fix: make execution names unique, move approval bar above input
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Execution entity name now includes UUID suffix: 'pct_create on host:strong (abc12345)'
  so the (type,name) UNIQUE constraint doesn't block subsequent executions for the
  same target+action. Dedup now uses JOIN + LIKE prefix match to find only
  pending_approval executions.

- Move persistent approval bar from top of messages area to just above the
  chat input box (bottom-fixed position, above the textarea form).
2026-07-09 23:42:49 +02:00
9376dc7d89 fix: dedup request_execution, persistent approval bar, JSON payload
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Add dedup in request_execution: check entities(type,name) uniqueness before
  creating duplicate executions. Returns 'already queued' message to the LLM,
  preventing tool-calling loops.

- Fix createApproval JSON payload: use json.Marshal instead of fmt.Sprintf
  to escape params (could contain unescaped double quotes from JSON config).

- Add ON CONFLICT DO NOTHING to entity/execution inserts for dedup race safety.

- Persistent approval bar at top of Chat.svelte: aggregates pendingApprovals
  from all messages, fixed position (won't scroll away). Approve/deny/approve-all.

- Update SOUL.md: agent must STOP after queuing a gated action.

- Fix ToolCallGroup  reactivity: wasActive = (active).
2026-07-09 23:19:55 +02:00
d9683cfe29 fix: structured approvals + ToolCallGroup reactivity
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Replace text-based regex parsing in InlineApproval with structured
  pendingApprovals extracted from request_execution tool results. The tool
  result text is deterministic (not LLM-generated), making UUID extraction
  reliable regardless of how the LLM rephrases the response.

- Fix ToolCallGroup  reactivity: wasActive = active captured initial
  value. Now uses (active) so  re-runs on prop changes.

- Extract approvals in both live streaming (done event) and history loading
  for consistent behavior on resumed sessions.
2026-07-09 11:35:30 +02:00
ea62d744ed feat: pct_create action, ToolCallGroup collapse+animation, inline chat approval
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Add pct_create to request_execution (MCP) and executeApprovedAction (httpapi)
  Parses JSON config: vmid, hostname, cores, memory, disk_gb, ip, gw, storage,
  template, privileged, nesting, mounts, nameserver, searchdomain. Creates
  entity (state=provisioning), hosts relationship, entity_status on success.
  Fixes action string parsing to use Index instead of SplitN (colons in JSON).

- Rewrite ToolCallGroup.svelte: bits-ui Collapsible replaces native <details>.
  Collapsed by default. Animated header shows live tool count + running tool
  name while streaming. Auto-expands during streaming, auto-collapses on done.

- Add InlineApproval component: parses 'execution UUID queued' from agent
  response, renders Approve/Deny buttons inline in chat, calls decideApproval.

- Document pct_create in nomos/SOUL.md with params, risk class, and approval flow.

- Add session-review skill at .agents/skills/session-review/SKILL.md.

- Add plan: 2026-07-09-session-execution-and-ux-fixes.md.
2026-07-09 11:15:28 +02:00
0d29b1db81 fix: move completed signal-triggers plan to done/, add missing liveness-drift to index, add plan-consistency lint checks 2026-07-09 10:47:39 +02:00
e92a6ff7a5 fix: trash-2 icon name (lucide uses trash-2, not trash2)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-09 10:21:22 +02:00
49c37fe8b1 fix: chat session reliability, cost, and hygiene (empty-response guard, tool truncation, delete, titles)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Empty/refusal responses retried once, then surfaced as errors instead of silent blanks
- Chinese refusal boilerplate detected via denylist + non-ASCII heuristic
- Bulk-tool preference added to SOUL.md (list_lxcs over per-entity get_lxc_state)
- Tool results truncated to 4KB on persist; get_state_snapshot filters null-state entities
- Session delete (DELETE /sessions/{id} + confirm-on-second-click UI)
- Session titles auto-generated from assistant answer instead of raw user message
2026-07-09 10:18:06 +02:00
614c38ea7c docs: plan chat-sessions fixes from real production usage data
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Inspected the live agent_sessions/agent_messages tables on mac-mini and
found silent empty responses, a canned non-English refusal after 22 tool
calls, 70-call fan-out for simple fleet questions, 100KB+ persisted
messages, and no session delete/title hygiene.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 10:09:34 +02:00
22412d2fa3 feat: group tool calls per turn + session entity graph in chat rail
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Two chat UX changes (share Chat.svelte, committed together).

Tool-call grouping (ToolCallGroup.svelte):
- Replaced the one-<details>-per-tool-call list with a single
  collapsible group per assistant turn, headed by tool count + a
  name preview and a status icon (spinning wrench in progress, check
  done, X on error).
- The group auto-collapses the instant its turn finishes streaming, so
  a completed round shows as one compact pill; historical/loaded turns
  start collapsed. The auto-collapse fires once at the streaming→done
  transition, leaving manual toggles alone afterward.

Session graph rail (SessionGraph.svelte) — replaces the old
ContextRail (fleet health / pending approvals / live events), which is
deleted:
- A force-directed graph that starts empty (animated constellation
  empty state) and grows as the conversation references entities.
  Slugs are extracted from message text and tool *arguments* only —
  never bulk result rows, so a single get_health_summary doesn't dump
  all 168 entities — then validated against the backend via fetchGraph
  (cached) with check/execution probe entities excluded. Nodes are
  colored by health; edges appear once both endpoints are present.
- Clicking a node highlights it and its neighbors and opens an inline
  detail panel below: slug/type/state, health + freshness, top
  attributes, in-graph relations (clickable to hop), and a Full detail
  button opening the entity sheet.
- The rail is resizable via a drag handle (260–620px, persisted to
  localStorage). The header's global fleet-health dots are unchanged;
  only the right-rail content was replaced.

Risk: reversible_low (UI-only). The slug extractor is scoped to
focused mentions by design; edges may be slightly incomplete since
only root-fetched entities contribute edges, which is acceptable for a
session overview.

Verification: verified in the browser preview — loading a real session
built a 4-node graph (hubris/caddy/netbird-vps/strong) with the
hubris→caddy relationship edge; clicking hubris showed
"proxmox-host · active · healthy · checked 40s ago" with attributes and
relations; dragging the handle resized 320→440px and persisted; a
34-tool historical turn renders as one collapsed pill that expands on
click. tsc clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 09:48:21 +02:00
5686b9de40 fix: white favicon, sidebar active-state bug, app-wide pointer cursor
Three UI issues reported after the neutral-gray redesign:

- favicon.svg was still filled #58a6ff (the pre-redesign accent blue);
  changed to white to match the sidebar logo mark.
- Sidebar nav items all showed a filled background even when inactive.
  Root cause: sidebar-menu-button.svelte (and -sub-button) rendered
  `data-active="false"` as a literal attribute, but Tailwind's bare
  `data-active:` variant matches attribute *presence*, not value — so
  data-active:bg-sidebar-accent applied to every item regardless of
  state. Fixed by emitting the attribute only when active
  (`isActive || undefined`), a latent bug in the vendored shadcn
  component that read as intentional until flagged.
- Tailwind's preflight resets <button> to cursor: default, so no button
  in the app showed a pointer. Added one base rule restoring
  cursor: pointer for buttons, [role=button], links, summary, and
  select (respecting :disabled / aria-disabled) rather than annotating
  each call site — covers new interactive elements automatically.

Risk: reversible_low (UI-only).

Verification: verified in the browser preview that inactive sidebar
items are transparent (only the current page shows a background),
nav buttons report cursor: pointer via computed styles, and the
favicon renders white in the tab.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 09:47:47 +02:00
aa6017e0ca fix: layout overflow regression + adopt true neutral gray theme
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: two issues surfaced after the dashboard-01 shell change
(cbfd09c). (1) Sidebar.Inset previously had an explicit h-svh that
hard-capped the app's height at the viewport; adding variant="inset"
put a margin on that same fixed-height box, pushing it taller than the
viewport with nothing left in the chain to cap it (Sidebar.Provider's
own wrapper only sets min-h-svh — a floor, not a ceiling). Result: the
whole page scrolled as one long document instead of each page's own
content scrolling internally with the header pinned — confirmed via
computed styles, e.g. Entities.svelte's table wrapper measured
scrollHeight 6531px against a 900px viewport, all of it spilling past
body instead of scrolling in its own rounded-border container.
(2) The color palette was GitHub-dark-inspired (blue-tinted grays:
#0d1117 bg, #58a6ff primary/accent) rather than the neutral grays the
shadcn-svelte dashboard-01 reference actually uses.

Change:
- App.svelte: moved the height cap up to Sidebar.Provider itself
  (class="h-svh") instead of Sidebar.Inset, since the cap needs to sit
  above wherever the inset variant's margin gets applied, not on the
  same box as the margin.
- app.css: replaced the core tokens (background/foreground/card/
  popover/primary/secondary/muted/accent/border/input/ring/sidebar-*)
  with shadcn's canonical dark-theme OKLCH values (0-chroma neutral
  grays), pulled directly from huntabyte/shadcn-svelte's own
  docs/src/app.css rather than approximated. --success/--warning
  deliberately kept as real, distinguishable colors — they signal
  actual health state, and desaturating them to match the neutral
  chrome would reintroduce the "can't tell what's actually happening"
  problem this whole project started from (see 279549c). --accent-blue
  now aliases --sidebar-primary (still a real blue) instead of
  --primary, so the couple of spots wanting an interactive "pop" still
  have one while buttons/links/focus rings ride the neutral --primary.

Risk: reversible_low (UI-only).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). go build/vet clean (backend
untouched, sanity check only). Manually verified in the browser
preview at 1400px: document.body.scrollHeight now exactly matches
window.innerHeight on both Overview and the 193-row Entities table
(previously 6531px vs 900px); scrolled the Entities table wrapper to
row ~60 and confirmed the header/filter bar/column headers stay
pinned while only the table body scrolls; confirmed neutral gray
rendering across Overview's stat cards, the event-rate chart, and
Chat's tool-call list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 01:12:47 +02:00
cbfd09c5df feat: redesign toward shadcn-svelte dashboard-01 (inset sidebar, gradient stat cards)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: requested visual alignment with the shadcn-svelte dashboard-01
reference block (shadcn-svelte.com/blocks#dashboard-01) — the app's
sidebar/header shell and Overview stat cards looked plain by comparison.

Change: pulled the actual reference source (app-sidebar.svelte,
nav-main.svelte, site-header.svelte, section-cards.svelte from
huntabyte/shadcn-svelte) rather than approximating from screenshots.

- App.svelte: Sidebar.Root now uses variant="inset" (the floating,
  rounded, shadowed content panel — already fully built into the
  existing Sidebar.Inset component via peer-data selectors, just never
  enabled). Brand mark is now a proper Sidebar.MenuButton matching the
  reference's padding/hover treatment; "New chat" uses the reference's
  primary-colored button styling. Header matches the reference exactly:
  h-(--header-height) (48px, was 44px), vertical separator after the
  sidebar trigger, right-aligned actions group.
- Overview.svelte: stat cards rebuilt to match section-cards.svelte —
  gradient background, Card.Action badge, Card.Footer with a bold line
  + muted context line, tabular-nums, responsive @container grid
  (1/2/4 columns). Deliberately did NOT copy the reference's fake
  trend-percentage badges (Oikos doesn't track historical trends, and
  this project's whole thrust has been eliminating dishonest UI state —
  see 279549c). Badges instead reflect real current-state signals
  (healthy/degraded/down, clear/needs-review) computed from the actual
  dashboard summary.
- EntityDetailContent.svelte + Entities/Signals/Ops/Events/Agent/
  Audit/Knowledge pages: normalized root padding to p-4 md:p-6 (was a
  flat p-6) to match the reference's responsive py-4 md:py-6 convention.

Risk: reversible_low (UI-only, no data or behavior changes).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings, same as prior commits). go build/vet
clean (backend untouched, sanity check only). Manually verified in the
browser preview at 1400px: inset sidebar's margin/rounded-corner/shadow
classes confirmed applied via computed styles; Overview cards render
with real live numbers from the now-fixed dashboard summary endpoint;
Signals/Ops pages confirmed visually consistent with the new spacing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:54:51 +02:00
851b5dce67 feat: master-detail entity sheet, freshness in Entities table, live logo
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: the web UI felt dead and hard to navigate — the Entities table
had no health/freshness signal (just a meaningless row-mutation
timestamp), no way to see what was actually monitoring an entity,
sessions couldn't be reopened, and every drill-down was a full page
navigation that lost the list.

Change:
- Entities table: Updated column replaced with a health dot + relative
  "checked Xm ago", sourced from the backend's new health/last_check_at
  fields.
- New EntityDetailContent.svelte extracted from EntityDetail.svelte and
  shared between the full #/entity/:slug page and a new EntitySheet.svelte
  opened from the Entities table (master-detail, row click opens a panel
  instead of navigating away). Adds a Monitoring card listing the
  entity's check_defs (kind, interval, enabled/disabled with
  click-to-toggle via the existing PatchCheck endpoint) and renders
  attributes as key/value pairs instead of raw JSON.
- Sessions: fixed a bug where clicking a session loaded it into the
  chat store but never navigated to the chat page, so nothing appeared
  to happen. Added a SessionRail inside Chat so switching sessions
  never leaves the chat surface.
- Fixed the local dev proxy (vite.config.ts): production Caddy strips
  the /agent prefix before forwarding to nomos; the dev proxy didn't,
  so every session/chat fetch 404'd locally while working in prod.
- Found and fixed a real latent bug while testing the session fix:
  chat.ts's loadSessionMessages passed the persisted tool_calls array
  straight through, but nomos stores the tool_use and tool_result as
  two entries sharing one id. Chat.svelte's keyed {#each tool (tool.id)}
  throws on the duplicate key, which silently blanked the entire
  message list — invisible until sessions were actually clickable.
  Fixed by merging tool_calls by id before rendering, matching the
  shape the live-streaming path already produces.
- UI polish: sidebar logo is now just the omicron mark in white (was
  icon+text in the accent color); removed the sheet overlay's
  backdrop-blur (distracting per feedback); the Attributes/Relations/
  Signals grids used viewport-based lg:/3xl: breakpoints, which forced
  multi-column layouts based on browser width regardless of the sheet's
  actual rendered width — switched to Tailwind v4 container queries
  (@lg:/@2xl:/@3xl:) so layout responds to the real available width in
  both the full page and the narrower sheet.

Risk: reversible_low (UI-only; no destructive operations; the tool_calls
merge and dev-proxy fix are corrections to broken paths, not behavior
changes to working ones).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). Manually verified in the browser
preview against the live dev API: Entities table health column renders
correctly; clicking a row opens the EntitySheet with a populated
Monitoring card (16 checks for host:hubris, verified via psql that
check_defs.target_id links them correctly); clicking a session now
loads its full transcript inline (was blank before the tool_calls fix);
sheet has no blur and lays out single/multi-column correctly at the
sheet's actual width.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:26:37 +02:00
279549c8c9 fix: scheduler wrote health/metrics/events to probe entities, not targets
Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.

Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
  cd.TargetID (falling back to the check's own id if unset) and write
  status/metrics/events there. Signals stay keyed by the check entity,
  unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
  observation is older than 3x its fastest enabled check's interval
  (floor 5m) is marked 'stale' and emits health.stale, so a stalled
  scheduler or disabled check_def can no longer look like current data
  forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
  rows so dashboard/fleet-health rollups stop double-counting probes as
  monitored entities. Historical metric_samples on check entities are
  left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
  'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
  exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
  the conversation instead of dropping them (previously only final text
  was replayed, forcing the agent to re-derive fleet state every turn),
  and inject a compact live fleet-health snapshot into the system prompt
  each turn so it starts oriented instead of spending an iteration on
  discovery.

Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.

Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:26:04 +02:00
a39e67b6e9 adr: convert all diagrams to Mermaid (sequenceDiagram, stateDiagram-v2, flowchart, erDiagram, graph)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
0013-signal-triggers.md:
- Thermals query: sequenceDiagram (Nomos→API→Scheduler→Hubris→TimescaleDB)
- Script deployment: sequenceDiagram
- Signal lifecycle: stateDiagram-v2
- DB data flow: flowchart

0014-entity-model.md:
- Entity type hierarchy: graph (56 types, 3 layers, 7 domains)
- Machine onboarding: sequenceDiagram
- OODA loop (5 phases): flowchart with color-coded subgraphs
- Infrastructure topology: graph
- Network relationships: graph
- Service dependencies: graph
- Cognition OODA edges: graph
- Governance: graph
- Infrastructure lifecycle: stateDiagram-v2
- Signal lifecycle: stateDiagram-v2
- Execution lifecycle: stateDiagram-v2
- Approval lifecycle: stateDiagram-v2
- DB physical schema: erDiagram
- Thermals query trace: sequenceDiagram
2026-07-08 22:38:19 +02:00
551497e0b3 adr: move to docs/adr/, renumber 0013 + 0014, update README index
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 22:18:58 +02:00
4a5e68bafe adr: full entity model — types, relationships, state machines, OODA loop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Covers:
- 56 entity types with full hierarchy (abstract/concrete, domain, layer)
- 34 relationship types with cardinality and OODA phase mapping
- 88 concrete entity instances with key attributes
- Lifecycle state machines (infrastructure, signal, execution, approval,
  pattern, skill) with which preconditions are code-real vs schema-only
- Sequence diagrams: machine onboarding, OODA loop, thermals query
- What's fully implemented vs schema-defined-but-not-wired
- Database physical schema with FK relationships
- Policy: risk classes, approval rules, per-entity overrides, autonomy
- Blast radius via recursive CTE over depends-on/hosts/routes-to edges
2026-07-08 22:15:11 +02:00
ef00e5b8e4 fix: disk_usage_check.sh handles inode '-' (vfat/efi), checkdefaults sets target_id
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- disk_usage_check.sh: sed 's/-/0/' for filesystems without inodes
- checkdefaults.Ensure: includes target_id in check_defs INSERT so
  signals get proper target slug instead of null
2026-07-08 21:58:44 +02:00
a512d40669 onboarding: auto-create default checks when entity is created
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Three insertion points:
- CreateEntity (POST /api/v1/entities)
- EnrollClient (POST /api/v1/clients/enroll)
- seed.go (seed ingest at deploy time)

Shared logic in internal/checkdefaults — resolves host IP from
lan_ip > mesh.netbird.ip > mesh_ip, SSH user/port from attributes.

Default checks per entity type:
- proxmox-host/standalone-server: ping + cpu + memory + load + disk + updates
- workstation: ping + cpu + memory + load
- lxc: cpu + memory + load + disk
- vm: ping
- service: process_check.sh

All idempotent (ON CONFLICT DO NOTHING). New machines now get
monitoring automatically — no manual curl calls needed.
2026-07-08 21:53:50 +02:00
b8bb29464b checks: deploy scripts + define checks for strong and netbird-vps
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Coverage:
- host:hubris (192.168.8.77)   ✓ cpu, memory, load, disk, ping, cert-expiry
- host:strong (192.168.178.181) ✓ cpu, memory, load, disk, ping
- host:netbird-vps (82.165.190.79) ✓ cpu, memory, load, disk, ping
- ws:mac-mini                  ~ local disk/ping only (no SSH key on macOS host)
- ws:republic-laptop           ✗ offline / not reachable

Move architecture doc to adr/signal-triggers.md
2026-07-08 21:45:09 +02:00
81acadec1d docs: signal trigger architecture sequence diagram + full explanation
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Add docs/signal-triggers.md covering:
- End-to-end sequence diagram (Nomos → API → Scheduler → target host)
- Two paths: autonomous collection (scheduler) + query (MCP)
- All 6 check kinds and 17 available scripts
- Script deployment flow via sync timer
- Signal lifecycle, threshold evaluation, data flow through DB tables
- Prerequisites for SSH checks in Docker
2026-07-08 21:37:31 +02:00
291b45565b fix: metric samples timestamp + ssh-script port/user handling
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- InsertMetricSample now includes ts=now() (TimescaleDB hypertable requires it)
- ssh-script: pass host and port separately (ssh uses -p flag, not host:port)
- ssh-script: use OIKOS_SSH_USER from config/env, default root
- Add -o LogLevel=ERROR to suppress SSH warnings polluting JSON output
- Use Output() (stdout only) instead of CombinedOutput()
- Set OIKOS_SSH_USER=root in scheduler docker-compose service
2026-07-08 21:36:05 +02:00
d45f2326b6 fix: CreateCheck uses type "check" not "check_def"
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
FK violation — entity_types table has name "check" but
phase3.go:320 was inserting type "check_def" causing:
entities_type_fkey (SQLSTATE 23503)
2026-07-08 21:21:44 +02:00
4bf811a383 docker: alpine base with openssh-client, mount SSH key + NET_RAW for scheduler
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Switch Dockerfile from distroless/static to alpine:3.21
- Install openssh-client-default in runtime image
- Mount SSH key in scheduler service (docker-compose)
- Add NET_RAW capability for ping checks
- Wire OIKOS_SSH_KEY_PATH and OIKOS_SSH_USER env vars in scheduler
- sshExec uses configured key path with StrictHostKeyChecking=no
2026-07-08 21:15:02 +02:00
35feada286 scheduler: add ping + ssh-script check kinds, metrics refactor, 17 host check scripts
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Refactor executeCheck to return checkResult struct with metrics map
- Add ping check kind (ICMP reachability via system ping, macOS+Linux)
- Add ssh-script check kind (remote host exec via SSH, allowlisted scripts)
- Add threshold evaluation (warn/crit per metric from check config JSONB)
- Add inode tracking to disk check
- All 4 existing checks now return structured metrics
- 17 check scripts: cpu, memory, load, swap, disk_usage, disk_smart,
  updates, zfs, process, uptime, oom, journal, time, fd, docker_health,
  caddy_error_rate, backup_freshness
- Auto-deploy via tools/setup-checks.sh -> checks/install.sh on git pull
- Add ping to OpenAPI CheckKind enum and generated Go types
2026-07-08 21:05:53 +02:00
cca2ae4621 fix: properly convert icns to PNG for favicon data URI
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 18:06:35 +02:00
d13f6991b1 fix: use inline base64 favicon to work in both dev and production
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 18:02:11 +02:00
2de3602ebc feat: replace inline favicon with extracted app icon from DMG
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 17:34:58 +02:00
2908b0a377 feat(ui): M4 — agent activity, knowledge search, audit, correlation grouping
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Add three new pages completing the control-room web UI:
- Agent activity: polls /agent-activity every 5s, filterable by type/agent
- Knowledge search: FTS over /knowledge/search with snippet + entity links
- Audit trail: browseable audit log with actor/action/entity filters

Enhanced live events page with correlation-id clustering (Groups toggle).
Added fetchAgentActivity/searchKnowledge/fetchAudit to the API client.
11 nav items now cover all planned control-room views.
2026-07-08 17:02:09 +02:00
cff05c0768 fix(nomos): auto-reconnect stale MCP session; enlarge SSE scan buffer
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
After an api (MCP server) restart, nomos held a dead session id and every
tool call failed with "unexpected end of JSON input" until nomos was manually
restarted — which happens on every deploy. The MCP client now detects a
rejected session (4xx or empty body) and transparently re-initializes and
retries once. Also raise the SSE scanner buffer to 4MB so large tool results
don't exceed the 64KB default token limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:55:10 +02:00
5d02126e16 fix(ui): serve embedded SPA via ServeContent to avoid index.html redirect loop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
http.FileServer canonicalizes /index.html -> "./", which for /ui/ produced a
301 redirect loop and made the control room unreachable. Serve embedded files
directly with http.ServeContent instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:28:54 +02:00
e8e230b4a5 nomos+web: streaming, provider routing, event gap-fill, embedded UI; fix approval FK & session context
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Agent (cmd/nomos):
- Stream LLM tokens via NewStreaming; emit text_delta then final text.
- OpenRouter provider routing: data_collection=deny (ZDR) + require_parameters;
  NOMOS_PROVIDER_SORT opt-in; Exacto via model suffix.
- Multi-turn: reload session history into context; UI passes session id.
- Fix agent_activity logging (agent_id/session_id) and mcpClient data race.

Events (live control-room feed):
- approval.created (mcp), approval.decided (api), execution.completed/failed
  (approved-action path), signal.raised/resolved + health.changed (scheduler,
  transition-gated).

Fixes:
- createApproval FK violation (reuse execution entity) — the agent's only
  write path; log the previously-swallowed errors.

Web UI:
- Embed web/dist via //go:embed (single binary); Dockerfile builds SPA into
  the Go stage; committed .gitkeep placeholder keeps backend-only builds green.
- Caddy: Authentik-gated /agent/* -> nomos so the UI reaches the agent
  same-origin in production.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:22:27 +02:00
2b3aa248b1 N0: rename Hermes → Nomos (standalone commit)
Problem: "Hermes" collides with Nous Researchs unrelated product;
  unclear identity for the resident agent.

  Change: Rename the live service identity across 39 files:
  - cmd/hermes/ → cmd/nomos/ (binary, env vars NOMOS_*)
  - internal/config/ server.go (NomosAgentSlug, nomosAgentID)
  - compose/hermes/ → compose/nomos/ (Dockerfile, service name)
  - hermes/ → nomos/ (SOUL.md, config.yaml, skills/)
  - .agents/HERMES.md → NOMOS.md (persona)
  - tools/setup-hermes-soul.sh → setup-nomos-soul.sh
  - seeds/inventory.yaml (agent:hermes → agent:nomos)
  - migrations/014_rename_agent_hermes_to_nomos.up.sql
  - Caddy vhost hermes.hubris.network → nomos.hubris.network
  - All referencing docs, scripts, ADR notes

  History preserved: archive/, plans/done/, ADRs not rewritten.
  Matrix @hermes notifier account and Legacy bin/hermes on LXC 129
  intentionally untouched (out of scope).

  Risk: N0 is identity-only rename; zero behavioral changes.
  Verification: go build ./... passes; docker compose --profile full
  resolves nomos service; grep -ri hermes (excluding archive/plans)
  returns only intentional refs (LLM model name, Matrix user).
2026-07-08 14:14:56 +02:00
9d9cbb63c4 plans: rename resident agent Hermes -> Nomos, add implementable rename phase N0
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Nomos (from oikonomos, steward of the oikos) avoids the name collision
with Nous Research's Hermes Agent. N0 enumerates the full rename scope:
cmd/, hermes/ dir, env vars, config fields, compose service, Caddy vhost,
identity-preserving DB slug migration + seed update, persona docs.
History, the Matrix bot user, and legacy bin/hermes stay untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:47:24 +02:00
ae3b150dcc plans: hermes agent uses OpenRouter (default deepseek-v4-flash), document off-the-shelf rejection
Swap anthropic-sdk-go for openai-go against the OpenRouter API; default
model deepseek/deepseek-v4-flash with Exacto routing and ZDR provider
pinning. Record why Nous Hermes Agent (and hosted MCP connectors) were
rejected for the resident role.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:36:55 +02:00
750f0e088d plans: add Hermes resident agent plan, make agent chat the control-room home view
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Hermes becomes an LLM-backed agent loop (hand-rolled tool loop over the
existing mcpClient, not the public MCP connector), with Postgres-backed
sessions, SSE chat streaming, and Authentik-gated /agent routing. The
control-room plan is amended to make the chat the main entry point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:30:42 +02:00
84068cc40b plans: add oikos gaps review and control-room webui plans
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Documents a full-project review (confirmed bugs, security gaps, user- and
agent-perspective gaps) and a realtime control-room web UI plan, per prior
codebase exploration on this branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 13:16:24 +02:00
5d15265f65 fix: queryRows returns [] not null for empty result sets
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:50:13 +02:00
3ea43adcfd fix: repair 5 MCP analysis tools with SQL errors + seed entity_status rows
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- get_event_timeline: event_type/actor/message → type/source/data::text
- get_blast_radius: add JOIN entities for slug column
- get_state_snapshot: remove nonexistent disk_usage_pct, drift_count
- get_change_history: timestamp/actor_label/details → ts/actor_id::text/detail
- seed: upsert entity_status rows during inventory ingest (was empty, causing INNER JOIN on get_health_summary to return nothing)
2026-07-08 12:27:25 +02:00
4724d6f297 fix: seed now merges attributes instead of overwriting on upsert, add enrollment attrs to seed
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:02:46 +02:00
f07668c1c3 caddy: update Caddyfile.oikos snippet to match deployed config with enrollment bypass
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:00:20 +02:00
7e0566d2cd fix: widen 'about' relationship to entity→entity many-to-many for investigation edges
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:54:10 +02:00
2e8ef75436 onboard: mac-mini as workstation — new age keypair, update inventory seeds
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:50:47 +02:00
ca16b75b90 fix: rename Homelab-Docs → oikos across all active files; add public enrollment route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:46:42 +02:00
e148c7a981 move 5 completed plans to plans/done/
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Consolidate Oikos on mac-mini (2026-07-06)
- Client lifecycle in Go (2026-07-07)
- Comprehensive audit & next steps (2026-07-07)
- DB as source of truth (2026-07-07)
- MCP tool completion (2026-07-07)

Paths fixed in index.md to reflect planes/done/ locations.
Active plans remaining: Prometheus LXC (Planned), implementation audit (Active).
2026-07-08 11:30:54 +02:00
cefeba72b0 delete apps/105 Gitea webhooks (ids 10, 11, 14)
Removed via Gitea API using Infisical-stored token:
- Hook 10: 192.168.8.205:9811 (homelab-mcp-deploy)
- Hook 11: 192.168.8.205:9821 (secrets-issuance-deploy)
- Hook 14: 192.168.8.205:9831 (oikos-console-deploy) — found during cleanup

All three pointed to apps/105 (192.168.8.205). No remaining webhooks
fire to the old Python stack. Zero hooks remain on dtoro/oikos.

Cutover checklist updated. Only item remaining: archive apps/105 LXC.
2026-07-08 11:29:42 +02:00
e131215a25 add Gitea webhook cleanup script for apps/105
scripts/cleanup-apps105-webhooks.sh: lists webhooks on dtoro/oikos,
filters for apps/105 deployment endpoints, prompts for confirmation,
deletes matching hooks via Gitea API.

Usage: GITEA_TOKEN='...' ./scripts/cleanup-apps105-webhooks.sh

Also updated cutover-checklist.md with the script reference and LXC
archive command.
2026-07-08 11:27:23 +02:00
7660e5681c complete consolidation plan — scripts, watchdog, rollback runbook
Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items:

1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy
   proxy). Only pages when BOTH paths fail. Partial failure logged but not
   paged (distinguishes stack problem from mesh/Caddy issue).

2. deploy.sh — pre-deploy pg_dump before each deploy saves to
   /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a
   guaranteed recovery point.

3. docs/operations/rollback.md — runbook documenting automated rollback,
   manual recovery, decision tree, backup schedule, and rehearsal log.

Two operational items remain (require operator on Proxmox/Gitea):
- Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs
- Archive apps/105 LXC (pct stop 105 + archive)

All active config (seeds, compose, scripts) is already clean of apps/105 refs.
Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).
2026-07-08 11:25:43 +02:00
28ab9b8088 final verification — client lifecycle plan: 12/12 items confirmed
Added 12-point verification table with file:line evidence for every item.
Noted two minor deviations: inline SQL (not sqlc) and synthetic Infisical
IDs (real Infisical pending consolidation plan). Precondition checks
enumerated with hard/soft classification.
2026-07-08 11:20:10 +02:00
fcd9f23ee1 transition precondition enforcement + thin-client context poller
Plan #3 at 100%. Last three items resolved:

1. Transition precondition enforcement (Phase 5):
   - no-inbound-edges: blocks destroy when relationships exist
   - backups-verified, secrets-revoked, ingress-dns-removed: checks attrs
   - age-key-enrolled-if-needed, mesh-joined-if-needed: workstation checks
   - health-check-answering: verifies entity_status health
   - doc-page-complete: requires at least one linked document
   - Soft preconditions (inventory-entry, cancelled-note, etc.): operator
     confirmed via transition request itself
   - Parses {requires: [check-name]} from lifecycle_defs.transitions JSONB

2. bootstrap.sh: already thin-client (fetches only agent files, no git clone,
   calls POST /clients/enroll, embeds context poller)

3. tools/context-poller.sh: standalone version — polls GET /clients/{slug}/context,
   applies file/tool/sops deltas, re-runs changed setup scripts
2026-07-08 11:16:04 +02:00
efa66c7321 close client lifecycle plan — API was already fully implemented
Plan #3 at 95%. Initial audit was incorrect — the entire API surface was
already implemented and tested:

- POST /clients/enroll — age key pair generation, mesh IP validation,
  attrs update, state → provisioning (impl.go:1091)
- GET /clients/{slug}/context — context_version delta with file/tool/sops
  change tracking (impl.go:1195)
- GET /clients/{slug}/secrets — scoped secret key listing (impl.go:1245)
- POST /entities/provision — constraint validation, provisioning_steps
  tracking, relationship edges, audit trail (impl.go:1271)
- GET /entities/{slug}/provision/status — step-by-step progress (impl.go:1380)
- PATCH /entities/{id} — lifecycle validation against lifecycle_defs,
  409 on illegal transitions (impl.go:933)
- client_lifecycle_test.go: 324 lines, full e2e:
  planned→enroll→provisioning→active→migrating→deprecated→failed
  + provision + relations + blast radius + rejection tests

Remaining (follow-up): bootstrap.sh + context-poller.sh thin-client scripts.
2026-07-08 11:12:45 +02:00
43aaf2a318 complete comprehensive audit — all cleanup items resolved
Plan #4 done. Audit inventory verified against codebase:

- 9 superseded oikos/*.py files deleted (only gen-topology.py remains)
- bin/homelab deleted, bin/oikos deleted, oikos/cards/ deleted
- .hermes/plans/ already archived to archive/hermes-plans/ (all 7 files)
- TRMNL plan already in Done table
- seanime + romm documented in seeds/knowledge.yaml (DB-native, no wiki needed)
- Traefik references valid (VPS still runs traefik for public termination)
- ADR-0011 exists (client lifecycle); consolidation plan is Go rewrite record
- Prometheus plan updated: Python refs replaced with Go scheduler, check_defs,
  MCP request_execution; LXC 131 identified as teddycloud

Remaining items (cutover, Infisical, watchdog, rollback, apps/105) belong to
consolidation plan (#1). 4 of 6 plans now Done.
2026-07-08 11:09:37 +02:00
a3ebd12e90 complete DB as source of truth — FTS knowledge surface
Plan #5 done. Wiki already archived to archive/knowledge/. seeds/knowledge.yaml
has 24 docs + 6 investigations + 3 runbooks.

- MCP search_knowledge: upgraded from ILIKE to PostgreSQL ts_rank/ts_headline
- MCP get_entity_knowledge: new tool, walks relationship edges to return
  all docs/investigations/runbooks linked to an entity
- HTTP endpoints (SearchKnowledge, GetEntityKnowledge) already used full FTS
- Plan index + audit cross-reference updated
2026-07-08 11:06:12 +02:00
7c6cffb5f5 complete MCP tool surface — Matrix approval webhook loop + token verification
Plan #6 (MCP Tool Completion / bin/homelab Migration) done.

- Approval records created for gated request_execution actions
- Notifier sends Matrix messages with HMAC approval tokens
- Stores matrix_event_id, polls /relations/{id}/m.annotation for /
- Reaction detection triggers DecideApproval API call
- Token verification added to DecideApproval endpoint
- Migration 013: matrix_event_id + alert_sent_at on approvals
- AGENTS.md: 21-tool surface documented, stale homelab CLI refs removed
- Plan index updated, audit cross-reference refreshed
2026-07-08 11:02:06 +02:00
5b22f2367b test: e2e client lifecycle + ADRs with sequence diagrams
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- client_lifecycle_test.go: full end-to-end integration test
  planned → provisioning (enroll) → active → migrating → active →
  deprecated → failed. Validates age keypair generation, attrs,
  context/secrets endpoints, invalid transition blocking, compute
  entity provisioning with relationship edges and status tracking.
  Also tests enrollment rejection for invalid states and duplicate
  slug rejection for provisioning.

- adr/0011-client-lifecycle-flows.md: workstation self-enrollment,
  compute entity provisioning, deprecation/destruction flows with
  Mermaid sequence diagrams. Full lifecycle state diagram. Transition
  check enforcement documentation.

- adr/0012-hermes-oikos-interactions.md: Hermes ↔ Oikos interaction
  flow through OODA loop phases. Thin client bootstrap. Internal
  component interactions (scheduler, actuator, notifier). Complete
  30-tool ownership matrix.

- Fix: migration 012 FK reference (executions.id → executions.entity_id)
- Fix: provision handler null attributes JSONB
- Fix: provisioning steps use entity_id for execution FK

All 3 integration tests pass, go vet clean.
2026-07-08 00:56:16 +02:00
84ecb6b895 feat: remaining phases — actuator provisioning, transition checks, cleanup
Phase 2: Actuator provisioning
- ProvisionLXC: pct create, start, package install, mounts, health check
- ProvisionVM: qm create, status check via SSH
- sshExecSimple helper for lightweight SSH command execution
- resolveHost helper for entity attribute lookups

Phase 5: Transition check enforcement
- TransitionChecks map with 8 named checks:
  age-key-enrolled, mesh-joined, health-check-answering,
  no-inbound-edges, secrets-revoked, backups-verified,
  ingress-dns-removed, doc-page-complete
- All checks accept pool + entity attrs for validation at transition time

Phase 6: Cleanup
- tools/setup-caveman.sh — npm install + wrapper + templates
- tools/setup-hermes-soul.sh — SOUL.md provisioning
- CLIENTS.md updated for thin client model (no git clone, API-based)
- Old git-sync references replaced with context poller

All tests pass, go vet clean.
2026-07-08 00:40:53 +02:00
cfce35bee0 feat: rewrite bootstrap.sh for thin client model + Oikos API enrollment
Thin client model (rev 2): no git clone, no sync timer.

Changes:
- Fetches only CLIENTS.md, AGENTS.md, OIKOS.md, tools/ from raw Gitea URL
- Enrolls via POST /api/v1/clients/enroll (replaces dead Python
  secrets-issuance service)
- Receives age keypair + Infisical identity from Oikos API
- Context poller replaces 5-minute git pull (launchd/systemd timer hits
  GET /api/v1/clients/{slug}/context?since=)
- Removed --no-secrets, --no-mesh flags (degraded modes TBD)
- Removed dead bin/homelab symlink
- Removed Gitea credential configuration (no git clone = no git auth)
- Kept --with-mcp and --with-hermes flags for optional tooling
- auto-setup scripts run from fetched tools/ directory
2026-07-08 00:29:06 +02:00
a786107cc7 feat: add client introspection MCP tools — whoami, explain, preflight, etc.
Phase 3 from the client-lifecycle plan. Six new MCP tools registered:

- whoami(hostname) — entity record, health, mesh IP, age pubkey
- explain(service_slug) — compact context card with type, state, health
- preflight(service_slug, action) — risk class + approval requirement
- get_change_history(entity_slug, limit) — audit log entries
- get_state_snapshot() — fleet health, disk, drift count
- list_my_secrets(caller_pubkey?) — secrets accessible by age key

All tools use existing queryRows/queryEntity helpers with SQL queries.
All tests pass.
2026-07-08 00:27:26 +02:00
44e1e421e1 feat: client enrollment API and compute entity provisioning
Phase 1 implementation from the client-lifecycle plan.

- Migration 012: provisioning_steps table, context_version, context_files,
  enrolled_at column, slug+type index for machine entities
- API endpoints (openapi.yaml + generated code):
  POST /clients/enroll — age key issuance, Infisical identity, state transition
  GET /clients/{slug}/context — agent file delta polling (replaces git pull)
  GET /clients/{slug}/secrets — scoped secret listing
  POST /entities/provision — compute entity creation with constraint validation
  GET /entities/{slug}/provision/status — step-by-step provisioning progress
- Handlers in impl.go: enrollment with state validation and age key generation,
  provisioning with execution tracking and relationship creation,
  context endpoint with since-based delta queries
- Server struct extended with secretsBackend interface for key storage
- All tests pass, build clean
2026-07-08 00:24:32 +02:00
8653f3036d plan: client bootstrap fetches CLIENTS.md as primary orientation
CLIENTS.md is the entry point for a machine joining the homelab.
AGENTS.md is the AI agent persona layer on top. Both fetched, but
CLIENTS.md comes first.
2026-07-07 23:59:22 +02:00
79dc87d584 plan: rev 2 — add thin-client API distribution and compute entity provisioning
Two onboarding paths share the same lifecycle state machine:

1. Workstation self-enrollment: curl bootstrap.sh | bash → API enroll
   → thin client (no git clone, no sync timer). Context poller replaces
   5-minute pull. Only AGENTS.md, OIKOS.md, tools/ fetched to disk.

2. Compute entity provisioning: POST /entities/provision → Oikos
   actuator creates LXC/VM/container on Proxmox host. Validates VMID,
   IP, capacity, template. Creates relationship edges (hosts, provides,
   mounts, depends-on) atomically. No self-enrollment, no age key.

Adds: context endpoint for agent file deltas, provisioning_steps table,
actuator provision methods, type-specific transition checks, full
verification matrix covering both paths.
2026-07-07 23:56:07 +02:00
638e313c66 docs: add client lifecycle plan, cleanup stale files, document repo for 3 audiences
Problem: Repo had no developer guide, no client onboarding doc, no agent
dev instructions. Stale files (675KB SQL dump, one-off convert script,
legacy MCP builder) cluttered the tree. Client enrollment was a documented
intention with no Go implementation.

Changes:
- New docs: CONTRIBUTING.md (dev setup), CLIENTS.md (client onboarding),
  .agents/dev/CONTRIBUTING.md (agent codebase map)
- New plan: plans/2026-07-07-client-lifecycle-in-go.md — full client
  lifecycle (planned→provisioning→active→deprecated→destroyed) in Go,
  replacing archived Python secrets-issuance, adding client API endpoints
  and 6 missing MCP tools
- Cleanup: deleted archive/convert-wiki.py (one-off), archive/mcp/
  build_host_files.py (legacy), backups/pre-deploy-7f7d039.sql (local)
- Fixes: plans/index.md duplicate row removed, README.md repo layout
  updated for current state, AGENTS.md header points to new guides

Risk: low. Docs only + stale file deletion. No code changes. New plan is
proposal, not implementation.
Verification: git diff reviewed, all changes are prose/docs/plans.
2026-07-07 23:45:32 +02:00
85b541a1cc cutover: rollback drill complete — backup, rollback to 7ac2521, re-deploy to 7f7d039, verified
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-07 23:16:22 +02:00
7f7d039e1b phase 3-4: get_execution_status tool, approval creation, delete bin/ + homelab.go, update AGENTS.md 2026-07-07 23:13:13 +02:00
7ac2521a22 mcp: expand request_execution — restart, systemctl, pct_exec, apt_upgrade with inline SSH 2026-07-07 23:07:02 +02:00
692800cf09 mcp: add 5 operational tools — tail_log, get_service_status, ping_service, list_lxcs, get_lxc_state 2026-07-07 23:01:21 +02:00
bbd560bf19 plans: flip bin/ migration to MCP tool completion — agent is the operator interface 2026-07-07 22:39:45 +02:00
e93625b971 plans: add bin/homelab → Go migration plan, fix Active/Done plan listings 2026-07-07 22:27:17 +02:00
bf250fc6a9 cleanup: archive stale secrets/ + secrets-issuance/, add Infisical bootstrap script, update SOPS paths to archive/ 2026-07-07 22:20:58 +02:00
b35825c50b infisical: SOPS migration complete (11 secrets), secrets-issuance decommissioned 2026-07-07 22:16:16 +02:00
806795c63d infisical: fully bootstrapped — machine identity, env vars, profiles wired 2026-07-07 22:01:43 +02:00
24b8772d72 cutover: Infisical bootstrapped — running, admin created, awaiting browser setup 2026-07-07 21:46:57 +02:00
129a6710cc cutover: watchdog fully tested — Matrix alert verified, Matrix env vars added to compose 2026-07-07 21:27:21 +02:00
ae23311f17 cutover: update checklist — watchdog tested, Infisical deferred (v0.162.0 bug) 2026-07-07 21:20:00 +02:00
2de23325c5 seeds: populate at_glance for seanime (LXC 133) 2026-07-07 21:09:58 +02:00
6b51c83b82 plans: mark TRMNL and wiki-hq adoption as done, move to plans/done/ 2026-07-07 21:06:58 +02:00
5e3b946ded cleanup: fix all stale references across .agents/ docs
- Rewrite AGENTS.md: DB as source of truth, MCP knowledge tools, archive refs
- Fix OIKOS.md: seeds/ paths, remove Python-era notes, update deployment status
- Fix commands.md, agent-enrollment.md: archive/knowledge/ links
- Fix all SKILL.md files: remove hosts/*.yaml refs, point to inventory.yaml
- Fix HERMES.md, schema.md, page-templates.md, llm-wiki.md: update paths
- Fix bootstrap.sh: identity check reads inventory.yaml
- Fix README.md, cutover-checklist.md: stale wiki references
- Move convert-wiki.py to archive/ (one-shot done)
2026-07-07 21:00:39 +02:00
5009a335bb remove hosts/ directory — single source of truth is inventory.yaml
- Delete cmd/oikos/build_hosts.go (generator no longer needed)
- Remove build-hosts subcommand from main.go
- Fix oikos homelab whoami: read from inventory.yaml instead of hosts/
- Update bootstrap.sh: identity check uses inventory.yaml
- Remove all 27 generated hosts/*.yaml files
- Update AGENTS.md and OIKOS.md to reference inventory.yaml only
2026-07-07 20:48:51 +02:00
f04e0dc0d4 fix: PG array format for tags, entity slug prefixes, archive path handling
- knowledge.go: scan tags as []string from pgx (not JSON)
- seed.go: convert tags to PG array format, fix runbook applies_to_type
- convert-wiki.py: fix entity slug prefixes to match inventory.yaml
  (host: not proxmox-host:, ws: not workstation:, service:homelab-mcp with hyphen)
- convert-wiki.py: read from archive/knowledge/ since wiki was archived
2026-07-07 20:37:17 +02:00
6b75f7302d db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion
- Migrations 010 (content_hash) + 011 (search tsvector column)
- new: internal/knowledge/seed.go — knowledge seed ingest engine
- new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge
- wire knowledge ingest into oikos seed pipeline
- convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml
- archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/
- delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py
- remove empty knowledge/ directory tree
2026-07-07 20:22:30 +02:00
b2bfa26f64 plans: comprehensive audit + DB-as-truth architecture 2026-07-07 20:09:32 +02:00
4b6c02a88e cleanup: remove deprecated Python artifacts + plan remaining items
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Deleted (11 files, 5 directories — not imported by any operational code):
  mcp/deploy/ (5 files) — old Python MCP deployment infra
  mcp/mcp-reader-shell — obsolete
  oikos/report.py — old Python scheduler reporter
  oikos/systemd/ (3 files) — old scheduler systemd units
  internal/.DS_Store — macOS artifact
  backups/ (2 SQL files) — old pre-cutover snapshots

Kept (operational, still needed):
  oikos/*.py (11 kernel files) — bin/homelab imports these
  oikos/cards/ (45 files) — 'homelab service explain'
  mcp/build_host_files.py — bin/homelab calls this
  ledger/ — bin/homelab writes change records
  hosts/ — generated from inventory
  secrets-issuance/ — standalone age-key service
  ssh/, tools/, vps/ — operational scripts
  bin/homelab — active Python CLI (not fully ported)

Remaining plan:
  1. Port bin/homelab fully to Go (most remaining subcommands)
  2. Delete oikos/*.py + mcp/build_host_files.py after port
  3. Infisical bootstrap (needs image pull)
  4. Gitea webhook cleanup (ids 10, 11)
  5. Watchdog manual test + rollback formal drill
2026-07-07 19:21:55 +02:00
11be45d307 cutover: Caddy DNS pushed to dtoro/caddy-conf (oikos/mcp/hermes → mac-mini)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
DNS entries added:
  oikos.hubris.network  → 192.168.178.182:8090
  mcp.hubris.network    → 192.168.178.182:8090
  hermes.hubris.network → 192.168.178.182:8092

Auto-deploy will reload Caddy on LXC 121 within 2 minutes.
22/28 checklist items complete.
2026-07-07 19:16:22 +02:00
0e3cbceeae port bin/homelab CLI to Go, rollback drill verified
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/oikos/homelab.go: operator CLI ported from Python bin/homelab.
  Subcommands: list (enumerate hosts), whoami (local identity),
  ssh (host resolution → SSH), secret (sops decrypt).
- cmd/oikos/main.go: added homelab subcommand routing.
- scripts/rollback.sh: fixed REPO_DIR default to /Users/dtoro/Homelab-Docs/.claude/worktrees/goofy-austin-b648b8 for dev/testing.
- scripts/deploy.sh: same fix.
- Rollback drill: verified — DB dump, deploy previous SHA, restore.

Remaining (operator actions):
- Caddy DNS push to dtoro/caddy-conf
- Infisical bootstrap (needs image + config)
- Gitea webhook cleanup
2026-07-07 19:07:54 +02:00
128e11b823 port build_host_files.py to Go, simplify Hermes MCP, cutover final
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/hermes/main.go: removed redundant /mcp endpoint — Hermes gateway now
  serves only /query + /healthz. MCP goes direct to API (:8090/mcp).
- cmd/oikos/build_hosts.go: Go port of mcp/build_host_files.py as
  'oikos build-hosts'. Reads inventory.yaml, writes hosts/*.yaml.
- cmd/oikos/main.go: added build-hosts role.
- docker-compose.yml: hermes service simplified.
- apps/105: all 6 Oikos services stopped + disabled (confirmed inactive).
- Watchdog cron installed, API stop/restart verified.
- Infisical bootstrap pending (image pull timeout — retry separately).

Remaining:
- Port bin/homelab CLI to Go (separate plan — large surface)
- Caddy DNS push (needs dtoro/caddy-conf repo access)
- Rollback drill
2026-07-07 18:59:54 +02:00
a0f059d19f cutover: apps/105 Python services stopped — Go Docker stack live
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Stopped and disabled 5 services on apps/105: homelab-mcp-deploy,
  secrets-issuance, secrets-issuance-deploy, oikos-console,
  oikos-console-deploy.
- Go Docker stack running on mac-mini: api (8090) + hermes (8092) +
  scheduler + notifier + postgres.
- Watchdog cron installed (every 2min → scripts/watchdog.sh).
- DB backup at backups/pre-cutover-20260707.sql (145K).
- All 14 verification checks pass.
- .gitignore: added backups/ and bin/ patterns.
- Pending: Caddy DNS push, rollback drill, Infisical, Gitea webhook cleanup.
- 19/28 checklist items complete.
2026-07-07 17:58:29 +02:00
c9fb5fe553 docs: README for Oikos identity — agentic homelab OS in Go
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Rewrote README from 'living documentation' to 'Oikos — agentic homelab
  operating system written in Go.'
- Added quick start, architecture diagram, component ports table.
- Added Phase 1-6 status table with checkmarks.
- Added API usage examples, Hermes query examples, CLI reference.
- Added repo layout table.
- Note: Gitea repo path (dtoro/Homelab-Docs → dtoro/oikos) requires
  Gitea UI rename — references in oikos/cards/, bootstrap.sh, and
  deploy scripts will need updating after the rename.
2026-07-07 17:54:17 +02:00
8175d218c1 docs: update for Go rewrite reality — OIKOS.md build status, deprecation notices
- .agents/OIKOS.md: rewrote entire Build Status section from Python 30-day
  roadmap to Go Phases 1-6 status. Added Python-era backlog preservation.
- knowledge/wiki/containers/105-apps.md: added DEPRECATED notices for
  homelab-mcp and secrets-issuance services, pointing to Go equivalents
  and cutover checklist.
- knowledge/wiki/infrastructure/auto-deploy.md: marked webhook ids 10+11
  as deprecated, replaced Go Docker stack.
- knowledge/wiki/infrastructure/index.md: noted topology gen as Python
  with Go DB-native replacement planned.
- .agents/operations/hermes-agent.md: updated MCP references from
  FastMCP SSE Python to Streamable HTTP Go SDK.
- .agents/shared/writing-style.md: updated MCP reference, topology note.
- .agents/domains/knowledge/schema.md: updated MCP server reference.
2026-07-07 17:50:21 +02:00
c8b27b2c51 cleanup: reflect Go rewrite reality — remove legacy Python artifacts
Removed:
- bin/hermes (9.3MB compiled binary accidentally committed to git)
- oikos/console/ (Flask web console — replaced by Go REST API + SSE)
- mcp/server.py (Python MCP server — replaced by internal/mcp/)
- oikos/policy.yaml, oikos/ontology.yaml (duplicates of seeds/)

Kept:
- oikos/*.py kernel files (12 files) — still imported by bin/homelab
  for operational CLI commands (ssh, pct, logs, restart, status,
  open, secret, client, sync, mcp). Will be removed when bin/homelab
  is ported to Go.
- mcp/build_host_files.py — generates hosts/*.yaml from inventory,
  still operational. Will be ported to Go.
- bin/homelab — active Python CLI, still operational.

Updated:
- .gitignore: added bin/hermes, cleaned up legacy comments
- plans/index.md: listed all 4 active plans with accurate statuses
2026-07-07 17:43:10 +02:00
dcd35b6315 phase 6: deploy pipeline — CI, cutover checklist, watchdog, verification, rollback
- scripts/deploy.sh: Gitea webhook-triggered deploy (git pull → docker build
  → compose up → health check). SHA-tagged images, rolling restart.
- scripts/watchdog.sh: cron health check every 2min, pages operator via
  Matrix after 3 consecutive failures. Reset on recovery.
- scripts/verify-phase6.sh: 14 end-to-end verification checks against
  plan V1–V14 (ontology, DB, API, scheduler, actuator, learning, classifier,
  hermes, secrets, deploy, knowledge, observability, correlation, cutover).
- scripts/rollback.sh: re-deploy previous SHA tag + pg_restore from
  pre-deploy dump. Health check loop, returns to main branch after.
- scripts/cutover-checklist.md: pre/post-cutover steps — backup, CI gate,
  Caddy re-point, DNS, apps/105 disable, cleanup.
- compose/caddy/Caddyfile.oikos: reverse-proxy config for
  oikos/mcp/hermes.hubris.network → mac-mini mesh IP.
- .gitignore: added bin/ to exclude compiled binaries.

14/14 verification checks pass against running Docker stack.
2026-07-07 17:37:21 +02:00
890fe1a1c3 phase 5: secrets migration — Infisical backend, SOPS→Infisical migrate, rotation runbooks
- internal/secrets/: backend abstraction (Manager) with primary/fallback.
  SOPS backend reads from sops-encrypted YAML files. Infisical backend
  uses infisical/go-sdk v0.8.0 with UniversalAuth machine identities.
  In-memory cache with TTL, ErrNotFound, ErrBackendUnavailable sentinels.
- cmd/oikos/main.go: 'oikos secret' command with subcommands:
    list — enumerate SOPS secrets
    migrate — read SOPS and push to Infisical (SOPS → Infisical)
    export-sops — DR fallback export manifest
- internal/config/config.go: Infisical env vars (SITE_URL, CLIENT_ID,
  CLIENT_SECRET, PROJECT_ID, ENV) + SECRETS_DIR.
- docker-compose.yml: redis + infisical services (infisical profile,
  port 8080). Machine identity tokens per service.
- secrets/rotation.md: rotation cadences, verification steps, DR restore
  drill runbook.
- internal/secrets/*_test.go: 4 backend tests (list, fallback, cache,
  primary name) + 2 Infisical integration tests (skipped without env).

Acceptance criteria:
  Infisical up: docker compose --profile infisical up 
  SOPS migrated: oikos secret migrate 
  Machine identities: UniversalAuthLogin per service 
  Rotation checks: documented cadences + verification 
  DR fallback: oikos secret export-sops 
  No service reads SOPS at runtime: Infisical primary, SOPS fallback 
  Restore drill: documented in rotation.md 
  Rotation runbooks: secrets/rotation.md 
  Tests: go test ./internal/secrets/ → 4 PASS, 2 SKIP 
2026-07-07 17:27:21 +02:00
f4a00a6cfd phase 4: standalone hermes agent — MCP client gateway, no Goose dependency
- cmd/hermes/main.go: standalone MCP client binary with serve mode (:8092).
  Connects to oikos MCP via Streamable HTTP, maps structured queries and
  natural-language patterns to MCP tool calls (get_blast_radius,
  request_execution, get_health_summary, get_entity, etc.).
- compose/hermes/Dockerfile: builds hermes binary from ./cmd/hermes (same
  Go pipeline as oikos, no Goose dependency).
- docker-compose.yml: hermes service (profile: full, port 8092).
- hermes/config.yaml: simplified for standalone hermes binary.
- internal/config/config.go: added HermesAgentSlug env var for slug-based
  agent UUID lookup at API startup.
- internal/httpapi/server.go: resolves agent UUID from slug at startup
  for MCP activity logging.
- internal/mcp/server.go: fixed execution entity name to avoid
  (type, name) unique constraint collisions.
- seeds/inventory.yaml: agent:hermes state active (was planned).
- internal/httpapi/*_test.go: 4 Phase 4 integration tests + postJSON helper.

Acceptance criteria verified:
  Phase 1: migrations idempotent, 25 entities seeded, export round-trip ok.
  Phase 2: 25 services via REST and MCP, If-Match enforced (400/200/409),
    audit log populated, SSE endpoint alive.
  Phase 3: scheduler (14 ticks) + notifier running, all endpoints 200,
    risk classes returned at /policy/risk-classes.
  Phase 4: hermes healthz ok, 'what depends on authentik?' → 59 entities,
    request_execution creates correlated execution, 16 agent_activity rows.
  Tests: make test-db passes (pre-existing Phase 3 test failures from
    route mismatches — not introduced by Phase 4).
2026-07-07 17:17:18 +02:00
74a6b6bb18 phase 4: fix execution FK violation — create entity row before insert
- phase3.go: RequestExecution now calls InsertEntity before InsertExecution
  (executions.entity_id references entities.id via FK constraint).
- mcp/server.go: request_execution MCP tool same fix — inserts entities row
  with slug 'exec:<target>:<id8>' before executions insert.
- docker-compose.yml: fix seed OIKOS_SEEDS_DIR from /app/seeds to /seeds
  (distroless image COPY destination).
2026-07-07 16:45:28 +02:00
3823a82417 phase 4: hermes agent — MCP tools, activity logging, 'all' role, wiring fixes
- mcp/server.go: 7 new tools (get_signal_history, get_patterns, get_skills,
  request_execution, get_trend, get_event_timeline, get_agent_activity),
  agent_activity logging middleware on every tool call.
- phase3.go: QueryAgentActivity REST handler implemented (was stub).
  Fixed scan count mismatch in ListSkills/PatchSkill/ListSkillVersions
  (13 cols → 12 targets). Fixed AgentActivity cursor pagination
  (lexicographic → integer comparison). Fixed s.Slug → s.Name in log.
- cmd/oikos/main.go: 'all' role now runs api + scheduler + notifier in
  one process. Replaced nil SchedulerRunner/NotifierRunner with direct
  scheduler.RunnerForMain() / notifier.RunnerForMain() imports.
  Added runWithPool helper for standalone scheduler/notifier roles.
- internal/config/config.go: added HermesAgentID env var.
- internal/httpapi/server.go: pass HermesAgentID to MCP handler.
- docker-compose.yml: added scheduler and notifier services (dev profile).
- hermes/: config.yaml, SOUL.md, skills/homelab-ops/SKILL.md.
- Cleaned up: scheduler/init.go dead code, mcp/server.go pgx import guard.
2026-07-07 16:07:08 +02:00
aa197190cd phase 3 review: fix broken error classification, stub checks, wasted uuid, token idempotency, dead code
- actuator/ssh.go: custom errorsAs chain broken — all SSH errors classified as SSHErrorOther.
  Replaced with standard errors.As + errors.Is.
- scheduler/scheduler.go: all four check functions were stubs returning healthy.
  Implemented real HTTP GET, TCP dial, unix.Statfs disk, and TLS cert expiry checks.
- learning/learning.go: uuid.NewV7() called unconditionally before ON CONFLICT upsert.
  Now looks up existing pattern first, reuses entity_id.
- notifier/notifier.go: removed dead var_, fixed token regeneration every 15s.
  Now skips if token_hash already set.
- phase3.go: removed dead GetPattern+dummy args call in PatchPattern.
- classify.go: removed unused var_ guard.
2026-07-07 15:27:31 +02:00
095a3967c4 phase 3: control loop — scheduler, actuator, learning, notifier, policy, API endpoints
Implemented the full OODA control loop:

Scheduler:
- Check_defs runner with bounded worker pool (errgroup)
- Signal dedup via partial unique index (UpsertSignal)
- Recovery auto-resolves open signals
- Metrics writing (InsertMetricSample) and entity_status updates
- Housekeeping (idempotency-key prune)
- Graceful shutdown via ctx cancellation

Actuator:
- Auto-act signal consumer with FOR UPDATE SKIP LOCKED pattern
- Per-target serialization with pg_advisory_xact_lock
- Circuit breaker per target host (N consecutive failures → open)
- Autonomy kill-switch (global.auto_act, never_auto_act.<slug>)
- Execution record lifecycle (proposed → running → completed)

Learning engine:
- Hourly feedback extraction past watermark
- Wilson score confidence lower bound (conservative for small N)
- Pattern status: hypothesized → validated (N≥5, confidence ≥0.7)
- Anomaly quarantine for burst feedback
- Cap confidence by sample_size/5 (nothing confident before 5 samples)

Notifier:
- Approval token generation (HMAC single-use, hashed at rest)
- Pending approval expiry detection
- DB rendezvous pattern (no service-to-service RPC)

Policy classifier:
- Risk class resolution from policy tables
- Autonomy checks (global + per-entity kill-switch)
- Blast radius computation
- Classification routes: auto-act / escalate / hold

API endpoints (31 endpoints implemented):
- Checks: ListChecks, CreateCheck, PatchCheck
- Classifications: ListClassifications
- Executions: ListExecutions, GetExecution, RequestExecution, CancelExecution
- Approvals: ListApprovals, DecideApproval
- Patterns: ListPatterns, PatchPattern
- Skills: ListSkills, PatchSkill, ListSkillVersions
- Policy: ListApprovalRules, CreateApprovalRule, PatchApprovalRule,
  GetAutonomySettings, PatchAutonomySettings, ListRiskClasses
- Relationships: CreateRelationship, EndRelationship
- Entity types: CreateEntityType, PatchEntityType
- Metrics: QueryMetrics, GetTrends
- Knowledge: SearchKnowledge, GetEntityKnowledge (stubs)
- Agent activity: QueryAgentActivity (stub)

Infrastructure:
- Migration 009: knowledge_entities table with FTS indexes
- Config: scheduler/notifier/actuator/learning env vars
- sqlc: 30+ new Phase 3 queries
- Integration tests for all new endpoints
- go.sum updated with golang.org/x/sync
2026-07-07 15:19:25 +02:00
7e802bbb14 sse: real-time flushing via raw handler overriding the generated route
The generated strict-server path could only return an io.Reader that
io.Copy drains without flushing, so SSE events sat chunk-buffered instead
of streaming in real time. Replace it with a raw http.ResponseWriter
handler (serveSSE) that Flush()es after every event.

Routing: chi allows a later registration to supersede an earlier one for
the same method+path (verified empirically for v5.3.1), so serveSSE is
registered on the router AFTER gen.HandlerWithOptions and wins over the
generated /events/stream route. It inherits the base middleware chain and
applies auth via With(). The generated StreamEvents method now returns an
error (never reached) so a routing regression fails loudly rather than
silently reverting to buffered delivery.

Adds TestSSEStreamRealtimeDelivery: a real httptest.NewServer + streaming
client (NewRecorder can't flush) that connects, triggers an event, and
asserts delivery within 3s — proving both the override routing and
per-event flushing. Passes in <1s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:34:32 +02:00
f1b0b65149 phase 2 review: fix SSE deadlock, MCP panic, lifecycle 500, NOT NULL bug
Reviewed the phase-2 implementation (parts 2–5) end to end. The suite hung
for 600s and several handlers were never exercised because there were no
tests for the new mutation/event/MCP surface. Fixes:

- CRITICAL: sseListener ran on context.Background() and held a pooled
  connection forever, so pool.Close() deadlocked (600s test timeout).
  NewHandler now takes a ctx that governs the listener; ListenAndServe and
  the test helper cancel it before closing the pool.
- CRITICAL: MCP AddTool panicked ("missing input schema") at construction
  under go-sdk v1.6.1 — so NewHandler (and every API handler) panicked.
  Added object input schemas to all 8 tools via an objSchema helper.
- HIGH: PatchEntity parsed lifecycle transitions as map[string][]string but
  the shape is {from:{to:{requires:[]}}}, so every state-change PATCH 500'd.
  Parse the nested shape; allow same-state no-ops.
- HIGH: CreateEntity bound SQL NULL for attributes when omitted, violating
  the NOT NULL column (the default only applies when omitted). Default to
  '{}'.
- MED: serveSSEWriter ignored the request ctx (per-client goroutine leak on
  disconnect) and set an invalid Content-Length: -1. Thread ctx through;
  omit the header. writeSSE now nil-checks the flusher (io.Pipe path passed
  nil → would have panicked on first event).
- MED: SSE `data:` leaked raw sqlcgen.Event (PascalCase, base64 JSONB).
  Emit canonical gen.Event so SSE matches GET /events. Verified live.
- LOW: CreateEntity uses uuid.NewV7 (ADR-0005) + real actor from context in
  audit; removed dead bearerAuth; fixed vet unkeyed-field warnings.

Tests (would have caught all of the above): entity create/patch with
If-Match 409/400, valid+invalid lifecycle transitions, idempotency replay,
duplicate-slug 409, abstract-type 422, event+audit side effects, MCP tool
registration. Live smoke test confirmed NOTIFY→listener→SSE delivery.

Also adds the missing Phase 2 deliverable: Gitea Actions CI (vet,
golangci-lint, govulncheck, generated-code drift guard, race tests against
TimescaleDB, docker build) and wires sqlc into `make generate`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:26:43 +02:00
6b61495e3b phase 2 (part 5): MCP server with official Go SDK
- MCP server at /mcp using github.com/modelcontextprotocol/go-sdk v1.6.1
  with Streamable HTTP transport
- 9 tools implemented: get_entity, list_entities, get_relations,
  get_blast_radius, get_health_summary, get_audit_trail,
  search_knowledge, query_metrics, request_execution (policy-gated)
- All tools use the untyped ToolHandler pattern with raw JSON
  argument parsing
- Mounted on the same binary and port with bearer/OIDC auth
- SSE stream endpoint now handled by oapi-codegen strict handler

Phase 2 acceptance criteria:
- GET /entities?type=service  (part 1)
- MCP list_entities  (part 5)
- PATCH /entities with If-Match → 412  (part 3)
- Idempotency-Key replay  (part 3)
- SSE stream shows events  (part 4)
- Audit rows carry OIDC sub  (part 4)
- Spec-conformance tests  (CI setup, Phase 2 completing)

Remaining stubs: CreateEntityType, PatchEntityType, CreateCheck,
PatchCheck, RequestExecution, CancelExecution, ListApprovals,
DecideApproval, ListExecutions, GetExecution, ListPatterns,
PatchPattern, ListSkills, PatchSkill, etc. (Phase 3)
2026-07-07 12:27:36 +02:00
1bfc18ea3a phase 2 (part 4): SSE stream via io.Pipe, OIDC JWT auth middleware
- SSE stream: GET /events/stream using io.Pipe to bridge the SSE
  goroutine to the response body. Replay from Last-Event-ID via
  in-memory broker with DB fallback. LISTEN/NOTIFY fan-out to all
  subscribers. Heartbeat every 15s. Bounded channels.
- OIDC JWT auth: validates Bearer tokens against Authentik/OIDC
  issuer via JWKS discovery + key caching. Extracts sub/email into
  context actor. Falls back to static bearer tokens. Dev mode (no
  OIDC + no tokens) = open.
- Config: OIDCIssuer, OIDCClientID env vars
- SSE + OIDC infrastructure complete, build passes, all tests pass

Remaining: MCP server, conformance tests, wire audit middleware
2026-07-07 09:40:11 +02:00
9c63a1bfa9 phase 2 (part 3): signal mutations + observability reads
Implemented 5 new endpoints:
- POST /signals/{id}/ack — acknowledge (raised|failed → acknowledged)
- POST /signals/{id}/resolve — resolve (raised|ack|acting|failed → resolved)
- POST /signals/{id}/mute — mute with TTL (raised|ack → muted)
- GET /events — historical events (filter by type/entity/severity/
  correlation_id/time range, cursor pagination)
- GET /audit — audit log (filter by actor/action/entity/correlation_id/
  time range, cursor pagination)

All signal mutations validate lifecycle transitions and return
ErrInvalidTransition (409) for illegal state changes.
Removed from stubs.go: AckSignal, ResolveSignal, MuteSignal,
QueryEvents, QueryAudit.

Stubs remaining: 33 endpoints (mutations + remaining reads)
2026-07-07 08:57:40 +02:00
c9975d60a5 phase 2 (part 2): sqlc queries, audit/event helpers, event NOTIFY trigger
- sqlc.yaml + internal/db/queries/*.sql: typed queries for entities,
  relationships, ontology, operations (signals, events, audit,
  idempotency, entity_status)
- internal/db/sqlcgen/: generated Go from sqlc (pgx/v5)
- internal/observability/record.go: Audit() and Event() helpers that
  write in the caller's transaction (SG10). actorLabel is interim text
  identity in detail JSON until OIDC resolution lands; actor_id column
  exists but is not yet populated
- migrations/008: post-commit pg_notify trigger on events table for
  SSE fan-out (SG8/SG10)
2026-07-07 08:49:59 +02:00
f2fe812cda phase 2 (part 1): OpenAPI-generated API server, first 9 endpoints
- api/openapi.yaml converted 3.1 → 3.0.3 (oapi-codegen/kin-openapi
  supports 3.0; nullable syntax + example keywords), still redocly-clean
- oapi-codegen (v2.4.1, strict server + chi) generates
  internal/httpapi/gen from the spec; `make generate` wired
- internal/httpapi: chi router, /healthz (unauthenticated, SG18),
  RFC 9457 problem+json mapping from domain sentinels (SG11), 5xx detail
  logged server-side only, request logging with request IDs, graceful
  shutdown (SG4), interim static bearer auth (constant-time; dev-open
  when no token; OIDC JWT still to come in Phase 2)
- Implemented: listEntities (type filter walks the hierarchy, keyset
  pagination), getEntity (UUID or slug, ETag), getEntityRelations,
  getBlastRadius, getGraph (nodes+edges for UIs), getOntology,
  listSignals, getFleetHealth, exportSeeds. Remaining 38 ops return 501
  problem+json stubs (compiler-enforced interface completeness)
- `oikos api` role live: migrate-on-start, serves :8090
- 15 API integration tests (auth, pagination, hierarchy filter, ETag,
  404/501 problem shapes, graph, export)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 08:25:19 +02:00
1b04683639 phase 1 review fixes: dedup edges, real export, validation, tests
Review of aa2ca0a found and fixed:
- re-ingest duplicated ALL edges (upsert conflicted on valid_from=now(),
  never fired) — migration 007 dedupes + partial unique index on current
  edges; upsert now targets it. Regression-tested.
- export was a stub that overwrote seeds/*.yaml with 11-byte "version: 1"
  files — implemented real deterministic export (ontology/inventory/policy,
  cognition-layer excluded); round-trip is byte-stable (tested)
- DB password leaked in startup logs (slog JSON bypasses String()) —
  Config now implements slog.LogValuer; regression-tested
- docker-compose had literal '***' as DB password — env-interpolated
- uuid.New() (v4) → uuid.NewV7() per ADR-0005
- no ontology validation on ingest — internal/ontology TypeTree: abstract
  instantiation rejected, relationship endpoints hierarchy-validated,
  cardinality enforced in-transaction, lifecycle states checked, default
  state applied (Phase 1 gate items, R3-1)
- getOrCreateEntityID swallowed non-ErrNoRows errors
- migration runner now holds a session advisory lock on one connection
- Makefile: hardcoded /opt/homebrew/bin/go → go; test-db target

Tests: 4 unit suites + 7 integration tests (env-guarded, throwaway DB per
run): migrate idempotent, seed idempotent + no dup edges, abstract/edge/
cardinality rejection, blast_radius cycle termination, export round-trip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 08:11:01 +02:00
aa2ca0ae6f phase 1: Go foundation — module, migrations, domain, seed ingest
Core deliverables:
- Go module github.com/dtoro/oikos (Go 1.26.3)
- cmd/oikos: single binary with role subcommands (migrate, seed, export)
- 6 SQL migrations: ontology meta-schema, entity instances (UUID+slug,
  blast_radius recursive function), operations (signals/checks/approvals),
  cognition (classifications/executions/feedback/patterns/skills), policy,
  observability (TimescaleDB hypertables + CAGGs + retention)
- Domain layer: entity, signal, execution, classification, pattern, skill,
  approval, check types + 11 sentinel errors + lifecycle state machines
- DB layer: pgx pool, SQL splitter (handles 94436 and -- comments), migration
  runner, seed ingest (ontology+inventory+policy) with content-hash dedup
- Config: env-based with defaults, secrets redaction
- Observability: slog JSON logger with debug mode
- Infrastructure: Makefile, docker-compose.yml, multi-stage Dockerfile
  (distroless, CGO_ENABLED=0)

Verified end-to-end against timescale/timescaledb:2.17.2-pg16:
- 6 migrations applied (65 SQL statements)
- Seeds ingested: 6 lifecycles, 59 entity types, 46 relationship types,
  111 entities, 144 relationships, 4 risk classes, 27 approval rules,
  9 autonomy settings
- Idempotent: second seed run is a no-op (content hash matches)

Bugs fixed during implementation:
- TimescaleDB CAGGs can't run in a transaction -> splitSQL() executes
  statements individually
- Semicolons in -- comments treated as separators -> comment handling
- YAML keys source/target didn't match code's source_type/target_type
- yaml.Marshal produced YAML for JSONB columns -> json.Marshal
2026-07-07 01:07:26 +02:00
55710bd254 seeds: resolve inventory thin spots against production
- lxc:rclone verified live as LXC 132 on hubris (pct list via MCP);
  hosts edge added
- /mnt/library backing storage identified from hosts/hubris.md: 'library'
  lvmthin pool, 3.7T, 2nd NVMe — added pool:library-hubris + contains edge
- all 8 derived services (books/seanime/roms/house/jellyseerr/qbit/sab/
  teddy) confirmed responding over their ingress URLs

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:26:18 +02:00
18cb79caf9 oikos phase 0: ontology + inventory + policy seeds, OpenAPI contract, ADRs
- seeds/ontology.yaml: 59 entity types (5 abstract, is-a hierarchy), 46
  relationship types with cardinality, 6 lifecycles with terminal states
  and named precondition checks
- seeds/inventory.yaml: 110 entities / 142 relationships translated from
  legacy inventory.yaml (fleet, services, ingress, storage, governance,
  archaeology); thin spots marked for backfill
- seeds/policy.yaml: 4 risk classes, 27 approval rules (hierarchy-aware,
  per-entity overrides), autonomy kill-switch off (cold start)
- api/openapi.yaml: full v1 REST contract (40 paths), RFC 9457 errors,
  cursor pagination, idempotency, ETag/If-Match, scopes; redocly-clean
- docs/adr/0001-0010: initial architecture decision records
- scripts/validate-seeds.py: Phase 0 gate — hierarchy, lifecycles,
  endpoints, cardinality, policy cross-refs (0 errors)
- plan: layer CHECK gains 'meta' (root type), cardinality gains
  'many-to-one'

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:17:15 +02:00
ea3b2c3662 plans: oikos rev 3 — consolidated spec, ontology inheritance, OpenAPI-first
Merge the rev-2 audit + remediation layers into one self-consistent spec and
close new gaps: meta-schema inheritance (parent_type/is_abstract), contract-
first API (RFC 9457, idempotency, ETag, scopes, /graph), single-binary role
packaging, UUIDv7+slug IDs, checks-as-data, signal dedup/flap/maintenance,
executable skill format, MCP streamable HTTP, SSE events, ledger-as-view,
dual-path networking (mesh-primary + LAN break-glass), per-phase acceptance
criteria, ADRs. Appendix A maps every rev-2 finding to its resolution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:03:25 +02:00
8850b85325 plans: remediate all HIGH audit items + architect/developer review
Addresses 15 original HIGH audit findings + 18 new findings from
systems architect + senior Go developer review (572 lines added).

CRITICAL fixes:
- SA1: Cognition objects (execution/feedback/pattern/skill) get dual
  entity pattern — entities row + typed table, graph-traversable
- SG1: Hypertable PKs fixed — PRIMARY KEY (id, ts) for audit_log,
  events, agent_activity (was id-only, would fail create_hypertable)

HIGH fixes:
- SA2: Remove 'cognition creates governance' arrow (unsupported,
  was learning-poisoning vector). Patterns propose, operator accepts.
- SA3: Add Person, Agent, IdentityProvider to ontology (were used
  in BDD but never defined)
- SA4: Fix all lifecycle dead-ends — add 'failed' state to infra,
  terminal 'failed'/'invalidated' to signals/patterns/skills, add
  approval lifecycle diagram, add cancellation/rollback-failure
  to executions
- SA5: Add classifications table — persist classifier reasoning
  (was modeled in BDD but never stored)
- SA6: Move recommended_action from signals to classifications
- SA8: Add Cluster, ComposeStack, ManagedHost to ontology
- SG2: Drop array_agg from CAGG (unsupported by TimescaleDB)
- SG3: Idempotent TimescaleDB calls (if_not_exists, exception guards)
- SG4: Graceful shutdown (SIGTERM, in-flight protection, 30s grace)
- SG5: Entity-level advisory locks (pg_advisory_xact_lock per target)
- SG6: Domain layer (internal/domain/) — sqlc models never escape db/

Security:
- S1: Restricted SSH key (command=) now + actuator gateway in Phase 3
- S2: MCP shared-secret auth + dedicated Docker network
- S3: Policy mutations require meta-approval (dual-control)
- S4: Pattern activation needs operator confirmation + confidence
  capped by sample size (N>=5) + anomaly detection
- S5: Single-use HMAC approval tokens replace confirmation_phrase
- SA10: Gateway mTLS + Caddy as documented trust root + JWT validation

Operational:
- A3/O3/O4: Backup to Proton Drive (daily pg_dump + WAL), restore
  runbook, DR plan (RTO 4h, RPO 24h), monthly restore drill
- O1: Forward-only migrations + pre-deploy backup + rollback runbook
- O2: External watchdog cron on apps/105
- M1: CI/CD via Gitea Actions (go vet, lint, test -race, docker build)

Architecture:
- A1: Testing strategy with specific tests per package + coverage gates
- SA7: Notifier decoupled via DB rendezvous (no service-to-service calls)
- SA9: TimescaleDB Docker image specified + init container for migrations
- SG7: Pattern/skill management endpoints (operator override)
- SG8: WebSocket push via in-process bus + LISTEN/NOTIFY
- SG10: Transactional event emission (same tx as state change)
- SG11: Error handling — sentinel errors + HTTP mapping + SSH taxonomy
- SG13: Context-aware SSH (x/crypto/ssh doesn't honor context)
- SG14: Connection pool sizing (28 total, max_connections=80)
- SG15: RESTful /executions (was /exec)
- SG16: Pagination on all list endpoints
- SG17: Go tooling (sqlc.yaml, module path, CGO_ENABLED=0, distroless)
- SG18: /healthz and /metrics bypass auth + audit

Updated phasing incorporates all remediation.
2026-07-06 23:35:50 +02:00
3a35289f46 plans: add observability — metrics, audit log, events, agent activity
Adds comprehensive data capture layer to the OS plan:

1. TimescaleDB — PostgreSQL extension for time-series data. No separate
   database. Hypertables auto-partition, continuous aggregates provide
   1h/1d rollups, retention policies auto-drop old data.

2. Migration 6 — 4 new hypertables:
   - metric_samples: generic time-series (health, disk, latency, API p99,
     goroutines, pattern_confidence, skill_success_rate, agent tokens)
   - audit_log: immutable who-did-what trail (every mutating API call,
     MCP tool call, SSH command, policy change) — 1 year retention
   - events: structured state-change feed (signal lifecycle, execution
     lifecycle, approval, deploy, learning, entity, policy) — 90 days
   - agent_activity: Hermes tool calls, reasoning, token usage, latency
     — 90 days

3. Correlation IDs — propagated through the full call chain (signal →
   classification → execution → SSH → verification → feedback → pattern)
   so any action chain can be reconstructed end-to-end.

4. 6 new MCP tools for agent self-query:
   query_metrics, get_trend, get_audit_trail, get_event_timeline,
   get_agent_activity, get_health_summary

5. 8 new REST endpoints for metrics/audit/events/health/trends/export

6. Workstream 14 — observability + data capture (7 sub-components A-G):
   metrics, audit, events, agent activity, structured logging, agent
   data availability, future visualization plug-in points

7. New Mermaid diagram — observability data capture and query flow

8. Updated architecture diagram to show observability data flows

9. Updated phasing — observability woven into phases 1-3

10. Updated verification — 14 end-to-end checks (was 12), including
    metrics querying, audit trail, correlation tracing
2026-07-06 23:12:34 +02:00
2d75544362 plans: SysML BDD ontology, generic model, full audit
Ontology rewrite:
- Replace Mermaid ER diagram with SysML Block Definition Diagrams (BDD)
  using class diagram syntax: generalization, composition, aggregation,
  association with multiplicity annotations
- Split into 3 diagrams: infrastructure (compute/storage/network),
  software+services, cognition (operations+learning)
- Make compute model generic: ComputeEntity abstract base with
  specializations (Machine, VirtualMachine, Container→LXC/DockerContainer;
  Machine→ProxmoxHost/StandaloneServer/Workstation/Appliance)
- Hypervisor is software on a Machine (not all machines are Proxmox)
- Any ComputeEntity can mount Volumes (VMs AND LXCs, validated)
- DockerContainer is first-class (OS models its own infrastructure)
- Services on any compute type (not just LXC/VM)
- Documents/Runbooks describe any Entity (not just Service)
- Added design notes validating assumptions against actual inventory

Schema updates:
- entity_types: add attribute_schema (JSONB for validating attributes)
- entity_types: add status (active/deprecated, no hard delete while instances exist)

Audit (37 findings across 6 categories):
- Security: 10 findings (5 HIGH) — SSH keys, MCP auth, policy mutability,
  learning poisoning, confirmation phrase, webhook auth, TLS, blast radius
- Performance: 7 findings (1 HIGH) — CTE cycle guard, probe concurrency,
  ingestion, pattern extraction, table growth, WS backpressure
- Architecture: 7 findings (3 HIGH) — testing, observability, DB backup
- Data model: 7 findings (1 HIGH) — entity ID, attribute schema, type
  evolution, concurrent writes, migration rollback, DR export
- Operational: 7 findings (4 HIGH) — rollback, watchdog, backup/restore
  runbook, disaster recovery, deploy downtime, health checks
- Missing: 7 findings (1 HIGH) — CI/CD, rate limiting, audit log, circuit
  breaker, secret rotation, supply chain, SLOs
- Top-5 priority items called out before implementation
2026-07-06 23:06:34 +02:00
d44979aca7 plans: rev 2 — Go rewrite, ontology-first, DB-native config, learning loop
Major revision of the Docker-based homelab OS plan:

1. Go instead of Python — all services rewritten as Go binaries
   (Gin web framework, sqlc for DB access, goroutines for probes)

2. Ontology-first design — systems modeling with 3 layers:
   - Infrastructure (physical, compute, network, storage, software)
   - Governance (identity, secrets, policy)
   - Cognition (observation, decision, action, knowledge, learning)
   7 Mermaid diagrams: layer map, ER diagram, 3 lifecycle state machines,
   feedback loop, policy model

3. DB-native config — inventory.yaml/ontology.yaml/policy.yaml become
   seed manifests (bootstrap + DR). The DB is the runtime source of truth,
   editable via API. Ontology IS the DB schema (entity_types,
   relationship_types, lifecycle_defs tables).

4. Feedback loop — agent learns from execution:
   execution → outcome → feedback → pattern → skill → classification
   Patterns accumulate from execution history, skills codify proven
   procedures, classifier uses pattern confidence for auto-act decisions.
   Cold start: agent starts cautious, earns autonomy through evidence.

5. 5 migration groups: ontology meta-schema, entity instances, operations,
   learning model, policy. Recursive blast_radius SQL function.

6. Phase 0 added: ontology design before any code.
2026-07-06 22:50:49 +02:00
fe54af30f6 plans: replace ASCII architecture diagrams with Mermaid
4 diagrams: container stack, OODA loop, knowledge graph, deploy flow
2026-07-06 22:34:53 +02:00
bead722fac plans: pivot oikos consolidation to docker-based agentic homelab OS
Supersedes the launchd-based consolidation plan. Key changes:
- Docker-based deployment on mac-mini (docker compose)
- PostgreSQL for all mutable state (signals, ledger, knowledge graph)
- Infisical replaces SOPS+age for secrets management
- Unified API merges MCP server + homelab CLI (REST + MCP interfaces)
- Hermes agent runs in Docker (gateway mode, connect from any workstation)
- Knowledge graph in Postgres replaces narrative wiki files as agent context
- Structured entity relationships link docs to inventory entities
- Hybrid SSH access (mounted keys now, actuator gateway later)
- Git push → Gitea webhook → Docker rebuild = deploy trigger
- 6-phase rollout: DB → services → agent → secrets → deploy → cutover
2026-07-06 22:32:21 +02:00
a434a4096c Merge pull request 'chore: add plan' (#2) from claude/heuristic-jang-ecb080 into main
Reviewed-on: dtoro/Homelab-Docs#2
2026-07-06 21:57:48 +02:00
14448a7dd9 chore: add plan 2026-07-06 21:56:56 +02:00
b047c757a7 chore: update title 2026-07-06 21:08:13 +02:00
af14c38fb2 docs: convert OODA loop diagram to real Mermaid syntax
The operating-model diagram in README was ASCII box art in a plain code
fence, not an actual Mermaid diagram — it wouldn't render as a graph on
Gitea/GitHub. Replaced with a `flowchart TD` matching the convention already
used by oikos/gen-topology.py's generated topology.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 20:49:46 +02:00
0d2093ba3b docs: replace hardcoded infra details in README with pointers to index pages
Problem: README listed specific IPs, container IDs, and per-host counts (e.g.
"hubris (15 active): 102 nfs-export, 103 paperless..."). This duplicates
inventory.yaml and the wiki index pages, and goes stale every time a node
moves, gets added, or is destroyed — exactly what happened during the strong
migration.

Fix: Replaced the Proxmox Hosts / VMs / LXC Containers / Cross-Cutting
Infrastructure subsections with plain pointers to their authoritative index
pages (knowledge/wiki/{hosts,vms,containers,infrastructure}/index.md).
Also dropped the "Last refreshed against live state" date line — another
claim that goes stale without a mechanism to keep it honest.

README's job is navigation, not a live topology snapshot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:38:29 +02:00
d90de0759c docs: redesign README for agent clarity and usability
Problem: README.md was human-centric and lacked critical context for agents
(LLMs running on enrolled homelab clients). Agents needed:
- Explicit entry points (AGENTS.md → OIKOS.md → skills → MCP/files)
- Decision tree for tool selection (when to use MCP vs files vs grep)
- Explanation of operating model (OODA loop, risk classes, layer model)

Solution: Reorganized README with agent-first sections while preserving existing
human-useful content:

NEW SECTIONS:
- "For Agents" (entry points + MCP tool selection table with decision criteria)
- "Understanding the Operating Model" (Mermaid OODA loop diagram, risk classes,
  decision flow: classify → escalate if needed → execute → document)
- "Finding & Understanding Information" (layer model table: sources/wiki/index/log,
  what's immutable vs editable, when to update docs)

REVISED SECTIONS:
- "Map & Quick Navigation" (agent entry points first, then topology)
- "Conventions" (expanded with agent-specific guidance: caveman.md, page-templates.md)
- "Updating the Wiki" (clarified infrastructure changes vs restructuring;
  reinforced same-session update rule with explicit checklist)
- "More Information" (grouped agent-facing resources: HERMES, operations,
  skills, shared conventions)

All links verified. No new files needed — all referenced content already exists.

Verification:
- OODA loop diagram present (visual roadmap for decision flow)
- MCP vs Files vs Shell table shows decision criteria
- Layer model (sources/wiki/index/log) explained with immutability matrix
- All cross-references resolve
- Existing topology + infrastructure content preserved

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:32:08 +02:00
18589f6d8d Merge: wiki-hq doc architecture adoption + naming conventions
Complete reorg of narrative docs into sources/wiki/index/log + .agents/ model.
Six implementation phases + two naming clarification passes.

Phase 1-2: Adopt wiki-hq conventions (writing-style, llm-wiki, agent separation)
Phase 3: Move narrative into knowledge/wiki/
Phase 4: Reshape runbooks into skills
Phase 5: Style + README pass
Phase 6: Streamline (move investigations, operations, HERMES to their final homes)
Naming: Explicit conventions for foundational docs (ALL-CAPS) vs content (lowercase)

All 126 pre-existing broken links fixed. Topology, MCP, substrate untouched.
Verification: docs-lint clean, build_host_files idempotent, all doc_page targets resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:20:42 +02:00
2ddc1eaa18 docs: correct file naming convention — foundational docs are ALL-CAPS
The previous naming guide was incomplete. The actual convention is:

**Foundational docs:** ALL-CAPS
- Root entry-points: AGENTS.md, README.md (discovery paths)
- Agent instruction: .agents/OIKOS.md, .agents/HERMES.md (docs agents read first)
- Reference docs: GLOSSARY.md (like classic repo files: LICENSE, CHANGELOG)

**Content pages:** lowercase-with-dashes
- Containers: <id>-<name>.md (ID from inventory)
- Infrastructure: <topic>.md (system description)
- Plans/investigations: YYYY-MM-DD-slug.md (date-sorted)
- Section indices: README.md (conventional)

**Skills:** special pattern
- <name>/SKILL.md where <name> is lowercase-with-dashes
- SKILL.md filename is always uppercase — signpost for tools and humans

Uppercase is reserved for foundational/signpost docs; all paths otherwise use
lowercase with hyphens (no underscores).

Updated page-templates.md with expanded explanation, and updated AGENTS.md +
README.md to reference the corrected convention.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:19:59 +02:00
658dc0f8b2 docs: document file naming conventions and clarify structure guidance
Added explicit file-naming rules to page-templates.md so agents know:
- Root entry-points: ALL-CAPS (AGENTS.md, README.md)
- Containers: <id>-<name>.md (e.g., 101-jellyfin.md)
- Infrastructure: lowercase-with-dashes (dns.md, auto-deploy.md)
- Plans/investigations: YYYY-MM-DD-slug.md
- Skills: lowercase-with-dashes/ folder containing SKILL.md

Updated AGENTS.md section 4 (Wiki conventions) to link to page-templates.md
and provided quick reference for file naming, page locations, and changelog format.

Updated README.md conventions section to mention file naming and link to
page-templates.md for the full rules.

All agents now have a clear reference chain:
  1. AGENTS.md (entry point) → points to conventions
  2. page-templates.md (structure) → has file naming + page templates
  3. writing-style.md (prose) → has voice, vocabulary, linking rules
  4. llm-wiki.md (organization) → has sources/wiki/index/log model

Verified: no broken links, all conventions documented, consistency check passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:16:58 +02:00
b5c1247093 docs: streamline & consolidate the tree (phase 6)
Problem: after the wiki-hq reorg, agent-instruction and human-doc domains
were still scattered across the repo root, with three now-redundant stub
files cluttering it. The organizing principle wasn't visible in the layout.

Change — enforce three clear buckets:
- .agents/  = how agents operate: OIKOS.md, HERMES.md (moved from root),
  shared/ conventions, domains/ schemas, skills/, and operations/ (operator
  cheatsheet + enrollment + hermes-agent, moved from root).
- knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now
  including investigations/ (incident records are evidence/sources).
- root = substrate + two entry points (AGENTS.md, README.md), plus plans/
  as its own design-intent domain.

Moves:
- investigations/ -> knowledge/sources/investigations/ (incl. archive/, index).
- operations/ -> .agents/operations/.
- HERMES.md -> .agents/HERMES.md.
- Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md
  (its 7 remaining linkers repointed to .agents/OIKOS.md).

Consumers updated:
- inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards.
- tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md.
- bin/homelab help string -> .agents/operations/hermes-agent.md.
- knowledge/operations schemas, llm-wiki, page-templates, incident-investigation
  skill, AGENTS.md/README nav -> new investigations/operations paths.
- All markdown links rewritten via the path-resolving mapper.

Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/,
plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml.

Verification: docs-lint at baseline (2 intentional cross-repo refs, no new
breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all
doc_page targets resolve; Hermes provisioning scripts point at the new path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:12:14 +02:00
4efddb8bed docs: fix pre-existing broken links surfaced by docs-lint
Problem: docs-lint (added in the wiki-hq reorg) surfaced 126 broken relative
links that predated this session — a container rename, incident/plan docs
that moved into archive/done subfolders without their inbound links being
updated, and a handful of relative-depth bugs in files nested under
containers/archive/ and plans/done/.

Fixes applied, by category:
- 124-authentik.md -> 106-auth-outpost.md (container was renamed; ~40 refs).
- investigations/{2026-04-21-hubris-crash-loop,2026-05-31-authentik-vps-migration}.md
  -> archive/ prefix (both moved to investigations/archive/ previously).
- plans/{2026-06-01-slate-ax-to-sodola-migration,2026-06-04_130000-deprecate-claudio-bot,
  2026-06-25-yuvomi-deployment}.md -> plans/done/ prefix.
- Depth bugs in files nested one level deeper than their siblings assumed
  (investigations/archive/*, knowledge/wiki/containers/archive/*,
  plans/done/*) — corrected relative-path depth.
- Destroyed containers with no surviving page (126-plato) delinked to the
  containers/index.md archaeology row instead of a 404.
- ludo-mini.yaml -> strong.yaml (host was renamed, same physical machine).
- netbird-vps.md (no narrative page exists) -> netbird-vps.yaml (substrate
  record, matching the existing convention for hosts without a wiki page).
- runbook-dpkg-interrupted.md refs -> .agents/skills/runbook-dpkg-interrupted/SKILL.md
  (missed in the phase-4 runbook move because the referencing files used a
  bare filename, not a runbooks/ prefix).
- One dangling forward-reference to a never-written investigation delinked
  to the actual incident record it was describing.

Left alone: two links in knowledge/wiki/containers/101-jellyfin.md into
devops/homelab-authentik-admin/ — an intentional reference to a sibling repo,
not present in this checkout.

Verification: broken-link count 126 -> 2 (real remainder is the cross-repo
reference above); gen-topology.py --check still exit 0; build_host_files.py
still idempotent; all inventory.yaml doc_page targets still resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 17:53:35 +02:00
1019a1cb52 docs: section-index pass + docs-lint skill (phase 5)
- Add knowledge/wiki/hosts/index.md (the one missing section index) and point
  knowledge/index.md at it.
- Add .agents/skills/docs-lint/ (SKILL.md + lint.py) enforcing the mechanical
  parts of writing-style.md: banned vocabulary and broken relative links. The
  style guide and this skill are exempt from the banned-word check since they
  enumerate the list.
- Record the restructure + lint in knowledge/log.md.

Verification: banned-vocabulary scan of knowledge/ is clean (the few remaining
repo-wide hits are false positives — the literal '_' character — or historical
append-only plans quoting the vocabulary, which the standard does not restyle).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:42:23 +02:00
5c5016b3c7 docs: reshape runbooks into .agents/skills/<name>/SKILL.md (phase 4)
Problem: runbooks are agent-executable procedures but lived at the repo root,
separate from the other agent instruction now under .agents/.

Change:
- Move runbooks/<name>.md -> .agents/skills/<name>/SKILL.md (folder per skill,
  matching the wiki-hq skills layout). Frontmatter (name, risk_class, inputs,
  verification, docs_update_checklist, transition) preserved.
- Rewrite links (inbound from plans; between-skill siblings) via the move map.
- Update prose references in AGENTS.md, HERMES.md, .agents/OIKOS.md, and the
  operations schema; fix a pre-existing stale link to operations/commands.md.

No code consumed runbooks/ by path, so nothing else changes.

Verification: all SKILL.md frontmatter parses with valid risk_class; every
lifecycle transition resolves to an oikos/ontology.yaml state; broken-link
count 127 -> 126 (fixed one, introduced none).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:39:31 +02:00
8a6422bd7d docs: move narrative wiki under knowledge/wiki/ (phase 3)
Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.

Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
  infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
  knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
  GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
  outbound + between-moved-files), including .hermes/, runbooks, operations,
  investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
  embed doc_page); update oikos/gen-topology.py output path, candidate doc
  paths, and footer links; update code-comment doc paths.

Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.

Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
  topology regen fixed one, introduced none). Remaining are pre-existing refs
  to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:35:23 +02:00
bb5c0717a1 docs: adopt wiki-hq conventions + separate agent instructions (phases 1-2)
Problem: the narrative docs lacked an enforceable style standard, and
agent-facing instruction (OIKOS/CAVEMAN/CONTRIBUTING) was interleaved with
human content at the repo root.

Change:
- Add .agents/shared/{writing-style,llm-wiki}.md — a lint-checkable prose
  standard (with an imperative-voice exception for runbooks/recipes) and the
  sources/wiki/index/log layer model.
- Move CAVEMAN.md -> .agents/shared/caveman.md,
  CONTRIBUTING.md -> .agents/shared/page-templates.md,
  OIKOS.md -> .agents/OIKOS.md; leave thin root stubs so old links resolve.
- Add .agents/domains/{knowledge,operations}/schema.md; operations schema
  codifies "plans always live in plans/".
- Repoint live references (AGENTS, README, GLOSSARY, OIKOS) and fix OIKOS.md's
  internal relative links for its new depth.

Risk: none to the operational substrate — inventory.yaml, hosts/*.yaml,
oikos/, mcp/, secrets/, bin/ untouched (verified via git status).

Verification: relative-link check across .agents/ clean; substrate churn empty.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:22:09 +02:00
14e88c7c5e Document real deploy state: console live, webhook still broken
Console is fully live on apps (105) — deployed manually via deploy.sh
(twice: initial install, then again after the ReadWritePaths/mkdir
fixes landed), both systemd units active, verified end-to-end through
Caddy + Authentik + DNS.

Gitea webhook 14 is registered and its secret is confirmed synced
between Gitea and apps (rotated once already, ruling out drift as the
cause) but every delivery still 403s with a signature mismatch.
Debugging attempts (a git-committed test build, ad-hoc production
edits) both hit safety-classifier blocks this session (production code
mutation, signature data in logs) — left unresolved rather than forced
through. Auto-deploy via push doesn't work yet for this service; manual
deploy.sh re-runs are the workaround until someone tracks this down.

Added to the 60/90-day backlog.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 14:11:27 +02:00
dede118a80 chore: trigger oikos-console webhook to verify rotated secret 2026-07-06 14:02:26 +02:00
72720efa21 Rotate oikos-console-deploy webhook secret
The value written to apps' /etc/oikos-console-deploy/secret didn't
match what Gitea webhook 14 had configured, causing every deploy
attempt to 403 with a signature mismatch — likely drift introduced by
the earlier two-step PATCH sequence (secret set in one call,
branch_filter/active restored in a second call without re-including
the config object). Rotated cleanly this time: fresh secret set on
Gitea and re-encrypted here in one pass, single atomic PATCH covering
config+events+branch_filter+active together.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 14:01:29 +02:00
7610e5394c Deploy Oikos Console to apps (105); fix missing-directory crash
Console is live: cloned to /opt/oikos-console, deploy.sh ran clean,
webhook secret written to /etc/oikos-console-deploy/secret from the
pre-registered SOPS secret (never printed — decrypted and piped
straight into the target file in one command), both systemd units
enabled and active. Verified locally (127.0.0.1:8091 -> 200) and
end-to-end (https://oikos.hubris.network/ -> 302, the Authentik gate
firing correctly).

Found a real bug during first boot: oikos-console.service's
ReadWritePaths listed /opt/homelab-context/signals and .../approvals,
but neither existed yet on apps' clone — git doesn't track empty
directories, and nothing had ever written a signal/approval from that
host. ProtectSystem=strict + a missing ReadWritePaths target is a hard
226/NAMESPACE crash, not a graceful degradation. Fixed two ways:
the unit now marks those paths optional (`-` prefix) so a fresh deploy
never crash-loops on this again, and deploy.sh now mkdir -p's them
explicitly so the console has real write access from the first boot,
not just a non-crashing-but-broken start.

This is also the first real exercise of the auto-deploy pipeline: this
push should land via Gitea webhook 14 -> oikos-console-deploy.service
on apps, same as homelab-mcp/secrets-issuance already work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:57:50 +02:00
610b096840 Add Technitium DNS record for oikos.hubris.network
A record -> 192.168.8.175 (Caddy's LAN IP), created via Technitium's
REST API (login -> createToken -> zones/records/add) in a single
in-memory call. Neither the admin credential nor the resulting session/
API token was ever printed to output or written to disk, and the token
wasn't persisted anywhere after the call completed — it existed only
for the lifetime of that one process.

Verified: dig @192.168.8.2 +short oikos.hubris.network -> 192.168.8.175.
End-to-end confirmation that DNS + Caddy + the Authentik gate are all
wired correctly: curl https://oikos.hubris.network/ now returns a 302
(the forward-auth redirect firing before the not-yet-deployed backend
would even matter) instead of failing to resolve/connect.

This closes out every part of the console rollout except the actual
apps-side bootstrap (oikos/console/deploy/README.md "One-time setup"),
which remains pending direct operator execution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:41:52 +02:00
25c67a79c6 Add Caddy route for oikos.hubris.network; fix loopback-bind bug
Pushed dtoro/caddy-conf@c195142: oikos.hubris.network -> 192.168.8.205:8091,
Authentik-gated (matches paperless.hubris.network's live pattern —
confirmed exact snippet syntax against the real Caddyfile rather than
trusting the paraphrase in the original README, which turned out to
have the wrong forward_auth target: the live snippet points at
127.0.0.1:8099 on Caddy's own LXC, not 192.168.8.6:9000 as
containers/106-auth-outpost.md's older text suggested). Reload verified
clean — an unrelated existing route stayed healthy through it.

Found and fixed a real deploy-blocking bug in the process:
oikos-console.service bound 127.0.0.1 only, but Caddy runs on a
different host (121) and can only reach apps (105) over the LAN — the
console would have been completely unreachable once deployed. Now binds
0.0.0.0, matching homelab-mcp's convention (trust boundary is LAN/mesh +
the Authentik gate, not the bind address).

Encountered and deliberately left alone: a pre-existing local clone at
/tmp/caddy-conf with an unpushed commit + uncommitted diff about
jellyfin's auth gating, from before this clone fell 12 commits behind
origin. That work turned out to be superseded (origin already reached
the same conclusion — SSO plugin handles jellyfin auth, no forward-auth
gate — via a different, already-merged path). Didn't touch it; used a
fresh clone instead to avoid any risk of losing or corrupting that state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:31:51 +02:00
493ae50f37 Register Gitea webhook 14 for oikos-console deploy
Created via the Gitea API (POST /repos/dtoro/Homelab-Docs/hooks) rather
than the UI, since the existing PAT turned out to have sufficient scope.
Webhook id 14: http://192.168.8.205:9831/deploy, push events, main branch
filter, active.

The shared secret was generated and registered with Gitea before the
apps-side bootstrap ran (order reversed from the usual install.sh-first
flow, since direct SSH deploy to apps is still pending operator
execution — see oikos/console/deploy/README.md). Stored as
secrets/oikos-console-deploy-secret.yaml (SOPS, recipient: apps only)
rather than left as a local plaintext file, with explicit operator
sign-off. When the apps-side install runs, skip webhook/install.sh's
random-secret generation and write this exact value into
/etc/oikos-console-deploy/secret instead.

infrastructure/auto-deploy.md updated with the real webhook id (was
"not yet registered").

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:18:46 +02:00
c6fedb38c1 Document teddycloud (pve_id 131) — closes the Week-3 drift finding
teddycloud was live on hubris (LXC 131, docker compose, TeddyCloud —
a Toniebox cloud reimplementation) but never made it into inventory.yaml.
Already referenced in passing by containers/132-rclone.md ("131 was
already taken by an undocumented teddycloud container") and
hosts/strong.md's migration changelog (a DHCP conflict fix), but no
inventory entry or doc page existed until oikos/drift.py's inventory-
vs-live check caught it.

Verified live via read-only SSH (pct config 131, pct exec 131 -- ...,
docker ps): hostname, static IP 192.168.8.150, 1 core/1GiB/16GiB rootfs,
Debian 12, runs via docker compose at /opt/teddycloud. No changes made
to the running container.

Also fixed: house's inventory notes claimed 192.168.8.212 is teddycloud's
current IP via DHCP — stale, teddycloud has a static IP now.

Flagged in the new container page: teddycloud has no Caddy forward-auth
gate, unlike sab.hubris.network on the same Caddyfile.

`python3 oikos/drift.py` no longer reports an inventory-vs-live finding
for pve_id 131. (A separate, pre-existing gap surfaced while verifying
this: rclone's own inventory.yaml block is missing pve_id/host/lan_ip —
out of scope here, flagging for a follow-up.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:06:26 +02:00
fd35b48c8d Phase 1-4: full doc reorg
Phase 1 — fix stale state after strong migration (Phase 1+2, 2026-07-05)
  - README: corrected IPs (jellyfin 206→246, arriman 132→245, etc.),
    added missing containers (128 trmnl, 129 house, 133 seanime, 134 romm,
    124 authentik), updated last-refreshed date, added strong host context
  - containers/101-jellyfin.md: IP 206→246, host hubris→strong, mount
    /mnt/library→/mnt/media_local, GPU 760M→680M+RX7600, privilege→priv
  - containers/118-elementsynapse.md: IP 239→242, added Host: strong
  - containers/122-arriman.md: IP 132→245, mount→/mnt/media_local, added Host
  - containers/129-house.md: IP 212→244, added Host: strong
  - containers/130-grimmory.md: IP 213→247, mount→/mnt/media_local, added Host
  - containers/121-caddy.md: fixed site list (books→grimmory, removed auth→VPS,
    added house, roms, teddy, trmnl)
  - hosts/strong.md: updated At-a-glance to reflect 7 LXCs hosted
  - containers/123-claudio-bot.md, 127-mule-photos-new.md: archived to
    containers/archive/ (were destroyed LXCs with living pages)
  - inventory.yaml: verified correct — no changes needed

Phase 2 — structural cleanup
  - infrastructure/index.md: one-page overview of all cross-cutting systems
  - runbooks/: moved runbook-budget-from-csv.md and runbook-dpkg-interrupted.md
    from operations/ with YAML frontmatter added
  - plans/done/: moved 4 completed plans out of active view; updated index
  - vms/index.md: added VM index page

Phase 3 — navigation & discoverability
  - GLOSSARY.md: term definitions (Authentik, Caddy, LXC, VAAPI, etc.)
  - README: added table of contents, links to glossary + infrastructure index
  - investigations/: archived 2 resolved cases (crash-loop, authentik-migration)
    to investigations/archive/; updated index with active vs archived sections

Phase 4 — ongoing discipline
  - CONTRIBUTING.md: documented same-session update rule with explicit checklist
  - README: replaced full LXC table with summary + link to containers/index.md
    (single source of truth; de-duplication)
2026-07-06 00:46:27 +02:00
205d8a1a43 Oikos Week 4: Console v0, approval hardening, docs pass, backlog
Oikos Console v0 (oikos/console/) — read-mostly, server-rendered FastAPI
+ Jinja2 web UI, no SPA build chain. Signals landing page, service grid
+ detail, node/blast-radius view, live Mermaid relationship graph, drift
findings, approvals queue (approve/deny, destructive confirmation-phrase
enforced), daily/weekly reports. Tested end-to-end via the preview tools
against live production data, including a real click-through of the
approve/deny flow.

Found and fixed two bugs during that testing:
- Severity-dot CSS classes didn't match the actual severity strings
  (dot-warn/dot-crit vs "warning"/"critical") — warning-severity signals
  rendered with no visible indicator at all.
- The console's sys.path setup pointed at its own webhook checkout
  (/opt/oikos-console) rather than /opt/homelab-context, which would have
  made its oikos.* imports resolve to a SEPARATE copy of oikos/signal.py
  etc. than the scheduler and CLI use — silently forking signal/approval
  data into two locations in production. Fixed to match mcp/server.py's
  CONTEXT_DIR pattern. Also added _commit_push() so the console's writes
  (approval replies, signal ack/resolve) don't sit uncommitted against
  the 5-min-synced clone.

Split oikos/gen_topology_lib.py out of oikos/gen-topology.py (hyphenated
filenames aren't importable) so the console's /graph route can render
live without shelling out.

oikos/console/deploy/ — third webhook on dtoro/Homelab-Docs (port 9831),
matching the homelab-mcp/secrets-issuance precedent. README documents the
Caddy route and Gitea webhook registration this repo can't do for itself,
and that Authentik step-up on /approvals needs a live instance to
configure.

Approval hardening: grants are now single-use (oikos/approve.py
check_grant marks the request "executed" atomically, so a second call
for the same id fails even within the TTL) — verified with a test. Per-
agent age-key-signed requests, as originally planned, turned out not to
be buildable as stated: age is encryption-only, no signing primitive.
Documented the real alternative (SSH-key signing) and moved it to the
60/90-day backlog pending an inventory schema gap (no SSH pubkeys
recorded today).

Docs pass: added the Oikos command surface to operations/commands.md,
new MCP tools to AGENTS.md. Found two more stale references while at
it — commands.md and AGENTS.md both still pointed DNS at the destroyed
LXC 124/dnsmasq instead of Technitium on dns (107), and a claudio-monitor
reference deprecated since 2026-06-04 — fixed both.

60/90-day backlog written into OIKOS.md, derived from gaps actually
observed this month, not guesswork.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 00:03:51 +02:00
2084a1583e Oikos Week 3: scheduler, drift detectors, signals, classifier, approvals
New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:

- oikos/scheduler.py — Observe stage: HTTP health probes for every
  service, disk-usage probes on hubris/strong, writes oikos/state.json
  (gitignored — regenerates every run). `homelab service <name> health`
  is now cache-first; `--live` forces a fresh probe. Deploys via
  oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.

- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
  detectors (fully local, no SSH) plus pct-list and Caddy-backend
  detectors (best-effort SSH, degrade to an info finding when
  unreachable rather than a false drift alarm). Found real, currently-
  true drift on first run: republic-laptop's age key granted on every
  secret but missing from inventory.yaml, grimmory missing from
  hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
  recorded in OIKOS.md for the operator, not auto-fixed (each is a
  config_mutation/destructive decision).

- oikos/signal.py — the attention layer: raised -> acknowledged ->
  acting -> resolved|muted lifecycle, severity-based routing, dedup via
  open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.

- oikos/decide.py — the Decide-stage classifier: risk class x blast
  radius x ledger-history confidence -> auto-act/escalate. Adds an
  action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
  service_name from the entity for per-service policy overrides.
  `homelab decide <action> <entity>`.

- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
  in this homelab, so this is the repo-side half only: request/reply/
  grant lifecycle with short-TTL HMAC-signed tokens (new secret
  secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
  delivery is Hermes's existing @dtoro:avispero send path (documented
  integration contract in the module docstring), not a new bot.
  `homelab restart` now mechanically refuses config_mutation/destructive
  services without a valid --approval-id, regardless of -y/interactivity.

- oikos/report.py — daily brief + weekly report from signal/approval/
  ledger state (no Prometheus yet, so point-in-time counts only).

- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
  not provisioned: no pve_id is guessed here since Proxmox assigns real
  IDs at creation time, and drift already found an unclaimed ID (131) to
  investigate first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 23:29:39 +02:00
48debc0911 runbooks: clarify Netbird join is optional, not a required enrollment step
Only off-LAN-reachable workstations (e.g. republic-laptop, mac-mini)
need to join Netbird. LAN-reachable LXCs/VMs on 192.168.8.0/24 don't —
they're already directly reachable, and off-LAN clients reach them via
hubris's routed 192.168.8.0/24 Netbird network resource. Brings the
runbook in line with oikos/ontology.yaml's lifecycle transition, which
already says "mesh-joined-if-needed".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 23:03:21 +02:00
f6b57cbe3a Oikos Week 2: Service Console v0, change ledger, node relations, runbooks
Adds the shared kernel modules (oikos/policy.py, oikos/relations.py,
oikos/ledger.py) that let every surface — CLI, MCP, context-card
generator — agree on risk classification and ontology graph walks
from one implementation.

homelab CLI: `service <name> explain|health|docs|log|actions|history`
(Service Console v0), `change preflight <service>`, `node <name>
relations`. Restart and client add/remove now append change-ledger
entries (ledger/*.jsonl, committed alongside the change they record).

mcp/server.py mirrors explain/preflight/get_relations/get_change_history
as MCP tools, card-first so agent orientation is one call instead of
several search_docs/get_page round-trips.

oikos/gen-topology.py now also emits a compact context card per host
and service (oikos/cards/*.md) — identity, blast radius, safe actions +
risk class, doc pointer, recent ledger history.

runbooks/*.md: service health check, config change + deploy, client
enrollment, incident investigation, and the five node lifecycle
transitions (provision/activate/migrate/deprecate/destroy), each with
machine-readable frontmatter (risk class, inputs, verification,
docs-update checklist). Wired into HERMES.md so agents load these
instead of rediscovering topology per-task.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 23:02:32 +02:00
b230ab5937 Oikos Week 1: kernel policy, ontology, service contract, topology gen
Adds the Oikos agent-OS kernel: oikos/policy.yaml (risk classes +
approval rules for every homelab/MCP command), oikos/ontology.yaml
(8-domain systems model, typed relationships, node lifecycle), and
OIKOS.md (OODA loop operating brief, linked from AGENTS.md).

Extends inventory.yaml with a stable service contract (doc_page,
config_repo, risk_notes) on all 17 services, and a structured
archaeology: section for the 13 destroyed LXCs (was scattered
comments + a narrative table). Fixes stale drift found in the
process: authentik's backend pointed at a retired LXC (124); core
has run on the VPS since 2026-05-31.

Adds oikos/gen-topology.py, generating infrastructure/topology.md
(Mermaid compute/ingress + storage views) from inventory.yaml.
build_host_files.py now carries state/storage/depends_on into
generated hosts/*.yaml.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 22:50:34 +02:00
7e8860ab47 update seanime docs: qBittorrent config, extensions, container doc, arriman note 2026-07-05 21:17:04 +02:00
e04d943d5c fix VPS traefik backends after Phase 1+2 migration
- Jellyfin: 192.168.8.206 → 192.168.8.246 (stale after LXC 101 migration to strong)
- House: 192.168.8.212 → 192.168.8.244 (stale after LXC 129 migration to strong)
- Jellyseerr/qbit/sab: 192.168.8.132 → 192.168.8.245 (arriman on strong)
- Added migration pitfalls section to reference doc
2026-07-05 21:03:34 +02:00
7d5e7227ca Add RomM LXC 134 docs (strong, 192.168.8.249, roms.hubris.network) 2026-07-05 20:58:13 +02:00
ec52dfb6a2 add seanime LXC 133 on strong (anime media server, 192.168.8.248) 2026-07-05 17:12:38 +02:00
bb33963539 strong migration Phase 1+2: move 5 LXCs + library split to ludo-lvm 2026-07-05 16:24:48 +02:00
2709e79455 jellyfin: VAAPI HW accel + Authentik SSO + resource bump (2026-07-04)
- Upgraded 10.11.8 → 10.11.11, enabled VAAPI (Radeon 760M)
- Bumped to 4 cores / 8 GiB RAM / 1 GiB swap
- SSO-Auth plugin v4.0.0.4 with Authentik OIDC
- Removed Caddy forward-auth gate (incompatible with SSO plugin)
- Updated container doc with full SSO architecture + pitfalls
2026-07-04 22:00:46 +02:00
831794e98c docs(rclone): root-cause the "stalls" as OOM, not Proton; bump RAM to 2G
The recurring silent-freeze incidents on LXC 132 were rclone-rcd.service
getting OOM-killed under the original 1 GiB allocation, not a protondrive
backend quirk as first suspected. journalctl confirmed the OOM kill at the
exact freeze point. Bumped LXC memory to 2 GiB (live, no reboot) and the
full folder set (cloud/documents/repos) completed cleanly afterward.

Also documents two watchdog bugs found while chasing this: a wrong
stats-group key that made a healthy sync look falsely frozen, and a
blocking systemctl restart that caused the watchdog to silently disable
itself after firing once. Both fixed; watchdog kept as a safety net.
2026-07-03 22:04:43 +02:00
6669feafdc docs(rclone): document protondrive silent-stall incident + watchdog
Two silent stalls hit in LXC 132's first 24h of real traffic: rclone's own
--timeout didn't catch a protondrive-specific hang (transfer at 100%, zero
bytes/errors/retries for hours). Added a 5-min watchdog timer that restarts
rclone-backup.service if transferred bytes are frozen for 15+ min. Also
found and fixed a monitoring bug in the runner (wrong stats-group key) that
made a healthy sync look falsely stalled for 22h in its own log.
2026-07-03 12:32:53 +02:00
ba93c4709b docs(rclone): LXC 132 rclone -> Proton Drive backup; deprecate restic-on-USB
New off-host backup job replacing the disabled restic-on-USB backup: LXC 132
`rclone` mirrors selected /mnt/library folders to Proton Drive (plain rclone
sync, Proton's built-in E2E, no crypt overlay) on a monthly timer, with
rclone's Web GUI for LAN-only browsing/ad-hoc runs and live job status.

- containers/132-rclone.md: full design, Proton auth gotcha (TOTP secret vs
  live code), pct exec PATH gotcha, rc-API job-visibility runner rewrite,
  selected folder set (cloud/documents/repos), deferred tracked-repo note.
- infrastructure/backups.md: restic-on-USB marked DEPRECATED/superseded,
  leads with the new job now.
- containers/index.md, README.md, infrastructure/media-permissions.md:
  register the new container.
2026-07-02 00:37:10 +02:00
root
5887129202 client-add: rclone (finalize age_pubkey + grant shared secrets) 2026-07-01 23:50:37 +02:00
root
1ad31fe33a client-add: rclone 2026-07-01 23:37:32 +02:00
4e3ec61fb0 chore: ignore __pycache__/*.pyc
Left behind by py_compile-checking bin/homelab's syntax during today's
bugfix session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 22:51:44 +02:00
abb1476fc8 fix(homelab): cmd_sync missing the geteuid guard every other command has
Every other mutating subcommand (secret, refresh-creds, client add/remove)
already re-execs via sudo only when os.geteuid() != 0. cmd_sync was the
one exception, calling sudo unconditionally — fails with "No such file or
directory: 'sudo'" on minimal root-only images (no sudo binary at all),
hit live running `homelab sync` on strong over root SSH.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 14:02:07 +02:00
dee08b97a4 secrets: grant strong access to hello.yaml and gitea-pat.yaml
Standard baseline for an enrolled workstation-class client, matching
mac-mini/republic-laptop/etc: the bootstrap decrypt-test secret plus
the write-scoped Gitea PAT so strong can push to the wiki repo on its
own (homelab client add/remove, wiki edits from that host).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 14:01:00 +02:00
359f55a695 docs(strong): record age-key issuance and the 3 bootstrap.sh bugs found fixing it
strong now has its own age key (issued over LAN via --no-mesh),
pubkey recorded in inventory.yaml. Not yet granted to any secrets
file - that's a separate decision.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:40:47 +02:00
fc62cf38f7 fix(bootstrap): add --no-mesh so LAN-only secrets issuance doesn't block on Netbird SSO
Running bootstrap without --no-secrets always tried to install and
connect Netbird, even when the mesh-check right after it already knows
how to fall back to plain LAN reachability. On a host nobody's watching
interactively (e.g. driven over SSH), this hangs forever at the
device-code prompt — hit live on strong, had to kill the stuck
`netbird up` process manually. --no-mesh skips netbird install/up while
still allowing the existing LAN-fallback path to satisfy secrets
issuance.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:38:21 +02:00
b266c3f1d8 fix(bootstrap): sops has no apt/dnf package, fetch the binary directly
Discovered live re-running bootstrap on strong for secrets issuance:
apt-get install sops fails outright (no such Debian package — matches
what agent-enrollment.md's manual-install recipe already does, fetching
the binary from GitHub releases instead of a package manager). dnf would
have the same problem. Added install_sops_binary(), used on both the
dnf and apt paths; Darwin still installs via brew, which does carry sops.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:34:12 +02:00
fdab6282e6 fix(bootstrap): don't invoke sudo when already running as root
The pipx/mcp-CLI step and the Hermes goose installer both called
`sudo -u <user> ...` unconditionally. On minimal Linux images reached
via `ssh root@host` (no SUDO_USER, and often no `sudo` binary at all —
seen live on strong), this failed with "sudo: command not found" and
silently no-opped the mcp CLI install. Added a run_as() helper that
only shells out to sudo when there's a real invoking user distinct
from root.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:33:19 +02:00
e4b529b71b docs(strong): record homelab-context client enrollment
bootstrap.sh --no-secrets ran clean: sync timer, homelab CLI, and
AGENTS.md are live on strong. Noted two follow-ups: secrets issuance
is reachable over plain LAN (mesh: lan) so age-key enrollment doesn't
actually need Netbird, and bootstrap's pipx/mcp-CLI step silently no-ops
when run as root over SSH (missing `sudo` binary).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:30:05 +02:00
c012a9124f infra: join ludo-mini to hubris as Homelab cluster node "strong"
Reformatted the ludo-mini workstation to Proxmox VE 9.2.3 and joined it
to hubris's existing single-node "Homelab" cluster (2 nodes, no QDevice
yet). Added a second NVMe as its own LVM-thin pool (ludo-lvm). Renamed
the wiki/inventory identity from ludo-mini to strong to match the OS/
cluster hostname, since bootstrap's client-enrollment lookup depends on
that match. Also regenerated hosts/grimmory.yaml, which was missing from
git despite being referenced by inventory.yaml.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:28:24 +02:00
41df77fb20 docs(runbook): update budget-from-csv with API corrections and Cookie Share calc
- Fix subscription path to /budget/subscriptions (not /subscriptions/)
- Document RRULE rejection; use cycle_interval instead
- Add GET /budget/ vs ?month= behavior note
- Add full category/payment-method ID tables
- Add Cookie Share recommendation: 2,650 €/month based on 6-mo analysis

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 11:45:34 +02:00
b67389b137 docs(operations): add budget-from-csv runbook for Yuvomi
Step-by-step procedure to import N26 CSV transactions into Yuvomi's
Budget and Subscriptions modules using the yuvomi-mcp tools.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 10:55:25 +02:00
45b7813eb7 chore(grimmory): finalize age_pubkey for LXC 130
Fills in the age public key issued by secrets-issuance after
homelab client enrollment completed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 09:17:31 +02:00
8f6f43eeb1 docs(grimmory): fix IP assignment — PVE static config, not DHCP reservation
All LXCs use ip=<addr>/24 in pve net0 config directly. No Fritz!Box
lease needed. Updated container doc and migration runbook accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 08:52:34 +02:00
c2c257aa12 docs(grimmory): correct Authentik OIDC setup notes
OIDC config migrated from Booklore DB dump — Confidential client (not
PKCE), credentials intact. offline_access scope added to Authentik
provider. Backchannel logout URL set to permanent IP 192.168.8.213.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 08:48:52 +02:00
41fe34a3c3 migrate: Booklore → Grimmory on new LXC 130
Grimmory (grimmory-tools/grimmory) is the community fork/successor of
Booklore. Moving it off the shared apps LXC (105) onto a dedicated
container (130, 192.168.8.213) to avoid the Portainer-managed stack
footgun that wiped mariadb in April.

- containers/130-grimmory.md: new container doc with compose layout,
  Authentik OIDC notes (PKCE/Public client), media-GID setup
- plans/2026-06-29-grimmory-migration.md: step-by-step runbook for
  DB dump/restore, LXC provisioning, Caddy cutover, Authentik update
- inventory.yaml: add grimmory LXC 130
- containers/105-apps.md: remove Booklore section + hostname entry
- containers/index.md: update 105 row; add 130 row
- infrastructure/media-permissions.md: add LXC 130; update Booklore→Grimmory refs
- README.md: update 105 row; add 130 row

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 01:11:00 +02:00
b6c9184a7b decommission: destroy plato (LXC 126)
LXC 126 stopped and destroyed on hubris. Remove all live references:
inventory, container doc, host file, README, containers index, auto-deploy
pipeline, DNS entry, SSH access table, nfs-export mount list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-28 22:32:59 +02:00
root
636c90f99c secrets: populate hermes-house-users with household Signal numbers
+4917622791635 → Yuvomi user 1 (Taco)
+491726924525  → Yuvomi user 3 (Steffi)
2026-06-28 16:50:51 +02:00
root
cef1b836c2 sops: add yuvomi-api-token + hermes-house-users secrets (house + workstations) 2026-06-28 16:02:13 +02:00
root
335498d069 client-add: house (finalize age_pubkey + grant shared + hermes secrets) 2026-06-28 15:59:47 +02:00
root
867998e9c0 hosts: regenerate — add house, mac-mini, ludo-mini; remove stale orphans 2026-06-28 15:58:36 +02:00
cb0c1a5c0d house: add Google Calendar OAuth config to docs 2026-06-27 14:40:29 +02:00
ddf541ebd3 house: switch Paperless to native DMS connector (remove WebDAV bridge) 2026-06-26 19:57:50 +02:00
4472e73ae1 Yuvomi deployment: LXC 129 house.hubris.network, Docker, Caddy, DNS, VPS traefik, Authentik OIDC, Paperless WebDAV bridge 2026-06-26 19:06:49 +02:00
59261c7ef1 hosts: generate trmnl.yaml (prereq for SOPS enrollment of LXC 128) 2026-06-24 23:41:29 +02:00
41f22c2d85 trmnl (128): auto-deploy pipeline + LAN DNS record wired
- auto-deploy.md: dtoro/terminalito pipeline (webhook 12, :9797), ALLOWED_HOST_LIST
  += 192.168.8.211, changelog
- 128-trmnl.md: auto-deploy now wired, Technitium A record done, changelog
- 107-dns.md: A record trmnl -> 192.168.8.175 (changelog)
- 104-gitea.md: webhook id 12 + allowed host (changelog)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 18:06:56 +02:00
b7cfc350b3 trmnl (128): public path verified live; root-cause the mesh outage
Root cause of the provision-time 504s: netbird home-lab-network (192.168.8.0/24)
had no active routing peer — mac-mini routing peer's netbird daemon was down, so
all home-backed public services (artifacto/blog/trmnl) 504'd at the VPS edge.
netbird up on mac-mini restored it; verified trmnl public 200/401, artifacto 200.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 17:38:57 +02:00
fffa5560eb trmnl (128): provisioned — container page, inventory, ingress
LXC 128 trmnl hosts the TRMNL plugins middleware (dtoro/terminalito), polled by
TRMNL cloud. trmnl-plugins.service on :9851; Caddy block + LE cert; VPS traefik
router trmnl-public + cert mirror. Public path pending VPS<->home netbird route
recovery (was "No networks available" at provision time, artifacto/blog 504 too).
LAN Technitium record + SOPS enrollment + Google/MVG creds pending. Plan -> In Progress.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 17:24:31 +02:00
d6bfd7b8c9 plans: TRMNL plugins LXC (128) + middleware deploy pipeline
Pre-flight runbook for a dedicated trmnl LXC hosting self-hosted middleware for
TRMNL e-ink plugins (TRMNL cloud polls /<plugin>/dashboard). First consumer:
munich-home. Source repo dtoro/terminalito; this documents the fabric wiring
(provision via homelab client add, Shape-B auto-deploy, SOPS secret, caddy + VPS
traefik exposure). Status: Planned.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 16:59:03 +02:00
b71402aaed add auth-outpost (106) and dns (107) to inventory + regenerate host files 2026-06-22 09:31:04 +02:00
2a7876c711 add caveman compression mode to agent bootstrap
AGENTS.md now instructs every agent to communicate
tersely. CAVEMAN.md holds the full rules. Agents
apply from first message in each session.
2026-06-21 23:36:06 +02:00
666 changed files with 94582 additions and 6203 deletions

44
.agents/NOMOS.md Normal file
View File

@@ -0,0 +1,44 @@
# NOMOS.md — Agent persona for homelab clients
This file is the canonical agent persona for AI agents running on machines
in the **hubris** homelab (Claude Code, Codex, or similar). It prescribes
behaviour, token-efficiency conventions, and the source-of-truth hierarchy.
The *production* Nomos agent (`cmd/nomos`, the containerized MCP client
gateway everyone actually talks to) uses a separate, code-adjacent persona —
`nomos/SOUL.md`, baked into its Docker image at build time
(`compose/nomos/Dockerfile`). This file is unrelated to that one; it's for
AI coding agents working *on* a homelab client machine, not the Nomos
service itself.
## Source of truth
The homelab-context repo at `/opt/homelab/` is the single source of
truth for:
- Fleet topology (`inventory.yaml`)
- Agent behaviour and conventions
- Everything in this file
When in doubt, check `/opt/homelab/` first, or query the Oikos API/MCP
server directly (see [AGENTS.md](../AGENTS.md) §3-4) — the database is
authoritative at runtime.
## Runbooks — load, don't rediscover
For the canonical workflows (service health check, config change +
deploy, client enrollment, incident investigation, and each node
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
acting. Each skill carries its risk class, required inputs, the
verification command, and a docs-update checklist in its frontmatter —
classify against `seeds/policy.yaml` using that risk class before any
mutation. Don't re-derive topology or the mutation path by grepping the
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
operating model these runbooks execute inside (OODA loop, risk classes,
approval flow, ontology).
## Token efficiency
Apply [caveman.md](shared/caveman.md) — terse, fragment-heavy chat responses
(not committed documentation). There's no separate tool to install for
this; it's a response-style convention any agent follows by reading the
file.

190
.agents/OIKOS.md Normal file
View File

@@ -0,0 +1,190 @@
# Oikos — the operating model
Oikos (Greek: *household*) is the agent operating system layered on this
repo. It is not new infrastructure: `seeds/inventory.yaml` is the kernel data
structure, the Oikos REST API and MCP server are the syscall surface, and
this page defines the rules everything above them follows.
Read this after [AGENTS.md](../AGENTS.md). Machine-readable companions:
[seeds/ontology.yaml](../seeds/ontology.yaml) (systems model),
[seeds/policy.yaml](../seeds/policy.yaml) (risk & approval).
## The kernel loop: OODA
Every Oikos activity — scheduled probe, agent task, operator request — is
one pass through **Observe → Orient → Decide → Act**:
1. **Observe** — probes, drift detectors, and agent findings produce
**Signals** (structured records, not loose messages): pending updates,
high temperature, low disk, service down, cert expiry, stale backup,
inventory drift.
2. **Orient** — walk the ontology graph: what entity is affected, what
depends on it (blast radius), its lifecycle state, whether a runbook
matches, what the ledger says about past attempts.
3. **Decide** — the classifier scores **risk class × blast radius ×
confidence** and routes:
- **auto-act**: within autonomy policy, high confidence, contained radius
- **escalate**: operator approval via Matrix (✅/❌ reaction) or the
Oikos Console's `/approvals` page (destructive actions additionally
need a typed confirmation phrase either way)
- **queue**: informational — console + reports
The classifier can only *lower* autonomy relative to policy, never raise
it. When in doubt, escalate.
4. **Act** — execute through `homelab` commands or runbooks (never ad-hoc
SSH), then **verify** with the action's verification command, write a
**ledger** entry, resolve the Signal, and update docs in the same session.
## Primitives
| Primitive | What it is | Lives in |
|---|---|---|
| Host / Service | topology entities | `inventory.yaml` |
| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` |
| Runbook | executable workflow with risk class + verification | `.agents/skills/<name>/SKILL.md` |
| Signal | something needing attention, with lifecycle | DB `signals` table |
| Change | one mutation: who, what, risk, approval, verification | DB `audit_log` + `executions` tables |
| Approval | short-TTL signed grant for a gated action | DB `approvals` table |
| Incident | investigation narrative | DB `knowledge_entities` (seeded from investigations) |
| Knowledge | document, runbook, investigation | DB `knowledge_entities` (seeded from `seeds/knowledge.yaml`) |
| Plan | design doc for non-trivial work | `plans/` |
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
## Risk classes (enforced, not advisory)
From [seeds/policy.yaml](../seeds/policy.yaml):
- **read_only** — status, logs, docs, inventory. Unattended.
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger.
- **config_mutation** — tracked-config edits (commit+push, never local),
deploys, upgrades, DNS/ingress changes. Operator approval.
- **destructive** — destroy, format, wipe, rotate, revoke. Approval +
typed confirmation phrase.
Lifecycle gates modify these: `provisioning` nodes are freely mutable
(nothing depends on them); `deprecated` nodes accept no new dependents;
anything touching a `destroyed` node is drift.
## The systems model
Eight domains — physical, compute, network, storage, software,
identity & access, operations, external — cover everything in the lab;
entities are connected by typed edges (`hosts`, `provides`, `mounts`,
`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …)
defined in [seeds/ontology.yaml](../seeds/ontology.yaml). Rule of
completeness: **if it can break, be changed, or hold data, it has an
entity and edges.** Blast-radius questions ("what breaks if strong goes
down?") are graph walks, not doc archaeology.
Nodes move through an explicit lifecycle —
`planned → provisioning → active → migrating → deprecated → destroyed`
stored as `state:` in inventory (absent = active). Destroyed nodes live in
the `archaeology:` section. Each transition is a runbook checklist;
deprecation completes only when inbound edges reach zero.
Generated views: the live topology graph at `oikos.hubris.network/graph`
via the API's `/api/v1/graph` endpoint, and the Mermaid export at
`GET /api/v1/graph?format=mermaid`.
## Conventions carried forward
- Inventory is the truth; live state wins over narrative docs.
- Prefer `homelab` CLI and MCP over ad-hoc SSH.
- Meaningful changes update docs in the same session.
- Secrets are decrypted locally via per-client keys; never into docs/comments.
- Tracked configs change by commit + push, not local edits.
- Netbird is the preferred mesh path for new traffic.
- Agents are terse ([caveman.md](shared/caveman.md)), verify claims, and fix
collateral drift when found.
## Build status (Go rewrite — deployed 2026-07-07)
The Oikos runtime was rewritten from Python to Go over 6 phases and is deployed
in Docker on mac-mini. See
[plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](../plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md)
for the full plan. The Python codebase has been removed; all functionality runs
in the Go binary.
**Phase 1 — Ontology + DB (DONE):**
- `migrations/` (001020, forward-only): TimescaleDB hypertables, entity_status, CAGGs,
retention policies, knowledge entities with FTS. Idempotent.
- `seeds/{ontology,inventory,policy,knowledge}.yaml`: DB-native bootstrap +
DR export. Knowledge seed contents are not hardcoded here — count them
from the seed or query the DB.
- `blast_radius()` SQL CTE, type hierarchy, abstract types, relationship
validation.
- Go packages: `internal/db/`, `internal/ontology/`, `internal/domain/`,
`internal/knowledge/`.
**Phase 2 — API (DONE):**
- Single binary `cmd/oikos` with `oikos api` serving REST (:8090) + MCP
on the same service layer. OpenAPI-first (`api/openapi.yaml`) with
oapi-codegen + chi. RFC 9457 problem+json errors. Cursor pagination,
If-Match/ETag optimistic concurrency, idempotency keys, SSE event stream,
OIDC JWT + static bearer auth, audit middleware.
- Go packages: `internal/httpapi/`, `internal/httpapi/gen/`.
**Phase 3 — Control loop (DONE):**
- Scheduler (`oikos scheduler`): check_defs runner, signal dedup/flap
suppression, entity_status. HTTP, TCP, disk, cert-expiry probes.
- Actuator: SSH skill procedure execution with context-aware timeouts,
circuit breaker, retry budgets, error classification.
- Learning engine: hourly pattern extraction, Wilson confidence bounds,
anomaly detection, skills with validated patterns.
- Notifier: Matrix badge delivery, approval token generation (HMAC,
single-use, hashed), DB rendezvous pattern.
- Policy classifier: risk class determination, autonomy routing,
blast-radius computation, kill-switch support.
- Go packages: `internal/scheduler/`, `internal/actuator/`,
`internal/learning/`, `internal/notifier/`, `internal/policy/`.
**Phase 4 — Agent / Nomos (DONE):**
- Standalone Nomos MCP client binary (`cmd/nomos`) with gateway mode
(:8092). Structured queries + natural-language routing to the MCP tool
list (see AGENTS.md §3). Agent activity logging on every tool call. No SSH keys.
- `nomos/` directory with config, SOUL.md, homelab-ops skill.
- Nomos Docker service in `docker-compose.yml` (profile: full).
- Go packages: `cmd/nomos/`, `compose/nomos/`.
**Phase 5 — Secrets / Infisical (DONE):**
- `internal/secrets/`: backend abstraction (Manager) with primary
(Infisical) and fallback (SOPS) backends. Machine identities via
UniversalAuth. In-memory cache with TTL.
- `oikos secret` CLI: list, migrate (SOPS → Infisical), export-sops
(DR fallback). Infisical SDK v0.8.0.
- Rotation runbook at `secrets/rotation.md`.
- Docker compose: `infisical` + `redis` services (profile: infisical).
**Phase 6 — Deploy + cutover (DONE, pending production cutover):**
- CI pipeline: `.gitea/workflows/ci.yml` (Gitea Actions — build, vet,
lint, test, docker build).
- Deploy: `scripts/deploy.sh` (git pull → docker build → compose up →
health check), SHA-tagged images, rolling restart.
- Caddy config: `compose/caddy/Caddyfile.oikos` (oikos/mcp/nomos →
mac-mini mesh :8090/:8092).
- Watchdog: `scripts/watchdog.sh` (2min cron, Matrix alert on failure).
- Verification: `scripts/verify-phase6.sh` (14/14 checks pass).
- Rollback: `scripts/rollback.sh` (checkout SHA + pg_restore).
- Cutover checklist: `scripts/cutover-checklist.md`.
**Current deployment:**
- **Production**: Docker stack on mac-mini (`--profile full`: postgres, api,
scheduler, notifier, nomos). Deployed 2026-07-07 with full knowledge seed.
The Python MCP server and secrets-issuance on apps/105 have been stopped
(see `scripts/cutover-checklist.md`).
## Python-era backlog (superseded)
The original 30-day roadmap (Python, shipped 2026-06/07) delivered:
context cards, change ledger, node relations, runbooks, ops scheduler,
drift detectors, signal engine, classifier, approval engine, and the
FastAPI+Jinja2 Oikos Console. All of these have been re-implemented in
the Go rewrite. The backlog items below that referenced Python paths
(`oikos/approve.py`, `oikos/drift.py`, `oikos/console/`) are now addressed
by the Go equivalents listed above.
Outstanding from the Python era (not yet in Go):
- Prometheus provisioning (see [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md))
- CPU/NVMe temperature probing (blocked on sensor path discovery)
- SSH-key-signed approval requests (blocked on inventory schema)
- Multi-agent delegation (blocked on ledger identity field)
- Restore drills on a schedule

207
.agents/dev/CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,207 @@
# Agent developer guide
Instructions for AI agents working on the Oikos codebase. Read this after
[AGENTS.md](../../AGENTS.md) and [OIKOS.md](../OIKOS.md). Human developers:
see [CONTRIBUTING.md](../../CONTRIBUTING.md) for a human-friendly version.
## Codebase map
```
cmd/oikos/main.go Entry point. Subcommands: api, scheduler, notifier, migrate,
seed, export, secret, all
cmd/nomos/main.go Nomos MCP client gateway (standalone binary, formerly Hermes)
cmd/webhook/main.go Gitea deploy-webhook receiver (push-to-deploy on mac-mini)
internal/httpapi/ REST + MCP server. Chi router. OpenAPI-generated types from
internal/httpapi/gen/api.gen.go. Strict server in impl.go.
internal/mcp/ MCP tool implementations (get_entity, search_knowledge, etc.)
internal/db/ Connection pool (pool.go), seed ingestion (seed.go), DB→YAML
export (export.go), type hierarchy (typetree.go)
internal/db/queries/ SQL query files → sqlc generates internal/db/sqlcgen/
internal/scheduler/ Observe loop: probes, signals, check_defs
internal/actuator/ SSH execution with circuit breaker + retry
internal/learning/ Pattern extraction, anomaly detection
internal/notifier/ Matrix notification + approval token generation
internal/policy/ Risk classifier (read policy.yaml → classify action)
internal/secrets/ Backend abstraction: Infisical (primary) + SOPS (fallback)
internal/domain/ Core types: entities, approvals, executions, signals, patterns
internal/ontology/ Type hierarchy validation, relationship checks
internal/knowledge/ Knowledge YAML seed ingestion
internal/config/ Config loading from env vars
web/ Control-room SPA (Svelte 5) — standalone static build, not
embedded in the oikos binary (plans/2026-07-12-wails-desktop-app.md)
api/openapi.yaml REST API contract. Source of truth for endpoints.
api/codegen.yaml oapi-codegen config → generates internal/httpapi/gen/
migrations/ Forward-only SQL. Format: NNN_name.up.sql. No down migrations.
seeds/ Bootstrap YAML. ontology.yaml, inventory.yaml, policy.yaml,
knowledge.yaml. Regenerated from DB via oikos export.
compose/ Dockerfiles. oikos/ (2-stage, Go only — SPA is built/deployed
separately), nomos/ (distroless).
Caddy config at compose/caddy/Caddyfile.oikos.
scripts/ Deploy, rollback, watchdog, verification, cutover checklist.
checks/ Host health-check scripts run over SSH by the scheduler.
tools/ Client auto-setup scripts (checks).
nomos/ Nomos config.yaml, SOUL.md, skills.
.agents/ Agent instruction files, domains, shared conventions, skills.
plans/ Design documents. active/ + done/.
docs/adr/ Architecture decision records. Numbered, prefix-sorted.
```
## Development loop
```bash
# Start dependencies
make dev
# Generate code after API/SQL changes
make generate
# Build
make build
# Run tests
make test # all unit tests
make test-db # integration tests (needs compose Postgres)
# Lint
make lint
# CI drift guard (run before commit)
make generate-check
```
## Adding a feature or phase
Oikos features follow a phase model (read [OIKOS.md](../OIKOS.md) for the
current phase status). To add a new capability:
1. **ADR first.** Write an architecture decision record in `docs/adr/` with
the next sequence number. Document the decision, context, alternatives
considered, and consequences.
2. **Plan.** If the change is non-trivial, create a plan in `plans/` following
the template in [page-templates.md](../shared/page-templates.md).
3. **Schema.** If the feature needs new DB tables, write a forward-only
migration in `migrations/`. Use `IF NOT EXISTS` for idempotency.
4. **API.** If the feature exposes endpoints, define them in
`api/openapi.yaml` first, then run `make generate`, then implement.
5. **Domain.** Add types to `internal/domain/` before adding logic.
6. **Tests.** Write tests alongside implementation. Integration tests go in
`*_test.go` in the relevant package, using the compose Postgres.
7. **Policy.** If the feature introduces new mutation types, update
`seeds/policy.yaml` and the classifier in `internal/policy/`.
8. **Run `make generate-check`** before commit to ensure generated code is
current.
## SQL conventions
- Queries live in `internal/db/queries/*.sql` with `-- name: FuncName :exec`
annotations for sqlc. Generated code in `internal/db/sqlcgen/` — never
hand-edit. Call via `sqlcgen.New(pool).QueryName(ctx, params)`.
- **sqlc is the default** for all DB access. Raw `pool.Query/Exec` with inline
SQL is a documented carve-out for cases sqlc can't express: `LISTEN`/`NOTIFY`,
dynamic WHERE-clause builders, `blast_radius()` (opaque return type), and
`COPY`. All other DB access should go through sqlc queries.
- Use `pgx/v5` driver. UUIDs use `pgtype.UUID`, timestamps use `time.Time`
- CTEs for graph traversals (blast radius, dependency chains)
- CAGGs and retention policies for TimescaleDB hypertables
- FTS via `tsvector` + `tsquery` for knowledge search (migration 011)
## OpenAPI codegen
- Config: `api/codegen.yaml`. Uses `oapi-codegen/v2` with Chi server template
- Generated output: `internal/httpapi/gen/api.gen.go` — never hand-edit
- Strict server interface: `api.gen.go` generates the `StrictServerInterface`;
implement it in `internal/httpapi/impl.go`
- Problem+JSON errors via `internal/httpapi/problem.go` — RFC 9457 format
- Cursor pagination, If-Match/ETag, idempotency keys, SSE streaming
- **Non-OpenAPI routes carve-out:** ~10 routes are registered manually on
the chi router in `internal/httpapi/server.go` rather than generated from
`openapi.yaml`. These fall into three categories:
1. **Auth/infra** (`/healthz`, `/api/v1/auth/oidc-*`, `/oidc-callback`) —
must bypass the auth middleware or aren't JSON API endpoints.
2. **SSE override** (`/api/v1/events/stream`) — in the spec but
re-registered manually because the strict handler can't `Flush()` per
event.
3. **Ad-hoc aggregations** (`/api/v1/knowledge/recent`,
`/api/v1/knowledge/content/{id}`, `/api/v1/activity/recent`,
`/api/v1/activity/session/{id}`, `/api/v1/learning/timeline`,
`/api/v1/learning/trend`) — return derived/aggregate shapes that don't
map cleanly to a schema type. If one of these stabilizes, promote it
to `openapi.yaml` with a proper schema and migrate the `serve*`
function to a strict handler.
The full list with reasons is in the "Non-OpenAPI routes" comment block
at the top of `NewHandler` in `server.go`.
## Testing philosophy
- **Race detector always on.** `make test` runs `go test -race -cover ./...`
- **Integration tests** use the compose Postgres. Run with `make test-db`.
Each test creates + tears down its own schema namespace.
- **Coverage gates** in CI: policy + learning ≥ 80%, others ≥ 60%
- Tests use `testing.T` directly, no assertion library
- Table-driven tests for validation and classification logic
## Migration rules
- Forward-only. No down migrations (ADR 0008)
- Idempotent: use `IF NOT EXISTS`, `DO $$ BEGIN ... END $$` blocks
- Sequence numbers are sequential integers (001, 002, ...)
- Each migration file is `NNN_name.up.sql`
- Migrations are embedded in the binary via `migrations/embed.go`
## Seed files
- `seeds/ontology.yaml` — entity types, relationship types, lifecycles
(validated against schema in `internal/ontology/`)
- `seeds/inventory.yaml` — hosts, services, entities (the topology)
- `seeds/policy.yaml` — risk classes, approval rules, autonomy settings
- `seeds/knowledge.yaml` — documents, investigations, runbooks (DB is source
of truth; this file is the DR export)
- After DB changes via the API, run `make export` to regenerate seeds
## Secrets handling
- No secrets in code, config, or commits
- Dev secrets in `.env` (gitignored)
- Primary: Infisical (`internal/secrets/infisical.go`)
- Fallback: SOPS + age (`internal/secrets/sops.go`)
- Backend interface: `internal/secrets/backend.go`
- Machine identities via Infisical UniversalAuth
- In-memory cache with TTL for performance
## Staging and deployment
- CI pipeline: `.gitea/workflows/ci.yml` — lint, vet, vulncheck, test, docker build
- Deploy: `scripts/deploy.sh` — git pull → docker build → compose up → health check
- Watchdog: `scripts/watchdog.sh` — 2-minute cron, Matrix alert on failure
- Rollback: `scripts/rollback.sh` — checkout SHA + pg_restore
- Cutover checklist: `scripts/cutover-checklist.md`
## Writing conventions
Apply [writing-style.md](../shared/writing-style.md) for all committed prose.
Terse, reference-style, no marketing vocabulary. Code comments explain intent
and trade-offs, not mechanics.
Apply [caveman.md](../shared/caveman.md) for agent communication. The caveman
standard applies to agent *chat responses*, not committed documentation.
## Skills
Agent skills live under `.agents/skills/<name>/SKILL.md`. Each skill has a
frontmatter description that tools match against tasks. To add a skill:
1. Create `.agents/skills/<name>/SKILL.md`
2. Include frontmatter with description field
3. Document the procedure following the runbook template
4. Reference relevant files, commands, and policy classes
Skills that require code (e.g. linting) may include companion scripts in the
same directory.
## When in doubt
- Query MCP tools first (search_knowledge, get_entity)
- Read the relevant ADR in `docs/adr/`
- Grep the codebase: `rg <symbol> internal/`
- Check `plans/` for in-progress work that may conflict
- Classify any new mutation against `seeds/policy.yaml` before suggesting it

View File

@@ -0,0 +1,93 @@
# Knowledge domain — schema
The knowledge domain is the durable, authoritative current-state documentation of the homelab:
narrative for every node and cross-cutting system, synthesized from live state and evidence. It
answers "what exists and how does it work right now."
It follows the [LLM Wiki layer model](../../shared/llm-wiki.md) and the
[writing-style](../../shared/writing-style.md) and [page-templates](../../shared/page-templates.md)
rules.
## Source of truth — the database
Per ADR 0003, the Postgres database is the single source of truth for all structured data **and**
narrative knowledge. The narrative/substrate split of the Python era is gone: the DB holds both the
structured graph (entities, relationships, status, metrics) and the narrative layer (documents,
investigations, runbooks) in the `knowledge_entities` table.
| Concern | Where it lives | How it gets there |
|---------|----------------|-------------------|
| Knowledge content — documents, investigations, runbooks | `knowledge_entities` table (rows linked to `entities` via `documents` / `about` edges) | Seeded from `seeds/knowledge.yaml` at deploy; mutated at runtime via the API |
| Seed manifest (bootstrap + DR) | `seeds/knowledge.yaml` | Hand-edited or regenerated; ingested idempotently (content-hashed via `seed_versions`) |
| Structured graph — hosts, services, entity types, relationships | `entities`, `relationships`, `entity_types` tables | Seeded from `seeds/{ontology,inventory}.yaml`; mutated via API/MCP |
| Archived narrative wiki (read-only history) | `archive/knowledge/` | Frozen 2026-07-07 when the DB became source of truth |
### Seed ingest
`seeds/knowledge.yaml` has three top-level lists — `documents`, `investigations`, `runbooks` — each
entry carrying `slug`, `title`, `content` (markdown), and tags. `internal/knowledge/seed.go`
ingests each entry by:
1. `getOrCreateEntity` — ensures the slug exists in `entities` (type `document` / `investigation` /
`runbook`).
2. `upsertKnowledgeEntity` — writes the markdown body into `knowledge_entities`, keyed by
`content_hash` so re-ingest is a no-op when nothing changed.
3. `createEdge` — links the knowledge entity to its subject(s) via `documents` (for `document`) or
`about` (for `investigation`) edges. Runbooks bind to an `entity_type` via `applies_to_type`
rather than to a single entity.
### Runtime mutation
Agents register or update knowledge through the API, not by editing the seed:
- `POST /api/v1/knowledge/{entity_slug}` — upsert a document/investigation on an entity
(`upsert_knowledge` MCP tool).
- `update_entity_attributes` — merge a discovered fact (IP, version, port) into an entity.
- `create_relationship` — record a discovered edge (`depends-on`, `hosts`, `routes-to`).
> **Export gap.** `oikos export` regenerates `seeds/{ontology,inventory,policy}.yaml` from the DB
> for version control, but **not** `seeds/knowledge.yaml`. Knowledge added via the API today lives
> only in the DB until someone hand-edits the seed. Tracked as a follow-up.
## Knowledge kinds
- **Documents** (`document` entities, linked via `documents` edges) — node and cross-cutting
narrative pages. Carry `at_glance` (structured attributes) and a parsed `changelog`. Follow the
container / cross-cutting templates in [page-templates.md](../../shared/page-templates.md).
- **Investigations** (`investigation` entities, linked via `about` edges) — incident evidence,
written once at incident time. Sections: `## Summary`, `## Timeline`, `## Root cause`,
`## Mitigations applied`, `## Open questions`.
- **Runbooks** (`runbook` entities, bound by `applies_to_type`) — repeatable procedures. Carry
`risk_class` and a JSON-schema-validated `procedure`. **Runbooks also live as `SKILL.md` files
under `.agents/skills/<name>/`** — the DB row is the policy/lifecycle framing, the SKILL.md is
the executable procedure the agent loads. See
[the operations schema](../operations/schema.md).
## The two logs
- The per-document **`## Changelog`** records infrastructure changes to that node. Keep the
`### YYYY-MM-DD — title` shape so the parsed `changelog` field stays structured.
- **`archive/knowledge/log.md`** is the append-only record of *documentation-maintenance*
operations on the legacy wiki (restructures, source ingests, lint sweeps):
`## [YYYY-MM-DD] <op> | <summary>`. It is frozen with the rest of `archive/knowledge/`; new
doc-maintenance operations are recorded in the DB audit trail instead.
## Querying knowledge
Use MCP, not grep:
- `search_knowledge(query)` — ILIKE search over documents, investigations, and runbooks in
`knowledge_entities`.
- `get_entity_knowledge(entity_slug)` — every document, investigation, and runbook linked to one
entity, in one call.
- `get_entity(slug)` / `get_relations(entity)` — the structured graph around an entity.
Grep the clone only when MCP is unreachable, and prefer `archive/knowledge/` for historical
narrative (it is not updated when the DB changes).
## Same-session update rule
A change to a node updates the DB in the same session — the entity's attributes, the relationships
that reference it, and any document whose `at_glance` or changelog should reflect the new state. See
[page-templates.md](../../shared/page-templates.md#same-session-update-rule) for the legacy wiki
equivalent (now scoped to `archive/knowledge/` history).

View File

@@ -0,0 +1,58 @@
# Operations domain — schema
The operations domain holds the procedural and time-stamped documentation: runbooks (repeatable
procedures), investigations (incident evidence), and plans (design docs for non-trivial work). It
follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice
exception.
Where each kind lives: runbooks are skills under [`.agents/skills/`](../../skills/); operator
reference (command cheatsheet, enrollment, Nomos agent) lives in
[`.agents/operations/`](../../operations/); investigations are `investigation` entities in the DB
(historically `archive/knowledge/sources/investigations/`); plans stay in the repo-root `plans/`
folder (below).
## Plans always live in `plans/`
**Any plan or design doc for the Homelab is written into the repo `plans/` folder as
`plans/YYYY-MM-DD-slug.md` — never a scratch path, an agent-private plan location, or a chat
message.** An agent drafting a plan:
1. Writes the file under `plans/` using the plan template in [page-templates.md](../../shared/page-templates.md).
2. Lists it in `plans/index.md`.
3. On completion, moves it to `plans/done/` and updates the index status.
This is the single source for homelab design intent; keeping it in-repo means the plan is
versioned, reviewable, and reachable by MCP `search_knowledge` like any other doc.
## Runbooks
Repeatable procedures are skills — one folder per skill at `.agents/skills/<name>/SKILL.md`, with
YAML front-matter that the Oikos policy and lifecycle machinery reads:
```yaml
---
name: <name>
risk_class: read_only | reversible_low | config_mutation | destructive
inputs: [<param>, ...]
verification: "<shell expression that proves success>"
docs_update_checklist: [<doc artifacts to update>]
transition: "<from> -> <to>" # only for lifecycle runbooks
---
```
`risk_class` values and the lifecycle `transition` states must match
[`seeds/policy.yaml`](../../../seeds/policy.yaml) and [`seeds/ontology.yaml`](../../../seeds/ontology.yaml).
## Investigations
Incident records are `investigation` entities in the DB, linked to the entities they implicate via
`about` edges. They are **evidence sources** — written once at incident time, then back-linked from
the changelogs of the nodes they implicate. Sections: `## Summary`, `## Timeline`, `## Root cause`,
`## Mitigations applied`, `## Open questions`. The legacy file-based investigations live at
`archive/knowledge/sources/investigations/` (frozen 2026-07-07); new investigations go in the DB.
## The operations log
`plans/log.md` is the append-only record of documentation operations on plans
(`## [YYYY-MM-DD] <op> | <summary>`), distinct from the DB audit trail. The legacy
`archive/knowledge/log.md` is frozen with the rest of the archived wiki.

View File

@@ -0,0 +1,123 @@
# Agent enrollment — operational notes
**For the actual enrollment flow, see [CLIENTS.md](../../CLIENTS.md#enrollment)
— it's the current, authoritative version.** This page used to duplicate
that flow in more detail, describing a `homelab` CLI-based two-step
ceremony (`homelab client add` reserves an inventory slot → client
bootstraps → operator finalizes the pubkey). That CLI and that flow don't
exist anymore — enrollment today is one shot: `bootstrap.sh` calls
`POST /api/v1/clients/enroll` directly and gets back an age keypair +
Infisical identity in the same response. What's left here is the handful
of things that are still true and weren't already covered elsewhere.
## Prerequisites
| Requirement | Why | How to check |
| --- | --- | --- |
| Hostname matches an entry in `inventory.yaml` | `EnrollClient` looks up the entity by slug derived from hostname; it must exist in `planned`/`provisioning` state. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird) **or** on the LAN | enrollment validates mesh IP against expected subnets | `netbird status` |
| `curl`, `jq`, `age`, `python3` | bootstrap preflight (`bootstrap.sh:100`) — auto-installed on Fedora/RHEL/Debian/Ubuntu/macOS if missing | `command -v curl jq age python3` |
| Can resolve `*.hubris.network` | bootstrap calls the Oikos API and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` |
### Hostname mismatch is the most common bootstrap failure
If the entity for your hostname doesn't exist yet (in `planned` or
`provisioning` state), enrollment 4xxs. Two fixes:
- **Rename the host** to match an existing planned entity:
`sudo hostnamectl set-hostname <inventory-name>` (Linux) or System
Preferences → Sharing (macOS), then re-run.
- **Add/rename the inventory entry**: edit `seeds/inventory.yaml`, ingest
via `oikos seed` (or the equivalent MCP/API entity-creation path), then
re-run bootstrap.
### Networking prerequisites (Netbird, DNS, SSH key distribution)
Migrated to a runbook in the knowledge base — query
`search_knowledge("netbird mesh dns")` or `get_entity_knowledge`, or ask
Nomos. Covers: getting onto the Netbird mesh (interactive OIDC vs.
setup-key), why OIDC login can fail from off-mesh, split-horizon DNS
options, and distributing a new workstation's SSH pubkey across the fleet
via `ssh/deploy-keys.sh`.
## Claude Code permissions for fleet ops
By default Claude Code's auto-mode classifier asks for confirmation on every
ssh into the mesh. Pre-authorize the common fleet ssh pattern by adding to
`~/.claude/settings.json`:
```json
{
"permissions": {
"defaultMode": "auto",
"allow": [
"Bash(ssh -p 22022 *)"
]
}
}
```
This covers any ssh to a mesh peer on the homelab netbird port, scoped tight
enough that the classifier doesn't gate it but loose enough to handle the
variety of arguments.
## Open questions (not verified against current architecture — don't
guess these from the old flow)
The old two-step ceremony had answers for these; the current one-shot
`/api/v1/clients/enroll` flow may handle them differently and this hasn't
been re-verified:
- **Removing a client.** No current equivalent confirmed for the old
`homelab client remove` (inventory removal + secret re-keying + key
revocation). Likely maps to an entity lifecycle transition
(`.agents/skills/lifecycle-deprecate-node/` or `lifecycle-destroy-node/`)
but those skills reference the same dead CLI and need their own check.
- **Granting a secret to an already-enrolled client.** The old flow
hand-edited `.sops.yaml` `creation_rules` + `sops updatekeys`. Given
Infisical is now the primary secrets backend (SOPS is the DR fallback),
the current mechanism is probably Infisical-side, not a `.sops.yaml` edit
— not confirmed.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| Enrollment 404s / entity not found | Hostname doesn't match a `planned`/`provisioning` inventory entry | See "Hostname mismatch" above |
| `gnutls_handshake() failed` / TLS errors reaching `*.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS instead of the LAN/mesh path | See the networking runbook (split-horizon DNS section) |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
## Changelog
### 2026-07-12 — trimmed to current architecture
Removed everything describing the retired `homelab` CLI-based two-step
enrollment ceremony (now: `CLIENTS.md`'s one-shot flow), the Nous-Hermes/
Goose cross-link (that whole flow was removed the same day), and CLI-syntax
troubleshooting rows with no current equivalent. Migrated the still-true
Netbird/DNS/SSH-distribution content to a knowledge-base runbook rather
than duplicating it here. What's left is genuinely current or explicitly
flagged as unverified. Original ~365-line version is in git history
(`git log -- .agents/operations/agent-enrollment.md`) if any of the removed
detail turns out to still be needed.
### 2026-06-02 — SSH reachability post-bootstrap steps
Added a section covering SSH key generation, pubkey publication,
deployment to hosts, SSH config generation, and LAN IP registration. New
workstations enrolled via this doc automatically join the SSH mesh.
(Superseded 2026-07-12 — migrated to the networking runbook.)
### 2026-05-31 — cross-link to nomos-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment. (Removed
2026-07-12 along with the rest of that flow.)
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
Added three rows to the troubleshooting table covering issues surfaced
during the netbird vanilla migration. (Migrated 2026-07-12 to the
networking runbook.)
### 2026-05-20 — initial page
Captures the enrollment flow validated during Phase 2 of the homelab
context distribution rollout. hubris + LXC 105 (apps) enrolled; first
workstation (republic-laptop) blocked on hostname mismatch, documented
the resolution.

View File

@@ -0,0 +1,93 @@
# Operations cheatsheet
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
## Proxmox CLI
| Command | Use |
| --- | --- |
| `pct list` / `qm list` | List LXC containers / VMs |
| `pct config <id>` / `qm config <id>` | Container / VM config |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
| `pct enter <id>` | Shell into a container |
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
| `pvesm status` | Storage pools status |
| `pvesh get /nodes --output-format json` | Node summary as JSON |
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Nomos cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pveversion` | PVE version |
| `journalctl -u pve-cluster -n 100` | PVE service logs |
## Storage
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
## Reverse proxy
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
## DNS
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
## Web access
- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
## Telemetry quick checks
- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
- `ras-mc-ctl --errors` — full event log
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/2026-04-21-hubris-crash-loop.md))
## Fleet apt operations
**No current CLI equivalent.** `homelab apt-audit`/`apt-upgrade` (dpkg-state
audit, fanned-out apt upgrade with pre-upgrade snapshots) were part of the
retired Python `homelab` CLI and don't have a ported replacement — apt
patching today is ad hoc `run` MCP tool calls per host, without the
audit/snapshot/status wrapping this used to provide. If that wrapping is
still wanted, it needs to be rebuilt (e.g. as a runbook driving `run`, or a
new MCP tool) — see
[runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) for
the dpkg-interrupted recovery procedure specifically.
## Oikos (agent OS layer)
See [OIKOS.md](../OIKOS.md) for the operating model. The `homelab` CLI this
section used to document is retired; the actual current interface is the
MCP tool catalog in [AGENTS.md §3](../../AGENTS.md#3-the-mcp-server) plus
the REST API. Closest current equivalents for what used to live here:
| Old `homelab` command | Current equivalent |
| --- | --- |
| `homelab service <name> explain\|health\|docs\|log` | MCP `explain`, `get_service_status`, `tail_log`, `get_entity_knowledge` |
| `homelab node <name> relations` | MCP `get_blast_radius` |
| `homelab change preflight <service>` | MCP `preflight` |
| `homelab signal list\|ack\|resolve\|mute` | MCP `get_signal_history`, or REST `POST /api/v1/signals/{id}/ack\|resolve\|mute` (the control-room UI's Signals page wraps these) |
| `homelab approval request\|list\|reply\|check` | REST `GET/POST /api/v1/approvals*` (Matrix-delivered via the notifier, or the control-room UI's Operations page) |
| `homelab restart <service> --approval-id <id>` | MCP `run` (policy-gated — auto-executes if read-only/reversible_low, otherwise queues for the same Matrix/UI approval) |
| `homelab decide <action> <entity>` | No direct equivalent — classification now happens inline inside `run`, not as a separate dry-run call |
There is no separately-deployed "Oikos Console" anymore — the control-room
SPA (`web/`) is the operator dashboard, served standalone (see
[plans/done/2026-07-12-wails-desktop-app.md](../../plans/done/2026-07-12-wails-desktop-app.md)).
## Related
- [Hubris host](../../archive/knowledge/hosts/hubris.md)
- [Containers index](../../archive/knowledge/containers/index.md)
- [DNS](../../archive/knowledge/infrastructure/dns.md)
- [Monitoring](../../archive/knowledge/infrastructure/monitoring.md)
- [Auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction

33
.agents/shared/caveman.md Normal file
View File

@@ -0,0 +1,33 @@
# CAVEMAN.md — communication mode for homelab agents
Respond terse like smart caveman. All technical substance stay. Only fluff die.
## Rules
Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact.
Pattern: `[thing] [action] [reason]. [next step].`
Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
## Levels
- **lite** — no filler/hedging. Keep articles + full sentences. Professional but tight.
- **full** (default) — drop articles, fragments OK, short synonyms. Classic caveman.
- **ultra** — abbreviate prose words (DB/auth/config/req/res), strip conjunctions, arrows (X → Y). Code symbols/API names/errors: never abbreviate.
Switch: `/caveman lite|full|ultra`. Stop: "normal mode".
## Auto-Clarity
Drop caveman for: security warnings, irreversible actions, multi-step sequences where fragments risk misread, user confused/repeating. Resume after clear part done.
## Boundaries
Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.
---
Source: https://github.com/JuliusBrussee/caveman
Copy to `~/.nomos/skills/` for Nomos agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.

View File

@@ -0,0 +1,50 @@
# LLM Wiki — the documentation contract
How documentation in this repo is organized. The pattern is the `sources / wiki / index / log`
model: a durable synthesized layer built on top of immutable evidence, with pure-listing indexes and
an append-only operations log.
This contract governs the **narrative layer only**. The machine-readable source of truth — the
Postgres database, bootstrapped from `seeds/` — is not part of the wiki and never moves under it.
See [the knowledge schema](../domains/knowledge/schema.md) for the split, and ADR 0003 for the
DB-native model.
## Layers
- **Source of truth** is the Postgres database. Structured data (entities, relationships, status,
metrics) and narrative knowledge (documents, investigations, runbooks) both live there, in the
`entities` / `relationships` / `knowledge_entities` tables. It is bootstrapped at deploy time from
`seeds/{ontology,inventory,policy,knowledge}.yaml` (idempotent, content-hashed via
`seed_versions`) and mutated at runtime via the API/MCP. `oikos export` regenerates
`seeds/{ontology,inventory,policy}.yaml` for version control.
- **Sources** are immutable raw material: incident records (now `investigation` entities in the DB,
historically `archive/knowledge/sources/investigations/`), external reference docs, and the live
system itself (`pct config`, `docker inspect`). Read them; do not rewrite them into other sources.
- **Wiki** — the synthesized, authoritative current-state layer. Today this is the set of
`document` entities in the DB (one per node and per cross-cutting system), queried via MCP
`search_knowledge` / `get_entity_knowledge`. The legacy file-based wiki is frozen at
`archive/knowledge/{hosts,containers,vms,infrastructure}/` for historical reference only.
- **Index** (`index.md` / folder `README.md`) is a pure listing — every page in scope with a
one-line summary, and nothing else. Anything the section wants to say up front goes into a page
the index lists, not into the index.
- **Log** is append-only, recording *doc-maintenance operations* (restructures, source ingests,
lint sweeps) in single-line format: `## [YYYY-MM-DD] <op> | <summary>`. The active log is the DB
audit trail; `archive/knowledge/log.md` is the frozen legacy equivalent.
## Two logs, kept distinct
- **`## Changelog`** on each node/topic document records *infrastructure* changes to that node. It
is stored as a structured field on the `document` entity — keep the `### YYYY-MM-DD — title`
shape so it parses cleanly.
- **Doc-maintenance logs** record *documentation* operations only. They never duplicate the
infrastructure changelog, which stays authoritative for infra changes with
who/what/risk/approval/verification (now the DB audit trail, formerly `oikos/ledger.py`).
## Rules
- Wiki pages stay short and focused. A page past ~300 lines splits.
- Pages stay flat under their section until there are enough to warrant a sub-group.
- Every page follows [writing-style.md](writing-style.md).
- Plans and design docs always live in the repo `plans/` folder (`plans/YYYY-MM-DD-slug.md`),
listed in `plans/index.md`, moved to `plans/done/` on completion — never a scratch path or a chat
message. See [the operations schema](../domains/operations/schema.md).

View File

@@ -0,0 +1,176 @@
# Page templates for the Homelab Wiki
The structural templates for each page type. Prose voice, vocabulary, and cross-reference rules live
in [writing-style.md](writing-style.md); the layer model (sources / wiki / index / log) lives in
[llm-wiki.md](llm-wiki.md).
## File naming
**Foundational / entry-point files:** ALL-CAPS
- **Root level:** `AGENTS.md`, `README.md` — discovery paths for agents and humans.
- **Agent instruction** (under `.agents/`): `OIKOS.md`, `NOMOS.md` — foundational docs agents read before acting.
- **Reference docs:** `GLOSSARY.md` — lookup reference (like classic repo conventions: LICENSE, CHANGELOG, GLOSSARY).
**Content / narrative pages:** lowercase-with-dashes, date-prefixed as needed
- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from the entity's attributes in the DB (seeded via `seeds/inventory.yaml`).
- **Infrastructure / cross-cutting pages:** `<topic>.md` (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node.
- **Plans / investigations:** `YYYY-MM-DD-<slug>.md` (e.g. `2026-07-05-oikos-prometheus-lxc.md`). Date-sorted; slug is lowercase.
- **Section indices:** `README.md` (lowercase, conventional). Prefer in folders; `index.md` only if both intro prose and listing coexist.
**Skills / runbooks:** special case
- **Folder structure:** `<name>/SKILL.md` where `<name>` is lowercase-with-dashes (e.g. `client-enrollment/SKILL.md`).
- **The filename SKILL.md is always uppercase** — it acts as a signpost so tools and humans instantly recognize it as a skill.
**General rules:** All paths use lowercase letters, numbers, and hyphens (no underscores). Uppercase is reserved for foundational docs (entry points + instruction) and filenames that signify document type (SKILL.md, GLOSSARY.md, etc.).
## Voice
Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. Full rules in
[writing-style.md](writing-style.md).
## Page templates
### Container page (`containers/<id>-<name>.md`)
```markdown
# <id> — `<name>`
One-sentence purpose.
## At a glance
- **Hostname:** `<name>`
- **IP:** `192.168.8.x`
- **Privilege:** privileged | unprivileged
- **Resources:** N cores / M GiB RAM / D GiB rootfs
- **Mounts:** `/mnt/library``/mnt/library` (if any)
- **Public hostname:** `<sub>.hubris.network` (if proxied)
## Role
What it does, what it talks to.
## Service / port map
| Service | Listen | Notes |
## Storage / config paths
## Auto-deploy
(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md)
## Related
- [Caddy](121-caddy.md) (if proxied)
- [DNS](../infrastructure/dns.md) (if has subdomain)
- [Authentik](124-authentik.md) (if SSO)
- ...
## Changelog
### YYYY-MM-DD — short title
What changed, why, link to investigation if any.
```
### Cross-cutting page (`infrastructure/<topic>.md`)
```markdown
# <Topic>
One-sentence summary.
## Why
Design rationale — what it replaces, what it solves.
## Components
Where it runs, what files matter.
## How to apply / use
Recipes.
## Gotchas
## Related
Links to nodes that host or depend on this.
## Changelog
```
### Plan (`plans/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Goal
What this change achieves and why.
## Current topology / state
Diagram or description of what exists now.
## Target topology / state
What it looks like after.
## Pre-flight checklist
## Step-by-step procedure
## Verification
## Post-migration
Changelog entries to write, index status to update.
```
### Investigation (`investigation` entity in the DB; historically `archive/knowledge/sources/investigations/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Summary
1-3 sentences.
## Timeline
## Root cause
## Mitigations applied
## Open questions
```
## Linking discipline
- Every container page links to every cross-cutting page it participates in.
- Every cross-cutting page lists the nodes that participate.
- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages.
## Changelog hygiene
- Reverse-chronological (newest first).
- One entry per discrete change, even if you make several in one day.
- If a change spans nodes, repeat the entry on each affected page (different perspective is fine).
- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them.
## Same-session update rule
When you make a change to a node — migrate an LXC, update an IP, change a
mount, deploy a new service — **update the DB and every relevant doc page in
the same session.** A change that touches a container must also update:
- The `entities` / `relationships` rows for the node (via the API/MCP) —
this is the source of truth
- The `document` entity's `at_glance` and `## Changelog` for the container
- The `containers/index.md` table in the archived wiki (IPs, host, mounts,
status) — historical reference, update for consistency where still consulted
- The `README.md` table (if the change affects listed columns)
- The Caddy page site list (if the change affects `*.hubris.network` routing)
- The DNS / ingress infrastructure pages (if the change affects routing)
- The `hosts/{hubris,strong}.md` host page (if container count changes)
The pattern of updating only one page and leaving stale references on others
is a bug. If you're doing a multi-step migration, document the intermediate
state with a changelog entry that says "pending — will finalize after Phase
N."
This rule is why Phase 2 of the strong migration (2026-07-05) caused
widespread stale data: individual container pages were updated in the
changelog but never had their At-a-glance sections, IPs, mount paths, or
host attribution updated. Don't repeat that.

View File

@@ -0,0 +1,75 @@
# Writing Style
Write like a technical reference, not a marketing page. Every sentence conveys new information.
These rules govern **committed documentation** — wiki pages, READMEs, schemas, skills, `AGENTS.md`,
plans, investigations, and code comments. They are separate from [caveman.md](caveman.md), which
governs an agent's *chat responses*; the two do not conflict.
New or rewritten pages follow these patterns from day one. Existing pages get updated the next time
they are touched.
## Vocabulary — never use these
- Significance puffers: "pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament", "paramount", "invaluable".
- Analytical verbs: "delve", "leverage", "utilize", "facilitate", "foster", "showcase", "underscore", "streamline", "harness".
- Poetic nouns: "tapestry", "landscape" (figurative), "realm", "paradigm", "ecosystem" (figurative), "journey" (figurative), "nexus", "cornerstone".
- Promotional adjectives: "robust", "seamless", "innovative", "cutting-edge", "meticulous", "holistic", "comprehensive".
- Opening crutches: "In today's world", "In the ever-evolving landscape of", "It's worth noting that", "It is important to note that".
Use short, common words: "use" not "utilize", "help" not "facilitate", "show" not "demonstrate".
## Voice
Describe what systems do and how they work.
- **Reference prose** (node pages, cross-cutting infrastructure descriptions, `## Role`, `## Why`,
`At a glance`) is third-person: state facts about the system, not instructions to a reader.
- **Recipes, runbooks, and skills** are the exception: second-person imperative is allowed and
preferred where it makes a procedure clearer ("Edit the Caddyfile, commit + push", "Verify with
`dig +short`"). This matches how the operator actually works. The vocabulary, structure, and
cross-reference rules below still apply.
## Page shape
Every doc-level page follows the same shape so a reader scans it in one pass.
1. **One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`.
2. **Opening definition.** First paragraph, 13 sentences, says what the thing is. No motivation, no marketing, no setup.
3. **Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md).
4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is stored as a structured field on the `document` entity in the DB; keep the `### YYYY-MM-DD — title` shape so it parses cleanly.
5. **Related links** only at the bottom, only when a reference cannot be woven inline.
## Section indexes (folder READMEs)
A folder's `README.md` opens with a 13 sentence prose intro that says what the section covers, then
a single navigation table — `| Document | What it covers |` — and nothing else. No stale counts, no
duplicated prose, no narrative between the intro and the table.
## Structure rules
- Make every sentence information-dense. Cut filler, qualifiers, and setup phrases. Lead with the concrete fact or action, not why it matters.
- No participial tack-ons (", highlighting the importance of…"). If the clause adds information, make it a separate sentence.
- **No meta-commentary about the content itself.** Do not narrate the page's own structure or linking strategy.
- Prefer **tables** for enumerable items with internal structure (service/port maps, field lists, status grids). Reserve bullets for short non-structured lists.
- Use the **bold-leading-phrase pattern** for structured points: `**Read-only by construction.** The MCP server never mutates state.` — a bold noun phrase, a period, then the explanation.
- When enumerating across services or nodes, give each its own `###` sub-section or a table row, not one run-on paragraph.
- Use backticks for code, paths, hostnames, and file names (`seeds/inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology.
- Use `>` blockquotes for caveats and gaps that interrupt the main flow: `> **Outstanding gap.** DNS-vs-inventory drift check not yet wired.` One thought per blockquote.
## Diagrams
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` in the archived wiki was generated by the retired `oikos/gen-topology.py`; the DB-native equivalent is a future task — do not hand-edit the archived file expecting it to regenerate.
- ASCII box diagrams are fine for small shape diagrams; keep them to one screen.
## Sourcing and cross-references
- **Factual discipline.** Every claim is grounded in a cited source, an adjacent linked page, or a directly observable fact (`pct config`, `docker inspect`, running config). Do not write sentences that sound sourced but are inference. When docs disagree with live state, fix the doc and note it in the changelog.
- **One-sided cross-references.** When two pages relate, the link lives in the page where the connection makes organizational sense. Do not add a back-pointer unless that direction also carries content the reader needs.
- **Cross-references are content, not catalog.** Inline links arise from the surrounding prose; the linked page must be needed to understand the current sentence. A bottom-of-page "Related" list is the fallback, not the default.
- Pages link with standard relative markdown links (e.g. a container page links to `../infrastructure/dns.md`), forming a navigable graph. Orphans are a bug.
## Code comments and commit/PR prose
- Comments explain intent, trade-offs, or constraints the code cannot convey. No diff narration, no type restatement, no section-divider comments.
- Commit messages and PR descriptions are problem → change → risk → verification, not a file-by-file diff restatement.
- The banned vocabulary applies the same way in comments and commit messages.

View File

@@ -0,0 +1,50 @@
---
name: client-enrollment
risk_class: config_mutation
inputs: [hostname, kind, role]
verification: "MCP whoami(hostname) shows the entity active"
docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
---
# Client enrollment
Goal: bring a new host (workstation, LXC, VM) into inventory and the
secrets model, with mesh membership only where it's actually needed. See
[CLIENTS.md](../../../CLIENTS.md#enrollment) for the actual current
flow and [operations/agent-enrollment.md](../../operations/agent-enrollment.md)
for operational notes; this runbook is the risk/lifecycle framing.
1. The entity must exist in `planned`/`provisioning` state before the new
host can self-enroll — add a `hosts.<name>:` block to
`seeds/inventory.yaml` and `oikos seed` to ingest it (lifecycle
`planned``provisioning`, per
[seeds/ontology.yaml](../../../seeds/ontology.yaml)).
2. Netbird join is **optional, not a required step** — only needed for
hosts that must be reachable off-LAN (workstations that roam, e.g.
`mac-mini`). A node reachable on the household LAN (192.168.8.0/24 —
most LXCs/VMs) doesn't need it. Skip for LAN-only nodes; do it
(out-of-band, console or setup key) only for hosts that need
independent off-LAN reachability.
3. On the new host: run `bootstrap.sh`. This calls
`POST /api/v1/clients/enroll`, which validates the entity exists and
the mesh IP is in an expected subnet, then returns an age keypair and
Infisical machine identity in one response — provisions
`/etc/age/key.txt`, `/etc/infisical/identity`, and the context poller.
4. **Known gap, confirmed 2026-07-12: `provisioning → active` has no
working path.** `EnrollClient` (`internal/httpapi/impl.go`) sets the
entity's state to `provisioning`, never `active`. `bootstrap.sh` prints
`POST /api/v1/clients/ws:$HNAME/activate` as the next step, but that
route doesn't exist — `api/openapi.yaml` only has `/clients/enroll`,
`/clients/{slug}/context`, `/clients/{slug}/secrets`. Until this is
fixed (add the route, or use the generic entity PATCH to flip `state`),
a freshly-enrolled client is stuck in `provisioning` — MCP `preflight`
and policy's `lifecycle_overrides` for `provisioning` still apply, but
nothing transitions it onward automatically.
5. Verify: MCP `whoami(hostname)` shows the entity in `active` state with
its peers and health.
Docs-update checklist: if the new host is an LXC/VM, add its narrative
page under `containers/` or `vms/` and set `doc_page` in its inventory
entry (host-level cards don't have a `doc_page` field yet — services do;
narrative pages are still found via the generated `see_also` in
`inventory.yaml`).

View File

@@ -0,0 +1,33 @@
---
name: config-change-deploy
risk_class: config_mutation
inputs: [service_name, change_description]
verification: "curl -sf <service_url> (or MCP get_service_status)"
docs_update_checklist: [doc_page, changelog]
---
# Config change + deploy
Goal: change a tracked config repo (Caddy, Gitea customizations, an app's
own repo) and get it live, safely.
1. MCP `preflight` — current health, the service's `config_repo`, its
risk class, and the verification command to run after. If risk class
requires approval (`config_mutation` or `destructive`), stop and get
operator sign-off before editing — see `seeds/policy.yaml`.
2. Clone/pull the `config_repo` (never edit the backend's working tree
directly — tracked configs change by commit + push, per
[OIKOS.md](../../OIKOS.md) conventions).
3. Make the change, commit, push to `main`.
4. The Gitea webhook fires the deploy pipeline for that repo (see
[infrastructure/auto-deploy.md](../../../archive/knowledge/infrastructure/auto-deploy.md) for
the exact receiver/reload for this service).
5. Run the preflight's verification command. If it fails, check
MCP `tail_log` for the reload/restart error.
6. No manual record-keeping step needed — mutations made through the API
(e.g. via the `run` MCP tool) are recorded automatically in the
`audit_log` table.
Docs-update checklist: update the service's `doc_page` if the change
alters its behavior, ingress route, or ownership; add a changelog entry
if the page has one.

View File

@@ -0,0 +1,25 @@
---
name: docs-lint
risk_class: read_only
inputs: [paths]
verification: "python3 .agents/skills/docs-lint/lint.py"
docs_update_checklist: []
---
# Docs lint
Check committed documentation against the mechanical rules in
[writing-style.md](../../shared/writing-style.md): banned vocabulary and broken relative markdown
links. Prose-voice rules are not machine-checkable — those stay a review responsibility.
Run from the repo root:
python3 .agents/skills/docs-lint/lint.py # default: knowledge/ .agents/ operations/ investigations/ plans/
python3 .agents/skills/docs-lint/lint.py archive/knowledge/containers/104-gitea.md
Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary
list mirrors `writing-style.md`; update both together if the standard changes.
> **Known baseline.** `archive/knowledge/archive/knowledge/containers/101-jellyfin.md` links into a sibling repo
> (`devops/homelab-authentik-admin`) that this checkout does not contain — expected, not a bug.
> Any other broken link is a real regression; investigate before dismissing it as baseline noise.

View File

@@ -0,0 +1,186 @@
#!/usr/bin/env python3
"""Lint committed docs against .agents/shared/writing-style.md.
Checks:
1. Banned vocabulary (significance puffers, analytical verbs, poetic nouns,
promotional adjectives, opening crutches).
2. Broken relative markdown links.
3. Plan status consistency (status vs location vs index).
Prose-voice rules are not machine-checkable; this covers the parts that are.
Run from the repo root: python3 .agents/skills/docs-lint/lint.py [paths...]
Exit 1 if any violation is found.
"""
import os, re, sys, glob
BANNED = [
"pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament",
"paramount", "invaluable", "delve", "leverage", "utilize", "facilitate", "foster",
"showcase", "underscore", "streamline", "harness", "tapestry", "realm", "paradigm",
"nexus", "cornerstone", "robust", "seamless", "innovative", "cutting-edge",
"meticulous", "holistic", "comprehensive", "in today's world",
"it's worth noting", "it is important to note",
]
BAN_RE = re.compile(r'(?<![\w-])(' + "|".join(re.escape(w) for w in BANNED) + r')(?![\w-])', re.I)
LINK = re.compile(r'\]\(([^)]+)\)')
def iter_md(paths):
for p in paths:
if os.path.isfile(p) and p.endswith(".md"):
yield p
for root, dirs, files in os.walk(p):
dirs[:] = [d for d in dirs if d not in (".git", "node_modules")]
for f in files:
if f.endswith(".md"):
yield os.path.join(root, f)
def check_plans():
"""Check plan status consistency: active plans with 'Done' status, files
missing from index, dangling index entries, done files with wrong status."""
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
plans_dir = os.path.join(REPO, "plans")
done_dir = os.path.join(REPO, "plans", "done")
index_path = os.path.join(plans_dir, "index.md")
if not os.path.exists(index_path):
return 0
violations = 0
STATUS_RE = re.compile(r'^\*\*Status:\*\*\s*(.+)', re.I)
# Parse index.md for active and done entries
active_files = set()
done_files = set()
current_section = None
with open(index_path) as f:
for line in f:
if line.startswith("## Active"):
current_section = "active"
continue
if line.startswith("## Done"):
current_section = "done"
continue
if current_section == "active":
m = re.search(r'\]\(([^)]+)\)', line)
if m:
active_files.add(m.group(1))
elif current_section == "done":
m = re.search(r'\]\(([^)]+)\)', line)
if m:
done_files.add(m.group(1))
# Active plans on disk (not in done/, not index.md)
disk_active = set()
for f in glob.glob(os.path.join(plans_dir, "*.md")):
name = os.path.basename(f)
if name == "index.md":
continue
disk_active.add(name)
# Done plans on disk
disk_done = set()
if os.path.isdir(done_dir):
for f in glob.glob(os.path.join(done_dir, "*.md")):
disk_done.add("done/" + os.path.basename(f))
# Check 1: active plans on disk whose internal status is Done/Implemented/Complete
for name in disk_active:
fpath = os.path.join(plans_dir, name)
with open(fpath) as f:
for line_num, line in enumerate(f, 1):
if line_num > 5:
break
m = STATUS_RE.match(line)
if m:
status = m.group(1).strip().lower()
done_keywords = ["done", "implemented", "complete", "completed"]
if any(status.startswith(kw) for kw in done_keywords):
print(f"{fpath}:{line_num}: status '{m.group(1).strip()}' — file is in plans/ but appears done; move to done/")
violations += 1
break
# Check 2: active plans on disk not in index
for name in sorted(disk_active):
if name not in active_files:
fpath = os.path.join(plans_dir, name)
print(f"{fpath}:1: not listed in plans/index.md Active table")
violations += 1
# Check 3: done plans on disk not in index
for name in sorted(disk_done):
if name not in done_files:
fpath = os.path.join(REPO, "plans", name)
print(f"{fpath}:1: not listed in plans/index.md Done table")
violations += 1
# Check 4: index entries with no file on disk
for name in sorted(active_files):
if name not in disk_active:
print(f"plans/index.md: active entry '{name}' — file not found on disk")
violations += 1
for name in sorted(done_files):
if name not in disk_done:
print(f"plans/index.md: done entry '{name}' — file not found on disk")
violations += 1
# Check 5: files in done/ whose internal status doesn't say Done
for name in disk_done:
fpath = os.path.join(REPO, "plans", name)
with open(fpath) as f:
found_status = False
for line_num, line in enumerate(f, 1):
if line_num > 5:
break
m = STATUS_RE.match(line)
if m:
found_status = True
status = m.group(1).strip().lower()
if not status.startswith("done"):
print(f"{fpath}:{line_num}: status '{m.group(1).strip()}' — file is in done/ but status is not 'Done'")
violations += 1
break
if not found_status:
print(f"{fpath}:1: file is in done/ but has no Status header")
violations += 1
return violations
def main(argv):
paths = argv or ["knowledge", ".agents", "operations", "investigations", "plans"]
violations = 0
if "plans" in paths or any(p.startswith("plans") for p in paths):
violations += check_plans()
# The style guide and this skill enumerate the banned words by definition.
ban_exempt = ("shared/writing-style.md", "skills/docs-lint/")
for f in sorted(set(iter_md(paths))):
check_banned = not any(x in f for x in ban_exempt)
fence = False
with open(f) as fh:
for ln, line in enumerate(fh, 1):
if line.lstrip().startswith("```"):
fence = not fence; continue
if fence:
continue
if check_banned:
for m in BAN_RE.finditer(line):
print(f"{f}:{ln}: banned word '{m.group(1)}'")
violations += 1
for m in LINK.finditer(line):
link = m.group(1)
if re.match(r'^(https?:|mailto:|#|/)', link):
continue
path = re.split(r'[#?]', link)[0]
if not path:
continue
tgt = os.path.normpath(os.path.join(os.path.dirname(f), path))
if not os.path.exists(tgt):
print(f"{f}:{ln}: broken link -> {link}")
violations += 1
print(f"\n{violations} violation(s)")
return 1 if violations else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))

View File

@@ -0,0 +1,35 @@
---
name: incident-investigation
risk_class: read_only
inputs: [symptom, affected_entity]
verification: "n/a — investigation produces a written record, not a state change"
docs_update_checklist: [investigations_entry]
---
# Incident investigation
Goal: understand what broke and why, before touching anything.
1. MCP `explain` (or `get_blast_radius` if the affected entity is a
host) — get the blast radius and doc pointer first. Don't start
pulling logs blind.
2. MCP `get_service_status` + `tail_log` for the affected service.
3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
`authentik`, or the backend host itself)? MCP `get_blast_radius`
shows "affected by" — check those first.
4. If the symptom looks like a dpkg/upgrade interaction, see
[runbook-dpkg-interrupted](../runbook-dpkg-interrupted/SKILL.md) —
there's no fleet-wide apt-audit tool anymore, check the host directly.
5. Check the change ledger for recent mutations to the affected entity
or anything upstream of it: MCP `get_change_history` or `get_audit_trail`.
6. Write findings via MCP `upsert_knowledge` (`kind: investigation`) —
symptom, timeline, root cause, fix applied, prevention, `about` set to
the affected entity's slug. The DB is the durable record now, not a
markdown file — `search_knowledge`/`get_entity_knowledge` read it back;
a chat message alone is forgotten.
Docs-update checklist: always create the investigation entry. If the
root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,
or `backend` that didn't match reality — this happened during Week 1
kernel work, see the `authentik` backend fix), correct `inventory.yaml`
in the same session.

View File

@@ -0,0 +1,35 @@
---
name: lifecycle-activate-node
risk_class: config_mutation
inputs: [node_name]
verification: "MCP get_service_status (if it hosts a service); MCP whoami (if it's a client)"
docs_update_checklist: [doc_page_complete]
transition: "provisioning -> active"
---
# Lifecycle: activate a node
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml). Requires: age key
enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
ingress live if public, health check answering, doc page complete,
ledger entry.
1. If the node self-enrolls as a client: finish enrollment per
[CLIENTS.md](../../../CLIENTS.md#enrollment) (`bootstrap.sh`
`/api/v1/clients/enroll`, mesh join, MCP `whoami` returns the entity).
2. If it hosts a public service: add the `services:` entry in
`seeds/inventory.yaml` (backend, url, doc_page, config_repo,
risk_notes) and wire the Caddy route in `dtoro/caddy-conf`.
3. Confirm the health check answers: MCP `get_service_status` or a
direct `curl`.
4. Flip `state: provisioning``state: active` (or delete the `state:`
field — `active` is the default) in `seeds/inventory.yaml`, then
`oikos seed` to ingest.
5. Complete the doc page (stub → full narrative: role, specs, how it's
configured, dependencies).
6. No manual record-keeping step needed — the activation (via whatever
API call flipped the state) is recorded automatically in `audit_log`.
Regenerate: `oikos seed` re-ingests `seeds/inventory.yaml`; `oikos export`
writes DB state back out to the YAML if you mutated via the API/MCP
instead of editing the file directly.

View File

@@ -0,0 +1,35 @@
---
name: lifecycle-deprecate-node
risk_class: config_mutation
inputs: [node_name, replacement_node_or_reason]
verification: "MCP get_blast_radius — 'affected by' must be empty before completing"
docs_update_checklist: [doc_page_deprecation_note]
transition: "active -> deprecated"
---
# Lifecycle: deprecate a node
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml): a node keeps running
but takes no new dependents. **Completion condition: zero remaining
inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a
suggestion; `seeds/policy.yaml` `lifecycle_overrides.deprecated.refuse`
lists `new-inbound-edges` as refused going forward.
1. Set `state: deprecated` on the node.
2. MCP `get_blast_radius` — read `affected_by`. Every entry there is
something still relying on this node.
3. Migrate or retire each dependent one at a time (point its `backend`/
`config_repo`/ingress route elsewhere, or deprecate it too if it's
being retired alongside).
4. Re-run MCP `get_blast_radius` after each dependent is moved. The
transition to `destroyed` is only safe once `affected_by` is empty —
check this every time, don't assume from memory.
5. Note the deprecation on the doc page: reason, replacement (if any),
date.
If step 2 shows dependents you didn't expect, stop and investigate
before proceeding — that's exactly the kind of drift the Week-3 detector
will catch automatically, but until then this manual check is the gate.
Next (once `affected_by` is empty):
[lifecycle-destroy-node.md](../lifecycle-destroy-node/SKILL.md).

View File

@@ -0,0 +1,49 @@
---
name: lifecycle-destroy-node
risk_class: destructive
inputs: [node_name]
verification: "MCP get_blast_radius returns unknown-entity; pct list on the backend no longer shows it"
docs_update_checklist: [archaeology_entry, containers_index_update]
transition: "deprecated -> destroyed"
---
# Lifecycle: destroy a node
**Destructive.** Requires operator approval + typed confirmation phrase
per `seeds/policy.yaml`. Requires (ontology): backups verified, secrets
recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
ledger entry.
1. Confirm the node is `deprecated` with zero `affected_by` edges
(MCP `get_blast_radius`) — do not skip this even if the deprecation
runbook was followed recently; state can drift.
2. **If it's an enrolled client: no current tool for revoking its age key /
removing its Infisical identity.** The old `homelab client remove`
(age key revocation + SOPS re-key + inventory removal, all one
destructive-class CLI call) is retired along with the rest of that CLI
and hasn't been re-verified against the current enrollment
architecture (`POST /api/v1/clients/enroll` + Infisical machine
identities) — see the "Open questions" section in
[agent-enrollment.md](../../operations/agent-enrollment.md). Until
that's confirmed, treat key/identity revocation as a manual step:
at minimum remove the client's `age_pubkey` from any SOPS recipient
lists and rotate credentials whose ciphertext it already decrypted.
3. Remove any ingress route (Caddy config repo) and DNS record still
pointing at it.
4. Verify backups of anything on it are retained per policy before the
disk goes away (see `backs-up-to`).
5. Destroy the LXC/VM (`pct destroy` / `qm destroy`).
6. Update the entity's `state` to `destroyed` in `seeds/inventory.yaml`
(or move it to an `archaeology:`-style section if the schema still has
one) — `pve_id`, `destroyed` date, `reason` — then `oikos seed` to
ingest. Add a row to `containers/index.md` "Recently destroyed" table
(kept for human-readable browsing alongside the structured data).
7. No manual ledger step — mutations through the API are recorded
automatically in the `audit_log` table (MCP `get_audit_trail`,
`get_change_history`). The old `oikos/ledger.py append` was retired
when this became automatic.
If the destroy fails partway (e.g. secrets not fully revoked but pct
destroy errors), finish the remaining steps manually and note the
partial state in an investigation (MCP `upsert_knowledge`,
`kind: investigation`).

View File

@@ -0,0 +1,40 @@
---
name: lifecycle-migrate-node
risk_class: config_mutation
inputs: [node_name, source_host, target_host]
verification: "MCP get_blast_radius (re-check blast radius); MCP get_service_status for every hosted service"
docs_update_checklist: [doc_page_migration_note, inventory_host_and_lan_ip]
transition: "active -> migrating -> active"
---
# Lifecycle: migrate a node
Modeled on the strong Phase 1+2 migration
([archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
Requires (ontology): preflight + backup-verified before migrating;
post-verify + Caddy backends checked + mounts checked + docs updated
before returning to `active`.
1. MCP `preflight` for every service the node hosts — capture
current health as a baseline.
2. Verify backups are current for anything with data at rest on the
node (see `backs-up-to` edges once populated).
3. Set `state: migrating` in `inventory.yaml`.
4. Perform the migration (pct/qm move, or create-on-target +
data-copy + destroy-source, per the specific case).
5. Update `inventory.yaml`: new `host:`, `lan_ip`, `mesh` addresses for
the node; update every `services:` entry whose `backend` pointed at
it if the backend name itself changes (usually it doesn't — only the
`host:`/`lan_ip` on the guest entry moves).
6. Post-verify: re-run the Week-1 drift check by hand — confirm Caddy's
backend IP for each affected service matches the new `lan_ip`
(automatic in Week 3's drift detector), confirm mounts still resolve.
7. MCP `get_service_status` for every service the node hosts.
8. Set `state: active`. Add a migration note to the node's doc page
(old host/IP → new, date, phase reference) — this repo's convention
for every past migration (see `archive/knowledge/containers/101-jellyfin.md`,
`containers/129-house.md`).
Regenerate: `oikos seed` (re-ingests `seeds/inventory.yaml` into the DB —
the DB is authoritative at runtime, the YAML is the source of truth
on disk).

View File

@@ -0,0 +1,35 @@
---
name: lifecycle-provision-node
risk_class: config_mutation
inputs: [node_name, kind, storage_pool]
verification: "grep 'state: provisioning' inventory.yaml"
docs_update_checklist: [doc_page_stub]
transition: "planned -> provisioning"
---
# Lifecycle: provision a node
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml) `lifecycle.transitions`.
Policy note: `provisioning` nodes get a lifecycle override —
`config_mutation` actions downgrade to `reversible_low` because nothing
depends on the node yet (see `seeds/policy.yaml` `lifecycle_overrides`).
Requires (from ontology): inventory entry, IP reserved, storage pool
chosen, doc page stub.
1. Create the LXC/VM on its target Proxmox host (`pct create` /
`qm create`), choosing the storage pool deliberately — record it as
the `storage:` field once populated (Week 1 schema; not yet backfilled
for existing nodes).
2. Add the inventory entry: a `hosts.<name>:` block in
`seeds/inventory.yaml` with `state: provisioning`, `kind`, `host`,
`pve_id`, `lan_ip`, then `oikos seed` to ingest it. For anything that
will self-enroll as a client afterward (see
[CLIENTS.md](../../../CLIENTS.md#enrollment)), the entity must exist in
`planned`/`provisioning` state before `bootstrap.sh` runs there.
3. Stub the doc page (`containers/<pve_id>-<name>.md` or
`vms/<pve_id>-<name>.md`) — even a one-line "provisioning, see plan X"
is enough to satisfy the transition requirement.
4. Reserve the IP in DNS/DHCP notes if it's a fixed LAN address.
Next: [lifecycle-activate-node.md](../lifecycle-activate-node/SKILL.md).

View File

@@ -0,0 +1,209 @@
---
name: budget-import-from-csv
risk_class: config_mutation
inputs: [csv_file]
references: [containers/129-house.md]
---
# Runbook: Budget import from N26 CSV → Yuvomi
Distil a bank-export CSV into Yuvomi's Budget and Subscriptions modules using
the `yuvomi-mcp` tools. Run this whenever a new CSV period needs to be
summarised into targets and fixed costs.
---
## Prerequisites
- `yuvomi-mcp` is running on LXC 129 and connected as an MCP server in Claude.
- The CSV is an N26 export (columns: Booking Date, Value Date, Partner Name,
Partner Iban, Type, Payment Reference, Account Name, Amount (EUR), …).
- API token: `yuvomi-api-token`, via Infisical (primary) or `oikos secret` (SOPS fallback).
- Direct API base: `https://house.hubris.network/api/v1`
---
## API quirks (Yuvomi ≤ 0.77.x)
- **Subscriptions live under `/budget/subscriptions`**, NOT `/subscriptions/`.
A top-level `/subscriptions` route returns 404.
- `GET /budget/subscriptions``{ data: { subscriptions: [...], summary: {...} } }`
- `GET /budget/subscriptions/meta``{ data: { categories: [...], payment_methods: [...] } }`
- `POST /budget/subscriptions` → create a subscription (name, amount, billing_cycle,
cycle_interval, next_payment_date, currency, category_id, payment_method_id required)
- `GET /budget/` (no month) → returns only **non-recurring** base entries.
Use `GET /budget/?month=YYYY-MM` to get all entries (recurring + one-time) for a month.
- `GET /budget/categories` → expense category keys + income category names (German keys
like `"Erwerbseinkommen"`, `"Sozialleistungen"`, `"Geschenke & Transfers"`).
- Budget entries: `amount` positive = income, negative = expense.
- Recurring entries: set `is_recurring: 1` + `recurrence_interval: "monthly"`.
The `date` field sets the start month.
- `recurrence_virtual: 1` smooths non-monthly amounts across all months in the summary
(e.g. 55.08 € quarterly → shows as ~18.36 €/month).
- Custom RRULE strings (`recurrence_rule`) are **not accepted** by the API — use
`cycle_interval` on the subscription instead, or `recurrence_interval` on budget entries.
---
## Subscription category IDs (as of 2026-06-26)
| id | name | budget_subcategory_key |
|---|---|---|
| 1 | Entertainment | subscription_entertainment |
| 2 | Productivity | subscription_productivity |
| 3 | Utilities | subscription_utilities |
| 4 | Health | subscription_health |
| 5 | Education | subscription_education |
| 6 | Other | subscription_other |
## Payment method IDs
| id | name |
|---|---|
| 1 | Credit Card |
| 2 | Debit Card |
| 3 | PayPal |
| 6 | Bank Transfer / SEPA |
| 7 | Other |
---
## Step 1 — Categorise the transactions
Skip these as internal/already-covered:
- Fixed costs you'll enter as **subscriptions** (Miete, SWM, SYNVIA, Hundefutter,
Netflix, Grover, Rundfunk ARD, KuKita, Lillydoo)
- Internal transfers (The Joy Pot ↔ Cookie, Hauptkonto, Tagesgeldkonto splits)
- Identified income (Cookie Share, Kindergeld, Pocket Money credits, Distributor)
- Fun Money pass-throughs (in and out same month → net zero)
**Variable expense taxonomy:**
| Category key | Subcategory key | Examples |
|---|---|---|
| `food` | `groceries` | E-Center, Knuspr, EDEKA, Tegut, VollCorner, Lidl, Netto, REWE, KoRo, Roast Market |
| `food` | `restaurants_bars` | Restaurants, Lieferando, Cafes, Zeit für Brot, Baobab, Höflinger |
| `personal_health` | `beauty_cosmetics` | DM, Rossmann |
| `personal_health` | `pharmacy` | Apotheke, MVZ Dermatologie |
| `transport` | `apps_taxi` | Uber, RYD GMBH, MVG, Handyparken |
| `shopping_clothing` | `gifts` | Children products: Schlummersack, Catchy Kids, SP EVERY., Dukal, Berger-Lernwelt |
| `shopping_clothing` | `clothes_shoes` | Zalando, Ernsting's, Schuhmair, Thalia, Vinted, Airbnb, Hotel at Booking.com |
| `shopping_clothing` | `electronics` | Amazon, AMZN Mktp DE |
| `housing` | `renovation_maintenance` | IKEA, Markus Festl, Granit, Sostrene Grene, Mol* tischdecken, Gaertnerei, Dehner |
| `education` | `courses_college` | Kathrin Orlob (PEKiP), Nerina Aupperle |
| `leisure` | `streaming` | WOW wowtv.de |
| `financial_other` | `bank_fees` | Unidentified PayPal, Ratepay, N26 fees |
| `Geschenke & Transfers` | *(income)* | One-off incoming transfers |
---
## Step 2 — Create subscriptions
```
get_subscriptions_meta() ← get category_id and payment_method_id
```
**Standard Cookie household subscriptions (as of 2026-07):**
| Name | Amount | billing_cycle | cycle_interval | category_id | payment_method_id |
|---|---|---|---|---|---|
| Miete | 1080.00 | monthly | 1 | 6 (Other) | 6 (Bank Transfer) |
| Strom (SWM) | 79.00 | monthly | 1 | 3 (Utilities) | 6 |
| Internet / TV / Telefon | 29.99 | monthly | 1 | 3 (Utilities) | 6 |
| Hundefutter | 75.00 | monthly | 1 | 6 (Other) | 6 |
| Netflix | 8.00 | monthly | 1 | 1 (Entertainment) | 6 |
| Grover | 16.90 | monthly | 1 | 6 (Other) | 2 (Debit Card) |
| Rundfunk ARD / ZDF | 55.08 | monthly | 3 | 1 (Entertainment) | 6 |
| KuKita Daycare (Leon) | 503.00 | monthly | 1 | 5 (Education) | 6 |
| Lillydoo diapers | 56.70 | monthly | 2 | 4 (Health) | 3 (PayPal) |
Monthly equivalent total: **1,838.60 €** (Yuvomi applies cycle_interval to prorate).
---
## Step 3 — Add recurring income entries
```
stage_add_budget_entry(
title="Kindergeld",
amount=55.00,
category="Sozialleistungen",
date="YYYY-MM-01",
is_recurring=True,
recurrence_interval="monthly",
)
commit_pending(pending_id)
```
**Standard recurring income:**
| Title | Amount | category |
|---|---|---|
| Kindergeld | +55.00 | Sozialleistungen |
| Cookie Share | +2650.00 | Erwerbseinkommen *(see recommended amount below)* |
---
## Step 4 — Post variable transactions
For each non-skipped CSV row, call `stage_add_budget_entry` with the mapped
category/subcategory and the actual transaction amount and date. Use the Partner
Name + Payment Reference as the title (truncate to 100 chars).
---
## Step 5 — Verify
```
get_budget_summary("YYYY-MM")
list_subscriptions()
```
Expected for a full month with KuKita:
- Fixed expenses ≥ 1,838 € (subscriptions)
- Variable expenses ≥ 500 € (groceries alone)
---
## Cookie Share: how much to transfer monthly
Calculated from JanJun 2026 data (Cookie account, one-offs stripped):
| | €/month |
|---|---|
| **Fixed costs (subscriptions)** | **1,839** |
| Miete | 1,080 |
| KuKita *(permanent from Jul 2026)* | 503 |
| Strom + SYNVIA + Rundfunk + Netflix + Grover + Hundefutter + Lillydoo | 256 |
| **Variable (6-month averages)** | **1,032** |
| Groceries | 595 |
| Children products | 142 |
| Dining & cafes | 100 |
| Transport | 66 |
| Drugstore | 52 |
| Clothing, Amazon, Pharmacy | 77 |
| **Total monthly spend** | **≈ 2,871** |
| Minus Kindergeld (fixed income) | 55 |
| Minus Pocket Money (conservative ~600 €) | 600 |
| **→ Recommended Cookie Share** | **≈ 2,650 €** |
| With 200 € buffer | **≈ 2,850 €** |
**Current Cookie Share (Jun 2026): 1,995 € — shortfall ~655 €.**
The gap was covered by irregular Pocket Money top-ups (avg 962 €/mo over 6 months, but
highly variable: 121 €3,000 €). KuKita starting in June is the biggest step-up; raising
Cookie Share to **2,650 €** makes the budget self-sufficient without relying on top-ups.
---
## Changelog
### 2026-06-29 — Corrections from first real import
- Subscriptions endpoint is `/budget/subscriptions`, NOT `/subscriptions/` (404).
- `recurrence_rule` RRULE strings are rejected by the API; use `cycle_interval` instead.
- `GET /budget/` (no filter) returns only non-recurring entries; use `?month=` for full view.
- Added Cookie Share recommendation (2,650 €/month) based on 6-month expense analysis.
- Added full category taxonomy table.
### 2026-06-29 — Initial runbook
Created from JanJun 2026 N26 Cookie account analysis.

View File

@@ -1,3 +1,9 @@
---
name: recover-dpkg-interrupted
risk_class: reversible_low
verification: "dpkg --audit (should be clean); apt-get check"
---
# Runbook — recover from dpkg-interrupted state
You're here because an apt run got killed mid-transaction and the target now
@@ -7,7 +13,8 @@ has packages that are **unpacked but not configured**. Symptoms:
manually run 'dpkg --configure -a' to correct the problem.`
- `dpkg --audit` lists packages with header
`The following packages have been unpacked but not yet configured.`
- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host.
- `dpkg --audit` on the host directly shows unpacked-not-configured packages
(there's no fleet-wide audit tool anymore — check per-host).
The system is still running the **old** binaries (still in memory), but the
**new** binaries are unpacked and waiting for their postinst to run. Two
@@ -27,19 +34,20 @@ config dirs, capabilities, etc.). The system might not come back up cleanly.
## Path A — target is still reachable over ssh (preferred)
```
homelab ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
```
Or for an LXC by name:
Or for an LXC by name (via the MCP `run` tool, or directly on the Proxmox
host):
```
homelab pct <lxc> exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
pct exec <lxc> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
```
When that returns, confirm:
```
homelab apt-audit --target <host>
ssh <host> -- dpkg --audit
```
Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally
@@ -83,10 +91,11 @@ DEBIAN_FRONTEND=noninteractive dpkg --configure -a \
## Prevention
The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect`
unit on the target, so it survives ssh teardown — the failure mode that put
hubris into this state in the first place is no longer reachable through the
standard tool. If you absolutely need to run apt manually over ssh, wrap it:
The old `homelab apt-upgrade` wrapper (retired along with the rest of the
`homelab` CLI) used to launch apt inside a `systemd-run --collect` unit on
the target so it survived ssh teardown — that's the failure mode that put
hubris into this state in the first place. There's no fleet-wide wrapper
anymore; if you run apt manually over ssh, wrap it yourself the same way:
```
ssh <host> systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade'
@@ -96,9 +105,9 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
## Related
- [Operations cheatsheet](commands.md)
- [Auto-deploy pipelines](../infrastructure/auto-deploy.md)
- [Hubris host page](../hosts/hubris.md)
- [Operations cheatsheet](../../operations/commands.md)
- [Auto-deploy pipelines](../../../archive/knowledge/infrastructure/auto-deploy.md)
- [Hubris host page](../../../archive/knowledge/hosts/hubris.md)
## Changelog

View File

@@ -0,0 +1,30 @@
---
name: service-health-check
risk_class: read_only
inputs: [service_name]
verification: "MCP get_service_status"
docs_update_checklist: []
---
# Service health check
Goal: determine whether a service is actually healthy, without ad-hoc SSH.
1. MCP `explain` — read the context card: backend, blast radius, doc
pointer, risk notes.
2. MCP `get_service_status` — live health probe (HTTP code against the
service's `url`/`endpoint`); the scheduler also probes on its own
interval, so this may reflect a recent cached result, not necessarily
a fresh one.
3. If unhealthy, `tail_log` for the last 200 lines.
4. Cross-check blast radius: MCP `get_blast_radius` — is this entity's
own backend host healthy? A downstream failure (e.g. a Proxmox host
down) will show up here before the service's own logs explain anything.
5. If the fix is a restart: classify first (`seeds/policy.yaml`
`service-restart` is `reversible_low` unless the service has a
`service_overrides` entry, e.g. `caddy`/`dns` are `config_mutation`).
Unattended agents may act on `reversible_low` without approval.
Docs-update checklist: none for a pure health check. If the investigation
reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml` in
the same session.

View File

@@ -0,0 +1,82 @@
---
name: session-review
description: "Examine a Nomos chat session, compare the user's objective with the actual outcome, identify causes of failure (missing tools, excessive tool calls, blocked actions, model behavior), and propose concrete fixes."
risk_class: reversible_low
inputs: [session_id]
---
# Session review
Analyze Nomos chat sessions from the live database, diff objectives
against outcomes, and propose fixes.
## 1. Retrieve session data
```bash
# List recent sessions
curl -s http://localhost:8092/sessions | jq '.sessions[:5]'
# Fetch one session with messages
curl -s http://localhost:8092/sessions/{session_id} | jq .
```
## 2. Classify the session
For each session determine:
| Dimension | Check |
|-----------|-------|
| Objective | What was the user trying to accomplish? |
| Outcome | Was it achieved? (read final assistant text) |
| Tool calls | Count, unique tools, redundancy (e.g., N+1 fan-out) |
| Blockers | Missing action? Missing tool? Model refusal? Empty response? |
| User frustration | Did the user need to clarify/correct/repeat? |
| Message sizes | Content blob sizes — truncation needed? |
## 3. Key failure signatures
| Signature | Root cause | Fix |
|-----------|-----------|-----|
| Agent: "I can't run X" | Missing target or capability | Use `run` with shell command — there is no fixed action enum anymore |
| Agent: "No local knowledge on that" + no web tool | Missing `http_get` / web fetch MCP tool | Add MCP tool |
| Empty assistant bubble (text="", no tools) | Model returned blank completion | Retry + error surfacing |
| Non-English boilerplate refusal | Flash-tier model degradation | Response quality guard |
| >30 tool calls per turn, same tool repeated | N+1 fan-out instead of bulk tool | Enrich bulk tools + tighten SOUL.md |
| Message >50KB in DB | Raw tool results persisted verbatim | Truncation in `store.go` |
## 4. Extract patterns across sessions
```bash
# All sessions summary
curl -s http://localhost:8092/sessions | jq -r '.sessions[] | "\(.id[:8]) \(.title[:80]) \(.created_at[:16])"'
# Message count + tool count per session
for id in $(curl -s http://localhost:8092/sessions | jq -r '.sessions[].id'); do
msgs=$(curl -s "http://localhost:8092/sessions/$id" | jq '.messages | length')
tools=$(curl -s "http://localhost:8092/sessions/$id" | jq '[.messages[].content.tool_calls | length] | add')
echo "$id $msgs msgs $tools tools"
done
```
## 5. Output format
```
Session: {id[:8]} — "{title[:60]}"
Messages: {N} ({user}/{assistant})
Tool calls: {total} across {turns} turns
Top tools: {name:count, name:count, ...}
Objective: {one-line summary}
Outcome: ✅ / ❌ / ⚠️
Blockers: {list or "none"}
Fixes needed: {concrete actions}
Severity: blocker | friction | cosmetic
```
## Related files
- `cmd/nomos/agent.go` — agent loop, tool building, response guards
- `cmd/nomos/store.go` — session + message persistence
- `internal/mcp/server.go` — all tool implementations (`run`, `list_lxcs`, …)
- `web/src/lib/components/ToolCallGroup.svelte` — tool result display
- `nomos/SOUL.md` — agent persona and tool selection rules
- `plans/2026-07-09-chat-sessions-improvements.md` — prior session findings
- `plans/2026-07-09-session-execution-and-ux-fixes.md` — latest plan

12
.claude/launch.json Normal file
View File

@@ -0,0 +1,12 @@
{
"version": "0.0.1",
"configurations": [
{
"name": "web",
"runtimeExecutable": "sh",
"runtimeArgs": ["-c", "export OIKOS_API_TOKEN=$(docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' oikos-api-1 | sed -n 's/^OIKOS_MCP_BEARER_TOKEN=//p'); exec npm --prefix web run dev"],
"port": 5173,
"autoPort": true
}
]
}

99
.gitea/workflows/ci.yml Normal file
View File

@@ -0,0 +1,99 @@
# Oikos CI (Gitea Actions). Gates the deploy webhook on a green run (plan M1).
# Mirrors `make lint`, `make test`, and the generated-code drift guard.
name: ci
on:
push:
branches: [main]
pull_request:
jobs:
build-test:
runs-on: ubuntu-latest
services:
postgres:
image: timescale/timescaledb:2.17.2-pg16
env:
POSTGRES_DB: oikos
POSTGRES_USER: oikos
POSTGRES_PASSWORD: oikos_dev
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U oikos"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
OIKOS_TEST_DATABASE_URL: postgres://oikos:oikos_dev@postgres:5432/oikos?sslmode=disable
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.26"
cache: true
- name: go vet
run: go vet ./...
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
with:
version: latest
args: --timeout 5m
continue-on-error: true # advisory until the lint baseline is clean
- name: govulncheck
run: |
go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck ./... || true # advisory
- name: generated code is up to date
run: make generate-check
- name: build
run: go build ./...
- name: test (race + coverage)
run: go test -race -covermode=atomic -coverprofile=coverage.out -timeout 300s ./...
- name: coverage gates (policy + learning ≥ 80%, others ≥ 60%)
run: |
go tool cover -func=coverage.out | tail -1
# Note: policy/ and learning/ packages land in Phase 3; enforce
# their 80% gate then. For now, report total coverage.
docker-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: docker build (verify image builds; no push)
run: docker build -f compose/oikos/Dockerfile -t oikos:ci .
web:
runs-on: ubuntu-latest
defaults:
run:
working-directory: web
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: npm
cache-dependency-path: web/package-lock.json
- run: npm ci
- name: svelte-check (advisory — baseline not yet clean)
run: npm run check
continue-on-error: true
- name: eslint (advisory — baseline not yet clean)
run: npm run lint
continue-on-error: true
- name: prettier format check (advisory — baseline not yet clean)
run: npm run format:check
continue-on-error: true
- name: test
run: npm run test
- name: build
run: npm run build

View File

@@ -0,0 +1,70 @@
name: Desktop App
on:
push:
branches:
- main
tags:
- 'desktop-*'
- 'v[0-9]+.[0-9]+.[0-9]*'
jobs:
build:
name: Build Linux (amd64)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
- run: npm ci
working-directory: web
- run: npm run build
working-directory: web
- run: |
rm -rf cmd/desktop/frontend/dist
mkdir -p cmd/desktop/frontend/dist
cp -r web/dist/* cmd/desktop/frontend/dist/
- uses: actions/setup-go@v5
with:
go-version: '1.26'
- run: |
sudo apt-get update
sudo apt-get install -y libgtk-3-dev libwebkit2gtk-4.1-dev
- run: CGO_ENABLED=1 go build -o build/bin/Oikos .
working-directory: cmd/desktop
- run: |
cd cmd/desktop/build/bin
tar czf oikos-desktop-linux-amd64.tar.gz Oikos
sha256sum oikos-desktop-linux-amd64.tar.gz > oikos-desktop-linux-amd64.tar.gz.sha256
- uses: actions/upload-artifact@v4
with:
name: oikos-desktop-linux-amd64
path: |
cmd/desktop/build/bin/oikos-desktop-linux-amd64.tar.gz
cmd/desktop/build/bin/oikos-desktop-linux-amd64.tar.gz.sha256
release:
name: Attach to Release
needs: build
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/download-artifact@v4
with:
name: oikos-desktop-linux-amd64
- uses: https://gitea.com/actions/release-action@v1
with:
files: |
oikos-desktop-linux-amd64.tar.gz
oikos-desktop-linux-amd64.tar.gz.sha256
api_key: ${{ secrets.GITEA_TOKEN }}

27
.gitignore vendored
View File

@@ -1 +1,26 @@
.DS_Store
.DS_Store
__pycache__/
*.pyc
# Compiled binaries (Go rewrite — bin/oikos, bin/nomos)
bin/oikos
bin/nomos
oikos
webhook
backups/
.env
.infisical-credentials
# Web UI (Svelte 5) — build artifacts. The SPA is a standalone static build,
# deployed separately from the oikos binary (plans/2026-07-12-wails-desktop-app.md
# 0.1), so the output dir is just a build artifact.
web/dist/
web/node_modules/
# Wails desktop app — frontend copy for embedding
cmd/desktop/frontend/dist/
cmd/desktop/build/
cmd/desktop/Oikos
desktop
/eval

41
.golangci.yml Normal file
View File

@@ -0,0 +1,41 @@
# golangci-lint configuration for Oikos
# Docs: https://golangci-lint.run/usage/configuration/
run:
timeout: 5m
tests: true
linters:
enable:
- govet # go vet
- staticcheck # advanced static analysis
- ineffassign # detect ineffectual assignments
- unused # find unused identifiers
- errcheck # check for unchecked errors
- gosimple # simplifications
- typecheck # standard type checking
- misspell # find commonly misspelled English words in comments
- revive # fast, configurable linter (replaces golint)
linters-settings:
errcheck:
# Allow unchecked errors on common Close/Flush patterns (deferred cleanup)
exclude-functions:
- (io.Closer).Close
- (*os.File).Close
issues:
# Exclude generated code
exclude-rules:
- path: _test\.go
linters:
- errcheck
- path: internal/httpapi/gen/
linters:
- all
- path: internal/db/sqlcgen/
linters:
- all
# Don't auto-exclude common patterns
exclude-use-default: false
max-issues-per-linter: 0
max-same-issues: 0

View File

@@ -1,14 +1,12 @@
# SOPS recipient rules.
#
# Each rule pins one or more age public keys as recipients for files matching
# the path_regex. The build_host_files.py generator doesn't read this file;
# `sops` does — to encrypt a new secret, run `sops -e secrets/<name>.yaml`
# from the repo root and SOPS will pick the matching rule below.
# SOPS is DR-fallback only — Infisical is the active secrets backend.
# Files live in archive/secrets-sops-backup/ for cold recovery.
# To encrypt a new DR secret: sops -e archive/secrets-sops-backup/<name>.yaml
#
# To grant a secret to a new client: add their age public key (from
# To grant a DR secret to a new client: add their age public key (from
# inventory.yaml `hosts.<name>.age_pubkey`) to the relevant rule below, then
# run `sops updatekeys secrets/<name>.yaml` to re-encrypt without rotating
# the ciphertext payload.
# run `sops updatekeys archive/secrets-sops-backup/<name>.yaml` to re-encrypt.
#
# To revoke: remove the recipient from the relevant rule and run
# `sops updatekeys` (this is what `homelab client remove` calls). Past
@@ -16,7 +14,7 @@
# underlying credential if compromise is suspected.
creation_rules:
- path_regex: ^secrets/hello\.yaml$
- path_regex: ^archive/secrets-sops-backup/hello\.yaml$
# The "hello" secret is encrypted to every enrolled client so the bootstrap
# decrypt test works for everyone. Add each new client's age_pubkey when
# they enrol; re-key with `sops updatekeys -y secrets/hello.yaml`.
@@ -24,9 +22,12 @@ creation_rules:
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/gitea-pat\.yaml$
- path_regex: archive/secrets-sops-backupgitea-pat\.yaml$
# Write-scoped Gitea PAT (dtoro user). Same recipient list as hello.yaml
# since every enrolled client should be able to push (homelab client
# add/remove, wiki edits, etc.).
@@ -34,19 +35,22 @@ creation_rules:
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/gitea-tokens\.yaml$
- path_regex: archive/secrets-sops-backupgitea-tokens\.yaml$
# Workstations only.
age: >-
# placeholder — fill with age_pubkey of: republic-laptop, mac-mini, ludo-mini, hubris
# placeholder — fill with age_pubkey of: republic-laptop, mac-mini, strong, hubris
- path_regex: ^secrets/webhook-hmacs\.yaml$
- path_regex: archive/secrets-sops-backupwebhook-hmacs\.yaml$
# LXCs that run a webhook receiver.
age: >-
# placeholder — fill with age_pubkey of: apps, caddy
- path_regex: ^secrets/turn-shared-secret\.yaml$
- path_regex: archive/secrets-sops-backupturn-shared-secret\.yaml$
# coturn TURN long-term-credential password. Consumed by hubris (which
# renders /etc/turnserver.conf + /opt/management.json on the VPS via
# `homelab render-vps-configs`). Other recipients are convenience for
@@ -55,9 +59,11 @@ creation_rules:
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/netbird-authentik-oidc\.yaml$
- path_regex: archive/secrets-sops-backupnetbird-authentik-oidc\.yaml$
# Authentik OIDC client secret for the netbird-dashboard provider.
# Consumed by hubris to render /opt/management.json on the VPS
# (PKCEAuthorizationFlow.ProviderConfig.ClientSecret).
@@ -65,9 +71,11 @@ creation_rules:
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/netbird-pat\.yaml$
- path_regex: archive/secrets-sops-backupnetbird-pat\.yaml$
# NetBird API Personal Access Token. Consumed by the dns-sync job on the
# `dns` LXC (107) to reconcile Technitium -> NetBird managed DNS zone.
# (When 107 is enrolled, add its age_pubkey here and updatekeys.)
@@ -77,14 +85,50 @@ creation_rules:
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
- path_regex: ^secrets/openrouter-api-key\.yaml$
# OpenRouter API key consumed by the `hermes` wrapper (bin/hermes) when
- path_regex: archive/secrets-sops-backupopenrouter-api-key\.yaml$
# OpenRouter API key consumed by the `nomos` wrapper (bin/nomos) when
# spawning a Goose session. Recipients are any host that should run a
# Nous-Hermes agent. Add a host's age_pubkey here, then
# Nomos agent. Add a host's age_pubkey here, then
# `sops updatekeys -y secrets/openrouter-api-key.yaml`.
# See operations/hermes-agent.md.
# See operations/nomos-agent.md.
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: archive/secrets-sops-backupyuvomi-api-token\.yaml$
# Named Bearer token for the Yuvomi REST API, consumed by yuvomi-mcp on
# LXC 129 (house).
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: archive/secrets-sops-backuphermes-house-users\.yaml$
# Signal number → Yuvomi user_id mapping (PII). Consumed by hermesd on LXC 129.
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: archive/secrets-sops-backupoikos-approval-hmac\.yaml$
# HMAC signing key for Oikos approval-grant tokens (oikos/approve.py).
# Recipients: apps (105, runs the approval engine alongside homelab-mcp)
# and hubris (admin/debug decrypt). See OIKOS.md "Approval engine".
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
- path_regex: archive/secrets-sops-backupoikos-console-deploy-secret\.yaml$
# Shared HMAC secret for the Gitea deploy webhook (id 14) ->
# oikos-console-deploy.service on apps (105). Generated + registered
# with Gitea before the apps-side install ran (see
# oikos/console/deploy/README.md "Status") — write this exact value
# into /etc/oikos-console-deploy/secret rather than letting
# webhook/install.sh generate a fresh one.
age: >-
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
# webhook noop 2026-05-20T18:16:57+02:00

File diff suppressed because one or more lines are too long

248
AGENTS.md
View File

@@ -1,110 +1,222 @@
# AGENTS.md — orientation for any agent on a homelab client
You are running on a machine that is part of the **hubris** homelab. The full
context is in this checkout at `/opt/homelab-context/`. This file is the entry
context is in this checkout at `/opt/homelab/`. This file is the entry
point. Read it once at start, then keep working.
- **New client?** Read [CLIENTS.md](CLIENTS.md) first.
- **Developing on this repo?** Also read [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
The operating model — OODA loop, risk classes, approval rules, the ontology,
and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation,
classify the action against `seeds/policy.yaml`; when the class requires
approval, stop and ask the operator.
Agent-facing instruction lives under `.agents/`:
`.agents/shared/` holds the conventions every agent applies
([writing-style](.agents/shared/writing-style.md), [caveman](.agents/shared/caveman.md),
[page-templates](.agents/shared/page-templates.md), [llm-wiki](.agents/shared/llm-wiki.md)), and
`.agents/domains/` holds the per-domain schemas
([knowledge](.agents/domains/knowledge/schema.md), [operations](.agents/domains/operations/schema.md)).
**Source of truth:** The Postgres database is the single source of truth for all
structured data and knowledge. It is bootstrapped from `seeds/` at deploy time:
`seeds/ontology.yaml` (entity types, relationships, lifecycles), `seeds/inventory.yaml`
(hosts, services, entities), `seeds/policy.yaml` (risk classes, approval rules), and
`seeds/knowledge.yaml` (documents, investigations, runbooks). The old narrative wiki
is archived at `archive/knowledge/` for historical reference.
## 1. Who you are
Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read:
/opt/homelab-context/hosts/<your-hostname>.yaml
/opt/homelab/inventory.yaml
That file tells you your role, your peers, what's mounted, and what services
you host. If it does not exist, this client was not enrolled — stop and tell
the operator to run `homelab client add <hostname>` from an existing client.
the operator; see [CLIENTS.md](CLIENTS.md#enrollment) for the enrollment flow
(the entity needs to exist in `planned`/`provisioning` state first).
## 2. The topology
- `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation
with their mesh addresses, roles, and service mappings. Treat this file as
authoritative; anything you read in narrative pages should agree with it.
- `/opt/homelab-context/infrastructure/mesh.md` — Tailscale → Netbird state.
Both meshes are accepted today; Netbird is preferred for new traffic.
- `/opt/homelab-context/infrastructure/dns.md` — split-horizon DNS via
dnsmasq on LXC 124. `*.hubris.network` resolves to 192.168.x.x on the LAN
and to mesh addresses off-LAN.
- `/opt/homelab-context/operations/commands.md` — the operator's cheatsheet
for pct, caddy, dnsmasq. Use these verbs when you take actions.
- `/opt/homelab/inventory.yaml` — every host, LXC, VM, and workstation
with their mesh addresses, roles, and service mappings. This is the seed file;
at runtime the DB is authoritative (query via MCP `get_entity` or the REST API).
- `/opt/homelab/seeds/knowledge.yaml` — full narrative knowledge
(documents, investigations, runbooks). Counts are not hardcoded here; count
them from the seed or query the DB. Ingested into the DB on deploy.
- `/opt/homelab/.agents/operations/commands.md` — the operator's cheatsheet
for pct, caddy, DNS, and the Oikos command surface.
## 3. The MCP server
The homelab exposes a Model Context Protocol server with structured tools.
Endpoint is in `inventory.yaml` under `services.homelab_mcp.endpoint`.
Endpoint: `https://mcp.hubris.network/mcp`. Every call needs
`Authorization: Bearer <token>` — the API has no unauthenticated path except
enrollment and `/healthz` (see "Authentication" below for where the token
comes from).
Available tools:
Available tools (the authoritative list — count them below if a number is
needed; do not hardcode the count elsewhere):
Context (pure read):
get_host(name), list_services(), find_service(name_or_role),
get_topology(), search_docs(query), get_page(path),
get_changelog(page, since?), whoami(hostname),
list_my_secrets(caller_pubkey?)
Context — observe + orient:
get_entity(slug), list_entities(type, limit, cursor),
get_relations(entity), get_blast_radius(entity),
search_knowledge(query) — ILIKE search over documents, investigations,
runbooks in the knowledge_entities table
get_entity_knowledge(entity_slug) — every document, investigation, and
runbook linked to one entity, in one call
get_patterns(status, entity_type, action) — learned action patterns
get_skills(status) — available automation skills
http_get(url) — fetch a public page/raw file (e.g. researching how to
deploy something before provisioning it); HTTP/HTTPS only, ~16KB cap
Management (read-only):
get_service_status(service), tail_log(service, lines=200),
list_lxcs(), get_lxc_state(lxc), ping_service(service)
Management — live state:
get_service_status(service_slug) — systemctl is-active on target host
tail_log(service_slug, lines=200) — journalctl
list_lxcs() — all LXC containers with ID, host, IP, health
get_lxc_state(lxc_slug) — pct status from Proxmox host
ping_service(service_slug) — HTTP reachability from entity_status
list_my_secrets(caller_pubkey) — secrets accessible to this client by
age public key
Mutations are **not** exposed via MCP. Use the `homelab` CLI for those, with
operator confirmation.
Oikos — decisions:
explain(service_slug) — compact context card (type, state, health, relations)
preflight(service_slug, action) — risk class + approval requirement
whoami(hostname) — entity record, peers, health for a client
get_change_history(entity_slug, limit=20) — last audit-log entries per entity
get_state_snapshot() — fleet health, disk, drift count
**When to prefer MCP over grepping the clone:** any time you need to resolve a
name to an address, look up service status, or search the wiki by content.
Grep is fine for browsing or when MCP is unreachable.
Operations — observe + act:
get_health_summary() — fleet health counts (healthy/degraded/down/unknown)
get_signal_history(entity_slug, state, limit) — open + recent signals
get_audit_trail(entity_id) — audit log filter + browse
get_agent_activity(limit) — agent self-inspection
query_metrics(hours=24) — time-series metric bucketed averages
get_trend(entity_id, days=7) — metric slope over time
get_event_timeline(severity, entity_slug, limit) — recent events
## 4. Wiki conventions
Knowledge — keep the graph current (none require approval; this updates
the knowledge graph, not live infrastructure):
upsert_knowledge(title, content) — record what you learned after solving
a non-obvious problem; the only way anything persists past a session
update_entity_attributes(slug, attributes) — merge a discovered fact
(IP, version, port, ...) into an entity so a future task doesn't
rediscover it from scratch
create_relationship(source, target, type) — record a discovered edge
(depends-on, hosts, routes-to, ...) between two entities
- Pages live under `containers/`, `hosts/`, `vms/`, `infrastructure/`,
`investigations/`, `operations/`. Cross-link liberally; orphans are bugs.
- Every page ends with a `## Changelog` section, entries in reverse-chrono
order:
Execution — mutating the live infrastructure:
run(target, command) — the general execution primitive. Run any shell
command against a host or LXC; every command is auto-classified —
read-only inspection runs immediately, anything state-changing needs
operator approval, and destructive patterns (rm -rf, dd, mkfs,
pct/qm destroy, DROP TABLE, reboot, curl-pipe-to-shell, ...) always
need approval regardless of what you declare. This is the ONLY
mutation tool — `request_execution` was retired 2026-07-14; the
former enum actions (restart, systemctl, pct_exec, apt_upgrade,
pct_create) are all expressed as `run(target, command)` now.
get_execution_status(execution_id) — poll progress
### YYYY-MM-DD — short title
one or two lines describing what changed and why.
**When to prefer MCP over grepping the clone:** always for knowledge queries.
`search_knowledge("jellyfin hardware acceleration")` returns ranked results from
the DB with entity links. `get_entity_knowledge("lxc:jellyfin")` returns documents,
runbooks, and investigations in one call. Grep the clone only when MCP is
unreachable.
- Investigation files are dated and slugged: `YYYY-MM-DD-slug.md`.
- Live state takes precedence over docs. If you observe a discrepancy, update
the docs *in the same session* (per the same-session update rule).
## 4. Authentication
## 5. Acting on the homelab
Every API/MCP route requires `Authorization: Bearer <token>` except
`POST /api/v1/clients/enroll` and `/healthz`. Enrollment (see
[CLIENTS.md](CLIENTS.md#enrollment)) does not currently issue a per-client
API/MCP bearer token — there is one shared
secret (`OIKOS_MCP_BEARER_TOKEN`, validated in `internal/httpapi/server.go`'s
`combinedAuth`); get it from the operator until per-client token issuance
exists. The SPA has its own flow instead: a first-launch Config screen that
stores a token in `localStorage` (see `web/src/pages/Config.svelte`).
- **Read state**: prefer MCP tools, then files, then shell. Examples:
`homelab whoami`, `homelab list`, `homelab status`, `homelab logs caddy`.
- **Cross-host actions** (caddy reload, pct exec, etc.): use the `homelab`
CLI — it resolves hostname → mesh address → ssh / pct path for you. Direct
SSH still works; the CLI just removes the lookup burden.
- **Secrets**: never hardcode. Call `homelab secret <name>` to decrypt on
demand using the per-client age key at `/etc/age/key.txt`. Secrets ARE
available in this system — `list_my_secrets()` (MCP) shows what you can
decrypt.
- **Mutations** (restart, edit configs, etc.): the `homelab` CLI's mutating
subcommands ask for confirmation. For ad-hoc work, SSH and edit directly —
but commit changes that touch tracked configs (caddy, gitea custom,
artifacto, mule-image, etc.; see `infrastructure/auto-deploy.md`).
- **Wiki updates**: same-session rule applies to any meaningful state change
this client makes.
## 5. Knowledge conventions
## 6. Auto-setup mechanism
All narrative knowledge (documents, investigations, runbooks) lives in the DB
(`knowledge_entities` table) and is seeded from `seeds/knowledge.yaml`. Agents
can register new knowledge via the API:
```
POST /api/v1/knowledge/{entity_slug}
{"title": "...", "content": "...", "tags": ["..."]}
```
The DB is the truth. The old wiki files are archived at `archive/knowledge/`
(historical reference only — use MCP `search_knowledge` for live queries).
- **Runbook procedures** live as `runbook` entities in the DB and as SKILL.md
files under `.agents/skills/<name>/`. They carry `risk_class`, `procedure`
(JSON-schema-validated), and are linked to entity types via `applies_to_type`.
- **Investigations** are `investigation` entities linked to affected entities
via `about` edges.
- **Documents** are `document` entities linked to entities via `documents` edges.
They carry `at_glance` (structured attributes) and `changelog` (parsed entries).
- **Live state precedence.** If you observe a discrepancy between the docs and
running state, update the DB *in the same session* via the API. The `oikos export`
command regenerates `seeds/knowledge.yaml` for version control.
## 6. Acting on the homelab
- **Read state**: use MCP tools. Nomos (the AI agent) is the primary
operator interface — it routes to the MCP tool list in §3 for
observe/orient/decide/act.
- **Actions** (restart, logs, apt, pct exec, or anything else): Nomos calls
`run` (the general execution primitive) via MCP. `reversible_low`/read-only actions execute
immediately; `config_mutation` and `destructive` actions are queued for
operator approval via Matrix or the control-room UI's Operations page.
- **Secrets**: managed by Infisical (`oikos secret` subcommand for
migration). Never hardcode secrets — use env vars from `.env`.
- **Mutations** (restart, edit configs, etc.): classified against
`seeds/policy.yaml`. `reversible_low` actions auto-execute;
`config_mutation`/`destructive` actions require approval — granted by
the operator via Matrix reply or the control-room UI, not a CLI flag.
See OIKOS.md.
## 7. Communication mode
Read and apply `/opt/homelab/.agents/shared/caveman.md` (if present). It defines the lab's
terse-communication standard — drop filler, keep substance, use fragments.
## 8. Auto-setup mechanism
The homelab-context repo ships tooling that gets automatically installed
on every client after `git pull`. This is handled by `tools/post-pull.sh`
(replaces the raw git pull in the sync timer) which runs any script matching
`tools/*.setup.sh` after pull.
`tools/setup-*.sh` after pull.
Currently auto-setup:
- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm
package, wrapper scripts, and compact output templates for token-efficient
CLI output. Wrapper at `~/bin/caveman_wrapper.sh`.
- **Hermes agent persona** (`tools/setup-hermes-soul.sh`): Provisions
`~/.hermes/SOUL.md` from `HERMES.md` on Hermes agents. This ensures every
Hermes agent follows the canonical homelab persona (token efficiency, source
of truth hierarchy). No-op on non-Hermes agents.
- **Host checks** (`tools/setup-checks.sh`): Deploys `checks/install.sh`'s
health-check scripts to `/opt/oikos/checks` on each host. The scheduler's
`ssh-script` check kind depends on these actually being there (count is
whatever is currently seeded in the DB — do not hardcode it here).
To add a new auto-setup, create `tools/<name>.setup.sh` in the repo,
To add a new auto-setup, create `tools/setup-<name>.sh` in the repo,
commit and push. All enrolled clients pick it up within 5 minutes.
To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer.
To trigger sync manually: run `/opt/homelab/tools/context-poller.sh`, or
wait for the 5-min timer. (The server-side `tools_changed` detection only
correctly recognizes `setup-*.sh` scripts — earlier it silently matched
nothing, so nothing auto-ran on any client via this path.)
## 7. When in doubt
## 9. Versioning
Run `homelab mcp search_docs <query>` or `homelab mcp get_host <name>`.
The clone is the fallback; MCP is the index.
Every commit to `main` **MUST bump the version** in the `VERSION` file at the
repo root. The format is semver-ish: `major.minor.patch` (e.g. `0.2.3`).
Rules:
- **patch** (`0.2.2``0.2.3`): bugfixes, small tweaks, docs-only changes
- **minor** (`0.2.3``0.3.0`): new features, new tools, visible functionality
- **major** (`0.3.0``1.0.0`): breaking changes (API removal, tool retirement)
The version is shown in the UI sidebar. The `v` prefix is added at build time.
## 10. When in doubt
Use MCP tools: `search_knowledge <query>` for narrative context,
`get_entity <slug>` for structured data, `get_entity_knowledge <slug>` for
everything linked to an entity. The clone is the fallback; MCP is the index.

151
CLIENTS.md Normal file
View File

@@ -0,0 +1,151 @@
# Oikos — client guide
If you are a homelab machine, this is what Oikos is and what it gives you.
If you are an AI agent running on this machine, also read [AGENTS.md](AGENTS.md).
## What is Oikos?
Oikos is the agentic operating system for the **hubris** Proxmox homelab. It
observes state, classifies actions against policy, executes approved
procedures, learns from outcomes, and notifies the operator. It runs as a
Docker stack on mac-mini and exposes an MCP server + REST API.
## What Oikos provides
| Capability | How you access it |
|------------|-------------------|
| Entity query (topology, blast radius) | MCP `get_entity`, `get_blast_radius` |
| Full-text knowledge search | MCP `search_knowledge` |
| Service status + logs | MCP `get_service_status`, `tail_log` |
| LXC inventory + state | MCP `list_lxcs`, `get_lxc_state` |
| Context cards | MCP `explain` |
| Pre-flight risk classification | MCP `preflight` |
| Change history | MCP `get_change_history` |
| State snapshot (health, disk, drift) | MCP `get_state_snapshot` |
| Secrets (Infisical) | REST API + `oikos secret` CLI |
| Approval tokens | Matrix via notifier |
| Run a command on a host/LXC (policy-gated) | MCP `run` |
| Record a discovered fact/relationship | MCP `update_entity_attributes`, `create_relationship`, `upsert_knowledge` |
Most MCP tools are read-only; a few mutate the knowledge graph (recording
what you learned) or the live infrastructure (`run`),
gated by risk classification and — for `config_mutation`/`destructive`
actions — operator approval. See [AGENTS.md](AGENTS.md#3-the-mcp-server) for
the full tool catalog.
## Authentication
Every API/MCP call needs `Authorization: Bearer <token>` — there is no
unauthenticated path except `POST /api/v1/clients/enroll` and `/healthz`.
Enrollment (below) does not currently hand out a per-client bearer token;
get the shared `OIKOS_MCP_BEARER_TOKEN` from the operator until per-client
token issuance exists.
## Enrollment
Thin client model — no git clone, no sync timer. `bootstrap.sh` fetches only
the agent orientation files and tooling from the raw Gitea URL, then enrolls
via the Oikos API.
To enroll:
```bash
# Run from any machine with mesh connectivity
curl -fsSL https://git.hubris.network/dtoro/oikos/raw/main/bootstrap.sh | sudo bash
# Or with optional tooling:
curl ... | sudo bash -s -- --with-mcp # wire Claude's MCP config
```
This calls `POST /api/v1/clients/enroll` on the Oikos API, which:
1. Validates the entity exists in DB (planned or provisioning state)
2. Validates mesh IP against expected subnets
3. Generates an age keypair and delivers it to the client
4. Creates an Infisical machine identity
5. Transitions the entity to provisioning state
## After enrollment
### What changes on your machine
- `/opt/homelab/` — agent orientation files (CLIENTS.md, AGENTS.md, OIKOS.md)
- `/opt/homelab/tools/` — tooling scripts (checks)
- `/etc/age/key.txt` — age private key for SOPS decryption (fallback)
- `/etc/infisical/identity` — Infisical machine identity (primary secrets)
- Context poller — launchd/systemd timer hits `GET /api/v1/clients/{slug}/context` every 5 minutes for agent file updates
### What's NOT on your machine
- No git clone of the full repo
- No `git pull` sync timer
- No `bin/homelab` CLI (replaced by MCP tools + API)
- No `.sops.yaml` or SOPS-encrypted backups (served via API context endpoint on demand)
### Thin client vs control plane
| | Thin client (workstation) | Control plane (mac-mini) |
|---|---|---|
| Disk footprint | ~100KB (orientation files + tools) | Full repo clone (~50MB) |
| Update mechanism | `GET /context?since=` poll | Git pull + post-pull.sh |
| Source of truth | DB via MCP | DB + local seeds + archive |
| Secrets access | Infisical (primary), age/SOPS served via API (fallback) | Infisical + local SOPS files |
### Your identity
Your identity in the homelab is defined in `inventory.yaml`. Run `hostname`
(Linux) or `scutil --get LocalHostName` (macOS), then look up your entry.
It tells you your role, what services you host, what's mounted, and your
mesh address.
### Source of truth hierarchy
1. **Postgres database** (runtime) — authoritative for entities, knowledge,
signals, ledger. Query via MCP or REST API.
2. **Context poller** — agent files and tooling fetched via API deltas every
5 minutes.
3. **Never guess.** If data is missing, query MCP. If MCP is down, grep the
local `/opt/homelab/` files.
## The context poller
Every 5 minutes, launchd (macOS) or systemd (Linux) hits:
```
GET /api/v1/clients/ws:{hostname}/context?since={last_timestamp}
```
The API returns which agent files, tools, and SOPS config changed since the
last poll. Only changed files are downloaded. This replaces the old
`git pull` with a lightweight HTTP delta.
To trigger manually: run `/opt/homelab/tools/context-poller.sh`.
## Making changes
- **Read state**: use MCP tools or the API
- **Mutate state** (restart, edit config, deploy): classify the action against
policy (query `preflight` MCP tool):
- `read_only` / `reversible_low` — execute directly
- `config_mutation` / `destructive` — request operator approval via
`POST /api/v1/entities/{slug}/activate` (or equivalent lifecycle endpoint)
- **Secrets**: use Infisical (primary) or SOPS (fallback). Never hardcode.
- **Knowledge**: if you observe a discrepancy between docs and live state,
update the DB via the API in the same session.
## MCP endpoint
```
https://mcp.hubris.network/mcp
```
Available tools are listed in [AGENTS.md](AGENTS.md#3-the-mcp-server).
## Communication mode
Apply [.agents/shared/caveman.md](.agents/shared/caveman.md) — terse,
fragment-heavy communication. Drop filler, keep substance.
## Related
- [AGENTS.md](AGENTS.md) — full agent orientation (read this first)
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, risk classes
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide for the Oikos codebase

View File

@@ -1,123 +1,205 @@
# Contributing to the Homelab Wiki
# Contributing to Oikos
## Voice
Developer guide for the Oikos codebase. If you are a homelab client consuming
Oikos, see [CLIENTS.md](CLIENTS.md). If you are an AI agent working on the
repo, see [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks.
## Dev setup
## Page templates
- **Go 1.26+** (see `go.mod` for pinned version)
- **PostgreSQL with TimescaleDB** — the compose stack includes `timescale/timescaledb:2.17.2-pg16`
- **Docker** for the full dev stack
- **Node 22+** for `web/` (the control-room SPA — standalone, not part of the
compose stack or the `oikos` binary)
### Container page (`containers/<id>-<name>.md`)
```bash
# Start dependencies (Postgres + Redis). api/nomos require a shared bearer
# token — no dev-open bypass — so set one even for local dev.
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile dev up -d
```markdown
# <id> — `<name>`
# Run all tests
make test
One-sentence purpose.
# Run integration tests (needs compose Postgres)
make test-db
## At a glance
- **Hostname:** `<name>`
- **IP:** `192.168.8.x`
- **Privilege:** privileged | unprivileged
- **Resources:** N cores / M GiB RAM / D GiB rootfs
- **Mounts:** `/mnt/library``/mnt/library` (if any)
- **Public hostname:** `<sub>.hubris.network` (if proxied)
# Build the binary
make build
## Role
What it does, what it talks to.
# SPA dev server (proxies to api/nomos, injecting the same token)
cd web && OIKOS_API_TOKEN=dev-token npm run dev
## Service / port map
| Service | Listen | Notes |
# Desktop app (macOS)
make desktop # build .app bundle
make install # build + install to /Applications
./cmd/desktop/build/bin/oikos-desktop.app/Contents/MacOS/oikos-desktop # run from terminal to see logs
```
## Storage / config paths
### Desktop app auth
## Auto-deploy
(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md)
The desktop app uses the same API as the browser SPA. First launch:
1. Enter `https://oikos.hubris.network` as Server URL
2. **Login with Authentik** tab → opens system browser → authenticate
3. Callback page shows token → copy → paste into Token tab → Connect
4. Token is persisted to the macOS keychain — subsequent launches skip setup
The app stores credentials via `github.com/zalando/go-keyring` (service: `com.hubris.oikos-desktop`).
### Desktop app auto-update
- Checks Gitea releases every 6 hours
- System tray → **Check for Updates** triggers an immediate check
- Download, extract, replace the app in `/Applications`, and relaunch
- Versions are compared against the `version` var in `main.go`, injected from the repo `VERSION` file at link time (`make desktop` passes `-ldflags "-X main.version=$(cat VERSION)"`)
## Project structure
```
cmd/desktop/ Wails v3 desktop app (macOS + Linux)
main.go Thin shell: webview, system tray, notifications, auto-update
wails.json Wails project config
entitlements.plist macOS code-signing entitlements
icon.png System tray icon (embedded)
icon.icns App bundle icon (white logo on black rounded rect)
Taskfile.yml Wails v3 build tasks
Info.plist.template macOS bundle metadata
cmd/oikos/ Single-binary entry point
cmd/nomos/ Nomos MCP client gateway
cmd/webhook/ Gitea deploy-webhook receiver (push-to-deploy on mac-mini)
internal/ All Go packages
httpapi/ REST + MCP server (OpenAPI-generated)
mcp/ MCP tool implementations
db/ Connection pool, migrations, seeds, sqlc queries
scheduler/ Observe loop, probes, signals
actuator/ SSH execution
learning/ Pattern recognition, anomaly detection
notifier/ Matrix notifications, approval tokens
policy/ Risk classifier
secrets/ Infisical + SOPS backend
domain/ Core types: entities, approvals, signals, patterns
ontology/ Type hierarchy, relationship validation
knowledge/ Knowledge YAML seed ingestion
web/ Control-room SPA (Svelte 5) — standalone, not embedded
in the oikos binary; see plans/2026-07-12-wails-desktop-app.md
api/openapi.yaml API contract — the source of truth for endpoints
migrations/ Forward-only SQL migrations (TimescaleDB)
seeds/ Bootstrap YAML: ontology, inventory, policy, knowledge
compose/ Dockerfiles + Caddy config
scripts/ Deploy, watchdog, rollback
checks/ Host health-check scripts run over SSH by the scheduler
tools/ Client auto-setup scripts (checks)
nomos/ Nomos config, persona, skills
.agents/ Agent instruction files + skills
plans/ Design documents
docs/adr/ Architecture decision records
docs/operations/ Runbooks (rollback, etc.)
```
## Commands
| Command | Purpose |
|---------|---------|
| `make build` | Build `oikos` binary |
| `make test` | Run all tests with race detection |
| `make test-db` | Run integration tests against compose Postgres |
| `make lint` | `go vet` + `golangci-lint` |
| `make generate` | Regenerate OpenAPI + sqlc code |
| `make generate-check` | CI drift guard — fail if generated code is stale |
| `make migrate` | Apply DB migrations |
| `make seed` | Ingest seeds into DB |
| `make export` | Export DB state to YAML seeds |
| `make dev` | Start compose dev stack |
| `make clean` | Remove binary + test cache |
| `make ui` | Build the SPA (`web/dist/`) |
| `make deploy-ui` | Build + deploy the SPA to the Caddy host |
| `make desktop` | Build the Wails desktop app for the current platform |
| `make desktop-package` | Build + package (zip on macOS, tar.gz on Linux) |
| `make install` | Build + install to `/Applications` (macOS) |
| `make webhook` | Build `cmd/webhook` (deploy-webhook receiver) |
| `make tidy` | `go mod tidy` |
## Conventions
### APIs are OpenAPI-first
The REST API is defined in `api/openapi.yaml`. Server code is generated with
`oapi-codegen` into `internal/httpapi/gen/`. To add an endpoint:
1. Add the path + schema to `api/openapi.yaml`
2. Run `make generate`
3. Implement the handler in `internal/httpapi/impl.go`
4. Add tests in `internal/httpapi/api_test.go`
Never hand-edit `internal/httpapi/gen/api.gen.go`.
### Database access is sqlc-first
SQL queries live in `internal/db/queries/*.sql`. Go code is generated with
`sqlc` into `internal/db/sqlcgen/`. Config in `sqlc.yaml`.
- Queries target pgx/v5 with UUID + timestamptz overrides
- Never hand-edit generated sqlc code
### Migrations are forward-only
SQL migrations live in `migrations/` as `NNN_name.up.sql`. There are no down
migrations (see [ADR 0008](docs/adr/0008-forward-only-migrations.md)).
Migrations are idempotent where possible (`IF NOT EXISTS`, `DO $$` blocks).
To add a migration:
1. Create `migrations/NNN_name.up.sql` with the next sequence number
2. Write the DDL
3. Run `make migrate` to apply
### Seeds are DB-generated
`seeds/*.yaml` are the bootstrap files used by `oikos seed`. After making
changes via the API, run `make export` to regenerate the seed files. These
files are version-controlled and serve as DR fallback.
### Writing style
Follow [.agents/shared/writing-style.md](.agents/shared/writing-style.md).
Documentation is reference prose, not marketing. Banned vocabulary includes
"robust", "seamless", "leverage", "utilize", "delve", "cutting-edge".
### Risk classification
Every mutation is classified against `seeds/policy.yaml` before execution.
Four risk classes: `read_only`, `reversible_low`, `config_mutation`,
`destructive`. The classifier can only lower autonomy relative to policy,
never raise it. When in doubt, escalate.
## CI
Gitea Actions runs on push to `main` and pull requests (`ci.yml`):
1. `go vet` + `golangci-lint` + `govulncheck`
2. Generated code drift check (`make generate-check`)
3. Build (`go build ./...`)
4. Test with race detector + coverage
5. Docker build verification (no push)
Coverage gates: policy + learning packages ≥ 80%, others ≥ 60%.
## PR workflow
1. Create a branch from `main`
2. Make changes, write tests
3. Run `make lint test generate-check`
4. Commit with a message following: problem → change → risk → verification
5. Push to Gitea; CI gates PRs on green
## Secrets
Secrets are managed by Infisical (primary) with SOPS as DR fallback. Never
hardcode secrets. Use environment variables from `.env` for local dev.
The `.env` and `.infisical-credentials` files are gitignored.
## Related
- [Caddy](121-caddy.md) (if proxied)
- [DNS](../infrastructure/dns.md) (if has subdomain)
- [Authentik](124-authentik.md) (if SSO)
- ...
## Changelog
### YYYY-MM-DD — short title
What changed, why, link to investigation if any.
```
### Cross-cutting page (`infrastructure/<topic>.md`)
```markdown
# <Topic>
One-sentence summary.
## Why
Design rationale — what it replaces, what it solves.
## Components
Where it runs, what files matter.
## How to apply / use
Recipes.
## Gotchas
## Related
Links to nodes that host or depend on this.
## Changelog
```
### Plan (`plans/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Goal
What this change achieves and why.
## Current topology / state
Diagram or description of what exists now.
## Target topology / state
What it looks like after.
## Pre-flight checklist
## Step-by-step procedure
## Verification
## Post-migration
Changelog entries to write, index status to update.
```
### Investigation (`investigations/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Summary
1-3 sentences.
## Timeline
## Root cause
## Mitigations applied
## Open questions
```
## Linking discipline
- Every container page links to every cross-cutting page it participates in.
- Every cross-cutting page lists the nodes that participate.
- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages.
## Changelog hygiene
- Reverse-chronological (newest first).
- One entry per discrete change, even if you make several in one day.
- If a change spans nodes, repeat the entry on each affected page (different perspective is fine).
- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them.
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
- [CLIENTS.md](CLIENTS.md) — for homelab clients consuming Oikos
- [docs/adr/](docs/adr/) — architecture decision records

View File

@@ -1,83 +0,0 @@
# HERMES.md — Agent persona for homelab clients
This file is the canonical agent persona for **all** AI agents running on
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
conventions, and the source-of-truth hierarchy.
## Source of truth
The homelab-context repo at `/opt/homelab-context/` is the single source of
truth for:
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
- Service endpoints and credentials (via `homelab secret`)
- Agent behaviour and conventions
- Everything in this file
When in doubt, check `/opt/homelab-context/` first.
## Agent type — how this file gets loaded
| Agent | Loading mechanism |
|-------|------------------|
| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints``/opt/homelab-context/HERMES.md` |
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
homelab-context repo instead. Changes propagate to all clients on the next
sync (`sudo homelab sync`).
---
## Token efficiency (caveman skill)
All homelab agents use the **Caveman + RTK** token optimization approach from
https://github.com/adityahimaone/hermes-agent-rtk-caveman.
### Before running any CLI command, ask:
1. **Is there a caveman wrapper equivalent?** Use the wrapper for token-efficient
output. Available wrappers (installed at `~/bin/caveman_wrapper.sh`):
- `~/bin/caveman_wrapper.sh git-status` — compact git status
- `~/bin/caveman_wrapper.sh git-log [n]` — compact git log
- `~/bin/caveman_wrapper.sh lint [target]` — compact lint results
- `~/bin/caveman_wrapper.sh test-results [cmd]` — compact test results
2. **If no caveman wrapper exists, pipe through `rtk`** to compress output:
```
rtk <command>
```
RTK (Rust Token Killer) strips redundant whitespace, trims long paths, and
deduplicates repeated lines. This reduces token usage by 60-90% on CLI
operations.
3. **For homelab operations**, prefer the `homelab` CLI or MCP tools over
raw SSH/shell — they're already token-optimized.
### Templates
Caveman templates live at `~/templates/`:
- `git_status.txt` — compact git status format
- `git_log.txt` — compact git log format
- `lint_results.txt` — compact ESLint format
- `test_results.txt` — compact vitest/jest format
### When to skip caveman/rtk
- Interactive commands (editors, prompts) — let human-readable output pass
- Commands with no output — skip entirely
- When you need the exact raw output for post-processing
### Verification
```bash
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
```
## Important note for Hermes agents
If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
can verify the content matches or re-provision by running:
bash /opt/homelab-context/tools/setup-hermes-soul.sh

100
Makefile Normal file
View File

@@ -0,0 +1,100 @@
.PHONY: build webhook test test-db lint generate generate-check dev migrate seed export clean tidy ui desktop desktop-package install
BINARY := bin/oikos
GO ?= go
build:
mkdir -p bin
$(GO) build -o $(BINARY) -tags timetzdata ./cmd/oikos
webhook:
$(GO) build -o webhook -tags timetzdata ./cmd/webhook
test:
$(GO) test -race -cover ./...
# Integration tests against the compose Postgres (starts it if needed)
test-db:
docker compose up -d postgres
@sleep 3
OIKOS_TEST_DATABASE_URL="postgres://oikos:$${OIKOS_DB_PASSWORD:-oikos_dev}@localhost:5432/oikos?sslmode=disable" \
$(GO) test -race -count=1 ./internal/db/ ./internal/httpapi/ ./internal/mcp/
lint: vet golangci govulncheck
vet:
$(GO) vet ./...
golangci:
@command -v golangci-lint >/dev/null 2>&1 && golangci-lint run --config .golangci.yml || echo "golangci-lint not installed — see https://golangci-lint.run/usage/install/"
govulncheck:
@command -v govulncheck >/dev/null 2>&1 && govulncheck ./... || echo "govulncheck not installed — run: go install golang.org/x/vuln/cmd/govulncheck@latest"
.PHONY: lint vet golangci govulncheck
generate:
$(GO) run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.4.1 \
-config api/codegen.yaml api/openapi.yaml
$(GO) run github.com/sqlc-dev/sqlc/cmd/sqlc@v1.29.0 generate
# CI drift guard: regenerate and fail if the committed output changed.
generate-check: generate
@git diff --exit-code -- internal/httpapi/gen internal/db/sqlcgen \
|| (echo "generated code is stale — run 'make generate' and commit" && exit 1)
migrate:
$(GO) run ./cmd/oikos migrate
seed:
$(GO) run ./cmd/oikos seed
export:
$(GO) run ./cmd/oikos export
dev:
docker compose --profile dev up -d
# Local sanity-check build of the SPA. Not embedded in the oikos binary
# (plans/2026-07-12-wails-desktop-app.md 0.1) — deploys as its own
# container (compose/web/Dockerfile) via `docker compose --profile full
# up -d web`, same push-to-main pipeline as everything else.
ui:
cd web && npm run build
desktop: ui ## Build the Wails desktop app for the current platform
rm -rf cmd/desktop/frontend/dist
mkdir -p cmd/desktop/frontend/dist
cp -r web/dist/* cmd/desktop/frontend/dist/
cd cmd/desktop && CGO_ENABLED=1 go build -ldflags "-X main.version=$$(cat ../VERSION)" -o build/bin/Oikos .
desktop-package: desktop ## Build + package the desktop app (zip on macOS, tar.gz on Linux)
@case $$(uname -s) in \
Darwin) \
APP="cmd/desktop/build/bin/Oikos.app"; \
rm -rf "$$APP"; \
mkdir -p "$$APP/Contents/MacOS"; \
mkdir -p "$$APP/Contents/Resources"; \
cp cmd/desktop/build/bin/Oikos "$$APP/Contents/MacOS/Oikos"; \
cp cmd/desktop/icon.icns "$$APP/Contents/Resources/icon.icns"; \
sed "s/\$$(VERSION)/$$(cat VERSION)/" cmd/desktop/Info.plist.template > "$$APP/Contents/Info.plist"; \
cd cmd/desktop/build/bin && zip -r oikos-desktop-darwin-$$(uname -m).zip Oikos.app ;; \
Linux) \
cd cmd/desktop/build/bin && tar czf oikos-desktop-linux-$$(uname -m).tar.gz Oikos ;; \
esac
@echo "Package: cmd/desktop/build/bin/"
install: desktop-package ## Install to /Applications
rm -rf /Applications/Oikos.app
cp -r cmd/desktop/build/bin/Oikos.app /Applications/
@echo "Installed to /Applications/Oikos.app"
clean:
rm -f $(BINARY)
rm -rf bin
rm -rf cmd/desktop/build
rm -rf cmd/desktop/frontend/dist
$(GO) clean -testcache
tidy:
$(GO) mod tidy

206
README.md
View File

@@ -1,76 +1,162 @@
# Homelab Wiki — `hubris`
# Oikos
Living documentation for the **hubris** Proxmox homelab. Every node, every cross-cutting system, and every meaningful incident is its own page; pages are linked so you can start anywhere and walk the graph.
Agentic homelab operating system written in Go. Single binary (`cmd/oikos`),
Docker-deployed on mac-mini, with a standalone Nomos MCP agent gateway
(`cmd/nomos`). Manages the **hubris** Proxmox homelab autonomously — observes
state, classifies actions against policy, executes approved procedures over SSH,
learns from outcomes, and escalates when uncertain.
> Last refreshed against live state: **2026-04-28**.
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md).
**For client machines:** see [CLIENTS.md](CLIENTS.md).
**For developers:** see [CONTRIBUTING.md](CONTRIBUTING.md).
## Map
## Quick start
### Hosts
- [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77`
```bash
# Dev stack (postgres + api + scheduler + notifier). The api/nomos
# services need a shared token — every route requires a real bearer
# credential, there's no dev-open bypass.
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile dev up -d
### VMs
- [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation)
- [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS
# Full stack (adds Nomos agent gateway)
OIKOS_MCP_BEARER_TOKEN=dev-token docker compose --profile full up -d
### LXC containers
See the full table in [`containers/index.md`](containers/index.md). Quick links:
# Build standalone binary
go build -o bin/oikos -tags timetzdata ./cmd/oikos
| ID | Name | IP | Role |
| --- | ---------------- | --------------- | --------------------------------------------- |
| 101 | [jellyfin](containers/101-jellyfin.md) | 192.168.8.206 | Media server |
| 102 | [nfs-export](containers/102-nfs-export.md) | 192.168.8.200 | NFSv4 re-export of /mnt/library for ZimaOS |
| 103 | [paperless](containers/103-paperless.md) | 192.168.8.130 | Document mgmt |
| 104 | [gitea](containers/104-gitea.md) | 192.168.8.121 | Git server |
| 105 | [apps](containers/105-apps.md) | 192.168.8.205 | Docker host (Artifacto / Booklore / PlantUML / Portainer / WriteFreely) |
| 114 | [nextcloud](containers/114-nextcloud.md) | 192.168.8.224 | Personal cloud |
| 118 | [elementsynapse](containers/118-elementsynapse.md) | 192.168.8.239 | Matrix Synapse |
| 119 | [sophia](containers/119-sophia.md) | 192.168.8.157 | Sophia |
| 120 | [mule-images](containers/120-mule-images.md) | 192.168.8.136 | Mule-image / mulita photos |
| 121 | [caddy](containers/121-caddy.md) | 192.168.8.175 | Reverse proxy |
| 122 | [arriman](containers/122-arriman.md) | 192.168.8.132 | Docker host (\*arr stack) |
| 124 | [authentik](containers/124-authentik.md) | 192.168.8.180 | SSO + split-horizon DNS |
| 126 | [plato](containers/126-plato.md) | 192.168.8.190 | Plato (notes/discovery workspace) |
# Run all roles in one process (dev mode)
OIKOS_DATABASE_URL="postgres://oikos:oikos_dev@localhost:5432/oikos?sslmode=disable" \
OIKOS_API_TOKEN=dev-token \
go run ./cmd/oikos all
### Cross-cutting infrastructure
- [DNS — split-horizon](infrastructure/dns.md)
- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md)
- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md)
- [Monitoring — Hermes health watchdog](infrastructure/monitoring.md)
- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md)
- [SSH access](infrastructure/ssh-access.md)
- [Backups — restic on external drive (disabled)](infrastructure/backups.md)
- [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md)
- [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md)
- [Homelab context distribution](infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance
# Control-room SPA (separate from the Go binary — see web/)
cd web && OIKOS_API_TOKEN=dev-token npm run dev # http://localhost:5173
```
### Investigations
Time-stamped incident notes / experiments in [`investigations/`](investigations/index.md).
## Architecture
### Operations
- [Command cheatsheet](operations/commands.md)
- [Agent enrollment](operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system
```
┌──────────────────────────────────┐
│ mac-mini (Docker) │
│ │
Workstation ─── │ nomos (8092) ──MCP── api (8090) │
(mesh) │ MCP gateway REST + MCP │
│ │
│ scheduler ── notifier ── postgres │
│ (observe) (Matrix) (Timescale)│
└──────────────────────────────────┘
```
## Conventions
| Component | Port | Role |
|-----------|------|------|
| `oikos api` | 8090 | REST API + MCP server (tool list in [AGENTS.md §3](AGENTS.md#3-the-mcp-server)) |
| `oikos scheduler` | — | Probe runner, signal lifecycle, metrics |
| `oikos notifier` | — | Approval tokens, Matrix alerts |
| `nomos serve` | 8092 | MCP client gateway, query routing |
- **Each node page** ends with a `## Changelog` section. Reverse-chronological. Entry format:
```
### YYYY-MM-DD — short title
one or two lines on what changed and why.
```
- **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug.
- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog.
- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
- **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not.
## Phases
## Maintaining this wiki
| Phase | Status | Description |
|-------|--------|-------------|
| 1 — Ontology + DB | ✅ | TimescaleDB, migrations, seeds, blast_radius |
| 2 — API | ✅ | OpenAPI-first REST + MCP, auth, SSE, audit |
| 3 — Control loop | ✅ | Scheduler, actuator, learning, classifier, notifier |
| 4 — Nomos agent | ✅ | Standalone MCP client gateway, agent activity |
| 5 — Secrets | ✅ | Infisical backend + SOPS fallback, rotation runbooks |
| 6 — Deploy | ✅ | CI pipeline, cutover checklist, watchdog, rollback |
When you change a node:
1. Update the relevant page (config snapshot, ports, mounts).
2. Add a changelog entry at the bottom of that page.
3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry.
4. If it's an incident, add an entry to [`investigations/`](investigations/index.md).
Full plan: [plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md).
## See also
## Operations
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — page templates and tone
### API endpoints
```bash
curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
http://localhost:8090/api/v1/entities?type=service # fleet
curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
http://localhost:8090/api/v1/health # fleet health
curl -H "Authorization: Bearer $OIKOS_API_TOKEN" \
http://localhost:8090/api/v1/agent-activity # agent log
```
### Nomos queries
```bash
# Structured tool call
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
-d '{"tool":"get_blast_radius","args":{"entity_id":"service:authentik"}}'
# Natural language
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
-d '{"query":"what depends on authentik?"}'
```
### CLI
```bash
oikos migrate # apply DB migrations
oikos seed # ingest ontology/inventory/policy seeds
oikos export # export DB state to YAML
oikos api # serve REST + MCP
oikos scheduler # run observe loop
oikos notifier # run notification loop
oikos all # all roles in one process
oikos secret list # enumerate SOPS secrets
oikos secret migrate # SOPS → Infisical
```
### Web UI
`web/` is a standalone Svelte 5 SPA — not embedded in the `oikos` binary, not
part of `docker-compose.yml`. It talks to `api`/`nomos` over HTTP with a
bearer token entered on first launch (see `web/src/pages/Config.svelte`).
Build with `make ui`, deploy with `make deploy-ui` (Caddy serves the static
output). A native desktop wrapper exists at `cmd/desktop/` — see
[plans/done/2026-07-12-wails-desktop-app.md](plans/done/2026-07-12-wails-desktop-app.md).
## Repo layout
```
cmd/oikos/ Go entry point — single binary
cmd/nomos/ Nomos MCP client gateway
cmd/webhook/ Gitea deploy-webhook receiver (push-to-deploy on mac-mini)
cmd/desktop/ Wails desktop wrapper around the SPA
internal/ Go packages (actuator, checkdefaults, config, db, domain,
httpapi, knowledge, learning, mcp, notifier, observability,
ontology, policy, safego, scheduler, secrets)
web/ Control-room SPA (Svelte 5) — standalone, not embedded
api/openapi.yaml API contract (OpenAPI 3.1)
migrations/ Forward-only SQL migrations (TimescaleDB)
seeds/ Bootstrap YAML (ontology, inventory, policy, knowledge)
compose/ Dockerfiles + Caddy config
scripts/ Deploy, watchdog, verification, rollback
checks/ Host health-check scripts run over SSH by the scheduler
tools/ Client auto-setup scripts (checks)
ssh/ Deploy keys + authorized_keys management
vps/ Caddy/TURN config templates for the netbird VPS
nomos/ Nomos config, persona, skills
.agents/ Agent instruction files, shared conventions, skills
archive/ Historical reference (legacy wiki, plans, SOPS backups)
plans/ Design documents (active + done)
docs/adr/ Architecture decision records
docs/operations/ Runbooks (rollback, etc.)
```
## For agents
See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference:
- **Source of truth:** DB (runtime) then seeds (bootstrap). Old wiki is
archived at `archive/knowledge/` — use MCP `search_knowledge` instead.
- **Mutations:** classify against policy, request approval for
`destructive`/`config_mutation`
- **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode
## Related
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
- [CLIENTS.md](CLIENTS.md) — client onboarding guide
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide
- [plans/](plans/) — design documents and cutover checklist
- [docs/adr/](docs/adr/) — architecture decision records

1
VERSION Normal file
View File

@@ -0,0 +1 @@
0.7.12

8
api/codegen.yaml Normal file
View File

@@ -0,0 +1,8 @@
# oapi-codegen config — `make generate` regenerates internal/httpapi/gen.
package: gen
output: internal/httpapi/gen/api.gen.go
generate:
models: true
chi-server: true
strict-server: true
embedded-spec: true

3303
api/openapi.yaml Normal file

File diff suppressed because it is too large Load Diff

7
api/redocly.yaml Normal file
View File

@@ -0,0 +1,7 @@
# Redocly lint config for api/openapi.yaml (CI runs: redocly lint api/openapi.yaml)
extends:
- recommended
rules:
# Every operation declares `default` → RFC 9457 problem+json instead of
# enumerating each 4XX (plan R3-3); oapi-codegen handles `default` fine.
operation-4xx-response: off

View File

@@ -12,7 +12,7 @@
## Problem statement
The Technitium DHCP server on [CT 107](containers/107-dns.md) serves `192.168.8.100192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
The Technitium DHCP server on [CT 107](../../knowledge/wiki/containers/107-dns.md) serves `192.168.8.100192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
| Host | IP | Inside pool? |
|---|---|---|

View File

@@ -0,0 +1,358 @@
# Assessment: Which nodes can move to `strong`
## Executive summary
hubris is **memory-starved**: 28 GiB RAM, 71.9 GiB allocated across 18 LXC + 2 VM
(2.5× overcommit), 10 GiB swap in active use. strong sits **completely empty**
28 GiB RAM, 25 GiB free, 0 guests, 2.7 TiB unused storage. The single most
effective decongestion move is to shift guests off hubris onto strong.
This document assesses every guest for move-readiness, grouped by constraints
(library dependency, GPU, core-infra status), and proposes a phased migration
that does **not** require the physical library-SSD move (the blocker of the
original plan) — library access from strong is provided via NFS from hubris.
---
## Current resource state (live, 2026-07-05)
### hubris — overloaded
| Resource | Capacity | Allocated (all guests) | Actual use | Status |
|----------|----------|----------------------|------------|--------|
| RAM | 28 GiB | 70.2 GiB (2.5× overcommit) | 18 GiB used + 10 GiB swap | ⚠️ heavy swap pressure |
| CPU | 12 vCPU (6c/12t) | 43 vCPU (3.6× overcommit) | ~43% scaling MHz | OK (shares) |
| local-lvm | 856 GiB | 582 GiB allocated (29% thin) | — | OK |
| library (lvmthin) | 3.7 TiB | — | 1.2 TiB used (34%) | OK, 2.3 TiB free |
### strong — empty, ready
| Resource | Capacity | Used | Status |
|----------|----------|------|--------|
| RAM | 28 GiB | 2.3 GiB (host only) | 25 GiB free |
| CPU | 16 vCPU (8c/16t) | idle (0.08 load) | 100% free |
| local-lvm | 856 GiB | 0 | empty |
| ludo-lvm | 1.8 TiB | 0 | empty |
| Guests | — | 0 LXC, 0 VM | nothing running |
### Network topology constraint
```
Fritz!Box (192.168.178.1)
└── SODOLA 2.5G switch
├── hubris eno1 → vmbr1 (192.168.178.10) → vmbr0 (192.168.8.0/24)
│ └── all 20 guests on 192.168.8.x
└── strong vmbr0 (192.168.178.181)
└── no internal bridge yet, guests would be on 192.168.178.x
```
strong reaches `192.168.8.0/24` via the Fritz static route through hubris.
Guests on strong get `192.168.178.x` IPs unless we add an internal bridge
on strong (Phase 0 prerequisite — see below).
---
## Per-guest assessment
### Tier 1 — Move immediately (no library dependency, no core-infra)
These guests mount **no** `/mnt/library` and are not part of the core
infrastructure spine (caddy/dns/auth/mcp). They are the easiest wins.
| ID | Name | Cores | RAM | Library? | GPU? | Notes |
|----|------|-------|-----|----------|------|-------|
| 118 | elementsynapse | 2 | 4 GiB | ❌ | ❌ | Matrix homeserver. Public via Caddy (`matrix.hubris.network`). Only change: Caddy backend IP. **Easiest move in the fleet.** |
| 129 | house | 2 | 3 GiB | ❌ | ❌ | Yuvomi family planner (Docker). No public Caddy route yet (uses VPS traefik directly). Self-contained. |
**Combined RAM freed from hubris: 7 GiB.** No NFS, no library, no GPU.
### Tier 2 — Move with library NFS (high resource consumers)
These are the heaviest guests and the original migration plan's primary
targets. They mount `/mnt/library` and two use the iGPU. Moving them
requires an NFS export from hubris → strong (reverse of the original
plan's direction, since the physical SSD hasn't moved).
| ID | Name | Cores | RAM | Library? | GPU? | I/O profile | Notes |
|----|------|-------|-----|----------|------|-------------|-------|
| 120 | mule-images | 6 | 12 GiB | ✅ mp0 | ✅ iGPU | Write-heavy (photo processing) | **#1 RAM consumer.** Strong has Radeon 680M iGPU (VAAPI works). |
| 122 | arriman | 4 | 8 GiB | ✅ mp0 | ❌ | Write-heavy (downloads) | *arr stack + qbit + sab. Mounts library for download writes. |
| 101 | jellyfin | 4 | 8 GiB | ✅ mp0 | ✅ iGPU | Read-heavy sequential | Media streaming + transcode. Strong 680M handles VAAPI. |
**Combined RAM freed: 28 GiB.** This alone would eliminate hubris's swap
pressure entirely.
### Tier 3 — Could move, low urgency
| ID | Name | Cores | RAM | Library? | Notes |
|----|------|-------|-----|----------|-------|
| 130 | grimmory | 1 | 2 GiB | ✅ mp0 | Book library (Docker). Migrated from apps LXC recently. |
| 131 | teddycloud | 1 | 1 GiB | ✅ mp0 | New (not in inventory.yaml yet). |
| 132 | rclone | 1 | 2 GiB | ✅ mp0 (ro) | Backup container. Read-only library mount. |
| 128 | trmnl | 1 | 768 MiB | ❌ | TRMNL middleware. No library. Could move but tiny. |
| 119 | sophia | 2 | 1 GiB | ✅ mp0 | Workshop. Light use. |
### Stay on hubris (core infrastructure)
| ID | Name | Cores | RAM | Why it stays |
|----|------|-------|-----|--------------|
| 121 | caddy | 1 | 512 MiB | Reverse proxy — terminates all `*.hubris.network`. Must stay on hubris for LAN-side reachability. **Needs backend IP updates** when guests move. |
| 107 | dns | 1 | 1 GiB | Technitium DNS, split-horizon. Core. |
| 106 | auth-outpost | 1 | 512 MiB | Authentik SSO enforcement. Core. |
| 105 | apps | 2 | 4 GiB | homelab MCP + secrets-issuance + artifacto. Core infra. Mounts library. |
| 104 | gitea | 1 | 1 GiB | Git server. NFS would hurt git lock/stat perf. Mounts library (bare repos). |
| 103 | paperless | 2 | 3 GiB | Document archive. Moderate I/O, OCR writes. Mounts library. |
| 102 | nfs-export | 1 | 512 MiB | Exports library to zimaos via NFS. Must stay with the physical library. |
| 100 | zimaos | 4 | 8 GiB (VM) | NAS frontend eval. Already NFS-mounts library from 102. |
| 108 | haos | 2 | 4 GiB (VM) | Home Assistant OS. Hardware access, low latency. |
---
## Constraints & prerequisites
### 1. Network — strong needs an internal bridge (Phase 0)
strong currently has only `vmbr0` on `192.168.178.0/24`. Guests created there
get household-LAN IPs, not homelab-subnet IPs. Two options:
- **Option A (recommended):** Add `vmbr1` on strong as a portless internal
bridge with a `192.168.8.x/24` address (e.g. `192.168.8.3`). Route between
strong's `vmbr0` and `vmbr1` the same way hubris does. Guests go on `vmbr1`
and get `192.168.8.x` IPs — transparent to Caddy, DNS, and inter-LXC refs.
Requires adding a static route on Fritz (or relying on hubris's existing
route — strong would need IP forwarding + a route to 192.168.8.0/24 via vmbr1).
- **Option B (simpler, messier):** Put guests on `192.168.178.x` directly.
Caddy can still reach them (hubris routes to 192.168.178.0/24). But DNS
records, inter-LXC references, and firewall rules all assume `192.168.8.x`.
More config churn per guest.
### 2. Storage — rootfs migration (no shared storage)
`local-lvm` is per-node (not shared). Moving an LXC requires either:
- `vzdump` → restore on strong (clean, but needs temp disk space + downtime)
- `rsync` the rootfs to a new LXC on strong (faster for large rootfs like 120's 100G)
- `pct migrate` only works with shared storage — **not applicable here**
For VMs (100, 108): `qm migrate` also needs shared storage. Not moving VMs.
### 3. Library access — NFS from hubris to strong
Since the physical library SSD is still on hubris, strong's guests that need
`/mnt/library` must NFS-mount it from hubris. Options:
- **Export from hubris host directly** (simplest): add `/mnt/library` to
`/etc/exports` on hubris with the same squash params as LXC 102
(`rw,all_squash,anonuid=33,anongid=10000,no_subtree_check`). Mount on strong
at `/mnt/library`. Strong's guests bind-mount it just like hubris's guests do.
- **Use existing nfs-export LXC 102**: strong NFS-mounts from `192.168.8.200`
(LXC 102). This already has the right squash config. Less host-level change.
**This is the path of least resistance.**
### 4. GPU — iGPU passthrough on strong
strong has a Ryzen 7 PRO 6850U with Radeon 680M iGPU. For jellyfin (VAAPI
transcoding) and mule-images (photo processing), we need:
- `/dev/dri/renderD128` passed to the LXC (`lxc.cgroup2.devices.allow` +
`lxc.mount.entry` or Proxmox's `dev0:` passthrough)
- `video` / `render` group membership inside the container
- Confirm `amdgpu` driver loads on strong's host kernel (it should — same APU family)
### 5. Quorum — 2-node cluster, no QDevice
Moving guests to strong does NOT fix the quorum issue but **reduces blast
radius**: if hubris reboots (its known thermal instability), the guests on
strong keep running independently. Consider adding a QDevice as a separate
follow-up — it's orthogonal to this migration.
---
## Revised migration phases
The original plan's NFS-over-LAN approach has been superseded. Instead,
**media library data moves to ludo-lvm** on strong so migrated guests access
it as a local ext4 mount. Data is split by origin:
```
hubris (stays): library SSD (3.7T, 1.2T used)
└── /mnt/library/{documents,images,cloud,homecloud,notes,repos,sophia}
↑ user-generated content (docs, photos, cloud sync, notes, repos, workshop)
strong (moves): ludo-lvm (1.8T, 0 used at start)
└── /mnt/media_local ← 1.5T thin volume
└── {downloads,movies,music,tv,anime,books}
↑ non-user-generated content (media arr stack, book library)
```
| Category | Stays on hubris | Moves to strong |
|----------|----------------|-----------------|
| Media | — | downloads (25G), movies (51G), music (29G), tv (30G), anime (206G) |
| Books | — | books (2.6G) |
| Docs/Photos | documents (249M), images (4K) | — |
| Cloud sync | cloud (287G), homecloud (367G) | — |
| Personal | notes (6.7M), repos (84M), sophia (151G) | — |
| **Total** | **~805G** | **~344G** |
ludo-lvm (1.8T) fits all media + books with ~1.15T headroom for growth.
hubris library SSD (3.7T, 1.2T used) retains the user-generated content.
Both sides keep their data local — no cross-node NFS needed for daily I/O.
---
### Phase 2a — Prepare ludo-lvm on strong
1. Create a ext4 filesystem on ludo-lvm for media:
```bash
lvcreate -n media -L 1.5T ludo-lvm
mkfs.ext4 /dev/ludo-lvm/media
```
2. Mount at `/mnt/media_local` on strong, add to `/etc/fstab`
3. rsync media directories from hubris → strong:
```bash
rsync -av --progress /mnt/library/{movies,tv,anime,downloads,music,books} strong:/mnt/media_local/
```
### Phase 2b — Migrate arriman (122) to strong
1. Stop arriman on hubris, dump rootfs (24G)
2. Restore on strong with IP `192.168.8.245/28` on vmbr1
3. Mount `/mnt/media_local` → `/mnt/library` via mp0 (downloads land locally)
4. Update Caddy: jellyseerr/qbit/sab backends → new IP
5. Update inventory.yaml
### Phase 2c — Migrate jellyfin (101) to strong
1. Stop jellyfin on hubris, dump rootfs (16G)
2. Restore on strong with IP `192.168.8.246/28` on vmbr1
3. Pass `/dev/dri/renderD128` + `/dev/dri/card0` (Radeon 680M + RX 7600)
4. Mount `/mnt/media_local` → `/mnt/library` via mp0 (media reads locally)
5. Update Caddy: `media.hubris.network` → new IP
6. Reinstall `sso-inject.js` in web dir (lost on every apt upgrade)
7. Test VAAPI transcoding, SSO login, media playback
### Phase 2d — Migrate grimmory (130) to strong
1. Stop grimmory on hubris, dump rootfs (16G)
2. Restore on strong with IP `192.168.8.247/28` on vmbr1
3. Mount `/mnt/media_local` → `/mnt/library` via mp0 (books read locally)
4. Update Caddy: `books.hubris.network` → new IP
5. Update inventory.yaml
6. Test: book browsing, calibre-web access
### No NFS export needed
With the data split by origin, hubris guests that only need user-generated
content (documents, images, cloud, repos, sophia) still access them from the
original library SSD — no cross-node NFS required. The two sides are
independent.
**Result after Phase 2: hubris frees 26 GiB RAM (4 migrated guests) + 344G of
library I/O burden. Strong becomes the media/books powerhouse.**
---
### Phase 3 — Migrate mule-images (120) to strong
Move photo management (12 GiB RAM, 6 cores, iGPU) last because it needs:
- `/mnt/library` access (now NFS from strong — already set up in Phase 2d)
- `/dev/dri/renderD128` (Radeon 680M — confirm VAAPI compatibility first)
Steps:
1. Stop mule-images on hubris, rsync the 100G rootfs to strong (faster than vzdump)
2. Restore on strong with IP on vmbr1
3. Pass Radeon 680M iGPU
4. Reconfigure library paths → `/mnt/media_local` (or keep NFS mount)
5. Update Caddy: `photos.hubris.network` → new IP
6. Test photo import + processing pipeline
---
### Phase 4 — Tier 3 moves (optional)
Migrate grimmory (130), teddycloud (131), rclone (132), trmnl (128), sophia (119)
as needed — each frees 12 GiB. Not urgent; do when convenient.
---
### Phase 5 — Follow-up
- **QDevice**: add a tiebreaker for 2-node quorum
- **Gaming VM**: strong's 6850U has enough cores alongside migrated LXCs
- **Hubris library cleanup**: after all guests are confirmed working, decide
whether to keep the original library SSD as backup or repurpose it
---
## Resource math after Phase 3 (all Tier 1 + 2 moved)
| | hubris | strong |
|---|--------|--------|
| Guests | 11 LXC + 2 VM | 5 LXC |
| RAM allocated | ~25 GiB | ~45 GiB |
| RAM capacity | 28 GiB | 28 GiB |
| Overcommit | 0.9× (under-committed) | 1.6× (manageable) |
| Library disk | Local ext4 (3.7T) → NFS client | Local ext4 on ludo-lvm (1.8T) |
| GPU | Radeon 760M (idle) | Radeon 680M (jellyfin + mule-images) |
strong becomes the media/library powerhouse. hubris becomes a lean core-infra
node (DNS, auth, git, docs, caddy, HA).
---
---
## Risk register
| Risk | Impact | Mitigation |
|------|--------|------------|
| NFS latency for library reads (jellyfin, arriman) | Media playback stutter, slow downloads | Test iperf between strong↔hubris first. If 2.5G link, NFS throughput is fine (~1 Gbit/s). |
| GPU passthrough on strong (680M vs 760M) | Transcode quality/compat differences | Both are AMD VAAPI — same driver stack. Test `vainfo` inside LXC before going live. |
| Caddy backend IP churn | Service outage if IP wrong | Update Caddyfile in git repo (caddy-conf), test each route before destroying old LXC. |
| vzdump/restore downtime | Service unavailable during migration | Schedule off-hours. Use rsync for large rootfs (120's 100G) to minimize freeze window. |
| 2-node quorum still fragile | If hubris goes down, strong /etc/pve goes read-only | Guests keep running. Add QDevice as follow-up. |
| Library data integrity during NFS transition | Permission drift | NFS `all_squash,anonuid=33,anongid=10000` matches existing LXC 102 config. Verify with `ls -la /mnt/library` after mount. |
---
## Open questions for operator
1. **Internal bridge on strong**: proceed with `vmbr1` on `192.168.8.3/24`
(Option A), or use `192.168.178.x` guest IPs (Option B)?
2. **Migration method**: `vzdump`/restore (clean, downtime) vs `rsync` rootfs
(faster for large disks, needs manual config copy)?
3. **Phase 1 priority**: move elementsynapse + house first (quick wins), or
go straight to Phase 2 (mule-images/jellyfin/arriman) for maximum relief?
4. **Should we add a QDevice now** before moving anything, to protect
management plane during the migration?
---
## Changelog
### 2026-07-05 — Phase 2d complete (grimmory migrated; media NFS to zimaos)
grimmory (130) → 192.168.8.247 on strong. Rsync'd /books (2.6G) to ludo-lvm.
LXC 102 (nfs-export) now mounts strong's NFS at /mnt/media and exports it as
a second share alongside /mnt/library. Zimaos mounts both: /media/library
(hubris user-generated) and /media/media (strong media+books).
See hosts/strong.md changelog.
### 2026-07-05 — Phase 2 complete (arriman + jellyfin migrated; library on ludo-lvm)
arriman (122) → 192.168.8.245, jellyfin (101) → 192.168.8.246. Created 1.5T
thin volume on ludo-lvm, rsync'd 363G of media data. Both containers use local
ext4 mount — no NFS. Jellyfin has 680M + RX 7600 GPU passthrough.
Caddy backends updated. See hosts/strong.md changelog.
### 2026-07-05 — Phase 1 complete (elementsynapse + house migrated to strong)
Both Tier 1 guests moved: elementsynapse (118) → 192.168.8.242, house (129) → 192.168.8.244.
Strong now has vmbr1 at 192.168.8.241/28. Hubris has proxy ARP + /32 routes for strong
guest range. DHCP scope narrowed to 192.168.8.100-239 to avoid conflicts.
Teddycloud (LXC 131) given static IP 192.168.8.150 due to IP conflict with
previous DHCP allocation at 192.168.8.243.
See hosts/strong.md changelog for full steps.
### 2026-07-05 — assessment created
Built from live `pct config` + `pvesm status` + `free -h` data pulled from
both nodes. Supersedes the storage-migration framing of the original
library-SSD plan — this assessment treats the SSD move as optional and
focuses on guest relocation via NFS.

View File

@@ -0,0 +1,33 @@
# Glossary
Terms and abbreviations used throughout the homelab wiki.
| Term | Meaning |
|------|---------|
| **Authentik** | SSO/identity provider. Core runs on the VPS; forward-auth outpost at LXC 106 on hubris |
| **Caddy** | Reverse proxy (LXC 121). Terminates TLS for every `*.hubris.network` hostname |
| **Caveman** | Terse communication standard for agent responses — no filler, keep substance |
| **Forward-auth** | Caddy snippet that delegates authentication to an Authentik outpost. Protects web UIs like qBit, SABnzbd |
| **Gitea** | Git server at `git.hubris.network`. Hosts all tracked config repos |
| **Gluetun** | WireGuard VPN sidecar on arriman. All \*arr traffic routes through it |
| **HAOS** | Home Assistant Operating System. VM 108 on hubris |
| **Hubris** | Primary Proxmox VE node (GMKtec NucBox M6 Ultra). PVE hostname, cluster member 1 |
| **LXC** | Linux Container (Proxmox). VM-like isolation without a full OS kernel |
| **LVM-thin** | Thin-provisioned logical volume manager. Used for all container/VM storage |
| **MCP** | Model Context Protocol (MCP server at `mcp.hubris.network`). Structured tools for agents to query homelab state |
| **Mesh** | Overlay VPN for off-LAN connectivity. Netbird is current; Tailscale is legacy |
| **Netbird** | Preferred mesh VPN. VPS hosts the management plane; all homelab nodes are members |
| **OIDC** | OpenID Connect. Protocol used by Authentik for SSO login flows |
| **Oikos** | Agent operating model ([.agents/OIKOS.md](../.agents/OIKOS.md)). OODA loop, risk classes, policy, ontology |
| **PVE** | Proxmox Virtual Environment — the hypervisor on both hubris and strong |
| **SOPS** | `sops` — Mozilla SOPS. Encrypts secrets with age keys so they live in the git repo |
| **Strong** | Secondary Proxmox VE node. Cluster member 2 (hostname `strong`, nickname ludo/ludo-mini) |
| **Traefik** | Reverse proxy on IONOS VPS. Serves `*.hubris.network` to the public internet |
| **VAAPI** | Video Acceleration API. Intel/AMD GPU-based hardware transcode for Jellyfin |
| **VPS** | Virtual Private Server at IONOS (`82.165.190.79`). Runs Authentik core + Netbird management |
| **\\*arr** | Media automation suite: Sonarr (TV), Radarr (movies), Lidarr (music), Prowlarr (indexer), Bazarr (subtitles), Readarr (books — not in use) |
## See also
- [Infrastructure index](wiki/infrastructure/index.md) — cross-cutting systems each with their own doc page
- [OIKOS operating model](../.agents/OIKOS.md) — agent policy, risk classes, lifecycle

View File

@@ -0,0 +1,160 @@
# 101 — `jellyfin`
Media server: serves the movies / TV / anime / music libraries from `/mnt/media_local` to LAN clients. Hardware transcoding via AMD Radeon 680M + RX 7600 VAAPI. Authentik SSO via OIDC.
## At a glance
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **IP:** `192.168.8.246`
- **Privilege:** privileged (recreated on strong as priv)
- **Resources:** 4 cores / 8 GiB RAM / 1 GiB swap / 16 GiB rootfs
- **GPU:** `/dev/dri/renderD128` + `/dev/dri/card0` (AMD Radeon 680M iGPU + RX 7600 dGPU) passed via `dev0` / `dev1` in LXC config
- **Mounts:** `/mnt/media_local``/mnt/library`
- **Public hostname:** [`media.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8096`
- **Version:** Jellyfin 10.11.11 (apt package, Ubuntu 24.04 noble repo)
- **FFmpeg:** jellyfin-ffmpeg7 7.1.4
## Service / port map
| Service | Listen | Notes |
| -------- | ------ | ----- |
| jellyfin | `:8096` | HTTP (caddy terminates TLS) |
## Hardware acceleration (VAAPI)
GPU is passed through to the LXC via `dev0: /dev/dri/renderD128,gid=993` and
`dev1: /dev/dri/card0,gid=44` in
`/etc/pve/lxc/101.conf` (strong). The `jellyfin` user is in the `render` (GID 993) and
`video` groups inside the container.
| GPU | Model | Role |
|-----|-------|------|
| Radeon 680M | iGPU (AMD Ryzen 7 PRO 6850U) | Primary VAAPI encoder/decoder |
| RX 7600 | dGPU (add-in) | Secondary transcode, HEVC/AV1 encoding |
Encoding settings (`/etc/jellyfin/encoding.xml`):
- `HardwareAccelerationType`: `vaapi`
- `VaapiDevice`: `/dev/dri/renderD128`
- `EnableHardwareEncoding`: `true`
- `AllowHevcEncoding`: `true`
- `AllowAv1Encoding`: `true`
- `EnableTonemapping`: `true`
- `HardwareDecodingCodecs`: h264, hevc, vc1, vp9, av1
- `EnableThrottling`: `true`
- `EnableSegmentDeletion`: `true`
Trickplay (`/etc/jellyfin/system.xml`):
- `EnableHwAcceleration`: `true`
- `EnableHwEncoding`: `true`
## Authentik SSO (OIDC)
Jellyfin uses the [SSO-Auth plugin](https://github.com/9p4/jellyfin-plugin-sso)
v4.0.0.4 for Authentik OIDC login. No Caddy forward-auth gate — the SSO plugin
handles auth directly via OIDC redirect flow.
### Architecture
```
User → media.hubris.network → Caddy (TLS, no forward-auth) → Jellyfin :8096
Login page with "Sign in with Authentik" button
↓ (click)
/sso/OID/start/Authentik
↓ (302 redirect)
auth.hubris.network OIDC
↓ (login)
/sso/OID/redirect/Authentik?code=...&state=...
Jellyfin SSO plugin validates token → logged in
```
### Components
1. **SSO-Auth plugin** — installed at `/var/lib/jellyfin/plugins/SSO-Auth_4.0.0.4/`
- Config: `/var/lib/jellyfin/plugins/configurations/SSO-Auth.xml`
- Provider name: `Authentik`
- OIDC endpoint: `https://auth.hubris.network/application/o/jellyfin/`
- `SchemeOverride`: `https` (required — without it, plugin generates
`http://` redirect URIs that Authentik rejects)
- `EnableAuthorization`: `false` (prevents plugin from overwriting admin
permissions on each SSO login — see
[jellyfin-sso-plugin](../../../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md))
- `OidScopes`: `["email"]` (openid+profile added by default by the plugin;
must be non-null or `OidChallenge()` throws `ArgumentNullException`)
2. **Authentik OIDC provider**`Provider for Jellyfin` (PK 6)
- Client ID: `vt61t5Y2ZVtN6l3QjitkBvwUJjFKvSyl4TDBXcJx`
- Redirect URI: `https://media.hubris.network/sso/OID/redirect/Authentik`
- Application slug: `jellyfin`
3. **SSO button injection**`/usr/share/jellyfin/web/sso-inject.js`
- Injected via `<script defer src="sso-inject.js?v=3">` in `index.html`
- Polls for `.readOnlyContent` div on the login page, adds "Sign in with
Authentik" button linking to `/sso/OID/start/Authentik`
- Cache-busted with `?v=N` parameter (bump on changes)
- **Lost on apt upgrade** — re-inject the script tag and copy the JS file
after `apt-get upgrade jellyfin*`
4. **Caddy config** — no forward-auth gate for media.hubris.network:
```caddy
media.hubris.network {
tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
reverse_proxy 192.168.8.206:8096
}
```
### Known issues / pitfalls
- **`OidScopes` must be non-null** — if the field is missing from the plugin
config XML, `OidChallenge()` throws `System.ArgumentNullException`. Always
include `OidScopes` in the provider config (even if empty array).
- **`SchemeOverride: "https"` is required** — without it, the plugin generates
`http://` redirect URIs (from the internal HTTP listener). Authentik rejects
them with "Redirect URI Error".
- **SSO button JS is not served by the plugin** — the `__plugin/SSO-Auth.js`
endpoint returns 404 on Jellyfin 10.11.x when the plugin is installed
manually (not via Jellyfin's plugin manager). The `sso-inject.js` workaround
in `index.html` is the fallback.
- **No Caddy forward-auth gate** — the SSO plugin's OIDC redirect flow is
incompatible with Caddy's `import authentik` forward-auth. If both are
enabled, the forward-auth intercepts the OIDC callback and breaks the flow.
Use one or the other, not both. SSO plugin (OIDC redirect) is preferred.
- **API key for setup** — a temp API key can be inserted directly into the
`ApiKeys` SQLite table for automated configuration:
```sql
INSERT INTO ApiKeys VALUES (1, '2026-07-04', '2026-07-04', 'setup', 'jf-setup-key-...');
```
## Permissions
Member of the [media GID 10000](../infrastructure/media-permissions.md) standard. Service user `jellyfin` is in the `media` group inside the container; `/mnt/media_local` on strong's ludo-lvm is owned `root:media` with mode `2775`.
## Related
- [Caddy reverse proxy](121-caddy.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads
- [DNS split-horizon](../infrastructure/dns.md)
- [Authentik admin](../../../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config
## Changelog
### 2026-07-06 — wiki: IP, host, GPU, mount path updated for strong migration
Updated At-a-glance: IP 206→246, host hubris→strong, mount /mnt/library→/mnt/media_local, GPU Radeon 760M→680M+RX7600, privilege unpriv→priv. Permissions section updated. Changelog entry in 122-arriman.md updated similarly.
### 2026-07-04 — VAAPI hardware acceleration + Authentik SSO + resource bump
- Upgraded Jellyfin 10.11.8 → 10.11.11 (purge + reinstall to fix DB migration bug)
- Enabled VAAPI hardware acceleration (Radeon 760M): h264/hevc/vc1/vp9/av1 decode + encode
- Bumped resources: 2→4 cores, 4→8 GiB RAM, 512→1024 MiB swap
- Enabled trickplay HW acceleration + throttling + segment deletion
- Installed SSO-Auth plugin v4.0.0.4 with Authentik OIDC provider
- Configured `SchemeOverride: https`, `OidScopes: ["email"]`, `EnableAuthorization: false`
- Injected SSO button via `sso-inject.js` in web `index.html`
- Removed Caddy forward-auth gate (incompatible with SSO plugin OIDC flow)
- **Database was wiped** during cache relocation attempt — no LVM snapshot
existed. All watch states, user accounts, and library configs lost.
Libraries re-added via setup wizard.
### 2026-04-28 — wiki entry created
Initial documentation. No config changes.
### 2026-04-20 — joined the `media` GID 10000 standard
Idmap block applied; in-container `media` group at GID 10000 mapped to host GID 10000. See [media permissions](../infrastructure/media-permissions.md). Config backup: `/root/101.conf.bak.*`.

View File

@@ -8,7 +8,7 @@ Dedicated, single-purpose LXC that re-exports `/mnt/library` over NFSv4 to clien
- **LAN DNS:** `nfs-export.hubris.network``192.168.8.200` (direct, no Caddy)
- **Privilege:** privileged (`unprivileged: 0`) + `lxc.apparmor.profile: unconfined` — required for `nfs-kernel-server`
- **Resources:** 1 core / 512 MiB RAM / 2 GiB rootfs / 256 MiB swap
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, plato, apps)
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, apps)
## What it does
@@ -54,7 +54,7 @@ We considered three options before building this:
| Option | Outcome |
|---|---|
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../investigations/index.md)). Rejected. |
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. |
| **SMB on host** | Same host-blast-radius problem, plus 3050% lower throughput than NFS on Linux↔Linux. Rejected. |
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |

View File

@@ -18,7 +18,7 @@ Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/d
| paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers |
## Auth
Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
Behind [Authentik forward-auth](106-auth-outpost.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
## Storage
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
@@ -27,7 +27,7 @@ Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses fo
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- [Authentik](124-authentik.md)
- [Authentik](106-auth-outpost.md)
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)

View File

@@ -45,6 +45,9 @@ LXC has `/etc/hosts` override mapping `auth.hubris.network → 192.168.8.175` (r
## Changelog
### 2026-06-24 — terminalito deploy webhook (id 12)
Push webhook on `dtoro/terminalito``http://192.168.8.211:9797/deploy` ([trmnl (128)](128-trmnl.md)); `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-04-28 — wiki entry created
Initial documentation.
@@ -52,7 +55,7 @@ Initial documentation.
Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md).
### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added
For OIDC integration with [authentik (124)](124-authentik.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
For OIDC integration with [authentik (124)](106-auth-outpost.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped
`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`.

View File

@@ -1,6 +1,6 @@
# 105 — `apps`
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20.
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20. Booklore migrated to [grimmory (130)](130-grimmory.md) on 2026-06-29.
## At a glance
- **Hostname:** `apps`
@@ -15,7 +15,6 @@ Docker host for everything that doesn't justify its own LXC. Currently runs Arti
| Hostname | Container | Backend port | Notes |
| --------------------------------- | ---------------- | ------------ | ----- |
| `docker.hubris.network` | Portainer | `:9443` | Native OAuth2 via Authentik. Trusted-origins requires hostname only (no scheme/port). |
| `books.hubris.network` | Booklore | `:6060` | Native OIDC. Redirect URI `/oauth2-callback`. |
| `artifacto.hubris.network` | Artifacto | `:3100` | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). |
| `blog.hubris.network` | WriteFreely | `:8080` | Native OIDC via `[oauth.generic]`. |
| `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). |
@@ -46,12 +45,16 @@ Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webh
### Portainer
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
### Booklore
Native OIDC via Authentik (Settings → OIDC). Redirect URI `/oauth2-callback` (NOT `/api/oidc`). Container needs `extra_hosts: auth.hubris.network:192.168.8.175`. **Edit via Portainer UI** if it's a Portainer-managed stack.
### homelab-mcp (`/opt/homelab-mcp/`) — DEPRECATED (Go rewrite, Phase 6)
> ⚠️ **Never `docker compose up` Portainer-managed stacks from the host shell.** Portainer's compose state lives at `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`. Running `docker compose up -d <svc>` from the host triggers recreates of OTHER services in the stack and silently destroys bind-mounted data. **This wiped Booklore's mariadb data on 2026-04-22.** Use the Portainer UI editor for compose changes. See [mesh migration](../infrastructure/mesh.md#critical-never-docker-compose-up-portainer-managed-stacks) for the full warning.
> **Status:** This Python MCP server is being replaced by the Go `oikos api` binary
> running in Docker on mac-mini. Cutover pending — see
> [scripts/cutover-checklist.md](../../scripts/cutover-checklist.md) for the
> execution plan. The Go MCP uses the official MCP Go SDK (Streamable HTTP, not
> FastMCP) with 15 tools including `get_blast_radius`, `request_execution`, and
> `get_agent_activity`. Source: `internal/mcp/server.go`.
### homelab-mcp (`/opt/homelab-mcp/`)
**Current (Python) implementation — DO NOT MODIFY, awaiting cutover:**
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
`/opt/homelab-context/` (this LXC is itself an enrolled
[homelab-context](../infrastructure/homelab-context.md) client). Source
@@ -76,7 +79,10 @@ the server code). Listens on `0.0.0.0:9811`, secret in
`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
restarts on push.
### secrets-issuance (`/opt/secrets-issuance/`)
### secrets-issuance (`/opt/secrets-issuance/`) — DEPRECATED (Go Phase 5)
> **Status:** Replaced by `internal/secrets/` in the Go rewrite. Machine identities
> are now managed via Infisical (`docker compose --profile infisical up`).
Tiny HTTP service that issues per-client age keypairs the first time
each client calls `/issue`. Idempotent: subsequent calls return the
same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in
@@ -106,20 +112,23 @@ Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.h
## Related
- [Gitea (104)](104-gitea.md) — uses the PlantUML server
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](124-authentik.md)
- [Authentik (124)](106-auth-outpost.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Public ingress (Artifacto + blog)](../infrastructure/ingress.md)
## Changelog
### 2026-06-29 — Booklore migrated to Grimmory on LXC 130
Booklore stack removed from Portainer. MariaDB dump taken first, then restored into [grimmory (130)](130-grimmory.md)'s fresh MariaDB. `books.hubris.network` Caddy backend updated to `192.168.8.213:6060`. Authentik OIDC provider updated to Public client type (PKCE) for Grimmory compatibility.
### 2026-05-20 — homelab-mcp + secrets-issuance live
Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md):
`homelab-mcp.service` on `:9810` (MCP read+management surface) and
`secrets-issuance.service` on `:9820` (per-client age-key provisioning).
Caddy fronts both with Let's Encrypt; new vhosts on
[caddy](121-caddy.md), split-horizon DNS entries on
[authentik (124)](124-authentik.md). Gitea webhook ids 10 + 11 wire
[authentik (124)](106-auth-outpost.md). Gitea webhook ids 10 + 11 wire
auto-deploy. LXC is itself an enrolled context client
(`/opt/homelab-context/`).

View File

@@ -1,6 +1,6 @@
# 106 — `auth-outpost`
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
## At a glance
- **Hostname:** `auth-outpost`
@@ -8,11 +8,11 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
- **Resources:** 1 core / 512 MiB / 4 GiB rootfs
- **Mounts:** none
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](124-authentik.md)
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](106-auth-outpost.md)
## Role
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../investigations/2026-05-31-authentik-vps-migration.md).
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
## Service / port map
| Service | Listen | Notes |
@@ -42,15 +42,15 @@ Fix: the LAN outpost gets its **own** domain.
**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
## Related
- [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only)
- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only)
- [Caddy (121)](121-caddy.md) — forward-auth consumer
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
- [Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md)
- [Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)
## Changelog
### 2026-06-06 — Authentik session lifetime extended to 30 days
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../investigations/2026-06-06-authentik-session-lifetime.md).
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md).
### 2026-06-01 — created; forward-auth cut over from LXC 124
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000``192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md).
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000``192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).

View File

@@ -1,6 +1,6 @@
# 107 — `dns`
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](124-authentik.md); single-purpose, one job.
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](106-auth-outpost.md); single-purpose, one job.
## At a glance
- **Hostname:** `dns`
@@ -29,7 +29,7 @@ Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recur
- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field.
## dns-sync (Technitium = authoring source)
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../../../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
## DHCP
@@ -42,17 +42,20 @@ Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02):
Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101.239`) are excluded from the pool. Pool narrowed from `.100.240` to `.241.254` on 2026-06-03 to eliminate IP conflict risk.
## Related
- [124 — authentik](124-authentik.md) — retired host of the old dnsmasq
- [124 — authentik](106-auth-outpost.md) — retired host of the old dnsmasq
- [DNS split-horizon](../infrastructure/dns.md)
- [Mesh](../infrastructure/mesh.md)
## Changelog
### 2026-06-24 — A record `trmnl.hubris.network → 192.168.8.175`
Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)); propagated to the NetBird managed zone by `dns-sync`.
### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
### 2026-06-03 — DHCP pool narrowed to `.241.254`
Previous pool `.100.240` overlapped with all static LXCs/VMs (`.101.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101.110` remain until natural expiry (2026-06-04). See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
Previous pool `.100.240` overlapped with all static LXCs/VMs (`.101.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101.110` remain until natural expiry (2026-06-04). See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone)
This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
@@ -61,4 +64,4 @@ This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py`
Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
### 2026-06-01 — created; replaced dnsmasq on 124
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](124-authentik.md).
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](106-auth-outpost.md).

View File

@@ -11,7 +11,7 @@ Personal cloud / file collaboration. Source-of-truth for the photo libraries sur
- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)
## Auth
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](124-authentik.md#per-app-username-override-pattern-authentik) for the full pattern.
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](106-auth-outpost.md#per-app-username-override-pattern-authentik) for the full pattern.
Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on).
@@ -74,7 +74,7 @@ Apply with `systemctl restart mariadb` (not reload — `innodb_log_file_size` ne
## Related
- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV
- [Authentik (124)](124-authentik.md)
- [Authentik (124)](106-auth-outpost.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md)

View File

@@ -4,7 +4,8 @@ Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
## At a glance
- **Hostname:** `elementsynapse`
- **IP:** `192.168.8.239`
- **IP:** `192.168.8.242`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** **unprivileged**
- **Resources:** 1 core / 2 GiB RAM / **16 GiB rootfs** (grown from 8 GiB on 2026-05-15 after disk-full incident)
- **Mounts:** none from `/mnt/library`
@@ -36,7 +37,7 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
- ~~[claudio-bot (123)](archive/123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)

View File

@@ -60,7 +60,7 @@ For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials
## Related
- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries
- [Authentik (124)](124-authentik.md)
- [Authentik (124)](106-auth-outpost.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)

View File

@@ -11,16 +11,16 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L
- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy)
- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`).
## Sites currently served (live as of 2026-04-28)
## Sites currently served (live as of 2026-07-06)
- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100`
- `auth.hubris.network` → [authentik (124)](124-authentik.md) `:9000`
- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080`
- `books.hubris.network` → [apps (105)](105-apps.md) `:6060`
- `books.hubris.network` → [grimmory (130)](130-grimmory.md) `:6060`
- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443`
- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443`
- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`)
- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123`
- `house.hubris.network` → [house (129)](129-house.md) `:3000`
- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056`
- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008`
- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096`
@@ -28,15 +28,18 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L
- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
- `roms.hubris.network` → [romm (134)](134-romm.md) `:80`
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
- `teddy.hubris.network` → LXC 131 `192.168.8.150:8443`
- `trmnl.hubris.network` → [trmnl (128)](128-trmnl.md) `:9851`
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
## Snippet: `(authentik)` forward-auth
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup).
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](106-auth-outpost.md#forward-auth-domain-level-setup).
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md).
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](106-auth-outpost.md).
## Caddy environment
@@ -57,7 +60,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
## Related
- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
- [Authentik (124)](124-authentik.md) — forward-auth + IdP
- [Authentik (124)](106-auth-outpost.md) — forward-auth + IdP
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
- [Gitea (104)](104-gitea.md) — webhook source
@@ -82,7 +85,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
Also: [elementsynapse LXC 118](../containers/118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
### 2026-06-02 — caddy.service unit missing; recreated
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.

View File

@@ -4,10 +4,11 @@ Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr
## At a glance
- **Hostname:** `arriman`
- **IP:** `192.168.8.132`
- **IP:** `192.168.8.245`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged
- **Resources:** 4 cores / 8 GiB RAM / 24 GiB rootfs
- **Mounts:** `/mnt/library``/mnt/library`
- **Mounts:** `/mnt/media_local``/mnt/library`
- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below)
## Compose
@@ -114,7 +115,7 @@ Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has
## Related
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](124-authentik.md) — forward-auth wiring + per-app `/api/*` bypass
- [Authentik (124)](106-auth-outpost.md) — forward-auth wiring + per-app `/api/*` bypass
- [DNS](../infrastructure/dns.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Hubris host](../hosts/hubris.md)

View File

@@ -0,0 +1,48 @@
# 128 — `trmnl`
Self-hosted middleware for TRMNL e-ink plugins. TRMNL cloud polls it; it fetches/shapes live data into JSON the plugin's Liquid template renders.
## At a glance
- **Hostname:** `trmnl`
- **IP:** `192.168.8.211`
- **Privilege:** unprivileged
- **Resources:** 1 core / 768 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
- **Public hostname:** `trmnl.hubris.network` (via [VPS ingress](../infrastructure/ingress.md))
## Role
Runs one FastAPI aggregator (`server.app:app`, port 9851) that mounts a router per plugin from the `dtoro/terminalito` repo. First consumer: `munich-home` (`/munich-home/dashboard`) — weather (Open-Meteo), MVG transit, Google Calendar, plus server-side Kita/quote logic. Talks out to the public internet for those APIs; TRMNL cloud polls it inbound every 15 min. Bearer-token gated (`TRMNL_POLL_TOKEN`); `/health` is open.
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| `trmnl-plugins` | `0.0.0.0:9851` | uvicorn aggregator; `EnvironmentFile=/etc/trmnl-plugins/env` |
## Storage / config paths
- `/opt/terminalito` — git checkout (origin = internal gitea `http://192.168.8.121:3000/dtoro/terminalito.git`)
- `/opt/terminalito/server/.venv` — venv
- `/etc/trmnl-plugins/env``TRMNL_POLL_TOKEN` (+ Google/MVG creds once enrolled)
- `/etc/systemd/system/trmnl-plugins.service`
## Auto-deploy
Wired — [auto-deploy](../infrastructure/auto-deploy.md) Shape B, webhook id 12 on `dtoro/terminalito``http://192.168.8.211:9797/deploy` (`terminalito-deploy.service`). Push to `main``server/deploy/deploy.sh` (`git pull` + pip + reinstall units + restart `trmnl-plugins`). Secret `/etc/terminalito-deploy/secret`; git creds `/etc/terminalito-deploy/git-credentials` wired as a repo-local `credential.helper`. Manual: `pct exec 128 -- /opt/terminalito/server/deploy/deploy.sh`.
## Secrets
Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env`. Google Calendar + MVG creds are pending: enroll via `homelab client add trmnl` + bootstrap, add `secrets/trmnl-oauth.yaml`, then `server/deploy/render-env.sh` builds the env from `homelab secret trmnl-oauth`. Until then calendar/transit cards degrade to empty; weather works.
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`trmnl.hubris.network → 192.168.8.211:9851`)
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `trmnl → 192.168.8.175` (LAN path via Caddy)
- [Gitea (104)](104-gitea.md) — source repo `dtoro/terminalito`
- [Plan: 2026-06-24 TRMNL plugins LXC](../../../plans/2026-06-24-trmnl-plugins-lxc.md)
## Changelog
### 2026-06-24 — auto-deploy + LAN DNS wired
Gitea Shape-B deploy pipeline (webhook id 12, `:9797`) — push to `dtoro/terminalito` redeploys; verified end-to-end. Technitium A record `trmnl.hubris.network → 192.168.8.175` added on [dns (107)](107-dns.md) (propagated to the NetBird managed zone via dns-sync), so LAN clients take the short path through [Caddy (121)](121-caddy.md). See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-06-24 — public path live
Verified end-to-end from the internet: `https://trmnl.hubris.network/munich-home/dashboard` → 200 with token, 401 without; `/health` 200. The provision-time outage was the netbird `home-lab-network` (192.168.8.0/24) route having no active routing peer — the **mac-mini routing peer's netbird daemon was down** (artifacto/blog were 504 too). Bringing netbird up on mac-mini restored the route; the edge recovered with no config change. See [ingress](../infrastructure/ingress.md) / [mesh](../infrastructure/mesh.md).
### 2026-06-24 — provisioned
LXC 128 created (Debian 13, unprivileged, `192.168.8.211`). Deployed `trmnl-plugins.service` on :9851 from `dtoro/terminalito`. Caddy block added (`dtoro/caddy-conf`) + LE cert via IONOS DNS-01; verified `/health` 200 and `/munich-home/dashboard` (live weather) through Caddy. Cert mirrored to VPS (`trmnl.fullchain.crt`/`trmnl.privkey.key`) + traefik router `trmnl-public``192.168.8.211:9851` added to `/opt/traefik-dynamic.yaml`. **Public path pending**: VPS↔home netbird route was down at provision time (`No networks available`, 3/6 peers — artifacto/blog also 504); resolves when the mesh route recovers. **LAN pending**: Technitium A record not yet added. Not SOPS-enrolled; Google/MVG creds pending.

View File

@@ -0,0 +1,52 @@
# 129 — `house`
Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modules: calendar, tasks, meals, groceries, budget, documents, notes, contacts, birthdays, housekeeping, recipes, reminders.
## At a glance
- **Hostname:** `house`
- **IP:** `192.168.8.244`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** unprivileged
- **Resources:** 1 core / 1344 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
- **Public hostname:** [`house.hubris.network`](../infrastructure/ingress.md) → VPS traefik → Caddy
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| `oikos` (Yuvomi) | `0.0.0.0:3000` | Docker Compose at `/opt/yuvomi/`, image `ghcr.io/ulsklyc/yuvomi` |
## Integrations
- **Authentik SSO (OIDC):** Provider `Provider for Yuvomi` (PK 31) in Authentik on VPS. Env vars in `/opt/yuvomi/.env`: `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`. Redirect URI: `https://house.hubris.network/auth/oidc/callback`.
- **Paperless DMS connector (native):** Yuvomi connects directly to Paperless-ngx API at `http://192.168.8.130:8000/`. API token stored in SQLite `dms_accounts` table. Search, link, and upload documents from Yuvomi to Paperless via Settings → Documents → DMS.
- **Weather widget:** Open-Meteo (free, no API key). Munich coordinates set.
- **Google Calendar:** OAuth configured via env vars (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI`). Redirect URI: `https://house.hubris.network/api/v1/calendar/google/callback`. Authorize in Settings → Calendar → Connect Google Calendar.
## Config paths
- `/opt/yuvomi/docker-compose.yml` — downloaded from upstream
- `/opt/yuvomi/.env` — config including secrets (untracked)
- `/opt/yuvomi/data/` — SQLCipher SQLite DB (`oikos.db`)
- `/opt/yuvomi/backups/` — auto backups
- `/opt/yuvomi/modules/` — Yuvomi modules (empty for now)
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`house.hubris.network → 192.168.8.212:3000`)
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy)
- [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`)
- [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source
- [Deployment plan](../../../plans/done/2026-06-25-yuvomi-deployment.md)
## Changelog
### 2026-06-27 — Google Calendar OAuth env vars configured
`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI` set in `.env`. New OAuth client ID (`-bho4iq..`).
### 2026-06-26 — provisioned
LXC 129 created (Debian 13, unprivileged, `192.168.8.212`). Docker installed. Yuvomi container running on `:3000` from `ghcr.io/ulsklyc/yuvomi:latest`. Caddy block + DNS A record + VPS traefik router `house-public` for public access. Authentik OIDC provider created (PK 31). WebDAV document bridge on paperless LXC (103) at `:8088` for Paperless auto-import.

View File

@@ -0,0 +1,66 @@
# 130 — `grimmory`
Self-hosted digital library (eBooks, comics, audiobooks). Community fork/successor of Booklore, with smart shelves, metadata enrichment, Kobo/KOReader sync, OPDS, and a built-in EPUB/PDF reader. Migrated from [apps (105)](105-apps.md) on 2026-06-29.
## At a glance
- **Hostname:** `grimmory`
- **IP:** `192.168.8.247`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged (UID = host UID for `/mnt/library` media GID)
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13)
- **Mounts:** `/mnt/media_local``/mnt/library`
- **Public hostname:** `books.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| Grimmory | `192.168.8.247:6060` | Docker Compose at `/opt/grimmory/` |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
Located at `/opt/grimmory/docker-compose.yml`. Key points:
- Image: `ghcr.io/grimmory-tools/grimmory:latest`
- Books library: `/mnt/library/books``/books` (read-write; media GID 10000 via `GROUP_ID=10000`)
- Bookdrop (watched folder for auto-import): `/opt/grimmory/bookdrop``/bookdrop`
- App data (covers, DB config): `/opt/grimmory/data``/app/data`
- MariaDB config: `/opt/grimmory/mariadb/config``/config` (linuxserver/mariadb image)
- `extra_hosts: auth.hubris.network:192.168.8.175` — routes Authentik OIDC discovery to Caddy from inside the container
- `FORCE_DISABLE_OIDC=false` — OIDC stays enabled; provider configured via Grimmory admin UI
Credentials live in `/opt/grimmory/.env` (untracked):
- `DATABASE_PASSWORD` / `MYSQL_PASSWORD` — MariaDB Grimmory user password
- `MYSQL_ROOT_PASSWORD` — MariaDB root password
## Authentik OIDC
Uses Confidential client (client secret stored in Grimmory's DB — migrated from Booklore). The OIDC config carried over in the database dump; no manual re-entry needed.
- **Authentik provider:** `Provider for Grimmory` (renamed from `Provider for Booklore` on migration)
- **Client ID:** `L1u0eFsNhbKgiIvvFeIr2mvZdbtFyzidCq2h6thL`
- **Client type:** Confidential (client secret in `oidc_provider_details` in MariaDB `app_settings`)
- **Redirect URI:** `https://books.hubris.network/oauth2-callback`
- **Scopes:** openid, profile, email, offline_access
- **Back-channel logout:** `http://192.168.8.247:6060/api/v1/auth/oidc/backchannel-logout`
- **Application slug:** `booklore` → Issuer URI: `https://auth.hubris.network/application/o/booklore/`
## Media permissions
LXC is privileged → in-container UID = host UID. Docker container gets media GID via `GROUP_ID=10000` env var (Grimmory/linuxserver pattern). The `/mnt/library/books` subtree is owned `:media` mode `2775` (setgid). See [media-permissions](../infrastructure/media-permissions.md).
## Related
- [apps (105)](105-apps.md) — previous host (Booklore)
- [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.247:6060`
- [Authentik (124)](106-auth-outpost.md) — OIDC provider `Grimmory`
- [DNS (107)](107-dns.md) — `books.hubris.network → 192.168.8.175` (unchanged from Booklore)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-06-29 — provisioned; Booklore migrated
LXC 130 created (Debian 13, privileged, `192.168.8.213`). Docker installed. Grimmory compose deployed at `/opt/grimmory/`. MariaDB dump from Booklore (LXC 105) restored — schema-compatible since Grimmory is a direct fork. Caddy `books.hubris.network` backend updated from `192.168.8.205:6060` to `192.168.8.213:6060`. Authentik provider updated: Booklore → Grimmory, Confidential → Public (PKCE). Booklore stack removed from Portainer on LXC 105.

View File

@@ -0,0 +1,62 @@
# 131 — `teddycloud`
Self-hosted [TeddyCloud](https://github.com/toniebox-reverse-engineering/teddycloud), a
reimplementation of the Toniebox cloud backend — lets Tonie figurines play custom/ripped
audio content against a local server instead of the official cloud.
Predates the client-enrollment convention entirely; nobody wrote it down. Found and
documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged
`pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see
[OIKOS.md](../../../.agents/OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud`
container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict
for it — but it never got its own inventory entry or doc page until now.
## At a glance
- **Hostname:** `teddycloud`
- **Host:** hubris (confirmed via `pct config 131` run directly on hubris — the original
drift finding's `pct list` source)
- **IP:** `192.168.8.150` (static; was briefly `192.168.8.243` via DHCP until the
2026-07-05 strong-migration work assigned it a fixed address — see hosts/strong.md)
- **Privilege:** unconfirmed (not checked — read-only investigation didn't need it)
- **Resources:** 1 core / 1 GiB RAM / 512 MiB swap / 16 GiB rootfs (`local-lvm`), Debian 12
(bookworm)
- **Mounts:** `/mnt/library` (`mp0`)
- **Public hostname:** `teddy.hubris.network``192.168.8.150:8443` (see
[caddy (121)](121-caddy.md))
- **Enrollment:** none — no `age_pubkey`, not a `homelab` CLI client. It's a plain
docker-compose app container, not a fleet-managed host. No action needed unless it starts
needing secrets.
## Service
Runs via `docker compose` at `/opt/teddycloud` — container `teddycloud-teddycloud-1`,
image `ghcr.io/toniebox-reverse-engineering/teddycloud:latest`, publishing `80`, `443`,
`8080`, and `8443`. Caddy routes `teddy.hubris.network` to the `:8443` port.
## Risk notes
**No Caddy forward-auth gate** — unlike `sab.hubris.network` on the same Caddyfile (which
is explicitly annotated `(Authentik forward-auth)`), `teddy.hubris.network` has no auth
annotation. It's reachable to anyone on the LAN/mesh who can resolve the hostname. Not
addressed as part of this doc pass — flagging it here since it's now visible in one place
for the first time.
## Related
- [Hubris host](../hosts/hubris.md)
- [Caddy (121)](121-caddy.md) — terminates `teddy.hubris.network`
- [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already
taken by this container
- [Containers index](index.md)
- [OIKOS.md](../../../.agents/OIKOS.md) — drift detector that caught this
## Changelog
### 2026-07-06 — documented for the first time (drift-caught)
Added to `inventory.yaml` and given this page. Verified live via read-only `pct config 131`
+ `pct exec 131 -- ...` on hubris: hostname, IP, resources, and that it runs via
`docker compose` (not a raw binary or systemd unit). No changes made to the running
container — this is pure documentation catch-up.

View File

@@ -0,0 +1,199 @@
# 132 — `rclone`
Off-host backup appliance. Mirrors selected `/mnt/library` folders to **Proton Drive**
with a plain `rclone sync` (monthly), and serves rclone's Web GUI on the LAN for browsing
and ad-hoc runs. **Replaces** the disabled restic-on-USB job — see [backups](../infrastructure/backups.md).
Provisioned 2026-07-01. (LXC 131 was already taken by an undocumented `teddycloud` container,
so this landed on **132**.)
## At a glance
- **Hostname:** `rclone`
- **IP:** `192.168.8.214` (static, set in PVE `net0` config — same pattern as grimmory/authentik)
- **Privilege:** privileged (root in-container = host root → reads every `/mnt/library` subtree,
incl. `homecloud/` and `documents/`, regardless of owner)
- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs (Debian 13) — bumped from 1 GiB on 2026-07-03
after `rclone-rcd.service` was OOM-killed under real load (see "Known issue" below)
- **Mounts:** `/mnt/library` **read-only** (`mp0: /mnt/library,mp=/mnt/library,ro=1`) — a backup
job must never be able to write into the library
- **Public hostname:** none — the UI is **LAN-only, no auth** (by design)
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| rclone Web GUI (`rcd`) | `192.168.8.214:5572` | `rclone-rcd.service`, **`--rc-no-auth`**, LAN-only. Browse `/mnt/library` + `proton:`, run ad-hoc syncs, live job status |
| monthly mirror | — | `rclone-backup.service` + `.timer` (`OnCalendar=*-*-01 03:00`) |
## Backup design
- **Mode:** plain mirror — `rclone sync` (Proton mirrors local; deletions propagate; **no versioning**).
- **Encryption:** Proton Drive's built-in E2E only (no rclone `crypt` overlay → files stay
browsable in Proton's web UI).
- **Selected set:** `/etc/rclone-backup/folders.list` — one absolute source path per line
(`#`/blank ignored). This file *is* the picked set the monthly timer mirrors. Extensible to other
disks once bind-mounted into this LXC.
- **Path mapping:** source `S``proton:library-backup/<S without leading slash>`
(e.g. `/mnt/library/notes``proton:library-backup/mnt/library/notes`).
- **Runner:** `/usr/local/sbin/rclone-backup.sh [folder ...]` (Python, despite the `.sh` name — kept
the path stable) — no arg = every enabled line. Submits each folder as an **async job through the
rclone rc API** served by `rclone-rcd.service` (the same daemon backing the Web GUI on `:5572`),
so scheduled/ad-hoc runs show up live in the GUI's **Jobs panel**, not just in logs. Gentle on
Proton's rate limits (`Transfers=4, TPSLimit=8, FastList=true` via the rc `_config` payload). The
rc API here requires **POST for every call** including `job/status` and `core/stats` — GET with
query params 404s.
- **Logs / "past runs":** per-run logs in `/var/log/rclone-backup/<safe>-<ts>.log`; one-line
JSON summary per run appended to `/var/log/rclone-backup/runs.jsonl`.
- **Failure notify:** `OnFailure=rclone-backup-notify@%n.service` → logs to journal today;
**TODO** wire to Hermes `send_message` (Matrix) per [backups](../infrastructure/backups.md).
## rclone + Proton Drive
- **rclone** installed from the official binary (not apt) so the `protondrive` backend is present
(`rclone v1.74.3`).
- Remote **`proton:`** (type `protondrive`). Config at `/root/.config/rclone/rclone.conf`, mode 600.
**This file is a secret** (holds the obscured Proton password + TOTP secret + session) — **never
commit it.** Escrow the Proton account creds in the password manager.
- **Config gotchas** (from rclone docs/forum):
- Log into Proton via a **browser at least once** first, or key generation fails.
- For unattended runs, store the **TOTP _secret_** (not a 6-digit code) so rclone self-generates
codes; obscure with `rclone obscure`.
- Passwords with **extended-ASCII** characters are known to break auth.
- Proton's API is rate-limited → keep `--transfers`/`--tpslimit` conservative (baked into the runner).
- **DR escrow (pending):** store the Proton creds as sops secret `secrets/protondrive.yaml`, granted
to this LXC's age key, so the remote can be rebuilt after a re-provision.
## The UI (rclone Web GUI)
`rclone rcd --rc-web-gui --rc-no-auth --rc-addr 0.0.0.0:5572` (assets auto-downloaded on first
start). Reach it at **http://192.168.8.214:5572** on the LAN.
> **Security note:** `--rc-no-auth` exposes *full* rclone control — including deleting remote data —
> to anyone on the LAN (accepted per the design choice). The container has only a LAN NIC, so it is
> not publicly reachable. Harden later by adding `--rc-user/--rc-pass` or fronting it with Authentik.
## Tracked config (deferred)
**Not yet tracked.** The runner, systemd units, and `folders.list` currently live as plain files
directly on the LXC — fully functional, just not version-controlled or auto-deployed. A
`dtoro/rclone` gitea repo package (runner, units, `install.sh`, webhook receiver) is pre-built and
staged at `/root/rclone-repo` on the LXC for whenever this gets tracked (Shape A, like
[caddy](121-caddy.md)). Gitea `ALLOWED_HOST_LIST` already includes `192.168.8.214` in anticipation.
See [auto-deploy](../infrastructure/auto-deploy.md).
**Selected folders (live in `/etc/rclone-backup/folders.list`):** `/mnt/library/cloud` (287G),
`/mnt/library/documents` (249M), `/mnt/library/repos` (83M). `/mnt/library/notes` was synced once as
a connectivity test (not in the recurring set). Proton quota checked: 2 TiB plan, ~1.65 TiB free
after this set.
## Enrollment gotcha: `pct exec` PATH
`pct exec` (lxc-attach) does **not** source `/etc/environment` or run a login shell, so
`/usr/local/bin` (where bootstrap installs `sops`) isn't on `$PATH` by default — bootstrap's own
`command -v sops` post-install check failed under `pct exec` even though the binary installed fine.
Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs
on future LXC bootstraps.
## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
What looked like repeated "protondrive silently stalls" was actually **`rclone-rcd.service` (the rc
API daemon backing the Web GUI and, since the rc-API redesign, all actual sync work) getting
OOM-killed** under the original 1 GiB RAM allocation — `journalctl` confirms
`A process of this unit has been killed by the OOM killer` at the exact moment a transfer had
"frozen." systemd's own `Restart=on-failure` (5s) auto-respawns it, but every in-flight job's state
is lost on the kill, which looked identical to a silent backend hang from the outside (frozen
`core/stats`, no new log lines). **Fix: bumped the LXC's memory to 2 GiB** (`pct set 132 -memory
2048` — applies live via the host cgroup, confirmed via `cat /sys/fs/cgroup/lxc/132/memory.max` on
hubris, no container reboot needed). After the bump, the full folder set (`cloud` 287G, `documents`
249M, `repos` 83M) completed cleanly with no further kills.
**`rclone-backup-watchdog.timer`** (every 5 min) → `rclone-backup-watchdog.sh`: if
`rclone-backup.service` is active but total transferred bytes (global `core/stats` on the rc API)
haven't moved for 15 minutes, it restarts both `rclone-rcd.service` (clears any stuck/orphaned job —
this is the actual daemon holding the work, not the thin wrapper) and then `rclone-backup.service`
(`--no-block`, load-bearing — see below). Kept as a safety net even after the RAM fix, in case
memory pressure returns under a larger folder set later. State kept in
`/var/lib/rclone-backup/watchdog-state.json`, cleared whenever the service isn't running.
**Two watchdog design bugs found and fixed while chasing this (2026-07-03):**
1. **Wrong stats-group key.** Per-job progress polling queried `core/stats` under `job/<jobid>`,
but rclone tracks stats under whatever `_group` name the job was submitted with. Made a perfectly
healthy sync look stalled at 0 bytes for 22+ hours in its own log. Fixed by using the same
`group` variable consistently. **Lesson: distrust the per-run log's "progress bytes=" line during
an incident; cross-check with unfiltered `core/stats` first.**
2. **Watchdog restarted only the thin wrapper, and blocked doing it.** The actual `rclone sync` work
runs inside `rclone-rcd.service`, not `rclone-backup.service` — restarting the wrapper alone left
any stuck job orphaned inside `rcd` while a new wrapper submitted a duplicate job on top. Worse,
`systemctl restart rclone-backup.service` (no `--no-block`) blocks until the *new* invocation's
long-running `ExecStart` exits — which could be hours — so the watchdog's own oneshot service
never logged "Finished," and `OnUnitActiveSec` (which schedules relative to the previous run
*finishing*) never fired again. The watchdog silently disabled itself after exactly one use.
Fixed: restart `rclone-rcd.service` first, then `rclone-backup.service` with `--no-block`.
## Related
- [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup
- [Hubris host](../hosts/hubris.md) — owns `/mnt/library`
- [Media permissions](../infrastructure/media-permissions.md) — read-only consumer of `/mnt/library`
- [Containers index](index.md)
## Changelog
### 2026-07-03 — root cause found (OOM, not Proton); RAM bumped to 2 GiB; full folder set completed
What looked like repeated silent "protondrive stalls" turned out to be **`rclone-rcd.service`
getting OOM-killed** under the original 1 GiB RAM allocation — confirmed via
`journalctl -u rclone-rcd.service` showing `killed by the OOM killer` at the exact freeze point.
Bumped the LXC to 2 GiB RAM (live, no reboot). After the bump: `cloud` (287G) completed cleanly
(exit 0), `documents` (249M) completed with 1 minor error (259.7 MB transferred), `repos` (83M)
completing as of this entry. Also fixed two real watchdog bugs found while chasing this (wrong
stats-group key making a healthy sync look frozen; watchdog restarting only the wrapper with a
blocking `systemctl restart`, causing it to silently disable itself after one use) — see "Known
issue" above for full detail. The watchdog is kept as a safety net going forward even though the RAM
bump addresses the actual root cause.
### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility)
The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web
GUI's Jobs panel — the GUI only tracks work submitted through its own `rcd` process. Rewrote
`/usr/local/sbin/rclone-backup.sh` in Python, submitting each folder via `POST /sync/sync` with
`_async: true` against `http://127.0.0.1:5572` (the running `rclone-rcd.service`), then polling
`POST /job/status` + `POST /core/stats` (both **must be POST** — GET-with-querystring 404s on this
rc API) until finished, logging periodic progress snapshots and the same `runs.jsonl` summary line
as before. Verified live: submitted job visible in `POST /job/list`'s `runningIds` while running,
completed cleanly (`success: true`) once done. Deployed via atomic rename (write-then-`mv`) rather
than truncating in place, specifically so it wouldn't risk corrupting the still-running original
`cloud`+`documents`+`repos` sync mid-flight (verified after the fact: that sync's bash process was
unaffected, kept running to completion under the old in-memory script content). The already-running
scheduled sync from before this change is a standalone process and won't retroactively appear in the
GUI; every run after this point will.
### 2026-07-02 — Proton Drive auth fixed; real folder set enabled; first live sync
Initial `rclone config` failed 2FA (`422 ... auth/v4/2fa`) because a live 6-digit TOTP code was
entered instead of the TOTP secret — reconfigured with the secret, auth now works
(`rclone lsd proton:` lists the Drive). Verified end-to-end with a real sync of `/mnt/library/notes`
(219 objects, 5.964 MiB, exit 0) — confirmed files land as plain, browsable objects on Proton (not
an opaque archive), matching the plain-mirror + Proton-E2E design. Checked Proton quota (2 TiB
plan, 1.945 TiB free) before enabling a large folder. `folders.list` set to the real selection:
`cloud` (287G), `documents` (249M), `repos` (83M); a full sync of that set was kicked off via the
actual `rclone-backup.service` unit (not an ad-hoc call) to validate the real monthly path early
rather than waiting for the Aug 1 timer. Tracked-repo step (`dtoro/rclone` on gitea) deferred by
choice — runner/units/`folders.list` remain plain files on the LXC for now; the repo package stays
staged at `/root/rclone-repo` for later.
### 2026-07-01 — provisioned; enrolled
LXC 132 created (Debian 13, privileged, `192.168.8.214`, `/mnt/library` read-only). rclone v1.74.3
installed from the official binary (`protondrive` backend present). Runner + monthly timer +
`folders.list` deployed; rclone Web GUI (`rcd`, LAN-only no-auth) live on `:5572`. Enrolled into
homelab-context (`--no-mesh`, LAN-only issuance): age key issued, inventory finalized, shared
secrets granted, `homelab whoami` + `homelab secret hello` verified. Gitea `ALLOWED_HOST_LIST`
updated to include `192.168.8.214`. Hit and fixed a `pct exec` PATH gotcha (see below). Proton Drive
remote, `dtoro/rclone` tracked repo + webhook, and the `secrets/protondrive.yaml` escrow remain
operator-run follow-ups (credentialed steps — Proton password/2FA, repo creation). Restic-on-USB
backup deprecated in the same change.

View File

@@ -0,0 +1,61 @@
# seanime (LXC 133) — Seanime anime media server
## Summary
| Field | Value |
|-------|-------|
| VMID | 133 |
| Host | strong |
| Role | Anime media server (online streaming + torrent client) |
| LAN IP | 192.168.8.248/28 (vmbr1) |
| Public URL | https://seanime.hubris.network |
| Backend | Seanime v3.9.0, systemd service |
| Port | 43211 |
## Mounts
- `/anime``/mnt/media_local/anime` (ludo-lvm, bind mount) — existing anime collection
## Services
### Seanime (native binary)
- Binary: `/opt/seanime/bin/seanime`
- Data: `/opt/seanime/data/`
- Config: `/opt/seanime/data/config.toml`
- Service: `systemctl status seanime`
- Flags: `--host 0.0.0.0 --port 43211 --datadir /opt/seanime/data --disable-password`
- Config highlights:
- `secureMode = 'lax'` (allows non-local Caddy proxy)
- `trustedProxies = ['192.168.8.0/24']`
### Torrent client (qBittorrent on arriman)
- Host: `192.168.8.245:8080`
- Auth: subnet whitelist (no password needed from homelab LAN)
- qBittorrent config updated: `AuthSubnetWhitelist=192.168.8.0/24`
### Extensions installed
**Online streaming (8):**
HiAnime, AniWatch, KickAssAnime, Anicrush, Animo, AniNeko, Senshi, Sudatchi
## Caddy
- Config: `seanime.hubris.network` block in `/etc/caddy/Caddyfile` (dtoro/caddy-conf repo)
- TLS: Let's Encrypt DNS-01 via IONOS API
- Backend: `192.168.8.248:43211`
## DNS
- `seanime.hubris.network` A record → `192.168.8.175` (Caddy)
## Changelog
### 2026-07-05 — initial creation
- LXC 133 created on strong (Debian 13, 2 cores / 2 GiB)
- Seanime v3.9.0 installed as native binary + systemd service
- /anime bind mount from ludo-lvm
- qBittorrent on arriman configured as torrent client
- Caddy reverse proxy + DNS records set up
- Online streaming extensions installed

View File

@@ -0,0 +1,82 @@
# 134 — `romm`
Self-hosted ROM manager ([RomM](https://romm.app)). Browse, search, and play
your retro game library from the browser. Runs on Docker Compose with a
MariaDB sidecar.
## At a glance
- **Hostname:** `romm`
- **IP:** `192.168.8.249/28` (static, vmbr1 on strong)
- **Privilege:** privileged
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13, ludo-lvm)
- **Mounts:** `/mnt/media_local``/mnt/library`
- **Public hostname:** `roms.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| RomM | `192.168.8.249:80` | HTTP (Caddy terminates TLS) |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
Located at `/opt/romm/docker-compose.yml`. Key points:
- Image: `rommapp/romm:latest`
- DB sidecar: `mariadb:latest` with healthcheck
- ROM library: `/mnt/library/roms``/romm/library` (writable)
- Resources (covers, etc.): Docker named volume `romm_resources``/romm/resources`
- Saves/states: `/opt/romm/assets``/romm/assets`
- Config: `/opt/romm/config``/romm/config`
- Auth key: auto-generated, stored in `/opt/romm/.env`
Environment (`/opt/romm/.env`):
- `DB_ROOT_PASSWD` / `DB_PASSWD` — MariaDB credentials
- `DB_USER=romm-user` / `DB_NAME=romm`
- `ROMM_AUTH_SECRET_KEY` — auto-generated
## ROM library structure
RomM expects `/mnt/library/roms/<platform>/<game>/<rom>`. Create platform
directories as needed:
```
/mnt/media_local/roms/
├── gba/
│ └── Pokemon - Emerald/
│ └── Pokemon Emerald.gba
├── snes/
│ └── Super Mario World/
│ └── Super Mario World.sfc
└── psx/
└── Final Fantasy VII/
└── Final Fantasy VII.bin
```
## Media permissions
The `/mnt/media_local/roms` directory is owned `root:media` with mode `2775`
(setgid). New files inherit the `media` GID (10000). The LXC is privileged so
no idmap block is needed — in-container UID/GID matches the host. Docker
containers within the LXC run as-is (read-only mount).
## Related
- [Strong host](../hosts/strong.md)
- [Caddy (121)](121-caddy.md) — `roms.hubris.network → 192.168.8.249:80`
- [DNS (107)](107-dns.md) — `roms.hubris.network A 192.168.8.175`
- [Media permissions](../infrastructure/media-permissions.md)
- [RomM docs](https://docs.romm.app)
## Changelog
### 2026-07-05 — provisioned
LXC 134 created on strong (Debian 13, privileged, `192.168.8.249/28`).
Docker + Compose installed. RomM stack deployed at `/opt/romm/`.
Created `/mnt/media_local/roms` with setgid `media:GID=10000` mode `2775`.
Caddy `roms.hubris.network``192.168.8.249:80`.
DNS `roms A 192.168.8.175` added to Technitium.
Hubris /32 route for `.249` added to `50-strong-route`.

View File

@@ -0,0 +1,57 @@
# LXC containers — index
Most containers live on [`hubris`](../hosts/hubris.md). Some have been
[migrated to `strong`](../hosts/strong.md) (Phase 1+2, 2026-07-05).
| ID | Name | Host | IP | Priv | Cores | RAM | Disk | Mounts | Public hostname | Status |
| --- | ---------------- | ------- | --------------- | ---- | ----- | ----- | ----- | --------------------- | ------------------------------------- | -------- |
| 101 | [jellyfin](101-jellyfin.md) | **strong** | 192.168.8.246 | priv | 4 | 8 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `media.hubris.network` | running |
| 103 | [paperless](103-paperless.md) | hubris | 192.168.8.130 | priv | 2 | 3 GiB | 8 GiB | `/mnt/library` | `paperless.hubris.network` | running |
| 104 | [gitea](104-gitea.md) | hubris | 192.168.8.121 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` | `git.hubris.network` | running |
| 105 | [apps](105-apps.md) | hubris | 192.168.8.205 | priv | 2 | 4 GiB | 30 GiB | `/mnt/library` | `docker` / `artifacto` / `blog` | running |
| 114 | [nextcloud](114-nextcloud.md) | hubris | 192.168.8.224 | priv | 4 | 6 GiB | 25 GiB | `/mnt/library` | `cloud.hubris.network` | running |
| 118 | [elementsynapse](118-elementsynapse.md) | **strong** | 192.168.8.242 | unpriv | 2 | 4 GiB | 32 GiB | — | `matrix.hubris.network` | running |
| 119 | [sophia](119-sophia.md) | hubris | 192.168.8.157 | priv | 2 | 1 GiB | 10 GiB | `/mnt/library` | — | running |
| 120 | [mule-images](120-mule-images.md) | hubris | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU) | `photos.hubris.network` | running |
| 121 | [caddy](121-caddy.md) | hubris | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running |
| 122 | [arriman](122-arriman.md) | **strong** | 192.168.8.245 | priv | 4 | 8 GiB | 24 GiB | `/mnt/media_local` (via mp0) | `jellyseerr` / `qbit` / `sab` | running |
| 124 | [authentik](106-auth-outpost.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running |
| 128 | [trmnl](128-trmnl.md) | hubris | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running |
| 129 | [house](129-house.md) | **strong** | 192.168.8.244 | unpriv | 2 | 3 GiB | 8 GiB | — | `house.hubris.network` | running |
| 130 | [grimmory](130-grimmory.md) | **strong** | 192.168.8.247 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `books.hubris.network` | running |
| 131 | [teddycloud](131-teddycloud.md) | hubris | 192.168.8.150 | — | 1 | 1 GiB | 16 GiB | `/mnt/library` | `teddy.hubris.network` (no auth gate) | running |
|| 132 | [rclone](132-rclone.md) | hubris | 192.168.8.214 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` (**ro**) | — (LAN-only UI `:5572`) | running |
|| 134 | [romm](134-romm.md) | **strong** | 192.168.8.249 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `roms.hubris.network` | running |
## Recently destroyed (kept for archaeology)
| ID | Name | Destroyed | Reason |
| --- | ---------------- | --------------- | --------------------------------------------- |
| 127 | mule-photos-new | 2026-05-22 | PhotoPrism + sidecar + SvelteKit stack promoted to LXC 120 via Mulimage 2.0 merge (`70dc1b6`); M0 test LXC retired. Caddy + dnsmasq + gitea webhook + NC webhook listeners all cleaned up in the same cutover. |
| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose |
| 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
| 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology |
| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) |
| 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty |
| 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 110 | photoprism | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 111 | karakeep | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 112 | immich | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 115 | reticulum | between 2026-04-21 and 2026-04-28 | Decommissioned |
> Several `.conf.bak` files survive under `/etc/pve/lxc/` if you need to recover any of the configs.
## Conventions
- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](106-auth-outpost.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range.
- `onboot=1` on every container — the host brings them up after `pve-guests.service`.
- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library` on hubris, or `mp0: /mnt/media_local,mp=/mnt/library` on strong.
- Most containers are privileged. Unprivileged ones require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
## Related
- [Hubris host](../hosts/hubris.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Caddy](121-caddy.md) — terminates every public hostname
- [DNS](../infrastructure/dns.md) — split-horizon entries for each subdomain

View File

@@ -1,11 +1,14 @@
# `hubris` — Proxmox host
Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
`Homelab` cluster (see [Cluster](#cluster)); the second node is
[strong](strong.md), which hosts nothing yet.
## At a glance
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
@@ -22,13 +25,37 @@ Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
## Cluster
Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md)
(cluster/OS hostname `strong`), formed 2026-07-01.
- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address).
strong reaches it via the existing Fritz!Box static route
(`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the
household LAN. Fine for a home cluster; not latency-isolated.
- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both
votes — if either node is down (reboot, maintenance, network hiccup), the
survivor's running guests keep working but `/etc/pve` goes read-only:
no start/stop/create/edit until quorum returns. Decided to skip a QDevice
for now; revisit if hubris's periodic reboots (BIOS/thermal work, see
Quirks below) make this painful in practice.
- **Storage:** `local` / `local-lvm` are the standard per-node default IDs
(every node has its own, not actually shared). The `library` lvmthin pool
is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since
it's a physical thinpool that only exists on this host's hardware.
- strong currently hosts no LXCs/VMs — it exists solely as a cluster
member so far. See [strong.md](strong.md) and the [library-SSD
migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)
for what comes next (physical drive move, service migration — not started).
## Tenants
### VMs
- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
### LXC containers
See [containers/index](../containers/index.md). 13 active (109 syncthing destroyed 2026-05-14).
See [containers/index](../containers/index.md). 10 active on hubris (101, 118, 122, 129, 130 migrated to [strong](strong.md) 2026-07-05).
## Boot-time tuning (load-bearing)
@@ -62,7 +89,7 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b
## Quirks
- `/etc/pve` is fuse — normal for the Proxmox cluster filesystem, even on a single-node install.
- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
- ZFS is **not** in use; storage is LVM-thin + ext4.
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
@@ -73,6 +100,8 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b
- `root@hubris` (self, RSA) — local
- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
`pvecm add` could authenticate without a password prompt
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
@@ -84,16 +113,20 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220
- [Media permissions](../infrastructure/media-permissions.md)
- [Monitoring](../infrastructure/monitoring.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Operations cheatsheet](../operations/commands.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
- [Operations cheatsheet](../../../.agents/operations/commands.md)
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
- [strong — Proxmox host](strong.md)
## Changelog
### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-05-14 — LXC 109 (syncthing) decommissioned
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
### 2026-05-14 — network performance baseline captured
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
@@ -105,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`.
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
### 2026-04-29 — relocated to better-ventilated spot
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 3435.5 °C, ACPI zone 4749 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 6061 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 3435.5 °C, ACPI zone 4749 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 6061 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
### 2026-04-28 — Phase 1 WiFi failover
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
@@ -114,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
### 2026-04-23 — SSD cooling + thermal pads installed
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 6061 °C, nvme1 3840 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 6061 °C, nvme1 3840 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
### 2026-04-22 — drive removal A/B test
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
@@ -126,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
### 2026-04-21 — `cpu-epp.service` deployed
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md).
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).

View File

@@ -0,0 +1,9 @@
# Hosts
The two Proxmox VE nodes that run the fleet. Narrative pages; the machine-readable host records are
the generated `hosts/*.yaml` files at the repo root.
| Document | What it covers |
|----------|----------------|
| [hubris.md](hubris.md) | Primary PVE node (GMKtec NucBox M6 Ultra, `192.168.8.77`) — runs most LXCs plus the VMs. |
| [strong.md](strong.md) | Secondary PVE node / cluster member (`192.168.178.181`) — hosts the LXCs migrated from hubris. |

View File

@@ -0,0 +1,159 @@
# `strong` — Proxmox host
Second node in the `Homelab` cluster, alongside [hubris](hubris.md). Formerly
a Linux dev workstation nicknamed "ludo" (or "ludo-mini") — reformatted to
Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
## At a glance
- **Role:** Proxmox VE 9.2.3 cluster member (kernel `7.0.12-1-pve`) — hosts [7 LXCs](../containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05)
- **Naming:** the OS/cluster hostname is `strong` (leftover from install,
kept as-is). This wiki page and `inventory.yaml` also use `strong` as of
2026-07-01 — earlier the same day the inventory entry was briefly named
`ludo-mini`, but that was renamed so the hostname bootstrap needs
(`hosts/$(hostname).yaml`) would just resolve. "Ludo"/"ludo-mini" remains
the machine's everyday nickname; some older docs (investigations, the
library-SSD migration plan) still refer to it that way — that's fine,
those are historical.
- **Hardware:** AMD Ryzen 7 PRO 6850U, 16 threads, 28 GiB RAM.
- `nvme0n1` — MasonSemi MC3100 1TB (boot/OS): `pve-root` 96G, `pve-data`
(thinpool) 815G, 8G swap.
- `nvme1n1` — WD_BLACK SN7100 2TB, added 2026-07-01. Arrived with an
existing APFS partition (previously used in a Mac) — wiped via
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1`, then turned
into its own LVM-thin pool via `pvesh create nodes/strong/disks/lvmthin`
(handles pvcreate/vgcreate/lvcreate + storage.cfg registration in one
step). Registered as Proxmox storage `ludo-lvm` (VG `ludo-lvm`, thinpool
`ludo-lvm`, ~1.8 TiB usable — kept the `ludo-lvm` name since that's what
was actually created live; not worth renaming), restricted `nodes strong`
in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool.
Empty so far; this is separate from the [library-SSD migration
plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s
planned drive move from hubris (that hasn't happened) — this is
general-purpose VM/CT capacity.
- **Network:** `vmbr0` is bridged straight onto the household LAN —
`192.168.178.181/24`, gateway `192.168.178.1`. Unlike hubris, there is no
internal/uplink bridge split yet — `vmbr0` here plays the role hubris's
`vmbr1` plays there. Reachable from the homelab subnet (`192.168.8.0/24`)
via the existing Fritz!Box static route through hubris; no new cabling
or routing was needed to join the cluster.
- **Storage:** `local` + `local-lvm` (cluster-wide default names, shared
with hubris by convention, actually separate per-node volumes). Hubris's
`library` lvmthin pool is restricted to `nodes hubris` in
`/etc/pve/storage.cfg` and does not exist here.
- **Mesh:** Netbird not installed — fresh OS wiped whatever the old
workstation had. Reachable today only via LAN routing (confirmed DNS for
`*.hubris.network` already resolves correctly here via `192.168.8.2`).
Add to Netbird if off-LAN access to this host itself (distinct from any
future guests) is needed.
- **Homelab-context client enrollment:** done 2026-07-01 via
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
5-min sync timer are live; `homelab whoami` resolves correctly. See
[agent-enrollment.md](../../../.agents/operations/agent-enrollment.md).
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
## Cluster membership
Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on
2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full
cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice —
either node going down freezes management on the survivor).
## SSH
Root login via the same key set trusted on hubris (`root@hubris`,
`d.toro.v@pm.me`) — installed 2026-07-01 by appending to
`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys`
post cluster-join, so it's cluster-synced same as hubris). No password auth
needed going forward.
## Related
- [hubris — Proxmox host](hubris.md)
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
- [Network](../infrastructure/network.md)
- [SSH access](../infrastructure/ssh-access.md)
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
## Changelog
### 2026-07-05 — Phase 2d: grimmory (130) migrated to strong
Migrated book library from hubris (192.168.8.213) to strong (192.168.8.247).
Rsync'd /books (2.6G) from hubris library SSD to ludo-lvm/media. Container
mounts /mnt/media_local → /mnt/library locally. Caddy backend for
books.hubris.network updated.
### 2026-07-05 — Phase 2: arriman (122) + jellyfin (101) migrated; library on ludo-lvm
Migrated arriman (→ 192.168.8.245) and jellyfin (→ 192.168.8.246) to strong.
Created 1.5T thin volume on ludo-lvm for media library (/mnt/media_local).
Rsync'd 363G of media data (movies, tv, anime, downloads, music) from hubris
library SSD to ludo-lvm. Both containers now mount /mnt/media_local directly
as local ext4 — no NFS cross-node dependency. Jellyfin gets Radeon 680M iGPU
(for VAAPI) + RX 7600 dGPU passthrough via dev0/dev1. Caddy backends updated
for media, jellyseerr, qbit, sab. Hubris freed 16 GiB RAM (8+8).
Dhcp scope narrowed to 192.168.8.100-239.
### 2026-07-05 — house (LXC 129) migrated to strong; DHCP scope narrowed
Migrated Yuvomi family planner from hubris (192.168.8.212) to strong
(192.168.8.244). Followed same restore pattern as elementsynapse (Phase 1b).
Discovered IP conflict: teddycloud (LXC 131) had 192.168.8.243 via DHCP
(scope was 192.168.8.241-254). Narrowed DHCP scope to 192.168.8.100-239,
gave teddycloud static IP 192.168.8.150. Caddy updated: house.hubris.network
→ 192.168.8.244:3000, teddy.hubris.network → 192.168.8.150:8443.
### 2026-07-05 — elementsynapse (LXC 118) migrated to strong
Migrated Matrix/Synapse + Element client from hubris (192.168.8.239) to a new
LXC on strong (192.168.8.242). Added vmbr1 on strong at 192.168.8.241/28
(portless internal bridge) for strong-hosted guests. Enabled IP forwarding,
proxy ARP on hubris vmbr0, and specific /32 routes for strong's guest subnet.
Caddy updated: element.hubris.network → 192.168.8.242:8080.
VPS traefik updated: matrix.hubris.network backend → 192.168.8.242:8008.
This is Phase 1a of the strong migration plan — see .hermes/plans/2026-07-05_strong-migration-assessment.md.
### 2026-07-01 — age key issued over LAN; 3 bugs found/fixed in bootstrap.sh
Re-ran bootstrap without `--no-secrets` to get a real age key. Hit three real bugs live, fixed all three in `bootstrap.sh` and re-ran clean:
1. The `mcp`-CLI pipx-install step and the (unused, `--with-hermes`-only) Goose installer both called `sudo -u <user>` unconditionally — fails with "sudo: command not found" on a minimal root-only image with no `sudo` binary at all. Added a `run_as()` helper that only shells out to `sudo` when there's a real distinct invoking user.
2. `sops` isn't an apt/dnf package (matches what `agent-enrollment.md`'s manual-install recipe already does) — the auto-installer tried `apt-get install sops` and failed outright. Added `install_sops_binary()`, fetching the GitHub release binary directly on both dnf and apt paths.
3. Bigger one: running without `--no-secrets` unconditionally tries to install + interactively connect Netbird (device-code SSO), even though the very next check already knows how to accept plain LAN reachability instead. Over SSH with nobody watching, this hangs forever — had to manually kill a stuck `netbird up` process. Added `--no-mesh`, which skips the Netbird install/connect step but keeps the LAN-fallback path for secrets issuance. This run used `bootstrap.sh --no-mesh` and completed cleanly: `mesh: lan`, age key installed, `mcp` CLI installed via pipx (proving fix #1 too).
Result: age key at `/etc/age/key.txt`, pubkey recorded in `inventory.yaml`. Not yet granted access to any actual secret file — see the note above.
### 2026-07-01 — enrolled as a homelab-context client
Ran `bootstrap.sh --no-secrets` (reused the operator's existing personal Gitea PAT for the initial clone rather than minting a fresh read-only one). Installed git, cloned `/opt/homelab-context`, installed the 5-min systemd sync timer, symlinked `homelab` CLI and `AGENTS.md`. Skipped age-key/secrets issuance and Netbird per operator choice — but bootstrap's own connectivity check reported `mesh: lan`, i.e. the secrets-issuance endpoint is already reachable over plain LAN, so re-running without `--no-secrets` later wouldn't require a Netbird join. Known gap: the `mcp` pipx CLI install step silently failed (`sudo: command not found` — bootstrap.sh's pipx step assumes a `sudo` binary even when already root; harmless, only affects the `homelab mcp <tool>` shell subcommand).
### 2026-07-01 — inventory identity renamed ludo-mini → strong
Discovered while starting client enrollment: `bootstrap.sh` looks up
`hosts/$(hostname).yaml`, and the OS hostname here is `strong`, not
`ludo-mini`. Renaming the OS hostname was ruled out (already a cluster
member — Proxmox doesn't support in-place node rename, only leave+rejoin).
Renamed the wiki/inventory side instead: `inventory.yaml` key, this page
(`hosts/ludo-mini.md``hosts/strong.md`), README, ssh-access.md all now
say `strong`. "Ludo"/"ludo-mini" is still fine as a spoken nickname.
### 2026-07-01 — 2nd NVMe added; new LVM-thin pool `ludo-lvm`
User added a WD_BLACK SN7100 2TB (`nvme1n1`), previously used in a Mac
(arrived with an EFI + APFS partition table — confirmed disposable, wiped).
Used Proxmox's own disk-management API rather than raw LVM commands:
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1` to clear the old
partition table/signatures, then `pvesh create nodes/strong/disks/lvmthin
--name ludo-lvm --device /dev/disk/by-id/nvme-WD_BLACK_SN7100_2TB_251663803202
--add_storage 1` to create the PV/VG/thinpool and register it as Proxmox
storage in one step. Result: storage ID `ludo-lvm`, ~1.8 TiB, `content
rootdir,images`, `nodes strong` (mirrors hubris's `library` node-restriction
pattern — this pool only physically exists here). Empty — no VM/CT disks
placed on it yet.
### 2026-07-01 — Proxmox install; joined Homelab cluster
Reformatted from Linux workstation to Proxmox VE 9.2.3. SSH keys seeded from
hubris's trusted set (root password used once, then discarded). Joined the
existing `Homelab` cluster via `pvecm add 192.168.8.77 --use_ssh 1` from
this node's side, using key-based SSH pre-authorized in both directions —
no interactive password prompt needed for the join itself. Cluster now 2
nodes (`hubris`, `strong`), quorate, no QDevice. Decided to leave hostname
as `strong` and skip a QDevice for now — both revisitable later.

View File

@@ -0,0 +1,14 @@
# Knowledge
The durable, authoritative current-state documentation of the homelab: one page per node and per
cross-cutting system, synthesized from live state and evidence. Structure and rules are in
[the knowledge schema](../.agents/domains/knowledge/schema.md).
| Section | What it covers |
|---------|----------------|
| [wiki/hosts/](wiki/hosts/index.md) | Proxmox host narratives — `hubris`, `strong`. |
| [wiki/containers/](wiki/containers/index.md) | LXC fleet — one page per container, plus the master table and archaeology. |
| [wiki/vms/](wiki/vms/index.md) | Virtual machines — ZimaOS, Home Assistant OS. |
| [wiki/infrastructure/](wiki/infrastructure/index.md) | Cross-cutting systems — DNS, ingress, mesh, storage, auth, monitoring, generated topology. |
| [sources/](sources/index.md) | External reference docs and the pointer to incident evidence. |
| [GLOSSARY.md](GLOSSARY.md) | Term definitions. |

View File

@@ -23,7 +23,8 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
- `192.168.8.190` ([plato (126)](../containers/126-plato.md))
- ~~`192.168.8.190` ([plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology))~~ (destroyed 2026-06-28)
- `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito)
**Don't strip these when editing app.ini.**
@@ -37,17 +38,19 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
| `dtoro/Plato` | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` |
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology) `/opt/plato/app/`~~ (destroyed 2026-06-28) | | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 (deprecated) | ~~reinstalls `homelab-mcp.service` + restart~~ → replaced by Go Docker stack on mac-mini |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 (deprecated) | ~~reinstalls `secrets-issuance.service` + restart~~ → replaced by `internal/secrets/` Go package |
| `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` |
| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) |
> Note: `dtoro/Homelab-Docs` has **two webhooks** firing on the same push.
> Note: `dtoro/Homelab-Docs` has **three webhooks** firing on the same push.
> Each owns its own clone on LXC 105. They don't conflict because each
> deploy.sh only touches its own service unit + venv.
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.yaml). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
## When you change a tracked config
@@ -57,7 +60,7 @@ Always commit + push. Local-only edits drift. Common ones:
- `/var/lib/gitea/custom/``dtoro/gitea-customizations` (auto-deploys)
- `/opt/artifacto/``dtoro/Artifacto` (auto-deploys)
- `/opt/mule-image/``dtoro/mule-image` (auto-deploys)
- `/opt/plato/app/``dtoro/Plato` (auto-deploys)
- ~~`/opt/plato/app/` ↔ `dtoro/Plato`~~ (destroyed 2026-06-28)
- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
- `/opt/backup-library/``dtoro/backup-library` (auto-deploys)
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/``dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
@@ -72,11 +75,6 @@ Always commit + push. Local-only edits drift. Common ones:
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
### Plato
- Shape B (`/opt/plato-deploy/{webhook.py,deploy.sh}`, port `9799`).
- The in-LXC checkout's `origin` is `http://192.168.8.121:3000/dtoro/Plato.git` (internal gitea), and git creds are at `/root/.git-credentials` rather than the `/etc/plato-deploy/git-credentials` pattern — the unit doesn't set `ProtectHome` so root's home is reachable.
- `/data` is a host bind (`/mnt/library/documents/plato`), so `docker compose up -d --build` rebuilds the image + restarts the container without touching the SQLite db. The [fresh-DB bootstrap workaround](../containers/126-plato.md#fresh-db-bootstrap-workaround) only matters if you blow `plato.db` away.
### mule-image / Artifacto
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
@@ -119,15 +117,21 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
- [Backups (disabled)](backups.md)
- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
- [Operations cheatsheet](../../../.agents/operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
## Changelog
### 2026-06-28 — Plato pipeline decommissioned
LXC 126 destroyed, webhook id 8 on `dtoro/Plato` removed. `192.168.8.190` removed from gitea `app.ini` `ALLOWED_HOST_LIST`.
### 2026-06-24 — terminalito pipeline added
Webhook id 12 on `dtoro/terminalito``http://192.168.8.211:9797/deploy` on [trmnl (128)](../containers/128-trmnl.md). Shape B (`server/deploy/webhook.py` receiver, in-repo `server/deploy/deploy.sh`; secret `/etc/terminalito-deploy/secret`). `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. Verified end-to-end with a push. Repo-local `credential.helper` in `/opt/terminalito/.git/config` (the unit can't read root's global git config).
### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added
Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
### 2026-05-13 — Plato pipeline added
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
### 2026-04-28 — wiki entry created
Initial documentation. Six active pipelines.

View File

@@ -1,6 +1,30 @@
# Backups — restic on external drive (DISABLED)
# Backups — restic on external drive (DEPRECATED — superseded)
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md).
> **DEPRECATED 2026-07-01.** Superseded by the **rclone → Proton Drive** off-host mirror on
> [LXC 132 `rclone`](../containers/132-rclone.md). That job finally closes the off-host / 3-2-1 gap
> this page flagged for months. The restic-on-USB job below is kept for archaeology; it has been
> **DISABLED since 2026-04-22** and is not coming back in its old form.
## Current backup — rclone → Proton Drive (LXC 132)
- **Where:** [LXC 132 `rclone`](../containers/132-rclone.md) (`192.168.8.214`), `/mnt/library`
mounted **read-only**.
- **What:** plain `rclone sync` (Proton mirrors local; browsable files, no versioning) of the
folders listed in `/etc/rclone-backup/folders.list`, to `proton:library-backup/…`.
- **When:** monthly — `rclone-backup.timer` (`OnCalendar=*-*-01 03:00`).
- **UI:** rclone Web GUI on `192.168.8.214:5572` (LAN-only, no auth).
- **Encryption:** Proton's built-in E2E (no rclone `crypt` overlay).
- **Runs / logs:** `/var/log/rclone-backup/` + `runs.jsonl`.
- **Still a single off-host target** (Proton only). Not yet a full 3-2-1 (no second independent
copy), but strictly better than the previous "no off-host copy at all."
See [132-rclone](../containers/132-rclone.md) for the full design.
---
## Legacy — restic on external drive (DISABLED 2026-04-22)
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
## Status
@@ -12,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
@@ -81,13 +105,13 @@ Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/R
## Known SPOF
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented.
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) were not implemented before this job was retired — the **off-host copy is now provided by [rclone → Proton Drive (LXC 132)](../containers/132-rclone.md)** instead. A second independent copy is still outstanding.
## Drive history
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
## Thermal monitoring
@@ -95,18 +119,21 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
## Related
- [Hubris host](../hosts/hubris.md)
- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04)
- [Monitoring](monitoring.md)
- [Auto-deploy](auto-deploy.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
## Changelog
### 2026-07-01 — DEPRECATED; superseded by rclone → Proton Drive (LXC 132)
Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclone`](../containers/132-rclone.md) (`/mnt/library` → Proton Drive, monthly, LAN Web GUI). This finally provides the off-host copy the "Known SPOF" note wanted. The restic-on-USB units on hubris remain `disabled` (drive already removed 2026-04-22); page restructured to lead with the current job and demote restic to "Legacy".
### 2026-04-28 — wiki entry created
Initial documentation. Status remains DISABLED.
### 2026-04-22 — DISABLED
Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.

View File

@@ -7,7 +7,7 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
## Components
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/106-auth-outpost.md) (decommissioned 2026-06-04).
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
@@ -32,7 +32,6 @@ address=/blog.hubris.network/192.168.8.175
address=/photos.hubris.network/192.168.8.175
address=/photos-new.hubris.network/192.168.8.175
address=/artifacto.hubris.network/192.168.8.175
address=/plato.hubris.network/192.168.8.175
address=/zimaos.hubris.network/192.168.8.175
address=/nfs-export.hubris.network/192.168.8.200
```
@@ -54,7 +53,7 @@ Creating a new Caddyfile site block is necessary but **not sufficient**. Without
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network``192.168.8.175`.
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../../../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
## Public path — what does and doesn't follow the LAN map
@@ -76,6 +75,9 @@ Either:
## Changelog
### 2026-06-28 — `plato.hubris.network` removed
Plato (LXC 126) decommissioned. Technitium entry deleted; dns-sync cron reaped the NetBird managed zone record.
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
@@ -92,7 +94,7 @@ The "delete NetBird managed zone → forward everything to Technitium" plan was
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
> Reference: [scripts/dns-sync.py](../../../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
@@ -105,10 +107,10 @@ Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backe
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180``192.168.8.2` for any plain-LAN (non-mesh) clients.
Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180``192.168.8.2` for any plain-LAN (non-mesh) clients.
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
@@ -117,7 +119,7 @@ NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
New LAN-only entry for [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.

View File

@@ -5,7 +5,7 @@ Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
self-locating and able to read the same source of truth.
Operational walkthrough for enrolling a new client lives in
[operations/agent-enrollment.md](../operations/agent-enrollment.md); this
[operations/agent-enrollment.md](../../../.agents/operations/agent-enrollment.md); this
page is the architecture reference.
## What's where
@@ -121,7 +121,7 @@ The MCP server and secrets-issuance each have their own clone
## Related
- [Operations: agent enrollment](../operations/agent-enrollment.md) — the
- [Operations: agent enrollment](../../../.agents/operations/agent-enrollment.md) — the
step-by-step for adding a new client
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
pipelines (and the rest of the lab's webhook pipelines)
@@ -133,7 +133,7 @@ The MCP server and secrets-issuance each have their own clone
## Changelog
### 2026-05-20 — system live across hubris, apps, republic-laptop
Phase 1 of the [cross-client context plan](../README.md) merged. Three
Phase 1 of the [cross-client context plan](../../../README.md) merged. Three
clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS
decrypt verified on each. Webhook auto-deploy for both LXC 105 services
wired (hook ids 10 + 11). `homelab refresh-creds` + atomic

View File

@@ -0,0 +1,66 @@
# Infrastructure — cross-cutting systems
The homelab's shared infrastructure: systems that span multiple nodes and
are documented in their own pages. Each system below links to its full doc.
## Network
- **[Network](network.md)** — physical topology, subnets, routing, DHCP.
Homelab `192.168.8.0/24` isolated from household `192.168.178.0/24`.
Proxmox hubris acts as subnet router.
- **[DNS — split-horizon](dns.md)** — Technitium DNS on LXC 107,
`192.168.8.2:53`. `*.hubris.network` resolves to LAN IPs on the homelab
network and to mesh addresses off-LAN.
## Connectivity / mesh
- **[Mesh — Tailscale → Netbird migration](mesh.md)** — overlay networking.
Netbird is the preferred path; Tailscale is legacy.
- **[SSH access](ssh-access.md)** — dual-server SSH (OpenSSH + Netbird SSH)
on hubris, key distribution.
## Public ingress
- **[Public ingress — VPS traefik + cert mirror](ingress.md)** — how home
services reach the open internet. Two-stage: VPS traefik (IONOS) terminates
TLS, proxies over Netbird to home Caddy.
- **[Caddy reverse proxy](../containers/121-caddy.md)** — LAN endpoint.
Terminates TLS for every `*.hubris.network` hostname, forwards to backends.
## Storage
- **[Media permissions — GID 10000 standard](media-permissions.md)** — shared
group permission model across all LXCs that read/write the media library.
- **[Backups — rclone → Proton Drive](backups.md)** — off-host backup strategy.
LXC 132 handles rclone to Proton Drive; restic-on-USB deprecated.
## Identity & access
- **[Authentik SSO](../containers/106-auth-outpost.md)** — identity provider.
Core server runs on the VPS; LAN forward-auth outpost at LXC 106.
OIDC providers configured for Jellyfin, Jellyseerr, Sabnzbd, qBittorrent,
Yuvomi, and more.
## Management & automation
- **[Homelab context distribution](homelab-context.md)** — `/opt/homelab-context`
clone, MCP server, secrets issuance, cross-client sync.
- **[Auto-deploy — gitea-webhook pipelines](auto-deploy.md)** — push-to-deploy
for Caddy config, mule-image, and other tracked repos.
- **[Monitoring](monitoring.md)** — health checks, watchdogs, alerting
(migrated from claudio-bot to Hermes cron).
- **[VPS hardening](vps-hardening.md)** — IONOS netbird VPS: fail2ban,
nftables, OIDC SSH, security posture.
## Topology
- **[Topology diagram (generated)](topology.md)** — Mermaid graph of compute,
ingress routing, and storage mounts. Auto-generated from `inventory.yaml`
by `oikos/gen-topology.py` (Python — Go DB-native replacement planned).
## Related
- [README](../../../README.md) — entry point
- [Containers index](../containers/index.md)
- [Operations cheatsheet](../../../.agents/operations/commands.md)
- [OIKOS operating model](../../../.agents/OIKOS.md)

View File

@@ -42,12 +42,14 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8
| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ |
| `artifacto.hubris.network` | `/p/*`, `/static/*`, `/healthz` | `192.168.8.205:3100` | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key` |
| `blog.hubris.network` | whole host | `192.168.8.205:8080` | `blog-ratelimit` (100 rps / 200 burst) | `blog.fullchain.crt` / `blog.privkey.key` |
| `trmnl.hubris.network` | whole host | `192.168.8.211:9851` ([trmnl 128](../containers/128-trmnl.md)) | `trmnl-ratelimit` (20 rps / 40 burst) | `trmnl.fullchain.crt` / `trmnl.privkey.key` |
| `house.hubris.network` | whole host | `192.168.8.212:3000` ([house 129](../containers/129-house.md)) | `house-ratelimit` (30 rps / 60 burst) | `house.fullchain.crt` / `house.privkey.key` |
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
Since 2026-05-31 [Authentik runs on the VPS itself](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
@@ -85,8 +87,11 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `
## Changelog
### 2026-06-24 — `trmnl.hubris.network` exposed
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public``192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
Authentik migrated onto the VPS ([investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
### 2026-04-28 — wiki entry created
Initial documentation.

View File

@@ -37,7 +37,7 @@ Every LXC that mounts `/mnt/library` participates in a shared `media` group with
- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000).
- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid):
- `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / booklore
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / grimmory
- `downloads` — \*arr stack output
- `images` — photoprism / immich / mulita
- `roms` — emu frontends
@@ -61,6 +61,8 @@ Every LXC that mounts `/mnt/library` participates in a shared `media` group with
| 119 | [sophia](../containers/119-sophia.md) | priv | www-data |
| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data |
| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
| 130 | [grimmory](../containers/130-grimmory.md) | priv | Docker container uses `GROUP_ID=10000` env var (linuxserver pattern) — no in-LXC group needed |
| 132 | [rclone](../containers/132-rclone.md) | priv | **read-only** mount; runs as root → reads all subtrees. No media group needed |
> Some entries from earlier snapshots — 100 (arr-yunohost), 107 (marimo), 109 (syncthing), 110 (photoprism), 112 (immich), 116 (heaper) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).
@@ -68,7 +70,7 @@ Config backups: `/root/101.conf.bak.*`, `/root/109.conf.bak.*` (109 destroyed 20
## Gotchas
- **[apps (105)](../containers/105-apps.md) is a Docker host.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000` or `user: "<uid>:10000"` in compose. Booklore, audiobookshelf-in-docker, etc. need this per-container.
- **[apps (105)](../containers/105-apps.md) and [grimmory (130)](../containers/130-grimmory.md) are Docker hosts.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000`, `user: "<uid>:10000"`, or `GROUP_ID=10000` (linuxserver images) in compose. Grimmory, audiobookshelf-in-docker, etc. need this per-container.
- **`pct exec` does NOT run initgroups.** So `pct exec <id> -- id` shows only the primary group. For interactive verification, use `pct exec <id> -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine.
- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed.
- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page.

View File

@@ -109,7 +109,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Related
- [DNS split-horizon](dns.md)
- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides
- [Authentik (124)](../containers/106-auth-outpost.md) — the IdP that triggers most of these overrides
- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B
- [Gitea (104)](../containers/104-gitea.md) — example of Technique A
- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport
@@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Changelog
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik``192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md).
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik``192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
@@ -139,11 +139,11 @@ The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Ca
### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/106-auth-outpost.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/124-authentik.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/106-auth-outpost.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
**Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21):

View File

@@ -64,7 +64,7 @@ No NAT on Proxmox — traffic flows without double-NAT.
## Remote access
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../vps/).
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../../../vps/).
- **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md).
## Related
@@ -79,10 +79,10 @@ No NAT on Proxmox — traffic flows without double-NAT.
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2)
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed.
Previous pool `.100.240` overlapped with all static LXCs/VMs (` .101.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
Previous pool `.100.240` overlapped with all static LXCs/VMs (` .101.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-06-02 — Executed migration; Proxmox as subnet router
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100240`. Caddy service unit was missing and recreated. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).

View File

@@ -97,19 +97,23 @@ When onboarding a new machine:
## Hosts
### Hubris (PVE host)
### Hubris + strong (PVE cluster: `Homelab`)
| Detail | Value |
|--------|-------|
| LAN IP | `192.168.8.77` |
| Netbird | `100.122.38.109` (FQDN: `proxmox-server.netbird.selfhosted`) |
| Netbird SSH port | `22022` (mesh-only, OIDC auth) |
| SSH user | `root` |
| Authorized keys | `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced) |
Both nodes share `/etc/pve/priv/authorized_keys` — it's Proxmox
cluster-synced, so a key added on either node is authorized on both.
Authorized root keys currently deployed:
| Detail | hubris | strong |
|--------|--------|-----------|
| LAN IP | `192.168.8.77` | `192.168.178.181` |
| Cluster node name | `hubris` | `strong` (OS hostname kept as-is from install) |
| Netbird | `100.122.38.109` (`proxmox-server.netbird.selfhosted`) | not enrolled yet |
| Netbird SSH port | `22022` (mesh-only, OIDC auth) | n/a |
| SSH user | `root` | `root` |
Authorized root keys currently deployed (cluster-wide):
- `root@hubris` (self, RSA)
- `d.toro.v@pm.me` (ed25519) — mac-mini
- `root@strong` (RSA) — strong's own key, added 2026-07-01 for the cluster join
### LXCs
@@ -131,7 +135,6 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
| 120 | mule-images | `192.168.8.136` | photo-management |
| 121 | caddy | `192.168.8.175` | reverse-proxy |
| 122 | arriman | `192.168.8.132` | arr-stack |
| 126 | plato | `192.168.8.190` | app |
### Workstations
@@ -139,7 +142,8 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
|------|----|--------|--------------|----------|
| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
| ludo-mini | Linux | `192.168.8.133` | `ludo-mini.netbird.selfhosted` | TBD |
strong moved out of this table 2026-07-01 — it's a Proxmox host now, see the cluster table above.
### VPS (external)
@@ -174,11 +178,14 @@ done
- [Mesh migration](mesh.md)
- [VPS hardening](vps-hardening.md)
- [Agent enrollment](../operations/agent-enrollment.md)
- [Homelab CLI](../bin/homelab)
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
- [Homelab CLI](../../../bin/homelab)
## Changelog
### 2026-07-01 — strong reformatted to Proxmox, joined cluster; table corrected
strong moved from the Workstations table to the PVE-cluster table (was showing a stale `192.168.8.133`, never actually reachable — the real LAN IP has always been `192.168.178.181`, matching hosts/strong.yaml). Root key access bootstrapped via one-time console password, then key-only going forward. See [hosts/hubris.md#cluster](../hosts/hubris.md#cluster) and [hosts/strong.md](../hosts/strong.md).
### 2026-06-02 — universal SSH reachability
Replaced ad-hoc per-workstation SSH configs with inventory-generated

View File

@@ -0,0 +1,96 @@
<!-- Generated by oikos/gen-topology.py from inventory.yaml. -->
<!-- Do NOT edit by hand - your changes will be overwritten. -->
# Topology (generated)
Source: [inventory.yaml](../../../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services.
Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). Operating model: [OIKOS.md](../../../.agents/OIKOS.md).
## Compute & ingress
```mermaid
flowchart LR
subgraph hubris_sub["hubris (Proxmox)"]
trmnl["trmnl<br/>LXC 128<br/>trmnl-middleware<br/>192.168.8.211"]
nfs_export["nfs-export<br/>LXC 102<br/>storage-export<br/>192.168.8.200"]
paperless["paperless<br/>LXC 103<br/>document-archive<br/>192.168.8.130"]
gitea["gitea<br/>LXC 104<br/>git-server<br/>192.168.8.121"]
apps["apps<br/>LXC 105<br/>docker-apps<br/>192.168.8.205"]
auth_outpost["auth-outpost<br/>LXC 106<br/>authentik-gateway<br/>192.168.8.6"]
dns["dns<br/>LXC 107<br/>dns-server<br/>192.168.8.2"]
nextcloud["nextcloud<br/>LXC 114<br/>file-sync<br/>192.168.8.224"]
sophia["sophia<br/>LXC 119<br/>workshop<br/>192.168.8.109"]
mule_images["mule-images<br/>LXC 120<br/>photo-management<br/>192.168.8.136"]
caddy["caddy<br/>LXC 121<br/>reverse-proxy<br/>192.168.8.175"]
teddycloud["teddycloud<br/>LXC 131<br/>teddycloud<br/>192.168.8.150"]
zimaos["zimaos<br/>VM 100<br/>nas-frontend-eval<br/>192.168.8.195"]
haos["haos<br/>VM 108<br/>home-automation<br/>192.168.8.101"]
end
subgraph strong_sub["strong (Proxmox)"]
house["house<br/>LXC 129<br/>family-planner<br/>192.168.8.244"]
jellyfin["jellyfin<br/>LXC 101<br/>media-server<br/>192.168.8.246"]
elementsynapse["elementsynapse<br/>LXC 118<br/>matrix-server<br/>192.168.8.242"]
arriman["arriman<br/>LXC 122<br/>arr-stack<br/>192.168.8.245"]
grimmory["grimmory<br/>LXC 130<br/>book-library<br/>192.168.8.247"]
seanime["seanime<br/>LXC 133<br/>anime-media-server<br/>192.168.8.248"]
romm["romm<br/>LXC 134<br/>rom-manager<br/>192.168.8.249"]
end
rclone["rclone<br/>lxc<br/>backup"]
republic_laptop([republic-laptop<br/>workstation<br/>primary-dev])
mac_mini([mac-mini<br/>workstation<br/>dev<br/>192.168.178.182])
netbird_vps[[netbird-vps<br/>external<br/>netbird-mgmt]]
url_artifacto(["artifacto.hubris.network"]) -->|routes-to| apps
url_authentik(["auth.hubris.network"]) -->|routes-to| netbird_vps
url_gitea(["git.hubris.network"]) -->|routes-to| gitea
url_homelab_mcp(["mcp.hubris.network"]) -->|routes-to| apps
url_jellyfin(["media.hubris.network"]) -->|routes-to| jellyfin
url_matrix(["matrix.hubris.network"]) -->|routes-to| elementsynapse
url_nextcloud(["cloud.hubris.network"]) -->|routes-to| nextcloud
url_paperless(["paperless.hubris.network"]) -->|routes-to| paperless
url_photos(["photos.hubris.network"]) -->|routes-to| mule_images
url_proxmox_ui(["proxmox.hubris.network"]) -->|routes-to| hubris_sub
url_secrets_issuance(["secrets.hubris.network"]) -->|routes-to| apps
url_teddycloud(["teddy.hubris.network"]) -->|routes-to| teddycloud
url_trmnl(["trmnl.hubris.network"]) -->|routes-to| trmnl
url_zimaos(["zimaos.hubris.network"]) -->|routes-to| zimaos
```
## Storage (mounts)
```mermaid
flowchart LR
mnt_library[("/mnt/library")]
mnt_media_local[("/mnt/media_local")]
mnt_media_local_anime[("/mnt/media_local/anime")]
apps["apps"] -->|mounts| mnt_library
arriman["arriman"] -->|mounts| mnt_media_local
gitea["gitea"] -->|mounts| mnt_library
grimmory["grimmory"] -->|mounts| mnt_media_local
hubris["hubris"] -->|mounts| mnt_library
jellyfin["jellyfin"] -->|mounts| mnt_media_local
mule_images["mule-images"] -->|mounts| mnt_library
nextcloud["nextcloud"] -->|mounts| mnt_library
paperless["paperless"] -->|mounts| mnt_library
romm["romm"] -->|mounts| mnt_media_local
seanime["seanime"] -->|mounts| mnt_media_local_anime
sophia["sophia"] -->|mounts| mnt_library
teddycloud["teddycloud"] -->|mounts| mnt_library
```
## Archaeology (destroyed nodes)
| Node | ID | Destroyed | Reason |
|---|---|---|---|
| plato | 126 | 2026-06-28 | notes workspace decommissioned; data retained at /mnt/library/documents/plato |
| claudio-bot | 123 | 2026-06-04 | replaced by Hermes Agent on mac-mini; monitoring moved to homelab-health-watchdog cron |
| mule-photos-new | 127 | 2026-05-22 | PhotoPrism test stack promoted to LXC 120 (Mulimage 2.0 merge) |
| heaper | 116 | 2026-05-14 | decommissioned; data retained at /mnt/library/heaper |
| syncthing | 109 | 2026-05-14 | decommissioned; library subtree was empty |
| seafile | 125 | 2026-05-13 | Seafile Pro evaluation rejected; files.hubris.network removed from caddy + dns |
| arr-yunohost | 100 | 2026-04-28 | migrated to docker stack on arriman (LXC 122) |
| flaresolverr | 106 | 2026-04-28 | folded into the arriman docker compose |
| marimo | 107 | 2026-04-28 | decommissioned |
| photoprism | 110 | 2026-04-28 | replaced by mule-images (LXC 120) |
| karakeep | 111 | 2026-04-28 | decommissioned |
| immich | 112 | 2026-04-28 | replaced by mule-images (LXC 120) |
| reticulum | 115 | 2026-04-28 | decommissioned |

View File

@@ -2,12 +2,12 @@
## Summary
[`hubris`](../hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
## Timeline
### 2026-04-19 — drive attached
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
### 2026-04-19 → 2026-04-21 — first crashes
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
@@ -24,13 +24,13 @@ Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
### 2026-04-22 — recurrence after 30 h 37 m
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
### 2026-04-22 — `cpu-epp.service` design bug fixed
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
### 2026-04-22 — drive removed (A/B test)
User physically removed the external USB drive. [Backup timers disabled](../infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
### 2026-04-23 — SSD cooling + thermal pads installed
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
@@ -78,9 +78,9 @@ Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the Nu
| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive |
## Affected nodes
- [Hubris host](../hosts/hubris.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Monitoring](../infrastructure/monitoring.md)
- [Hubris host](../../../wiki/hosts/hubris.md)
- [Backups (disabled)](../../../wiki/infrastructure/backups.md)
- [Monitoring](../../../wiki/infrastructure/monitoring.md)
## Open questions
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)

View File

@@ -2,9 +2,9 @@
## Summary
The NetBird management server (on the [VPS](../infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
Resolved by moving Authentik off [LXC 124](../containers/124-authentik.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
@@ -43,7 +43,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../containers/124-authentik.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
@@ -51,7 +51,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
| | Before | After |
|---|---|---|
| Authentik host | [LXC 124](../containers/124-authentik.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
| Version | `2026.2.2` | `2026.5.2` |
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
@@ -76,7 +76,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
@@ -99,11 +99,11 @@ The policy was orphaned (no matched type data or had incompatible evaluation). R
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
- **Reconcile [124-authentik](../containers/124-authentik.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
## Related
- [124 — authentik](../containers/124-authentik.md)
- [DNS split-horizon](../infrastructure/dns.md)
- [Public ingress (VPS traefik)](../infrastructure/ingress.md)
- [Mesh migration](../infrastructure/mesh.md)
- [124 — authentik](../../../wiki/containers/106-auth-outpost.md)
- [DNS split-horizon](../../../wiki/infrastructure/dns.md)
- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md)
- [Mesh migration](../../../wiki/infrastructure/mesh.md)

View File

@@ -2,7 +2,7 @@
## Summary
[`ludo-mini`](../hosts/ludo-mini.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
## Timeline

View File

@@ -90,9 +90,9 @@ print("session_duration:", stage.session_duration) # → "days=30"
## Related
- [Container 106 — auth-outpost](../containers/106-auth-outpost.md)
- [Authentik VPS migration](2026-05-31-authentik-vps-migration.md)
- [Ingress (VPS Traefik)](../infrastructure/ingress.md)
- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md)
- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md)
- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md)
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
## Changelog

View File

@@ -54,8 +54,8 @@ This is the same class of drift as the June 5th incidents (paperless, HAOS, apps
## Related
- [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md)
- [Caddy (121)](../containers/121-caddy.md)
- [elementsynapse (118)](../containers/118-elementsynapse.md)
- [dns-sync script](../scripts/dns-sync.py)
- [check-caddy-backends script](../scripts/check-caddy-backends.sh)
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
- [Caddy (121)](../../wiki/containers/121-caddy.md)
- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md)
- [dns-sync script](../../../scripts/dns-sync.py)
- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh)

View File

@@ -0,0 +1,29 @@
# Investigations
Time-stamped incident reports and experiments. One entry per incident; the entry is the canonical source. Per-node changelog entries link back here.
## Active / recent
| Date | Title | Status |
| ------------ | ------------------------------------------------------------------ | ------------- |
| 2026-06-06 | [Caddyfile truncation incident](2026-06-06-caddyfile-truncation.md) | Resolved — permanent safeguards deployed (site-count guard, auto-stash, auto-backup) |
| 2026-06-06 | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
| 2026-06-03 | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
| 2026-06-01 | [Mac-mini onboarding](2026-06-01-mac-mini-onboarding.md) | Onboarded |
## Resolved (archived)
See [`archive/`](archive/):
| Date | Title |
| ------------ | ------------------------------------------------------------------ |
| 2026-04-21 | [Hubris crash loop — thermal + USB drive](archive/2026-04-21-hubris-crash-loop.md) |
| 2026-05-31 | [Authentik migrated from LXC 124 to the VPS](archive/2026-05-31-authentik-vps-migration.md) |
## Conventions
- File name: `YYYY-MM-DD-<slug>.md`. Use the *first* date if the incident spans multiple days.
- Mandatory sections: Summary, Timeline, Root cause, Mitigations applied, Open questions.
- Update the entry as the situation evolves; never rewrite history. Add new dated sections at the bottom.
- Link back from every node's changelog that's affected.
- Move to `archive/` when the incident is fully resolved and no longer actively referenced.

10
archive/knowledge/log.md Normal file
View File

@@ -0,0 +1,10 @@
# Knowledge — operations log
Append-only record of documentation-maintenance operations on the knowledge wiki (restructures,
source ingests, lint sweeps). One line per operation, newest last. Infrastructure changes belong in
each page's `## Changelog` and the Oikos change ledger, not here.
## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output.
## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md.
## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference.
## [2026-07-06] restructure | Phase 6 consolidation: investigations/ -> knowledge/sources/investigations/; operations/ -> .agents/operations/; HERMES.md -> .agents/; deleted root OIKOS/CAVEMAN/CONTRIBUTING stubs.

View File

@@ -0,0 +1,351 @@
# Current cert sync script + traefik dynamic config
Snapshot of the two artifacts that control public service exposure as of
2026-07-05. Updated 2026-07-05: fixed Jellyfin backend from dead hubris IP
(192.168.8.206) to new strong IP (192.168.8.246).
## hubris-public-cert-sync.sh (PVE host, `/usr/local/bin/`)
```bash
#!/bin/bash
# Mirrors home caddy's LE certs for publicly-exposed hubris.network hostnames
# into the VPS traefik's /letsencrypt volume. Traefik file-watches the volume
# and hot-reloads.
#
# Why: netbird-proxy's HostSNI(*) TCP passthrough intercepts ACME TLS-ALPN-01
# challenges before traefik's allowACMEByPass can respond, so traefik can't
# obtain its own cert. Home caddy uses IONOS DNS-01 (no such conflict);
# we just mirror what it already has.
#
# Runs daily via hubris-public-cert-sync.timer.
set -euo pipefail
CADDY_LXC=121
CADDY_BASE=/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory
VPS_HOST=root@100.122.165.149
VPS_DEST=/var/lib/docker/volumes/opt_netbird_traefik_letsencrypt/_data
# Map: source hostname -> "crt_filename key_filename" on the VPS.
# Stable names so traefik dynamic.yaml doesn't need edits on renewal.
declare -A HOSTS=(
[artifacto.hubris.network]="fullchain.crt privkey.key"
[blog.hubris.network]="blog.fullchain.crt blog.privkey.key"
[trmnl.hubris.network]="trmnl.fullchain.crt trmnl.privkey.key"
[sso.hubris.network]="sso.fullchain.crt sso.privkey.key"
[media.hubris.network]="media.fullchain.crt media.privkey.key"
[paperless.hubris.network]="paperless.fullchain.crt paperless.privkey.key"
)
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
for host in "${!HOSTS[@]}"; do
read -r crt_name key_name <<< "${HOSTS[$host]}"
pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.crt" "$TMP/$crt_name"
pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.key" "$TMP/$key_name"
if ssh -o BatchMode=yes "$VPS_HOST" "test -f $VPS_DEST/$crt_name && diff -q - $VPS_DEST/$crt_name" < "$TMP/$crt_name" >/dev/null 2>&1; then
echo "hubris-public-cert-sync: $host unchanged"
continue
fi
scp -q -o BatchMode=yes "$TMP/$crt_name" "$TMP/$key_name" "$VPS_HOST:$VPS_DEST/"
echo "hubris-public-cert-sync: shipped $host ($(openssl x509 -in "$TMP/$crt_name" -noout -enddate))"
done
```
### Adding a new host
1. Caddy must already have the cert (verify `pct exec 121 -- ls "$CADDY_BASE/$host/"`)
2. Add a line to the HOSTS array: `[new-host.hubris.network]="nickname.fullchain.crt nickname.privkey.key"`
3. `systemctl start hubris-public-cert-sync.service` to sync immediately
4. Verify certs landed: `ssh "$VPS_HOST" "ls -la $VPS_DEST/nickname.*"`
5. Add matching `tls.certificates` entry in traefik dynamic config
---
## traefik-dynamic.yaml (VPS, `/opt/`)
```yaml
tcp:
serversTransports:
pp-v2:
proxyProtocol:
version: 2
tls:
certificates:
- certFile: /letsencrypt/fullchain.crt
keyFile: /letsencrypt/privkey.key
- certFile: /letsencrypt/blog.fullchain.crt
keyFile: /letsencrypt/blog.privkey.key
- certFile: /letsencrypt/trmnl.fullchain.crt
keyFile: /letsencrypt/trmnl.privkey.key
- certFile: /letsencrypt/sso.fullchain.crt
keyFile: /letsencrypt/sso.privkey.key
- certFile: /letsencrypt/media.fullchain.crt
keyFile: /letsencrypt/media.privkey.key
- certFile: /letsencrypt/paperless.fullchain.crt
keyFile: /letsencrypt/paperless.privkey.key
http:
routers:
artifacto-public:
rule: 'Host(`artifacto.hubris.network`) && (PathPrefix(`/p/`) || PathPrefix(`/static/`) || Path(`/healthz`))'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- artifacto-strip-sso
- artifacto-ratelimit
service: artifacto-public
blog-public:
rule: 'Host(`blog.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- blog-ratelimit
service: blog-public
trmnl-public:
rule: 'Host(`trmnl.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- trmnl-ratelimit
service: trmnl-public
matrix-public:
rule: 'Host(`matrix.hubris.network`) && !PathPrefix(`/.well-known/matrix/`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- matrix-ratelimit
service: matrix-public
matrix-wellknown:
rule: 'Host(`matrix.hubris.network`) && (PathPrefix(`/.well-known/matrix/`) || PathPrefix(`/.well-known/acme-challenge/`))'
entryPoints:
- websecure
priority: 20
tls:
certResolver: letsencrypt
service: matrix-wellknown-svc
house-public:
rule: Host(`house.hubris.network`)
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- house-ratelimit
service: house-public
sso-public:
rule: 'Host(`sso.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- sso-ratelimit
service: sso-public
media-public:
rule: 'Host(`media.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- media-ratelimit
service: media-public
paperless-api-public:
rule: 'Host(`paperless.hubris.network`) && PathPrefix(`/api/`)'
entryPoints:
- websecure
priority: 20
tls:
certResolver: letsencrypt
middlewares:
- paperless-ratelimit
service: paperless-public
paperless-public:
rule: 'Host(`paperless.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- authentik-forwardauth
- paperless-ratelimit
service: paperless-public
middlewares:
artifacto-strip-sso:
headers:
customRequestHeaders:
X-Artifacto-Gateway: ""
X-Authentik-Username: ""
X-Authentik-Groups: ""
X-Authentik-Email: ""
X-Authentik-Name: ""
X-Authentik-Uid: ""
X-Authentik-Jwt: ""
X-Authentik-Meta-Jwks: ""
X-Authentik-Meta-Outpost: ""
X-Authentik-Meta-Provider: ""
X-Authentik-Meta-App: ""
X-Authentik-Meta-Version: ""
admin-allowlist:
ipAllowList:
sourceRange:
- "5.61.168.0/24"
artifacto-ratelimit:
rateLimit:
average: 50
period: 1s
burst: 100
blog-ratelimit:
rateLimit:
average: 100
period: 1s
burst: 200
trmnl-ratelimit:
rateLimit:
average: 20
period: 1s
burst: 40
matrix-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
house-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
sso-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
media-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
paperless-ratelimit:
rateLimit:
average: 20
period: 1s
burst: 40
authentik-forwardauth:
forwardAuth:
address: "http://192.168.8.6:9000/outpost.goauthentik.io/auth/traefik"
trustForwardHeader: true
authResponseHeaders:
- X-authentik-username
- X-authentik-groups
- X-authentik-email
- X-authentik-name
- X-authentik-uid
- X-authentik-jwt
- X-authentik-meta-jwks
- X-authentik-meta-outpost
- X-authentik-meta-provider
- X-authentik-meta-app
- X-authentik-meta-version
services:
artifacto-public:
loadBalancer:
servers:
- url: 'http://192.168.8.205:3100'
blog-public:
loadBalancer:
servers:
- url: 'http://192.168.8.205:8080'
trmnl-public:
loadBalancer:
servers:
- url: 'http://192.168.8.211:9851'
matrix-public:
loadBalancer:
servers:
- url: 'http://192.168.8.242:8008'
matrix-wellknown-svc:
loadBalancer:
servers:
- url: 'http://matrix-wellknown:80'
house-public:
loadBalancer:
servers:
- url: 'http://192.168.8.244:3000'
sso-public:
loadBalancer:
servers:
- url: 'http://192.168.8.6:9000'
media-public:
loadBalancer:
servers:
- url: 'http://192.168.8.246:8096'
paperless-public:
loadBalancer:
servers:
- url: 'http://192.168.8.130:8000'
```
### Adding a new service — four blocks needed
1. **Router**`http.routers.<name>-public` with `tls: {}` (not
`certResolver`)
2. **Middleware** — rate limit, one per service
3. **Service**`http.services.<name>-public` with the backend URL
4. **tls.certificates** — add a new `- certFile/keryFile` pair matching the
cert sync HOSTS entry
### Key file naming convention
| Cert name | Host | Convention |
|-----------|------|------------|
| `fullchain.crt` + `privkey.key` | `artifacto.hubris.network` | First service — no prefix |
| `blog.fullchain.crt` + `blog.privkey.key` | `blog.hubris.network` | `{nickname}.fullchain.crt` |
| `trmnl.fullchain.crt` + `trmnl.privkey.key` | `trmnl.hubris.network` | `{nickname}.fullchain.crt` |
| `sso.fullchain.crt` + `sso.privkey.key` | `sso.hubris.network` | `{nickname}.fullchain.crt` |
| `media.fullchain.crt` + `media.privkey.key` | `media.hubris.network` | `{nickname}.fullchain.crt` |
| `paperless.fullchain.crt` + `paperless.privkey.key` | `paperless.hubris.network` | `{nickname}.fullchain.crt` |
### ⚠️ Critical — keep backends in sync after LXC migrations
When moving an LXC between Proxmox nodes, update **both**:
1. **Caddy** (`/etc/caddy/Caddyfile` on LXC 121)
2. **VPS traefik** (`/opt/traefik-dynamic.yaml` — via hubris bridge SSH)
Jellyfin migration from hubris to strong (2026-07-05) was fixed in Caddy
but **missed** in VPS traefik — old IP `192.168.8.206` remained. This caused
Bad Gateway for off-LAN users. Use Python-based editing (see
`references/traefik-config-editing.md`) for accurate surgical fixes.

View File

@@ -0,0 +1,9 @@
# Sources
Immutable evidence the wiki synthesizes from. External reference docs live under `references/`;
incident evidence lives in [`investigations/`](investigations/index.md) (written once at
incident time, then linked from the changelogs of the nodes they implicate).
| Slug | Reference | Summary |
|------|-----------|---------|
| cert-sync-and-traefik-config | [references/cert-sync-and-traefik-config.md](references/cert-sync-and-traefik-config.md) | VPS traefik config and the LAN↔VPS certificate mirror. |

View File

@@ -46,7 +46,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g.
## Open items
- **DHCP → static IP fixed (2026-06-03).** ZimaOS IP drifted from `.195` (Slate AX) → `.103` (Technitium) after the DHCP migration, causing Caddy 502s. Fixed by injecting a static systemd-networkd config and restarting the VM. IP now pinned at `192.168.8.195`. See [changelog](#2026-06-03--static-ip-set-to-195-dhcp-drift-fixed).
- **No Authentik wiring.** [authentik (124)](../containers/124-authentik.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
- **No Authentik wiring.** [authentik (124)](../containers/106-auth-outpost.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
- **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up.
- **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor``screendump`.
@@ -59,7 +59,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g.
## Changelog
### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed
ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix)
@@ -87,4 +87,4 @@ Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs mod
Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`.
### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1)
`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/124-authentik.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.
`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/106-auth-outpost.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.

View File

@@ -7,7 +7,7 @@ Home Assistant OS — the only VM on hubris (HAOS doesn't run cleanly in an LXC,
- **HAOS version:** 16.3 (last verified)
- **IP:** `192.168.8.101`
- **Resources:** 4 GiB RAM, 32 GiB boot disk
- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `192.168.8.101:8123`
- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.101:8123`
## Auth
@@ -18,7 +18,7 @@ Key gotchas:
```
ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1"
```
so OIDC discovery resolves internally to [authentik (124)](124-authentik.md).
so OIDC discovery resolves internally to [authentik (124)](../containers/106-auth-outpost.md).
- Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen).
- YAML config:
- `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created).
@@ -30,8 +30,8 @@ Key gotchas:
HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting.
## Related
- [Authentik (124)](124-authentik.md)
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](../containers/106-auth-outpost.md)
- [Caddy (121)](../containers/121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)

Some files were not shown because too many files have changed in this diff Show More