156 Commits

Author SHA1 Message Date
614c38ea7c docs: plan chat-sessions fixes from real production usage data
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Inspected the live agent_sessions/agent_messages tables on mac-mini and
found silent empty responses, a canned non-English refusal after 22 tool
calls, 70-call fan-out for simple fleet questions, 100KB+ persisted
messages, and no session delete/title hygiene.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 10:09:34 +02:00
22412d2fa3 feat: group tool calls per turn + session entity graph in chat rail
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Two chat UX changes (share Chat.svelte, committed together).

Tool-call grouping (ToolCallGroup.svelte):
- Replaced the one-<details>-per-tool-call list with a single
  collapsible group per assistant turn, headed by tool count + a
  name preview and a status icon (spinning wrench in progress, check
  done, X on error).
- The group auto-collapses the instant its turn finishes streaming, so
  a completed round shows as one compact pill; historical/loaded turns
  start collapsed. The auto-collapse fires once at the streaming→done
  transition, leaving manual toggles alone afterward.

Session graph rail (SessionGraph.svelte) — replaces the old
ContextRail (fleet health / pending approvals / live events), which is
deleted:
- A force-directed graph that starts empty (animated constellation
  empty state) and grows as the conversation references entities.
  Slugs are extracted from message text and tool *arguments* only —
  never bulk result rows, so a single get_health_summary doesn't dump
  all 168 entities — then validated against the backend via fetchGraph
  (cached) with check/execution probe entities excluded. Nodes are
  colored by health; edges appear once both endpoints are present.
- Clicking a node highlights it and its neighbors and opens an inline
  detail panel below: slug/type/state, health + freshness, top
  attributes, in-graph relations (clickable to hop), and a Full detail
  button opening the entity sheet.
- The rail is resizable via a drag handle (260–620px, persisted to
  localStorage). The header's global fleet-health dots are unchanged;
  only the right-rail content was replaced.

Risk: reversible_low (UI-only). The slug extractor is scoped to
focused mentions by design; edges may be slightly incomplete since
only root-fetched entities contribute edges, which is acceptable for a
session overview.

Verification: verified in the browser preview — loading a real session
built a 4-node graph (hubris/caddy/netbird-vps/strong) with the
hubris→caddy relationship edge; clicking hubris showed
"proxmox-host · active · healthy · checked 40s ago" with attributes and
relations; dragging the handle resized 320→440px and persisted; a
34-tool historical turn renders as one collapsed pill that expands on
click. tsc clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 09:48:21 +02:00
5686b9de40 fix: white favicon, sidebar active-state bug, app-wide pointer cursor
Three UI issues reported after the neutral-gray redesign:

- favicon.svg was still filled #58a6ff (the pre-redesign accent blue);
  changed to white to match the sidebar logo mark.
- Sidebar nav items all showed a filled background even when inactive.
  Root cause: sidebar-menu-button.svelte (and -sub-button) rendered
  `data-active="false"` as a literal attribute, but Tailwind's bare
  `data-active:` variant matches attribute *presence*, not value — so
  data-active:bg-sidebar-accent applied to every item regardless of
  state. Fixed by emitting the attribute only when active
  (`isActive || undefined`), a latent bug in the vendored shadcn
  component that read as intentional until flagged.
- Tailwind's preflight resets <button> to cursor: default, so no button
  in the app showed a pointer. Added one base rule restoring
  cursor: pointer for buttons, [role=button], links, summary, and
  select (respecting :disabled / aria-disabled) rather than annotating
  each call site — covers new interactive elements automatically.

Risk: reversible_low (UI-only).

Verification: verified in the browser preview that inactive sidebar
items are transparent (only the current page shows a background),
nav buttons report cursor: pointer via computed styles, and the
favicon renders white in the tab.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-09 09:47:47 +02:00
aa6017e0ca fix: layout overflow regression + adopt true neutral gray theme
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: two issues surfaced after the dashboard-01 shell change
(cbfd09c). (1) Sidebar.Inset previously had an explicit h-svh that
hard-capped the app's height at the viewport; adding variant="inset"
put a margin on that same fixed-height box, pushing it taller than the
viewport with nothing left in the chain to cap it (Sidebar.Provider's
own wrapper only sets min-h-svh — a floor, not a ceiling). Result: the
whole page scrolled as one long document instead of each page's own
content scrolling internally with the header pinned — confirmed via
computed styles, e.g. Entities.svelte's table wrapper measured
scrollHeight 6531px against a 900px viewport, all of it spilling past
body instead of scrolling in its own rounded-border container.
(2) The color palette was GitHub-dark-inspired (blue-tinted grays:
#0d1117 bg, #58a6ff primary/accent) rather than the neutral grays the
shadcn-svelte dashboard-01 reference actually uses.

Change:
- App.svelte: moved the height cap up to Sidebar.Provider itself
  (class="h-svh") instead of Sidebar.Inset, since the cap needs to sit
  above wherever the inset variant's margin gets applied, not on the
  same box as the margin.
- app.css: replaced the core tokens (background/foreground/card/
  popover/primary/secondary/muted/accent/border/input/ring/sidebar-*)
  with shadcn's canonical dark-theme OKLCH values (0-chroma neutral
  grays), pulled directly from huntabyte/shadcn-svelte's own
  docs/src/app.css rather than approximated. --success/--warning
  deliberately kept as real, distinguishable colors — they signal
  actual health state, and desaturating them to match the neutral
  chrome would reintroduce the "can't tell what's actually happening"
  problem this whole project started from (see 279549c). --accent-blue
  now aliases --sidebar-primary (still a real blue) instead of
  --primary, so the couple of spots wanting an interactive "pop" still
  have one while buttons/links/focus rings ride the neutral --primary.

Risk: reversible_low (UI-only).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). go build/vet clean (backend
untouched, sanity check only). Manually verified in the browser
preview at 1400px: document.body.scrollHeight now exactly matches
window.innerHeight on both Overview and the 193-row Entities table
(previously 6531px vs 900px); scrolled the Entities table wrapper to
row ~60 and confirmed the header/filter bar/column headers stay
pinned while only the table body scrolls; confirmed neutral gray
rendering across Overview's stat cards, the event-rate chart, and
Chat's tool-call list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 01:12:47 +02:00
cbfd09c5df feat: redesign toward shadcn-svelte dashboard-01 (inset sidebar, gradient stat cards)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: requested visual alignment with the shadcn-svelte dashboard-01
reference block (shadcn-svelte.com/blocks#dashboard-01) — the app's
sidebar/header shell and Overview stat cards looked plain by comparison.

Change: pulled the actual reference source (app-sidebar.svelte,
nav-main.svelte, site-header.svelte, section-cards.svelte from
huntabyte/shadcn-svelte) rather than approximating from screenshots.

- App.svelte: Sidebar.Root now uses variant="inset" (the floating,
  rounded, shadowed content panel — already fully built into the
  existing Sidebar.Inset component via peer-data selectors, just never
  enabled). Brand mark is now a proper Sidebar.MenuButton matching the
  reference's padding/hover treatment; "New chat" uses the reference's
  primary-colored button styling. Header matches the reference exactly:
  h-(--header-height) (48px, was 44px), vertical separator after the
  sidebar trigger, right-aligned actions group.
- Overview.svelte: stat cards rebuilt to match section-cards.svelte —
  gradient background, Card.Action badge, Card.Footer with a bold line
  + muted context line, tabular-nums, responsive @container grid
  (1/2/4 columns). Deliberately did NOT copy the reference's fake
  trend-percentage badges (Oikos doesn't track historical trends, and
  this project's whole thrust has been eliminating dishonest UI state —
  see 279549c). Badges instead reflect real current-state signals
  (healthy/degraded/down, clear/needs-review) computed from the actual
  dashboard summary.
- EntityDetailContent.svelte + Entities/Signals/Ops/Events/Agent/
  Audit/Knowledge pages: normalized root padding to p-4 md:p-6 (was a
  flat p-6) to match the reference's responsive py-4 md:py-6 convention.

Risk: reversible_low (UI-only, no data or behavior changes).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings, same as prior commits). go build/vet
clean (backend untouched, sanity check only). Manually verified in the
browser preview at 1400px: inset sidebar's margin/rounded-corner/shadow
classes confirmed applied via computed styles; Overview cards render
with real live numbers from the now-fixed dashboard summary endpoint;
Signals/Ops pages confirmed visually consistent with the new spacing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:54:51 +02:00
851b5dce67 feat: master-detail entity sheet, freshness in Entities table, live logo
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Problem: the web UI felt dead and hard to navigate — the Entities table
had no health/freshness signal (just a meaningless row-mutation
timestamp), no way to see what was actually monitoring an entity,
sessions couldn't be reopened, and every drill-down was a full page
navigation that lost the list.

Change:
- Entities table: Updated column replaced with a health dot + relative
  "checked Xm ago", sourced from the backend's new health/last_check_at
  fields.
- New EntityDetailContent.svelte extracted from EntityDetail.svelte and
  shared between the full #/entity/:slug page and a new EntitySheet.svelte
  opened from the Entities table (master-detail, row click opens a panel
  instead of navigating away). Adds a Monitoring card listing the
  entity's check_defs (kind, interval, enabled/disabled with
  click-to-toggle via the existing PatchCheck endpoint) and renders
  attributes as key/value pairs instead of raw JSON.
- Sessions: fixed a bug where clicking a session loaded it into the
  chat store but never navigated to the chat page, so nothing appeared
  to happen. Added a SessionRail inside Chat so switching sessions
  never leaves the chat surface.
- Fixed the local dev proxy (vite.config.ts): production Caddy strips
  the /agent prefix before forwarding to nomos; the dev proxy didn't,
  so every session/chat fetch 404'd locally while working in prod.
- Found and fixed a real latent bug while testing the session fix:
  chat.ts's loadSessionMessages passed the persisted tool_calls array
  straight through, but nomos stores the tool_use and tool_result as
  two entries sharing one id. Chat.svelte's keyed {#each tool (tool.id)}
  throws on the duplicate key, which silently blanked the entire
  message list — invisible until sessions were actually clickable.
  Fixed by merging tool_calls by id before rendering, matching the
  shape the live-streaming path already produces.
- UI polish: sidebar logo is now just the omicron mark in white (was
  icon+text in the accent color); removed the sheet overlay's
  backdrop-blur (distracting per feedback); the Attributes/Relations/
  Signals grids used viewport-based lg:/3xl: breakpoints, which forced
  multi-column layouts based on browser width regardless of the sheet's
  actual rendered width — switched to Tailwind v4 container queries
  (@lg:/@2xl:/@3xl:) so layout responds to the real available width in
  both the full page and the narrower sheet.

Risk: reversible_low (UI-only; no destructive operations; the tool_calls
merge and dev-proxy fix are corrections to broken paths, not behavior
changes to working ones).

Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). Manually verified in the browser
preview against the live dev API: Entities table health column renders
correctly; clicking a row opens the EntitySheet with a populated
Monitoring card (16 checks for host:hubris, verified via psql that
check_defs.target_id links them correctly); clicking a session now
loads its full transcript inline (was blank before the tool_calls fix);
sheet has no blur and lays out single/multi-column correctly at the
sheet's actual width.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:26:37 +02:00
279549c8c9 fix: scheduler wrote health/metrics/events to probe entities, not targets
Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.

Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
  cd.TargetID (falling back to the check's own id if unset) and write
  status/metrics/events there. Signals stay keyed by the check entity,
  unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
  observation is older than 3x its fastest enabled check's interval
  (floor 5m) is marked 'stale' and emits health.stale, so a stalled
  scheduler or disabled check_def can no longer look like current data
  forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
  rows so dashboard/fleet-health rollups stop double-counting probes as
  monitored entities. Historical metric_samples on check entities are
  left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
  'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
  exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
  the conversation instead of dropping them (previously only final text
  was replayed, forcing the agent to re-derive fleet state every turn),
  and inject a compact live fleet-health snapshot into the system prompt
  each turn so it starts oriented instead of spending an iteration on
  discovery.

Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.

Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 00:26:04 +02:00
a39e67b6e9 adr: convert all diagrams to Mermaid (sequenceDiagram, stateDiagram-v2, flowchart, erDiagram, graph)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
0013-signal-triggers.md:
- Thermals query: sequenceDiagram (Nomos→API→Scheduler→Hubris→TimescaleDB)
- Script deployment: sequenceDiagram
- Signal lifecycle: stateDiagram-v2
- DB data flow: flowchart

0014-entity-model.md:
- Entity type hierarchy: graph (56 types, 3 layers, 7 domains)
- Machine onboarding: sequenceDiagram
- OODA loop (5 phases): flowchart with color-coded subgraphs
- Infrastructure topology: graph
- Network relationships: graph
- Service dependencies: graph
- Cognition OODA edges: graph
- Governance: graph
- Infrastructure lifecycle: stateDiagram-v2
- Signal lifecycle: stateDiagram-v2
- Execution lifecycle: stateDiagram-v2
- Approval lifecycle: stateDiagram-v2
- DB physical schema: erDiagram
- Thermals query trace: sequenceDiagram
2026-07-08 22:38:19 +02:00
551497e0b3 adr: move to docs/adr/, renumber 0013 + 0014, update README index
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 22:18:58 +02:00
4a5e68bafe adr: full entity model — types, relationships, state machines, OODA loop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Covers:
- 56 entity types with full hierarchy (abstract/concrete, domain, layer)
- 34 relationship types with cardinality and OODA phase mapping
- 88 concrete entity instances with key attributes
- Lifecycle state machines (infrastructure, signal, execution, approval,
  pattern, skill) with which preconditions are code-real vs schema-only
- Sequence diagrams: machine onboarding, OODA loop, thermals query
- What's fully implemented vs schema-defined-but-not-wired
- Database physical schema with FK relationships
- Policy: risk classes, approval rules, per-entity overrides, autonomy
- Blast radius via recursive CTE over depends-on/hosts/routes-to edges
2026-07-08 22:15:11 +02:00
ef00e5b8e4 fix: disk_usage_check.sh handles inode '-' (vfat/efi), checkdefaults sets target_id
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- disk_usage_check.sh: sed 's/-/0/' for filesystems without inodes
- checkdefaults.Ensure: includes target_id in check_defs INSERT so
  signals get proper target slug instead of null
2026-07-08 21:58:44 +02:00
a512d40669 onboarding: auto-create default checks when entity is created
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Three insertion points:
- CreateEntity (POST /api/v1/entities)
- EnrollClient (POST /api/v1/clients/enroll)
- seed.go (seed ingest at deploy time)

Shared logic in internal/checkdefaults — resolves host IP from
lan_ip > mesh.netbird.ip > mesh_ip, SSH user/port from attributes.

Default checks per entity type:
- proxmox-host/standalone-server: ping + cpu + memory + load + disk + updates
- workstation: ping + cpu + memory + load
- lxc: cpu + memory + load + disk
- vm: ping
- service: process_check.sh

All idempotent (ON CONFLICT DO NOTHING). New machines now get
monitoring automatically — no manual curl calls needed.
2026-07-08 21:53:50 +02:00
b8bb29464b checks: deploy scripts + define checks for strong and netbird-vps
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Coverage:
- host:hubris (192.168.8.77)   ✓ cpu, memory, load, disk, ping, cert-expiry
- host:strong (192.168.178.181) ✓ cpu, memory, load, disk, ping
- host:netbird-vps (82.165.190.79) ✓ cpu, memory, load, disk, ping
- ws:mac-mini                  ~ local disk/ping only (no SSH key on macOS host)
- ws:republic-laptop           ✗ offline / not reachable

Move architecture doc to adr/signal-triggers.md
2026-07-08 21:45:09 +02:00
81acadec1d docs: signal trigger architecture sequence diagram + full explanation
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Add docs/signal-triggers.md covering:
- End-to-end sequence diagram (Nomos → API → Scheduler → target host)
- Two paths: autonomous collection (scheduler) + query (MCP)
- All 6 check kinds and 17 available scripts
- Script deployment flow via sync timer
- Signal lifecycle, threshold evaluation, data flow through DB tables
- Prerequisites for SSH checks in Docker
2026-07-08 21:37:31 +02:00
291b45565b fix: metric samples timestamp + ssh-script port/user handling
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- InsertMetricSample now includes ts=now() (TimescaleDB hypertable requires it)
- ssh-script: pass host and port separately (ssh uses -p flag, not host:port)
- ssh-script: use OIKOS_SSH_USER from config/env, default root
- Add -o LogLevel=ERROR to suppress SSH warnings polluting JSON output
- Use Output() (stdout only) instead of CombinedOutput()
- Set OIKOS_SSH_USER=root in scheduler docker-compose service
2026-07-08 21:36:05 +02:00
d45f2326b6 fix: CreateCheck uses type "check" not "check_def"
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
FK violation — entity_types table has name "check" but
phase3.go:320 was inserting type "check_def" causing:
entities_type_fkey (SQLSTATE 23503)
2026-07-08 21:21:44 +02:00
4bf811a383 docker: alpine base with openssh-client, mount SSH key + NET_RAW for scheduler
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Switch Dockerfile from distroless/static to alpine:3.21
- Install openssh-client-default in runtime image
- Mount SSH key in scheduler service (docker-compose)
- Add NET_RAW capability for ping checks
- Wire OIKOS_SSH_KEY_PATH and OIKOS_SSH_USER env vars in scheduler
- sshExec uses configured key path with StrictHostKeyChecking=no
2026-07-08 21:15:02 +02:00
35feada286 scheduler: add ping + ssh-script check kinds, metrics refactor, 17 host check scripts
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Refactor executeCheck to return checkResult struct with metrics map
- Add ping check kind (ICMP reachability via system ping, macOS+Linux)
- Add ssh-script check kind (remote host exec via SSH, allowlisted scripts)
- Add threshold evaluation (warn/crit per metric from check config JSONB)
- Add inode tracking to disk check
- All 4 existing checks now return structured metrics
- 17 check scripts: cpu, memory, load, swap, disk_usage, disk_smart,
  updates, zfs, process, uptime, oom, journal, time, fd, docker_health,
  caddy_error_rate, backup_freshness
- Auto-deploy via tools/setup-checks.sh -> checks/install.sh on git pull
- Add ping to OpenAPI CheckKind enum and generated Go types
2026-07-08 21:05:53 +02:00
cca2ae4621 fix: properly convert icns to PNG for favicon data URI
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 18:06:35 +02:00
d13f6991b1 fix: use inline base64 favicon to work in both dev and production
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 18:02:11 +02:00
2de3602ebc feat: replace inline favicon with extracted app icon from DMG
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 17:34:58 +02:00
2908b0a377 feat(ui): M4 — agent activity, knowledge search, audit, correlation grouping
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Add three new pages completing the control-room web UI:
- Agent activity: polls /agent-activity every 5s, filterable by type/agent
- Knowledge search: FTS over /knowledge/search with snippet + entity links
- Audit trail: browseable audit log with actor/action/entity filters

Enhanced live events page with correlation-id clustering (Groups toggle).
Added fetchAgentActivity/searchKnowledge/fetchAudit to the API client.
11 nav items now cover all planned control-room views.
2026-07-08 17:02:09 +02:00
cff05c0768 fix(nomos): auto-reconnect stale MCP session; enlarge SSE scan buffer
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
After an api (MCP server) restart, nomos held a dead session id and every
tool call failed with "unexpected end of JSON input" until nomos was manually
restarted — which happens on every deploy. The MCP client now detects a
rejected session (4xx or empty body) and transparently re-initializes and
retries once. Also raise the SSE scanner buffer to 4MB so large tool results
don't exceed the 64KB default token limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:55:10 +02:00
5d02126e16 fix(ui): serve embedded SPA via ServeContent to avoid index.html redirect loop
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
http.FileServer canonicalizes /index.html -> "./", which for /ui/ produced a
301 redirect loop and made the control room unreachable. Serve embedded files
directly with http.ServeContent instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:28:54 +02:00
e8e230b4a5 nomos+web: streaming, provider routing, event gap-fill, embedded UI; fix approval FK & session context
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Agent (cmd/nomos):
- Stream LLM tokens via NewStreaming; emit text_delta then final text.
- OpenRouter provider routing: data_collection=deny (ZDR) + require_parameters;
  NOMOS_PROVIDER_SORT opt-in; Exacto via model suffix.
- Multi-turn: reload session history into context; UI passes session id.
- Fix agent_activity logging (agent_id/session_id) and mcpClient data race.

Events (live control-room feed):
- approval.created (mcp), approval.decided (api), execution.completed/failed
  (approved-action path), signal.raised/resolved + health.changed (scheduler,
  transition-gated).

Fixes:
- createApproval FK violation (reuse execution entity) — the agent's only
  write path; log the previously-swallowed errors.

Web UI:
- Embed web/dist via //go:embed (single binary); Dockerfile builds SPA into
  the Go stage; committed .gitkeep placeholder keeps backend-only builds green.
- Caddy: Authentik-gated /agent/* -> nomos so the UI reaches the agent
  same-origin in production.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 15:22:27 +02:00
2b3aa248b1 N0: rename Hermes → Nomos (standalone commit)
Problem: "Hermes" collides with Nous Researchs unrelated product;
  unclear identity for the resident agent.

  Change: Rename the live service identity across 39 files:
  - cmd/hermes/ → cmd/nomos/ (binary, env vars NOMOS_*)
  - internal/config/ server.go (NomosAgentSlug, nomosAgentID)
  - compose/hermes/ → compose/nomos/ (Dockerfile, service name)
  - hermes/ → nomos/ (SOUL.md, config.yaml, skills/)
  - .agents/HERMES.md → NOMOS.md (persona)
  - tools/setup-hermes-soul.sh → setup-nomos-soul.sh
  - seeds/inventory.yaml (agent:hermes → agent:nomos)
  - migrations/014_rename_agent_hermes_to_nomos.up.sql
  - Caddy vhost hermes.hubris.network → nomos.hubris.network
  - All referencing docs, scripts, ADR notes

  History preserved: archive/, plans/done/, ADRs not rewritten.
  Matrix @hermes notifier account and Legacy bin/hermes on LXC 129
  intentionally untouched (out of scope).

  Risk: N0 is identity-only rename; zero behavioral changes.
  Verification: go build ./... passes; docker compose --profile full
  resolves nomos service; grep -ri hermes (excluding archive/plans)
  returns only intentional refs (LLM model name, Matrix user).
2026-07-08 14:14:56 +02:00
9d9cbb63c4 plans: rename resident agent Hermes -> Nomos, add implementable rename phase N0
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Nomos (from oikonomos, steward of the oikos) avoids the name collision
with Nous Research's Hermes Agent. N0 enumerates the full rename scope:
cmd/, hermes/ dir, env vars, config fields, compose service, Caddy vhost,
identity-preserving DB slug migration + seed update, persona docs.
History, the Matrix bot user, and legacy bin/hermes stay untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:47:24 +02:00
ae3b150dcc plans: hermes agent uses OpenRouter (default deepseek-v4-flash), document off-the-shelf rejection
Swap anthropic-sdk-go for openai-go against the OpenRouter API; default
model deepseek/deepseek-v4-flash with Exacto routing and ZDR provider
pinning. Record why Nous Hermes Agent (and hosted MCP connectors) were
rejected for the resident role.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:36:55 +02:00
750f0e088d plans: add Hermes resident agent plan, make agent chat the control-room home view
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Hermes becomes an LLM-backed agent loop (hand-rolled tool loop over the
existing mcpClient, not the public MCP connector), with Postgres-backed
sessions, SSE chat streaming, and Authentik-gated /agent routing. The
control-room plan is amended to make the chat the main entry point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 13:30:42 +02:00
84068cc40b plans: add oikos gaps review and control-room webui plans
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Documents a full-project review (confirmed bugs, security gaps, user- and
agent-perspective gaps) and a realtime control-room web UI plan, per prior
codebase exploration on this branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 13:16:24 +02:00
5d15265f65 fix: queryRows returns [] not null for empty result sets
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:50:13 +02:00
3ea43adcfd fix: repair 5 MCP analysis tools with SQL errors + seed entity_status rows
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- get_event_timeline: event_type/actor/message → type/source/data::text
- get_blast_radius: add JOIN entities for slug column
- get_state_snapshot: remove nonexistent disk_usage_pct, drift_count
- get_change_history: timestamp/actor_label/details → ts/actor_id::text/detail
- seed: upsert entity_status rows during inventory ingest (was empty, causing INNER JOIN on get_health_summary to return nothing)
2026-07-08 12:27:25 +02:00
4724d6f297 fix: seed now merges attributes instead of overwriting on upsert, add enrollment attrs to seed
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:02:46 +02:00
f07668c1c3 caddy: update Caddyfile.oikos snippet to match deployed config with enrollment bypass
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 12:00:20 +02:00
7e0566d2cd fix: widen 'about' relationship to entity→entity many-to-many for investigation edges
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:54:10 +02:00
2e8ef75436 onboard: mac-mini as workstation — new age keypair, update inventory seeds
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:50:47 +02:00
ca16b75b90 fix: rename Homelab-Docs → oikos across all active files; add public enrollment route
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-08 11:46:42 +02:00
e148c7a981 move 5 completed plans to plans/done/
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Consolidate Oikos on mac-mini (2026-07-06)
- Client lifecycle in Go (2026-07-07)
- Comprehensive audit & next steps (2026-07-07)
- DB as source of truth (2026-07-07)
- MCP tool completion (2026-07-07)

Paths fixed in index.md to reflect planes/done/ locations.
Active plans remaining: Prometheus LXC (Planned), implementation audit (Active).
2026-07-08 11:30:54 +02:00
cefeba72b0 delete apps/105 Gitea webhooks (ids 10, 11, 14)
Removed via Gitea API using Infisical-stored token:
- Hook 10: 192.168.8.205:9811 (homelab-mcp-deploy)
- Hook 11: 192.168.8.205:9821 (secrets-issuance-deploy)
- Hook 14: 192.168.8.205:9831 (oikos-console-deploy) — found during cleanup

All three pointed to apps/105 (192.168.8.205). No remaining webhooks
fire to the old Python stack. Zero hooks remain on dtoro/oikos.

Cutover checklist updated. Only item remaining: archive apps/105 LXC.
2026-07-08 11:29:42 +02:00
e131215a25 add Gitea webhook cleanup script for apps/105
scripts/cleanup-apps105-webhooks.sh: lists webhooks on dtoro/oikos,
filters for apps/105 deployment endpoints, prompts for confirmation,
deletes matching hooks via Gitea API.

Usage: GITEA_TOKEN='...' ./scripts/cleanup-apps105-webhooks.sh

Also updated cutover-checklist.md with the script reference and LXC
archive command.
2026-07-08 11:27:23 +02:00
7660e5681c complete consolidation plan — scripts, watchdog, rollback runbook
Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items:

1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy
   proxy). Only pages when BOTH paths fail. Partial failure logged but not
   paged (distinguishes stack problem from mesh/Caddy issue).

2. deploy.sh — pre-deploy pg_dump before each deploy saves to
   /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a
   guaranteed recovery point.

3. docs/operations/rollback.md — runbook documenting automated rollback,
   manual recovery, decision tree, backup schedule, and rehearsal log.

Two operational items remain (require operator on Proxmox/Gitea):
- Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs
- Archive apps/105 LXC (pct stop 105 + archive)

All active config (seeds, compose, scripts) is already clean of apps/105 refs.
Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).
2026-07-08 11:25:43 +02:00
28ab9b8088 final verification — client lifecycle plan: 12/12 items confirmed
Added 12-point verification table with file:line evidence for every item.
Noted two minor deviations: inline SQL (not sqlc) and synthetic Infisical
IDs (real Infisical pending consolidation plan). Precondition checks
enumerated with hard/soft classification.
2026-07-08 11:20:10 +02:00
fcd9f23ee1 transition precondition enforcement + thin-client context poller
Plan #3 at 100%. Last three items resolved:

1. Transition precondition enforcement (Phase 5):
   - no-inbound-edges: blocks destroy when relationships exist
   - backups-verified, secrets-revoked, ingress-dns-removed: checks attrs
   - age-key-enrolled-if-needed, mesh-joined-if-needed: workstation checks
   - health-check-answering: verifies entity_status health
   - doc-page-complete: requires at least one linked document
   - Soft preconditions (inventory-entry, cancelled-note, etc.): operator
     confirmed via transition request itself
   - Parses {requires: [check-name]} from lifecycle_defs.transitions JSONB

2. bootstrap.sh: already thin-client (fetches only agent files, no git clone,
   calls POST /clients/enroll, embeds context poller)

3. tools/context-poller.sh: standalone version — polls GET /clients/{slug}/context,
   applies file/tool/sops deltas, re-runs changed setup scripts
2026-07-08 11:16:04 +02:00
efa66c7321 close client lifecycle plan — API was already fully implemented
Plan #3 at 95%. Initial audit was incorrect — the entire API surface was
already implemented and tested:

- POST /clients/enroll — age key pair generation, mesh IP validation,
  attrs update, state → provisioning (impl.go:1091)
- GET /clients/{slug}/context — context_version delta with file/tool/sops
  change tracking (impl.go:1195)
- GET /clients/{slug}/secrets — scoped secret key listing (impl.go:1245)
- POST /entities/provision — constraint validation, provisioning_steps
  tracking, relationship edges, audit trail (impl.go:1271)
- GET /entities/{slug}/provision/status — step-by-step progress (impl.go:1380)
- PATCH /entities/{id} — lifecycle validation against lifecycle_defs,
  409 on illegal transitions (impl.go:933)
- client_lifecycle_test.go: 324 lines, full e2e:
  planned→enroll→provisioning→active→migrating→deprecated→failed
  + provision + relations + blast radius + rejection tests

Remaining (follow-up): bootstrap.sh + context-poller.sh thin-client scripts.
2026-07-08 11:12:45 +02:00
43aaf2a318 complete comprehensive audit — all cleanup items resolved
Plan #4 done. Audit inventory verified against codebase:

- 9 superseded oikos/*.py files deleted (only gen-topology.py remains)
- bin/homelab deleted, bin/oikos deleted, oikos/cards/ deleted
- .hermes/plans/ already archived to archive/hermes-plans/ (all 7 files)
- TRMNL plan already in Done table
- seanime + romm documented in seeds/knowledge.yaml (DB-native, no wiki needed)
- Traefik references valid (VPS still runs traefik for public termination)
- ADR-0011 exists (client lifecycle); consolidation plan is Go rewrite record
- Prometheus plan updated: Python refs replaced with Go scheduler, check_defs,
  MCP request_execution; LXC 131 identified as teddycloud

Remaining items (cutover, Infisical, watchdog, rollback, apps/105) belong to
consolidation plan (#1). 4 of 6 plans now Done.
2026-07-08 11:09:37 +02:00
a3ebd12e90 complete DB as source of truth — FTS knowledge surface
Plan #5 done. Wiki already archived to archive/knowledge/. seeds/knowledge.yaml
has 24 docs + 6 investigations + 3 runbooks.

- MCP search_knowledge: upgraded from ILIKE to PostgreSQL ts_rank/ts_headline
- MCP get_entity_knowledge: new tool, walks relationship edges to return
  all docs/investigations/runbooks linked to an entity
- HTTP endpoints (SearchKnowledge, GetEntityKnowledge) already used full FTS
- Plan index + audit cross-reference updated
2026-07-08 11:06:12 +02:00
7c6cffb5f5 complete MCP tool surface — Matrix approval webhook loop + token verification
Plan #6 (MCP Tool Completion / bin/homelab Migration) done.

- Approval records created for gated request_execution actions
- Notifier sends Matrix messages with HMAC approval tokens
- Stores matrix_event_id, polls /relations/{id}/m.annotation for /
- Reaction detection triggers DecideApproval API call
- Token verification added to DecideApproval endpoint
- Migration 013: matrix_event_id + alert_sent_at on approvals
- AGENTS.md: 21-tool surface documented, stale homelab CLI refs removed
- Plan index updated, audit cross-reference refreshed
2026-07-08 11:02:06 +02:00
5b22f2367b test: e2e client lifecycle + ADRs with sequence diagrams
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- client_lifecycle_test.go: full end-to-end integration test
  planned → provisioning (enroll) → active → migrating → active →
  deprecated → failed. Validates age keypair generation, attrs,
  context/secrets endpoints, invalid transition blocking, compute
  entity provisioning with relationship edges and status tracking.
  Also tests enrollment rejection for invalid states and duplicate
  slug rejection for provisioning.

- adr/0011-client-lifecycle-flows.md: workstation self-enrollment,
  compute entity provisioning, deprecation/destruction flows with
  Mermaid sequence diagrams. Full lifecycle state diagram. Transition
  check enforcement documentation.

- adr/0012-hermes-oikos-interactions.md: Hermes ↔ Oikos interaction
  flow through OODA loop phases. Thin client bootstrap. Internal
  component interactions (scheduler, actuator, notifier). Complete
  30-tool ownership matrix.

- Fix: migration 012 FK reference (executions.id → executions.entity_id)
- Fix: provision handler null attributes JSONB
- Fix: provisioning steps use entity_id for execution FK

All 3 integration tests pass, go vet clean.
2026-07-08 00:56:16 +02:00
84ecb6b895 feat: remaining phases — actuator provisioning, transition checks, cleanup
Phase 2: Actuator provisioning
- ProvisionLXC: pct create, start, package install, mounts, health check
- ProvisionVM: qm create, status check via SSH
- sshExecSimple helper for lightweight SSH command execution
- resolveHost helper for entity attribute lookups

Phase 5: Transition check enforcement
- TransitionChecks map with 8 named checks:
  age-key-enrolled, mesh-joined, health-check-answering,
  no-inbound-edges, secrets-revoked, backups-verified,
  ingress-dns-removed, doc-page-complete
- All checks accept pool + entity attrs for validation at transition time

Phase 6: Cleanup
- tools/setup-caveman.sh — npm install + wrapper + templates
- tools/setup-hermes-soul.sh — SOUL.md provisioning
- CLIENTS.md updated for thin client model (no git clone, API-based)
- Old git-sync references replaced with context poller

All tests pass, go vet clean.
2026-07-08 00:40:53 +02:00
cfce35bee0 feat: rewrite bootstrap.sh for thin client model + Oikos API enrollment
Thin client model (rev 2): no git clone, no sync timer.

Changes:
- Fetches only CLIENTS.md, AGENTS.md, OIKOS.md, tools/ from raw Gitea URL
- Enrolls via POST /api/v1/clients/enroll (replaces dead Python
  secrets-issuance service)
- Receives age keypair + Infisical identity from Oikos API
- Context poller replaces 5-minute git pull (launchd/systemd timer hits
  GET /api/v1/clients/{slug}/context?since=)
- Removed --no-secrets, --no-mesh flags (degraded modes TBD)
- Removed dead bin/homelab symlink
- Removed Gitea credential configuration (no git clone = no git auth)
- Kept --with-mcp and --with-hermes flags for optional tooling
- auto-setup scripts run from fetched tools/ directory
2026-07-08 00:29:06 +02:00
a786107cc7 feat: add client introspection MCP tools — whoami, explain, preflight, etc.
Phase 3 from the client-lifecycle plan. Six new MCP tools registered:

- whoami(hostname) — entity record, health, mesh IP, age pubkey
- explain(service_slug) — compact context card with type, state, health
- preflight(service_slug, action) — risk class + approval requirement
- get_change_history(entity_slug, limit) — audit log entries
- get_state_snapshot() — fleet health, disk, drift count
- list_my_secrets(caller_pubkey?) — secrets accessible by age key

All tools use existing queryRows/queryEntity helpers with SQL queries.
All tests pass.
2026-07-08 00:27:26 +02:00
44e1e421e1 feat: client enrollment API and compute entity provisioning
Phase 1 implementation from the client-lifecycle plan.

- Migration 012: provisioning_steps table, context_version, context_files,
  enrolled_at column, slug+type index for machine entities
- API endpoints (openapi.yaml + generated code):
  POST /clients/enroll — age key issuance, Infisical identity, state transition
  GET /clients/{slug}/context — agent file delta polling (replaces git pull)
  GET /clients/{slug}/secrets — scoped secret listing
  POST /entities/provision — compute entity creation with constraint validation
  GET /entities/{slug}/provision/status — step-by-step provisioning progress
- Handlers in impl.go: enrollment with state validation and age key generation,
  provisioning with execution tracking and relationship creation,
  context endpoint with since-based delta queries
- Server struct extended with secretsBackend interface for key storage
- All tests pass, build clean
2026-07-08 00:24:32 +02:00
8653f3036d plan: client bootstrap fetches CLIENTS.md as primary orientation
CLIENTS.md is the entry point for a machine joining the homelab.
AGENTS.md is the AI agent persona layer on top. Both fetched, but
CLIENTS.md comes first.
2026-07-07 23:59:22 +02:00
79dc87d584 plan: rev 2 — add thin-client API distribution and compute entity provisioning
Two onboarding paths share the same lifecycle state machine:

1. Workstation self-enrollment: curl bootstrap.sh | bash → API enroll
   → thin client (no git clone, no sync timer). Context poller replaces
   5-minute pull. Only AGENTS.md, OIKOS.md, tools/ fetched to disk.

2. Compute entity provisioning: POST /entities/provision → Oikos
   actuator creates LXC/VM/container on Proxmox host. Validates VMID,
   IP, capacity, template. Creates relationship edges (hosts, provides,
   mounts, depends-on) atomically. No self-enrollment, no age key.

Adds: context endpoint for agent file deltas, provisioning_steps table,
actuator provision methods, type-specific transition checks, full
verification matrix covering both paths.
2026-07-07 23:56:07 +02:00
638e313c66 docs: add client lifecycle plan, cleanup stale files, document repo for 3 audiences
Problem: Repo had no developer guide, no client onboarding doc, no agent
dev instructions. Stale files (675KB SQL dump, one-off convert script,
legacy MCP builder) cluttered the tree. Client enrollment was a documented
intention with no Go implementation.

Changes:
- New docs: CONTRIBUTING.md (dev setup), CLIENTS.md (client onboarding),
  .agents/dev/CONTRIBUTING.md (agent codebase map)
- New plan: plans/2026-07-07-client-lifecycle-in-go.md — full client
  lifecycle (planned→provisioning→active→deprecated→destroyed) in Go,
  replacing archived Python secrets-issuance, adding client API endpoints
  and 6 missing MCP tools
- Cleanup: deleted archive/convert-wiki.py (one-off), archive/mcp/
  build_host_files.py (legacy), backups/pre-deploy-7f7d039.sql (local)
- Fixes: plans/index.md duplicate row removed, README.md repo layout
  updated for current state, AGENTS.md header points to new guides

Risk: low. Docs only + stale file deletion. No code changes. New plan is
proposal, not implementation.
Verification: git diff reviewed, all changes are prose/docs/plans.
2026-07-07 23:45:32 +02:00
85b541a1cc cutover: rollback drill complete — backup, rollback to 7ac2521, re-deploy to 7f7d039, verified
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
2026-07-07 23:16:22 +02:00
7f7d039e1b phase 3-4: get_execution_status tool, approval creation, delete bin/ + homelab.go, update AGENTS.md 2026-07-07 23:13:13 +02:00
7ac2521a22 mcp: expand request_execution — restart, systemctl, pct_exec, apt_upgrade with inline SSH 2026-07-07 23:07:02 +02:00
692800cf09 mcp: add 5 operational tools — tail_log, get_service_status, ping_service, list_lxcs, get_lxc_state 2026-07-07 23:01:21 +02:00
bbd560bf19 plans: flip bin/ migration to MCP tool completion — agent is the operator interface 2026-07-07 22:39:45 +02:00
e93625b971 plans: add bin/homelab → Go migration plan, fix Active/Done plan listings 2026-07-07 22:27:17 +02:00
bf250fc6a9 cleanup: archive stale secrets/ + secrets-issuance/, add Infisical bootstrap script, update SOPS paths to archive/ 2026-07-07 22:20:58 +02:00
b35825c50b infisical: SOPS migration complete (11 secrets), secrets-issuance decommissioned 2026-07-07 22:16:16 +02:00
806795c63d infisical: fully bootstrapped — machine identity, env vars, profiles wired 2026-07-07 22:01:43 +02:00
24b8772d72 cutover: Infisical bootstrapped — running, admin created, awaiting browser setup 2026-07-07 21:46:57 +02:00
129a6710cc cutover: watchdog fully tested — Matrix alert verified, Matrix env vars added to compose 2026-07-07 21:27:21 +02:00
ae23311f17 cutover: update checklist — watchdog tested, Infisical deferred (v0.162.0 bug) 2026-07-07 21:20:00 +02:00
2de23325c5 seeds: populate at_glance for seanime (LXC 133) 2026-07-07 21:09:58 +02:00
6b51c83b82 plans: mark TRMNL and wiki-hq adoption as done, move to plans/done/ 2026-07-07 21:06:58 +02:00
5e3b946ded cleanup: fix all stale references across .agents/ docs
- Rewrite AGENTS.md: DB as source of truth, MCP knowledge tools, archive refs
- Fix OIKOS.md: seeds/ paths, remove Python-era notes, update deployment status
- Fix commands.md, agent-enrollment.md: archive/knowledge/ links
- Fix all SKILL.md files: remove hosts/*.yaml refs, point to inventory.yaml
- Fix HERMES.md, schema.md, page-templates.md, llm-wiki.md: update paths
- Fix bootstrap.sh: identity check reads inventory.yaml
- Fix README.md, cutover-checklist.md: stale wiki references
- Move convert-wiki.py to archive/ (one-shot done)
2026-07-07 21:00:39 +02:00
5009a335bb remove hosts/ directory — single source of truth is inventory.yaml
- Delete cmd/oikos/build_hosts.go (generator no longer needed)
- Remove build-hosts subcommand from main.go
- Fix oikos homelab whoami: read from inventory.yaml instead of hosts/
- Update bootstrap.sh: identity check uses inventory.yaml
- Remove all 27 generated hosts/*.yaml files
- Update AGENTS.md and OIKOS.md to reference inventory.yaml only
2026-07-07 20:48:51 +02:00
f04e0dc0d4 fix: PG array format for tags, entity slug prefixes, archive path handling
- knowledge.go: scan tags as []string from pgx (not JSON)
- seed.go: convert tags to PG array format, fix runbook applies_to_type
- convert-wiki.py: fix entity slug prefixes to match inventory.yaml
  (host: not proxmox-host:, ws: not workstation:, service:homelab-mcp with hyphen)
- convert-wiki.py: read from archive/knowledge/ since wiki was archived
2026-07-07 20:37:17 +02:00
6b75f7302d db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion
- Migrations 010 (content_hash) + 011 (search tsvector column)
- new: internal/knowledge/seed.go — knowledge seed ingest engine
- new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge
- wire knowledge ingest into oikos seed pipeline
- convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml
- archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/
- delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py
- remove empty knowledge/ directory tree
2026-07-07 20:22:30 +02:00
b2bfa26f64 plans: comprehensive audit + DB-as-truth architecture 2026-07-07 20:09:32 +02:00
4b6c02a88e cleanup: remove deprecated Python artifacts + plan remaining items
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Deleted (11 files, 5 directories — not imported by any operational code):
  mcp/deploy/ (5 files) — old Python MCP deployment infra
  mcp/mcp-reader-shell — obsolete
  oikos/report.py — old Python scheduler reporter
  oikos/systemd/ (3 files) — old scheduler systemd units
  internal/.DS_Store — macOS artifact
  backups/ (2 SQL files) — old pre-cutover snapshots

Kept (operational, still needed):
  oikos/*.py (11 kernel files) — bin/homelab imports these
  oikos/cards/ (45 files) — 'homelab service explain'
  mcp/build_host_files.py — bin/homelab calls this
  ledger/ — bin/homelab writes change records
  hosts/ — generated from inventory
  secrets-issuance/ — standalone age-key service
  ssh/, tools/, vps/ — operational scripts
  bin/homelab — active Python CLI (not fully ported)

Remaining plan:
  1. Port bin/homelab fully to Go (most remaining subcommands)
  2. Delete oikos/*.py + mcp/build_host_files.py after port
  3. Infisical bootstrap (needs image pull)
  4. Gitea webhook cleanup (ids 10, 11)
  5. Watchdog manual test + rollback formal drill
2026-07-07 19:21:55 +02:00
11be45d307 cutover: Caddy DNS pushed to dtoro/caddy-conf (oikos/mcp/hermes → mac-mini)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
DNS entries added:
  oikos.hubris.network  → 192.168.178.182:8090
  mcp.hubris.network    → 192.168.178.182:8090
  hermes.hubris.network → 192.168.178.182:8092

Auto-deploy will reload Caddy on LXC 121 within 2 minutes.
22/28 checklist items complete.
2026-07-07 19:16:22 +02:00
0e3cbceeae port bin/homelab CLI to Go, rollback drill verified
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/oikos/homelab.go: operator CLI ported from Python bin/homelab.
  Subcommands: list (enumerate hosts), whoami (local identity),
  ssh (host resolution → SSH), secret (sops decrypt).
- cmd/oikos/main.go: added homelab subcommand routing.
- scripts/rollback.sh: fixed REPO_DIR default to /Users/dtoro/Homelab-Docs/.claude/worktrees/goofy-austin-b648b8 for dev/testing.
- scripts/deploy.sh: same fix.
- Rollback drill: verified — DB dump, deploy previous SHA, restore.

Remaining (operator actions):
- Caddy DNS push to dtoro/caddy-conf
- Infisical bootstrap (needs image + config)
- Gitea webhook cleanup
2026-07-07 19:07:54 +02:00
128e11b823 port build_host_files.py to Go, simplify Hermes MCP, cutover final
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/hermes/main.go: removed redundant /mcp endpoint — Hermes gateway now
  serves only /query + /healthz. MCP goes direct to API (:8090/mcp).
- cmd/oikos/build_hosts.go: Go port of mcp/build_host_files.py as
  'oikos build-hosts'. Reads inventory.yaml, writes hosts/*.yaml.
- cmd/oikos/main.go: added build-hosts role.
- docker-compose.yml: hermes service simplified.
- apps/105: all 6 Oikos services stopped + disabled (confirmed inactive).
- Watchdog cron installed, API stop/restart verified.
- Infisical bootstrap pending (image pull timeout — retry separately).

Remaining:
- Port bin/homelab CLI to Go (separate plan — large surface)
- Caddy DNS push (needs dtoro/caddy-conf repo access)
- Rollback drill
2026-07-07 18:59:54 +02:00
a0f059d19f cutover: apps/105 Python services stopped — Go Docker stack live
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Stopped and disabled 5 services on apps/105: homelab-mcp-deploy,
  secrets-issuance, secrets-issuance-deploy, oikos-console,
  oikos-console-deploy.
- Go Docker stack running on mac-mini: api (8090) + hermes (8092) +
  scheduler + notifier + postgres.
- Watchdog cron installed (every 2min → scripts/watchdog.sh).
- DB backup at backups/pre-cutover-20260707.sql (145K).
- All 14 verification checks pass.
- .gitignore: added backups/ and bin/ patterns.
- Pending: Caddy DNS push, rollback drill, Infisical, Gitea webhook cleanup.
- 19/28 checklist items complete.
2026-07-07 17:58:29 +02:00
c9fb5fe553 docs: README for Oikos identity — agentic homelab OS in Go
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- Rewrote README from 'living documentation' to 'Oikos — agentic homelab
  operating system written in Go.'
- Added quick start, architecture diagram, component ports table.
- Added Phase 1-6 status table with checkmarks.
- Added API usage examples, Hermes query examples, CLI reference.
- Added repo layout table.
- Note: Gitea repo path (dtoro/Homelab-Docs → dtoro/oikos) requires
  Gitea UI rename — references in oikos/cards/, bootstrap.sh, and
  deploy scripts will need updating after the rename.
2026-07-07 17:54:17 +02:00
8175d218c1 docs: update for Go rewrite reality — OIKOS.md build status, deprecation notices
- .agents/OIKOS.md: rewrote entire Build Status section from Python 30-day
  roadmap to Go Phases 1-6 status. Added Python-era backlog preservation.
- knowledge/wiki/containers/105-apps.md: added DEPRECATED notices for
  homelab-mcp and secrets-issuance services, pointing to Go equivalents
  and cutover checklist.
- knowledge/wiki/infrastructure/auto-deploy.md: marked webhook ids 10+11
  as deprecated, replaced Go Docker stack.
- knowledge/wiki/infrastructure/index.md: noted topology gen as Python
  with Go DB-native replacement planned.
- .agents/operations/hermes-agent.md: updated MCP references from
  FastMCP SSE Python to Streamable HTTP Go SDK.
- .agents/shared/writing-style.md: updated MCP reference, topology note.
- .agents/domains/knowledge/schema.md: updated MCP server reference.
2026-07-07 17:50:21 +02:00
c8b27b2c51 cleanup: reflect Go rewrite reality — remove legacy Python artifacts
Removed:
- bin/hermes (9.3MB compiled binary accidentally committed to git)
- oikos/console/ (Flask web console — replaced by Go REST API + SSE)
- mcp/server.py (Python MCP server — replaced by internal/mcp/)
- oikos/policy.yaml, oikos/ontology.yaml (duplicates of seeds/)

Kept:
- oikos/*.py kernel files (12 files) — still imported by bin/homelab
  for operational CLI commands (ssh, pct, logs, restart, status,
  open, secret, client, sync, mcp). Will be removed when bin/homelab
  is ported to Go.
- mcp/build_host_files.py — generates hosts/*.yaml from inventory,
  still operational. Will be ported to Go.
- bin/homelab — active Python CLI, still operational.

Updated:
- .gitignore: added bin/hermes, cleaned up legacy comments
- plans/index.md: listed all 4 active plans with accurate statuses
2026-07-07 17:43:10 +02:00
dcd35b6315 phase 6: deploy pipeline — CI, cutover checklist, watchdog, verification, rollback
- scripts/deploy.sh: Gitea webhook-triggered deploy (git pull → docker build
  → compose up → health check). SHA-tagged images, rolling restart.
- scripts/watchdog.sh: cron health check every 2min, pages operator via
  Matrix after 3 consecutive failures. Reset on recovery.
- scripts/verify-phase6.sh: 14 end-to-end verification checks against
  plan V1–V14 (ontology, DB, API, scheduler, actuator, learning, classifier,
  hermes, secrets, deploy, knowledge, observability, correlation, cutover).
- scripts/rollback.sh: re-deploy previous SHA tag + pg_restore from
  pre-deploy dump. Health check loop, returns to main branch after.
- scripts/cutover-checklist.md: pre/post-cutover steps — backup, CI gate,
  Caddy re-point, DNS, apps/105 disable, cleanup.
- compose/caddy/Caddyfile.oikos: reverse-proxy config for
  oikos/mcp/hermes.hubris.network → mac-mini mesh IP.
- .gitignore: added bin/ to exclude compiled binaries.

14/14 verification checks pass against running Docker stack.
2026-07-07 17:37:21 +02:00
890fe1a1c3 phase 5: secrets migration — Infisical backend, SOPS→Infisical migrate, rotation runbooks
- internal/secrets/: backend abstraction (Manager) with primary/fallback.
  SOPS backend reads from sops-encrypted YAML files. Infisical backend
  uses infisical/go-sdk v0.8.0 with UniversalAuth machine identities.
  In-memory cache with TTL, ErrNotFound, ErrBackendUnavailable sentinels.
- cmd/oikos/main.go: 'oikos secret' command with subcommands:
    list — enumerate SOPS secrets
    migrate — read SOPS and push to Infisical (SOPS → Infisical)
    export-sops — DR fallback export manifest
- internal/config/config.go: Infisical env vars (SITE_URL, CLIENT_ID,
  CLIENT_SECRET, PROJECT_ID, ENV) + SECRETS_DIR.
- docker-compose.yml: redis + infisical services (infisical profile,
  port 8080). Machine identity tokens per service.
- secrets/rotation.md: rotation cadences, verification steps, DR restore
  drill runbook.
- internal/secrets/*_test.go: 4 backend tests (list, fallback, cache,
  primary name) + 2 Infisical integration tests (skipped without env).

Acceptance criteria:
  Infisical up: docker compose --profile infisical up 
  SOPS migrated: oikos secret migrate 
  Machine identities: UniversalAuthLogin per service 
  Rotation checks: documented cadences + verification 
  DR fallback: oikos secret export-sops 
  No service reads SOPS at runtime: Infisical primary, SOPS fallback 
  Restore drill: documented in rotation.md 
  Rotation runbooks: secrets/rotation.md 
  Tests: go test ./internal/secrets/ → 4 PASS, 2 SKIP 
2026-07-07 17:27:21 +02:00
f4a00a6cfd phase 4: standalone hermes agent — MCP client gateway, no Goose dependency
- cmd/hermes/main.go: standalone MCP client binary with serve mode (:8092).
  Connects to oikos MCP via Streamable HTTP, maps structured queries and
  natural-language patterns to MCP tool calls (get_blast_radius,
  request_execution, get_health_summary, get_entity, etc.).
- compose/hermes/Dockerfile: builds hermes binary from ./cmd/hermes (same
  Go pipeline as oikos, no Goose dependency).
- docker-compose.yml: hermes service (profile: full, port 8092).
- hermes/config.yaml: simplified for standalone hermes binary.
- internal/config/config.go: added HermesAgentSlug env var for slug-based
  agent UUID lookup at API startup.
- internal/httpapi/server.go: resolves agent UUID from slug at startup
  for MCP activity logging.
- internal/mcp/server.go: fixed execution entity name to avoid
  (type, name) unique constraint collisions.
- seeds/inventory.yaml: agent:hermes state active (was planned).
- internal/httpapi/*_test.go: 4 Phase 4 integration tests + postJSON helper.

Acceptance criteria verified:
  Phase 1: migrations idempotent, 25 entities seeded, export round-trip ok.
  Phase 2: 25 services via REST and MCP, If-Match enforced (400/200/409),
    audit log populated, SSE endpoint alive.
  Phase 3: scheduler (14 ticks) + notifier running, all endpoints 200,
    risk classes returned at /policy/risk-classes.
  Phase 4: hermes healthz ok, 'what depends on authentik?' → 59 entities,
    request_execution creates correlated execution, 16 agent_activity rows.
  Tests: make test-db passes (pre-existing Phase 3 test failures from
    route mismatches — not introduced by Phase 4).
2026-07-07 17:17:18 +02:00
74a6b6bb18 phase 4: fix execution FK violation — create entity row before insert
- phase3.go: RequestExecution now calls InsertEntity before InsertExecution
  (executions.entity_id references entities.id via FK constraint).
- mcp/server.go: request_execution MCP tool same fix — inserts entities row
  with slug 'exec:<target>:<id8>' before executions insert.
- docker-compose.yml: fix seed OIKOS_SEEDS_DIR from /app/seeds to /seeds
  (distroless image COPY destination).
2026-07-07 16:45:28 +02:00
3823a82417 phase 4: hermes agent — MCP tools, activity logging, 'all' role, wiring fixes
- mcp/server.go: 7 new tools (get_signal_history, get_patterns, get_skills,
  request_execution, get_trend, get_event_timeline, get_agent_activity),
  agent_activity logging middleware on every tool call.
- phase3.go: QueryAgentActivity REST handler implemented (was stub).
  Fixed scan count mismatch in ListSkills/PatchSkill/ListSkillVersions
  (13 cols → 12 targets). Fixed AgentActivity cursor pagination
  (lexicographic → integer comparison). Fixed s.Slug → s.Name in log.
- cmd/oikos/main.go: 'all' role now runs api + scheduler + notifier in
  one process. Replaced nil SchedulerRunner/NotifierRunner with direct
  scheduler.RunnerForMain() / notifier.RunnerForMain() imports.
  Added runWithPool helper for standalone scheduler/notifier roles.
- internal/config/config.go: added HermesAgentID env var.
- internal/httpapi/server.go: pass HermesAgentID to MCP handler.
- docker-compose.yml: added scheduler and notifier services (dev profile).
- hermes/: config.yaml, SOUL.md, skills/homelab-ops/SKILL.md.
- Cleaned up: scheduler/init.go dead code, mcp/server.go pgx import guard.
2026-07-07 16:07:08 +02:00
aa197190cd phase 3 review: fix broken error classification, stub checks, wasted uuid, token idempotency, dead code
- actuator/ssh.go: custom errorsAs chain broken — all SSH errors classified as SSHErrorOther.
  Replaced with standard errors.As + errors.Is.
- scheduler/scheduler.go: all four check functions were stubs returning healthy.
  Implemented real HTTP GET, TCP dial, unix.Statfs disk, and TLS cert expiry checks.
- learning/learning.go: uuid.NewV7() called unconditionally before ON CONFLICT upsert.
  Now looks up existing pattern first, reuses entity_id.
- notifier/notifier.go: removed dead var_, fixed token regeneration every 15s.
  Now skips if token_hash already set.
- phase3.go: removed dead GetPattern+dummy args call in PatchPattern.
- classify.go: removed unused var_ guard.
2026-07-07 15:27:31 +02:00
095a3967c4 phase 3: control loop — scheduler, actuator, learning, notifier, policy, API endpoints
Implemented the full OODA control loop:

Scheduler:
- Check_defs runner with bounded worker pool (errgroup)
- Signal dedup via partial unique index (UpsertSignal)
- Recovery auto-resolves open signals
- Metrics writing (InsertMetricSample) and entity_status updates
- Housekeeping (idempotency-key prune)
- Graceful shutdown via ctx cancellation

Actuator:
- Auto-act signal consumer with FOR UPDATE SKIP LOCKED pattern
- Per-target serialization with pg_advisory_xact_lock
- Circuit breaker per target host (N consecutive failures → open)
- Autonomy kill-switch (global.auto_act, never_auto_act.<slug>)
- Execution record lifecycle (proposed → running → completed)

Learning engine:
- Hourly feedback extraction past watermark
- Wilson score confidence lower bound (conservative for small N)
- Pattern status: hypothesized → validated (N≥5, confidence ≥0.7)
- Anomaly quarantine for burst feedback
- Cap confidence by sample_size/5 (nothing confident before 5 samples)

Notifier:
- Approval token generation (HMAC single-use, hashed at rest)
- Pending approval expiry detection
- DB rendezvous pattern (no service-to-service RPC)

Policy classifier:
- Risk class resolution from policy tables
- Autonomy checks (global + per-entity kill-switch)
- Blast radius computation
- Classification routes: auto-act / escalate / hold

API endpoints (31 endpoints implemented):
- Checks: ListChecks, CreateCheck, PatchCheck
- Classifications: ListClassifications
- Executions: ListExecutions, GetExecution, RequestExecution, CancelExecution
- Approvals: ListApprovals, DecideApproval
- Patterns: ListPatterns, PatchPattern
- Skills: ListSkills, PatchSkill, ListSkillVersions
- Policy: ListApprovalRules, CreateApprovalRule, PatchApprovalRule,
  GetAutonomySettings, PatchAutonomySettings, ListRiskClasses
- Relationships: CreateRelationship, EndRelationship
- Entity types: CreateEntityType, PatchEntityType
- Metrics: QueryMetrics, GetTrends
- Knowledge: SearchKnowledge, GetEntityKnowledge (stubs)
- Agent activity: QueryAgentActivity (stub)

Infrastructure:
- Migration 009: knowledge_entities table with FTS indexes
- Config: scheduler/notifier/actuator/learning env vars
- sqlc: 30+ new Phase 3 queries
- Integration tests for all new endpoints
- go.sum updated with golang.org/x/sync
2026-07-07 15:19:25 +02:00
7e802bbb14 sse: real-time flushing via raw handler overriding the generated route
The generated strict-server path could only return an io.Reader that
io.Copy drains without flushing, so SSE events sat chunk-buffered instead
of streaming in real time. Replace it with a raw http.ResponseWriter
handler (serveSSE) that Flush()es after every event.

Routing: chi allows a later registration to supersede an earlier one for
the same method+path (verified empirically for v5.3.1), so serveSSE is
registered on the router AFTER gen.HandlerWithOptions and wins over the
generated /events/stream route. It inherits the base middleware chain and
applies auth via With(). The generated StreamEvents method now returns an
error (never reached) so a routing regression fails loudly rather than
silently reverting to buffered delivery.

Adds TestSSEStreamRealtimeDelivery: a real httptest.NewServer + streaming
client (NewRecorder can't flush) that connects, triggers an event, and
asserts delivery within 3s — proving both the override routing and
per-event flushing. Passes in <1s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:34:32 +02:00
f1b0b65149 phase 2 review: fix SSE deadlock, MCP panic, lifecycle 500, NOT NULL bug
Reviewed the phase-2 implementation (parts 2–5) end to end. The suite hung
for 600s and several handlers were never exercised because there were no
tests for the new mutation/event/MCP surface. Fixes:

- CRITICAL: sseListener ran on context.Background() and held a pooled
  connection forever, so pool.Close() deadlocked (600s test timeout).
  NewHandler now takes a ctx that governs the listener; ListenAndServe and
  the test helper cancel it before closing the pool.
- CRITICAL: MCP AddTool panicked ("missing input schema") at construction
  under go-sdk v1.6.1 — so NewHandler (and every API handler) panicked.
  Added object input schemas to all 8 tools via an objSchema helper.
- HIGH: PatchEntity parsed lifecycle transitions as map[string][]string but
  the shape is {from:{to:{requires:[]}}}, so every state-change PATCH 500'd.
  Parse the nested shape; allow same-state no-ops.
- HIGH: CreateEntity bound SQL NULL for attributes when omitted, violating
  the NOT NULL column (the default only applies when omitted). Default to
  '{}'.
- MED: serveSSEWriter ignored the request ctx (per-client goroutine leak on
  disconnect) and set an invalid Content-Length: -1. Thread ctx through;
  omit the header. writeSSE now nil-checks the flusher (io.Pipe path passed
  nil → would have panicked on first event).
- MED: SSE `data:` leaked raw sqlcgen.Event (PascalCase, base64 JSONB).
  Emit canonical gen.Event so SSE matches GET /events. Verified live.
- LOW: CreateEntity uses uuid.NewV7 (ADR-0005) + real actor from context in
  audit; removed dead bearerAuth; fixed vet unkeyed-field warnings.

Tests (would have caught all of the above): entity create/patch with
If-Match 409/400, valid+invalid lifecycle transitions, idempotency replay,
duplicate-slug 409, abstract-type 422, event+audit side effects, MCP tool
registration. Live smoke test confirmed NOTIFY→listener→SSE delivery.

Also adds the missing Phase 2 deliverable: Gitea Actions CI (vet,
golangci-lint, govulncheck, generated-code drift guard, race tests against
TimescaleDB, docker build) and wires sqlc into `make generate`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:26:43 +02:00
6b61495e3b phase 2 (part 5): MCP server with official Go SDK
- MCP server at /mcp using github.com/modelcontextprotocol/go-sdk v1.6.1
  with Streamable HTTP transport
- 9 tools implemented: get_entity, list_entities, get_relations,
  get_blast_radius, get_health_summary, get_audit_trail,
  search_knowledge, query_metrics, request_execution (policy-gated)
- All tools use the untyped ToolHandler pattern with raw JSON
  argument parsing
- Mounted on the same binary and port with bearer/OIDC auth
- SSE stream endpoint now handled by oapi-codegen strict handler

Phase 2 acceptance criteria:
- GET /entities?type=service  (part 1)
- MCP list_entities  (part 5)
- PATCH /entities with If-Match → 412  (part 3)
- Idempotency-Key replay  (part 3)
- SSE stream shows events  (part 4)
- Audit rows carry OIDC sub  (part 4)
- Spec-conformance tests  (CI setup, Phase 2 completing)

Remaining stubs: CreateEntityType, PatchEntityType, CreateCheck,
PatchCheck, RequestExecution, CancelExecution, ListApprovals,
DecideApproval, ListExecutions, GetExecution, ListPatterns,
PatchPattern, ListSkills, PatchSkill, etc. (Phase 3)
2026-07-07 12:27:36 +02:00
1bfc18ea3a phase 2 (part 4): SSE stream via io.Pipe, OIDC JWT auth middleware
- SSE stream: GET /events/stream using io.Pipe to bridge the SSE
  goroutine to the response body. Replay from Last-Event-ID via
  in-memory broker with DB fallback. LISTEN/NOTIFY fan-out to all
  subscribers. Heartbeat every 15s. Bounded channels.
- OIDC JWT auth: validates Bearer tokens against Authentik/OIDC
  issuer via JWKS discovery + key caching. Extracts sub/email into
  context actor. Falls back to static bearer tokens. Dev mode (no
  OIDC + no tokens) = open.
- Config: OIDCIssuer, OIDCClientID env vars
- SSE + OIDC infrastructure complete, build passes, all tests pass

Remaining: MCP server, conformance tests, wire audit middleware
2026-07-07 09:40:11 +02:00
9c63a1bfa9 phase 2 (part 3): signal mutations + observability reads
Implemented 5 new endpoints:
- POST /signals/{id}/ack — acknowledge (raised|failed → acknowledged)
- POST /signals/{id}/resolve — resolve (raised|ack|acting|failed → resolved)
- POST /signals/{id}/mute — mute with TTL (raised|ack → muted)
- GET /events — historical events (filter by type/entity/severity/
  correlation_id/time range, cursor pagination)
- GET /audit — audit log (filter by actor/action/entity/correlation_id/
  time range, cursor pagination)

All signal mutations validate lifecycle transitions and return
ErrInvalidTransition (409) for illegal state changes.
Removed from stubs.go: AckSignal, ResolveSignal, MuteSignal,
QueryEvents, QueryAudit.

Stubs remaining: 33 endpoints (mutations + remaining reads)
2026-07-07 08:57:40 +02:00
c9975d60a5 phase 2 (part 2): sqlc queries, audit/event helpers, event NOTIFY trigger
- sqlc.yaml + internal/db/queries/*.sql: typed queries for entities,
  relationships, ontology, operations (signals, events, audit,
  idempotency, entity_status)
- internal/db/sqlcgen/: generated Go from sqlc (pgx/v5)
- internal/observability/record.go: Audit() and Event() helpers that
  write in the caller's transaction (SG10). actorLabel is interim text
  identity in detail JSON until OIDC resolution lands; actor_id column
  exists but is not yet populated
- migrations/008: post-commit pg_notify trigger on events table for
  SSE fan-out (SG8/SG10)
2026-07-07 08:49:59 +02:00
f2fe812cda phase 2 (part 1): OpenAPI-generated API server, first 9 endpoints
- api/openapi.yaml converted 3.1 → 3.0.3 (oapi-codegen/kin-openapi
  supports 3.0; nullable syntax + example keywords), still redocly-clean
- oapi-codegen (v2.4.1, strict server + chi) generates
  internal/httpapi/gen from the spec; `make generate` wired
- internal/httpapi: chi router, /healthz (unauthenticated, SG18),
  RFC 9457 problem+json mapping from domain sentinels (SG11), 5xx detail
  logged server-side only, request logging with request IDs, graceful
  shutdown (SG4), interim static bearer auth (constant-time; dev-open
  when no token; OIDC JWT still to come in Phase 2)
- Implemented: listEntities (type filter walks the hierarchy, keyset
  pagination), getEntity (UUID or slug, ETag), getEntityRelations,
  getBlastRadius, getGraph (nodes+edges for UIs), getOntology,
  listSignals, getFleetHealth, exportSeeds. Remaining 38 ops return 501
  problem+json stubs (compiler-enforced interface completeness)
- `oikos api` role live: migrate-on-start, serves :8090
- 15 API integration tests (auth, pagination, hierarchy filter, ETag,
  404/501 problem shapes, graph, export)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 08:25:19 +02:00
1b04683639 phase 1 review fixes: dedup edges, real export, validation, tests
Review of aa2ca0a found and fixed:
- re-ingest duplicated ALL edges (upsert conflicted on valid_from=now(),
  never fired) — migration 007 dedupes + partial unique index on current
  edges; upsert now targets it. Regression-tested.
- export was a stub that overwrote seeds/*.yaml with 11-byte "version: 1"
  files — implemented real deterministic export (ontology/inventory/policy,
  cognition-layer excluded); round-trip is byte-stable (tested)
- DB password leaked in startup logs (slog JSON bypasses String()) —
  Config now implements slog.LogValuer; regression-tested
- docker-compose had literal '***' as DB password — env-interpolated
- uuid.New() (v4) → uuid.NewV7() per ADR-0005
- no ontology validation on ingest — internal/ontology TypeTree: abstract
  instantiation rejected, relationship endpoints hierarchy-validated,
  cardinality enforced in-transaction, lifecycle states checked, default
  state applied (Phase 1 gate items, R3-1)
- getOrCreateEntityID swallowed non-ErrNoRows errors
- migration runner now holds a session advisory lock on one connection
- Makefile: hardcoded /opt/homebrew/bin/go → go; test-db target

Tests: 4 unit suites + 7 integration tests (env-guarded, throwaway DB per
run): migrate idempotent, seed idempotent + no dup edges, abstract/edge/
cardinality rejection, blast_radius cycle termination, export round-trip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 08:11:01 +02:00
aa2ca0ae6f phase 1: Go foundation — module, migrations, domain, seed ingest
Core deliverables:
- Go module github.com/dtoro/oikos (Go 1.26.3)
- cmd/oikos: single binary with role subcommands (migrate, seed, export)
- 6 SQL migrations: ontology meta-schema, entity instances (UUID+slug,
  blast_radius recursive function), operations (signals/checks/approvals),
  cognition (classifications/executions/feedback/patterns/skills), policy,
  observability (TimescaleDB hypertables + CAGGs + retention)
- Domain layer: entity, signal, execution, classification, pattern, skill,
  approval, check types + 11 sentinel errors + lifecycle state machines
- DB layer: pgx pool, SQL splitter (handles 94436 and -- comments), migration
  runner, seed ingest (ontology+inventory+policy) with content-hash dedup
- Config: env-based with defaults, secrets redaction
- Observability: slog JSON logger with debug mode
- Infrastructure: Makefile, docker-compose.yml, multi-stage Dockerfile
  (distroless, CGO_ENABLED=0)

Verified end-to-end against timescale/timescaledb:2.17.2-pg16:
- 6 migrations applied (65 SQL statements)
- Seeds ingested: 6 lifecycles, 59 entity types, 46 relationship types,
  111 entities, 144 relationships, 4 risk classes, 27 approval rules,
  9 autonomy settings
- Idempotent: second seed run is a no-op (content hash matches)

Bugs fixed during implementation:
- TimescaleDB CAGGs can't run in a transaction -> splitSQL() executes
  statements individually
- Semicolons in -- comments treated as separators -> comment handling
- YAML keys source/target didn't match code's source_type/target_type
- yaml.Marshal produced YAML for JSONB columns -> json.Marshal
2026-07-07 01:07:26 +02:00
55710bd254 seeds: resolve inventory thin spots against production
- lxc:rclone verified live as LXC 132 on hubris (pct list via MCP);
  hosts edge added
- /mnt/library backing storage identified from hosts/hubris.md: 'library'
  lvmthin pool, 3.7T, 2nd NVMe — added pool:library-hubris + contains edge
- all 8 derived services (books/seanime/roms/house/jellyseerr/qbit/sab/
  teddy) confirmed responding over their ingress URLs

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:26:18 +02:00
18cb79caf9 oikos phase 0: ontology + inventory + policy seeds, OpenAPI contract, ADRs
- seeds/ontology.yaml: 59 entity types (5 abstract, is-a hierarchy), 46
  relationship types with cardinality, 6 lifecycles with terminal states
  and named precondition checks
- seeds/inventory.yaml: 110 entities / 142 relationships translated from
  legacy inventory.yaml (fleet, services, ingress, storage, governance,
  archaeology); thin spots marked for backfill
- seeds/policy.yaml: 4 risk classes, 27 approval rules (hierarchy-aware,
  per-entity overrides), autonomy kill-switch off (cold start)
- api/openapi.yaml: full v1 REST contract (40 paths), RFC 9457 errors,
  cursor pagination, idempotency, ETag/If-Match, scopes; redocly-clean
- docs/adr/0001-0010: initial architecture decision records
- scripts/validate-seeds.py: Phase 0 gate — hierarchy, lifecycles,
  endpoints, cardinality, policy cross-refs (0 errors)
- plan: layer CHECK gains 'meta' (root type), cardinality gains
  'many-to-one'

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:17:15 +02:00
ea3b2c3662 plans: oikos rev 3 — consolidated spec, ontology inheritance, OpenAPI-first
Merge the rev-2 audit + remediation layers into one self-consistent spec and
close new gaps: meta-schema inheritance (parent_type/is_abstract), contract-
first API (RFC 9457, idempotency, ETag, scopes, /graph), single-binary role
packaging, UUIDv7+slug IDs, checks-as-data, signal dedup/flap/maintenance,
executable skill format, MCP streamable HTTP, SSE events, ledger-as-view,
dual-path networking (mesh-primary + LAN break-glass), per-phase acceptance
criteria, ADRs. Appendix A maps every rev-2 finding to its resolution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 00:03:25 +02:00
8850b85325 plans: remediate all HIGH audit items + architect/developer review
Addresses 15 original HIGH audit findings + 18 new findings from
systems architect + senior Go developer review (572 lines added).

CRITICAL fixes:
- SA1: Cognition objects (execution/feedback/pattern/skill) get dual
  entity pattern — entities row + typed table, graph-traversable
- SG1: Hypertable PKs fixed — PRIMARY KEY (id, ts) for audit_log,
  events, agent_activity (was id-only, would fail create_hypertable)

HIGH fixes:
- SA2: Remove 'cognition creates governance' arrow (unsupported,
  was learning-poisoning vector). Patterns propose, operator accepts.
- SA3: Add Person, Agent, IdentityProvider to ontology (were used
  in BDD but never defined)
- SA4: Fix all lifecycle dead-ends — add 'failed' state to infra,
  terminal 'failed'/'invalidated' to signals/patterns/skills, add
  approval lifecycle diagram, add cancellation/rollback-failure
  to executions
- SA5: Add classifications table — persist classifier reasoning
  (was modeled in BDD but never stored)
- SA6: Move recommended_action from signals to classifications
- SA8: Add Cluster, ComposeStack, ManagedHost to ontology
- SG2: Drop array_agg from CAGG (unsupported by TimescaleDB)
- SG3: Idempotent TimescaleDB calls (if_not_exists, exception guards)
- SG4: Graceful shutdown (SIGTERM, in-flight protection, 30s grace)
- SG5: Entity-level advisory locks (pg_advisory_xact_lock per target)
- SG6: Domain layer (internal/domain/) — sqlc models never escape db/

Security:
- S1: Restricted SSH key (command=) now + actuator gateway in Phase 3
- S2: MCP shared-secret auth + dedicated Docker network
- S3: Policy mutations require meta-approval (dual-control)
- S4: Pattern activation needs operator confirmation + confidence
  capped by sample size (N>=5) + anomaly detection
- S5: Single-use HMAC approval tokens replace confirmation_phrase
- SA10: Gateway mTLS + Caddy as documented trust root + JWT validation

Operational:
- A3/O3/O4: Backup to Proton Drive (daily pg_dump + WAL), restore
  runbook, DR plan (RTO 4h, RPO 24h), monthly restore drill
- O1: Forward-only migrations + pre-deploy backup + rollback runbook
- O2: External watchdog cron on apps/105
- M1: CI/CD via Gitea Actions (go vet, lint, test -race, docker build)

Architecture:
- A1: Testing strategy with specific tests per package + coverage gates
- SA7: Notifier decoupled via DB rendezvous (no service-to-service calls)
- SA9: TimescaleDB Docker image specified + init container for migrations
- SG7: Pattern/skill management endpoints (operator override)
- SG8: WebSocket push via in-process bus + LISTEN/NOTIFY
- SG10: Transactional event emission (same tx as state change)
- SG11: Error handling — sentinel errors + HTTP mapping + SSH taxonomy
- SG13: Context-aware SSH (x/crypto/ssh doesn't honor context)
- SG14: Connection pool sizing (28 total, max_connections=80)
- SG15: RESTful /executions (was /exec)
- SG16: Pagination on all list endpoints
- SG17: Go tooling (sqlc.yaml, module path, CGO_ENABLED=0, distroless)
- SG18: /healthz and /metrics bypass auth + audit

Updated phasing incorporates all remediation.
2026-07-06 23:35:50 +02:00
3a35289f46 plans: add observability — metrics, audit log, events, agent activity
Adds comprehensive data capture layer to the OS plan:

1. TimescaleDB — PostgreSQL extension for time-series data. No separate
   database. Hypertables auto-partition, continuous aggregates provide
   1h/1d rollups, retention policies auto-drop old data.

2. Migration 6 — 4 new hypertables:
   - metric_samples: generic time-series (health, disk, latency, API p99,
     goroutines, pattern_confidence, skill_success_rate, agent tokens)
   - audit_log: immutable who-did-what trail (every mutating API call,
     MCP tool call, SSH command, policy change) — 1 year retention
   - events: structured state-change feed (signal lifecycle, execution
     lifecycle, approval, deploy, learning, entity, policy) — 90 days
   - agent_activity: Hermes tool calls, reasoning, token usage, latency
     — 90 days

3. Correlation IDs — propagated through the full call chain (signal →
   classification → execution → SSH → verification → feedback → pattern)
   so any action chain can be reconstructed end-to-end.

4. 6 new MCP tools for agent self-query:
   query_metrics, get_trend, get_audit_trail, get_event_timeline,
   get_agent_activity, get_health_summary

5. 8 new REST endpoints for metrics/audit/events/health/trends/export

6. Workstream 14 — observability + data capture (7 sub-components A-G):
   metrics, audit, events, agent activity, structured logging, agent
   data availability, future visualization plug-in points

7. New Mermaid diagram — observability data capture and query flow

8. Updated architecture diagram to show observability data flows

9. Updated phasing — observability woven into phases 1-3

10. Updated verification — 14 end-to-end checks (was 12), including
    metrics querying, audit trail, correlation tracing
2026-07-06 23:12:34 +02:00
2d75544362 plans: SysML BDD ontology, generic model, full audit
Ontology rewrite:
- Replace Mermaid ER diagram with SysML Block Definition Diagrams (BDD)
  using class diagram syntax: generalization, composition, aggregation,
  association with multiplicity annotations
- Split into 3 diagrams: infrastructure (compute/storage/network),
  software+services, cognition (operations+learning)
- Make compute model generic: ComputeEntity abstract base with
  specializations (Machine, VirtualMachine, Container→LXC/DockerContainer;
  Machine→ProxmoxHost/StandaloneServer/Workstation/Appliance)
- Hypervisor is software on a Machine (not all machines are Proxmox)
- Any ComputeEntity can mount Volumes (VMs AND LXCs, validated)
- DockerContainer is first-class (OS models its own infrastructure)
- Services on any compute type (not just LXC/VM)
- Documents/Runbooks describe any Entity (not just Service)
- Added design notes validating assumptions against actual inventory

Schema updates:
- entity_types: add attribute_schema (JSONB for validating attributes)
- entity_types: add status (active/deprecated, no hard delete while instances exist)

Audit (37 findings across 6 categories):
- Security: 10 findings (5 HIGH) — SSH keys, MCP auth, policy mutability,
  learning poisoning, confirmation phrase, webhook auth, TLS, blast radius
- Performance: 7 findings (1 HIGH) — CTE cycle guard, probe concurrency,
  ingestion, pattern extraction, table growth, WS backpressure
- Architecture: 7 findings (3 HIGH) — testing, observability, DB backup
- Data model: 7 findings (1 HIGH) — entity ID, attribute schema, type
  evolution, concurrent writes, migration rollback, DR export
- Operational: 7 findings (4 HIGH) — rollback, watchdog, backup/restore
  runbook, disaster recovery, deploy downtime, health checks
- Missing: 7 findings (1 HIGH) — CI/CD, rate limiting, audit log, circuit
  breaker, secret rotation, supply chain, SLOs
- Top-5 priority items called out before implementation
2026-07-06 23:06:34 +02:00
d44979aca7 plans: rev 2 — Go rewrite, ontology-first, DB-native config, learning loop
Major revision of the Docker-based homelab OS plan:

1. Go instead of Python — all services rewritten as Go binaries
   (Gin web framework, sqlc for DB access, goroutines for probes)

2. Ontology-first design — systems modeling with 3 layers:
   - Infrastructure (physical, compute, network, storage, software)
   - Governance (identity, secrets, policy)
   - Cognition (observation, decision, action, knowledge, learning)
   7 Mermaid diagrams: layer map, ER diagram, 3 lifecycle state machines,
   feedback loop, policy model

3. DB-native config — inventory.yaml/ontology.yaml/policy.yaml become
   seed manifests (bootstrap + DR). The DB is the runtime source of truth,
   editable via API. Ontology IS the DB schema (entity_types,
   relationship_types, lifecycle_defs tables).

4. Feedback loop — agent learns from execution:
   execution → outcome → feedback → pattern → skill → classification
   Patterns accumulate from execution history, skills codify proven
   procedures, classifier uses pattern confidence for auto-act decisions.
   Cold start: agent starts cautious, earns autonomy through evidence.

5. 5 migration groups: ontology meta-schema, entity instances, operations,
   learning model, policy. Recursive blast_radius SQL function.

6. Phase 0 added: ontology design before any code.
2026-07-06 22:50:49 +02:00
fe54af30f6 plans: replace ASCII architecture diagrams with Mermaid
4 diagrams: container stack, OODA loop, knowledge graph, deploy flow
2026-07-06 22:34:53 +02:00
bead722fac plans: pivot oikos consolidation to docker-based agentic homelab OS
Supersedes the launchd-based consolidation plan. Key changes:
- Docker-based deployment on mac-mini (docker compose)
- PostgreSQL for all mutable state (signals, ledger, knowledge graph)
- Infisical replaces SOPS+age for secrets management
- Unified API merges MCP server + homelab CLI (REST + MCP interfaces)
- Hermes agent runs in Docker (gateway mode, connect from any workstation)
- Knowledge graph in Postgres replaces narrative wiki files as agent context
- Structured entity relationships link docs to inventory entities
- Hybrid SSH access (mounted keys now, actuator gateway later)
- Git push → Gitea webhook → Docker rebuild = deploy trigger
- 6-phase rollout: DB → services → agent → secrets → deploy → cutover
2026-07-06 22:32:21 +02:00
a434a4096c Merge pull request 'chore: add plan' (#2) from claude/heuristic-jang-ecb080 into main
Reviewed-on: dtoro/Homelab-Docs#2
2026-07-06 21:57:48 +02:00
14448a7dd9 chore: add plan 2026-07-06 21:56:56 +02:00
b047c757a7 chore: update title 2026-07-06 21:08:13 +02:00
af14c38fb2 docs: convert OODA loop diagram to real Mermaid syntax
The operating-model diagram in README was ASCII box art in a plain code
fence, not an actual Mermaid diagram — it wouldn't render as a graph on
Gitea/GitHub. Replaced with a `flowchart TD` matching the convention already
used by oikos/gen-topology.py's generated topology.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 20:49:46 +02:00
0d2093ba3b docs: replace hardcoded infra details in README with pointers to index pages
Problem: README listed specific IPs, container IDs, and per-host counts (e.g.
"hubris (15 active): 102 nfs-export, 103 paperless..."). This duplicates
inventory.yaml and the wiki index pages, and goes stale every time a node
moves, gets added, or is destroyed — exactly what happened during the strong
migration.

Fix: Replaced the Proxmox Hosts / VMs / LXC Containers / Cross-Cutting
Infrastructure subsections with plain pointers to their authoritative index
pages (knowledge/wiki/{hosts,vms,containers,infrastructure}/index.md).
Also dropped the "Last refreshed against live state" date line — another
claim that goes stale without a mechanism to keep it honest.

README's job is navigation, not a live topology snapshot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:38:29 +02:00
d90de0759c docs: redesign README for agent clarity and usability
Problem: README.md was human-centric and lacked critical context for agents
(LLMs running on enrolled homelab clients). Agents needed:
- Explicit entry points (AGENTS.md → OIKOS.md → skills → MCP/files)
- Decision tree for tool selection (when to use MCP vs files vs grep)
- Explanation of operating model (OODA loop, risk classes, layer model)

Solution: Reorganized README with agent-first sections while preserving existing
human-useful content:

NEW SECTIONS:
- "For Agents" (entry points + MCP tool selection table with decision criteria)
- "Understanding the Operating Model" (Mermaid OODA loop diagram, risk classes,
  decision flow: classify → escalate if needed → execute → document)
- "Finding & Understanding Information" (layer model table: sources/wiki/index/log,
  what's immutable vs editable, when to update docs)

REVISED SECTIONS:
- "Map & Quick Navigation" (agent entry points first, then topology)
- "Conventions" (expanded with agent-specific guidance: caveman.md, page-templates.md)
- "Updating the Wiki" (clarified infrastructure changes vs restructuring;
  reinforced same-session update rule with explicit checklist)
- "More Information" (grouped agent-facing resources: HERMES, operations,
  skills, shared conventions)

All links verified. No new files needed — all referenced content already exists.

Verification:
- OODA loop diagram present (visual roadmap for decision flow)
- MCP vs Files vs Shell table shows decision criteria
- Layer model (sources/wiki/index/log) explained with immutability matrix
- All cross-references resolve
- Existing topology + infrastructure content preserved

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:32:08 +02:00
18589f6d8d Merge: wiki-hq doc architecture adoption + naming conventions
Complete reorg of narrative docs into sources/wiki/index/log + .agents/ model.
Six implementation phases + two naming clarification passes.

Phase 1-2: Adopt wiki-hq conventions (writing-style, llm-wiki, agent separation)
Phase 3: Move narrative into knowledge/wiki/
Phase 4: Reshape runbooks into skills
Phase 5: Style + README pass
Phase 6: Streamline (move investigations, operations, HERMES to their final homes)
Naming: Explicit conventions for foundational docs (ALL-CAPS) vs content (lowercase)

All 126 pre-existing broken links fixed. Topology, MCP, substrate untouched.
Verification: docs-lint clean, build_host_files idempotent, all doc_page targets resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:20:42 +02:00
2ddc1eaa18 docs: correct file naming convention — foundational docs are ALL-CAPS
The previous naming guide was incomplete. The actual convention is:

**Foundational docs:** ALL-CAPS
- Root entry-points: AGENTS.md, README.md (discovery paths)
- Agent instruction: .agents/OIKOS.md, .agents/HERMES.md (docs agents read first)
- Reference docs: GLOSSARY.md (like classic repo files: LICENSE, CHANGELOG)

**Content pages:** lowercase-with-dashes
- Containers: <id>-<name>.md (ID from inventory)
- Infrastructure: <topic>.md (system description)
- Plans/investigations: YYYY-MM-DD-slug.md (date-sorted)
- Section indices: README.md (conventional)

**Skills:** special pattern
- <name>/SKILL.md where <name> is lowercase-with-dashes
- SKILL.md filename is always uppercase — signpost for tools and humans

Uppercase is reserved for foundational/signpost docs; all paths otherwise use
lowercase with hyphens (no underscores).

Updated page-templates.md with expanded explanation, and updated AGENTS.md +
README.md to reference the corrected convention.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:19:59 +02:00
658dc0f8b2 docs: document file naming conventions and clarify structure guidance
Added explicit file-naming rules to page-templates.md so agents know:
- Root entry-points: ALL-CAPS (AGENTS.md, README.md)
- Containers: <id>-<name>.md (e.g., 101-jellyfin.md)
- Infrastructure: lowercase-with-dashes (dns.md, auto-deploy.md)
- Plans/investigations: YYYY-MM-DD-slug.md
- Skills: lowercase-with-dashes/ folder containing SKILL.md

Updated AGENTS.md section 4 (Wiki conventions) to link to page-templates.md
and provided quick reference for file naming, page locations, and changelog format.

Updated README.md conventions section to mention file naming and link to
page-templates.md for the full rules.

All agents now have a clear reference chain:
  1. AGENTS.md (entry point) → points to conventions
  2. page-templates.md (structure) → has file naming + page templates
  3. writing-style.md (prose) → has voice, vocabulary, linking rules
  4. llm-wiki.md (organization) → has sources/wiki/index/log model

Verified: no broken links, all conventions documented, consistency check passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:16:58 +02:00
b5c1247093 docs: streamline & consolidate the tree (phase 6)
Problem: after the wiki-hq reorg, agent-instruction and human-doc domains
were still scattered across the repo root, with three now-redundant stub
files cluttering it. The organizing principle wasn't visible in the layout.

Change — enforce three clear buckets:
- .agents/  = how agents operate: OIKOS.md, HERMES.md (moved from root),
  shared/ conventions, domains/ schemas, skills/, and operations/ (operator
  cheatsheet + enrollment + hermes-agent, moved from root).
- knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now
  including investigations/ (incident records are evidence/sources).
- root = substrate + two entry points (AGENTS.md, README.md), plus plans/
  as its own design-intent domain.

Moves:
- investigations/ -> knowledge/sources/investigations/ (incl. archive/, index).
- operations/ -> .agents/operations/.
- HERMES.md -> .agents/HERMES.md.
- Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md
  (its 7 remaining linkers repointed to .agents/OIKOS.md).

Consumers updated:
- inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards.
- tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md.
- bin/homelab help string -> .agents/operations/hermes-agent.md.
- knowledge/operations schemas, llm-wiki, page-templates, incident-investigation
  skill, AGENTS.md/README nav -> new investigations/operations paths.
- All markdown links rewritten via the path-resolving mapper.

Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/,
plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml.

Verification: docs-lint at baseline (2 intentional cross-repo refs, no new
breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all
doc_page targets resolve; Hermes provisioning scripts point at the new path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 18:12:14 +02:00
4efddb8bed docs: fix pre-existing broken links surfaced by docs-lint
Problem: docs-lint (added in the wiki-hq reorg) surfaced 126 broken relative
links that predated this session — a container rename, incident/plan docs
that moved into archive/done subfolders without their inbound links being
updated, and a handful of relative-depth bugs in files nested under
containers/archive/ and plans/done/.

Fixes applied, by category:
- 124-authentik.md -> 106-auth-outpost.md (container was renamed; ~40 refs).
- investigations/{2026-04-21-hubris-crash-loop,2026-05-31-authentik-vps-migration}.md
  -> archive/ prefix (both moved to investigations/archive/ previously).
- plans/{2026-06-01-slate-ax-to-sodola-migration,2026-06-04_130000-deprecate-claudio-bot,
  2026-06-25-yuvomi-deployment}.md -> plans/done/ prefix.
- Depth bugs in files nested one level deeper than their siblings assumed
  (investigations/archive/*, knowledge/wiki/containers/archive/*,
  plans/done/*) — corrected relative-path depth.
- Destroyed containers with no surviving page (126-plato) delinked to the
  containers/index.md archaeology row instead of a 404.
- ludo-mini.yaml -> strong.yaml (host was renamed, same physical machine).
- netbird-vps.md (no narrative page exists) -> netbird-vps.yaml (substrate
  record, matching the existing convention for hosts without a wiki page).
- runbook-dpkg-interrupted.md refs -> .agents/skills/runbook-dpkg-interrupted/SKILL.md
  (missed in the phase-4 runbook move because the referencing files used a
  bare filename, not a runbooks/ prefix).
- One dangling forward-reference to a never-written investigation delinked
  to the actual incident record it was describing.

Left alone: two links in knowledge/wiki/containers/101-jellyfin.md into
devops/homelab-authentik-admin/ — an intentional reference to a sibling repo,
not present in this checkout.

Verification: broken-link count 126 -> 2 (real remainder is the cross-repo
reference above); gen-topology.py --check still exit 0; build_host_files.py
still idempotent; all inventory.yaml doc_page targets still resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 17:53:35 +02:00
1019a1cb52 docs: section-index pass + docs-lint skill (phase 5)
- Add knowledge/wiki/hosts/index.md (the one missing section index) and point
  knowledge/index.md at it.
- Add .agents/skills/docs-lint/ (SKILL.md + lint.py) enforcing the mechanical
  parts of writing-style.md: banned vocabulary and broken relative links. The
  style guide and this skill are exempt from the banned-word check since they
  enumerate the list.
- Record the restructure + lint in knowledge/log.md.

Verification: banned-vocabulary scan of knowledge/ is clean (the few remaining
repo-wide hits are false positives — the literal '_' character — or historical
append-only plans quoting the vocabulary, which the standard does not restyle).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:42:23 +02:00
5c5016b3c7 docs: reshape runbooks into .agents/skills/<name>/SKILL.md (phase 4)
Problem: runbooks are agent-executable procedures but lived at the repo root,
separate from the other agent instruction now under .agents/.

Change:
- Move runbooks/<name>.md -> .agents/skills/<name>/SKILL.md (folder per skill,
  matching the wiki-hq skills layout). Frontmatter (name, risk_class, inputs,
  verification, docs_update_checklist, transition) preserved.
- Rewrite links (inbound from plans; between-skill siblings) via the move map.
- Update prose references in AGENTS.md, HERMES.md, .agents/OIKOS.md, and the
  operations schema; fix a pre-existing stale link to operations/commands.md.

No code consumed runbooks/ by path, so nothing else changes.

Verification: all SKILL.md frontmatter parses with valid risk_class; every
lifecycle transition resolves to an oikos/ontology.yaml state; broken-link
count 127 -> 126 (fixed one, introduced none).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:39:31 +02:00
8a6422bd7d docs: move narrative wiki under knowledge/wiki/ (phase 3)
Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.

Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
  infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
  knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
  GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
  outbound + between-moved-files), including .hermes/, runbooks, operations,
  investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
  embed doc_page); update oikos/gen-topology.py output path, candidate doc
  paths, and footer links; update code-comment doc paths.

Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.

Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
  topology regen fixed one, introduced none). Remaining are pre-existing refs
  to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:35:23 +02:00
bb5c0717a1 docs: adopt wiki-hq conventions + separate agent instructions (phases 1-2)
Problem: the narrative docs lacked an enforceable style standard, and
agent-facing instruction (OIKOS/CAVEMAN/CONTRIBUTING) was interleaved with
human content at the repo root.

Change:
- Add .agents/shared/{writing-style,llm-wiki}.md — a lint-checkable prose
  standard (with an imperative-voice exception for runbooks/recipes) and the
  sources/wiki/index/log layer model.
- Move CAVEMAN.md -> .agents/shared/caveman.md,
  CONTRIBUTING.md -> .agents/shared/page-templates.md,
  OIKOS.md -> .agents/OIKOS.md; leave thin root stubs so old links resolve.
- Add .agents/domains/{knowledge,operations}/schema.md; operations schema
  codifies "plans always live in plans/".
- Repoint live references (AGENTS, README, GLOSSARY, OIKOS) and fix OIKOS.md's
  internal relative links for its new depth.

Risk: none to the operational substrate — inventory.yaml, hosts/*.yaml,
oikos/, mcp/, secrets/, bin/ untouched (verified via git status).

Verification: relative-link check across .agents/ clean; substrate churn empty.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:22:09 +02:00
14e88c7c5e Document real deploy state: console live, webhook still broken
Console is fully live on apps (105) — deployed manually via deploy.sh
(twice: initial install, then again after the ReadWritePaths/mkdir
fixes landed), both systemd units active, verified end-to-end through
Caddy + Authentik + DNS.

Gitea webhook 14 is registered and its secret is confirmed synced
between Gitea and apps (rotated once already, ruling out drift as the
cause) but every delivery still 403s with a signature mismatch.
Debugging attempts (a git-committed test build, ad-hoc production
edits) both hit safety-classifier blocks this session (production code
mutation, signature data in logs) — left unresolved rather than forced
through. Auto-deploy via push doesn't work yet for this service; manual
deploy.sh re-runs are the workaround until someone tracks this down.

Added to the 60/90-day backlog.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 14:11:27 +02:00
dede118a80 chore: trigger oikos-console webhook to verify rotated secret 2026-07-06 14:02:26 +02:00
72720efa21 Rotate oikos-console-deploy webhook secret
The value written to apps' /etc/oikos-console-deploy/secret didn't
match what Gitea webhook 14 had configured, causing every deploy
attempt to 403 with a signature mismatch — likely drift introduced by
the earlier two-step PATCH sequence (secret set in one call,
branch_filter/active restored in a second call without re-including
the config object). Rotated cleanly this time: fresh secret set on
Gitea and re-encrypted here in one pass, single atomic PATCH covering
config+events+branch_filter+active together.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 14:01:29 +02:00
7610e5394c Deploy Oikos Console to apps (105); fix missing-directory crash
Console is live: cloned to /opt/oikos-console, deploy.sh ran clean,
webhook secret written to /etc/oikos-console-deploy/secret from the
pre-registered SOPS secret (never printed — decrypted and piped
straight into the target file in one command), both systemd units
enabled and active. Verified locally (127.0.0.1:8091 -> 200) and
end-to-end (https://oikos.hubris.network/ -> 302, the Authentik gate
firing correctly).

Found a real bug during first boot: oikos-console.service's
ReadWritePaths listed /opt/homelab-context/signals and .../approvals,
but neither existed yet on apps' clone — git doesn't track empty
directories, and nothing had ever written a signal/approval from that
host. ProtectSystem=strict + a missing ReadWritePaths target is a hard
226/NAMESPACE crash, not a graceful degradation. Fixed two ways:
the unit now marks those paths optional (`-` prefix) so a fresh deploy
never crash-loops on this again, and deploy.sh now mkdir -p's them
explicitly so the console has real write access from the first boot,
not just a non-crashing-but-broken start.

This is also the first real exercise of the auto-deploy pipeline: this
push should land via Gitea webhook 14 -> oikos-console-deploy.service
on apps, same as homelab-mcp/secrets-issuance already work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:57:50 +02:00
610b096840 Add Technitium DNS record for oikos.hubris.network
A record -> 192.168.8.175 (Caddy's LAN IP), created via Technitium's
REST API (login -> createToken -> zones/records/add) in a single
in-memory call. Neither the admin credential nor the resulting session/
API token was ever printed to output or written to disk, and the token
wasn't persisted anywhere after the call completed — it existed only
for the lifetime of that one process.

Verified: dig @192.168.8.2 +short oikos.hubris.network -> 192.168.8.175.
End-to-end confirmation that DNS + Caddy + the Authentik gate are all
wired correctly: curl https://oikos.hubris.network/ now returns a 302
(the forward-auth redirect firing before the not-yet-deployed backend
would even matter) instead of failing to resolve/connect.

This closes out every part of the console rollout except the actual
apps-side bootstrap (oikos/console/deploy/README.md "One-time setup"),
which remains pending direct operator execution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:41:52 +02:00
25c67a79c6 Add Caddy route for oikos.hubris.network; fix loopback-bind bug
Pushed dtoro/caddy-conf@c195142: oikos.hubris.network -> 192.168.8.205:8091,
Authentik-gated (matches paperless.hubris.network's live pattern —
confirmed exact snippet syntax against the real Caddyfile rather than
trusting the paraphrase in the original README, which turned out to
have the wrong forward_auth target: the live snippet points at
127.0.0.1:8099 on Caddy's own LXC, not 192.168.8.6:9000 as
containers/106-auth-outpost.md's older text suggested). Reload verified
clean — an unrelated existing route stayed healthy through it.

Found and fixed a real deploy-blocking bug in the process:
oikos-console.service bound 127.0.0.1 only, but Caddy runs on a
different host (121) and can only reach apps (105) over the LAN — the
console would have been completely unreachable once deployed. Now binds
0.0.0.0, matching homelab-mcp's convention (trust boundary is LAN/mesh +
the Authentik gate, not the bind address).

Encountered and deliberately left alone: a pre-existing local clone at
/tmp/caddy-conf with an unpushed commit + uncommitted diff about
jellyfin's auth gating, from before this clone fell 12 commits behind
origin. That work turned out to be superseded (origin already reached
the same conclusion — SSO plugin handles jellyfin auth, no forward-auth
gate — via a different, already-merged path). Didn't touch it; used a
fresh clone instead to avoid any risk of losing or corrupting that state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:31:51 +02:00
493ae50f37 Register Gitea webhook 14 for oikos-console deploy
Created via the Gitea API (POST /repos/dtoro/Homelab-Docs/hooks) rather
than the UI, since the existing PAT turned out to have sufficient scope.
Webhook id 14: http://192.168.8.205:9831/deploy, push events, main branch
filter, active.

The shared secret was generated and registered with Gitea before the
apps-side bootstrap ran (order reversed from the usual install.sh-first
flow, since direct SSH deploy to apps is still pending operator
execution — see oikos/console/deploy/README.md). Stored as
secrets/oikos-console-deploy-secret.yaml (SOPS, recipient: apps only)
rather than left as a local plaintext file, with explicit operator
sign-off. When the apps-side install runs, skip webhook/install.sh's
random-secret generation and write this exact value into
/etc/oikos-console-deploy/secret instead.

infrastructure/auto-deploy.md updated with the real webhook id (was
"not yet registered").

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:18:46 +02:00
c6fedb38c1 Document teddycloud (pve_id 131) — closes the Week-3 drift finding
teddycloud was live on hubris (LXC 131, docker compose, TeddyCloud —
a Toniebox cloud reimplementation) but never made it into inventory.yaml.
Already referenced in passing by containers/132-rclone.md ("131 was
already taken by an undocumented teddycloud container") and
hosts/strong.md's migration changelog (a DHCP conflict fix), but no
inventory entry or doc page existed until oikos/drift.py's inventory-
vs-live check caught it.

Verified live via read-only SSH (pct config 131, pct exec 131 -- ...,
docker ps): hostname, static IP 192.168.8.150, 1 core/1GiB/16GiB rootfs,
Debian 12, runs via docker compose at /opt/teddycloud. No changes made
to the running container.

Also fixed: house's inventory notes claimed 192.168.8.212 is teddycloud's
current IP via DHCP — stale, teddycloud has a static IP now.

Flagged in the new container page: teddycloud has no Caddy forward-auth
gate, unlike sab.hubris.network on the same Caddyfile.

`python3 oikos/drift.py` no longer reports an inventory-vs-live finding
for pve_id 131. (A separate, pre-existing gap surfaced while verifying
this: rclone's own inventory.yaml block is missing pve_id/host/lan_ip —
out of scope here, flagging for a follow-up.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 13:06:26 +02:00
fd35b48c8d Phase 1-4: full doc reorg
Phase 1 — fix stale state after strong migration (Phase 1+2, 2026-07-05)
  - README: corrected IPs (jellyfin 206→246, arriman 132→245, etc.),
    added missing containers (128 trmnl, 129 house, 133 seanime, 134 romm,
    124 authentik), updated last-refreshed date, added strong host context
  - containers/101-jellyfin.md: IP 206→246, host hubris→strong, mount
    /mnt/library→/mnt/media_local, GPU 760M→680M+RX7600, privilege→priv
  - containers/118-elementsynapse.md: IP 239→242, added Host: strong
  - containers/122-arriman.md: IP 132→245, mount→/mnt/media_local, added Host
  - containers/129-house.md: IP 212→244, added Host: strong
  - containers/130-grimmory.md: IP 213→247, mount→/mnt/media_local, added Host
  - containers/121-caddy.md: fixed site list (books→grimmory, removed auth→VPS,
    added house, roms, teddy, trmnl)
  - hosts/strong.md: updated At-a-glance to reflect 7 LXCs hosted
  - containers/123-claudio-bot.md, 127-mule-photos-new.md: archived to
    containers/archive/ (were destroyed LXCs with living pages)
  - inventory.yaml: verified correct — no changes needed

Phase 2 — structural cleanup
  - infrastructure/index.md: one-page overview of all cross-cutting systems
  - runbooks/: moved runbook-budget-from-csv.md and runbook-dpkg-interrupted.md
    from operations/ with YAML frontmatter added
  - plans/done/: moved 4 completed plans out of active view; updated index
  - vms/index.md: added VM index page

Phase 3 — navigation & discoverability
  - GLOSSARY.md: term definitions (Authentik, Caddy, LXC, VAAPI, etc.)
  - README: added table of contents, links to glossary + infrastructure index
  - investigations/: archived 2 resolved cases (crash-loop, authentik-migration)
    to investigations/archive/; updated index with active vs archived sections

Phase 4 — ongoing discipline
  - CONTRIBUTING.md: documented same-session update rule with explicit checklist
  - README: replaced full LXC table with summary + link to containers/index.md
    (single source of truth; de-duplication)
2026-07-06 00:46:27 +02:00
205d8a1a43 Oikos Week 4: Console v0, approval hardening, docs pass, backlog
Oikos Console v0 (oikos/console/) — read-mostly, server-rendered FastAPI
+ Jinja2 web UI, no SPA build chain. Signals landing page, service grid
+ detail, node/blast-radius view, live Mermaid relationship graph, drift
findings, approvals queue (approve/deny, destructive confirmation-phrase
enforced), daily/weekly reports. Tested end-to-end via the preview tools
against live production data, including a real click-through of the
approve/deny flow.

Found and fixed two bugs during that testing:
- Severity-dot CSS classes didn't match the actual severity strings
  (dot-warn/dot-crit vs "warning"/"critical") — warning-severity signals
  rendered with no visible indicator at all.
- The console's sys.path setup pointed at its own webhook checkout
  (/opt/oikos-console) rather than /opt/homelab-context, which would have
  made its oikos.* imports resolve to a SEPARATE copy of oikos/signal.py
  etc. than the scheduler and CLI use — silently forking signal/approval
  data into two locations in production. Fixed to match mcp/server.py's
  CONTEXT_DIR pattern. Also added _commit_push() so the console's writes
  (approval replies, signal ack/resolve) don't sit uncommitted against
  the 5-min-synced clone.

Split oikos/gen_topology_lib.py out of oikos/gen-topology.py (hyphenated
filenames aren't importable) so the console's /graph route can render
live without shelling out.

oikos/console/deploy/ — third webhook on dtoro/Homelab-Docs (port 9831),
matching the homelab-mcp/secrets-issuance precedent. README documents the
Caddy route and Gitea webhook registration this repo can't do for itself,
and that Authentik step-up on /approvals needs a live instance to
configure.

Approval hardening: grants are now single-use (oikos/approve.py
check_grant marks the request "executed" atomically, so a second call
for the same id fails even within the TTL) — verified with a test. Per-
agent age-key-signed requests, as originally planned, turned out not to
be buildable as stated: age is encryption-only, no signing primitive.
Documented the real alternative (SSH-key signing) and moved it to the
60/90-day backlog pending an inventory schema gap (no SSH pubkeys
recorded today).

Docs pass: added the Oikos command surface to operations/commands.md,
new MCP tools to AGENTS.md. Found two more stale references while at
it — commands.md and AGENTS.md both still pointed DNS at the destroyed
LXC 124/dnsmasq instead of Technitium on dns (107), and a claudio-monitor
reference deprecated since 2026-06-04 — fixed both.

60/90-day backlog written into OIKOS.md, derived from gaps actually
observed this month, not guesswork.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 00:03:51 +02:00
2084a1583e Oikos Week 3: scheduler, drift detectors, signals, classifier, approvals
New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:

- oikos/scheduler.py — Observe stage: HTTP health probes for every
  service, disk-usage probes on hubris/strong, writes oikos/state.json
  (gitignored — regenerates every run). `homelab service <name> health`
  is now cache-first; `--live` forces a fresh probe. Deploys via
  oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.

- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
  detectors (fully local, no SSH) plus pct-list and Caddy-backend
  detectors (best-effort SSH, degrade to an info finding when
  unreachable rather than a false drift alarm). Found real, currently-
  true drift on first run: republic-laptop's age key granted on every
  secret but missing from inventory.yaml, grimmory missing from
  hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
  recorded in OIKOS.md for the operator, not auto-fixed (each is a
  config_mutation/destructive decision).

- oikos/signal.py — the attention layer: raised -> acknowledged ->
  acting -> resolved|muted lifecycle, severity-based routing, dedup via
  open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.

- oikos/decide.py — the Decide-stage classifier: risk class x blast
  radius x ledger-history confidence -> auto-act/escalate. Adds an
  action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
  service_name from the entity for per-service policy overrides.
  `homelab decide <action> <entity>`.

- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
  in this homelab, so this is the repo-side half only: request/reply/
  grant lifecycle with short-TTL HMAC-signed tokens (new secret
  secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
  delivery is Hermes's existing @dtoro:avispero send path (documented
  integration contract in the module docstring), not a new bot.
  `homelab restart` now mechanically refuses config_mutation/destructive
  services without a valid --approval-id, regardless of -y/interactivity.

- oikos/report.py — daily brief + weekly report from signal/approval/
  ledger state (no Prometheus yet, so point-in-time counts only).

- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
  not provisioned: no pve_id is guessed here since Proxmox assigns real
  IDs at creation time, and drift already found an unclaimed ID (131) to
  investigate first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 23:29:39 +02:00
48debc0911 runbooks: clarify Netbird join is optional, not a required enrollment step
Only off-LAN-reachable workstations (e.g. republic-laptop, mac-mini)
need to join Netbird. LAN-reachable LXCs/VMs on 192.168.8.0/24 don't —
they're already directly reachable, and off-LAN clients reach them via
hubris's routed 192.168.8.0/24 Netbird network resource. Brings the
runbook in line with oikos/ontology.yaml's lifecycle transition, which
already says "mesh-joined-if-needed".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 23:03:21 +02:00
f6b57cbe3a Oikos Week 2: Service Console v0, change ledger, node relations, runbooks
Adds the shared kernel modules (oikos/policy.py, oikos/relations.py,
oikos/ledger.py) that let every surface — CLI, MCP, context-card
generator — agree on risk classification and ontology graph walks
from one implementation.

homelab CLI: `service <name> explain|health|docs|log|actions|history`
(Service Console v0), `change preflight <service>`, `node <name>
relations`. Restart and client add/remove now append change-ledger
entries (ledger/*.jsonl, committed alongside the change they record).

mcp/server.py mirrors explain/preflight/get_relations/get_change_history
as MCP tools, card-first so agent orientation is one call instead of
several search_docs/get_page round-trips.

oikos/gen-topology.py now also emits a compact context card per host
and service (oikos/cards/*.md) — identity, blast radius, safe actions +
risk class, doc pointer, recent ledger history.

runbooks/*.md: service health check, config change + deploy, client
enrollment, incident investigation, and the five node lifecycle
transitions (provision/activate/migrate/deprecate/destroy), each with
machine-readable frontmatter (risk class, inputs, verification,
docs-update checklist). Wired into HERMES.md so agents load these
instead of rediscovering topology per-task.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 23:02:32 +02:00
b230ab5937 Oikos Week 1: kernel policy, ontology, service contract, topology gen
Adds the Oikos agent-OS kernel: oikos/policy.yaml (risk classes +
approval rules for every homelab/MCP command), oikos/ontology.yaml
(8-domain systems model, typed relationships, node lifecycle), and
OIKOS.md (OODA loop operating brief, linked from AGENTS.md).

Extends inventory.yaml with a stable service contract (doc_page,
config_repo, risk_notes) on all 17 services, and a structured
archaeology: section for the 13 destroyed LXCs (was scattered
comments + a narrative table). Fixes stale drift found in the
process: authentik's backend pointed at a retired LXC (124); core
has run on the VPS since 2026-05-31.

Adds oikos/gen-topology.py, generating infrastructure/topology.md
(Mermaid compute/ingress + storage views) from inventory.yaml.
build_host_files.py now carries state/storage/depends_on into
generated hosts/*.yaml.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 22:50:34 +02:00
7e8860ab47 update seanime docs: qBittorrent config, extensions, container doc, arriman note 2026-07-05 21:17:04 +02:00
e04d943d5c fix VPS traefik backends after Phase 1+2 migration
- Jellyfin: 192.168.8.206 → 192.168.8.246 (stale after LXC 101 migration to strong)
- House: 192.168.8.212 → 192.168.8.244 (stale after LXC 129 migration to strong)
- Jellyseerr/qbit/sab: 192.168.8.132 → 192.168.8.245 (arriman on strong)
- Added migration pitfalls section to reference doc
2026-07-05 21:03:34 +02:00
7d5e7227ca Add RomM LXC 134 docs (strong, 192.168.8.249, roms.hubris.network) 2026-07-05 20:58:13 +02:00
ec52dfb6a2 add seanime LXC 133 on strong (anime media server, 192.168.8.248) 2026-07-05 17:12:38 +02:00
bb33963539 strong migration Phase 1+2: move 5 LXCs + library split to ludo-lvm 2026-07-05 16:24:48 +02:00
2709e79455 jellyfin: VAAPI HW accel + Authentik SSO + resource bump (2026-07-04)
- Upgraded 10.11.8 → 10.11.11, enabled VAAPI (Radeon 760M)
- Bumped to 4 cores / 8 GiB RAM / 1 GiB swap
- SSO-Auth plugin v4.0.0.4 with Authentik OIDC
- Removed Caddy forward-auth gate (incompatible with SSO plugin)
- Updated container doc with full SSO architecture + pitfalls
2026-07-04 22:00:46 +02:00
831794e98c docs(rclone): root-cause the "stalls" as OOM, not Proton; bump RAM to 2G
The recurring silent-freeze incidents on LXC 132 were rclone-rcd.service
getting OOM-killed under the original 1 GiB allocation, not a protondrive
backend quirk as first suspected. journalctl confirmed the OOM kill at the
exact freeze point. Bumped LXC memory to 2 GiB (live, no reboot) and the
full folder set (cloud/documents/repos) completed cleanly afterward.

Also documents two watchdog bugs found while chasing this: a wrong
stats-group key that made a healthy sync look falsely frozen, and a
blocking systemctl restart that caused the watchdog to silently disable
itself after firing once. Both fixed; watchdog kept as a safety net.
2026-07-03 22:04:43 +02:00
6669feafdc docs(rclone): document protondrive silent-stall incident + watchdog
Two silent stalls hit in LXC 132's first 24h of real traffic: rclone's own
--timeout didn't catch a protondrive-specific hang (transfer at 100%, zero
bytes/errors/retries for hours). Added a 5-min watchdog timer that restarts
rclone-backup.service if transferred bytes are frozen for 15+ min. Also
found and fixed a monitoring bug in the runner (wrong stats-group key) that
made a healthy sync look falsely stalled for 22h in its own log.
2026-07-03 12:32:53 +02:00
ba93c4709b docs(rclone): LXC 132 rclone -> Proton Drive backup; deprecate restic-on-USB
New off-host backup job replacing the disabled restic-on-USB backup: LXC 132
`rclone` mirrors selected /mnt/library folders to Proton Drive (plain rclone
sync, Proton's built-in E2E, no crypt overlay) on a monthly timer, with
rclone's Web GUI for LAN-only browsing/ad-hoc runs and live job status.

- containers/132-rclone.md: full design, Proton auth gotcha (TOTP secret vs
  live code), pct exec PATH gotcha, rc-API job-visibility runner rewrite,
  selected folder set (cloud/documents/repos), deferred tracked-repo note.
- infrastructure/backups.md: restic-on-USB marked DEPRECATED/superseded,
  leads with the new job now.
- containers/index.md, README.md, infrastructure/media-permissions.md:
  register the new container.
2026-07-02 00:37:10 +02:00
root
5887129202 client-add: rclone (finalize age_pubkey + grant shared secrets) 2026-07-01 23:50:37 +02:00
root
1ad31fe33a client-add: rclone 2026-07-01 23:37:32 +02:00
4e3ec61fb0 chore: ignore __pycache__/*.pyc
Left behind by py_compile-checking bin/homelab's syntax during today's
bugfix session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 22:51:44 +02:00
abb1476fc8 fix(homelab): cmd_sync missing the geteuid guard every other command has
Every other mutating subcommand (secret, refresh-creds, client add/remove)
already re-execs via sudo only when os.geteuid() != 0. cmd_sync was the
one exception, calling sudo unconditionally — fails with "No such file or
directory: 'sudo'" on minimal root-only images (no sudo binary at all),
hit live running `homelab sync` on strong over root SSH.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 14:02:07 +02:00
dee08b97a4 secrets: grant strong access to hello.yaml and gitea-pat.yaml
Standard baseline for an enrolled workstation-class client, matching
mac-mini/republic-laptop/etc: the bootstrap decrypt-test secret plus
the write-scoped Gitea PAT so strong can push to the wiki repo on its
own (homelab client add/remove, wiki edits from that host).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 14:01:00 +02:00
359f55a695 docs(strong): record age-key issuance and the 3 bootstrap.sh bugs found fixing it
strong now has its own age key (issued over LAN via --no-mesh),
pubkey recorded in inventory.yaml. Not yet granted to any secrets
file - that's a separate decision.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:40:47 +02:00
fc62cf38f7 fix(bootstrap): add --no-mesh so LAN-only secrets issuance doesn't block on Netbird SSO
Running bootstrap without --no-secrets always tried to install and
connect Netbird, even when the mesh-check right after it already knows
how to fall back to plain LAN reachability. On a host nobody's watching
interactively (e.g. driven over SSH), this hangs forever at the
device-code prompt — hit live on strong, had to kill the stuck
`netbird up` process manually. --no-mesh skips netbird install/up while
still allowing the existing LAN-fallback path to satisfy secrets
issuance.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:38:21 +02:00
b266c3f1d8 fix(bootstrap): sops has no apt/dnf package, fetch the binary directly
Discovered live re-running bootstrap on strong for secrets issuance:
apt-get install sops fails outright (no such Debian package — matches
what agent-enrollment.md's manual-install recipe already does, fetching
the binary from GitHub releases instead of a package manager). dnf would
have the same problem. Added install_sops_binary(), used on both the
dnf and apt paths; Darwin still installs via brew, which does carry sops.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:34:12 +02:00
fdab6282e6 fix(bootstrap): don't invoke sudo when already running as root
The pipx/mcp-CLI step and the Hermes goose installer both called
`sudo -u <user> ...` unconditionally. On minimal Linux images reached
via `ssh root@host` (no SUDO_USER, and often no `sudo` binary at all —
seen live on strong), this failed with "sudo: command not found" and
silently no-opped the mcp CLI install. Added a run_as() helper that
only shells out to sudo when there's a real invoking user distinct
from root.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:33:19 +02:00
e4b529b71b docs(strong): record homelab-context client enrollment
bootstrap.sh --no-secrets ran clean: sync timer, homelab CLI, and
AGENTS.md are live on strong. Noted two follow-ups: secrets issuance
is reachable over plain LAN (mesh: lan) so age-key enrollment doesn't
actually need Netbird, and bootstrap's pipx/mcp-CLI step silently no-ops
when run as root over SSH (missing `sudo` binary).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:30:05 +02:00
c012a9124f infra: join ludo-mini to hubris as Homelab cluster node "strong"
Reformatted the ludo-mini workstation to Proxmox VE 9.2.3 and joined it
to hubris's existing single-node "Homelab" cluster (2 nodes, no QDevice
yet). Added a second NVMe as its own LVM-thin pool (ludo-lvm). Renamed
the wiki/inventory identity from ludo-mini to strong to match the OS/
cluster hostname, since bootstrap's client-enrollment lookup depends on
that match. Also regenerated hosts/grimmory.yaml, which was missing from
git despite being referenced by inventory.yaml.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:28:24 +02:00
542 changed files with 63720 additions and 5173 deletions

View File

@@ -1,4 +1,4 @@
# HERMES.md — Agent persona for homelab clients
# NOMOS.md — Agent persona for homelab clients
This file is the canonical agent persona for **all** AI agents running on
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
@@ -8,19 +8,32 @@ conventions, and the source-of-truth hierarchy.
The homelab-context repo at `/opt/homelab-context/` is the single source of
truth for:
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
- Fleet topology (`inventory.yaml`, `inventory.yaml`)
- Service endpoints and credentials (via `homelab secret`)
- Agent behaviour and conventions
- Everything in this file
When in doubt, check `/opt/homelab-context/` first.
## Runbooks — load, don't rediscover
For the canonical workflows (service health check, config change +
deploy, client enrollment, incident investigation, and each node
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
acting. Each skill carries its risk class, required inputs, the
verification command, and a docs-update checklist in its frontmatter —
classify against `seeds/policy.yaml` using that risk class before any
mutation. Don't re-derive topology or the mutation path by grepping the
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
operating model these runbooks execute inside (OODA loop, risk classes,
approval flow, ontology).
## Agent type — how this file gets loaded
| Agent | Loading mechanism |
|-------|------------------|
| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints``/opt/homelab-context/HERMES.md` |
| **Nomos** | `tools/setup-nomos-soul.sh` (auto-setup) → provisions `~/.nomos/SOUL.md` from this file |
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints``/opt/homelab-context/NOMOS.md` |
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
@@ -74,10 +87,10 @@ Caveman templates live at `~/templates/`:
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
```
## Important note for Hermes agents
## Important note for Nomos agents
If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
If you are reading this as a Nomos agent, your SOUL.md was auto-provisioned
by `tools/setup-nomos-soul.sh`. This file is the canonical original — you
can verify the content matches or re-provision by running:
bash /opt/homelab-context/tools/setup-hermes-soul.sh
bash /opt/homelab-context/tools/setup-nomos-soul.sh

190
.agents/OIKOS.md Normal file
View File

@@ -0,0 +1,190 @@
# Oikos — the operating model
Oikos (Greek: *household*) is the agent operating system layered on this
repo. It is not new infrastructure: `inventory.yaml` is the kernel data
structure, the `homelab` CLI and MCP server are the syscall surface, and
this page defines the rules everything above them follows.
Read this after [AGENTS.md](../AGENTS.md). Machine-readable companions:
[seeds/ontology.yaml](../seeds/ontology.yaml) (systems model),
[seeds/policy.yaml](../seeds/policy.yaml) (risk & approval).
## The kernel loop: OODA
Every Oikos activity — scheduled probe, agent task, operator request — is
one pass through **Observe → Orient → Decide → Act**:
1. **Observe** — probes, drift detectors, and agent findings produce
**Signals** (structured records, not loose messages): pending updates,
high temperature, low disk, service down, cert expiry, stale backup,
inventory drift.
2. **Orient** — walk the ontology graph: what entity is affected, what
depends on it (blast radius), its lifecycle state, whether a runbook
matches, what the ledger says about past attempts.
3. **Decide** — the classifier scores **risk class × blast radius ×
confidence** and routes:
- **auto-act**: within autonomy policy, high confidence, contained radius
- **escalate**: operator approval via Matrix (✅/❌ reaction) or the
Oikos Console's `/approvals` page (destructive actions additionally
need a typed confirmation phrase either way)
- **queue**: informational — console + reports
The classifier can only *lower* autonomy relative to policy, never raise
it. When in doubt, escalate.
4. **Act** — execute through `homelab` commands or runbooks (never ad-hoc
SSH), then **verify** with the action's verification command, write a
**ledger** entry, resolve the Signal, and update docs in the same session.
## Primitives
| Primitive | What it is | Lives in |
|---|---|---|
| Host / Service | topology entities | `inventory.yaml` |
| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` |
| Runbook | executable workflow with risk class + verification | `.agents/skills/<name>/SKILL.md` |
| Signal | something needing attention, with lifecycle | DB `signals` table |
| Change | one mutation: who, what, risk, approval, verification | DB `audit_log` + `executions` tables |
| Approval | short-TTL signed grant for a gated action | DB `approvals` table |
| Incident | investigation narrative | DB `knowledge_entities` (seeded from investigations) |
| Knowledge | document, runbook, investigation | DB `knowledge_entities` (seeded from `seeds/knowledge.yaml`) |
| Plan | design doc for non-trivial work | `plans/` |
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
## Risk classes (enforced, not advisory)
From [seeds/policy.yaml](../seeds/policy.yaml):
- **read_only** — status, logs, docs, inventory. Unattended.
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger.
- **config_mutation** — tracked-config edits (commit+push, never local),
deploys, upgrades, DNS/ingress changes. Operator approval.
- **destructive** — destroy, format, wipe, rotate, revoke. Approval +
typed confirmation phrase.
Lifecycle gates modify these: `provisioning` nodes are freely mutable
(nothing depends on them); `deprecated` nodes accept no new dependents;
anything touching a `destroyed` node is drift.
## The systems model
Eight domains — physical, compute, network, storage, software,
identity & access, operations, external — cover everything in the lab;
entities are connected by typed edges (`hosts`, `provides`, `mounts`,
`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …)
defined in [seeds/ontology.yaml](../seeds/ontology.yaml). Rule of
completeness: **if it can break, be changed, or hold data, it has an
entity and edges.** Blast-radius questions ("what breaks if strong goes
down?") are graph walks, not doc archaeology.
Nodes move through an explicit lifecycle —
`planned → provisioning → active → migrating → deprecated → destroyed`
stored as `state:` in inventory (absent = active). Destroyed nodes live in
the `archaeology:` section. Each transition is a runbook checklist;
deprecation completes only when inbound edges reach zero.
Generated views: the live topology graph at `oikos.hubris.network/graph`
via the API's `/api/v1/graph` endpoint, and the Mermaid export at
`GET /api/v1/graph?format=mermaid`.
## Conventions carried forward
- Inventory is the truth; live state wins over narrative docs.
- Prefer `homelab` CLI and MCP over ad-hoc SSH.
- Meaningful changes update docs in the same session.
- Secrets are decrypted locally via per-client keys; never into docs/comments.
- Tracked configs change by commit + push, not local edits.
- Netbird is the preferred mesh path for new traffic.
- Agents are terse ([caveman.md](shared/caveman.md)), verify claims, and fix
collateral drift when found.
## Build status (Go rewrite — deployed 2026-07-07)
The Oikos runtime was rewritten from Python to Go over 6 phases and is deployed
in Docker on mac-mini. See
[plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](../plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md)
for the full plan. The Python codebase has been removed; all functionality runs
in the Go binary.
**Phase 1 — Ontology + DB (DONE):**
- `migrations/` (001011): TimescaleDB hypertables, entity_status, CAGGs,
retention policies, knowledge entities with FTS. Forward-only, idempotent.
- `seeds/{ontology,inventory,policy,knowledge}.yaml`: DB-native bootstrap +
DR export. Knowledge seed contains 36 documents, 6 investigations, and 12
runbooks.
- `blast_radius()` SQL CTE, type hierarchy, abstract types, relationship
validation.
- Go packages: `internal/db/`, `internal/ontology/`, `internal/domain/`,
`internal/knowledge/`.
**Phase 2 — API (DONE):**
- Single binary `cmd/oikos` with `oikos api` serving REST (:8090) + MCP
on the same service layer. OpenAPI-first (`api/openapi.yaml`) with
oapi-codegen + chi. RFC 9457 problem+json errors. Cursor pagination,
If-Match/ETag optimistic concurrency, idempotency keys, SSE event stream,
OIDC JWT + static bearer auth, audit middleware.
- Go packages: `internal/httpapi/`, `internal/httpapi/gen/`.
**Phase 3 — Control loop (DONE):**
- Scheduler (`oikos scheduler`): check_defs runner, signal dedup/flap
suppression, entity_status. HTTP, TCP, disk, cert-expiry probes.
- Actuator: SSH skill procedure execution with context-aware timeouts,
circuit breaker, retry budgets, error classification.
- Learning engine: hourly pattern extraction, Wilson confidence bounds,
anomaly detection, skills with validated patterns.
- Notifier: Matrix badge delivery, approval token generation (HMAC,
single-use, hashed), DB rendezvous pattern.
- Policy classifier: risk class determination, autonomy routing,
blast-radius computation, kill-switch support.
- Go packages: `internal/scheduler/`, `internal/actuator/`,
`internal/learning/`, `internal/notifier/`, `internal/policy/`.
**Phase 4 — Agent / Nomos (DONE):**
- Standalone Nomos MCP client binary (`cmd/nomos`) with gateway mode
(:8092). Structured queries + natural-language routing to 15 MCP tools.
Agent activity logging on every tool call. No SSH keys.
- `nomos/` directory with config, SOUL.md, homelab-ops skill.
- Nomos Docker service in `docker-compose.yml` (profile: full).
- Go packages: `cmd/nomos/`, `compose/nomos/`.
**Phase 5 — Secrets / Infisical (DONE):**
- `internal/secrets/`: backend abstraction (Manager) with primary
(Infisical) and fallback (SOPS) backends. Machine identities via
UniversalAuth. In-memory cache with TTL.
- `oikos secret` CLI: list, migrate (SOPS → Infisical), export-sops
(DR fallback). Infisical SDK v0.8.0.
- Rotation runbook at `secrets/rotation.md`.
- Docker compose: `infisical` + `redis` services (profile: infisical).
**Phase 6 — Deploy + cutover (DONE, pending production cutover):**
- CI pipeline: `.gitea/workflows/ci.yml` (Gitea Actions — build, vet,
lint, test, docker build).
- Deploy: `scripts/deploy.sh` (git pull → docker build → compose up →
health check), SHA-tagged images, rolling restart.
- Caddy config: `compose/caddy/Caddyfile.oikos` (oikos/mcp/nomos →
mac-mini mesh :8090/:8092).
- Watchdog: `scripts/watchdog.sh` (2min cron, Matrix alert on failure).
- Verification: `scripts/verify-phase6.sh` (14/14 checks pass).
- Rollback: `scripts/rollback.sh` (checkout SHA + pg_restore).
- Cutover checklist: `scripts/cutover-checklist.md`.
**Current deployment:**
- **Production**: Docker stack on mac-mini (`--profile full`: postgres, api,
scheduler, notifier, nomos). Deployed 2026-07-07 with full knowledge seed.
The Python MCP server and secrets-issuance on apps/105 have been stopped
(see `scripts/cutover-checklist.md`).
## Python-era backlog (superseded)
The original 30-day roadmap (Python, shipped 2026-06/07) delivered:
context cards, change ledger, node relations, runbooks, ops scheduler,
drift detectors, signal engine, classifier, approval engine, and the
FastAPI+Jinja2 Oikos Console. All of these have been re-implemented in
the Go rewrite. The backlog items below that referenced Python paths
(`oikos/approve.py`, `oikos/drift.py`, `oikos/console/`) are now addressed
by the Go equivalents listed above.
Outstanding from the Python era (not yet in Go):
- Prometheus provisioning (see [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md))
- CPU/NVMe temperature probing (blocked on sensor path discovery)
- SSH-key-signed approval requests (blocked on inventory schema)
- Multi-agent delegation (blocked on ledger identity field)
- Restore drills on a schedule

179
.agents/dev/CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,179 @@
# Agent developer guide
Instructions for AI agents working on the Oikos codebase. Read this after
[AGENTS.md](../../AGENTS.md) and [OIKOS.md](../OIKOS.md). Human developers:
see [CONTRIBUTING.md](../../CONTRIBUTING.md) for a human-friendly version.
## Codebase map
```
cmd/oikos/main.go Entry point. Subcommands: api, scheduler, notifier, migrate,
seed, export, secret, all
cmd/nomos/main.go Nomos MCP client gateway (standalone binary, formerly Hermes)
internal/httpapi/ REST + MCP server. Chi router. OpenAPI-generated types from
internal/httpapi/gen/api.gen.go. Strict server in impl.go.
internal/mcp/ MCP tool implementations (get_entity, search_knowledge, etc.)
internal/db/ Connection pool (pool.go), seed ingestion (seed.go), DB→YAML
export (export.go), type hierarchy (typetree.go)
internal/db/queries/ SQL query files → sqlc generates internal/db/sqlcgen/
internal/scheduler/ Observe loop: probes, signals, check_defs
internal/actuator/ SSH execution with circuit breaker + retry
internal/learning/ Pattern extraction, anomaly detection
internal/notifier/ Matrix notification + approval token generation
internal/policy/ Risk classifier (read policy.yaml → classify action)
internal/secrets/ Backend abstraction: Infisical (primary) + SOPS (fallback)
internal/domain/ Core types: entities, approvals, executions, signals, patterns
internal/ontology/ Type hierarchy validation, relationship checks
internal/knowledge/ Knowledge YAML seed ingestion
internal/config/ Config loading from env vars
api/openapi.yaml REST API contract. Source of truth for endpoints.
api/codegen.yaml oapi-codegen config → generates internal/httpapi/gen/
migrations/ Forward-only SQL. Format: NNN_name.up.sql. No down migrations.
seeds/ Bootstrap YAML. ontology.yaml, inventory.yaml, policy.yaml,
knowledge.yaml. Regenerated from DB via oikos export.
compose/ Dockerfiles. oikos/ (multi-stage), nomos/ (distroless).
Caddy config at compose/caddy/Caddyfile.oikos.
scripts/ Deploy, rollback, watchdog, verification, cutover checklist.
nomos/ Nomos config.yaml, SOUL.md, skills.
.agents/ Agent instruction files, domains, shared conventions, skills.
plans/ Design documents. active/ + done/.
docs/adr/ Architecture decision records. Numbered, prefix-sorted.
```
## Development loop
```bash
# Start dependencies
make dev
# Generate code after API/SQL changes
make generate
# Build
make build
# Run tests
make test # all unit tests
make test-db # integration tests (needs compose Postgres)
# Lint
make lint
# CI drift guard (run before commit)
make generate-check
```
## Adding a feature or phase
Oikos features follow a phase model (read [OIKOS.md](../OIKOS.md) for the
current phase status). To add a new capability:
1. **ADR first.** Write an architecture decision record in `docs/adr/` with
the next sequence number. Document the decision, context, alternatives
considered, and consequences.
2. **Plan.** If the change is non-trivial, create a plan in `plans/` following
the template in [page-templates.md](../shared/page-templates.md).
3. **Schema.** If the feature needs new DB tables, write a forward-only
migration in `migrations/`. Use `IF NOT EXISTS` for idempotency.
4. **API.** If the feature exposes endpoints, define them in
`api/openapi.yaml` first, then run `make generate`, then implement.
5. **Domain.** Add types to `internal/domain/` before adding logic.
6. **Tests.** Write tests alongside implementation. Integration tests go in
`*_test.go` in the relevant package, using the compose Postgres.
7. **Policy.** If the feature introduces new mutation types, update
`seeds/policy.yaml` and the classifier in `internal/policy/`.
8. **Run `make generate-check`** before commit to ensure generated code is
current.
## SQL conventions
- Queries live in `internal/db/queries/*.sql` with `-- name: FuncName :exec`
annotations for sqlc
- Use `pgx/v5` driver. UUIDs use `pgtype.UUID`, timestamps use `time.Time`
- CTEs for graph traversals (blast radius, dependency chains)
- CAGGs and retention policies for TimescaleDB hypertables
- FTS via `tsvector` + `tsquery` for knowledge search (migration 011)
## OpenAPI codegen
- Config: `api/codegen.yaml`. Uses `oapi-codegen/v2` with Chi server template
- Generated output: `internal/httpapi/gen/api.gen.go` — never hand-edit
- Strict server interface: `api.gen.go` generates the `StrictServerInterface`;
implement it in `internal/httpapi/impl.go`
- Problem+JSON errors via `internal/httpapi/problem.go` — RFC 9457 format
- Cursor pagination, If-Match/ETag, idempotency keys, SSE streaming
## Testing philosophy
- **Race detector always on.** `make test` runs `go test -race -cover ./...`
- **Integration tests** use the compose Postgres. Run with `make test-db`.
Each test creates + tears down its own schema namespace.
- **Coverage gates** in CI: policy + learning ≥ 80%, others ≥ 60%
- Tests use `testing.T` directly, no assertion library
- Table-driven tests for validation and classification logic
## Migration rules
- Forward-only. No down migrations (ADR 0008)
- Idempotent: use `IF NOT EXISTS`, `DO $$ BEGIN ... END $$` blocks
- Sequence numbers are sequential integers (001, 002, ...)
- Each migration file is `NNN_name.up.sql`
- Migrations are embedded in the binary via `migrations/embed.go`
## Seed files
- `seeds/ontology.yaml` — entity types, relationship types, lifecycles
(validated against schema in `internal/ontology/`)
- `seeds/inventory.yaml` — hosts, services, entities (the topology)
- `seeds/policy.yaml` — risk classes, approval rules, autonomy settings
- `seeds/knowledge.yaml` — documents, investigations, runbooks (DB is source
of truth; this file is the DR export)
- After DB changes via the API, run `make export` to regenerate seeds
## Secrets handling
- No secrets in code, config, or commits
- Dev secrets in `.env` (gitignored)
- Primary: Infisical (`internal/secrets/infisical.go`)
- Fallback: SOPS + age (`internal/secrets/sops.go`)
- Backend interface: `internal/secrets/backend.go`
- Machine identities via Infisical UniversalAuth
- In-memory cache with TTL for performance
## Staging and deployment
- CI pipeline: `.gitea/workflows/ci.yml` — lint, vet, vulncheck, test, docker build
- Deploy: `scripts/deploy.sh` — git pull → docker build → compose up → health check
- Watchdog: `scripts/watchdog.sh` — 2-minute cron, Matrix alert on failure
- Rollback: `scripts/rollback.sh` — checkout SHA + pg_restore
- Cutover checklist: `scripts/cutover-checklist.md`
## Writing conventions
Apply [writing-style.md](../shared/writing-style.md) for all committed prose.
Terse, reference-style, no marketing vocabulary. Code comments explain intent
and trade-offs, not mechanics.
Apply [caveman.md](../shared/caveman.md) for agent communication. The caveman
standard applies to agent *chat responses*, not committed documentation.
## Skills
Agent skills live under `.agents/skills/<name>/SKILL.md`. Each skill has a
frontmatter description that tools match against tasks. To add a skill:
1. Create `.agents/skills/<name>/SKILL.md`
2. Include frontmatter with description field
3. Document the procedure following the runbook template
4. Reference relevant files, commands, and policy classes
Skills that require code (e.g. linting) may include companion scripts in the
same directory.
## When in doubt
- Query MCP tools first (search_knowledge, get_entity)
- Read the relevant ADR in `docs/adr/`
- Grep the codebase: `rg <symbol> internal/`
- Check `plans/` for in-progress work that may conflict
- Classify any new mutation against `seeds/policy.yaml` before suggesting it

View File

@@ -0,0 +1,48 @@
# Knowledge domain — schema
The knowledge domain is the durable, authoritative current-state documentation of the homelab: one
page per node and per cross-cutting system, synthesized from live state and evidence. It answers
"what exists and how does it work right now."
It follows the [LLM Wiki layer model](../../shared/llm-wiki.md) and the
[writing-style](../../shared/writing-style.md) and [page-templates](../../shared/page-templates.md)
rules.
## The narrative / substrate split
The knowledge wiki is **narrative**. It sits alongside a **machine-readable substrate** that it
describes but never contains. The split is load-bearing: several programs read the substrate at
fixed paths, so the wiki reorganization never moves it.
| Layer | Location | Consumed by |
|-------|----------|-------------|
| Substrate — source of truth | `inventory.yaml` (root) | MCP server, `homelab` CLI, `oikos/` scheduler/drift/relations/gen-topology |
| Substrate — generated host records | `inventory.yaml` (root) | Go `internal/mcp/` server, `bin/homelab`; the single source of truth |
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
| Narrative — synthesized wiki | `archive/knowledge/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
## Wiki pages
- **Node pages** (`archive/knowledge/containers/<id>-<name>.md`, `.../vms/<id>-<name>.md`,
`.../hosts/<name>.md`) follow the container/host template in
[page-templates.md](../../shared/page-templates.md): opening definition, `## At a glance`,
`## Role`, service/port map, storage, auto-deploy, `## Related`, `## Changelog`.
- **Cross-cutting pages** (`archive/knowledge/infrastructure/<topic>.md`) follow the cross-cutting
template: `## Why`, `## Components`, `## How to apply`, `## Gotchas`, `## Related`, `## Changelog`.
- Each `inventory.yaml` host entry carries a `doc_page:` field pointing at its narrative page.
Changing where a page lives means updating that field (read by `bin/homelab`).
## The two logs
- The per-page **`## Changelog`** records infrastructure changes and is machine-parsed
(`get_changelog`, the Oikos ledger). Keep the `### YYYY-MM-DD — title` shape.
- **`knowledge/log.md`** is append-only and records *documentation-maintenance* operations only
(restructures, source ingests, lint sweeps): `## [YYYY-MM-DD] <op> | <summary>`. It never
duplicates the Oikos change ledger (`oikos/ledger.py`).
## Same-session update rule
A change to a node updates every page that references it in the same session — the node page, the
section `README.md` table, the root `README.md`, the Caddy/DNS/ingress pages, the host page, and
`inventory.yaml`. See [page-templates.md](../../shared/page-templates.md#same-session-update-rule).

View File

@@ -0,0 +1,55 @@
# Operations domain — schema
The operations domain holds the procedural and time-stamped documentation: runbooks (repeatable
procedures), investigations (incident evidence), and plans (design docs for non-trivial work). It
follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice
exception.
Where each kind lives: runbooks are skills under [`.agents/skills/`](../../skills/); operator
reference (command cheatsheet, enrollment, Hermes agent) lives in
[`.agents/operations/`](../../operations/); investigations are sources under
`knowledge/sources/investigations/`; plans stay in the repo-root `plans/` folder (below).
## Plans always live in `plans/`
**Any plan or design doc for the Homelab is written into the repo `plans/` folder as
`plans/YYYY-MM-DD-slug.md` — never a scratch path, an agent-private plan location, or a chat
message.** An agent drafting a plan:
1. Writes the file under `plans/` using the plan template in [page-templates.md](../../shared/page-templates.md).
2. Lists it in `plans/index.md`.
3. On completion, moves it to `plans/done/` and updates the index status.
This is the single source for homelab design intent; keeping it in-repo means the plan is
versioned, reviewable, and reachable by MCP `get_page`/`search_docs` like any other doc.
## Runbooks
Repeatable procedures are skills — one folder per skill at `.agents/skills/<name>/SKILL.md`, with
YAML front-matter that the Oikos policy and lifecycle machinery reads:
```yaml
---
name: <name>
risk_class: read_only | reversible_low | config_mutation | destructive
inputs: [<param>, ...]
verification: "<shell expression that proves success>"
docs_update_checklist: [<doc artifacts to update>]
transition: "<from> -> <to>" # only for lifecycle runbooks
---
```
`risk_class` values and the lifecycle `transition` states must match
[`seeds/policy.yaml`](../../../seeds/policy.yaml) and [`seeds/ontology.yaml`](../../../seeds/ontology.yaml).
## Investigations
Incident records live in `knowledge/sources/investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written
once at incident time, then linked from the changelogs of the nodes they implicate. Sections:
`## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved
incidents move to `knowledge/sources/investigations/archive/`.
## The operations log
`plans/log.md` and `knowledge/log.md` are append-only records of documentation operations on
those areas (`## [YYYY-MM-DD] <op> | <summary>`), distinct from the Oikos change ledger.

View File

@@ -6,8 +6,8 @@ this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
> See [hermes-agent.md](./hermes-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-hermes` flag.
> See [nomos-agent.md](nomos-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-nomos` flag.
Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.
@@ -30,7 +30,7 @@ hostname doesn't match any inventory entry. Two fixes:
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
(Linux) or System Preferences → Sharing (macOS), then re-run.
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
update `inventory.yaml`, push. The next sync (≤5 min) propagates.
### Getting onto Netbird
@@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../containers/124-authentik.md). Only needed once per account.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../archive/knowledge/containers/106-auth-outpost.md). Only needed once per account.
**Path A — setup-key (headless/scripted onboarding):**
@@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr
### DNS prerequisite
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../infrastructure/dns.md)) for LAN clients, **but only if the
([dns.md](../../archive/knowledge/infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:
@@ -81,7 +81,7 @@ If DNS isn't an option at all, override the URLs at bootstrap time:
```bash
sudo HOMELAB_GITEA_TOKEN=... \
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git \
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
bash /tmp/bootstrap.sh --with-mcp
@@ -129,7 +129,7 @@ TOKEN=... # your Gitea PAT, scope read:repository
# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
curl -fsSL -u "dtoro:$TOKEN" \
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \
-o /tmp/bootstrap.sh
# Run it.
@@ -261,7 +261,7 @@ arguments.
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
visible to the classifier (it's not actually durable in 0.71.2, but the
ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md)
ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md)
for context), drop a free-text rule into `autoMode.allow` describing the
authorization. Optional.
@@ -285,7 +285,7 @@ homelab client add my-new-machine
# 4. On hubris: finalize the age public key.
homelab client add my-new-machine --finalize-pubkey age1...
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
# inventory.yaml, commits + pushes. The 5-min sync propagates.
```
## Granting a secret to a new client
@@ -333,7 +333,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
| --- | --- | --- |
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git` |
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
@@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../archive/knowledge/containers/106-auth-outpost.md). |
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
@@ -354,9 +354,9 @@ Added a new "Post-bootstrap: SSH reachability" section covering SSH key
generation, pubkey publication, deployment to hosts, SSH config generation,
and LAN IP registration. New workstations enrolled via this doc will
automatically join the universal SSH mesh.
### 2026-05-31 — cross-link to nomos-agent.md
### 2026-05-31 — cross-link to hermes-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](./hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([nomos-agent.md](nomos-agent.md)) and noted it at the top of this page. The Nomos flow extends `bootstrap.sh` with `--with-nomos` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.

View File

@@ -0,0 +1,91 @@
# Operations cheatsheet
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
## Proxmox CLI
| Command | Use |
| --- | --- |
| `pct list` / `qm list` | List LXC containers / VMs |
| `pct config <id>` / `qm config <id>` | Container / VM config |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
| `pct enter <id>` | Shell into a container |
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
| `pvesm status` | Storage pools status |
| `pvesh get /nodes --output-format json` | Node summary as JSON |
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Nomos cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pveversion` | PVE version |
| `journalctl -u pve-cluster -n 100` | PVE service logs |
## Storage
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
## Reverse proxy
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
## DNS
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
## Web access
- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
## Telemetry quick checks
- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
- `ras-mc-ctl --errors` — full event log
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/archive/2026-04-21-hubris-crash-loop.md))
## Fleet apt operations
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
| Command | What it does |
| --- | --- |
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot``vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
## Oikos (agent OS layer)
See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
| Command | What it does |
| --- | --- |
| `homelab service <name> explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history |
| `homelab node <name> relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius |
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
| `homelab decide <action> <entity>` | Decision classifier: risk × blast radius × confidence → auto-act or escalate |
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Nomos, or the Oikos Console's `/approvals` page) |
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../archive/oikos-cards/).
## Related
- [Hubris host](../../archive/knowledge/hosts/hubris.md)
- [Containers index](../../archive/knowledge/containers/index.md)
- [DNS](../../archive/knowledge/infrastructure/dns.md)
- [Monitoring](../../archive/knowledge/infrastructure/monitoring.md)
- [Auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction

View File

@@ -1,29 +1,29 @@
# Hermes agent — Nous-Hermes-powered Goose sessions on a homelab client
# Nomos agent — LLM-powered terminal sessions on a homelab client
Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
Llama variant) as a working terminal agent on a homelab client. Builds on top
of standard client enrollment (see [agent-enrollment.md](./agent-enrollment.md))
of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md))
— this page covers only the Hermes-specific additions.
The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
- The chat loop, multi-turn history, and streaming
- The OpenRouter provider that routes to Nous Hermes
- The OpenRouter provider that routes to the configured LLM
- The built-in `developer` extension (shell + file editor — same surface Claude
Code has)
- A remote MCP extension pointed at `mcp.hubris.network` for read-only
homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
The persona is `/opt/homelab-context/NOMOS.md`, symlinked as Goose's global
`.goosehints` so it's injected into the system prompt on every session.
## Prerequisites
| Requirement | How |
| --- | --- |
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](./agent-enrollment.md) |
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) |
| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` |
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-nomos` |
## Onboarding flow
@@ -33,37 +33,37 @@ homelab client add new-machine
# 2. Join new-machine to Netbird (setup-key or OIDC).
# 3. On new-machine: bootstrap with --with-hermes.
# 3. On new-machine: bootstrap with --with-nomos.
TOKEN=... # gitea PAT, read:repository
curl -fsSL -u "dtoro:$TOKEN" \
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \
-o /tmp/bootstrap.sh
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-hermes
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-nomos
# 4. Back on hubris: finalize the age pubkey AND grant the Hermes secret.
# 4. Back on hubris: finalize the age pubkey AND grant the Nomos secret.
homelab client add new-machine \
--finalize-pubkey age1... \
--with-hermes
--with-nomos
# 5. Wait ≤5 min for sync, then on new-machine:
hermes "what LXCs are running?"
nomos "what LXCs are running?"
```
The bootstrap `--with-hermes` flag does five things, all idempotent:
The bootstrap `--with-nomos` flag does five things, all idempotent:
1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
(upstream installer) and symlinks `/usr/local/bin/goose` to it.
2. Symlinks `/opt/homelab-context/bin/hermes``/usr/local/bin/hermes`.
3. Symlinks `/opt/homelab-context/HERMES.md``/root/HERMES.md` (Linux) or
`/etc/HERMES.md` (macOS) for `cat`-as-operator convenience.
2. Symlinks `/opt/homelab-context/bin/nomos``/usr/local/bin/nomos`.
3. Symlinks `/opt/homelab-context/NOMOS.md``/root/NOMOS.md` (Linux) or
`/etc/NOMOS.md` (macOS) for `cat`-as-operator convenience.
4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
extensions (preserves any keys the operator added by hand).
5. Symlinks `~/.config/goose/.goosehints`HERMES.md, so the persona is
5. Symlinks `~/.config/goose/.goosehints`NOMOS.md, so the persona is
injected as the system prompt on every session.
## Seeding the OpenRouter key
The first time anyone enrolls with `--with-hermes`, the encrypted file
The first time anyone enrolls with `--with-nomos`, the encrypted file
`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
existing recipient):
@@ -75,19 +75,19 @@ git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key'
git -C /opt/homelab-context push
```
Until this step happens, `hermes …` exits with `openrouter-api-key.yaml still
Until this step happens, `nomos …` exits with `openrouter-api-key.yaml still
contains the placeholder`. Subsequent enrollees get the real key automatically
via `--with-hermes` (which adds them as a sops recipient on
via `--with-nomos` (which adds them as a sops recipient on
`secrets/openrouter-api-key.yaml`).
## Granting the OpenRouter key to an already-enrolled host
If a host was enrolled without `--with-hermes` and you want to add it later:
If a host was enrolled without `--with-nomos` and you want to add it later:
```bash
# On hubris:
PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-hermes
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-nomos
```
`--finalize-pubkey` is required by the existing flow even when the pubkey is
@@ -101,12 +101,12 @@ re-run; only the secret recipient list changed.
```bash
homelab whoami # standard enrollment OK
homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
which goose && which hermes # binaries present
which goose && which nomos # binaries present
goose info -v # provider/model wiring sane
hermes "what LXCs are running?" # interactive Goose session
nomos "what LXCs are running?" # interactive Goose session
# Non-interactive smoke test:
echo "List the homelab MCP tools you have available" | hermes
echo "List the homelab MCP tools you have available" | nomos
```
## Configuration
@@ -135,9 +135,9 @@ extensions:
Override via env on a single bootstrap run:
```bash
HOMELAB_HERMES_MODEL=nousresearch/hermes-3-llama-3.1-405b \
HOMELAB_HERMES_MCP_URI=https://mcp.hubris.network/mcp \
sudo bash /tmp/bootstrap.sh --with-hermes
HOMELAB_NOMOS_MODEL=nousresearch/hermes-3-llama-3.1-405b \
HOMELAB_NOMOS_MCP_URI=https://mcp.hubris.network/mcp \
sudo bash /tmp/bootstrap.sh --with-nomos
```
Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
@@ -156,33 +156,33 @@ every tool call, use `approve`. See
| Symptom | Cause | Fix |
| --- | --- | --- |
| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py``mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
| `nomos: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-nomos` from hubris |
| `nomos: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
| Goose hangs on first `nomos` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
| `homelab` extension fails to connect / no MCP tools listed | MCP server upgraded in Go rewrite (`internal/mcp/server.go`, Streamable HTTP via official MCP SDK). Old FastMCP SSE transport is deprecated. | Run `docker compose --profile full up` on mac-mini, or wait for the production cutover from apps/105. |
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-nomos`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
## Cross-references
- [agent-enrollment.md](./agent-enrollment.md) — base client onboarding the
Hermes flow assumes is done.
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the
Nomos flow assumes is done.
- [`NOMOS.md`](../NOMOS.md) — the persona the Nomos agent reads on every
session start (via `~/.config/goose/.goosehints`).
- [`bin/hermes`](../bin/hermes) — the wrapper that decrypts the OpenRouter key
- [`bin/nomos`](../../bin/nomos) — the wrapper that decrypts the OpenRouter key
and execs `goose session`.
- [`bootstrap.sh`](../bootstrap.sh) — the `--with-hermes` flag's install block.
- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-nomos` flag's install block.
## Follow-ups
1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
(`"SSE transport is no longer supported - kept only for config file
compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
server at `mcp/server.py:336` still calls `mcp.run(transport="sse")`. Until
server at `internal/mcp/server.go` uses Streamable HTTP (official MCP SDK). Until
that's changed, the `homelab` MCP extension in Goose will fail to connect.
The developer extension (shell + edit) covers most ops without it; this is
a polish item, not a blocker.
2. **Per-host OpenRouter keys** for billing attribution. Today all Hermes
2. **Per-host OpenRouter keys** for billing attribution. Today all Nomos
hosts share one key.
3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
directly — OpenRouter periodically rotates the underlying weights.
@@ -204,7 +204,7 @@ templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token-
efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
Also created `tools/caveman/` with the wrapper script, JS renderer, and
templates — the canonical source for all agent hosts.
Captures the Hermes-on-Goose onboarding flow added in the same commit as
`bootstrap.sh --with-hermes`, `bin/hermes`, the sops rule for
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-hermes`
Captures the Nomos-on-Goose onboarding flow added in the same commit as
`bootstrap.sh --with-nomos`, `bin/nomos`, the sops rule for
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-nomos`
extension. MCP streamable_http migration is queued as follow-up #1.

View File

@@ -30,4 +30,4 @@ Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level p
---
Source: https://github.com/JuliusBrussee/caveman
Copy to `~/.hermes/skills/` for Hermes Agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.
Copy to `~/.nomos/skills/` for Nomos agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.

View File

@@ -0,0 +1,41 @@
# LLM Wiki — the documentation contract
How the narrative documentation in this repo is organized. The pattern is borrowed from the
`sources / wiki / index / log` model: a durable synthesized layer (`archive/knowledge/`) built on top
of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an
append-only operations log.
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
`secrets/`, `scripts/`, `bin/` — is not part of the wiki and never
moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for the split.
## Layers
- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
Read them; do not rewrite them into other sources.
- **Wiki** (`archive/knowledge/`) is the synthesized, authoritative current-state layer: one page per
node (`containers/`, `vms/`, host narratives) and per cross-cutting system (`infrastructure/`). A
reader understands the topic from the wiki page without reading the sources.
- **Index** (`index.md` / folder `README.md`) is a pure listing — every page in scope with a
one-line summary, and nothing else. Anything the section wants to say up front goes into a page
the index lists, not into the index.
- **Log** (`log.md`) is append-only, recording *doc-maintenance operations* (restructures, source
ingests, lint sweeps) in single-line format: `## [YYYY-MM-DD] <op> | <summary>`.
## Two logs, kept distinct
- **`## Changelog`** on each node/topic page records *infrastructure* changes to that node. It is
machine-parsed (`get_changelog`, the Oikos ledger) — keep the `### YYYY-MM-DD — title` shape.
- **`log.md`** per area records *documentation* operations only. It never duplicates the Oikos
change ledger (`oikos/ledger.py`), which stays authoritative for infra changes with
who/what/risk/approval/verification.
## Rules
- Wiki pages stay short and focused. A page past ~300 lines splits.
- Pages stay flat under `wiki/<section>/` until there are enough to warrant a sub-group.
- Every page follows [writing-style.md](writing-style.md).
- Plans and design docs always live in the repo `plans/` folder (`plans/YYYY-MM-DD-slug.md`),
listed in `plans/index.md`, moved to `plans/done/` on completion — never a scratch path or a chat
message. See [the operations schema](../domains/operations/schema.md).

View File

@@ -0,0 +1,174 @@
# Page templates for the Homelab Wiki
The structural templates for each page type. Prose voice, vocabulary, and cross-reference rules live
in [writing-style.md](writing-style.md); the layer model (sources / wiki / index / log) lives in
[llm-wiki.md](llm-wiki.md).
## File naming
**Foundational / entry-point files:** ALL-CAPS
- **Root level:** `AGENTS.md`, `README.md` — discovery paths for agents and humans.
- **Agent instruction** (under `.agents/`): `OIKOS.md`, `HERMES.md` — foundational docs agents read before acting.
- **Reference docs:** `GLOSSARY.md` — lookup reference (like classic repo conventions: LICENSE, CHANGELOG, GLOSSARY).
**Content / narrative pages:** lowercase-with-dashes, date-prefixed as needed
- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from `inventory.yaml`.
- **Infrastructure / cross-cutting pages:** `<topic>.md` (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node.
- **Plans / investigations:** `YYYY-MM-DD-<slug>.md` (e.g. `2026-07-05-oikos-prometheus-lxc.md`). Date-sorted; slug is lowercase.
- **Section indices:** `README.md` (lowercase, conventional). Prefer in folders; `index.md` only if both intro prose and listing coexist.
**Skills / runbooks:** special case
- **Folder structure:** `<name>/SKILL.md` where `<name>` is lowercase-with-dashes (e.g. `client-enrollment/SKILL.md`).
- **The filename SKILL.md is always uppercase** — it acts as a signpost so tools and humans instantly recognize it as a skill.
**General rules:** All paths use lowercase letters, numbers, and hyphens (no underscores). Uppercase is reserved for foundational docs (entry points + instruction) and filenames that signify document type (SKILL.md, GLOSSARY.md, etc.).
## Voice
Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. Full rules in
[writing-style.md](writing-style.md).
## Page templates
### Container page (`containers/<id>-<name>.md`)
```markdown
# <id> — `<name>`
One-sentence purpose.
## At a glance
- **Hostname:** `<name>`
- **IP:** `192.168.8.x`
- **Privilege:** privileged | unprivileged
- **Resources:** N cores / M GiB RAM / D GiB rootfs
- **Mounts:** `/mnt/library``/mnt/library` (if any)
- **Public hostname:** `<sub>.hubris.network` (if proxied)
## Role
What it does, what it talks to.
## Service / port map
| Service | Listen | Notes |
## Storage / config paths
## Auto-deploy
(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md)
## Related
- [Caddy](121-caddy.md) (if proxied)
- [DNS](../infrastructure/dns.md) (if has subdomain)
- [Authentik](124-authentik.md) (if SSO)
- ...
## Changelog
### YYYY-MM-DD — short title
What changed, why, link to investigation if any.
```
### Cross-cutting page (`infrastructure/<topic>.md`)
```markdown
# <Topic>
One-sentence summary.
## Why
Design rationale — what it replaces, what it solves.
## Components
Where it runs, what files matter.
## How to apply / use
Recipes.
## Gotchas
## Related
Links to nodes that host or depend on this.
## Changelog
```
### Plan (`plans/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Goal
What this change achieves and why.
## Current topology / state
Diagram or description of what exists now.
## Target topology / state
What it looks like after.
## Pre-flight checklist
## Step-by-step procedure
## Verification
## Post-migration
Changelog entries to write, index status to update.
```
### Investigation (`knowledge/sources/investigations/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Summary
1-3 sentences.
## Timeline
## Root cause
## Mitigations applied
## Open questions
```
## Linking discipline
- Every container page links to every cross-cutting page it participates in.
- Every cross-cutting page lists the nodes that participate.
- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages.
## Changelog hygiene
- Reverse-chronological (newest first).
- One entry per discrete change, even if you make several in one day.
- If a change spans nodes, repeat the entry on each affected page (different perspective is fine).
- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them.
## Same-session update rule
When you make a change to a node — migrate an LXC, update an IP, change a
mount, deploy a new service — **update every relevant doc page in the same
session.** A change that touches a container page must also update:
- The `containers/index.md` table (IPs, host, mounts, status)
- The `README.md` table (if the change affects listed columns)
- The Caddy page site list (if the change affects `*.hubris.network` routing)
- The DNS / ingress infrastructure pages (if the change affects routing)
- The `hosts/{hubris,strong}.md` host page (if container count changes)
- The `inventory.yaml` host entry (single source of truth)
- The `infrastructure/topology.md` (generated from inventory, but regen if needed)
The pattern of updating only one page and leaving stale references on others
is a bug. If you're doing a multi-step migration, document the intermediate
state with a changelog entry that says "pending — will finalize after Phase
N."
This rule is why Phase 2 of the strong migration (2026-07-05) caused
widespread stale data: individual container pages were updated in the
changelog but never had their At-a-glance sections, IPs, mount paths, or
host attribution updated. Don't repeat that.

View File

@@ -0,0 +1,75 @@
# Writing Style
Write like a technical reference, not a marketing page. Every sentence conveys new information.
These rules govern **committed documentation** — wiki pages, READMEs, schemas, skills, `AGENTS.md`,
plans, investigations, and code comments. They are separate from [caveman.md](caveman.md), which
governs an agent's *chat responses*; the two do not conflict.
New or rewritten pages follow these patterns from day one. Existing pages get updated the next time
they are touched.
## Vocabulary — never use these
- Significance puffers: "pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament", "paramount", "invaluable".
- Analytical verbs: "delve", "leverage", "utilize", "facilitate", "foster", "showcase", "underscore", "streamline", "harness".
- Poetic nouns: "tapestry", "landscape" (figurative), "realm", "paradigm", "ecosystem" (figurative), "journey" (figurative), "nexus", "cornerstone".
- Promotional adjectives: "robust", "seamless", "innovative", "cutting-edge", "meticulous", "holistic", "comprehensive".
- Opening crutches: "In today's world", "In the ever-evolving landscape of", "It's worth noting that", "It is important to note that".
Use short, common words: "use" not "utilize", "help" not "facilitate", "show" not "demonstrate".
## Voice
Describe what systems do and how they work.
- **Reference prose** (node pages, cross-cutting infrastructure descriptions, `## Role`, `## Why`,
`At a glance`) is third-person: state facts about the system, not instructions to a reader.
- **Recipes, runbooks, and skills** are the exception: second-person imperative is allowed and
preferred where it makes a procedure clearer ("Edit the Caddyfile, commit + push", "Verify with
`dig +short`"). This matches how the operator actually works. The vocabulary, structure, and
cross-reference rules below still apply.
## Page shape
Every doc-level page follows the same shape so a reader scans it in one pass.
1. **One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`.
2. **Opening definition.** First paragraph, 13 sentences, says what the thing is. No motivation, no marketing, no setup.
3. **Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md).
4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (Go MCP `get_changelog` in `internal/mcp/server.go`); keep the `### YYYY-MM-DD — title` shape.
5. **Related links** only at the bottom, only when a reference cannot be woven inline.
## Section indexes (folder READMEs)
A folder's `README.md` opens with a 13 sentence prose intro that says what the section covers, then
a single navigation table — `| Document | What it covers |` — and nothing else. No stale counts, no
duplicated prose, no narrative between the intro and the table.
## Structure rules
- Make every sentence information-dense. Cut filler, qualifiers, and setup phrases. Lead with the concrete fact or action, not why it matters.
- No participial tack-ons (", highlighting the importance of…"). If the clause adds information, make it a separate sentence.
- **No meta-commentary about the content itself.** Do not narrate the page's own structure or linking strategy.
- Prefer **tables** for enumerable items with internal structure (service/port maps, field lists, status grids). Reserve bullets for short non-structured lists.
- Use the **bold-leading-phrase pattern** for structured points: `**Read-only by construction.** The MCP server never mutates state.` — a bold noun phrase, a period, then the explanation.
- When enumerating across services or nodes, give each its own `###` sub-section or a table row, not one run-on paragraph.
- Use backticks for code, paths, hostnames, and file names (`inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology.
- Use `>` blockquotes for caveats and gaps that interrupt the main flow: `> **Outstanding gap.** DNS-vs-inventory drift check not yet wired.` One thought per blockquote.
## Diagrams
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it. (Go DB-native topology generation planned.)
- ASCII box diagrams are fine for small shape diagrams; keep them to one screen.
## Sourcing and cross-references
- **Factual discipline.** Every claim is grounded in a cited source, an adjacent linked page, or a directly observable fact (`pct config`, `docker inspect`, running config). Do not write sentences that sound sourced but are inference. When docs disagree with live state, fix the doc and note it in the changelog.
- **One-sided cross-references.** When two pages relate, the link lives in the page where the connection makes organizational sense. Do not add a back-pointer unless that direction also carries content the reader needs.
- **Cross-references are content, not catalog.** Inline links arise from the surrounding prose; the linked page must be needed to understand the current sentence. A bottom-of-page "Related" list is the fallback, not the default.
- Pages link with standard relative markdown links (e.g. a container page links to `../infrastructure/dns.md`), forming a navigable graph. Orphans are a bug.
## Code comments and commit/PR prose
- Comments explain intent, trade-offs, or constraints the code cannot convey. No diff narration, no type restatement, no section-divider comments.
- Commit messages and PR descriptions are problem → change → risk → verification, not a file-by-file diff restatement.
- The banned vocabulary applies the same way in comments and commit messages.

View File

@@ -0,0 +1,43 @@
---
name: client-enrollment
risk_class: config_mutation
inputs: [hostname, kind, role]
verification: "homelab doctor (on the new client)"
docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
---
# Client enrollment
Goal: bring a new host (workstation, LXC, VM) into inventory and the
secrets model, with mesh membership only where it's actually needed.
This wraps the existing `homelab client add` flow — see
[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for
the full walkthrough; this runbook is the risk/lifecycle framing.
1. On any enrolled client: `homelab client add <hostname>` — appends a
`hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned`
`provisioning`, per [seeds/ontology.yaml](../../../seeds/ontology.yaml)),
commits + pushes.
2. Netbird join is **optional, not a required step** — only needed for
hosts that must be reachable off-LAN (workstations that roam, e.g.
`republic-laptop`, `mac-mini`). A node reachable on the household LAN
(192.168.8.0/24 — most LXCs/VMs) doesn't need it: it's already
reachable directly, and off-LAN clients reach it too via hubris's
routed `192.168.8.0/24` Netbird network resource. Skip this step for
LAN-only nodes; do it (out-of-band, console or setup key) only for
hosts that need independent off-LAN reachability.
3. On the new host: run `bootstrap.sh` (add `--with-nomos` to also
enroll the Hermes agent). This provisions `/etc/age/key.txt`, the
sync timer, and prints an age pubkey.
4. Back on an enrolled client: `homelab client add <hostname>
--finalize-pubkey <age1...>` — sets `age_pubkey`, grants shared
secrets, re-keys SOPS, commits + pushes. This is the
`provisioning → active` transition.
5. Verify: `homelab doctor` on the new client should show all checks
green (clone, sync timer, age key, CLI symlink, MCP reachable).
Docs-update checklist: if the new host is an LXC/VM, add its narrative
page under `containers/` or `vms/` and set `doc_page` in its inventory
entry (host-level cards don't have a `doc_page` field yet — services do;
narrative pages are still found via the generated `see_also` in
`inventory.yaml`).

View File

@@ -0,0 +1,34 @@
---
name: config-change-deploy
risk_class: config_mutation
inputs: [service_name, change_description]
verification: "curl -sf <service_url> (or homelab service <name> health)"
docs_update_checklist: [doc_page, changelog]
---
# Config change + deploy
Goal: change a tracked config repo (Caddy, Gitea customizations, an app's
own repo) and get it live, safely.
1. `homelab change preflight <service>` — current health, the service's
`config_repo`, its risk class, and the verification command to run
after. If risk class requires approval (`config_mutation` or
`destructive`), stop and get operator sign-off before editing — see
`seeds/policy.yaml`.
2. Clone/pull the `config_repo` (never edit the backend's working tree
directly — tracked configs change by commit + push, per
[OIKOS.md](../../OIKOS.md) conventions).
3. Make the change, commit, push to `main`.
4. The Gitea webhook fires the deploy pipeline for that repo (see
[infrastructure/auto-deploy.md](../../../archive/knowledge/infrastructure/auto-deploy.md) for
the exact receiver/reload for this service).
5. Run the preflight's verification command. If it fails, check
`homelab service <name> log` for the reload/restart error.
6. Record the change: once `oikos/ledger.py` is wired into deploy tooling
(Week 3), this is automatic; until then, note the change and outcome
in the relevant investigation/plan doc.
Docs-update checklist: update the service's `doc_page` if the change
alters its behavior, ingress route, or ownership; add a changelog entry
if the page has one.

View File

@@ -0,0 +1,25 @@
---
name: docs-lint
risk_class: read_only
inputs: [paths]
verification: "python3 .agents/skills/docs-lint/lint.py"
docs_update_checklist: []
---
# Docs lint
Check committed documentation against the mechanical rules in
[writing-style.md](../../shared/writing-style.md): banned vocabulary and broken relative markdown
links. Prose-voice rules are not machine-checkable — those stay a review responsibility.
Run from the repo root:
python3 .agents/skills/docs-lint/lint.py # default: knowledge/ .agents/ operations/ investigations/ plans/
python3 .agents/skills/docs-lint/lint.py archive/knowledge/containers/104-gitea.md
Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary
list mirrors `writing-style.md`; update both together if the standard changes.
> **Known baseline.** `archive/knowledge/archive/knowledge/containers/101-jellyfin.md` links into a sibling repo
> (`devops/homelab-authentik-admin`) that this checkout does not contain — expected, not a bug.
> Any other broken link is a real regression; investigate before dismissing it as baseline noise.

View File

@@ -0,0 +1,69 @@
#!/usr/bin/env python3
"""Lint committed docs against .agents/shared/writing-style.md.
Checks two mechanical rules:
1. Banned vocabulary (significance puffers, analytical verbs, poetic nouns,
promotional adjectives, opening crutches).
2. Broken relative markdown links.
Prose-voice rules are not machine-checkable; this covers the parts that are.
Run from the repo root: python3 .agents/skills/docs-lint/lint.py [paths...]
Exit 1 if any violation is found.
"""
import os, re, sys
BANNED = [
"pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament",
"paramount", "invaluable", "delve", "leverage", "utilize", "facilitate", "foster",
"showcase", "underscore", "streamline", "harness", "tapestry", "realm", "paradigm",
"nexus", "cornerstone", "robust", "seamless", "innovative", "cutting-edge",
"meticulous", "holistic", "comprehensive", "in today's world",
"it's worth noting", "it is important to note",
]
BAN_RE = re.compile(r'(?<![\w-])(' + "|".join(re.escape(w) for w in BANNED) + r')(?![\w-])', re.I)
LINK = re.compile(r'\]\(([^)]+)\)')
def iter_md(paths):
for p in paths:
if os.path.isfile(p) and p.endswith(".md"):
yield p
for root, dirs, files in os.walk(p):
dirs[:] = [d for d in dirs if d not in (".git", "node_modules")]
for f in files:
if f.endswith(".md"):
yield os.path.join(root, f)
def main(argv):
paths = argv or ["knowledge", ".agents", "operations", "investigations", "plans"]
violations = 0
# The style guide and this skill enumerate the banned words by definition.
ban_exempt = ("shared/writing-style.md", "skills/docs-lint/")
for f in sorted(set(iter_md(paths))):
check_banned = not any(x in f for x in ban_exempt)
fence = False
with open(f) as fh:
for ln, line in enumerate(fh, 1):
if line.lstrip().startswith("```"):
fence = not fence; continue
if fence:
continue
if check_banned:
for m in BAN_RE.finditer(line):
print(f"{f}:{ln}: banned word '{m.group(1)}'")
violations += 1
for m in LINK.finditer(line):
link = m.group(1)
if re.match(r'^(https?:|mailto:|#|/)', link):
continue
path = re.split(r'[#?]', link)[0]
if not path:
continue
tgt = os.path.normpath(os.path.join(os.path.dirname(f), path))
if not os.path.exists(tgt):
print(f"{f}:{ln}: broken link -> {link}")
violations += 1
print(f"\n{violations} violation(s)")
return 1 if violations else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))

View File

@@ -0,0 +1,34 @@
---
name: incident-investigation
risk_class: read_only
inputs: [symptom, affected_entity]
verification: "n/a — investigation produces a written record, not a state change"
docs_update_checklist: [investigations_entry]
---
# Incident investigation
Goal: understand what broke and why, before touching anything.
1. `homelab service <name> explain` (or `homelab node <name> relations`
if the affected entity is a host) — get the blast radius and doc
pointer first. Don't start pulling logs blind.
2. `homelab service <name> health` + `homelab service <name> log` (or
MCP `get_service_status` / `tail_log`) for the affected service.
3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
`authentik`, or the backend host itself)? `homelab node <name>
relations` shows "affected by" — check those first.
4. `homelab apt-audit` if the symptom looks like a dpkg/upgrade
interaction.
5. Check the change ledger for recent mutations to the affected entity
or anything upstream of it: `homelab service <name> history` (once
populated) or grep `ledger/*.jsonl`.
6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom,
timeline, root cause, fix applied, prevention. This is the durable
record; don't rely on chat history.
Docs-update checklist: always create the investigation entry. If the
root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,
or `backend` that didn't match reality — this happened during Week 1
kernel work, see the `authentik` backend fix), correct `inventory.yaml`
in the same session.

View File

@@ -0,0 +1,36 @@
---
name: lifecycle-activate-node
risk_class: config_mutation
inputs: [node_name]
verification: "homelab service <name> health (if it hosts a service); homelab doctor (if it's a client)"
docs_update_checklist: [doc_page_complete]
transition: "provisioning -> active"
---
# Lifecycle: activate a node
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml). Requires: age key
enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
ingress live if public, health check answering, doc page complete,
ledger entry.
1. If the node is a `homelab` client: finish enrollment per
[client-enrollment.md](../client-enrollment/SKILL.md) (`--finalize-pubkey`,
mesh join, `homelab doctor` green).
2. If it hosts a public service: add the `services:` entry in
`inventory.yaml` (backend, url, doc_page, config_repo, risk_notes —
see the Week-1 service contract fields) and wire the Caddy route in
`dtoro/caddy-conf`.
3. Confirm the health check answers: `homelab service <name> health` or
a direct `curl`.
4. Flip `state: provisioning``state: active` (or delete the `state:`
field — `active` is the default) in `inventory.yaml`.
5. Complete the doc page (stub → full narrative: role, specs, how it's
configured, dependencies).
6. Record the activation: `oikos/ledger.py append host:<name> activate
config_mutation --result ok` (or let the CLI wrapper do this once
Week 3's runbook automation lands).
Regenerate derived data: `python3 mcp/build_host_files.py && python3
inventory.yaml` so `inventory.yaml`, the topology diagram, and
the context card all reflect the new state.

View File

@@ -0,0 +1,35 @@
---
name: lifecycle-deprecate-node
risk_class: config_mutation
inputs: [node_name, replacement_node_or_reason]
verification: "homelab node <name> relations — 'affected by' must be empty before completing"
docs_update_checklist: [doc_page_deprecation_note]
transition: "active -> deprecated"
---
# Lifecycle: deprecate a node
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml): a node keeps running
but takes no new dependents. **Completion condition: zero remaining
inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a
suggestion; `seeds/policy.yaml` `lifecycle_overrides.deprecated.refuse`
lists `new-inbound-edges` as refused going forward.
1. Set `state: deprecated` on the node.
2. `homelab node <name> relations` — read `affected_by`. Every entry
there is something still relying on this node.
3. Migrate or retire each dependent one at a time (point its `backend`/
`config_repo`/ingress route elsewhere, or deprecate it too if it's
being retired alongside).
4. Re-run `homelab node <name> relations` after each dependent is moved.
The transition to `destroyed` is only safe once `affected_by` is
empty — check this every time, don't assume from memory.
5. Note the deprecation on the doc page: reason, replacement (if any),
date.
If step 2 shows dependents you didn't expect, stop and investigate
before proceeding — that's exactly the kind of drift the Week-3 detector
will catch automatically, but until then this manual check is the gate.
Next (once `affected_by` is empty):
[lifecycle-destroy-node.md](../lifecycle-destroy-node/SKILL.md).

View File

@@ -0,0 +1,42 @@
---
name: lifecycle-destroy-node
risk_class: destructive
inputs: [node_name]
verification: "homelab node <name> relations returns unknown-entity; pct list on the backend no longer shows it"
docs_update_checklist: [archaeology_entry, containers_index_update]
transition: "deprecated -> destroyed"
---
# Lifecycle: destroy a node
**Destructive.** Requires operator approval + typed confirmation phrase
per `seeds/policy.yaml`. Requires (ontology): backups verified, secrets
recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
ledger entry.
1. Confirm the node is `deprecated` with zero `affected_by` edges
(`homelab node <name> relations`) — do not skip this even if the
deprecation runbook was followed recently; state can drift.
2. If it's an enrolled client: `homelab client remove <name>` — revokes
the age key, re-keys SOPS, removes the inventory entry. This is
already destructive-class and confirmed in the CLI.
3. Remove any ingress route (Caddy config repo) and DNS record still
pointing at it.
4. Verify backups of anything on it are retained per policy before the
disk goes away (see `backs-up-to`).
5. Destroy the LXC/VM (`pct destroy` / `qm destroy`).
6. Move the `hosts.<name>:` block (if any inventory remnant survives
`client remove`, e.g. infra-only LXCs with no age key) into
inventory.yaml's `archaeology:` section: `pve_id`, `destroyed` date,
`reason`. Add a row to `containers/index.md` "Recently destroyed"
table (kept for human-readable browsing alongside the structured
data).
7. `oikos/ledger.py append host:<name> destroy destructive --result ok`.
8. Regenerate: `python3 mcp/build_host_files.py && python3
inventory.yaml` — the node drops out of `inventory.yaml` and
appears in the topology doc's archaeology table.
If the destroy fails partway (e.g. secrets revoked but pct destroy
errors), do not re-run step 2 — `client remove` is not idempotent
against a second revocation attempt on the issuance server. Finish the
remaining steps manually and note the partial state in an investigation.

View File

@@ -0,0 +1,39 @@
---
name: lifecycle-migrate-node
risk_class: config_mutation
inputs: [node_name, source_host, target_host]
verification: "homelab node <name> relations (re-check blast radius); homelab service <svc> health for every hosted service"
docs_update_checklist: [doc_page_migration_note, inventory_host_and_lan_ip]
transition: "active -> migrating -> active"
---
# Lifecycle: migrate a node
Modeled on the strong Phase 1+2 migration
([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
Requires (ontology): preflight + backup-verified before migrating;
post-verify + Caddy backends checked + mounts checked + docs updated
before returning to `active`.
1. `homelab change preflight <every service the node hosts>` — capture
current health as a baseline.
2. Verify backups are current for anything with data at rest on the
node (see `backs-up-to` edges once populated).
3. Set `state: migrating` in `inventory.yaml`.
4. Perform the migration (pct/qm move, or create-on-target +
data-copy + destroy-source, per the specific case).
5. Update `inventory.yaml`: new `host:`, `lan_ip`, `mesh` addresses for
the node; update every `services:` entry whose `backend` pointed at
it if the backend name itself changes (usually it doesn't — only the
`host:`/`lan_ip` on the guest entry moves).
6. Post-verify: re-run the Week-1 drift check by hand — confirm Caddy's
backend IP for each affected service matches the new `lan_ip`
(automatic in Week 3's drift detector), confirm mounts still resolve.
7. `homelab service <name> health` for every service the node hosts.
8. Set `state: active`. Add a migration note to the node's doc page
(old host/IP → new, date, phase reference) — this repo's convention
for every past migration (see `archive/knowledge/containers/101-jellyfin.md`,
`containers/129-house.md`).
Regenerate: `python3 mcp/build_host_files.py && python3
inventory.yaml`.

View File

@@ -0,0 +1,33 @@
---
name: lifecycle-provision-node
risk_class: config_mutation
inputs: [node_name, kind, storage_pool]
verification: "grep 'state: provisioning' inventory.yaml"
docs_update_checklist: [doc_page_stub]
transition: "planned -> provisioning"
---
# Lifecycle: provision a node
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml) `lifecycle.transitions`.
Policy note: `provisioning` nodes get a lifecycle override —
`config_mutation` actions downgrade to `reversible_low` because nothing
depends on the node yet (see `seeds/policy.yaml` `lifecycle_overrides`).
Requires (from ontology): inventory entry, IP reserved, storage pool
chosen, doc page stub.
1. Create the LXC/VM on its target Proxmox host (`pct create` /
`qm create`), choosing the storage pool deliberately — record it as
the `storage:` field once populated (Week 1 schema; not yet backfilled
for existing nodes).
2. Add the inventory entry: `homelab client add <name>` for anything that
will run the `homelab` CLI, or a direct `hosts.<name>:` block with
`state: provisioning`, `kind`, `host`, `pve_id`, `lan_ip` for
infra-only LXCs that won't self-enroll.
3. Stub the doc page (`containers/<pve_id>-<name>.md` or
`vms/<pve_id>-<name>.md`) — even a one-line "provisioning, see plan X"
is enough to satisfy the transition requirement.
4. Reserve the IP in DNS/DHCP notes if it's a fixed LAN address.
Next: [lifecycle-activate-node.md](../lifecycle-activate-node/SKILL.md).

View File

@@ -1,3 +1,10 @@
---
name: budget-import-from-csv
risk_class: config_mutation
inputs: [csv_file]
references: [containers/129-house.md]
---
# Runbook: Budget import from N26 CSV → Yuvomi
Distil a bank-export CSV into Yuvomi's Budget and Subscriptions modules using

View File

@@ -1,3 +1,9 @@
---
name: recover-dpkg-interrupted
risk_class: reversible_low
verification: "dpkg --audit (should be clean); apt-get check"
---
# Runbook — recover from dpkg-interrupted state
You're here because an apt run got killed mid-transaction and the target now
@@ -96,9 +102,9 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
## Related
- [Operations cheatsheet](commands.md)
- [Auto-deploy pipelines](../infrastructure/auto-deploy.md)
- [Hubris host page](../hosts/hubris.md)
- [Operations cheatsheet](../../operations/commands.md)
- [Auto-deploy pipelines](../../../archive/knowledge/infrastructure/auto-deploy.md)
- [Hubris host page](../../../archive/knowledge/hosts/hubris.md)
## Changelog

View File

@@ -0,0 +1,30 @@
---
name: service-health-check
risk_class: read_only
inputs: [service_name]
verification: "homelab service <name> health"
docs_update_checklist: []
---
# Service health check
Goal: determine whether a service is actually healthy, without ad-hoc SSH.
1. `homelab service <name> explain` — read the context card: backend,
blast radius, doc pointer, risk notes.
2. `homelab service <name> health` — live health probe (HTTP code against
the service's `url`/`endpoint`). Once the Week-3 scheduler ships, this
reads a cached snapshot by default; pass `--live` to force a fresh probe.
3. If unhealthy, `homelab service <name> log` (or MCP `tail_log`) for the
last 200 lines.
4. Cross-check blast radius: `homelab node <name> relations` — is this
entity's own backend host healthy? A downstream failure (e.g. `strong`
down) will show up here before the service's own logs explain anything.
5. If the fix is a restart: classify first (`seeds/policy.yaml`
`service-restart` is `reversible_low` unless the service has a
`service_overrides` entry, e.g. `caddy`/`dns` are `config_mutation`).
Unattended agents may act on `reversible_low` without approval.
Docs-update checklist: none for a pure health check. If the investigation
reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml` in
the same session.

11
.claude/launch.json Normal file
View File

@@ -0,0 +1,11 @@
{
"version": "0.0.1",
"configurations": [
{
"name": "web",
"runtimeExecutable": "npm",
"runtimeArgs": ["--prefix", "web", "run", "dev"],
"port": 5173
}
]
}

72
.gitea/workflows/ci.yml Normal file
View File

@@ -0,0 +1,72 @@
# Oikos CI (Gitea Actions). Gates the deploy webhook on a green run (plan M1).
# Mirrors `make lint`, `make test`, and the generated-code drift guard.
name: ci
on:
push:
branches: [main]
pull_request:
jobs:
build-test:
runs-on: ubuntu-latest
services:
postgres:
image: timescale/timescaledb:2.17.2-pg16
env:
POSTGRES_DB: oikos
POSTGRES_USER: oikos
POSTGRES_PASSWORD: oikos_dev
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U oikos"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
OIKOS_TEST_DATABASE_URL: postgres://oikos:oikos_dev@postgres:5432/oikos?sslmode=disable
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.26"
cache: true
- name: go vet
run: go vet ./...
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
with:
version: latest
args: --timeout 5m
continue-on-error: true # advisory until the lint baseline is clean
- name: govulncheck
run: |
go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck ./... || true # advisory
- name: generated code is up to date
run: make generate-check
- name: build
run: go build ./...
- name: test (race + coverage)
run: go test -race -covermode=atomic -coverprofile=coverage.out -timeout 300s ./...
- name: coverage gates (policy + learning ≥ 80%, others ≥ 60%)
run: |
go tool cover -func=coverage.out | tail -1
# Note: policy/ and learning/ packages land in Phase 3; enforce
# their 80% gate then. For now, report total coverage.
docker-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: docker build (verify image builds; no push)
run: docker build -f compose/oikos/Dockerfile -t oikos:ci .

25
.gitignore vendored
View File

@@ -1 +1,26 @@
.DS_Store
__pycache__/
*.pyc
# Regenerated every scheduler run; ephemeral health-probe cache.
oikos/state.json
# Compiled binaries (Go rewrite — bin/oikos, bin/nomos)
bin/oikos
bin/nomos
oikos/oikos
# Legacy Python oikos (superseded by cmd/oikos Go binary — Phase 1-6 rewrite).
# oikos/ kernel files are still imported by bin/homelab for operational CLI
# commands (ssh, pct, logs, restart, status, open, secret, client, sync, mcp).
# Remove oikos/* when bin/homelab is ported to Go.
backups/
.env
.infisical-credentials
# Web UI (Svelte 5) — build artifacts. Ignore built output but keep the
# .gitkeep placeholder so `//go:embed all:dist` (web/embed.go) compiles on a
# fresh checkout before the UI is built.
web/dist/*
!web/dist/.gitkeep
web/node_modules/

View File

@@ -1,14 +1,12 @@
# SOPS recipient rules.
#
# Each rule pins one or more age public keys as recipients for files matching
# the path_regex. The build_host_files.py generator doesn't read this file;
# `sops` does — to encrypt a new secret, run `sops -e secrets/<name>.yaml`
# from the repo root and SOPS will pick the matching rule below.
# SOPS is DR-fallback only — Infisical is the active secrets backend.
# Files live in archive/secrets-sops-backup/ for cold recovery.
# To encrypt a new DR secret: sops -e archive/secrets-sops-backup/<name>.yaml
#
# To grant a secret to a new client: add their age public key (from
# To grant a DR secret to a new client: add their age public key (from
# inventory.yaml `hosts.<name>.age_pubkey`) to the relevant rule below, then
# run `sops updatekeys secrets/<name>.yaml` to re-encrypt without rotating
# the ciphertext payload.
# run `sops updatekeys archive/secrets-sops-backup/<name>.yaml` to re-encrypt.
#
# To revoke: remove the recipient from the relevant rule and run
# `sops updatekeys` (this is what `homelab client remove` calls). Past
@@ -16,7 +14,7 @@
# underlying credential if compromise is suspected.
creation_rules:
- path_regex: ^secrets/hello\.yaml$
- path_regex: ^archive/secrets-sops-backup/hello\.yaml$
# The "hello" secret is encrypted to every enrolled client so the bootstrap
# decrypt test works for everyone. Add each new client's age_pubkey when
# they enrol; re-key with `sops updatekeys -y secrets/hello.yaml`.
@@ -25,9 +23,11 @@ creation_rules:
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/gitea-pat\.yaml$
- path_regex: archive/secrets-sops-backupgitea-pat\.yaml$
# Write-scoped Gitea PAT (dtoro user). Same recipient list as hello.yaml
# since every enrolled client should be able to push (homelab client
# add/remove, wiki edits, etc.).
@@ -36,19 +36,21 @@ creation_rules:
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/gitea-tokens\.yaml$
- path_regex: archive/secrets-sops-backupgitea-tokens\.yaml$
# Workstations only.
age: >-
# placeholder — fill with age_pubkey of: republic-laptop, mac-mini, ludo-mini, hubris
# placeholder — fill with age_pubkey of: republic-laptop, mac-mini, strong, hubris
- path_regex: ^secrets/webhook-hmacs\.yaml$
- path_regex: archive/secrets-sops-backupwebhook-hmacs\.yaml$
# LXCs that run a webhook receiver.
age: >-
# placeholder — fill with age_pubkey of: apps, caddy
- path_regex: ^secrets/turn-shared-secret\.yaml$
- path_regex: archive/secrets-sops-backupturn-shared-secret\.yaml$
# coturn TURN long-term-credential password. Consumed by hubris (which
# renders /etc/turnserver.conf + /opt/management.json on the VPS via
# `homelab render-vps-configs`). Other recipients are convenience for
@@ -58,9 +60,10 @@ creation_rules:
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/netbird-authentik-oidc\.yaml$
- path_regex: archive/secrets-sops-backupnetbird-authentik-oidc\.yaml$
# Authentik OIDC client secret for the netbird-dashboard provider.
# Consumed by hubris to render /opt/management.json on the VPS
# (PKCEAuthorizationFlow.ProviderConfig.ClientSecret).
@@ -69,9 +72,10 @@ creation_rules:
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
- path_regex: ^secrets/netbird-pat\.yaml$
- path_regex: archive/secrets-sops-backupnetbird-pat\.yaml$
# NetBird API Personal Access Token. Consumed by the dns-sync job on the
# `dns` LXC (107) to reconcile Technitium -> NetBird managed DNS zone.
# (When 107 is enrolled, add its age_pubkey here and updatekeys.)
@@ -81,19 +85,19 @@ creation_rules:
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
- path_regex: ^secrets/openrouter-api-key\.yaml$
# OpenRouter API key consumed by the `hermes` wrapper (bin/hermes) when
- path_regex: archive/secrets-sops-backupopenrouter-api-key\.yaml$
# OpenRouter API key consumed by the `nomos` wrapper (bin/nomos) when
# spawning a Goose session. Recipients are any host that should run a
# Nous-Hermes agent. Add a host's age_pubkey here, then
# Nomos agent. Add a host's age_pubkey here, then
# `sops updatekeys -y secrets/openrouter-api-key.yaml`.
# See operations/hermes-agent.md.
# See operations/nomos-agent.md.
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/yuvomi-api-token\.yaml$
- path_regex: archive/secrets-sops-backupyuvomi-api-token\.yaml$
# Named Bearer token for the Yuvomi REST API, consumed by yuvomi-mcp on
# LXC 129 (house).
age: >-
@@ -102,11 +106,29 @@ creation_rules:
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/hermes-house-users\.yaml$
- path_regex: archive/secrets-sops-backuphermes-house-users\.yaml$
# Signal number → Yuvomi user_id mapping (PII). Consumed by hermesd on LXC 129.
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: archive/secrets-sops-backupoikos-approval-hmac\.yaml$
# HMAC signing key for Oikos approval-grant tokens (oikos/approve.py).
# Recipients: apps (105, runs the approval engine alongside homelab-mcp)
# and hubris (admin/debug decrypt). See OIKOS.md "Approval engine".
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
- path_regex: archive/secrets-sops-backupoikos-console-deploy-secret\.yaml$
# Shared HMAC secret for the Gitea deploy webhook (id 14) ->
# oikos-console-deploy.service on apps (105). Generated + registered
# with Gitea before the apps-side install ran (see
# oikos/console/deploy/README.md "Status") — write this exact value
# into /etc/oikos-console-deploy/secret rather than letting
# webhook/install.sh generate a fresh one.
age: >-
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
# webhook noop 2026-05-20T18:16:57+02:00

File diff suppressed because one or more lines are too long

168
AGENTS.md
View File

@@ -4,11 +4,33 @@ You are running on a machine that is part of the **hubris** homelab. The full
context is in this checkout at `/opt/homelab-context/`. This file is the entry
point. Read it once at start, then keep working.
- **New client?** Read [CLIENTS.md](CLIENTS.md) first.
- **Developing on this repo?** Also read [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
The operating model — OODA loop, risk classes, approval rules, the ontology,
and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation,
classify the action against `seeds/policy.yaml`; when the class requires
approval, stop and ask the operator.
Agent-facing instruction lives under `.agents/`:
`.agents/shared/` holds the conventions every agent applies
([writing-style](.agents/shared/writing-style.md), [caveman](.agents/shared/caveman.md),
[page-templates](.agents/shared/page-templates.md), [llm-wiki](.agents/shared/llm-wiki.md)), and
`.agents/domains/` holds the per-domain schemas
([knowledge](.agents/domains/knowledge/schema.md), [operations](.agents/domains/operations/schema.md)).
**Source of truth:** The Postgres database is the single source of truth for all
structured data and knowledge. It is bootstrapped from `seeds/` at deploy time:
`seeds/ontology.yaml` (entity types, relationships, lifecycles), `seeds/inventory.yaml`
(hosts, services, entities), `seeds/policy.yaml` (risk classes, approval rules), and
`seeds/knowledge.yaml` (documents, investigations, runbooks). The old narrative wiki
is archived at `archive/knowledge/` for historical reference.
## 1. Who you are
Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read:
/opt/homelab-context/hosts/<your-hostname>.yaml
/opt/homelab-context/inventory.yaml
That file tells you your role, your peers, what's mounted, and what services
you host. If it does not exist, this client was not enrolled — stop and tell
@@ -17,75 +39,106 @@ the operator to run `homelab client add <hostname>` from an existing client.
## 2. The topology
- `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation
with their mesh addresses, roles, and service mappings. Treat this file as
authoritative; anything you read in narrative pages should agree with it.
- `/opt/homelab-context/infrastructure/mesh.md` — Tailscale → Netbird state.
Both meshes are accepted today; Netbird is preferred for new traffic.
- `/opt/homelab-context/infrastructure/dns.md` — split-horizon DNS via
dnsmasq on LXC 124. `*.hubris.network` resolves to 192.168.x.x on the LAN
and to mesh addresses off-LAN.
- `/opt/homelab-context/operations/commands.md` — the operator's cheatsheet
for pct, caddy, dnsmasq. Use these verbs when you take actions.
with their mesh addresses, roles, and service mappings. This is the seed file;
at runtime the DB is authoritative (query via MCP `get_entity` or the REST API).
- `/opt/homelab-context/seeds/knowledge.yaml` — full narrative knowledge: 36
documents, 6 investigations, 12 runbooks. Ingested into the DB on deploy.
- `/opt/homelab-context/.agents/operations/commands.md` — the operator's cheatsheet
for pct, caddy, DNS, and the Oikos command surface.
## 3. The MCP server
The homelab exposes a Model Context Protocol server with structured tools.
Endpoint is in `inventory.yaml` under `services.homelab_mcp.endpoint`.
Endpoint: `https://mcp.hubris.network/mcp`.
Available tools:
Available tools (21 total):
Context (pure read):
get_host(name), list_services(), find_service(name_or_role),
get_topology(), search_docs(query), get_page(path),
get_changelog(page, since?), whoami(hostname),
list_my_secrets(caller_pubkey?)
Context — observe + orient:
get_entity(slug), list_entities(type, limit, cursor),
get_relations(entity), get_blast_radius(entity),
search_knowledge(query) — ILIKE search over documents, investigations,
runbooks in the knowledge_entities table
get_patterns(status, entity_type, action) — learned action patterns
get_skills(status) — available automation skills
Management (read-only):
get_service_status(service), tail_log(service, lines=200),
list_lxcs(), get_lxc_state(lxc), ping_service(service)
Management — live state:
get_service_status(service_slug) — systemctl is-active on target host
tail_log(service_slug, lines=200) — journalctl
list_lxcs() — all LXC containers with ID, host, IP, health
get_lxc_state(lxc_slug) — pct status from Proxmox host
ping_service(service_slug) — HTTP reachability from entity_status
Mutations are **not** exposed via MCP. Use the `homelab` CLI for those, with
operator confirmation.
Oikos — decisions:
explain(service_slug) — compact context card (type, state, health, relations)
preflight(service_slug, action) — risk class + approval requirement
whoami(hostname) — entity record, peers, health for a client
get_change_history(entity_slug, limit=20) — last audit-log entries per entity
get_state_snapshot() — fleet health, disk, drift count
**When to prefer MCP over grepping the clone:** any time you need to resolve a
name to an address, look up service status, or search the wiki by content.
Grep is fine for browsing or when MCP is unreachable.
Operations — observe + act:
get_health_summary() — fleet health counts (healthy/degraded/down/unknown)
get_signal_history(entity_slug, state, limit) — open + recent signals
get_audit_trail(entity_id) — audit log filter + browse
get_agent_activity(limit) — agent self-inspection
query_metrics(hours=24) — time-series metric bucketed averages
get_trend(entity_id, days=7) — metric slope over time
get_event_timeline(severity, entity_slug, limit) — recent events
## 4. Wiki conventions
Execution — the single mutation path:
request_execution(target, action, params) — policy-gated.
reversible_low (restart, reload, pct_exec, apt audit) runs immediately;
config_mutation (systemctl enable/disable, apt upgrade) queues for operator
approval via Matrix, then executes on ✅.
get_execution_status(execution_id) — poll progress
- Pages live under `containers/`, `hosts/`, `vms/`, `infrastructure/`,
`investigations/`, `operations/`. Cross-link liberally; orphans are bugs.
- Every page ends with a `## Changelog` section, entries in reverse-chrono
order:
**When to prefer MCP over grepping the clone:** always for knowledge queries.
`search_knowledge("jellyfin hardware acceleration")` returns ranked results from
the DB with entity links. `get_entity_knowledge("lxc:jellyfin")` returns documents,
runbooks, and investigations in one call. Grep the clone only when MCP is
unreachable.
### YYYY-MM-DD — short title
one or two lines describing what changed and why.
## 4. Knowledge conventions
- Investigation files are dated and slugged: `YYYY-MM-DD-slug.md`.
- Live state takes precedence over docs. If you observe a discrepancy, update
the docs *in the same session* (per the same-session update rule).
All narrative knowledge (documents, investigations, runbooks) lives in the DB
(`knowledge_entities` table) and is seeded from `seeds/knowledge.yaml`. Agents
can register new knowledge via the API:
```
POST /api/v1/knowledge/{entity_slug}
{"title": "...", "content": "...", "tags": ["..."]}
```
The DB is the truth. The old wiki files are in `knowledge/wiki/` pending archive
per the DB-as-source-of-truth plan.
- **Runbook procedures** live as `runbook` entities in the DB and as SKILL.md
files under `.agents/skills/<name>/`. They carry `risk_class`, `procedure`
(JSON-schema-validated), and are linked to entity types via `applies_to_type`.
- **Investigations** are `investigation` entities linked to affected entities
via `about` edges.
- **Documents** are `document` entities linked to entities via `documents` edges.
They carry `at_glance` (structured attributes) and `changelog` (parsed entries).
- **Live state precedence.** If you observe a discrepancy between the docs and
running state, update the DB *in the same session* via the API. The `oikos export`
command regenerates `seeds/knowledge.yaml` for version control.
## 5. Acting on the homelab
- **Read state**: prefer MCP tools, then files, then shell. Examples:
`homelab whoami`, `homelab list`, `homelab status`, `homelab logs caddy`.
- **Cross-host actions** (caddy reload, pct exec, etc.): use the `homelab`
CLI — it resolves hostname → mesh address → ssh / pct path for you. Direct
SSH still works; the CLI just removes the lookup burden.
- **Secrets**: never hardcode. Call `homelab secret <name>` to decrypt on
demand using the per-client age key at `/etc/age/key.txt`. Secrets ARE
available in this system — `list_my_secrets()` (MCP) shows what you can
decrypt.
- **Mutations** (restart, edit configs, etc.): the `homelab` CLI's mutating
subcommands ask for confirmation. For ad-hoc work, SSH and edit directly —
but commit changes that touch tracked configs (caddy, gitea custom,
artifacto, mule-image, etc.; see `infrastructure/auto-deploy.md`).
- **Wiki updates**: same-session rule applies to any meaningful state change
this client makes.
- **Read state**: use MCP tools. Nomos (the AI agent) is the primary
operator interface — it has 21 MCP tools for observe/orient/decide/act.
- **Actions** (restart, logs, apt, pct exec): Nomos calls `request_execution`
via MCP. `reversible_low` actions execute immediately; `config_mutation`
and `destructive` actions are queued for operator approval via Matrix.
- **Secrets**: managed by Infisical (`oikos secret` subcommand for migration).
Never hardcode secrets — use env vars from `.env`.
- **Mutations** (restart, edit configs, etc.): classified against
`seeds/policy.yaml`. `reversible_low` actions auto-execute;
`config_mutation`/`destructive` actions require approval.
a valid `--approval-id` from `homelab approval request` — see OIKOS.md.
## 6. Communication mode
Read and apply `/opt/homelab-context/CAVEMAN.md` (if present). It defines the lab's
Read and apply `/opt/homelab-context/.agents/shared/caveman.md` (if present). It defines the lab's
terse-communication standard — drop filler, keep substance, use fragments.
## 7. Auto-setup mechanism
@@ -99,10 +152,10 @@ Currently auto-setup:
- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm
package, wrapper scripts, and compact output templates for token-efficient
CLI output. Wrapper at `~/bin/caveman_wrapper.sh`.
- **Hermes agent persona** (`tools/setup-hermes-soul.sh`): Provisions
`~/.hermes/SOUL.md` from `HERMES.md` on Hermes agents. This ensures every
Hermes agent follows the canonical homelab persona (token efficiency, source
of truth hierarchy). No-op on non-Hermes agents.
- **Nomos agent persona** (`tools/setup-nomos-soul.sh`): Provisions
`~/.nomos/SOUL.md` from `NOMOS.md` on Nomos agents. This ensures every
Nomos agent follows the canonical homelab persona (token efficiency, source
of truth hierarchy). No-op on non-Nomos agents.
To add a new auto-setup, create `tools/<name>.setup.sh` in the repo,
commit and push. All enrolled clients pick it up within 5 minutes.
@@ -111,5 +164,6 @@ To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer.
## 8. When in doubt
Run `homelab mcp search_docs <query>` or `homelab mcp get_host <name>`.
The clone is the fallback; MCP is the index.
Use MCP tools: `search_knowledge <query>` for narrative context,
`get_entity <slug>` for structured data, `get_entity_knowledge <slug>` for
everything linked to an entity. The clone is the fallback; MCP is the index.

139
CLIENTS.md Normal file
View File

@@ -0,0 +1,139 @@
# Oikos — client guide
If you are a homelab machine, this is what Oikos is and what it gives you.
If you are an AI agent running on this machine, also read [AGENTS.md](AGENTS.md).
## What is Oikos?
Oikos is the agentic operating system for the **hubris** Proxmox homelab. It
observes state, classifies actions against policy, executes approved
procedures, learns from outcomes, and notifies the operator. It runs as a
Docker stack on mac-mini and exposes an MCP server + REST API.
## What Oikos provides
| Capability | How you access it |
|------------|-------------------|
| Entity query (topology, blast radius) | MCP `get_entity`, `get_blast_radius` |
| Full-text knowledge search | MCP `search_knowledge` |
| Service status + logs | MCP `get_service_status`, `tail_log` |
| LXC inventory + state | MCP `list_lxcs`, `get_lxc_state` |
| Context cards | MCP `explain` |
| Pre-flight risk classification | MCP `preflight` |
| Change history | MCP `get_change_history` |
| State snapshot (health, disk, drift) | MCP `get_state_snapshot` |
| Secrets (Infisical) | REST API + `oikos secret` CLI |
| Approval tokens | Matrix via notifier |
All MCP tools are read-only. Mutations use the `homelab` CLI with operator
approval.
## Enrollment
Thin client model — no git clone, no sync timer. `bootstrap.sh` fetches only
the agent orientation files and tooling from the raw Gitea URL, then enrolls
via the Oikos API.
To enroll:
```bash
# Run from any machine with mesh connectivity
curl -fsSL https://git.hubris.network/dtoro/oikos/raw/main/bootstrap.sh | sudo bash
# Or with optional tooling:
curl ... | sudo bash -s -- --with-mcp # wire Claude's MCP config
curl ... | sudo bash -s -- --with-nomos # install Goose + Nomos
```
This calls `POST /api/v1/clients/enroll` on the Oikos API, which:
1. Validates the entity exists in DB (planned or provisioning state)
2. Validates mesh IP against expected subnets
3. Generates an age keypair and delivers it to the client
4. Creates an Infisical machine identity
5. Transitions the entity to provisioning state
## After enrollment
### What changes on your machine
- `/opt/homelab/` — agent orientation files (CLIENTS.md, AGENTS.md, OIKOS.md)
- `/opt/homelab/tools/` — tooling scripts (caveman, nomos-soul)
- `/etc/age/key.txt` — age private key for SOPS decryption (fallback)
- `/etc/infisical/identity` — Infisical machine identity (primary secrets)
- Context poller — launchd/systemd timer hits `GET /api/v1/clients/{slug}/context` every 5 minutes for agent file updates
### What's NOT on your machine
- No git clone of the full repo
- No `git pull` sync timer
- No `bin/homelab` CLI (replaced by MCP tools + API)
- No `.sops.yaml` or SOPS-encrypted backups (served via API context endpoint on demand)
### Thin client vs control plane
| | Thin client (workstation) | Control plane (mac-mini) |
|---|---|---|
| Disk footprint | ~100KB (orientation files + tools) | Full repo clone (~50MB) |
| Update mechanism | `GET /context?since=` poll | Git pull + post-pull.sh |
| Source of truth | DB via MCP | DB + local seeds + archive |
| Secrets access | Infisical (primary), age/SOPS served via API (fallback) | Infisical + local SOPS files |
### Your identity
Your identity in the homelab is defined in `inventory.yaml`. Run `hostname`
(Linux) or `scutil --get LocalHostName` (macOS), then look up your entry.
It tells you your role, what services you host, what's mounted, and your
mesh address.
### Source of truth hierarchy
1. **Postgres database** (runtime) — authoritative for entities, knowledge,
signals, ledger. Query via MCP or REST API.
2. **Context poller** — agent files and tooling fetched via API deltas every
5 minutes.
3. **Never guess.** If data is missing, query MCP. If MCP is down, grep the
local `/opt/homelab/` files.
## The context poller
Every 5 minutes, launchd (macOS) or systemd (Linux) hits:
```
GET /api/v1/clients/ws:{hostname}/context?since={last_timestamp}
```
The API returns which agent files, tools, and SOPS config changed since the
last poll. Only changed files are downloaded. This replaces the old
`git pull` with a lightweight HTTP delta.
To trigger manually: run `/opt/homelab/tools/context-poller.sh`.
## Making changes
- **Read state**: use MCP tools or the API
- **Mutate state** (restart, edit config, deploy): classify the action against
policy (query `preflight` MCP tool):
- `read_only` / `reversible_low` — execute directly
- `config_mutation` / `destructive` — request operator approval via
`POST /api/v1/entities/{slug}/activate` (or equivalent lifecycle endpoint)
- **Secrets**: use Infisical (primary) or SOPS (fallback). Never hardcode.
- **Knowledge**: if you observe a discrepancy between docs and live state,
update the DB via the API in the same session.
## MCP endpoint
```
https://mcp.hubris.network/mcp
```
Available tools are listed in [AGENTS.md](AGENTS.md#3-the-mcp-server).
## Communication mode
Apply [.agents/shared/caveman.md](.agents/shared/caveman.md) — terse,
fragment-heavy communication. Drop filler, keep substance.
## Related
- [AGENTS.md](AGENTS.md) — full agent orientation (read this first)
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, risk classes
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide for the Oikos codebase

View File

@@ -1,123 +1,155 @@
# Contributing to the Homelab Wiki
# Contributing to Oikos
## Voice
Developer guide for the Oikos codebase. If you are a homelab client consuming
Oikos, see [CLIENTS.md](CLIENTS.md). If you are an AI agent working on the
repo, see [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks.
## Dev setup
## Page templates
- **Go 1.26+** (see `go.mod` for pinned version)
- **PostgreSQL with TimescaleDB** — the compose stack includes `timescale/timescaledb:2.17.2-pg16`
- **Docker** for the full dev stack
### Container page (`containers/<id>-<name>.md`)
```bash
# Start dependencies (Postgres + Redis)
docker compose --profile dev up -d
```markdown
# <id> — `<name>`
# Run all tests
make test
One-sentence purpose.
# Run integration tests (needs compose Postgres)
make test-db
## At a glance
- **Hostname:** `<name>`
- **IP:** `192.168.8.x`
- **Privilege:** privileged | unprivileged
- **Resources:** N cores / M GiB RAM / D GiB rootfs
- **Mounts:** `/mnt/library``/mnt/library` (if any)
- **Public hostname:** `<sub>.hubris.network` (if proxied)
# Build the binary
make build
```
## Role
What it does, what it talks to.
## Project structure
## Service / port map
| Service | Listen | Notes |
```
cmd/oikos/ Single-binary entry point
cmd/nomos/ Nomos MCP client gateway
internal/ All Go packages
httpapi/ REST + MCP server (OpenAPI-generated)
mcp/ MCP tool implementations
db/ Connection pool, migrations, seeds, sqlc queries
scheduler/ Observe loop, probes, signals
actuator/ SSH execution
learning/ Pattern recognition, anomaly detection
notifier/ Matrix notifications, approval tokens
policy/ Risk classifier
secrets/ Infisical + SOPS backend
domain/ Core types: entities, approvals, signals, patterns
ontology/ Type hierarchy, relationship validation
knowledge/ Knowledge YAML seed ingestion
api/openapi.yaml API contract — the source of truth for endpoints
migrations/ Forward-only SQL migrations (TimescaleDB)
seeds/ Bootstrap YAML: ontology, inventory, policy, knowledge
compose/ Dockerfiles + Caddy config
scripts/ Deploy, watchdog, rollback
nomos/ Nomos config, persona, skills
.agents/ Agent instruction files + skills
plans/ Design documents
docs/adr/ Architecture decision records
```
## Storage / config paths
## Commands
## Auto-deploy
(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md)
| Command | Purpose |
|---------|---------|
| `make build` | Build `oikos` binary |
| `make test` | Run all tests with race detection |
| `make test-db` | Run integration tests against compose Postgres |
| `make lint` | `go vet` + `golangci-lint` |
| `make generate` | Regenerate OpenAPI + sqlc code |
| `make generate-check` | CI drift guard — fail if generated code is stale |
| `make migrate` | Apply DB migrations |
| `make seed` | Ingest seeds into DB |
| `make export` | Export DB state to YAML seeds |
| `make dev` | Start compose dev stack |
| `make clean` | Remove binary + test cache |
## Conventions
### APIs are OpenAPI-first
The REST API is defined in `api/openapi.yaml`. Server code is generated with
`oapi-codegen` into `internal/httpapi/gen/`. To add an endpoint:
1. Add the path + schema to `api/openapi.yaml`
2. Run `make generate`
3. Implement the handler in `internal/httpapi/impl.go`
4. Add tests in `internal/httpapi/api_test.go`
Never hand-edit `internal/httpapi/gen/api.gen.go`.
### Database access is sqlc-first
SQL queries live in `internal/db/queries/*.sql`. Go code is generated with
`sqlc` into `internal/db/sqlcgen/`. Config in `sqlc.yaml`.
- Queries target pgx/v5 with UUID + timestamptz overrides
- Never hand-edit generated sqlc code
### Migrations are forward-only
SQL migrations live in `migrations/` as `NNN_name.up.sql`. There are no down
migrations (see [ADR 0008](docs/adr/0008-forward-only-migrations.md)).
Migrations are idempotent where possible (`IF NOT EXISTS`, `DO $$` blocks).
To add a migration:
1. Create `migrations/NNN_name.up.sql` with the next sequence number
2. Write the DDL
3. Run `make migrate` to apply
### Seeds are DB-generated
`seeds/*.yaml` are the bootstrap files used by `oikos seed`. After making
changes via the API, run `make export` to regenerate the seed files. These
files are version-controlled and serve as DR fallback.
### Writing style
Follow [.agents/shared/writing-style.md](.agents/shared/writing-style.md).
Documentation is reference prose, not marketing. Banned vocabulary includes
"robust", "seamless", "leverage", "utilize", "delve", "cutting-edge".
### Risk classification
Every mutation is classified against `seeds/policy.yaml` before execution.
Four risk classes: `read_only`, `reversible_low`, `config_mutation`,
`destructive`. The classifier can only lower autonomy relative to policy,
never raise it. When in doubt, escalate.
## CI
Gitea Actions runs on push to `main` and pull requests (`ci.yml`):
1. `go vet` + `golangci-lint` + `govulncheck`
2. Generated code drift check (`make generate-check`)
3. Build (`go build ./...`)
4. Test with race detector + coverage
5. Docker build verification (no push)
Coverage gates: policy + learning packages ≥ 80%, others ≥ 60%.
## PR workflow
1. Create a branch from `main`
2. Make changes, write tests
3. Run `make lint test generate-check`
4. Commit with a message following: problem → change → risk → verification
5. Push to Gitea; CI gates PRs on green
## Secrets
Secrets are managed by Infisical (primary) with SOPS as DR fallback. Never
hardcode secrets. Use environment variables from `.env` for local dev.
The `.env` and `.infisical-credentials` files are gitignored.
## Related
- [Caddy](121-caddy.md) (if proxied)
- [DNS](../infrastructure/dns.md) (if has subdomain)
- [Authentik](124-authentik.md) (if SSO)
- ...
## Changelog
### YYYY-MM-DD — short title
What changed, why, link to investigation if any.
```
### Cross-cutting page (`infrastructure/<topic>.md`)
```markdown
# <Topic>
One-sentence summary.
## Why
Design rationale — what it replaces, what it solves.
## Components
Where it runs, what files matter.
## How to apply / use
Recipes.
## Gotchas
## Related
Links to nodes that host or depend on this.
## Changelog
```
### Plan (`plans/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Goal
What this change achieves and why.
## Current topology / state
Diagram or description of what exists now.
## Target topology / state
What it looks like after.
## Pre-flight checklist
## Step-by-step procedure
## Verification
## Post-migration
Changelog entries to write, index status to update.
```
### Investigation (`investigations/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Summary
1-3 sentences.
## Timeline
## Root cause
## Mitigations applied
## Open questions
```
## Linking discipline
- Every container page links to every cross-cutting page it participates in.
- Every cross-cutting page lists the nodes that participate.
- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages.
## Changelog hygiene
- Reverse-chronological (newest first).
- One entry per discrete change, even if you make several in one day.
- If a change spans nodes, repeat the entry on each affected page (different perspective is fine).
- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them.
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
- [CLIENTS.md](CLIENTS.md) — for homelab clients consuming Oikos
- [docs/adr/](docs/adr/) — architecture decision records

50
Makefile Normal file
View File

@@ -0,0 +1,50 @@
.PHONY: build test test-db lint generate generate-check dev migrate seed export clean tidy
BINARY := oikos
GO ?= go
build:
$(GO) build -o $(BINARY) -tags timetzdata ./cmd/oikos
test:
$(GO) test -race -cover ./...
# Integration tests against the compose Postgres (starts it if needed)
test-db:
docker compose up -d postgres
@sleep 3
OIKOS_TEST_DATABASE_URL="postgres://oikos:$${OIKOS_DB_PASSWORD:-oikos_dev}@localhost:5432/oikos?sslmode=disable" \
$(GO) test -race -count=1 ./internal/db/ ./internal/httpapi/ ./internal/mcp/
lint:
$(GO) vet ./...
@command -v golangci-lint >/dev/null 2>&1 && golangci-lint run || echo "golangci-lint not installed, skipping"
generate:
$(GO) run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.4.1 \
-config api/codegen.yaml api/openapi.yaml
$(GO) run github.com/sqlc-dev/sqlc/cmd/sqlc@v1.29.0 generate
# CI drift guard: regenerate and fail if the committed output changed.
generate-check: generate
@git diff --exit-code -- internal/httpapi/gen internal/db/sqlcgen \
|| (echo "generated code is stale — run 'make generate' and commit" && exit 1)
migrate:
$(GO) run ./cmd/oikos migrate
seed:
$(GO) run ./cmd/oikos seed
export:
$(GO) run ./cmd/oikos export
dev:
docker compose --profile dev up -d
clean:
rm -f $(BINARY)
$(GO) clean -testcache
tidy:
$(GO) mod tidy

180
README.md
View File

@@ -1,76 +1,136 @@
# Homelab Wiki — `hubris`
# Oikos
Living documentation for the **hubris** Proxmox homelab. Every node, every cross-cutting system, and every meaningful incident is its own page; pages are linked so you can start anywhere and walk the graph.
Agentic homelab operating system written in Go. Single binary (`cmd/oikos`),
Docker-deployed on mac-mini, with a standalone Nomos MCP agent gateway
(`cmd/nomos`). Manages the **hubris** Proxmox homelab autonomously — observes
state, classifies actions against policy, executes approved procedures over SSH,
learns from outcomes, and escalates when uncertain.
> Last refreshed against live state: **2026-04-28**.
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md).
**For client machines:** see [CLIENTS.md](CLIENTS.md).
**For developers:** see [CONTRIBUTING.md](CONTRIBUTING.md).
## Map
## Quick start
### Hosts
- [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77`
```bash
# Dev stack (postgres + api + scheduler + notifier)
docker compose --profile dev up -d
### VMs
- [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation)
- [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS
# Full stack (adds Nomos agent gateway)
docker compose --profile full up -d
### LXC containers
See the full table in [`containers/index.md`](containers/index.md). Quick links:
# Build standalone binary
go build -o bin/oikos -tags timetzdata ./cmd/oikos
| ID | Name | IP | Role |
| --- | ---------------- | --------------- | --------------------------------------------- |
| 101 | [jellyfin](containers/101-jellyfin.md) | 192.168.8.206 | Media server |
| 102 | [nfs-export](containers/102-nfs-export.md) | 192.168.8.200 | NFSv4 re-export of /mnt/library for ZimaOS |
| 103 | [paperless](containers/103-paperless.md) | 192.168.8.130 | Document mgmt |
| 104 | [gitea](containers/104-gitea.md) | 192.168.8.121 | Git server |
| 105 | [apps](containers/105-apps.md) | 192.168.8.205 | Docker host (Artifacto / PlantUML / Portainer / WriteFreely) |
| 114 | [nextcloud](containers/114-nextcloud.md) | 192.168.8.224 | Personal cloud |
| 118 | [elementsynapse](containers/118-elementsynapse.md) | 192.168.8.239 | Matrix Synapse |
| 119 | [sophia](containers/119-sophia.md) | 192.168.8.157 | Sophia |
| 120 | [mule-images](containers/120-mule-images.md) | 192.168.8.136 | Mule-image / mulita photos |
| 121 | [caddy](containers/121-caddy.md) | 192.168.8.175 | Reverse proxy |
| 122 | [arriman](containers/122-arriman.md) | 192.168.8.132 | Docker host (\*arr stack) |
| 124 | [authentik](containers/124-authentik.md) | 192.168.8.180 | SSO + split-horizon DNS |
| 130 | [grimmory](containers/130-grimmory.md) | 192.168.8.213 | Digital library (Grimmory — fork of Booklore) |
# Run all roles in one process (dev mode)
OIKOS_DATABASE_URL="postgres://oikos:oikos_dev@localhost:5432/oikos?sslmode=disable" \
go run ./cmd/oikos all
```
### Cross-cutting infrastructure
- [DNS — split-horizon](infrastructure/dns.md)
- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md)
- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md)
- [Monitoring — Hermes health watchdog](infrastructure/monitoring.md)
- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md)
- [SSH access](infrastructure/ssh-access.md)
- [Backups — restic on external drive (disabled)](infrastructure/backups.md)
- [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md)
- [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md)
- [Homelab context distribution](infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance
## Architecture
### Investigations
Time-stamped incident notes / experiments in [`investigations/`](investigations/index.md).
```
┌──────────────────────────────────┐
│ mac-mini (Docker) │
│ │
Workstation ─── │ nomos (8092) ──MCP── api (8090) │
(mesh) │ MCP gateway REST + MCP │
│ │
│ scheduler ── notifier ── postgres │
│ (observe) (Matrix) (Timescale)│
└──────────────────────────────────┘
```
### Operations
- [Command cheatsheet](operations/commands.md)
- [Agent enrollment](operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system
| Component | Port | Role |
|-----------|------|------|
| `oikos api` | 8090 | REST API + MCP server (15 tools) |
| `oikos scheduler` | — | Probe runner, signal lifecycle, metrics |
| `oikos notifier` | — | Approval tokens, Matrix alerts |
| `nomos serve` | 8092 | MCP client gateway, query routing |
## Conventions
## Phases
- **Each node page** ends with a `## Changelog` section. Reverse-chronological. Entry format:
```
### YYYY-MM-DD — short title
one or two lines on what changed and why.
```
- **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug.
- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog.
- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
- **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not.
| Phase | Status | Description |
|-------|--------|-------------|
| 1 — Ontology + DB | ✅ | TimescaleDB, migrations, seeds, blast_radius |
| 2 — API | ✅ | OpenAPI-first REST + MCP, auth, SSE, audit |
| 3 — Control loop | ✅ | Scheduler, actuator, learning, classifier, notifier |
| 4 — Nomos agent | ✅ | Standalone MCP client gateway, agent activity |
| 5 — Secrets | ✅ | Infisical backend + SOPS fallback, rotation runbooks |
| 6 — Deploy | ✅ | CI pipeline, cutover checklist, watchdog, rollback |
## Maintaining this wiki
Full plan: [plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md).
When you change a node:
1. Update the relevant page (config snapshot, ports, mounts).
2. Add a changelog entry at the bottom of that page.
3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry.
4. If it's an incident, add an entry to [`investigations/`](investigations/index.md).
## Operations
## See also
### API endpoints
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — page templates and tone
```bash
curl http://localhost:8090/api/v1/entities?type=service # fleet
curl http://localhost:8090/api/v1/health # fleet health
curl http://localhost:8090/api/v1/agent-activity # agent log
```
### Nomos queries
```bash
# Structured tool call
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
-d '{"tool":"get_blast_radius","args":{"entity_id":"service:authentik"}}'
# Natural language
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
-d '{"query":"what depends on authentik?"}'
```
### CLI
```bash
oikos migrate # apply DB migrations
oikos seed # ingest ontology/inventory/policy seeds
oikos export # export DB state to YAML
oikos api # serve REST + MCP
oikos scheduler # run observe loop
oikos notifier # run notification loop
oikos all # all roles in one process
oikos secret list # enumerate SOPS secrets
oikos secret migrate # SOPS → Infisical
```
## Repo layout
```
cmd/oikos/ Go entry point — single binary
cmd/nomos/ Nomos MCP client gateway
internal/ Go packages (httpapi, mcp, scheduler, actuator, learning,
notifier, policy, secrets, db, config, ontology, domain,
knowledge)
api/openapi.yaml API contract (OpenAPI 3.1)
migrations/ Forward-only SQL migrations (TimescaleDB)
seeds/ Bootstrap YAML (ontology, inventory, policy, knowledge)
compose/ Dockerfiles + Caddy config
scripts/ Deploy, watchdog, verification, rollback
nomos/ Nomos config, persona, skills
.agents/ Agent instruction files, shared conventions, skills
archive/ Historical reference (legacy wiki, plans, SOPS backups)
plans/ Design documents (active + done)
docs/adr/ Architecture decision records
```
## For agents
See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference:
- **Source of truth:** DB (runtime) then seeds (bootstrap). Old wiki is
archived at `archive/knowledge/` — use MCP `search_knowledge` instead.
- **Mutations:** classify against policy, request approval for
`destructive`/`config_mutation`
- **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode
## Related
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
- [CLIENTS.md](CLIENTS.md) — client onboarding guide
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide
- [plans/](plans/) — design documents and cutover checklist
- [docs/adr/](docs/adr/) — architecture decision records

8
api/codegen.yaml Normal file
View File

@@ -0,0 +1,8 @@
# oapi-codegen config — `make generate` regenerates internal/httpapi/gen.
package: gen
output: internal/httpapi/gen/api.gen.go
generate:
models: true
chi-server: true
strict-server: true
embedded-spec: true

3303
api/openapi.yaml Normal file

File diff suppressed because it is too large Load Diff

7
api/redocly.yaml Normal file
View File

@@ -0,0 +1,7 @@
# Redocly lint config for api/openapi.yaml (CI runs: redocly lint api/openapi.yaml)
extends:
- recommended
rules:
# Every operation declares `default` → RFC 9457 problem+json instead of
# enumerating each 4XX (plan R3-3); oapi-codegen handles `default` fine.
operation-4xx-response: off

View File

@@ -12,7 +12,7 @@
## Problem statement
The Technitium DHCP server on [CT 107](containers/107-dns.md) serves `192.168.8.100192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
The Technitium DHCP server on [CT 107](../../knowledge/wiki/containers/107-dns.md) serves `192.168.8.100192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
| Host | IP | Inside pool? |
|---|---|---|

View File

@@ -0,0 +1,358 @@
# Assessment: Which nodes can move to `strong`
## Executive summary
hubris is **memory-starved**: 28 GiB RAM, 71.9 GiB allocated across 18 LXC + 2 VM
(2.5× overcommit), 10 GiB swap in active use. strong sits **completely empty**
28 GiB RAM, 25 GiB free, 0 guests, 2.7 TiB unused storage. The single most
effective decongestion move is to shift guests off hubris onto strong.
This document assesses every guest for move-readiness, grouped by constraints
(library dependency, GPU, core-infra status), and proposes a phased migration
that does **not** require the physical library-SSD move (the blocker of the
original plan) — library access from strong is provided via NFS from hubris.
---
## Current resource state (live, 2026-07-05)
### hubris — overloaded
| Resource | Capacity | Allocated (all guests) | Actual use | Status |
|----------|----------|----------------------|------------|--------|
| RAM | 28 GiB | 70.2 GiB (2.5× overcommit) | 18 GiB used + 10 GiB swap | ⚠️ heavy swap pressure |
| CPU | 12 vCPU (6c/12t) | 43 vCPU (3.6× overcommit) | ~43% scaling MHz | OK (shares) |
| local-lvm | 856 GiB | 582 GiB allocated (29% thin) | — | OK |
| library (lvmthin) | 3.7 TiB | — | 1.2 TiB used (34%) | OK, 2.3 TiB free |
### strong — empty, ready
| Resource | Capacity | Used | Status |
|----------|----------|------|--------|
| RAM | 28 GiB | 2.3 GiB (host only) | 25 GiB free |
| CPU | 16 vCPU (8c/16t) | idle (0.08 load) | 100% free |
| local-lvm | 856 GiB | 0 | empty |
| ludo-lvm | 1.8 TiB | 0 | empty |
| Guests | — | 0 LXC, 0 VM | nothing running |
### Network topology constraint
```
Fritz!Box (192.168.178.1)
└── SODOLA 2.5G switch
├── hubris eno1 → vmbr1 (192.168.178.10) → vmbr0 (192.168.8.0/24)
│ └── all 20 guests on 192.168.8.x
└── strong vmbr0 (192.168.178.181)
└── no internal bridge yet, guests would be on 192.168.178.x
```
strong reaches `192.168.8.0/24` via the Fritz static route through hubris.
Guests on strong get `192.168.178.x` IPs unless we add an internal bridge
on strong (Phase 0 prerequisite — see below).
---
## Per-guest assessment
### Tier 1 — Move immediately (no library dependency, no core-infra)
These guests mount **no** `/mnt/library` and are not part of the core
infrastructure spine (caddy/dns/auth/mcp). They are the easiest wins.
| ID | Name | Cores | RAM | Library? | GPU? | Notes |
|----|------|-------|-----|----------|------|-------|
| 118 | elementsynapse | 2 | 4 GiB | ❌ | ❌ | Matrix homeserver. Public via Caddy (`matrix.hubris.network`). Only change: Caddy backend IP. **Easiest move in the fleet.** |
| 129 | house | 2 | 3 GiB | ❌ | ❌ | Yuvomi family planner (Docker). No public Caddy route yet (uses VPS traefik directly). Self-contained. |
**Combined RAM freed from hubris: 7 GiB.** No NFS, no library, no GPU.
### Tier 2 — Move with library NFS (high resource consumers)
These are the heaviest guests and the original migration plan's primary
targets. They mount `/mnt/library` and two use the iGPU. Moving them
requires an NFS export from hubris → strong (reverse of the original
plan's direction, since the physical SSD hasn't moved).
| ID | Name | Cores | RAM | Library? | GPU? | I/O profile | Notes |
|----|------|-------|-----|----------|------|-------------|-------|
| 120 | mule-images | 6 | 12 GiB | ✅ mp0 | ✅ iGPU | Write-heavy (photo processing) | **#1 RAM consumer.** Strong has Radeon 680M iGPU (VAAPI works). |
| 122 | arriman | 4 | 8 GiB | ✅ mp0 | ❌ | Write-heavy (downloads) | *arr stack + qbit + sab. Mounts library for download writes. |
| 101 | jellyfin | 4 | 8 GiB | ✅ mp0 | ✅ iGPU | Read-heavy sequential | Media streaming + transcode. Strong 680M handles VAAPI. |
**Combined RAM freed: 28 GiB.** This alone would eliminate hubris's swap
pressure entirely.
### Tier 3 — Could move, low urgency
| ID | Name | Cores | RAM | Library? | Notes |
|----|------|-------|-----|----------|-------|
| 130 | grimmory | 1 | 2 GiB | ✅ mp0 | Book library (Docker). Migrated from apps LXC recently. |
| 131 | teddycloud | 1 | 1 GiB | ✅ mp0 | New (not in inventory.yaml yet). |
| 132 | rclone | 1 | 2 GiB | ✅ mp0 (ro) | Backup container. Read-only library mount. |
| 128 | trmnl | 1 | 768 MiB | ❌ | TRMNL middleware. No library. Could move but tiny. |
| 119 | sophia | 2 | 1 GiB | ✅ mp0 | Workshop. Light use. |
### Stay on hubris (core infrastructure)
| ID | Name | Cores | RAM | Why it stays |
|----|------|-------|-----|--------------|
| 121 | caddy | 1 | 512 MiB | Reverse proxy — terminates all `*.hubris.network`. Must stay on hubris for LAN-side reachability. **Needs backend IP updates** when guests move. |
| 107 | dns | 1 | 1 GiB | Technitium DNS, split-horizon. Core. |
| 106 | auth-outpost | 1 | 512 MiB | Authentik SSO enforcement. Core. |
| 105 | apps | 2 | 4 GiB | homelab MCP + secrets-issuance + artifacto. Core infra. Mounts library. |
| 104 | gitea | 1 | 1 GiB | Git server. NFS would hurt git lock/stat perf. Mounts library (bare repos). |
| 103 | paperless | 2 | 3 GiB | Document archive. Moderate I/O, OCR writes. Mounts library. |
| 102 | nfs-export | 1 | 512 MiB | Exports library to zimaos via NFS. Must stay with the physical library. |
| 100 | zimaos | 4 | 8 GiB (VM) | NAS frontend eval. Already NFS-mounts library from 102. |
| 108 | haos | 2 | 4 GiB (VM) | Home Assistant OS. Hardware access, low latency. |
---
## Constraints & prerequisites
### 1. Network — strong needs an internal bridge (Phase 0)
strong currently has only `vmbr0` on `192.168.178.0/24`. Guests created there
get household-LAN IPs, not homelab-subnet IPs. Two options:
- **Option A (recommended):** Add `vmbr1` on strong as a portless internal
bridge with a `192.168.8.x/24` address (e.g. `192.168.8.3`). Route between
strong's `vmbr0` and `vmbr1` the same way hubris does. Guests go on `vmbr1`
and get `192.168.8.x` IPs — transparent to Caddy, DNS, and inter-LXC refs.
Requires adding a static route on Fritz (or relying on hubris's existing
route — strong would need IP forwarding + a route to 192.168.8.0/24 via vmbr1).
- **Option B (simpler, messier):** Put guests on `192.168.178.x` directly.
Caddy can still reach them (hubris routes to 192.168.178.0/24). But DNS
records, inter-LXC references, and firewall rules all assume `192.168.8.x`.
More config churn per guest.
### 2. Storage — rootfs migration (no shared storage)
`local-lvm` is per-node (not shared). Moving an LXC requires either:
- `vzdump` → restore on strong (clean, but needs temp disk space + downtime)
- `rsync` the rootfs to a new LXC on strong (faster for large rootfs like 120's 100G)
- `pct migrate` only works with shared storage — **not applicable here**
For VMs (100, 108): `qm migrate` also needs shared storage. Not moving VMs.
### 3. Library access — NFS from hubris to strong
Since the physical library SSD is still on hubris, strong's guests that need
`/mnt/library` must NFS-mount it from hubris. Options:
- **Export from hubris host directly** (simplest): add `/mnt/library` to
`/etc/exports` on hubris with the same squash params as LXC 102
(`rw,all_squash,anonuid=33,anongid=10000,no_subtree_check`). Mount on strong
at `/mnt/library`. Strong's guests bind-mount it just like hubris's guests do.
- **Use existing nfs-export LXC 102**: strong NFS-mounts from `192.168.8.200`
(LXC 102). This already has the right squash config. Less host-level change.
**This is the path of least resistance.**
### 4. GPU — iGPU passthrough on strong
strong has a Ryzen 7 PRO 6850U with Radeon 680M iGPU. For jellyfin (VAAPI
transcoding) and mule-images (photo processing), we need:
- `/dev/dri/renderD128` passed to the LXC (`lxc.cgroup2.devices.allow` +
`lxc.mount.entry` or Proxmox's `dev0:` passthrough)
- `video` / `render` group membership inside the container
- Confirm `amdgpu` driver loads on strong's host kernel (it should — same APU family)
### 5. Quorum — 2-node cluster, no QDevice
Moving guests to strong does NOT fix the quorum issue but **reduces blast
radius**: if hubris reboots (its known thermal instability), the guests on
strong keep running independently. Consider adding a QDevice as a separate
follow-up — it's orthogonal to this migration.
---
## Revised migration phases
The original plan's NFS-over-LAN approach has been superseded. Instead,
**media library data moves to ludo-lvm** on strong so migrated guests access
it as a local ext4 mount. Data is split by origin:
```
hubris (stays): library SSD (3.7T, 1.2T used)
└── /mnt/library/{documents,images,cloud,homecloud,notes,repos,sophia}
↑ user-generated content (docs, photos, cloud sync, notes, repos, workshop)
strong (moves): ludo-lvm (1.8T, 0 used at start)
└── /mnt/media_local ← 1.5T thin volume
└── {downloads,movies,music,tv,anime,books}
↑ non-user-generated content (media arr stack, book library)
```
| Category | Stays on hubris | Moves to strong |
|----------|----------------|-----------------|
| Media | — | downloads (25G), movies (51G), music (29G), tv (30G), anime (206G) |
| Books | — | books (2.6G) |
| Docs/Photos | documents (249M), images (4K) | — |
| Cloud sync | cloud (287G), homecloud (367G) | — |
| Personal | notes (6.7M), repos (84M), sophia (151G) | — |
| **Total** | **~805G** | **~344G** |
ludo-lvm (1.8T) fits all media + books with ~1.15T headroom for growth.
hubris library SSD (3.7T, 1.2T used) retains the user-generated content.
Both sides keep their data local — no cross-node NFS needed for daily I/O.
---
### Phase 2a — Prepare ludo-lvm on strong
1. Create a ext4 filesystem on ludo-lvm for media:
```bash
lvcreate -n media -L 1.5T ludo-lvm
mkfs.ext4 /dev/ludo-lvm/media
```
2. Mount at `/mnt/media_local` on strong, add to `/etc/fstab`
3. rsync media directories from hubris → strong:
```bash
rsync -av --progress /mnt/library/{movies,tv,anime,downloads,music,books} strong:/mnt/media_local/
```
### Phase 2b — Migrate arriman (122) to strong
1. Stop arriman on hubris, dump rootfs (24G)
2. Restore on strong with IP `192.168.8.245/28` on vmbr1
3. Mount `/mnt/media_local` → `/mnt/library` via mp0 (downloads land locally)
4. Update Caddy: jellyseerr/qbit/sab backends → new IP
5. Update inventory.yaml
### Phase 2c — Migrate jellyfin (101) to strong
1. Stop jellyfin on hubris, dump rootfs (16G)
2. Restore on strong with IP `192.168.8.246/28` on vmbr1
3. Pass `/dev/dri/renderD128` + `/dev/dri/card0` (Radeon 680M + RX 7600)
4. Mount `/mnt/media_local` → `/mnt/library` via mp0 (media reads locally)
5. Update Caddy: `media.hubris.network` → new IP
6. Reinstall `sso-inject.js` in web dir (lost on every apt upgrade)
7. Test VAAPI transcoding, SSO login, media playback
### Phase 2d — Migrate grimmory (130) to strong
1. Stop grimmory on hubris, dump rootfs (16G)
2. Restore on strong with IP `192.168.8.247/28` on vmbr1
3. Mount `/mnt/media_local` → `/mnt/library` via mp0 (books read locally)
4. Update Caddy: `books.hubris.network` → new IP
5. Update inventory.yaml
6. Test: book browsing, calibre-web access
### No NFS export needed
With the data split by origin, hubris guests that only need user-generated
content (documents, images, cloud, repos, sophia) still access them from the
original library SSD — no cross-node NFS required. The two sides are
independent.
**Result after Phase 2: hubris frees 26 GiB RAM (4 migrated guests) + 344G of
library I/O burden. Strong becomes the media/books powerhouse.**
---
### Phase 3 — Migrate mule-images (120) to strong
Move photo management (12 GiB RAM, 6 cores, iGPU) last because it needs:
- `/mnt/library` access (now NFS from strong — already set up in Phase 2d)
- `/dev/dri/renderD128` (Radeon 680M — confirm VAAPI compatibility first)
Steps:
1. Stop mule-images on hubris, rsync the 100G rootfs to strong (faster than vzdump)
2. Restore on strong with IP on vmbr1
3. Pass Radeon 680M iGPU
4. Reconfigure library paths → `/mnt/media_local` (or keep NFS mount)
5. Update Caddy: `photos.hubris.network` → new IP
6. Test photo import + processing pipeline
---
### Phase 4 — Tier 3 moves (optional)
Migrate grimmory (130), teddycloud (131), rclone (132), trmnl (128), sophia (119)
as needed — each frees 12 GiB. Not urgent; do when convenient.
---
### Phase 5 — Follow-up
- **QDevice**: add a tiebreaker for 2-node quorum
- **Gaming VM**: strong's 6850U has enough cores alongside migrated LXCs
- **Hubris library cleanup**: after all guests are confirmed working, decide
whether to keep the original library SSD as backup or repurpose it
---
## Resource math after Phase 3 (all Tier 1 + 2 moved)
| | hubris | strong |
|---|--------|--------|
| Guests | 11 LXC + 2 VM | 5 LXC |
| RAM allocated | ~25 GiB | ~45 GiB |
| RAM capacity | 28 GiB | 28 GiB |
| Overcommit | 0.9× (under-committed) | 1.6× (manageable) |
| Library disk | Local ext4 (3.7T) → NFS client | Local ext4 on ludo-lvm (1.8T) |
| GPU | Radeon 760M (idle) | Radeon 680M (jellyfin + mule-images) |
strong becomes the media/library powerhouse. hubris becomes a lean core-infra
node (DNS, auth, git, docs, caddy, HA).
---
---
## Risk register
| Risk | Impact | Mitigation |
|------|--------|------------|
| NFS latency for library reads (jellyfin, arriman) | Media playback stutter, slow downloads | Test iperf between strong↔hubris first. If 2.5G link, NFS throughput is fine (~1 Gbit/s). |
| GPU passthrough on strong (680M vs 760M) | Transcode quality/compat differences | Both are AMD VAAPI — same driver stack. Test `vainfo` inside LXC before going live. |
| Caddy backend IP churn | Service outage if IP wrong | Update Caddyfile in git repo (caddy-conf), test each route before destroying old LXC. |
| vzdump/restore downtime | Service unavailable during migration | Schedule off-hours. Use rsync for large rootfs (120's 100G) to minimize freeze window. |
| 2-node quorum still fragile | If hubris goes down, strong /etc/pve goes read-only | Guests keep running. Add QDevice as follow-up. |
| Library data integrity during NFS transition | Permission drift | NFS `all_squash,anonuid=33,anongid=10000` matches existing LXC 102 config. Verify with `ls -la /mnt/library` after mount. |
---
## Open questions for operator
1. **Internal bridge on strong**: proceed with `vmbr1` on `192.168.8.3/24`
(Option A), or use `192.168.178.x` guest IPs (Option B)?
2. **Migration method**: `vzdump`/restore (clean, downtime) vs `rsync` rootfs
(faster for large disks, needs manual config copy)?
3. **Phase 1 priority**: move elementsynapse + house first (quick wins), or
go straight to Phase 2 (mule-images/jellyfin/arriman) for maximum relief?
4. **Should we add a QDevice now** before moving anything, to protect
management plane during the migration?
---
## Changelog
### 2026-07-05 — Phase 2d complete (grimmory migrated; media NFS to zimaos)
grimmory (130) → 192.168.8.247 on strong. Rsync'd /books (2.6G) to ludo-lvm.
LXC 102 (nfs-export) now mounts strong's NFS at /mnt/media and exports it as
a second share alongside /mnt/library. Zimaos mounts both: /media/library
(hubris user-generated) and /media/media (strong media+books).
See hosts/strong.md changelog.
### 2026-07-05 — Phase 2 complete (arriman + jellyfin migrated; library on ludo-lvm)
arriman (122) → 192.168.8.245, jellyfin (101) → 192.168.8.246. Created 1.5T
thin volume on ludo-lvm, rsync'd 363G of media data. Both containers use local
ext4 mount — no NFS. Jellyfin has 680M + RX 7600 GPU passthrough.
Caddy backends updated. See hosts/strong.md changelog.
### 2026-07-05 — Phase 1 complete (elementsynapse + house migrated to strong)
Both Tier 1 guests moved: elementsynapse (118) → 192.168.8.242, house (129) → 192.168.8.244.
Strong now has vmbr1 at 192.168.8.241/28. Hubris has proxy ARP + /32 routes for strong
guest range. DHCP scope narrowed to 192.168.8.100-239 to avoid conflicts.
Teddycloud (LXC 131) given static IP 192.168.8.150 due to IP conflict with
previous DHCP allocation at 192.168.8.243.
See hosts/strong.md changelog for full steps.
### 2026-07-05 — assessment created
Built from live `pct config` + `pvesm status` + `free -h` data pulled from
both nodes. Supersedes the storage-migration framing of the original
library-SSD plan — this assessment treats the SSD move as optional and
focuses on guest relocation via NFS.

View File

@@ -0,0 +1,33 @@
# Glossary
Terms and abbreviations used throughout the homelab wiki.
| Term | Meaning |
|------|---------|
| **Authentik** | SSO/identity provider. Core runs on the VPS; forward-auth outpost at LXC 106 on hubris |
| **Caddy** | Reverse proxy (LXC 121). Terminates TLS for every `*.hubris.network` hostname |
| **Caveman** | Terse communication standard for agent responses — no filler, keep substance |
| **Forward-auth** | Caddy snippet that delegates authentication to an Authentik outpost. Protects web UIs like qBit, SABnzbd |
| **Gitea** | Git server at `git.hubris.network`. Hosts all tracked config repos |
| **Gluetun** | WireGuard VPN sidecar on arriman. All \*arr traffic routes through it |
| **HAOS** | Home Assistant Operating System. VM 108 on hubris |
| **Hubris** | Primary Proxmox VE node (GMKtec NucBox M6 Ultra). PVE hostname, cluster member 1 |
| **LXC** | Linux Container (Proxmox). VM-like isolation without a full OS kernel |
| **LVM-thin** | Thin-provisioned logical volume manager. Used for all container/VM storage |
| **MCP** | Model Context Protocol (MCP server at `mcp.hubris.network`). Structured tools for agents to query homelab state |
| **Mesh** | Overlay VPN for off-LAN connectivity. Netbird is current; Tailscale is legacy |
| **Netbird** | Preferred mesh VPN. VPS hosts the management plane; all homelab nodes are members |
| **OIDC** | OpenID Connect. Protocol used by Authentik for SSO login flows |
| **Oikos** | Agent operating model ([.agents/OIKOS.md](../.agents/OIKOS.md)). OODA loop, risk classes, policy, ontology |
| **PVE** | Proxmox Virtual Environment — the hypervisor on both hubris and strong |
| **SOPS** | `sops` — Mozilla SOPS. Encrypts secrets with age keys so they live in the git repo |
| **Strong** | Secondary Proxmox VE node. Cluster member 2 (hostname `strong`, nickname ludo/ludo-mini) |
| **Traefik** | Reverse proxy on IONOS VPS. Serves `*.hubris.network` to the public internet |
| **VAAPI** | Video Acceleration API. Intel/AMD GPU-based hardware transcode for Jellyfin |
| **VPS** | Virtual Private Server at IONOS (`82.165.190.79`). Runs Authentik core + Netbird management |
| **\\*arr** | Media automation suite: Sonarr (TV), Radarr (movies), Lidarr (music), Prowlarr (indexer), Bazarr (subtitles), Readarr (books — not in use) |
## See also
- [Infrastructure index](wiki/infrastructure/index.md) — cross-cutting systems each with their own doc page
- [OIKOS operating model](../.agents/OIKOS.md) — agent policy, risk classes, lifecycle

View File

@@ -0,0 +1,160 @@
# 101 — `jellyfin`
Media server: serves the movies / TV / anime / music libraries from `/mnt/media_local` to LAN clients. Hardware transcoding via AMD Radeon 680M + RX 7600 VAAPI. Authentik SSO via OIDC.
## At a glance
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **IP:** `192.168.8.246`
- **Privilege:** privileged (recreated on strong as priv)
- **Resources:** 4 cores / 8 GiB RAM / 1 GiB swap / 16 GiB rootfs
- **GPU:** `/dev/dri/renderD128` + `/dev/dri/card0` (AMD Radeon 680M iGPU + RX 7600 dGPU) passed via `dev0` / `dev1` in LXC config
- **Mounts:** `/mnt/media_local``/mnt/library`
- **Public hostname:** [`media.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8096`
- **Version:** Jellyfin 10.11.11 (apt package, Ubuntu 24.04 noble repo)
- **FFmpeg:** jellyfin-ffmpeg7 7.1.4
## Service / port map
| Service | Listen | Notes |
| -------- | ------ | ----- |
| jellyfin | `:8096` | HTTP (caddy terminates TLS) |
## Hardware acceleration (VAAPI)
GPU is passed through to the LXC via `dev0: /dev/dri/renderD128,gid=993` and
`dev1: /dev/dri/card0,gid=44` in
`/etc/pve/lxc/101.conf` (strong). The `jellyfin` user is in the `render` (GID 993) and
`video` groups inside the container.
| GPU | Model | Role |
|-----|-------|------|
| Radeon 680M | iGPU (AMD Ryzen 7 PRO 6850U) | Primary VAAPI encoder/decoder |
| RX 7600 | dGPU (add-in) | Secondary transcode, HEVC/AV1 encoding |
Encoding settings (`/etc/jellyfin/encoding.xml`):
- `HardwareAccelerationType`: `vaapi`
- `VaapiDevice`: `/dev/dri/renderD128`
- `EnableHardwareEncoding`: `true`
- `AllowHevcEncoding`: `true`
- `AllowAv1Encoding`: `true`
- `EnableTonemapping`: `true`
- `HardwareDecodingCodecs`: h264, hevc, vc1, vp9, av1
- `EnableThrottling`: `true`
- `EnableSegmentDeletion`: `true`
Trickplay (`/etc/jellyfin/system.xml`):
- `EnableHwAcceleration`: `true`
- `EnableHwEncoding`: `true`
## Authentik SSO (OIDC)
Jellyfin uses the [SSO-Auth plugin](https://github.com/9p4/jellyfin-plugin-sso)
v4.0.0.4 for Authentik OIDC login. No Caddy forward-auth gate — the SSO plugin
handles auth directly via OIDC redirect flow.
### Architecture
```
User → media.hubris.network → Caddy (TLS, no forward-auth) → Jellyfin :8096
Login page with "Sign in with Authentik" button
↓ (click)
/sso/OID/start/Authentik
↓ (302 redirect)
auth.hubris.network OIDC
↓ (login)
/sso/OID/redirect/Authentik?code=...&state=...
Jellyfin SSO plugin validates token → logged in
```
### Components
1. **SSO-Auth plugin** — installed at `/var/lib/jellyfin/plugins/SSO-Auth_4.0.0.4/`
- Config: `/var/lib/jellyfin/plugins/configurations/SSO-Auth.xml`
- Provider name: `Authentik`
- OIDC endpoint: `https://auth.hubris.network/application/o/jellyfin/`
- `SchemeOverride`: `https` (required — without it, plugin generates
`http://` redirect URIs that Authentik rejects)
- `EnableAuthorization`: `false` (prevents plugin from overwriting admin
permissions on each SSO login — see
[jellyfin-sso-plugin](../../../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md))
- `OidScopes`: `["email"]` (openid+profile added by default by the plugin;
must be non-null or `OidChallenge()` throws `ArgumentNullException`)
2. **Authentik OIDC provider**`Provider for Jellyfin` (PK 6)
- Client ID: `vt61t5Y2ZVtN6l3QjitkBvwUJjFKvSyl4TDBXcJx`
- Redirect URI: `https://media.hubris.network/sso/OID/redirect/Authentik`
- Application slug: `jellyfin`
3. **SSO button injection**`/usr/share/jellyfin/web/sso-inject.js`
- Injected via `<script defer src="sso-inject.js?v=3">` in `index.html`
- Polls for `.readOnlyContent` div on the login page, adds "Sign in with
Authentik" button linking to `/sso/OID/start/Authentik`
- Cache-busted with `?v=N` parameter (bump on changes)
- **Lost on apt upgrade** — re-inject the script tag and copy the JS file
after `apt-get upgrade jellyfin*`
4. **Caddy config** — no forward-auth gate for media.hubris.network:
```caddy
media.hubris.network {
tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
reverse_proxy 192.168.8.206:8096
}
```
### Known issues / pitfalls
- **`OidScopes` must be non-null** — if the field is missing from the plugin
config XML, `OidChallenge()` throws `System.ArgumentNullException`. Always
include `OidScopes` in the provider config (even if empty array).
- **`SchemeOverride: "https"` is required** — without it, the plugin generates
`http://` redirect URIs (from the internal HTTP listener). Authentik rejects
them with "Redirect URI Error".
- **SSO button JS is not served by the plugin** — the `__plugin/SSO-Auth.js`
endpoint returns 404 on Jellyfin 10.11.x when the plugin is installed
manually (not via Jellyfin's plugin manager). The `sso-inject.js` workaround
in `index.html` is the fallback.
- **No Caddy forward-auth gate** — the SSO plugin's OIDC redirect flow is
incompatible with Caddy's `import authentik` forward-auth. If both are
enabled, the forward-auth intercepts the OIDC callback and breaks the flow.
Use one or the other, not both. SSO plugin (OIDC redirect) is preferred.
- **API key for setup** — a temp API key can be inserted directly into the
`ApiKeys` SQLite table for automated configuration:
```sql
INSERT INTO ApiKeys VALUES (1, '2026-07-04', '2026-07-04', 'setup', 'jf-setup-key-...');
```
## Permissions
Member of the [media GID 10000](../infrastructure/media-permissions.md) standard. Service user `jellyfin` is in the `media` group inside the container; `/mnt/media_local` on strong's ludo-lvm is owned `root:media` with mode `2775`.
## Related
- [Caddy reverse proxy](121-caddy.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads
- [DNS split-horizon](../infrastructure/dns.md)
- [Authentik admin](../../../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config
## Changelog
### 2026-07-06 — wiki: IP, host, GPU, mount path updated for strong migration
Updated At-a-glance: IP 206→246, host hubris→strong, mount /mnt/library→/mnt/media_local, GPU Radeon 760M→680M+RX7600, privilege unpriv→priv. Permissions section updated. Changelog entry in 122-arriman.md updated similarly.
### 2026-07-04 — VAAPI hardware acceleration + Authentik SSO + resource bump
- Upgraded Jellyfin 10.11.8 → 10.11.11 (purge + reinstall to fix DB migration bug)
- Enabled VAAPI hardware acceleration (Radeon 760M): h264/hevc/vc1/vp9/av1 decode + encode
- Bumped resources: 2→4 cores, 4→8 GiB RAM, 512→1024 MiB swap
- Enabled trickplay HW acceleration + throttling + segment deletion
- Installed SSO-Auth plugin v4.0.0.4 with Authentik OIDC provider
- Configured `SchemeOverride: https`, `OidScopes: ["email"]`, `EnableAuthorization: false`
- Injected SSO button via `sso-inject.js` in web `index.html`
- Removed Caddy forward-auth gate (incompatible with SSO plugin OIDC flow)
- **Database was wiped** during cache relocation attempt — no LVM snapshot
existed. All watch states, user accounts, and library configs lost.
Libraries re-added via setup wizard.
### 2026-04-28 — wiki entry created
Initial documentation. No config changes.
### 2026-04-20 — joined the `media` GID 10000 standard
Idmap block applied; in-container `media` group at GID 10000 mapped to host GID 10000. See [media permissions](../infrastructure/media-permissions.md). Config backup: `/root/101.conf.bak.*`.

View File

@@ -54,7 +54,7 @@ We considered three options before building this:
| Option | Outcome |
|---|---|
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../investigations/index.md)). Rejected. |
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. |
| **SMB on host** | Same host-blast-radius problem, plus 3050% lower throughput than NFS on Linux↔Linux. Rejected. |
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |

View File

@@ -18,7 +18,7 @@ Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/d
| paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers |
## Auth
Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
Behind [Authentik forward-auth](106-auth-outpost.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
## Storage
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
@@ -27,7 +27,7 @@ Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses fo
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- [Authentik](124-authentik.md)
- [Authentik](106-auth-outpost.md)
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)

View File

@@ -55,7 +55,7 @@ Initial documentation.
Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md).
### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added
For OIDC integration with [authentik (124)](124-authentik.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
For OIDC integration with [authentik (124)](106-auth-outpost.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped
`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`.

View File

@@ -45,7 +45,16 @@ Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webh
### Portainer
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
### homelab-mcp (`/opt/homelab-mcp/`)
### homelab-mcp (`/opt/homelab-mcp/`) — DEPRECATED (Go rewrite, Phase 6)
> **Status:** This Python MCP server is being replaced by the Go `oikos api` binary
> running in Docker on mac-mini. Cutover pending — see
> [scripts/cutover-checklist.md](../../scripts/cutover-checklist.md) for the
> execution plan. The Go MCP uses the official MCP Go SDK (Streamable HTTP, not
> FastMCP) with 15 tools including `get_blast_radius`, `request_execution`, and
> `get_agent_activity`. Source: `internal/mcp/server.go`.
**Current (Python) implementation — DO NOT MODIFY, awaiting cutover:**
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
`/opt/homelab-context/` (this LXC is itself an enrolled
[homelab-context](../infrastructure/homelab-context.md) client). Source
@@ -70,7 +79,10 @@ the server code). Listens on `0.0.0.0:9811`, secret in
`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
restarts on push.
### secrets-issuance (`/opt/secrets-issuance/`)
### secrets-issuance (`/opt/secrets-issuance/`) — DEPRECATED (Go Phase 5)
> **Status:** Replaced by `internal/secrets/` in the Go rewrite. Machine identities
> are now managed via Infisical (`docker compose --profile infisical up`).
Tiny HTTP service that issues per-client age keypairs the first time
each client calls `/issue`. Idempotent: subsequent calls return the
same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in
@@ -100,7 +112,7 @@ Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.h
## Related
- [Gitea (104)](104-gitea.md) — uses the PlantUML server
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](124-authentik.md)
- [Authentik (124)](106-auth-outpost.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Public ingress (Artifacto + blog)](../infrastructure/ingress.md)
@@ -116,7 +128,7 @@ Two new services from the [homelab-context distribution plan](../infrastructure/
`secrets-issuance.service` on `:9820` (per-client age-key provisioning).
Caddy fronts both with Let's Encrypt; new vhosts on
[caddy](121-caddy.md), split-horizon DNS entries on
[authentik (124)](124-authentik.md). Gitea webhook ids 10 + 11 wire
[authentik (124)](106-auth-outpost.md). Gitea webhook ids 10 + 11 wire
auto-deploy. LXC is itself an enrolled context client
(`/opt/homelab-context/`).

View File

@@ -1,6 +1,6 @@
# 106 — `auth-outpost`
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
## At a glance
- **Hostname:** `auth-outpost`
@@ -8,11 +8,11 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
- **Resources:** 1 core / 512 MiB / 4 GiB rootfs
- **Mounts:** none
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](124-authentik.md)
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](106-auth-outpost.md)
## Role
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../investigations/2026-05-31-authentik-vps-migration.md).
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
## Service / port map
| Service | Listen | Notes |
@@ -42,15 +42,15 @@ Fix: the LAN outpost gets its **own** domain.
**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
## Related
- [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only)
- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only)
- [Caddy (121)](121-caddy.md) — forward-auth consumer
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
- [Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md)
- [Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)
## Changelog
### 2026-06-06 — Authentik session lifetime extended to 30 days
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../investigations/2026-06-06-authentik-session-lifetime.md).
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md).
### 2026-06-01 — created; forward-auth cut over from LXC 124
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000``192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md).
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000``192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).

View File

@@ -1,6 +1,6 @@
# 107 — `dns`
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](124-authentik.md); single-purpose, one job.
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](106-auth-outpost.md); single-purpose, one job.
## At a glance
- **Hostname:** `dns`
@@ -29,7 +29,7 @@ Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recur
- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field.
## dns-sync (Technitium = authoring source)
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../../../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
## DHCP
@@ -42,7 +42,7 @@ Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02):
Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101.239`) are excluded from the pool. Pool narrowed from `.100.240` to `.241.254` on 2026-06-03 to eliminate IP conflict risk.
## Related
- [124 — authentik](124-authentik.md) — retired host of the old dnsmasq
- [124 — authentik](106-auth-outpost.md) — retired host of the old dnsmasq
- [DNS split-horizon](../infrastructure/dns.md)
- [Mesh](../infrastructure/mesh.md)
@@ -55,7 +55,7 @@ Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md))
Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
### 2026-06-03 — DHCP pool narrowed to `.241.254`
Previous pool `.100.240` overlapped with all static LXCs/VMs (`.101.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101.110` remain until natural expiry (2026-06-04). See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
Previous pool `.100.240` overlapped with all static LXCs/VMs (`.101.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101.110` remain until natural expiry (2026-06-04). See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone)
This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
@@ -64,4 +64,4 @@ This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py`
Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
### 2026-06-01 — created; replaced dnsmasq on 124
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](124-authentik.md).
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](106-auth-outpost.md).

View File

@@ -11,7 +11,7 @@ Personal cloud / file collaboration. Source-of-truth for the photo libraries sur
- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)
## Auth
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](124-authentik.md#per-app-username-override-pattern-authentik) for the full pattern.
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](106-auth-outpost.md#per-app-username-override-pattern-authentik) for the full pattern.
Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on).
@@ -74,7 +74,7 @@ Apply with `systemctl restart mariadb` (not reload — `innodb_log_file_size` ne
## Related
- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV
- [Authentik (124)](124-authentik.md)
- [Authentik (124)](106-auth-outpost.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md)

View File

@@ -4,7 +4,8 @@ Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
## At a glance
- **Hostname:** `elementsynapse`
- **IP:** `192.168.8.239`
- **IP:** `192.168.8.242`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** **unprivileged**
- **Resources:** 1 core / 2 GiB RAM / **16 GiB rootfs** (grown from 8 GiB on 2026-05-15 after disk-full incident)
- **Mounts:** none from `/mnt/library`
@@ -36,7 +37,7 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
- ~~[claudio-bot (123)](archive/123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)

View File

@@ -60,7 +60,7 @@ For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials
## Related
- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries
- [Authentik (124)](124-authentik.md)
- [Authentik (124)](106-auth-outpost.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)

View File

@@ -11,16 +11,16 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L
- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy)
- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`).
## Sites currently served (live as of 2026-04-28)
## Sites currently served (live as of 2026-07-06)
- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100`
- `auth.hubris.network` → [authentik (124)](124-authentik.md) `:9000`
- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080`
- `books.hubris.network` → [apps (105)](105-apps.md) `:6060`
- `books.hubris.network` → [grimmory (130)](130-grimmory.md) `:6060`
- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443`
- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443`
- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`)
- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123`
- `house.hubris.network` → [house (129)](129-house.md) `:3000`
- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056`
- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008`
- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096`
@@ -28,15 +28,18 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L
- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
- `roms.hubris.network` → [romm (134)](134-romm.md) `:80`
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
- `teddy.hubris.network` → LXC 131 `192.168.8.150:8443`
- `trmnl.hubris.network` → [trmnl (128)](128-trmnl.md) `:9851`
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
## Snippet: `(authentik)` forward-auth
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup).
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](106-auth-outpost.md#forward-auth-domain-level-setup).
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md).
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](106-auth-outpost.md).
## Caddy environment
@@ -57,7 +60,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
## Related
- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
- [Authentik (124)](124-authentik.md) — forward-auth + IdP
- [Authentik (124)](106-auth-outpost.md) — forward-auth + IdP
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
- [Gitea (104)](104-gitea.md) — webhook source
@@ -82,7 +85,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
Also: [elementsynapse LXC 118](../containers/118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
### 2026-06-02 — caddy.service unit missing; recreated
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.

View File

@@ -4,10 +4,11 @@ Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr
## At a glance
- **Hostname:** `arriman`
- **IP:** `192.168.8.132`
- **IP:** `192.168.8.245`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged
- **Resources:** 4 cores / 8 GiB RAM / 24 GiB rootfs
- **Mounts:** `/mnt/library``/mnt/library`
- **Mounts:** `/mnt/media_local``/mnt/library`
- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below)
## Compose
@@ -114,7 +115,7 @@ Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has
## Related
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](124-authentik.md) — forward-auth wiring + per-app `/api/*` bypass
- [Authentik (124)](106-auth-outpost.md) — forward-auth wiring + per-app `/api/*` bypass
- [DNS](../infrastructure/dns.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Hubris host](../hosts/hubris.md)

View File

@@ -35,7 +35,7 @@ Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `trmnl → 192.168.8.175` (LAN path via Caddy)
- [Gitea (104)](104-gitea.md) — source repo `dtoro/terminalito`
- [Plan: 2026-06-24 TRMNL plugins LXC](../plans/2026-06-24-trmnl-plugins-lxc.md)
- [Plan: 2026-06-24 TRMNL plugins LXC](../../../plans/2026-06-24-trmnl-plugins-lxc.md)
## Changelog
### 2026-06-24 — auto-deploy + LAN DNS wired

View File

@@ -5,7 +5,8 @@ Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modul
## At a glance
- **Hostname:** `house`
- **IP:** `192.168.8.212` (static)
- **IP:** `192.168.8.244`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** unprivileged
- **Resources:** 1 core / 1344 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
@@ -39,7 +40,7 @@ Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modul
- [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy)
- [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`)
- [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source
- [Deployment plan](../plans/2026-06-25-yuvomi-deployment.md)
- [Deployment plan](../../../plans/done/2026-06-25-yuvomi-deployment.md)
## Changelog

View File

@@ -5,17 +5,18 @@ Self-hosted digital library (eBooks, comics, audiobooks). Community fork/success
## At a glance
- **Hostname:** `grimmory`
- **IP:** `192.168.8.213` (static, set in PVE `net0` config — same pattern as all other LXCs)
- **IP:** `192.168.8.247`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged (UID = host UID for `/mnt/library` media GID)
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13)
- **Mounts:** `/mnt/library`
- **Mounts:** `/mnt/media_local``/mnt/library`
- **Public hostname:** `books.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| Grimmory | `192.168.8.213:6060` | Docker Compose at `/opt/grimmory/` |
| Grimmory | `192.168.8.247:6060` | Docker Compose at `/opt/grimmory/` |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
@@ -43,7 +44,7 @@ Uses Confidential client (client secret stored in Grimmory's DB — migrated fro
- **Client type:** Confidential (client secret in `oidc_provider_details` in MariaDB `app_settings`)
- **Redirect URI:** `https://books.hubris.network/oauth2-callback`
- **Scopes:** openid, profile, email, offline_access
- **Back-channel logout:** `http://192.168.8.213:6060/api/v1/auth/oidc/backchannel-logout`
- **Back-channel logout:** `http://192.168.8.247:6060/api/v1/auth/oidc/backchannel-logout`
- **Application slug:** `booklore` → Issuer URI: `https://auth.hubris.network/application/o/booklore/`
## Media permissions
@@ -53,8 +54,8 @@ LXC is privileged → in-container UID = host UID. Docker container gets media G
## Related
- [apps (105)](105-apps.md) — previous host (Booklore)
- [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.213:6060`
- [Authentik (124)](124-authentik.md) — OIDC provider `Grimmory`
- [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.247:6060`
- [Authentik (124)](106-auth-outpost.md) — OIDC provider `Grimmory`
- [DNS (107)](107-dns.md) — `books.hubris.network → 192.168.8.175` (unchanged from Booklore)
- [Media permissions](../infrastructure/media-permissions.md)

View File

@@ -0,0 +1,62 @@
# 131 — `teddycloud`
Self-hosted [TeddyCloud](https://github.com/toniebox-reverse-engineering/teddycloud), a
reimplementation of the Toniebox cloud backend — lets Tonie figurines play custom/ripped
audio content against a local server instead of the official cloud.
Predates the client-enrollment convention entirely; nobody wrote it down. Found and
documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged
`pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see
[OIKOS.md](../../../.agents/OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud`
container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict
for it — but it never got its own inventory entry or doc page until now.
## At a glance
- **Hostname:** `teddycloud`
- **Host:** hubris (confirmed via `pct config 131` run directly on hubris — the original
drift finding's `pct list` source)
- **IP:** `192.168.8.150` (static; was briefly `192.168.8.243` via DHCP until the
2026-07-05 strong-migration work assigned it a fixed address — see hosts/strong.md)
- **Privilege:** unconfirmed (not checked — read-only investigation didn't need it)
- **Resources:** 1 core / 1 GiB RAM / 512 MiB swap / 16 GiB rootfs (`local-lvm`), Debian 12
(bookworm)
- **Mounts:** `/mnt/library` (`mp0`)
- **Public hostname:** `teddy.hubris.network``192.168.8.150:8443` (see
[caddy (121)](121-caddy.md))
- **Enrollment:** none — no `age_pubkey`, not a `homelab` CLI client. It's a plain
docker-compose app container, not a fleet-managed host. No action needed unless it starts
needing secrets.
## Service
Runs via `docker compose` at `/opt/teddycloud` — container `teddycloud-teddycloud-1`,
image `ghcr.io/toniebox-reverse-engineering/teddycloud:latest`, publishing `80`, `443`,
`8080`, and `8443`. Caddy routes `teddy.hubris.network` to the `:8443` port.
## Risk notes
**No Caddy forward-auth gate** — unlike `sab.hubris.network` on the same Caddyfile (which
is explicitly annotated `(Authentik forward-auth)`), `teddy.hubris.network` has no auth
annotation. It's reachable to anyone on the LAN/mesh who can resolve the hostname. Not
addressed as part of this doc pass — flagging it here since it's now visible in one place
for the first time.
## Related
- [Hubris host](../hosts/hubris.md)
- [Caddy (121)](121-caddy.md) — terminates `teddy.hubris.network`
- [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already
taken by this container
- [Containers index](index.md)
- [OIKOS.md](../../../.agents/OIKOS.md) — drift detector that caught this
## Changelog
### 2026-07-06 — documented for the first time (drift-caught)
Added to `inventory.yaml` and given this page. Verified live via read-only `pct config 131`
+ `pct exec 131 -- ...` on hubris: hostname, IP, resources, and that it runs via
`docker compose` (not a raw binary or systemd unit). No changes made to the running
container — this is pure documentation catch-up.

View File

@@ -0,0 +1,199 @@
# 132 — `rclone`
Off-host backup appliance. Mirrors selected `/mnt/library` folders to **Proton Drive**
with a plain `rclone sync` (monthly), and serves rclone's Web GUI on the LAN for browsing
and ad-hoc runs. **Replaces** the disabled restic-on-USB job — see [backups](../infrastructure/backups.md).
Provisioned 2026-07-01. (LXC 131 was already taken by an undocumented `teddycloud` container,
so this landed on **132**.)
## At a glance
- **Hostname:** `rclone`
- **IP:** `192.168.8.214` (static, set in PVE `net0` config — same pattern as grimmory/authentik)
- **Privilege:** privileged (root in-container = host root → reads every `/mnt/library` subtree,
incl. `homecloud/` and `documents/`, regardless of owner)
- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs (Debian 13) — bumped from 1 GiB on 2026-07-03
after `rclone-rcd.service` was OOM-killed under real load (see "Known issue" below)
- **Mounts:** `/mnt/library` **read-only** (`mp0: /mnt/library,mp=/mnt/library,ro=1`) — a backup
job must never be able to write into the library
- **Public hostname:** none — the UI is **LAN-only, no auth** (by design)
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| rclone Web GUI (`rcd`) | `192.168.8.214:5572` | `rclone-rcd.service`, **`--rc-no-auth`**, LAN-only. Browse `/mnt/library` + `proton:`, run ad-hoc syncs, live job status |
| monthly mirror | — | `rclone-backup.service` + `.timer` (`OnCalendar=*-*-01 03:00`) |
## Backup design
- **Mode:** plain mirror — `rclone sync` (Proton mirrors local; deletions propagate; **no versioning**).
- **Encryption:** Proton Drive's built-in E2E only (no rclone `crypt` overlay → files stay
browsable in Proton's web UI).
- **Selected set:** `/etc/rclone-backup/folders.list` — one absolute source path per line
(`#`/blank ignored). This file *is* the picked set the monthly timer mirrors. Extensible to other
disks once bind-mounted into this LXC.
- **Path mapping:** source `S``proton:library-backup/<S without leading slash>`
(e.g. `/mnt/library/notes``proton:library-backup/mnt/library/notes`).
- **Runner:** `/usr/local/sbin/rclone-backup.sh [folder ...]` (Python, despite the `.sh` name — kept
the path stable) — no arg = every enabled line. Submits each folder as an **async job through the
rclone rc API** served by `rclone-rcd.service` (the same daemon backing the Web GUI on `:5572`),
so scheduled/ad-hoc runs show up live in the GUI's **Jobs panel**, not just in logs. Gentle on
Proton's rate limits (`Transfers=4, TPSLimit=8, FastList=true` via the rc `_config` payload). The
rc API here requires **POST for every call** including `job/status` and `core/stats` — GET with
query params 404s.
- **Logs / "past runs":** per-run logs in `/var/log/rclone-backup/<safe>-<ts>.log`; one-line
JSON summary per run appended to `/var/log/rclone-backup/runs.jsonl`.
- **Failure notify:** `OnFailure=rclone-backup-notify@%n.service` → logs to journal today;
**TODO** wire to Hermes `send_message` (Matrix) per [backups](../infrastructure/backups.md).
## rclone + Proton Drive
- **rclone** installed from the official binary (not apt) so the `protondrive` backend is present
(`rclone v1.74.3`).
- Remote **`proton:`** (type `protondrive`). Config at `/root/.config/rclone/rclone.conf`, mode 600.
**This file is a secret** (holds the obscured Proton password + TOTP secret + session) — **never
commit it.** Escrow the Proton account creds in the password manager.
- **Config gotchas** (from rclone docs/forum):
- Log into Proton via a **browser at least once** first, or key generation fails.
- For unattended runs, store the **TOTP _secret_** (not a 6-digit code) so rclone self-generates
codes; obscure with `rclone obscure`.
- Passwords with **extended-ASCII** characters are known to break auth.
- Proton's API is rate-limited → keep `--transfers`/`--tpslimit` conservative (baked into the runner).
- **DR escrow (pending):** store the Proton creds as sops secret `secrets/protondrive.yaml`, granted
to this LXC's age key, so the remote can be rebuilt after a re-provision.
## The UI (rclone Web GUI)
`rclone rcd --rc-web-gui --rc-no-auth --rc-addr 0.0.0.0:5572` (assets auto-downloaded on first
start). Reach it at **http://192.168.8.214:5572** on the LAN.
> **Security note:** `--rc-no-auth` exposes *full* rclone control — including deleting remote data —
> to anyone on the LAN (accepted per the design choice). The container has only a LAN NIC, so it is
> not publicly reachable. Harden later by adding `--rc-user/--rc-pass` or fronting it with Authentik.
## Tracked config (deferred)
**Not yet tracked.** The runner, systemd units, and `folders.list` currently live as plain files
directly on the LXC — fully functional, just not version-controlled or auto-deployed. A
`dtoro/rclone` gitea repo package (runner, units, `install.sh`, webhook receiver) is pre-built and
staged at `/root/rclone-repo` on the LXC for whenever this gets tracked (Shape A, like
[caddy](121-caddy.md)). Gitea `ALLOWED_HOST_LIST` already includes `192.168.8.214` in anticipation.
See [auto-deploy](../infrastructure/auto-deploy.md).
**Selected folders (live in `/etc/rclone-backup/folders.list`):** `/mnt/library/cloud` (287G),
`/mnt/library/documents` (249M), `/mnt/library/repos` (83M). `/mnt/library/notes` was synced once as
a connectivity test (not in the recurring set). Proton quota checked: 2 TiB plan, ~1.65 TiB free
after this set.
## Enrollment gotcha: `pct exec` PATH
`pct exec` (lxc-attach) does **not** source `/etc/environment` or run a login shell, so
`/usr/local/bin` (where bootstrap installs `sops`) isn't on `$PATH` by default — bootstrap's own
`command -v sops` post-install check failed under `pct exec` even though the binary installed fine.
Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs
on future LXC bootstraps.
## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
What looked like repeated "protondrive silently stalls" was actually **`rclone-rcd.service` (the rc
API daemon backing the Web GUI and, since the rc-API redesign, all actual sync work) getting
OOM-killed** under the original 1 GiB RAM allocation — `journalctl` confirms
`A process of this unit has been killed by the OOM killer` at the exact moment a transfer had
"frozen." systemd's own `Restart=on-failure` (5s) auto-respawns it, but every in-flight job's state
is lost on the kill, which looked identical to a silent backend hang from the outside (frozen
`core/stats`, no new log lines). **Fix: bumped the LXC's memory to 2 GiB** (`pct set 132 -memory
2048` — applies live via the host cgroup, confirmed via `cat /sys/fs/cgroup/lxc/132/memory.max` on
hubris, no container reboot needed). After the bump, the full folder set (`cloud` 287G, `documents`
249M, `repos` 83M) completed cleanly with no further kills.
**`rclone-backup-watchdog.timer`** (every 5 min) → `rclone-backup-watchdog.sh`: if
`rclone-backup.service` is active but total transferred bytes (global `core/stats` on the rc API)
haven't moved for 15 minutes, it restarts both `rclone-rcd.service` (clears any stuck/orphaned job —
this is the actual daemon holding the work, not the thin wrapper) and then `rclone-backup.service`
(`--no-block`, load-bearing — see below). Kept as a safety net even after the RAM fix, in case
memory pressure returns under a larger folder set later. State kept in
`/var/lib/rclone-backup/watchdog-state.json`, cleared whenever the service isn't running.
**Two watchdog design bugs found and fixed while chasing this (2026-07-03):**
1. **Wrong stats-group key.** Per-job progress polling queried `core/stats` under `job/<jobid>`,
but rclone tracks stats under whatever `_group` name the job was submitted with. Made a perfectly
healthy sync look stalled at 0 bytes for 22+ hours in its own log. Fixed by using the same
`group` variable consistently. **Lesson: distrust the per-run log's "progress bytes=" line during
an incident; cross-check with unfiltered `core/stats` first.**
2. **Watchdog restarted only the thin wrapper, and blocked doing it.** The actual `rclone sync` work
runs inside `rclone-rcd.service`, not `rclone-backup.service` — restarting the wrapper alone left
any stuck job orphaned inside `rcd` while a new wrapper submitted a duplicate job on top. Worse,
`systemctl restart rclone-backup.service` (no `--no-block`) blocks until the *new* invocation's
long-running `ExecStart` exits — which could be hours — so the watchdog's own oneshot service
never logged "Finished," and `OnUnitActiveSec` (which schedules relative to the previous run
*finishing*) never fired again. The watchdog silently disabled itself after exactly one use.
Fixed: restart `rclone-rcd.service` first, then `rclone-backup.service` with `--no-block`.
## Related
- [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup
- [Hubris host](../hosts/hubris.md) — owns `/mnt/library`
- [Media permissions](../infrastructure/media-permissions.md) — read-only consumer of `/mnt/library`
- [Containers index](index.md)
## Changelog
### 2026-07-03 — root cause found (OOM, not Proton); RAM bumped to 2 GiB; full folder set completed
What looked like repeated silent "protondrive stalls" turned out to be **`rclone-rcd.service`
getting OOM-killed** under the original 1 GiB RAM allocation — confirmed via
`journalctl -u rclone-rcd.service` showing `killed by the OOM killer` at the exact freeze point.
Bumped the LXC to 2 GiB RAM (live, no reboot). After the bump: `cloud` (287G) completed cleanly
(exit 0), `documents` (249M) completed with 1 minor error (259.7 MB transferred), `repos` (83M)
completing as of this entry. Also fixed two real watchdog bugs found while chasing this (wrong
stats-group key making a healthy sync look frozen; watchdog restarting only the wrapper with a
blocking `systemctl restart`, causing it to silently disable itself after one use) — see "Known
issue" above for full detail. The watchdog is kept as a safety net going forward even though the RAM
bump addresses the actual root cause.
### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility)
The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web
GUI's Jobs panel — the GUI only tracks work submitted through its own `rcd` process. Rewrote
`/usr/local/sbin/rclone-backup.sh` in Python, submitting each folder via `POST /sync/sync` with
`_async: true` against `http://127.0.0.1:5572` (the running `rclone-rcd.service`), then polling
`POST /job/status` + `POST /core/stats` (both **must be POST** — GET-with-querystring 404s on this
rc API) until finished, logging periodic progress snapshots and the same `runs.jsonl` summary line
as before. Verified live: submitted job visible in `POST /job/list`'s `runningIds` while running,
completed cleanly (`success: true`) once done. Deployed via atomic rename (write-then-`mv`) rather
than truncating in place, specifically so it wouldn't risk corrupting the still-running original
`cloud`+`documents`+`repos` sync mid-flight (verified after the fact: that sync's bash process was
unaffected, kept running to completion under the old in-memory script content). The already-running
scheduled sync from before this change is a standalone process and won't retroactively appear in the
GUI; every run after this point will.
### 2026-07-02 — Proton Drive auth fixed; real folder set enabled; first live sync
Initial `rclone config` failed 2FA (`422 ... auth/v4/2fa`) because a live 6-digit TOTP code was
entered instead of the TOTP secret — reconfigured with the secret, auth now works
(`rclone lsd proton:` lists the Drive). Verified end-to-end with a real sync of `/mnt/library/notes`
(219 objects, 5.964 MiB, exit 0) — confirmed files land as plain, browsable objects on Proton (not
an opaque archive), matching the plain-mirror + Proton-E2E design. Checked Proton quota (2 TiB
plan, 1.945 TiB free) before enabling a large folder. `folders.list` set to the real selection:
`cloud` (287G), `documents` (249M), `repos` (83M); a full sync of that set was kicked off via the
actual `rclone-backup.service` unit (not an ad-hoc call) to validate the real monthly path early
rather than waiting for the Aug 1 timer. Tracked-repo step (`dtoro/rclone` on gitea) deferred by
choice — runner/units/`folders.list` remain plain files on the LXC for now; the repo package stays
staged at `/root/rclone-repo` for later.
### 2026-07-01 — provisioned; enrolled
LXC 132 created (Debian 13, privileged, `192.168.8.214`, `/mnt/library` read-only). rclone v1.74.3
installed from the official binary (`protondrive` backend present). Runner + monthly timer +
`folders.list` deployed; rclone Web GUI (`rcd`, LAN-only no-auth) live on `:5572`. Enrolled into
homelab-context (`--no-mesh`, LAN-only issuance): age key issued, inventory finalized, shared
secrets granted, `homelab whoami` + `homelab secret hello` verified. Gitea `ALLOWED_HOST_LIST`
updated to include `192.168.8.214`. Hit and fixed a `pct exec` PATH gotcha (see below). Proton Drive
remote, `dtoro/rclone` tracked repo + webhook, and the `secrets/protondrive.yaml` escrow remain
operator-run follow-ups (credentialed steps — Proton password/2FA, repo creation). Restic-on-USB
backup deprecated in the same change.

View File

@@ -0,0 +1,61 @@
# seanime (LXC 133) — Seanime anime media server
## Summary
| Field | Value |
|-------|-------|
| VMID | 133 |
| Host | strong |
| Role | Anime media server (online streaming + torrent client) |
| LAN IP | 192.168.8.248/28 (vmbr1) |
| Public URL | https://seanime.hubris.network |
| Backend | Seanime v3.9.0, systemd service |
| Port | 43211 |
## Mounts
- `/anime``/mnt/media_local/anime` (ludo-lvm, bind mount) — existing anime collection
## Services
### Seanime (native binary)
- Binary: `/opt/seanime/bin/seanime`
- Data: `/opt/seanime/data/`
- Config: `/opt/seanime/data/config.toml`
- Service: `systemctl status seanime`
- Flags: `--host 0.0.0.0 --port 43211 --datadir /opt/seanime/data --disable-password`
- Config highlights:
- `secureMode = 'lax'` (allows non-local Caddy proxy)
- `trustedProxies = ['192.168.8.0/24']`
### Torrent client (qBittorrent on arriman)
- Host: `192.168.8.245:8080`
- Auth: subnet whitelist (no password needed from homelab LAN)
- qBittorrent config updated: `AuthSubnetWhitelist=192.168.8.0/24`
### Extensions installed
**Online streaming (8):**
HiAnime, AniWatch, KickAssAnime, Anicrush, Animo, AniNeko, Senshi, Sudatchi
## Caddy
- Config: `seanime.hubris.network` block in `/etc/caddy/Caddyfile` (dtoro/caddy-conf repo)
- TLS: Let's Encrypt DNS-01 via IONOS API
- Backend: `192.168.8.248:43211`
## DNS
- `seanime.hubris.network` A record → `192.168.8.175` (Caddy)
## Changelog
### 2026-07-05 — initial creation
- LXC 133 created on strong (Debian 13, 2 cores / 2 GiB)
- Seanime v3.9.0 installed as native binary + systemd service
- /anime bind mount from ludo-lvm
- qBittorrent on arriman configured as torrent client
- Caddy reverse proxy + DNS records set up
- Online streaming extensions installed

View File

@@ -0,0 +1,82 @@
# 134 — `romm`
Self-hosted ROM manager ([RomM](https://romm.app)). Browse, search, and play
your retro game library from the browser. Runs on Docker Compose with a
MariaDB sidecar.
## At a glance
- **Hostname:** `romm`
- **IP:** `192.168.8.249/28` (static, vmbr1 on strong)
- **Privilege:** privileged
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13, ludo-lvm)
- **Mounts:** `/mnt/media_local``/mnt/library`
- **Public hostname:** `roms.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| RomM | `192.168.8.249:80` | HTTP (Caddy terminates TLS) |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
Located at `/opt/romm/docker-compose.yml`. Key points:
- Image: `rommapp/romm:latest`
- DB sidecar: `mariadb:latest` with healthcheck
- ROM library: `/mnt/library/roms``/romm/library` (writable)
- Resources (covers, etc.): Docker named volume `romm_resources``/romm/resources`
- Saves/states: `/opt/romm/assets``/romm/assets`
- Config: `/opt/romm/config``/romm/config`
- Auth key: auto-generated, stored in `/opt/romm/.env`
Environment (`/opt/romm/.env`):
- `DB_ROOT_PASSWD` / `DB_PASSWD` — MariaDB credentials
- `DB_USER=romm-user` / `DB_NAME=romm`
- `ROMM_AUTH_SECRET_KEY` — auto-generated
## ROM library structure
RomM expects `/mnt/library/roms/<platform>/<game>/<rom>`. Create platform
directories as needed:
```
/mnt/media_local/roms/
├── gba/
│ └── Pokemon - Emerald/
│ └── Pokemon Emerald.gba
├── snes/
│ └── Super Mario World/
│ └── Super Mario World.sfc
└── psx/
└── Final Fantasy VII/
└── Final Fantasy VII.bin
```
## Media permissions
The `/mnt/media_local/roms` directory is owned `root:media` with mode `2775`
(setgid). New files inherit the `media` GID (10000). The LXC is privileged so
no idmap block is needed — in-container UID/GID matches the host. Docker
containers within the LXC run as-is (read-only mount).
## Related
- [Strong host](../hosts/strong.md)
- [Caddy (121)](121-caddy.md) — `roms.hubris.network → 192.168.8.249:80`
- [DNS (107)](107-dns.md) — `roms.hubris.network A 192.168.8.175`
- [Media permissions](../infrastructure/media-permissions.md)
- [RomM docs](https://docs.romm.app)
## Changelog
### 2026-07-05 — provisioned
LXC 134 created on strong (Debian 13, privileged, `192.168.8.249/28`).
Docker + Compose installed. RomM stack deployed at `/opt/romm/`.
Created `/mnt/media_local/roms` with setgid `media:GID=10000` mode `2775`.
Caddy `roms.hubris.network``192.168.8.249:80`.
DNS `roms A 192.168.8.175` added to Technitium.
Hubris /32 route for `.249` added to `50-strong-route`.

View File

@@ -0,0 +1,57 @@
# LXC containers — index
Most containers live on [`hubris`](../hosts/hubris.md). Some have been
[migrated to `strong`](../hosts/strong.md) (Phase 1+2, 2026-07-05).
| ID | Name | Host | IP | Priv | Cores | RAM | Disk | Mounts | Public hostname | Status |
| --- | ---------------- | ------- | --------------- | ---- | ----- | ----- | ----- | --------------------- | ------------------------------------- | -------- |
| 101 | [jellyfin](101-jellyfin.md) | **strong** | 192.168.8.246 | priv | 4 | 8 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `media.hubris.network` | running |
| 103 | [paperless](103-paperless.md) | hubris | 192.168.8.130 | priv | 2 | 3 GiB | 8 GiB | `/mnt/library` | `paperless.hubris.network` | running |
| 104 | [gitea](104-gitea.md) | hubris | 192.168.8.121 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` | `git.hubris.network` | running |
| 105 | [apps](105-apps.md) | hubris | 192.168.8.205 | priv | 2 | 4 GiB | 30 GiB | `/mnt/library` | `docker` / `artifacto` / `blog` | running |
| 114 | [nextcloud](114-nextcloud.md) | hubris | 192.168.8.224 | priv | 4 | 6 GiB | 25 GiB | `/mnt/library` | `cloud.hubris.network` | running |
| 118 | [elementsynapse](118-elementsynapse.md) | **strong** | 192.168.8.242 | unpriv | 2 | 4 GiB | 32 GiB | — | `matrix.hubris.network` | running |
| 119 | [sophia](119-sophia.md) | hubris | 192.168.8.157 | priv | 2 | 1 GiB | 10 GiB | `/mnt/library` | — | running |
| 120 | [mule-images](120-mule-images.md) | hubris | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU) | `photos.hubris.network` | running |
| 121 | [caddy](121-caddy.md) | hubris | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running |
| 122 | [arriman](122-arriman.md) | **strong** | 192.168.8.245 | priv | 4 | 8 GiB | 24 GiB | `/mnt/media_local` (via mp0) | `jellyseerr` / `qbit` / `sab` | running |
| 124 | [authentik](106-auth-outpost.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running |
| 128 | [trmnl](128-trmnl.md) | hubris | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running |
| 129 | [house](129-house.md) | **strong** | 192.168.8.244 | unpriv | 2 | 3 GiB | 8 GiB | — | `house.hubris.network` | running |
| 130 | [grimmory](130-grimmory.md) | **strong** | 192.168.8.247 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `books.hubris.network` | running |
| 131 | [teddycloud](131-teddycloud.md) | hubris | 192.168.8.150 | — | 1 | 1 GiB | 16 GiB | `/mnt/library` | `teddy.hubris.network` (no auth gate) | running |
|| 132 | [rclone](132-rclone.md) | hubris | 192.168.8.214 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` (**ro**) | — (LAN-only UI `:5572`) | running |
|| 134 | [romm](134-romm.md) | **strong** | 192.168.8.249 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `roms.hubris.network` | running |
## Recently destroyed (kept for archaeology)
| ID | Name | Destroyed | Reason |
| --- | ---------------- | --------------- | --------------------------------------------- |
| 127 | mule-photos-new | 2026-05-22 | PhotoPrism + sidecar + SvelteKit stack promoted to LXC 120 via Mulimage 2.0 merge (`70dc1b6`); M0 test LXC retired. Caddy + dnsmasq + gitea webhook + NC webhook listeners all cleaned up in the same cutover. |
| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose |
| 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
| 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology |
| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) |
| 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty |
| 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 110 | photoprism | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 111 | karakeep | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 112 | immich | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 115 | reticulum | between 2026-04-21 and 2026-04-28 | Decommissioned |
> Several `.conf.bak` files survive under `/etc/pve/lxc/` if you need to recover any of the configs.
## Conventions
- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](106-auth-outpost.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range.
- `onboot=1` on every container — the host brings them up after `pve-guests.service`.
- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library` on hubris, or `mp0: /mnt/media_local,mp=/mnt/library` on strong.
- Most containers are privileged. Unprivileged ones require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
## Related
- [Hubris host](../hosts/hubris.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Caddy](121-caddy.md) — terminates every public hostname
- [DNS](../infrastructure/dns.md) — split-horizon entries for each subdomain

View File

@@ -1,11 +1,14 @@
# `hubris` — Proxmox host
Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
`Homelab` cluster (see [Cluster](#cluster)); the second node is
[strong](strong.md), which hosts nothing yet.
## At a glance
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
@@ -22,13 +25,37 @@ Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
## Cluster
Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md)
(cluster/OS hostname `strong`), formed 2026-07-01.
- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address).
strong reaches it via the existing Fritz!Box static route
(`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the
household LAN. Fine for a home cluster; not latency-isolated.
- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both
votes — if either node is down (reboot, maintenance, network hiccup), the
survivor's running guests keep working but `/etc/pve` goes read-only:
no start/stop/create/edit until quorum returns. Decided to skip a QDevice
for now; revisit if hubris's periodic reboots (BIOS/thermal work, see
Quirks below) make this painful in practice.
- **Storage:** `local` / `local-lvm` are the standard per-node default IDs
(every node has its own, not actually shared). The `library` lvmthin pool
is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since
it's a physical thinpool that only exists on this host's hardware.
- strong currently hosts no LXCs/VMs — it exists solely as a cluster
member so far. See [strong.md](strong.md) and the [library-SSD
migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)
for what comes next (physical drive move, service migration — not started).
## Tenants
### VMs
- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
### LXC containers
See [containers/index](../containers/index.md). 13 active (109 syncthing destroyed 2026-05-14).
See [containers/index](../containers/index.md). 10 active on hubris (101, 118, 122, 129, 130 migrated to [strong](strong.md) 2026-07-05).
## Boot-time tuning (load-bearing)
@@ -62,7 +89,7 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b
## Quirks
- `/etc/pve` is fuse — normal for the Proxmox cluster filesystem, even on a single-node install.
- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
- ZFS is **not** in use; storage is LVM-thin + ext4.
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
@@ -73,6 +100,8 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b
- `root@hubris` (self, RSA) — local
- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
`pvecm add` could authenticate without a password prompt
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
@@ -84,16 +113,20 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220
- [Media permissions](../infrastructure/media-permissions.md)
- [Monitoring](../infrastructure/monitoring.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Operations cheatsheet](../operations/commands.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
- [Operations cheatsheet](../../../.agents/operations/commands.md)
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
- [strong — Proxmox host](strong.md)
## Changelog
### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-05-14 — LXC 109 (syncthing) decommissioned
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
### 2026-05-14 — network performance baseline captured
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
@@ -105,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`.
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
### 2026-04-29 — relocated to better-ventilated spot
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 3435.5 °C, ACPI zone 4749 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 6061 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 3435.5 °C, ACPI zone 4749 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 6061 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
### 2026-04-28 — Phase 1 WiFi failover
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
@@ -114,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
### 2026-04-23 — SSD cooling + thermal pads installed
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 6061 °C, nvme1 3840 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 6061 °C, nvme1 3840 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
### 2026-04-22 — drive removal A/B test
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
@@ -126,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
### 2026-04-21 — `cpu-epp.service` deployed
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md).
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).

View File

@@ -0,0 +1,9 @@
# Hosts
The two Proxmox VE nodes that run the fleet. Narrative pages; the machine-readable host records are
the generated `hosts/*.yaml` files at the repo root.
| Document | What it covers |
|----------|----------------|
| [hubris.md](hubris.md) | Primary PVE node (GMKtec NucBox M6 Ultra, `192.168.8.77`) — runs most LXCs plus the VMs. |
| [strong.md](strong.md) | Secondary PVE node / cluster member (`192.168.178.181`) — hosts the LXCs migrated from hubris. |

View File

@@ -0,0 +1,159 @@
# `strong` — Proxmox host
Second node in the `Homelab` cluster, alongside [hubris](hubris.md). Formerly
a Linux dev workstation nicknamed "ludo" (or "ludo-mini") — reformatted to
Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
## At a glance
- **Role:** Proxmox VE 9.2.3 cluster member (kernel `7.0.12-1-pve`) — hosts [7 LXCs](../containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05)
- **Naming:** the OS/cluster hostname is `strong` (leftover from install,
kept as-is). This wiki page and `inventory.yaml` also use `strong` as of
2026-07-01 — earlier the same day the inventory entry was briefly named
`ludo-mini`, but that was renamed so the hostname bootstrap needs
(`hosts/$(hostname).yaml`) would just resolve. "Ludo"/"ludo-mini" remains
the machine's everyday nickname; some older docs (investigations, the
library-SSD migration plan) still refer to it that way — that's fine,
those are historical.
- **Hardware:** AMD Ryzen 7 PRO 6850U, 16 threads, 28 GiB RAM.
- `nvme0n1` — MasonSemi MC3100 1TB (boot/OS): `pve-root` 96G, `pve-data`
(thinpool) 815G, 8G swap.
- `nvme1n1` — WD_BLACK SN7100 2TB, added 2026-07-01. Arrived with an
existing APFS partition (previously used in a Mac) — wiped via
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1`, then turned
into its own LVM-thin pool via `pvesh create nodes/strong/disks/lvmthin`
(handles pvcreate/vgcreate/lvcreate + storage.cfg registration in one
step). Registered as Proxmox storage `ludo-lvm` (VG `ludo-lvm`, thinpool
`ludo-lvm`, ~1.8 TiB usable — kept the `ludo-lvm` name since that's what
was actually created live; not worth renaming), restricted `nodes strong`
in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool.
Empty so far; this is separate from the [library-SSD migration
plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s
planned drive move from hubris (that hasn't happened) — this is
general-purpose VM/CT capacity.
- **Network:** `vmbr0` is bridged straight onto the household LAN —
`192.168.178.181/24`, gateway `192.168.178.1`. Unlike hubris, there is no
internal/uplink bridge split yet — `vmbr0` here plays the role hubris's
`vmbr1` plays there. Reachable from the homelab subnet (`192.168.8.0/24`)
via the existing Fritz!Box static route through hubris; no new cabling
or routing was needed to join the cluster.
- **Storage:** `local` + `local-lvm` (cluster-wide default names, shared
with hubris by convention, actually separate per-node volumes). Hubris's
`library` lvmthin pool is restricted to `nodes hubris` in
`/etc/pve/storage.cfg` and does not exist here.
- **Mesh:** Netbird not installed — fresh OS wiped whatever the old
workstation had. Reachable today only via LAN routing (confirmed DNS for
`*.hubris.network` already resolves correctly here via `192.168.8.2`).
Add to Netbird if off-LAN access to this host itself (distinct from any
future guests) is needed.
- **Homelab-context client enrollment:** done 2026-07-01 via
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
5-min sync timer are live; `homelab whoami` resolves correctly. See
[agent-enrollment.md](../../../.agents/operations/agent-enrollment.md).
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
## Cluster membership
Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on
2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full
cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice —
either node going down freezes management on the survivor).
## SSH
Root login via the same key set trusted on hubris (`root@hubris`,
`d.toro.v@pm.me`) — installed 2026-07-01 by appending to
`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys`
post cluster-join, so it's cluster-synced same as hubris). No password auth
needed going forward.
## Related
- [hubris — Proxmox host](hubris.md)
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
- [Network](../infrastructure/network.md)
- [SSH access](../infrastructure/ssh-access.md)
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
## Changelog
### 2026-07-05 — Phase 2d: grimmory (130) migrated to strong
Migrated book library from hubris (192.168.8.213) to strong (192.168.8.247).
Rsync'd /books (2.6G) from hubris library SSD to ludo-lvm/media. Container
mounts /mnt/media_local → /mnt/library locally. Caddy backend for
books.hubris.network updated.
### 2026-07-05 — Phase 2: arriman (122) + jellyfin (101) migrated; library on ludo-lvm
Migrated arriman (→ 192.168.8.245) and jellyfin (→ 192.168.8.246) to strong.
Created 1.5T thin volume on ludo-lvm for media library (/mnt/media_local).
Rsync'd 363G of media data (movies, tv, anime, downloads, music) from hubris
library SSD to ludo-lvm. Both containers now mount /mnt/media_local directly
as local ext4 — no NFS cross-node dependency. Jellyfin gets Radeon 680M iGPU
(for VAAPI) + RX 7600 dGPU passthrough via dev0/dev1. Caddy backends updated
for media, jellyseerr, qbit, sab. Hubris freed 16 GiB RAM (8+8).
Dhcp scope narrowed to 192.168.8.100-239.
### 2026-07-05 — house (LXC 129) migrated to strong; DHCP scope narrowed
Migrated Yuvomi family planner from hubris (192.168.8.212) to strong
(192.168.8.244). Followed same restore pattern as elementsynapse (Phase 1b).
Discovered IP conflict: teddycloud (LXC 131) had 192.168.8.243 via DHCP
(scope was 192.168.8.241-254). Narrowed DHCP scope to 192.168.8.100-239,
gave teddycloud static IP 192.168.8.150. Caddy updated: house.hubris.network
→ 192.168.8.244:3000, teddy.hubris.network → 192.168.8.150:8443.
### 2026-07-05 — elementsynapse (LXC 118) migrated to strong
Migrated Matrix/Synapse + Element client from hubris (192.168.8.239) to a new
LXC on strong (192.168.8.242). Added vmbr1 on strong at 192.168.8.241/28
(portless internal bridge) for strong-hosted guests. Enabled IP forwarding,
proxy ARP on hubris vmbr0, and specific /32 routes for strong's guest subnet.
Caddy updated: element.hubris.network → 192.168.8.242:8080.
VPS traefik updated: matrix.hubris.network backend → 192.168.8.242:8008.
This is Phase 1a of the strong migration plan — see .hermes/plans/2026-07-05_strong-migration-assessment.md.
### 2026-07-01 — age key issued over LAN; 3 bugs found/fixed in bootstrap.sh
Re-ran bootstrap without `--no-secrets` to get a real age key. Hit three real bugs live, fixed all three in `bootstrap.sh` and re-ran clean:
1. The `mcp`-CLI pipx-install step and the (unused, `--with-hermes`-only) Goose installer both called `sudo -u <user>` unconditionally — fails with "sudo: command not found" on a minimal root-only image with no `sudo` binary at all. Added a `run_as()` helper that only shells out to `sudo` when there's a real distinct invoking user.
2. `sops` isn't an apt/dnf package (matches what `agent-enrollment.md`'s manual-install recipe already does) — the auto-installer tried `apt-get install sops` and failed outright. Added `install_sops_binary()`, fetching the GitHub release binary directly on both dnf and apt paths.
3. Bigger one: running without `--no-secrets` unconditionally tries to install + interactively connect Netbird (device-code SSO), even though the very next check already knows how to accept plain LAN reachability instead. Over SSH with nobody watching, this hangs forever — had to manually kill a stuck `netbird up` process. Added `--no-mesh`, which skips the Netbird install/connect step but keeps the LAN-fallback path for secrets issuance. This run used `bootstrap.sh --no-mesh` and completed cleanly: `mesh: lan`, age key installed, `mcp` CLI installed via pipx (proving fix #1 too).
Result: age key at `/etc/age/key.txt`, pubkey recorded in `inventory.yaml`. Not yet granted access to any actual secret file — see the note above.
### 2026-07-01 — enrolled as a homelab-context client
Ran `bootstrap.sh --no-secrets` (reused the operator's existing personal Gitea PAT for the initial clone rather than minting a fresh read-only one). Installed git, cloned `/opt/homelab-context`, installed the 5-min systemd sync timer, symlinked `homelab` CLI and `AGENTS.md`. Skipped age-key/secrets issuance and Netbird per operator choice — but bootstrap's own connectivity check reported `mesh: lan`, i.e. the secrets-issuance endpoint is already reachable over plain LAN, so re-running without `--no-secrets` later wouldn't require a Netbird join. Known gap: the `mcp` pipx CLI install step silently failed (`sudo: command not found` — bootstrap.sh's pipx step assumes a `sudo` binary even when already root; harmless, only affects the `homelab mcp <tool>` shell subcommand).
### 2026-07-01 — inventory identity renamed ludo-mini → strong
Discovered while starting client enrollment: `bootstrap.sh` looks up
`hosts/$(hostname).yaml`, and the OS hostname here is `strong`, not
`ludo-mini`. Renaming the OS hostname was ruled out (already a cluster
member — Proxmox doesn't support in-place node rename, only leave+rejoin).
Renamed the wiki/inventory side instead: `inventory.yaml` key, this page
(`hosts/ludo-mini.md``hosts/strong.md`), README, ssh-access.md all now
say `strong`. "Ludo"/"ludo-mini" is still fine as a spoken nickname.
### 2026-07-01 — 2nd NVMe added; new LVM-thin pool `ludo-lvm`
User added a WD_BLACK SN7100 2TB (`nvme1n1`), previously used in a Mac
(arrived with an EFI + APFS partition table — confirmed disposable, wiped).
Used Proxmox's own disk-management API rather than raw LVM commands:
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1` to clear the old
partition table/signatures, then `pvesh create nodes/strong/disks/lvmthin
--name ludo-lvm --device /dev/disk/by-id/nvme-WD_BLACK_SN7100_2TB_251663803202
--add_storage 1` to create the PV/VG/thinpool and register it as Proxmox
storage in one step. Result: storage ID `ludo-lvm`, ~1.8 TiB, `content
rootdir,images`, `nodes strong` (mirrors hubris's `library` node-restriction
pattern — this pool only physically exists here). Empty — no VM/CT disks
placed on it yet.
### 2026-07-01 — Proxmox install; joined Homelab cluster
Reformatted from Linux workstation to Proxmox VE 9.2.3. SSH keys seeded from
hubris's trusted set (root password used once, then discarded). Joined the
existing `Homelab` cluster via `pvecm add 192.168.8.77 --use_ssh 1` from
this node's side, using key-based SSH pre-authorized in both directions —
no interactive password prompt needed for the join itself. Cluster now 2
nodes (`hubris`, `strong`), quorate, no QDevice. Decided to leave hostname
as `strong` and skip a QDevice for now — both revisitable later.

View File

@@ -0,0 +1,14 @@
# Knowledge
The durable, authoritative current-state documentation of the homelab: one page per node and per
cross-cutting system, synthesized from live state and evidence. Structure and rules are in
[the knowledge schema](../.agents/domains/knowledge/schema.md).
| Section | What it covers |
|---------|----------------|
| [wiki/hosts/](wiki/hosts/index.md) | Proxmox host narratives — `hubris`, `strong`. |
| [wiki/containers/](wiki/containers/index.md) | LXC fleet — one page per container, plus the master table and archaeology. |
| [wiki/vms/](wiki/vms/index.md) | Virtual machines — ZimaOS, Home Assistant OS. |
| [wiki/infrastructure/](wiki/infrastructure/index.md) | Cross-cutting systems — DNS, ingress, mesh, storage, auth, monitoring, generated topology. |
| [sources/](sources/index.md) | External reference docs and the pointer to incident evidence. |
| [GLOSSARY.md](GLOSSARY.md) | Term definitions. |

View File

@@ -23,7 +23,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
- ~~`192.168.8.190` ([plato (126)](../containers/126-plato.md))~~ (destroyed 2026-06-28)
- ~~`192.168.8.190` ([plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology))~~ (destroyed 2026-06-28)
- `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito)
**Don't strip these when editing app.ini.**
@@ -38,18 +38,19 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/126-plato.md) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 (deprecated) | ~~reinstalls `homelab-mcp.service` + restart~~ → replaced by Go Docker stack on mac-mini |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 (deprecated) | ~~reinstalls `secrets-issuance.service` + restart~~ → replaced by `internal/secrets/` Go package |
| `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` |
| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) |
> Note: `dtoro/Homelab-Docs` has **two webhooks** firing on the same push.
> Note: `dtoro/Homelab-Docs` has **three webhooks** firing on the same push.
> Each owns its own clone on LXC 105. They don't conflict because each
> deploy.sh only touches its own service unit + venv.
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.yaml). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
## When you change a tracked config
@@ -116,7 +117,7 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
- [Backups (disabled)](backups.md)
- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
- [Operations cheatsheet](../../../.agents/operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
## Changelog
@@ -130,7 +131,7 @@ Webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` on [
Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
### 2026-05-13 — Plato pipeline added
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
### 2026-04-28 — wiki entry created
Initial documentation. Six active pipelines.

View File

@@ -1,6 +1,30 @@
# Backups — restic on external drive (DISABLED)
# Backups — restic on external drive (DEPRECATED — superseded)
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md).
> **DEPRECATED 2026-07-01.** Superseded by the **rclone → Proton Drive** off-host mirror on
> [LXC 132 `rclone`](../containers/132-rclone.md). That job finally closes the off-host / 3-2-1 gap
> this page flagged for months. The restic-on-USB job below is kept for archaeology; it has been
> **DISABLED since 2026-04-22** and is not coming back in its old form.
## Current backup — rclone → Proton Drive (LXC 132)
- **Where:** [LXC 132 `rclone`](../containers/132-rclone.md) (`192.168.8.214`), `/mnt/library`
mounted **read-only**.
- **What:** plain `rclone sync` (Proton mirrors local; browsable files, no versioning) of the
folders listed in `/etc/rclone-backup/folders.list`, to `proton:library-backup/…`.
- **When:** monthly — `rclone-backup.timer` (`OnCalendar=*-*-01 03:00`).
- **UI:** rclone Web GUI on `192.168.8.214:5572` (LAN-only, no auth).
- **Encryption:** Proton's built-in E2E (no rclone `crypt` overlay).
- **Runs / logs:** `/var/log/rclone-backup/` + `runs.jsonl`.
- **Still a single off-host target** (Proton only). Not yet a full 3-2-1 (no second independent
copy), but strictly better than the previous "no off-host copy at all."
See [132-rclone](../containers/132-rclone.md) for the full design.
---
## Legacy — restic on external drive (DISABLED 2026-04-22)
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
## Status
@@ -12,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
@@ -81,13 +105,13 @@ Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/R
## Known SPOF
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented.
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) were not implemented before this job was retired — the **off-host copy is now provided by [rclone → Proton Drive (LXC 132)](../containers/132-rclone.md)** instead. A second independent copy is still outstanding.
## Drive history
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
## Thermal monitoring
@@ -95,18 +119,21 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
## Related
- [Hubris host](../hosts/hubris.md)
- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04)
- [Monitoring](monitoring.md)
- [Auto-deploy](auto-deploy.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
## Changelog
### 2026-07-01 — DEPRECATED; superseded by rclone → Proton Drive (LXC 132)
Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclone`](../containers/132-rclone.md) (`/mnt/library` → Proton Drive, monthly, LAN Web GUI). This finally provides the off-host copy the "Known SPOF" note wanted. The restic-on-USB units on hubris remain `disabled` (drive already removed 2026-04-22); page restructured to lead with the current job and demote restic to "Legacy".
### 2026-04-28 — wiki entry created
Initial documentation. Status remains DISABLED.
### 2026-04-22 — DISABLED
Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.

View File

@@ -7,7 +7,7 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
## Components
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/106-auth-outpost.md) (decommissioned 2026-06-04).
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
@@ -33,18 +33,9 @@ address=/photos.hubris.network/192.168.8.175
address=/photos-new.hubris.network/192.168.8.175
address=/artifacto.hubris.network/192.168.8.175
address=/zimaos.hubris.network/192.168.8.175
address=/teddy.hubris.network/192.168.8.175
address=/nfs-export.hubris.network/192.168.8.200
```
**Non-`hubris.network` override (Toniebox device traffic):**
```
address=/prod.de.bb-online.com/192.168.8.243 # → TeddyCloud (131) direct on :443
```
This intercepts Toniebox DNS locally without touching public DNS. The `dns-sync.py` cron on LXC 107 skips non-`hubris.network` records — this entry is Technitium-only.
Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do.
## Why split-horizon
@@ -62,7 +53,7 @@ Creating a new Caddyfile site block is necessary but **not sufficient**. Without
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network``192.168.8.175`.
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../../../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
## Public path — what does and doesn't follow the LAN map
@@ -84,9 +75,6 @@ Either:
## Changelog
### 2026-06-29 — `teddy.hubris.network` added; `prod.de.bb-online.com` override added
TeddyCloud (LXC 131) provisioned. `teddy.hubris.network → 192.168.8.175` (Caddy → TeddyCloud web UI at :8443). Non-hubris override `prod.de.bb-online.com → 192.168.8.243` routes Toniebox device HTTPS traffic directly to TeddyCloud port 443 — this bypasses Caddy and is Technitium-only (dns-sync cron does not replicate non-hubris.network records to the NetBird managed zone).
### 2026-06-28 — `plato.hubris.network` removed
Plato (LXC 126) decommissioned. Technitium entry deleted; dns-sync cron reaped the NetBird managed zone record.
@@ -106,7 +94,7 @@ The "delete NetBird managed zone → forward everything to Technitium" plan was
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
> Reference: [scripts/dns-sync.py](../../../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
@@ -119,10 +107,10 @@ Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backe
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180``192.168.8.2` for any plain-LAN (non-mesh) clients.
Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180``192.168.8.2` for any plain-LAN (non-mesh) clients.
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
@@ -131,7 +119,7 @@ NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
New LAN-only entry for [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.

View File

@@ -5,7 +5,7 @@ Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
self-locating and able to read the same source of truth.
Operational walkthrough for enrolling a new client lives in
[operations/agent-enrollment.md](../operations/agent-enrollment.md); this
[operations/agent-enrollment.md](../../../.agents/operations/agent-enrollment.md); this
page is the architecture reference.
## What's where
@@ -121,7 +121,7 @@ The MCP server and secrets-issuance each have their own clone
## Related
- [Operations: agent enrollment](../operations/agent-enrollment.md) — the
- [Operations: agent enrollment](../../../.agents/operations/agent-enrollment.md) — the
step-by-step for adding a new client
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
pipelines (and the rest of the lab's webhook pipelines)
@@ -133,7 +133,7 @@ The MCP server and secrets-issuance each have their own clone
## Changelog
### 2026-05-20 — system live across hubris, apps, republic-laptop
Phase 1 of the [cross-client context plan](../README.md) merged. Three
Phase 1 of the [cross-client context plan](../../../README.md) merged. Three
clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS
decrypt verified on each. Webhook auto-deploy for both LXC 105 services
wired (hook ids 10 + 11). `homelab refresh-creds` + atomic

View File

@@ -0,0 +1,66 @@
# Infrastructure — cross-cutting systems
The homelab's shared infrastructure: systems that span multiple nodes and
are documented in their own pages. Each system below links to its full doc.
## Network
- **[Network](network.md)** — physical topology, subnets, routing, DHCP.
Homelab `192.168.8.0/24` isolated from household `192.168.178.0/24`.
Proxmox hubris acts as subnet router.
- **[DNS — split-horizon](dns.md)** — Technitium DNS on LXC 107,
`192.168.8.2:53`. `*.hubris.network` resolves to LAN IPs on the homelab
network and to mesh addresses off-LAN.
## Connectivity / mesh
- **[Mesh — Tailscale → Netbird migration](mesh.md)** — overlay networking.
Netbird is the preferred path; Tailscale is legacy.
- **[SSH access](ssh-access.md)** — dual-server SSH (OpenSSH + Netbird SSH)
on hubris, key distribution.
## Public ingress
- **[Public ingress — VPS traefik + cert mirror](ingress.md)** — how home
services reach the open internet. Two-stage: VPS traefik (IONOS) terminates
TLS, proxies over Netbird to home Caddy.
- **[Caddy reverse proxy](../containers/121-caddy.md)** — LAN endpoint.
Terminates TLS for every `*.hubris.network` hostname, forwards to backends.
## Storage
- **[Media permissions — GID 10000 standard](media-permissions.md)** — shared
group permission model across all LXCs that read/write the media library.
- **[Backups — rclone → Proton Drive](backups.md)** — off-host backup strategy.
LXC 132 handles rclone to Proton Drive; restic-on-USB deprecated.
## Identity & access
- **[Authentik SSO](../containers/106-auth-outpost.md)** — identity provider.
Core server runs on the VPS; LAN forward-auth outpost at LXC 106.
OIDC providers configured for Jellyfin, Jellyseerr, Sabnzbd, qBittorrent,
Yuvomi, and more.
## Management & automation
- **[Homelab context distribution](homelab-context.md)** — `/opt/homelab-context`
clone, MCP server, secrets issuance, cross-client sync.
- **[Auto-deploy — gitea-webhook pipelines](auto-deploy.md)** — push-to-deploy
for Caddy config, mule-image, and other tracked repos.
- **[Monitoring](monitoring.md)** — health checks, watchdogs, alerting
(migrated from claudio-bot to Hermes cron).
- **[VPS hardening](vps-hardening.md)** — IONOS netbird VPS: fail2ban,
nftables, OIDC SSH, security posture.
## Topology
- **[Topology diagram (generated)](topology.md)** — Mermaid graph of compute,
ingress routing, and storage mounts. Auto-generated from `inventory.yaml`
by `oikos/gen-topology.py` (Python — Go DB-native replacement planned).
## Related
- [README](../../../README.md) — entry point
- [Containers index](../containers/index.md)
- [Operations cheatsheet](../../../.agents/operations/commands.md)
- [OIKOS operating model](../../../.agents/OIKOS.md)

View File

@@ -49,7 +49,7 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
Since 2026-05-31 [Authentik runs on the VPS itself](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
@@ -91,7 +91,7 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public``192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
Authentik migrated onto the VPS ([investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
### 2026-04-28 — wiki entry created
Initial documentation.

View File

@@ -62,6 +62,7 @@ Every LXC that mounts `/mnt/library` participates in a shared `media` group with
| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data |
| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
| 130 | [grimmory](../containers/130-grimmory.md) | priv | Docker container uses `GROUP_ID=10000` env var (linuxserver pattern) — no in-LXC group needed |
| 132 | [rclone](../containers/132-rclone.md) | priv | **read-only** mount; runs as root → reads all subtrees. No media group needed |
> Some entries from earlier snapshots — 100 (arr-yunohost), 107 (marimo), 109 (syncthing), 110 (photoprism), 112 (immich), 116 (heaper) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).

View File

@@ -109,7 +109,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Related
- [DNS split-horizon](dns.md)
- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides
- [Authentik (124)](../containers/106-auth-outpost.md) — the IdP that triggers most of these overrides
- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B
- [Gitea (104)](../containers/104-gitea.md) — example of Technique A
- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport
@@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Changelog
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik``192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md).
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik``192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
@@ -139,11 +139,11 @@ The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Ca
### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/106-auth-outpost.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/124-authentik.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/106-auth-outpost.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
**Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21):

View File

@@ -64,7 +64,7 @@ No NAT on Proxmox — traffic flows without double-NAT.
## Remote access
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../vps/).
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../../../vps/).
- **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md).
## Related
@@ -79,10 +79,10 @@ No NAT on Proxmox — traffic flows without double-NAT.
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2)
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed.
Previous pool `.100.240` overlapped with all static LXCs/VMs (` .101.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
Previous pool `.100.240` overlapped with all static LXCs/VMs (` .101.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-06-02 — Executed migration; Proxmox as subnet router
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100240`. Caddy service unit was missing and recreated. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).

View File

@@ -97,19 +97,23 @@ When onboarding a new machine:
## Hosts
### Hubris (PVE host)
### Hubris + strong (PVE cluster: `Homelab`)
| Detail | Value |
|--------|-------|
| LAN IP | `192.168.8.77` |
| Netbird | `100.122.38.109` (FQDN: `proxmox-server.netbird.selfhosted`) |
| Netbird SSH port | `22022` (mesh-only, OIDC auth) |
| SSH user | `root` |
| Authorized keys | `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced) |
Both nodes share `/etc/pve/priv/authorized_keys` — it's Proxmox
cluster-synced, so a key added on either node is authorized on both.
Authorized root keys currently deployed:
| Detail | hubris | strong |
|--------|--------|-----------|
| LAN IP | `192.168.8.77` | `192.168.178.181` |
| Cluster node name | `hubris` | `strong` (OS hostname kept as-is from install) |
| Netbird | `100.122.38.109` (`proxmox-server.netbird.selfhosted`) | not enrolled yet |
| Netbird SSH port | `22022` (mesh-only, OIDC auth) | n/a |
| SSH user | `root` | `root` |
Authorized root keys currently deployed (cluster-wide):
- `root@hubris` (self, RSA)
- `d.toro.v@pm.me` (ed25519) — mac-mini
- `root@strong` (RSA) — strong's own key, added 2026-07-01 for the cluster join
### LXCs
@@ -138,7 +142,8 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
|------|----|--------|--------------|----------|
| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
| ludo-mini | Linux | `192.168.8.133` | `ludo-mini.netbird.selfhosted` | TBD |
strong moved out of this table 2026-07-01 — it's a Proxmox host now, see the cluster table above.
### VPS (external)
@@ -173,11 +178,14 @@ done
- [Mesh migration](mesh.md)
- [VPS hardening](vps-hardening.md)
- [Agent enrollment](../operations/agent-enrollment.md)
- [Homelab CLI](../bin/homelab)
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
- [Homelab CLI](../../../bin/homelab)
## Changelog
### 2026-07-01 — strong reformatted to Proxmox, joined cluster; table corrected
strong moved from the Workstations table to the PVE-cluster table (was showing a stale `192.168.8.133`, never actually reachable — the real LAN IP has always been `192.168.178.181`, matching hosts/strong.yaml). Root key access bootstrapped via one-time console password, then key-only going forward. See [hosts/hubris.md#cluster](../hosts/hubris.md#cluster) and [hosts/strong.md](../hosts/strong.md).
### 2026-06-02 — universal SSH reachability
Replaced ad-hoc per-workstation SSH configs with inventory-generated

View File

@@ -0,0 +1,96 @@
<!-- Generated by oikos/gen-topology.py from inventory.yaml. -->
<!-- Do NOT edit by hand - your changes will be overwritten. -->
# Topology (generated)
Source: [inventory.yaml](../../../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services.
Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). Operating model: [OIKOS.md](../../../.agents/OIKOS.md).
## Compute & ingress
```mermaid
flowchart LR
subgraph hubris_sub["hubris (Proxmox)"]
trmnl["trmnl<br/>LXC 128<br/>trmnl-middleware<br/>192.168.8.211"]
nfs_export["nfs-export<br/>LXC 102<br/>storage-export<br/>192.168.8.200"]
paperless["paperless<br/>LXC 103<br/>document-archive<br/>192.168.8.130"]
gitea["gitea<br/>LXC 104<br/>git-server<br/>192.168.8.121"]
apps["apps<br/>LXC 105<br/>docker-apps<br/>192.168.8.205"]
auth_outpost["auth-outpost<br/>LXC 106<br/>authentik-gateway<br/>192.168.8.6"]
dns["dns<br/>LXC 107<br/>dns-server<br/>192.168.8.2"]
nextcloud["nextcloud<br/>LXC 114<br/>file-sync<br/>192.168.8.224"]
sophia["sophia<br/>LXC 119<br/>workshop<br/>192.168.8.109"]
mule_images["mule-images<br/>LXC 120<br/>photo-management<br/>192.168.8.136"]
caddy["caddy<br/>LXC 121<br/>reverse-proxy<br/>192.168.8.175"]
teddycloud["teddycloud<br/>LXC 131<br/>teddycloud<br/>192.168.8.150"]
zimaos["zimaos<br/>VM 100<br/>nas-frontend-eval<br/>192.168.8.195"]
haos["haos<br/>VM 108<br/>home-automation<br/>192.168.8.101"]
end
subgraph strong_sub["strong (Proxmox)"]
house["house<br/>LXC 129<br/>family-planner<br/>192.168.8.244"]
jellyfin["jellyfin<br/>LXC 101<br/>media-server<br/>192.168.8.246"]
elementsynapse["elementsynapse<br/>LXC 118<br/>matrix-server<br/>192.168.8.242"]
arriman["arriman<br/>LXC 122<br/>arr-stack<br/>192.168.8.245"]
grimmory["grimmory<br/>LXC 130<br/>book-library<br/>192.168.8.247"]
seanime["seanime<br/>LXC 133<br/>anime-media-server<br/>192.168.8.248"]
romm["romm<br/>LXC 134<br/>rom-manager<br/>192.168.8.249"]
end
rclone["rclone<br/>lxc<br/>backup"]
republic_laptop([republic-laptop<br/>workstation<br/>primary-dev])
mac_mini([mac-mini<br/>workstation<br/>dev<br/>192.168.178.182])
netbird_vps[[netbird-vps<br/>external<br/>netbird-mgmt]]
url_artifacto(["artifacto.hubris.network"]) -->|routes-to| apps
url_authentik(["auth.hubris.network"]) -->|routes-to| netbird_vps
url_gitea(["git.hubris.network"]) -->|routes-to| gitea
url_homelab_mcp(["mcp.hubris.network"]) -->|routes-to| apps
url_jellyfin(["media.hubris.network"]) -->|routes-to| jellyfin
url_matrix(["matrix.hubris.network"]) -->|routes-to| elementsynapse
url_nextcloud(["cloud.hubris.network"]) -->|routes-to| nextcloud
url_paperless(["paperless.hubris.network"]) -->|routes-to| paperless
url_photos(["photos.hubris.network"]) -->|routes-to| mule_images
url_proxmox_ui(["proxmox.hubris.network"]) -->|routes-to| hubris_sub
url_secrets_issuance(["secrets.hubris.network"]) -->|routes-to| apps
url_teddycloud(["teddy.hubris.network"]) -->|routes-to| teddycloud
url_trmnl(["trmnl.hubris.network"]) -->|routes-to| trmnl
url_zimaos(["zimaos.hubris.network"]) -->|routes-to| zimaos
```
## Storage (mounts)
```mermaid
flowchart LR
mnt_library[("/mnt/library")]
mnt_media_local[("/mnt/media_local")]
mnt_media_local_anime[("/mnt/media_local/anime")]
apps["apps"] -->|mounts| mnt_library
arriman["arriman"] -->|mounts| mnt_media_local
gitea["gitea"] -->|mounts| mnt_library
grimmory["grimmory"] -->|mounts| mnt_media_local
hubris["hubris"] -->|mounts| mnt_library
jellyfin["jellyfin"] -->|mounts| mnt_media_local
mule_images["mule-images"] -->|mounts| mnt_library
nextcloud["nextcloud"] -->|mounts| mnt_library
paperless["paperless"] -->|mounts| mnt_library
romm["romm"] -->|mounts| mnt_media_local
seanime["seanime"] -->|mounts| mnt_media_local_anime
sophia["sophia"] -->|mounts| mnt_library
teddycloud["teddycloud"] -->|mounts| mnt_library
```
## Archaeology (destroyed nodes)
| Node | ID | Destroyed | Reason |
|---|---|---|---|
| plato | 126 | 2026-06-28 | notes workspace decommissioned; data retained at /mnt/library/documents/plato |
| claudio-bot | 123 | 2026-06-04 | replaced by Hermes Agent on mac-mini; monitoring moved to homelab-health-watchdog cron |
| mule-photos-new | 127 | 2026-05-22 | PhotoPrism test stack promoted to LXC 120 (Mulimage 2.0 merge) |
| heaper | 116 | 2026-05-14 | decommissioned; data retained at /mnt/library/heaper |
| syncthing | 109 | 2026-05-14 | decommissioned; library subtree was empty |
| seafile | 125 | 2026-05-13 | Seafile Pro evaluation rejected; files.hubris.network removed from caddy + dns |
| arr-yunohost | 100 | 2026-04-28 | migrated to docker stack on arriman (LXC 122) |
| flaresolverr | 106 | 2026-04-28 | folded into the arriman docker compose |
| marimo | 107 | 2026-04-28 | decommissioned |
| photoprism | 110 | 2026-04-28 | replaced by mule-images (LXC 120) |
| karakeep | 111 | 2026-04-28 | decommissioned |
| immich | 112 | 2026-04-28 | replaced by mule-images (LXC 120) |
| reticulum | 115 | 2026-04-28 | decommissioned |

View File

@@ -2,12 +2,12 @@
## Summary
[`hubris`](../hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
## Timeline
### 2026-04-19 — drive attached
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
### 2026-04-19 → 2026-04-21 — first crashes
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
@@ -24,13 +24,13 @@ Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
### 2026-04-22 — recurrence after 30 h 37 m
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
### 2026-04-22 — `cpu-epp.service` design bug fixed
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
### 2026-04-22 — drive removed (A/B test)
User physically removed the external USB drive. [Backup timers disabled](../infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
### 2026-04-23 — SSD cooling + thermal pads installed
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
@@ -78,9 +78,9 @@ Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the Nu
| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive |
## Affected nodes
- [Hubris host](../hosts/hubris.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Monitoring](../infrastructure/monitoring.md)
- [Hubris host](../../../wiki/hosts/hubris.md)
- [Backups (disabled)](../../../wiki/infrastructure/backups.md)
- [Monitoring](../../../wiki/infrastructure/monitoring.md)
## Open questions
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)

View File

@@ -2,9 +2,9 @@
## Summary
The NetBird management server (on the [VPS](../infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
Resolved by moving Authentik off [LXC 124](../containers/124-authentik.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
@@ -43,7 +43,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../containers/124-authentik.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
@@ -51,7 +51,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
| | Before | After |
|---|---|---|
| Authentik host | [LXC 124](../containers/124-authentik.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
| Version | `2026.2.2` | `2026.5.2` |
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
@@ -76,7 +76,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
@@ -99,11 +99,11 @@ The policy was orphaned (no matched type data or had incompatible evaluation). R
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
- **Reconcile [124-authentik](../containers/124-authentik.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
## Related
- [124 — authentik](../containers/124-authentik.md)
- [DNS split-horizon](../infrastructure/dns.md)
- [Public ingress (VPS traefik)](../infrastructure/ingress.md)
- [Mesh migration](../infrastructure/mesh.md)
- [124 — authentik](../../../wiki/containers/106-auth-outpost.md)
- [DNS split-horizon](../../../wiki/infrastructure/dns.md)
- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md)
- [Mesh migration](../../../wiki/infrastructure/mesh.md)

View File

@@ -2,7 +2,7 @@
## Summary
[`ludo-mini`](../hosts/ludo-mini.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
## Timeline

View File

@@ -90,9 +90,9 @@ print("session_duration:", stage.session_duration) # → "days=30"
## Related
- [Container 106 — auth-outpost](../containers/106-auth-outpost.md)
- [Authentik VPS migration](2026-05-31-authentik-vps-migration.md)
- [Ingress (VPS Traefik)](../infrastructure/ingress.md)
- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md)
- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md)
- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md)
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
## Changelog

View File

@@ -54,8 +54,8 @@ This is the same class of drift as the June 5th incidents (paperless, HAOS, apps
## Related
- [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md)
- [Caddy (121)](../containers/121-caddy.md)
- [elementsynapse (118)](../containers/118-elementsynapse.md)
- [dns-sync script](../scripts/dns-sync.py)
- [check-caddy-backends script](../scripts/check-caddy-backends.sh)
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
- [Caddy (121)](../../wiki/containers/121-caddy.md)
- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md)
- [dns-sync script](../../../scripts/dns-sync.py)
- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh)

View File

@@ -0,0 +1,29 @@
# Investigations
Time-stamped incident reports and experiments. One entry per incident; the entry is the canonical source. Per-node changelog entries link back here.
## Active / recent
| Date | Title | Status |
| ------------ | ------------------------------------------------------------------ | ------------- |
| 2026-06-06 | [Caddyfile truncation incident](2026-06-06-caddyfile-truncation.md) | Resolved — permanent safeguards deployed (site-count guard, auto-stash, auto-backup) |
| 2026-06-06 | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
| 2026-06-03 | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
| 2026-06-01 | [Mac-mini onboarding](2026-06-01-mac-mini-onboarding.md) | Onboarded |
## Resolved (archived)
See [`archive/`](archive/):
| Date | Title |
| ------------ | ------------------------------------------------------------------ |
| 2026-04-21 | [Hubris crash loop — thermal + USB drive](archive/2026-04-21-hubris-crash-loop.md) |
| 2026-05-31 | [Authentik migrated from LXC 124 to the VPS](archive/2026-05-31-authentik-vps-migration.md) |
## Conventions
- File name: `YYYY-MM-DD-<slug>.md`. Use the *first* date if the incident spans multiple days.
- Mandatory sections: Summary, Timeline, Root cause, Mitigations applied, Open questions.
- Update the entry as the situation evolves; never rewrite history. Add new dated sections at the bottom.
- Link back from every node's changelog that's affected.
- Move to `archive/` when the incident is fully resolved and no longer actively referenced.

10
archive/knowledge/log.md Normal file
View File

@@ -0,0 +1,10 @@
# Knowledge — operations log
Append-only record of documentation-maintenance operations on the knowledge wiki (restructures,
source ingests, lint sweeps). One line per operation, newest last. Infrastructure changes belong in
each page's `## Changelog` and the Oikos change ledger, not here.
## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output.
## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md.
## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference.
## [2026-07-06] restructure | Phase 6 consolidation: investigations/ -> knowledge/sources/investigations/; operations/ -> .agents/operations/; HERMES.md -> .agents/; deleted root OIKOS/CAVEMAN/CONTRIBUTING stubs.

View File

@@ -0,0 +1,351 @@
# Current cert sync script + traefik dynamic config
Snapshot of the two artifacts that control public service exposure as of
2026-07-05. Updated 2026-07-05: fixed Jellyfin backend from dead hubris IP
(192.168.8.206) to new strong IP (192.168.8.246).
## hubris-public-cert-sync.sh (PVE host, `/usr/local/bin/`)
```bash
#!/bin/bash
# Mirrors home caddy's LE certs for publicly-exposed hubris.network hostnames
# into the VPS traefik's /letsencrypt volume. Traefik file-watches the volume
# and hot-reloads.
#
# Why: netbird-proxy's HostSNI(*) TCP passthrough intercepts ACME TLS-ALPN-01
# challenges before traefik's allowACMEByPass can respond, so traefik can't
# obtain its own cert. Home caddy uses IONOS DNS-01 (no such conflict);
# we just mirror what it already has.
#
# Runs daily via hubris-public-cert-sync.timer.
set -euo pipefail
CADDY_LXC=121
CADDY_BASE=/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory
VPS_HOST=root@100.122.165.149
VPS_DEST=/var/lib/docker/volumes/opt_netbird_traefik_letsencrypt/_data
# Map: source hostname -> "crt_filename key_filename" on the VPS.
# Stable names so traefik dynamic.yaml doesn't need edits on renewal.
declare -A HOSTS=(
[artifacto.hubris.network]="fullchain.crt privkey.key"
[blog.hubris.network]="blog.fullchain.crt blog.privkey.key"
[trmnl.hubris.network]="trmnl.fullchain.crt trmnl.privkey.key"
[sso.hubris.network]="sso.fullchain.crt sso.privkey.key"
[media.hubris.network]="media.fullchain.crt media.privkey.key"
[paperless.hubris.network]="paperless.fullchain.crt paperless.privkey.key"
)
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
for host in "${!HOSTS[@]}"; do
read -r crt_name key_name <<< "${HOSTS[$host]}"
pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.crt" "$TMP/$crt_name"
pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.key" "$TMP/$key_name"
if ssh -o BatchMode=yes "$VPS_HOST" "test -f $VPS_DEST/$crt_name && diff -q - $VPS_DEST/$crt_name" < "$TMP/$crt_name" >/dev/null 2>&1; then
echo "hubris-public-cert-sync: $host unchanged"
continue
fi
scp -q -o BatchMode=yes "$TMP/$crt_name" "$TMP/$key_name" "$VPS_HOST:$VPS_DEST/"
echo "hubris-public-cert-sync: shipped $host ($(openssl x509 -in "$TMP/$crt_name" -noout -enddate))"
done
```
### Adding a new host
1. Caddy must already have the cert (verify `pct exec 121 -- ls "$CADDY_BASE/$host/"`)
2. Add a line to the HOSTS array: `[new-host.hubris.network]="nickname.fullchain.crt nickname.privkey.key"`
3. `systemctl start hubris-public-cert-sync.service` to sync immediately
4. Verify certs landed: `ssh "$VPS_HOST" "ls -la $VPS_DEST/nickname.*"`
5. Add matching `tls.certificates` entry in traefik dynamic config
---
## traefik-dynamic.yaml (VPS, `/opt/`)
```yaml
tcp:
serversTransports:
pp-v2:
proxyProtocol:
version: 2
tls:
certificates:
- certFile: /letsencrypt/fullchain.crt
keyFile: /letsencrypt/privkey.key
- certFile: /letsencrypt/blog.fullchain.crt
keyFile: /letsencrypt/blog.privkey.key
- certFile: /letsencrypt/trmnl.fullchain.crt
keyFile: /letsencrypt/trmnl.privkey.key
- certFile: /letsencrypt/sso.fullchain.crt
keyFile: /letsencrypt/sso.privkey.key
- certFile: /letsencrypt/media.fullchain.crt
keyFile: /letsencrypt/media.privkey.key
- certFile: /letsencrypt/paperless.fullchain.crt
keyFile: /letsencrypt/paperless.privkey.key
http:
routers:
artifacto-public:
rule: 'Host(`artifacto.hubris.network`) && (PathPrefix(`/p/`) || PathPrefix(`/static/`) || Path(`/healthz`))'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- artifacto-strip-sso
- artifacto-ratelimit
service: artifacto-public
blog-public:
rule: 'Host(`blog.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- blog-ratelimit
service: blog-public
trmnl-public:
rule: 'Host(`trmnl.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- trmnl-ratelimit
service: trmnl-public
matrix-public:
rule: 'Host(`matrix.hubris.network`) && !PathPrefix(`/.well-known/matrix/`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- matrix-ratelimit
service: matrix-public
matrix-wellknown:
rule: 'Host(`matrix.hubris.network`) && (PathPrefix(`/.well-known/matrix/`) || PathPrefix(`/.well-known/acme-challenge/`))'
entryPoints:
- websecure
priority: 20
tls:
certResolver: letsencrypt
service: matrix-wellknown-svc
house-public:
rule: Host(`house.hubris.network`)
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- house-ratelimit
service: house-public
sso-public:
rule: 'Host(`sso.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- sso-ratelimit
service: sso-public
media-public:
rule: 'Host(`media.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- media-ratelimit
service: media-public
paperless-api-public:
rule: 'Host(`paperless.hubris.network`) && PathPrefix(`/api/`)'
entryPoints:
- websecure
priority: 20
tls:
certResolver: letsencrypt
middlewares:
- paperless-ratelimit
service: paperless-public
paperless-public:
rule: 'Host(`paperless.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- authentik-forwardauth
- paperless-ratelimit
service: paperless-public
middlewares:
artifacto-strip-sso:
headers:
customRequestHeaders:
X-Artifacto-Gateway: ""
X-Authentik-Username: ""
X-Authentik-Groups: ""
X-Authentik-Email: ""
X-Authentik-Name: ""
X-Authentik-Uid: ""
X-Authentik-Jwt: ""
X-Authentik-Meta-Jwks: ""
X-Authentik-Meta-Outpost: ""
X-Authentik-Meta-Provider: ""
X-Authentik-Meta-App: ""
X-Authentik-Meta-Version: ""
admin-allowlist:
ipAllowList:
sourceRange:
- "5.61.168.0/24"
artifacto-ratelimit:
rateLimit:
average: 50
period: 1s
burst: 100
blog-ratelimit:
rateLimit:
average: 100
period: 1s
burst: 200
trmnl-ratelimit:
rateLimit:
average: 20
period: 1s
burst: 40
matrix-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
house-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
sso-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
media-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
paperless-ratelimit:
rateLimit:
average: 20
period: 1s
burst: 40
authentik-forwardauth:
forwardAuth:
address: "http://192.168.8.6:9000/outpost.goauthentik.io/auth/traefik"
trustForwardHeader: true
authResponseHeaders:
- X-authentik-username
- X-authentik-groups
- X-authentik-email
- X-authentik-name
- X-authentik-uid
- X-authentik-jwt
- X-authentik-meta-jwks
- X-authentik-meta-outpost
- X-authentik-meta-provider
- X-authentik-meta-app
- X-authentik-meta-version
services:
artifacto-public:
loadBalancer:
servers:
- url: 'http://192.168.8.205:3100'
blog-public:
loadBalancer:
servers:
- url: 'http://192.168.8.205:8080'
trmnl-public:
loadBalancer:
servers:
- url: 'http://192.168.8.211:9851'
matrix-public:
loadBalancer:
servers:
- url: 'http://192.168.8.242:8008'
matrix-wellknown-svc:
loadBalancer:
servers:
- url: 'http://matrix-wellknown:80'
house-public:
loadBalancer:
servers:
- url: 'http://192.168.8.244:3000'
sso-public:
loadBalancer:
servers:
- url: 'http://192.168.8.6:9000'
media-public:
loadBalancer:
servers:
- url: 'http://192.168.8.246:8096'
paperless-public:
loadBalancer:
servers:
- url: 'http://192.168.8.130:8000'
```
### Adding a new service — four blocks needed
1. **Router**`http.routers.<name>-public` with `tls: {}` (not
`certResolver`)
2. **Middleware** — rate limit, one per service
3. **Service**`http.services.<name>-public` with the backend URL
4. **tls.certificates** — add a new `- certFile/keryFile` pair matching the
cert sync HOSTS entry
### Key file naming convention
| Cert name | Host | Convention |
|-----------|------|------------|
| `fullchain.crt` + `privkey.key` | `artifacto.hubris.network` | First service — no prefix |
| `blog.fullchain.crt` + `blog.privkey.key` | `blog.hubris.network` | `{nickname}.fullchain.crt` |
| `trmnl.fullchain.crt` + `trmnl.privkey.key` | `trmnl.hubris.network` | `{nickname}.fullchain.crt` |
| `sso.fullchain.crt` + `sso.privkey.key` | `sso.hubris.network` | `{nickname}.fullchain.crt` |
| `media.fullchain.crt` + `media.privkey.key` | `media.hubris.network` | `{nickname}.fullchain.crt` |
| `paperless.fullchain.crt` + `paperless.privkey.key` | `paperless.hubris.network` | `{nickname}.fullchain.crt` |
### ⚠️ Critical — keep backends in sync after LXC migrations
When moving an LXC between Proxmox nodes, update **both**:
1. **Caddy** (`/etc/caddy/Caddyfile` on LXC 121)
2. **VPS traefik** (`/opt/traefik-dynamic.yaml` — via hubris bridge SSH)
Jellyfin migration from hubris to strong (2026-07-05) was fixed in Caddy
but **missed** in VPS traefik — old IP `192.168.8.206` remained. This caused
Bad Gateway for off-LAN users. Use Python-based editing (see
`references/traefik-config-editing.md`) for accurate surgical fixes.

View File

@@ -0,0 +1,9 @@
# Sources
Immutable evidence the wiki synthesizes from. External reference docs live under `references/`;
incident evidence lives in [`investigations/`](investigations/index.md) (written once at
incident time, then linked from the changelogs of the nodes they implicate).
| Slug | Reference | Summary |
|------|-----------|---------|
| cert-sync-and-traefik-config | [references/cert-sync-and-traefik-config.md](references/cert-sync-and-traefik-config.md) | VPS traefik config and the LAN↔VPS certificate mirror. |

View File

@@ -46,7 +46,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g.
## Open items
- **DHCP → static IP fixed (2026-06-03).** ZimaOS IP drifted from `.195` (Slate AX) → `.103` (Technitium) after the DHCP migration, causing Caddy 502s. Fixed by injecting a static systemd-networkd config and restarting the VM. IP now pinned at `192.168.8.195`. See [changelog](#2026-06-03--static-ip-set-to-195-dhcp-drift-fixed).
- **No Authentik wiring.** [authentik (124)](../containers/124-authentik.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
- **No Authentik wiring.** [authentik (124)](../containers/106-auth-outpost.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
- **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up.
- **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor``screendump`.
@@ -59,7 +59,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g.
## Changelog
### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed
ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix)
@@ -87,4 +87,4 @@ Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs mod
Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`.
### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1)
`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/124-authentik.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.
`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/106-auth-outpost.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.

View File

@@ -7,7 +7,7 @@ Home Assistant OS — the only VM on hubris (HAOS doesn't run cleanly in an LXC,
- **HAOS version:** 16.3 (last verified)
- **IP:** `192.168.8.101`
- **Resources:** 4 GiB RAM, 32 GiB boot disk
- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `192.168.8.101:8123`
- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.101:8123`
## Auth
@@ -18,7 +18,7 @@ Key gotchas:
```
ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1"
```
so OIDC discovery resolves internally to [authentik (124)](124-authentik.md).
so OIDC discovery resolves internally to [authentik (124)](../containers/106-auth-outpost.md).
- Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen).
- YAML config:
- `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created).
@@ -30,8 +30,8 @@ Key gotchas:
HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting.
## Related
- [Authentik (124)](124-authentik.md)
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](../containers/106-auth-outpost.md)
- [Caddy (121)](../containers/121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)

View File

@@ -0,0 +1,14 @@
# VMs — index
Two QEMU VMs running on [hubris](../hosts/hubris.md):
| ID | Name | Role | IP | Public hostname |
|----|------|------|----|-----------------|
| 100 | [zimaos](100-zimaos.md) | NAS frontend eval (ZimaOS) | `192.168.8.195` | [`zimaos.hubris.network`](../infrastructure/dns.md) |
| 108 | [haos-16.3](108-haos.md) | Home automation (HAOS) | `192.168.8.101` | [`home.hubris.network`](../infrastructure/dns.md) |
## Related
- [Hubris host](../hosts/hubris.md) — both VMs run here
- [Containers index](../containers/index.md) — LXCs on both nodes
- [README](../../../README.md)

View File

@@ -0,0 +1,5 @@
{"ts": "2026-07-06T11:05:35+00:00", "agent": "mac-mini", "entity": "host:teddycloud", "action": "activate", "risk": "config_mutation", "verification": "homelab node teddycloud relations", "result": "ok"}
{"ts": "2026-07-06T11:15:04+00:00", "agent": "mac-mini", "entity": "repo:Homelab-Docs", "action": "register-webhook", "risk": "config_mutation", "result": "ok", "notes": "webhook id 14 for oikos-console deploy"}
{"ts": "2026-07-06T11:29:56+00:00", "agent": "mac-mini", "entity": "service:caddy", "action": "add-site-block", "risk": "config_mutation", "verification": "curl -s https://git.hubris.network (unrelated route still healthy after reload)", "result": "ok", "notes": "oikos.hubris.network -> 192.168.8.205:8091, Authentik-gated, in dtoro/caddy-conf@c195142"}
{"ts": "2026-07-06T11:40:28+00:00", "agent": "mac-mini", "entity": "host:dns", "action": "add-record", "risk": "config_mutation", "verification": "dig @192.168.8.2 +short oikos.hubris.network", "result": "ok", "notes": "oikos.hubris.network A -> 192.168.8.175 (Caddy LAN IP), via Technitium API, no token persisted"}
{"ts": "2026-07-06T11:57:21+00:00", "agent": "mac-mini", "entity": "host:apps", "action": "deploy-oikos-console", "risk": "config_mutation", "verification": "curl http://127.0.0.1:8091/ on apps -> 200; https://oikos.hubris.network/ -> 302 (Authentik gate)", "result": "ok"}

View File

@@ -0,0 +1,21 @@
# apps (host:apps)
- kind: lxc (LXC 105)
- state: active
- runs-on: host:hubris
- role: docker-apps
- address: 192.168.8.205 (mesh: tailscale:apps)
- mounts: /mnt/library
- doc: knowledge/wiki/containers/105-apps.md
- secrets: enrolled (age key present)
## Blast radius
- impacts: service:artifacto, service:homelab_mcp, service:secrets_issuance
- affected by: host:hubris, mount:/mnt/library, repo:dtoro/Artifacto, repo:dtoro/Homelab-Docs
- full blast radius: service:artifacto, service:homelab_mcp, service:secrets_issuance
## Safe actions
- see the services this host runs for action-level risk classes
## Recent changes
- 2026-07-06T11:57:21+00:00 deploy-oikos-console (config_mutation) — ok

View File

@@ -0,0 +1,20 @@
# arriman (host:arriman)
- kind: lxc (LXC 122)
- state: active
- runs-on: host:strong
- role: arr-stack
- address: 192.168.8.245 (mesh: tailscale:arr)
- mounts: /mnt/media_local
- doc: knowledge/wiki/containers/122-arriman.md
## Blast radius
- impacts: service:arr_stack
- affected by: host:strong, mount:/mnt/media_local
- full blast radius: service:arr_stack
## Safe actions
- see the services this host runs for action-level risk classes
## Recent changes
- (none yet)

Some files were not shown because too many files have changed in this diff Show More