- ApprovalService (core/app/approval.go) + ApprovalRepo (postgres adapter) with
full decide transaction: HMAC token verify, approval flip, execution un-gate,
session-scoped window keys (+session suffix matching GovernanceStore gate),
nomos session flip, audit+event on failure abort. httpapi DecideApproval now
a thin presenter delegating to the service. ListPending payload format fixed
(json.Unmarshal not raw-wrap).
- execlog folded into postgres adapter: internal/execlog deleted, NewExecutionLog
/ ReadExecutionLog live in the db package, callers updated (mcp, httpapi).
- execworker poller over ExecutionService.DispatchQueued: advisory lock leak
fixed (defer/recover per execution), correlation_id preserved via Finalize
event emission (ExecRunRepo.Finalize now emits execution.{status} with
correlation_id from the row).
- Phase 8 session export-rename completed: Store, New, and all 53 methods
exported; cmd/nomos/ agent.go fixed to use session.PendingContinuation etc.
- Coverage gates: ExecutionService.Submit 93.1%, PolicyService.Decide 100%.
- Plans index updated, VERSION bumped to 0.36.0.
archive/ contained the old narrative wiki (superseded by DB as source
of truth), hermes-plans, oikos-cards, ledger, secrets-issuance, and
SOPS backups — all Python-era artifacts with no ongoing value.
Updated all cross-references in:
- AGENTS.md, README.md
- .agents/operations/commands.md (point to docs/infrastructure/)
- .agents/shared/llm-wiki.md, page-templates.md
- .agents/domains/knowledge/schema.md, operations/schema.md
- .agents/skills/*/SKILL.md
- docs/infrastructure/*.md (removed archive link targets)
- docs-lint/SKILL.md known-baseline note
Infisical is the sole source of truth for secrets; SOPS DR backup
is no longer needed. Docs (README, rotation.md) already moved to
docs/secrets/ in prior commit.
Architecture has changed drastically (hexagonal refactor, web client
extraction). Every active plan has been reviewed, annotated with
'Completed' or 'Won't do' status, and moved to plans/done/.
Completed (7): gaps-and-improvements, liveness-drift, gated-execution,
nomos-code-review, codebase-cleanup, mascot-physics, backend-eval
Won't do (7): prometheus-lxc, control-room-webui, activity-gaps,
activity-timeline, frontend-os-apps, haos-capability-gaps, arr-audit
Mechanical extraction of nomos internal components into plain-Go subpackages
per the hexagonal plan (ADR 0016 §3.1 rule 3):
turngate/ — per-session turn serialization (plan 2026-08-03 F1)
retrycap/ — per-turn run retry cap (maxRunRetries=3)
messagequeue/ — operator-message queue for busy-turn re-entry (F2)
assent/ — chat-assent detection (isAssent, isTypedConfirmation,
ExtractPendingApprovals), decoupled from agent via
[]string input instead of persistedCall
session/ — store (chat sessions, plan execution, DB persistence),
migration runner + local emitEvent to break adapter
dependency
internal/migrate/ — shared migration runner extracted from postgres pool,
used by both the oikos postgres adapter and session tests.
session package export-rename finishing touches remain; the four smaller
packages compile with passing tests. Depguard rules and ADR-0016 leaf-note
update deferred to a followup. VERSION 0.35.1.
classifyAndGate's decision pipeline moves to core: PolicyService runs the
full gate order (classify + transport escalation, plan-first, syntax,
host-only/host-lxc, VM QGA preflight, dedup, approval-flood, window
routing) over ports.GovernanceStore; ExecutionService records and
dispatches (auto-run via ssh.CommandExecutor + TargetResolver, queue via
ExecutionRecorder) with one converged path for run/docker_exec. Gating
matrix test added (risk x window x declared risk -> outcome); pair
coverage 95.6%.
Bug fix surfaced by the matrix: the flag-space syntax regex was inverted
— it refused valid 'tail -n 3' and missed the actual 'head - n' typo.
Fixed to match dash-space-value only.
Remaining Phase 4 items tracked in the plan: ApprovalService.Decide
convergence, execlog fold, execworker poller. VERSION 0.35.0.
Final phase of the hexagonal architecture refactor (plans/2026-08-15-hexagonal-architecture.md). Verifies depguard rules, updates CONTRIBUTING.md to reflect final architecture, marks plan as complete, and bumps minor version to 0.34.0.
Problem: knowledge and learning operations were scattered across
httpapi and mcp handlers with no shared service layer. The hexagonal
refactor needs a single use-case service for both surfaces.
Change:
- app/knowledge.go: KnowledgeService (Search, Upsert, GetContent,
SoftDelete, Restore) and LearningService (ListPatterns,
UpsertPattern, Validate, Quarantine) wrapping the port interfaces.
- adapters/postgres/knowledge.go: KnowledgeRepo implements
KnowledgeRepository — Search, GetContent, Upsert, SoftDelete,
Restore with inline SQL matching the existing handler patterns
(full-text search ILIKE, upsert on conflict, soft-delete).
Verification: go build/vet, full test suite (18 pkgs), DB integration
(postgres + mcp — green).
Problem: signal lifecycle (upsert, resolve, health aggregation) and
observe-pass orchestration (load checks, resolve targets, run probes,
aggregate health) were embedded in scheduler/scheduler.go — 1095 lines
of monolith with no port abstraction.
Change:
- app/signals.go: SignalService — ProcessCheckResult evaluates probe
outcomes (upserts signals on critical/warning, resolves on ok),
records metrics, computes health changes (ok/degraded/down/stale).
WorstHealthForTarget aggregates open signals into entity health.
- app/observation.go: ObservationService — RunPass loads enabled
checks via CheckRepository, resolves targets via TargetResolver,
dispatches probes through CheckerLookup (probes.Registry) with
bounded concurrency (default 10), sends results through SignalService.
- adapters/postgres/signals.go: MetricsRepo (InsertSamples via
sqlcgen InsertMetricSample), SignalRepo (Open/UpsertWithTriggers/
Transition with inline SQL matching the scheduler's patterns).
Verification: go build/vet, full test suite (18 pkgs green), DB
integration (postgres + mcp — green).
Problem: probe logic (checkHTTP, checkTCP, checkPing, checkDNS,
checkSSHScript, etc.) was embedded inside scheduler/scheduler.go as
unexported functions coupled to sqlcgen types — unreachable from the
core ObservationService the hexagonal refactor needs.
Change:
- adapters/probes/network.go: HTTP, TCP, ping, and DNS probe adapters
implementing ports.Checker. Each parses CheckDef.Config (json map),
runs the probe against the Target, and returns a ports.CheckResult.
configMap helper unmarshals config JSON; parseStr/parseFloat extract
typed values.
- adapters/probes/ssh.go: SSHChecker wraps actuator.Dial +
RunCombinedOutput with a SignerSource for key resolution. Registry
(map[string]ports.Checker) with NewRegistry() pre-populating all
known kinds (ssh-script, vm-status, backup-freshness, cert-expiry
set to nil — filled by the ObservationService when signers are
available).
Verification: go build/vet, full test suite (19 pkgs), DB integration
(postgres + mcp — green).
Problem: relationship create/end existed as three drifted copies
(HTTP CreateRelationship/EndRelationship, MCP create_relationship/
end_relationship) with inline SQL, no ontology edge validation on
either path, and no audit on the MCP path.
Change:
- Internal/adapters/postgres/repositories.go: RelRepo implements
ports.RelationshipRepository (Create/End/ListFor) over the pool,
with in-tx upsert + audit/event side effects on Create.
- Internal/core/app/relationships.go: RelationshipService validates
edges against the cached ontology TypeTree (tree.ValidateEdge) and
delegates the tx to the repository. The adapter resolves slug→entity
and extracts types before calling the service.
- HTTP CreateRelationship: resolves source/target via ReadModels,
passes resolved types to RelationshipService for edge validation.
EndRelationship calls the service directly (audit stays in the
adapter for End — a simple toggle with no ontology check).
- MCP create_relationship/end_relationship: rewired to the service
(pool resolves entity IDs inline for the tool handlers; the service
validates edges and writes audit). The MCP path now gets ontology
validation and audit coverage for the first time.
- Composition root: RelationshipService built with RelRepo + Ontology
and wired through httpapi.NewHandler, ListenAndServe, and MCP
constructors.
Verification: go build/vet, full test suite (19 pkgs, DB integration
postgres+mcp green).
Problem: httpapi.NewHandler built its service dependencies internally
(entity repo, entity service, read models), making the handler a
god-object that knew how to construct its own dependencies. ADR 0016
§3.5 wants cmd/oikos/main.go to be the composition root.
Change:
- httpapi.NewHandler: services (EntityService, EntityRepo, ReadModels)
are now injected as parameters instead of constructed inside.
- httpapi.ListenAndServe: passes the injected services to NewHandler.
- cmd/oikos/main.go runAPI + the 'all' role handler: construct
entityRepo, readModels, and EntityService at the composition root
and pass them down. The router stays in httpapi for now; ownership
moves to main in a later phase.
- Tests: newTestHandler updated to construct and inject test doubles.
Verification: full build/vet, non-DB suite (19 pkgs), DB integration
(postgres + mcp — green). httpapi DB tests have the pre-existing set
of failures (TestAPIEndToEnd, TestPhase3* — verified at ec11956).
Problem: entity/relationship/graph/blast-radius reads were embedded
as inline SQL in the httpapi handlers, duplicating the recursive type
tree CTE, the blast_radius function call, and the topology-picking
query across the REST and MCP surfaces with no port abstraction.
Change:
- ports.ReadModels interface: ListEntities, GetEntity, GetEntityBySlug,
GetEntityRelations, GetBlastRadius, GetGraph (with health),
ListEntityTypes. Returns EntityWithHealth (domain.Entity + health
from entity_status join) and domain.Relationship — no gen types
in the port.
- adapters/postgres/readmodels.go: EntityReader implements ReadModels
with the existing SQL verbatim (recursive type filter, blast_radius,
most-connected-first topology, graph edge listing).
- httpapi/entities.go: ListEntities, GetEntity, GetEntityRelations,
GetBlastRadius, GetGraph rewired to ReadModels. SQL moved to the
adapter; handlers map domain/ports types to gen wire shapes.
entityWithHealthToGen, sqlcEntityToGen helpers added.
- Old sqlcEntityToGen (sqlcgen.Entity → gen.Entity) preserved for
client_lifecycle.go; mutation handlers use domainToGen.
Verification: go build/vet, full test suite (19 pkgs), DB integration
(postgres + mcp — both green). httpapi DB tests have the pre-existing
set of failures (TestAPIEndToEnd entity_types=60/501, TestPhase3*)
verified at ec11956.
Problem: entity mutations (create/update/state) existed as three drifted
copies — HTTP CreateEntity/PatchEntity, MCP create_entity/
update_entity_attributes/set_entity_state — each with its own inline
SQL, its own validation subset (MCP validated lifecycle states, HTTP
did not; HTTP patched attributes without regenerating derived checks,
MCP did; MCP wrote no audit trail), the exact drift ADR 0016's first
vertical slice exists to collapse.
Change:
- internal/core/ports: DerivedCheck, Idempotency (adapter-owned request
hash + cached-body renderer so the replay record commits in the
create's transaction), IdempotentResponse + GetIdempotent read,
AuditEntry gains Method/Path/CorrelationID, Event gains
CorrelationID; EntityUpdateInput carries ExpectedVersion +
RederiveChecks (derivation for updates runs repo-side: the graph
host fallback reads relationships through the open tx).
- internal/adapters/postgres/repositories.go: EntityRepo (Create/
Update/SetState/reads/idempotency) preserving the load-bearing
check-then-act invariants in-tx: version WHERE-clause, declared
transitions + preconditions (ValidateTransition), duplicate-slug
mapping, audit/event/writeCheck all inside one BEGIN…COMMIT.
OntologyRepo: TTL-cached OntologyStore.
- internal/core/app/entities.go: EntityService — ontology validation
(type exists, concrete, state declared — the stricter MCP rule now
governs both surfaces), default-state resolution, id generation,
derivation for creates, audit/event construction, idempotency
pass-through.
- httpapi CreateEntity/PatchEntity rewired to the service; PATCH now
regenerates derived checks (the A2 parity gap). MCP create/update/
set-state tools call the same service — and now write audit + event
rows like the HTTP surface always did.
- Integration-test seed paths fixed for the adapters/postgres package
depth (../../../seeds).
Pre-existing failures documented: TestAPIEndToEnd (entity_types 60 vs
59; 501-endpoint now 200), TestClientLifecycleEndToEnd, TestPhase3*
rows — verified failing identically at ec11956 (scratch approval-
notifier commit test drift), unrelated to this change. All mutation
integration tests (create/patch/idempotency/audit/regeneration) pass.
Verification: make test-db (postgres + mcp green, httpapi green except
the pre-existing set), full non-DB suite (19 pkgs), golangci on new
packages — 0 issues.
Problem: check derivation logic lived in internal/checkdefaults with
the pure decision logic (buildKind, address/user/port resolution)
interleaved with tx I/O (entity_status insert, graph host fallback,
check upserts) — and internal/db importing it was the plan's called-out
inverted dependency.
Change:
- internal/core/app/checkdefaults.go: Derive(tree, target, lookup) —
the full derivation (monitoring overrides, host fallback via an
injected HostLookup thunk, per-kind builders) with zero I/O imports.
Types renamed for the app surface: CheckTarget, CheckDef,
DeriveResult, Skip; LogDeriveResult.
- internal/adapters/postgres/checks.go absorbs the I/O half:
EnsureChecks (entity_status row + upsert loop), writeCheck, and
hostViaGraph. The db→checkdefaults edge is gone — adapters→core is
the ADR 0016 direction (the Phase 7 SeedService note anticipated
this; the inversion is fixed a phase early).
- seed.go pending-checks loop uses app.CheckTarget + EnsureChecks;
mcp formatting/tests follow the renamed types; both test files moved
to internal/core/app.
- Deliberate behavior note: a hostViaGraph read failure inside the
thunk now logs a warning and degrades to 'skipped: no address'
instead of aborting the whole entity-create tx — a monitoring
derivation gap is visible (warn log + coverage sweep) and self-heals
on the next mutation; failing the create over a graph-read blip was
disproportionate.
Verification: go build/vet, full test suite green (app tests exercise
every buildKind branch at their new home).
Problem: the repo carried cruft that predates the Phase 1 client split:
tracked web/node_modules and web/dist content (committed before the
ignore rules existed — ~2.6M lines), seven stale Claude worktrees plus
two stale Agent Manager worktrees (1.5GB on disk, all fully merged),
their 29 merged experiment branches, a pre-DB root inventory.yaml,
Playwright MCP session logs, a config screenshot, and the executed
one-shot apps/105 webhook-cleanup script. The dockerignore's own
comment documents how this cruft once starved the mac-mini disk
mid-build.
Change:
- git rm: web/ + cmd/desktop/ tracked remnants (node_modules, dist),
root inventory.yaml (stale pre-DB copy; seeds/inventory.yaml is
authoritative and what tests read), .playwright-mcp/ logs,
config-screen.png, .claude/launch.json,
scripts/cleanup-apps105-webhooks.sh (job done; pattern lives on in
oikos-web's webhook setup).
- Removed 9 stale worktrees (nested-first) + pruned; deleted 29 fully
merged branches (claude/*, feature/*, frontend-os-apps,
judicious-freckle, code-quality-* pair, impartial-height). The one
branch with an unmerged commit, chore/vendor-orby-engine, vendored
web/vendor — that work moved to dtoro/oikos-web in Phase 1, so it is
superseded.
- Disk cleanup: web/, cmd/desktop/, stray desktop binary, bin/, build
artifacts. Kept: root oikos + webhook binaries (referenced by the
launchd deploy unit and oikos-web's installer), .env bootstrap,
.infisical-credentials.
- .gitignore: .claude/, .playwright-mcp/, config-screen.png now
clone-safe instead of relying on local info/exclude.
Risk: none functional — deletions are either merged history (branches
recoverable from reflog) or content that moved repos; build, vet,
tests, and generate-check all green post-purge.
Verification: go build/vet, make test (19 pkgs ok), generate-check,
git ls-files web cmd/desktop → 0; worktree list → main only.
Problem: the hexagon's Phase 2 (plans/2026-08-15-hexagonal-architecture.md)
must give the use-cases-to-be their contract surface: driven-port
interfaces, test fakes, the secrets interface moved into core, and the
postgres package inside the adapters tree — before the first vertical
slice (Phase 3) can wire a composition root.
Change:
- internal/core/ports: full driven-port catalog per plan §3.3 —
repositories as transaction-scoped aggregates whose inputs carry
derived checks, audit, and events (§3.6), plus CommandExecutor,
TargetResolver, Checker, Secrets, EventPublisher, Provisioner.
Port-local payload types (Event, AuditEntry, CheckDef, KnowledgeEntry,
ExecResult) keep signatures off infrastructure; TypeTree aliases
internal/ontology (pure over domain) until checkdefaults is absorbed.
ReadModels intentionally not declared yet — it materializes with the
Phase 3 slice and grows as report handlers rewire.
- secrets.Backend is now an alias of ports.Secrets; implementations
(Infisical, SOPS, Manager) unchanged. mcp's local secretBackend
subset is deleted; tool constructors take ports.Secrets.
- internal/db → internal/adapters/postgres (mechanical import rewrite;
package identifier stays db until the Phase 3 repository split).
sqlc.yaml, Makefile, golangci exclusions, and docs follow the move;
make generate-check verified.
- internal/adapters/ssh: Executor implements ports.CommandExecutor over
the actuator dial pool + RunStreaming (10-min default timeout carried
over from the httpapi path).
- internal/adapters/remote: Resolver implements ports.TargetResolver
delegating to internal/remote (still pool-based; drops onto
ports.EntityRepository when repositories land in Phase 3 — documented
transitional import).
- internal/core/ports/portstest: importable fakes — in-memory
EntityRepo (with check-then-act SetState, side-effect recording),
RecordingExecutor, FakeChecker, SpyPublisher; port-satisfaction
guards; tests.
Risk: ports are declared ahead of implementations — signatures firm up
per phase as slices land (documented in the package doc); the
remote→postgres transitional import is explicit and dissolves in
Phase 3.
Verification: go vet, make test (race, 19 packages), generate-check,
golangci on core+adapters — 0 issues; full-repo baseline down
365→344.
Problem: the hexagonal refactor churns the backend tree for nine more
phases; the UI delivery stack (web/ SPA, cmd/desktop Wails wrapper,
compose/web image) must move to its own repo first so doc/layout
rewrites land once on a backend-only tree.
Change:
- New repo git.hubris.network/dtoro/oikos-web (v0.33.0): web/, desktop/
(updateURL repointed to oikos-web releases), compose/, own CI (web +
desktop jobs), own deploy script (CI-green gate, TOCTOU guard,
version-tagged images, prune-to-3), own webhook receiver on :9798 +
launchd unit, own compose project publishing the same 8091:80.
- Cutover executed on mac-mini in order: oikos stack's web service
stopped+removed, oikos-web project brought up on 8091; outer Caddy
untouched (targets the published port) — serving + Authentik flow +
/wails 404 quirk verified post-cutover.
- Stripped from oikos: web/, cmd/desktop/, compose/web/, desktop CI
workflow, ci.yml web job, Makefile ui/desktop/desktop-package/install
targets, the compose web service, oikos-web from deploy.sh's fallback
prune list; wails + go-keyring dropped from go.mod, vendor synced.
- README / CONTRIBUTING / AGENTS.md / .agents dev+operations docs now
point at the new repo; mbse + mascot design docs carry a path note.
Risk: production SPA serving depends on the new pipeline now; rollback
is versioned-image re-up of the old web service from a pre-split
checkout (port 8091). Desktop builds installed before the split still
check dtoro/oikos releases — one manual reinstall, noted in the
oikos-web release notes.
Verification: go vet, make test (race), make generate-check, golangci
(no new findings; baseline down 400→365); post-cutover curls —
localhost:8091 200, /wails/runtime.js 404, outer Caddy 302 Authentik.
Problem: the hexagonal-architecture plan (plans/2026-08-15-hexagonal-
architecture.md) needs its foundation — an accepted ADR, the target
directory tree, and machine-checked dependency rules — before any
service extraction starts. Also folds the four outstanding review
findings (F3.1/F5/F6/F7) into the plan: ObservationService owns the
bounded probe-concurrency contract (scheduler.go:133), Phase 9 gates
ExecutionService+PolicyService ≥ 90% with a gating-matrix test,
per-phase abort criteria, and the §3.2 internal/config note.
Change:
- docs/adr/0016-hexagonal-ports-adapters.md records context, decision,
and consequences of the ports & adapters migration.
- internal/domain → internal/core/domain (mechanical import rewrite,
20 files), new internal/core/{ports,app}, internal/adapters trees
with package docs.
- .golangci.yml: depguard rules for §3.1 (core purity, no agent-client
tech in core, nomos isolation — the nomos rules self-activate when
internal/nomos exists in Phase 8). Config migrated to golangci-lint
v2 format so it loads at all (the v1 config errored under v2, masked
by CI's advisory continue-on-error). Verified depguard fires on a
planted openai-go import in internal/core/app.
- CONTRIBUTING.md layout section now shows the core/adapters tree.
Risk: import path churn is mechanical and tests pass unchanged; the
lint config migration surfaces the pre-existing 400-issue baseline
(advisory in CI, unchanged policy) — new/moved packages lint clean.
Verification: go vet ./..., make test (race, core/domain at 100%
coverage), make generate-check, golangci-lint on internal/core/... and
internal/adapters/... — 0 issues; depguard violation probe confirmed.
Removes the entire Matrix-based notifier (internal/notifier/) that polled
for pending approvals, sent Matrix alerts, and checked for reaction-based
approve/deny. Approval decisions now work on any chat platform (Hermes
desktop, Telegram, Discord, WhatsApp, CLI) via two new MCP tools:
- list_approvals — query pending/recent approvals by status or entity
- decide_approval — approve/deny via same API endpoint as UI + nomos
Config fields removed: MatrixHomeserver, MatrixUserID, MatrixToken,
MatrixRoomID, ApprovalHMACSecret. Docker notifier: service removed.
Approval HMAC token generation removed (unused by code).
The existing chat-assent path in nomos (cmd/nomos/assent.go) and the
control-room Approve button keep working unchanged — both call the
shared POST /api/v1/approvals/{id}/decision endpoint.
New tool batches session discoveries into the knowledge graph:
- Creates a session-audit knowledge entry with summary
- Links it to all touched entities via 'documents' relationships
- Creates individual discovery knowledge entries
- Stamps each entity with last_agent_session attribute
- Updates AGENTS.md with tool listing
P0: Transport escalation in classifyAndGate (server.go:745) now exempts
log-inspection commands (tail/head/cat/journalctl on *.log or /logs/)
from the /opt//etc//var/lib/ gating on LXC targets. Fixes the
'tail -3 /opt/seanime/data/logs/seanime.log queued for approval' bug.
P1: queryEntity returns actionable error when slug_or_id param is empty
('slug_or_id is required') instead of silent 'entity not found: '.
P2: Added slugArg() helper (server.go) so get_entity, get_relations,
get_blast_radius, and explain accept 'slug' as an alias for their
declared param name. Solves the discoverability inconsistency where
every tool used a different param name for the same concept.
P3: Two new MCP tools:
- restart_service(target, service) — systemctl restart wrapper,
correctly classified config_mutation (requires approval)
- push_file(target, source_path, dest_path, backup=true) — pct push
from Proxmox host into LXC, with optional backup. Classified
config_mutation. LXC-only for now.
P4: Updated homelab-lxc-ops skill with MCP tools preference table.
Plus: Wails desktop build now uses build-tag approach for frontend embed
(assets_embed.go + assets_stub.go), so go build ./... works on
clean checkout without the frontend built first.
Version: 0.31.0 → 0.32.0
Adds pvecm_quorum_check.sh probe script and wires it into the
checkdefaults system as a new 'quorum' monitoring kind on proxmox-host
entities. Runs every 60s via ssh-script, surfaces unhealthy signal when
cluster loses quorum.
Closes the monitoring blind spot that let the 2026-08-12 3.5h corosync
flapping outage go undetected (ping passed, cluster was non-quorate).
Changes:
- seeds/ontology.yaml: proxmox-host declares monitoring: [quorum]
- internal/checkdefaults/defaults.go: KindQuorum builder
- internal/checkdefaults/build_test.go: 2 new test cases
- checks/pvecm_quorum_check.sh: new probe (deployed to hubris + strong)
- VERSION: 0.30.2 -> 0.31.0
Two fixes from the deploy pipeline audit:
1. Infisical tag v0.99.1 no longer exists on Docker Hub — bumped to
v0.162.19 (latest available). This was silently breaking the full
deploy pipeline (docker compose up failed on image pull).
2. Deploy failures now notify via two channels:
- Oikos API event (deploy.failed, severity=critical) — picked up by
the scheduler's notifier for Matrix alert
- Matrix webhook URL if MATRIX_WEBHOOK_URL is configured
Uses a trap with _ok flag to catch any non-zero exit path,
including CI gate rejections and health check timeouts.
Webhook now resolves and passes OIKOS_API_TOKEN to deploy.sh.
Registers a handler that serves the embedded OpenAPI 3.0 spec
(compiled into the binary via oapi-codegen) at a browseable
endpoint. Uses gen.GetSwagger() to deserialize the embedded
base64+gzip spec and returns it as JSON.
46 paths, 42 schemas — agents and humans can now introspect the
full API surface without reading Go source.
Read-only diagnostic commands ethtool, lsmod, lspci, modinfo, and dkms
were missing from the readOnlyLeadPattern in the command classifier,
causing compound diagnostic commands (e.g. 'uname -r && ethtool -i eno1
&& lsmod | grep r8169') to be misclassified as config_mutation instead
of read_only. This forced operator approval for simple hardware/driver
inspection during the 2026-08-12 hubris NIC cutover session.
Added regression test with the exact compound command from that session.
New EntityGraph.svelte app (sigma.js + graphology):
- Theme-aware Gruvbox palettes (light/dark) with reactive color switching
- Lucide icons rendered synchronously via Path2D canvas per entity type
- Color mode toggle (Health / Type) with per-type distinct colors
- Health distribution bar with clickable filters
- Entity type filters grouped by ontological layer (collapsible)
- Relationship type edge filters with color-coded swatches
- Quick presets: All / Problems / Infra
- Node selection with live blast radius from API
- Hover neighborhood highlighting with muted fade
- Isolated node hiding, edge alpha tuning, dot-grid background
- Search with camera focus on highest-degree match
createTaskEntity creates the task entity but never adds any graph
edges. The involves edges are only added by the run handler, so
sessions that call set_goal but never make a run call leave orphan
task entities with zero relationships.
Adds an idempotent involves edge from the new task to agent:nomos
at creation time, matching the same pattern used for run's involves
edges in server.go.
Every run call creates a classification entity, but it was never
connected to the execution via a graph edge — only via a DB column
(executions.classification_id). The ontology requires:
classification —precedes→ execution
Without this edge, all 53 classification entities had zero
relationships, making them invisible to get_relations and
blast-radius analysis.
Adds an idempotent INSERT into relationships after the existing
classification_id update, matching the same pattern used for
targets edges.
The healthcheck subcommand was reachable only after main()'s Infisical secrets
resolution (4x retries/key, ~30s when Infisical is down), which blew the 5s
Docker healthcheck timeout — so nomos stayed docker-unhealthy despite serving
/healthz fine. Short-circuit 'nomos healthcheck' at the top of main() before
any secrets init; measured 0.58s, no Infisical retries.
The nomos runtime image is gcr.io/distroless/static (no shell/wget), so the
wget-based healthcheck (D5) could never run — nomos showed docker-unhealthy
despite serving /healthz fine. Add a 'nomos healthcheck' subcommand that
self-probes NOMOS_LISTEN/healthz (exit 0 on 200), and point the compose
healthcheck at ["/nomos", "healthcheck"].
Measured startup is ~93s: NewHandler stalls on Infisical auth retries (~40s)
and OIDC discovery timeouts to auth.hubris.network (~35s) before binding
:8090. The api IS healthy once bound (serves /healthz); the window just needs
to clear both external-timeout phases so nomos (depends_on: api-healthy) can
start and the deploy completes.
The api retries Infisical at startup (4x with backoff) before binding :8090.
When Infisical is unreachable that adds ~60s, and the old start_period (10s)
+ 10 retries (~60s grace) ran out just before the bind — marking the api
unhealthy and failing the deploy (nomos depends_on api-healthy). 90s covers
the slow-startup window; the api genuinely serves /healthz once bound.
The procedural-glyph-engine dep pointed at a non-portable file:/private/tmp/orby-pkg
path, breaking npm ci in Docker and every main deploy since v0.20.0 (the build
cache masked it until it busted ~Aug 5). Vendor Orby v5.0.0 into web/vendor/,
switch the dep to file:../vendor, and use npm install in the web Dockerfile
(file: deps need install, not ci). Cherry-picked from 3cd4cf9.
- Wire secretsManager in NewHandler() — instantiate InfisicalBackend
when OIKOS_INFISICAL_SITE_URL is set (previously always nil)
- Add get_secret, list_secrets, set_secret MCP tools with nil-backend
graceful degradation
- Add oikos secret get|set|list CLI subcommands for Infisical
- Fix Set() bug: create-before-update so new keys are created;
add Type: "shared" to Update so it finds the right secret;
disable SDK cache so Get returns fresh data after Set
- Clean enrollment response: remove fake infisical_client_id/
infisical_client_secret stubs, store age key in Infisical for real
- classifyAndGate: escalate read-only commands on lxc: targets that
touch /opt/, /etc/, /var/lib/ to config_mutation. The classifier
scores command text only, not the SSH transport layer — SSH-ing into
a container to read config is riskier than pct exec from the host.
- ontology: standalone-server monitoring override from inherited
[ping, resource, updates] to [http]. VPS-like machines may not be
SSH/ICMP-reachable from the scheduler; HTTP is the LCD liveness
signal. Entities with full SSH can override per-entity.
- get_health_summary: filter out state=destroyed entities (was noise
from 20+ destroyed test LXCs, deprecated services, etc.)
- create_entity: document the monitoring footgun in the tool description
(creating a type=check entity does NOT wire a check_def; the correct
path is update_entity_attributes with monitoring + url attributes)
Adds a new 'dns' semantic monitoring kind that probes whether a DNS name
resolves. Uses net.LookupNS (NS records) with fallback to net.LookupHost
(A/AAAA). Supports an explicit server config for split-horizon resolution.
Changes:
- seeds/ontology.yaml: dns-zone monitoring: none → [dns] (was deferred
since 2026-06 with a comment 'no dns checker exists yet')
- seeds/inventory.yaml: host:netbird-vps monitoring: [http] (was none;
VPS was invisible for 7 days during the 2026-07-29 outage)
- internal/checkdefaults/defaults.go: add KindDNS, buildKind case for 'dns'
that creates a check_def at 5-minute intervals
- internal/scheduler/scheduler.go: add checkDNS probe + wire in executeCheck
The DNS checker catches stale/unreachable zones (e.g. matrix.hubris.network
pointing to a dead VPS IP). The VPS HTTP check probes the public endpoint
every 60s, closing the 7-day monitoring gap.
I — run pre-flights QEMU guest agent before queueing VM execution
classifyAndGate now checks vm: targets for qemu_guest_agent attribute.
If not_running/missing, returns immediate error instead of queuing forever.
II — policy.yaml: documented host-mutation classifier rule
Added comment clarifying that host-level package/kernel mutations
(apt-get install, dpkg, systemctl enable) always classify as
config_mutation and thus need operator approval.
III — health attribute read-only in update_entity_attributes
Strips scheduler-owned keys (health, last_check_at, last_check) from
attribute updates with a clear message directing agents to
get_health_summary / list_checks instead.
IV — Recorded discovered dependency edges
vm:zimaos → depends-on → lxc:nfs-export (NFS /media/library mount)
vm:zimaos → depends-on → host:strong (NFS /media/ludo-library mount)
Also updated the run tool description to mention both guardrails.
get_relations now accepts an optional 'types' (comma-separated) parameter
to filter relationship types — filters out the noisy exec/targets edges
that previously drowned useful host/provides edges.
get_health_summary now accepts an optional 'health' (comma-separated)
parameter to return only entities in specific health states (e.g.
'health=down,stale') instead of the full 100+ entity list.
ping_service now:
- Falls back to e.attributes->>'public_host' when 'url' is not set
(covers LXCs that only have public_host in the graph)
- Performs a live HTTP HEAD probe against the resolved URL, returning
the actual status code instead of just the scheduler's stale health
state
Also: fixed matrix.hubris.network DNS record (was pointing to dead VPS),
pruned 6 dead graph edges, wired url attributes on 7 LXCs, added VPS
HTTP monitoring check, and resolved the 18k-occurrence unmonitored signal.
This session's audit is documented as
document:nomos/2026-08-05-dns-monitoring-improvements-for-strong-hosted-services.
- Add MCP tool — lightweight connectivity check returning server
identity, no DB hit (resolves agent connection-test friction)
- Tighten 6 tool descriptions (get_relations, get_health_summary,
query_metrics, get_trend, get_event_timeline, ping) to be searchable
in the first 8-12 words
- Document Hermes MCP client setup in ADR-0012 with token security caveat
- Move completed plan to plans/done/
Backend:
- Add isThinking flag to agentEvent for text before tool calls
- Separate thinking from response text in runChatTurn and continue.go
- Persist thinking in a dedicated field in message content
Frontend:
- Add thinking field to MessageContent, ChatMessage, ChatTextEvent types
- Create ThinkingBlock.svelte — collapsible block with brain icon
- SSE handler moves text_delta content to thinking on isThinking flag
- Render thinking block between tools and response in ChatThread
- Fix chat window scroll reset on focus change (stable windowKeys order)
- Remove redundant #key id wrapper in WindowLayer
- Enlarge sidebar rail (24→32 default, 40→60 max)
- Remove glyph from sidebar, square graph at top
- Replace AgentTrace/ToolCallCard/UnifiedTimeline with TurnTrace/ToolLine
The "Working — message will queue" strip lived inside the sized input Pane, so
appearing/disappearing ate the textarea's fixed height and clipped it, forcing
a resize. Move it into the message Pane alongside the connection/error banners
— those correctly consume transcript space (flex-1) rather than the input's
fixed height. The input Pane is now stable whether or not a background turn is
running. Styling/idiom unchanged.
VERSION: 0.17.2 -> 0.17.3
The background-working hint was a plain muted text line bolted above the
textarea — misaligned with the input column and off-idiom. Restyle it as a
terminal status strip: spinner + uppercase fg "Working" label + muted detail,
hairline primary-tinted border (matching .trace.running), square, aligned to
the textarea's max-w-3xl column. Reads as part of the working state now.
VERSION: 0.17.1 -> 0.17.2
git mv staged the pre-edit index content; the status edits to these two
files landed in the working tree but not the archive commit. Amending the
status now so the archived copies reflect Implemented.
Move ten completed plans from plans/ to plans/done/ and update the index:
- 2026-07-18 session-review-three-sessions, 2026-07-20 desktop-mascot,
2026-07-20 session-review-ten-sessions, 2026-07-21 chat-full-polish,
2026-07-29 health-check-reality-and-knowledge-graph,
2026-07-30 session-review-plan-drift, and the four 2026-08-03 chat plans
(changes-review, reliability-and-ux-audit, cyberspace-style-adoption,
working-visibility).
- Refresh two stale statuses: cyberspace-style-adoption ("Draft" -> shipped as
full replacement in v0.16.0/757ef2f) and health-check-reality ("ready for
implementation" -> shipped across the v0.14.x-0.16.x check commits).
- .gitignore: ignore local tooling artifacts (.playwright-mcp/, config-screen.png).
No code change. index.md Active/Done tables now match the filesystem (no orphans).
VERSION: 0.17.0 -> 0.17.1
Make background/long/desynced turns visible and queueable, fixing the four
symptoms that survived the v0.15.0 chat reliability pass.
F1 - status-driven working signal (workspace.ts taskWorking/currentWorking =
streaming OR status in {planning,executing}). Drives the chat trace, indicator,
and activity spinner so a turn with no live stream (background resume, a dropped
SSE, an idle-close mid long turn) still looks alive.
F2 - operator messages sent during an in-flight turn are now QUEUED and
auto-run when the gate frees, replacing the "still finishing a previous step...
send it again" rejection. Per-session in-memory FIFO (messagequeue.go, capped at
20) drained one-at-a-time under the turn gate; a `queued` SSE event drives a
"Queued" hint. drainQueued releases via a per-iteration deferred closure so a
runChatTurn panic can't deadlock the session's gate.
F3 - SSE keepalive (12s `:keepalive` comment) in handleChat so 20-40s
inter-iteration gaps no longer trip a proxy/browser idle close (the desync root
cause). All SSE writes serialized through one mutex.
F4 - generation-aware activity timeline (only the last propose_plan renders;
superseded ones collapse to one "Earlier plan revised" marker; step-attribution
follows only the current generation) + debounced plan refetch on lifecycle
events so a missed plan.proposed self-heals.
Verified against the last session (23da10db: 6m33s turn, operator "status"
deferred at 19:48:05). go test ./cmd/nomos/ green (new messagequeue tests);
web vitest 72/72 (new F4 generation tests); vite build clean.
VERSION: 0.16.0 -> 0.17.0
Rebrand light/dark themes to the cyberspace.online look: warm cream-on-black
palette (light/dark are exact inverses), self-hosted JetBrains Mono + VT323,
square corners, border-driven surfaces with no soft shadows. Adds a terminal
design-system CSS layer (DOS double-border modals with hatched corner, fg focus,
inversion-on-hover), a theme-aware <RasterImage> (Atkinson-dithered canvas with
img fallback), and unifies desktop icons, taskbar, window controls, pills and
links under one idiom. Pins window titlebars to a fixed height and switches chat
auto-scroll off scrollIntoView to avoid titlebar reflow.
VERSION 0.15.1 -> 0.16.0
The Send button's disabled={!input.trim()} threw "trim is not a function" when
input was initialized from a non-string initialDraft (a Svelte 5 prop-init edge
where a null/undefined draft reached $state). Coerce at init so the composer
state is always a string.
VERSION: 0.15.0 -> 0.15.1
The agent could run two turns for one session at once (a reconnect resumed
while the live turn was still going), and their interleaved tool calls
corrupted the activity panel, fabricated a confusing "parallel/nested"
sequence, and made tasks feel stuck/never-ending. Several UX gaps compounded it.
Turn serialization (F1):
- turnGate: at most one in-flight turn per session. Background resume paths
(continuation worker, idle sweep, answer-question, /resume, reconnect)
skip non-blocking when busy; the live chat path waits briefly then bails
cleanly instead of stacking a second turn.
- resumeSession returns whether it ran; continueSession marks an execution
"continued" only after a real run (review P0) so a busy-skip can't lose a
finished-execution result. Idle nudge bumps only after delivery (P1).
Connection state (F2/F3, web):
- humanize/bucket raw errors ("model connection dropped..."); one surface
per drop; a terminal task.status event clears stuck streaming/disconnected
state and dismisses the connection toast. Reconnect no longer spawns turns.
Streaming where you look (F4, web):
- live command output in the global activity timeline and in the inline
tool card (auto-opened, tail-pinned) -- not just the per-window rail.
Other (web): artifact/knowledge deep links (F5); step-first stable
"thinking" headline (F6); stable chat layout, no empty->content reflow (F7);
lazy event sync (P2.2); reconnect skips a terminal session (P2.1).
VERSION: 0.14.2 -> 0.15.0
The plan recorded false history after a re-plan and the activity panel
showed fabricated, churning timestamps. Two bugs compounding on one event
stream.
Plan drift (P0.1):
- proposePlan seq is now 1..N per generation; (session,generation,seq) is
the addressing key. The model's 1-based update_plan_step calls always map
to the CURRENT plan after a re-plan, instead of resurrecting a superseded
`replaced` row as done while the live work went unrecorded.
- updatePlanStep resolves against MAX(generation); a stale/out-of-range seq
returns errPlanStepNotFound (never touches a superseded generation).
- getPlanSteps returns only the current generation by default; ?all=true
keeps the audit/eval view (plan_generations assertion).
- completeTask auto-close scopes to the current gen, stamps started_at, and
emits one plan.step.finished per closed step so the panel converges
instead of freezing on "running" after completion (P1.1).
- propose_plan result enumerates step seqs; writeback detector matches
"write back"/"writeback"/"upsert_knowledge" so a natural-language final
step isn't doubled (P1.2).
- migration 029 renumbers existing seq per generation + unique index.
Activity panel (P0.2 / P1.1, web):
- computeActivityLog uses the real message created_at for tool calls; live
entries fall back to wall-clock frozen on first sight, killing the 3s
poll churn. Steps use real started_at.
- dropped plan-step events warn + count instead of a silent no-op.
Tests: TestProposePlan updated; + generation-relative-seq and auto-close
event-emission regression tests; + web activity purity/timestamp tests.
VERSION: 0.14.0 -> 0.14.1
Every docker build sent the whole repo root as context, including every git
worktree under .claude/worktrees/ (200-300MB each) — that crossed 390MB of
cruft and starved mac-mini's disk mid-build on 2026-07-27 (873b00a). None of
it belongs in an image.
A `monitoring` attribute on an entity now overrides its type's declaration:
"none" opts out, a list overrides the kinds. service:haos uses it to opt out —
haos blocks SSH (no process probe can reach it) and the VM is already covered
by vm:haos's vm-status check, so the redundant process check only ever reported
false-down. vm:haos -> service:haos via provides confirms the coverage.
VMs declared monitoring [ping], but many block ICMP and lack a guest agent
(haos), so ping was the wrong probe — a powered-on VM reported "down". Add a
vm-status check: `qm status <pve_id>` on the VM's Proxmox host, which tests
"powered on" without needing the VM's network at all. vm type monitoring is
now [vm-status].
matrix.hubris.network is a public hostname (federation) resolving to
netbird-vps, not served by the lab Caddy — so its cert-expiry check's
dial=caddy IP failed. Drop the dial for matrix; it dials by name (DNS ->
public) like wget already proved works.
The ontology's stated intent was "http when it has a url, else a process
check", but the implementation emitted BOTH for every url-service — so ~17
fronted services carried a redundant process check that, under worst-of
aggregation, let a fragile supplementary probe (wrong unit name, unreachable
host, no guest agent) veto two healthy http checks and report the service
"down" while it was up (authentik, zimaos, house, matrix, ...).
buildKind now emits a process check only for services WITHOUT a url, or when
an explicit probe_unit opts into binary-level depth. http is the canonical
service-liveness probe (tests the real endpoint through the TLS terminator);
the redundant process checks were removed.
process_check.sh ran `systemctl is-active <entity-name>`, but a service's name
is a logical label, not its unit/container name — matrix is matrix-synapse.service
+ element-web/mautrix-* containers, authentik is authentik-server/-worker
containers. So every multi-component or docker service reported "inactive"
while up (authentik, matrix, photos, house, arr-stack, …).
Resolve in order: exact systemd unit, a unit with the name as prefix
(matrix -> matrix-synapse.service), or a running docker container whose name
contains it. checkdefaults passes a declared probe_unit/systemd_unit/container
attribute when set, for precision.
The DB-only audit_knowledge_graph can't see guests running in Proxmox that
have no entity, or entities whose pve_id is no longer live — the drift that
the stray test LXCs were a symptom of. discover_infra_drift enumerates running
guests via pct/qm list on every proxmox host (over the same SSH/pct path the
checks use) and diffs against the DB: returns missing (live, no entity) and
ghost (DB, not live). Read-only.
Companion to audit_knowledge_graph; the skill now runs both and treats the
remaining checks (misplaced parent, undeployed scripts, seed drift) as manual.
resolveProxmoxHostSlug trusted attributes.host verbatim, so a value polluted
with prose — lxc:teddycloud carried host="hubris (confirmed via pct config…)" —
became a slug that never resolved, leaving its checks 'down' despite a correct
`hosts` edge. Treat an attribute containing whitespace/parens as invalid and
fall back to the canonical hosts edge.
The audit now reports `polluted_attrs` — entities whose routing-critical
attributes carry prose — so this class is visible instead of a silent
resolution failure.
The B1 target-state filter `tgt.state NOT IN ('deprecated','destroyed')`
evaluates to NULL (unknown) when a target's state is NULL, which the WHERE
clause treats as false — so freshly-seeded entities without an explicit state
(the 20 TLS certificates) were silently dropped from ListEnabledCheckDefs and
never monitored. Treat NULL state as active (only explicit deprecated/
destroyed is excluded): `tgt.state IS NULL OR tgt.state NOT IN (...)`.
checkCertExpiry now accepts a `dial` address and sets ServerName to the
hostname — it connects to the terminator's IP while SNI/cert-read use the
hostname. The scheduler container has no mesh interface and the host resolver
doesn't know the split-horizon zone, so *.hubris.network can't be dialed by
name from there; dialing Caddy's lab IP (reachable on the LAN) makes the probe
work. The builder passes through a cert entity's `dial` attribute.
Re-seed the 20 *.hubris.network certificate entities with dial=192.168.8.175
(Caddy) and uses-certificate edges; cert-expiry monitoring now has real data.
disk_usage_check.sh built its mount list with `df`, which blocks on a wedged
filesystem (stale NFS export, a stuck ZFS pool) — and that stalled the whole
check past the scheduler's 30s budget, leaving host:hubris:4 perpetually down.
Build the mount list from /proc/mounts (a read that never stats anything), and
bound every per-mount `df` with `timeout 8` so a single stuck mount is skipped
instead of hanging the probe. Degrades to plain `df` on hosts without
`timeout`//proc/mounts (macOS), whose local mounts don't hang.
The cert-expiry builder and checkCertExpiry probe are correct, but the
scheduler container can't reach *.hubris.network:443 — its DNS forwards to the
host resolver, which doesn't know the split-horizon zone, and overriding the
container DNS would break docker service-name resolution. Seeding the 20 cert
entities now produced 20 false-down certificates.
Keep the builder (committed), drop the entities + edges until the scheduler can
reach Caddy (extra_hosts mapping, or a SNI-dial enhancement) — then re-add them.
The ontology declared monitoring [cert-expiry] on the certificate type and a
working checkCertExpiry probe existed, but checkdefaults had no cert-expiry
builder and no certificate entities were seeded — so certificate expiry, a
real failure mode, was invisible.
Add a KindCertExpiry builder (dials the cert's hostname on :443 hourly, warns
at 30d / crit at 7d) and seed certificate entities for the 20 public
*.hubris.network routes plus uses-certificate edges from each ingress route.
A service check used to bake its hosting LXC's lan_ip and SSH it directly as
root, which failed because the scheduler key is authorized on the Proxmox hosts
but not inside every guest — leaving all 8 service process checks 'down' even
after the guest routing and scripts were fixed.
ResolveExecTargetForCheck now, for a non-guest target, walks the
provides/runs-on/hosts edges to the compute entity that runs it and routes
through that: pct/qm exec if the host is a guest, direct SSH with the host's
correct user (workstation `user` attr) if it's a machine. The guest-resolution
path is shared via resolveGuest, and the scheduler no longer needs an
isMachine special case — one resolver handles guest, machine, and service.
Older writeCheck inserts omitted target_type, so every seed-created check_def
had a NULL/empty target_type. checkSSHScript's IsGuest check then never matched,
and guest checks silently fell back to their baked (often mesh-only) address —
keeping them 'down' even after the pct-exec routing and deployed scripts were
in place. rclone stayed down for exactly this reason after the host-hop fix.
writeCheck now writes target_type, and checkSSHScript resolves the type from
the target_id when the column is blank (a runtime safety net for existing rows;
the seed rows were also backfilled in the live DB).
Graph view: raise the node cap 500 -> 2000 and exclude execution/task audit
rows from the default whole-graph view so the cap is spent on actual topology
rather than ~380 cognition records that crowded out every host/lxc/service.
dns-zone monitoring [dns] -> none: no dns checker exists, so the declaration
only produced unresolvable `unmonitored` noise (requires ontology re-ingest;
coverageSweep now auto-clears the stale signals). Flip back to [dns] when a
checker lands.
Operator tooling: tools/deploy-checks.sh pushes check scripts into guests via
pct push (a pct-exec-routed check runs the script INSIDE the guest), wired
into the post-pull setup-checks hook so guests stay in sync on Proxmox hosts;
scripts/cleanup-orphan-checks.sh (dry-run by default) and
report-stray-test-lxcs.sh retire legacy cruft. VERSION 0.13.0 -> 0.14.0.
Plan: plans/2026-07-29-health-check-reality-and-knowledge-graph.md.
Adds audit_knowledge_graph (MCP tool) and GET /api/v1/audit/drift (endpoint)
backed by a shared internal/audit package. One pass surfaces the structural
gaps an operator otherwise finds by accident: orphan check entities, checks
targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored
declared types, and live edges pointing at destroyed targets. Each finding
carries a suggested remediation runbook. Read-only and safe to run unattended.
Ships the knowledge-graph-audit skill (SKILL.md + seeded runbook) that
interprets the report and routes findings to the lifecycle runbooks.
ListEnabledCheckDefs now LEFT JOINs the target entity and excludes rows whose
target is deprecated or destroyed, so retired things (secrets-issuance,
homelab-mcp, the dead secrets ingress route) stop generating permanent false
alarms instead of waiting for an operator to disable the check_def by hand.
coverageSweep's None() branch previously did nothing, so a type changed from
declared monitoring to `monitoring: none` (dns-zone) left its open
`unmonitored` signals lingering forever — a None() entity never gains a check,
so the hasCheck resolution path never fired. It now resolves those signals.
The scheduler SSHed each guest directly and assumed a deployed probe script
plus working root SSH at the guest's address — false for headless (nfs-export),
keyless (teddycloud), mesh-only (rclone), and macOS (mac-mini) targets, which
left 49 enabled checks stuck "down" on a healthy fleet.
Extract the MCP run tool's resolveExecTarget into a shared internal/remote
package and make it the single execution path for both the scheduler and MCP.
LXC/VM checks now host-hop via pct exec / qm guest exec through the owning
Proxmox host (no per-guest lan_ip, sshd, or authorized key needed); hosts and
workstations resolve their address and user live, so mac-mini's `user: dtoro`
is honored without a re-seed. Address preference now prefers public_ipv4 over
mesh, so netbird-vps is probeable from the scheduler container.
cpu_check.sh gains a real Darwin branch (it reported cpu_pct 0 before).
checkdefaults.resolveSSHUser reads the top-level `user` attribute too.
A machine-target resolution failure is now logged before falling back to baked
config, so a broken probe-config is distinguishable from a real outage.
Two things the entity window redesign surfaced but deliberately left alone.
**blast_radius answered the wrong question.** It walked source→target for every
relationship type, but which end of an edge is the dependent differs per type:
"machine hosts container" means the target breaks, while "service depends-on
service" and "ingress routes-to service" mean the SOURCE breaks. Walking
everything forwards was right for hosts/provides and backwards for everything
else — and swept in 2,800+ documents/involves/targets edges of pure bookkeeping,
so the result contained tasks and executions that cannot break.
Direction is now declared per relationship type in seeds/ontology.yaml
(blast_direction: forward | backward | none), the same shape as the entity
types' monitoring: declaration, and defaults to none so an undeclared edge
contributes nothing rather than a confidently wrong answer.
It also needed a modelling fix: `routes-to` names an ingress's BACKEND, so
nothing recorded that all 21 public hostnames are terminated by caddy. A
`served-by` edge type now says so.
pool:ludo-lvm 2 -> 23 (every container storing on it, then their services)
lxc:caddy 4 -> 22 (service:caddy, then all 21 ingress routes)
service:authentik 7 (what authenticates via it)
**Every ping check was reporting down.** Not a host:strong false positive: all
seven, including ws:mac-mini — the Docker host itself. The scheduler runs in
Docker on macOS, whose VM does not route ICMP to the LAN; loopback pings succeed
and every LAN ping fails. Under health aggregation each broken probe dragged its
entity to down.
The question the check exists to answer is "is it reachable", and ICMP is only
one way to ask it. checkPing now falls back to a TCP connect before concluding
anything, which restores an honest verdict for the four hosts that are genuinely
up while leaving the genuinely unreachable ones down.
TestBlastRadiusTerminatesOnCycles asserted the old direction (caddy=1,
authentik=2 — the cycle walked the wrong way); it now asserts the corrected
depths, and its exact-node-count check is relaxed because walking the right way
also surfaces the seed's own real dependents, which are correct answers.
Co-Authored-By: Claude <noreply@anthropic.com>
The window rendered the same 13 collapsible sections for every entity, sorted
only by "does it have content". Audit trail carried the same visual weight as
Health, and the window answered "what data do we hold about X?" rather than
"what do I need to know, and what should I do?".
Measured against prod: host:hubris has 223 relations, 1,601 events, 2.7M metric
samples and 148 executions; an ingress route has three facts. Both got 13
identical headers. Expanding a host put ~540 interactive elements on screen.
- **A verdict header that never collapses.** Not just "down" but *why*:
"ping failing · 5 of 6 checks passing". That line did not previously exist
and could not have — checks rendered as configuration, never as results.
- **Sections composed per type.** A document has no checks, metrics or blast
radius; a signal or execution is a record, not a thing. Infrastructure gets
Status/Impact/Activity/Metrics/Reference, knowledge types lead with Content,
records get a minimal view. Unknown types fall back to infrastructure so a
new entity type is never a blank window.
- **Status replaces Monitoring**, showing each check's own verdict and when it
last ran — the section that answers the header's "why".
- **Impact** finally calls /entities/{id}/blast-radius. The endpoint has existed
since the first API and had no frontend caller anywhere, despite
.agents/OIKOS.md naming blast radius as the reason the ontology exists. Its
outgoing-edges-only limitation is stated in the UI rather than hidden.
- **Activity merges four lists** (executions, signals, events, agent activity)
that were telling one story in four places.
- **Relations cap at 8 with a drill-in** — 540 interactive elements down to 126.
- **Ask Nomos** opens a task pre-scoped to what you are looking at, seeded with
the verdict just computed, via an optional draft threaded through
openNewTaskWindow -> NewTaskChat -> ChatThread.
Requires exposing check_defs.last_health/last_run_at through the API (the
columns landed with the health-aggregation work but were never surfaced).
Adding a fourth enum containing "unknown" made oapi-codegen disambiguate all
enum constants by type prefix, so metrics.go moves to gen.TrendDirection*.
Verdict derivation and type->section composition live in $lib/entityView.ts as
pure functions with 15 unit tests, including the host:strong case that
motivated this.
Co-Authored-By: Claude <noreply@anthropic.com>
host:strong logged 226 health.changed events in one hour, oscillating
down/healthy while the host was fine throughout. host:hubris did it 126 times.
runCheck wrote entity_status.health on every check completion, so an entity's
health was simply whichever of its checks finished most recently. A host with
six checks reported whichever facet happened to be sampled last, and one
failing probe alternating with five passing ones flapped forever. resolveSignal
forced "healthy" too, a second path by which one passing probe erased another
probe's genuine failure.
On this fleet the trigger is a known false positive: the scheduler's network
vantage point cannot ICMP host:strong, so its ping check fails while every
ssh-script check succeeds. Under last-writer-wins that single probe declared
the whole host down, twice a minute.
Each check now records its own verdict (check_defs.last_health, migration 027)
and the entity's health is the worst across its enabled checks. A failing probe
now degrades the entity honestly and *stably*, without erasing what the other
five report, and health.changed fires only when that aggregate actually moves.
Checks that have never run are ignored rather than counted as unknown, so
adding a check cannot drag a known-good entity down before it has a verdict.
Also declares service:oikos in the seed. The previous commit re-pointed the mcp
ingress at it, but the entity only ever existed in the production database — so
a fresh seed (a new install, or a DR restore) failed on an unresolvable edge.
Caught by seeding an empty database rather than a copy of prod, which is the
only way that class of bug shows up.
Co-Authored-By: Claude <noreply@anthropic.com>
Two unrelated console errors.
effect_update_depth_exceeded — mine, from the previous commit. The live-update
effects both read and wrote the same state: FleetMap's health patch builds a
new `graph` object every run, and EntityDetailContent's refreshExecutions()
assigns a fresh `executions` array. Svelte tracked those reads, so each write
re-triggered the effect, which wrote again, until it gave up. The effects now
depend on liveEvents alone and do their work inside untrack(). Applied to all
four live effects, including the two that happened to settle on their own —
relying on "applyHealthEvent returns the same reference when nothing changed"
to break a feedback loop is far too subtle to leave implicit.
SyntaxError: expected expression, got '<' — pre-existing, and unrelated to the
live-update work. index.html loads /wails/runtime.js unconditionally; that file
only exists inside the Wails desktop wrapper, which serves the same dist/ from
its own asset handler. In a browser it is missing, and the SPA fallback
answered it with index.html — so the browser parsed "<!doctype html>" as
JavaScript on every single page load. The web Caddyfile now returns a real 404
for /wails/*, and more generally serves asset extensions without the SPA
fallback: a missing .js or .css answered with HTML is always a confusing parse
error rather than an honest 404.
Co-Authored-By: Claude <noreply@anthropic.com>
nomos declared `depends_on: api: condition: service_started`, which only waits
for the container to exist. It came up while the API was still binding :8090,
failed its MCP initialize with "connection refused", exited 1, and crash-looped
for ~25 seconds on every single deploy. It always recovered on its own, which
is precisely why it went unnoticed.
service_healthy waits for the API to answer, so this needs api to declare a
healthcheck — wget is BusyBox's, already present in the alpine runtime image,
so nothing new is installed. /healthz pings the database, so "healthy" means
genuinely able to serve rather than merely listening.
Co-Authored-By: Claude <noreply@anthropic.com>
Refetching everything on a health event was wasteful and churned the UI: one
container going degraded pulled down the entire fleet entity list (plus its
parent-grouping pass), or the whole fleet graph, to learn something the event
had already delivered.
health.changed / health.stale carry the new value in their payload, so the
views that hold the entity just patch it:
- Fleet table: patch the row. Only entity.* changes which entities exist, so
only that still refetches.
- Fleet map: patch the node AND graph.health[id] — healthOf() reads the side
map in preference to the node's own field, so patching only the nodes would
have left the rendered colour unchanged.
- Entity detail: patch the open entity. Signals still need a read (the event
says one was raised, not what the list now contains) but only the signals,
not the entity and checks alongside them.
Shared in $lib/health.ts, which returns the original array when an event does
not apply so unrelated rows keep their identity and do not re-render. Note it
matches on entity_id, never data.slug: the scheduler emits health.changed with
entity_id = the observed entity but slug = the *check's* slug.
Separately, events.ts had no reconnect. onerror was empty on the assumption
the browser retries, but EventSource only does that for a transient failure --
once it reaches CLOSED (an HTTP error on connect, e.g. the API restarting
during a deploy) it stays closed forever. A single blip silently froze every
live surface in the app with nothing on screen to say so. Now reconnects with
capped exponential backoff, and exports eventsConnected so a future indicator
can show when the stream is down.
Verified against live prod: flipping lxc:apps health recoloured the map node
and moved its counts (30 healthy -> 29, 9 down -> 10) with ZERO network
requests.
Co-Authored-By: Claude <noreply@anthropic.com>
The SSE stream already carried health.changed, health.stale, signal.raised,
signal.resolved and coverage.unmonitored, but two of the places that render
health never listened for them.
- The Fleet table refreshed only on entity.*, so its Health column sat at
whatever it was when the page mounted while the map view beside it — which
did listen — updated live. Health arrives on its own events, not entity.*.
Coalesced on a 400ms timer because health.stale fires once per entity during
a sweep, and refetching the whole fleet per event would mean a burst of
identical requests.
- The entity detail window loaded health, signals and monitoring once on open
and never again, so a window left on screen kept showing the health it had
at mount. That is the same staleness this whole change set has been about,
reproduced one window at a time. Now scoped by entity_id, and re-reads only
what a health or signal event can actually change rather than re-running the
full 11-request load().
Verified against live prod: flipping lxc:apps healthy -> degraded -> healthy
updated the Fleet table and an open detail window together, without a reload.
Co-Authored-By: Claude <noreply@anthropic.com>
`systemctl is-active` prints the state AND exits non-zero when a unit is not
active, so `... || echo unknown` appended a second line: STATE became
"inactive\nunknown" and the script emitted a raw newline inside a JSON string.
The scheduler rejected all 14 process checks with "invalid character '\n' in
string literal".
Latent since the script was written — process checks never actually ran,
because checkdefaults wrote an `args` config the ssh-script checker ignored.
Passing args through finally executed them and exposed it.
- head -1 keeps the state, and the fallback only fires on empty output.
- Quotes are stripped from both the unit name and the state; either would
break the hand-built JSON just as thoroughly.
- signalKind is now the constant "process" rather than "$SERVICE". Emitting
the service name minted a distinct signal kind per service (kind=paperless,
kind=qbit, …) — nothing an approval_rule can match, and it makes "how many
process checks are failing?" unanswerable.
Co-Authored-By: Claude <noreply@anthropic.com>
Rehearsing the deploy against a full copy of prod surfaced 40+ cardinality
violations that would have failed the seed. Since api/scheduler/notifier all
depend on `seed: service_completed_successfully`, and this change alters the
seed files (so the content hash changes and a full re-ingest runs for the
first time in months), that failure would have stopped those services from
starting at all.
None of them are new. The foreign-key bug in checkdefaults was aborting the
ingest earlier, during entity ingest, so ValidateCardinality at the end never
got the chance to run. Fixing the first failure revealed the next.
- `documents` was declared many-to-one, meaning a document may document at
most one entity. Nomos has been writing docs that cover several (a
fleet-wide apt audit documents every host it touched) for months, which is
reasonable — the ontology was the strict one. Now many-to-many.
- The mcp ingress still routed to service:homelab-mcp, which prod marks
deprecated: the Python MCP server on apps/105 was stopped at the Go cutover.
Nomos re-pointed it at service:oikos on 2026-07-12 and was right; the seed
was stale, and re-asserting the old edge alongside the new one is what made
it a violation.
Remaining after this: one genuine drift, `hosts target=lxc:caddy (2 edges)`,
which needs a prod data fix rather than a code change — see the follow-up.
Co-Authored-By: Claude <noreply@anthropic.com>
ListEnabledCheckDefs selected interval_s but never filtered on it, so every
enabled check ran on every 30s pass and the declared per-check intervals were
decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126
SSH connections every 30s (~363k/day) and `apt update` on every machine every
30 seconds — 14,400 mirror hits a day to answer a question that changes daily.
- check_defs.last_run_at (migration 026) + a due-ness predicate in the query.
A column rather than scheduler memory because this control plane restarts on
every deploy, and an in-memory map would re-fire every check on each restart.
- runCheck stamps last_run_at before processing the result, so a permanently
failing check backs off to its interval instead of re-running every pass.
- updates and backup-freshness drop to daily. Both answer questions whose
answers change about once a day; 60s was just the shared ssh-script default.
- last_run_at is seeded to a random offset within the interval so checks
created by the same seed do not stay in lockstep — otherwise ~165 probes
land in the same instant each minute instead of spread across it.
Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them.
Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day
(down from 14,400), with each 60s check landing at its own point in the minute.
Also renumbers 022→023, 023→024, 024→025: origin/main added its own
022_knowledge_revisions, and prod has already applied version 22. Left
colliding, prod would have skipped the monitoring_spec migration entirely and
then failed the seed on a missing column.
Co-Authored-By: Claude <noreply@anthropic.com>
Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by
discarded errors in checkdefaults:
- writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT
(slug) DO NOTHING, then wrote a check_defs row referencing it. On any
re-seed the slug already existed, the entity insert no-oped, and the FK
violated — aborting the ingest transaction and surfacing as an unrelated
failure several entities later. Re-seeding has been broken since; prod's
coverage was frozen at its first successful seed. This is what
TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting.
- shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed
to ".network" and overwrote each other; service:jellyfin collided with
lxc:jellyfin.
- The ssh-script checker never read the `args` config checkdefaults wrote, so
process_check.sh always ran without its unit name and returned "unknown".
Coverage is now 75/89. Monitoring is declared per entity type in
seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say
it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap.
coverageSweep raises an `unmonitored` signal only where a type declares
monitoring it lacks — 8 real gaps, no false positives.
Also:
- entity_types.attribute_schema was never ingested: the seed loader read
"attribute_schema" but the YAML says "attributes", so all 60 types stored
JSON null.
- ListExecutions ignored its declared target/action/correlation_id filters and
paginated on a non-unique target slug, dropping and repeating rows.
- started_at was captured but only written at terminal state, so a running
execution reported NULL for its whole life. The three MCP auto-run copies
wrote no timing at all; they are now one autoRun helper.
- SSH output was buffered to completion and discarded entirely on timeout.
Both sshExec copies now stream through a shared execlog sink into
execution_logs, and keep partial output when a command is cancelled.
- executions.correlation_id was a random per-execution uuid that correlated
nothing; it is now the chat session id, which is what lets the chat tail
live output.
- reversible_low had no auto-run branch despite policy declaring it
unattended. Since computeCommandRisk never returns it, the class only arises
when an agent declares it over a read_only command — so gating it penalised
candor without adding safety.
- backup-target gains a backup-freshness checker (portable find -mmin, since
the first target is on macOS), resolving its host by walking backs-up-to
backwards. The pre-deploy pg_dump is now a tracked backup target.
UI: an Executions section on entity detail with live output tailing, and
streamed output under a running `run` call in the chat timeline.
Migrations 022-024. Ops.svelte and context.ts exclude execution.output from
their refetch triggers, which would otherwise fire once a second per command.
Co-Authored-By: Claude <noreply@anthropic.com>
The six loading states across the Knowledge wiki (initial app load, the
reader's note/history fetches, and the four Cleanup tabs) all showed a
centered spinner with no relation to what was about to render — costing
a full reflow the instant real content landed. Replaces each with a
skeleton shaped like its actual content (tree rows, reader header +
prose, revision list + diff, cluster cards, table rows, flat lists)
using the existing shadcn Skeleton primitive already used elsewhere.
Verified each of the six by temporarily injecting a delay into
fetchWithAuth and screenshotting the transient state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the read-only stats dashboard with a three-pane wiki: a
navigator tree (group by folder/type/tag/entity), a reader/editor with
bare-slug auto-linking and revision history + diff, and a context rail
for backlinks and related notes. Adds a Cleanup mode for the drift
tools (duplicates, tag manager, orphans, trash) and a Cmd+K quick-open.
Also:
- Adds a real landing view (hero count, KPI row, Nomos-share meter,
recently-updated, busiest tags) in place of the old "Select a note"
empty state, and extends the design pass across the tree/reader/rail
(kind icons instead of repeated text badges, accent-bar selection,
constrained prose measure).
- Guards every note-selection path behind a confirm when there's an
unsaved edit in progress, so switching notes can no longer silently
discard a draft.
- Extracts the markdown-rendering CSS duplicated across ChatThread,
EntityDetailContent, and the new WikiReader into a shared
.markdown-body class in app.css, with ChatThread keeping only its
decorative deltas.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Knowledge page was read-only from the HTTP API — the only writer was
the agent's MCP upsert_knowledge tool. Adds create/update/soft-delete/
restore/trash endpoints, a DB-trigger-backed revision history (catches
both the web UI and the MCP tool), and maintenance endpoints: duplicate
detection (pg_trgm + complete-linkage clustering), tag rename/normalize,
orphan detection, and merge.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.prettierrc.json was missing "semi": false, so prettier wanted to add
semicolons to a codebase written without them (763 semicolon-free
statements vs. 150 with, in hand-written .ts; zero hand-written .svelte
files use them at all). That's why prettier --check failed on 249 files
— not because the code was unformatted, but because the config didn't
match the actual house style. Added "semi": false; left printWidth/etc
as configured (printWidth barely moves the failure count: 218/213/212
files at 100/120/140).
Ran `prettier --write .` with the corrected config. Verified
semantics-preserving before and after:
- eslint: 142 problems both before and after, byte-identical
- build passes, 38/38 tests pass
- token-stream diff (whitespace/semicolons/quotes normalized) on all
218 changed files: only 52 had any remaining token change, all either
trailing-comma removal (matching trailingComma: "none") or import/
ternary reflow — no semantic changes
- live smoke test: Knowledge, Tasks, Fleet map, and a chat window
(AgentTrace, markdown, Scope graph, activity rail) all render
correctly, no console errors
Most of the diff is shadcn/ui vendor files (lib/components/ui/) moving
from the CLI's own style (double quotes, tabs, semicolons) to house
style; re-running `shadcn-svelte add` on a component will need a
follow-up format pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- drop the unused KnowledgeItem type import
- the svelte/no-at-html-tags disable comment sat on the wrong line: the
multi-line Card.Description opening tag meant "next line" wasn't the
line with {@html}, so it never suppressed. Reformatted so the {@html}
is on its own line, directly after the disable comment. Sanitization
(DOMPurify with ALLOWED_TAGS: ['b']) is unchanged — this was a false
positive, verified live (search results still render <b> highlights,
no script/attribute injection).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Taskbar: the theme toggle is icon-only now, matching the Settings
button beside it. The theme name moves into the title/aria-label so
an icon-only control still has an accessible name and the current
theme stays discoverable on hover.
- Pages: Fleet, Knowledge, Learning, Ops and Signals used p-4 (or
p-4 md:p-6) while Tasks used p-2, so windows didn't line up. All now
p-2. App Store and Settings are deliberately untouched — they have no
root padding, using a full-bleed header whose border spans the window;
insetting them would break that.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The chat rendered one card per tool call, so a 20-call turn buried the
answer under 20 stacked cards. Merge them with the "thinking" indicator
into a single collapsible strip above the answer:
- collapsed: the live activity while running, a count once finished
- expanded: the turn's work in humanized language (reuses the activity
log's toolActivityLabel, so ten identical "run · target: host:strong"
rows now read as what they actually did)
- per row: the raw args/result, one more click in
Also flip the Activity rail to newest-first with the current step on top:
- follow-mode/auto-scroll re-anchored to the top to match, or it would
jump to the oldest entry on every new event
- pending plan steps park at the tail rather than sorting above the
running step and pushing it off the top; the goal anchors the bottom
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- FleetMap: service-centric host -> container -> service graph replacing
the WebGL 3D force graph, with health coloring, hover-to-trace blast
radius, and click-to-open
- Desktop background: configurable CSS pattern picker in Settings ->
Appearance (8 patterns, color/fill/opacity/fade/size/rotation),
replacing the hardcoded ambient graph background
- Fix missing data-orientation/data-disabled Tailwind custom variants so
the shadcn Slider's track actually renders
- Rename "Knowledge Base" app to "Fleet"; scope its table to the same
fleet entities as the graph (compute-entity descendants + service)
instead of all entities
- Remove dead code: EntityGraph, GraphBackground, categories.ts,
MultiSelectFilter (all superseded by the above)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Windows: make floating windows fully opaque (drop backdrop-blur/color-mix
transparency), add margin around windows, reduce Overview padding to p-2.
- DataTable: fix Toolbar always rendering an empty padded bar (children
slot was always truthy regardless of actual content); split header/body
into separate tables so the scrollbar no longer overlaps the sticky
header; make sort work for derived/synthetic columns by sorting on the
column's accessor instead of a nonexistent row key.
- Overview: enable sorting on Status and Task columns via accessors.
- TaskContextPanel: give the Activity pane more height by default (Scope
30% / Activity 70%), fixing that the saved split sizes were never
actually applied to the bound Pane sizes.
- windows.ts: clamp new/resized windows to the desktop viewport so
content-heavy entity windows can't grow taller than the visible screen;
fixes a bad defaultSize.height ('30vh', an invalid non-numeric value)
that had silently left window height unconstrained.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problem: the mascot's right-click menu was non-interactive — RadialMenu's
root div rendered inside MascotLayer's pointer-events-none root (and the
new DockedLayer wrapper compounded it) without re-enabling pointer-events,
so clicks passed straight through. The desktop right-click menu was a
hand-rolled positioned div, inconsistent with the rest of the UI.
Change: both menus now use the shadcn-svelte context-menu primitive
(bits-ui, portaled to <body>).
- Mascot: MascotMenu.svelte renders the action tree recursively —
children become ContextMenu.Sub (native hover sub-menu navigation,
replacing the manual breadcrumb stack), leaves become ContextMenu.Item
with onSelect. MascotLayer wraps <Mascot> in a ContextMenu.Trigger;
visibility predicates read reactively off ctx.model so items
appear/disappear live. Removed the manual menuPos/openMenu/closeMenu
machinery. RadialMenu.svelte deleted.
- Desktop: the surface's bare-desktop hit area is now a
ContextMenu.Trigger layer (absolute inset-0, pointer-events-auto)
placed before the icons/windows in the DOM. The DOM-structure gate
(icons/windows are pointer-events-auto siblings that paint on top and
intercept their own right-clicks; bare desktop falls through to the
trigger) replaces the old fragile e.currentTarget === e.target check.
Left-click blur moved onto the trigger; Undo/Redo disabled state
snapshotted via onOpenChange (canUndo/canRedo are wmkit methods).
Risk: the blocker that made the mascot menu non-interactive in the first
place — Mascot.svelte's handleContextMenu called e.stopPropagation(),
which would have prevented a ContextMenu.Trigger wrapper from ever
seeing the right-click. Removed that handler; bits-ui now owns
right-click on the mascot, left-click drag/pet passes through. The
context-menu content portals to <body>, escaping the pointer-events-none
mascot and docked layers entirely — the structural fix, not just a
component swap.
Verification: vitest 38/38; svelte-check + tsc clean for changed files;
eslint clean (the shadcn-generated ui/context-menu/* files carry the
same baseline custom_element_props_identifier warnings as the rest of
the ui/ folder, not from this change); vite build green; runtime
confirmed — right-click mascot opens the action tree with hover
sub-menus, right-click bare desktop opens Cascade/Tile/Show/Reset/
Undo/Redo, right-click on an icon or window does not.
Problem: the frontend had an implicit OS+Apps metaphor (desktop, floating
windows, an app registry) but the contract was informal — the mascot was
hardcoded into the shell, all apps were statically imported into one
800KB bundle, and there was no install/uninstall path.
Change: three phases landed.
- Phase 1 (contract + docked kind): AppDef extended with docked/noIcon
and optional geometry; the mascot registered as a docked app via a
generic DockedLayer that replaces the hardcoded <MascotLayer />;
openAppWindow branches on docked → toggleDocked; persisted docked
visibility store (absent key = visible, no APPS import to avoid a
static cycle).
- Phase 2 (lazy loading): AppDef.component is now a dynamic-import
loader; LazyApp renders with a loading skeleton; Vite code-splits
each app (main bundle 800KB→485KB); the LazyMascot wrapper is gone
since the lazy loader breaks the import cycle directly.
- Phase 3 (installable apps, local bundles): AppManifest + catalog +
installApp/uninstallApp + localStorage persistence; reactive apps
store (built-in + installed) and derived appById; App Store page;
Notes demo app; icons.ts and WindowLayer's orphan-close react to
registration so installs appear without a reload.
- Structure: data-table casing unified to PascalCase; the mislabeled
DataTable.svelte.ts (pure types, not runes) renamed to types.ts;
LazyApp colocated with its desktop-shell consumers; app-store moved
under lib/ so the dependency direction is consistent.
Risk: the app registry is now a reactive store, not a static array, so
every consumer (Desktop, DockedLayer, Taskbar, icons, windows) reads
from derived stores. Two static-cycle traps are documented in
docs/mbse/components.md §9: docked.ts must not import APPS (it would
fire a TDZ at init via the apps.ts→pages→windows.ts→here path), and
apps.ts must not statically import the mascot (the lazy loader defers
its module graph). Remote bundle loading, the /api/v1/apps endpoint,
and permission enforcement are deliberately NOT in this commit — they
are security-critical and deferred to Phase 4 with an ADR.
Verification: vitest 38/38; svelte-check + tsc clean for changed files;
eslint clean; vite build green; runtime smoke confirmed (install
Notes → icon appears → open → uninstall → icon + window gone; survives
reload). docs/mbse/components.md Component 9 and the plan updated.
Plan: plans/2026-07-21-frontend-os-apps-architecture.md
A config_mutation/destructive run() queued for approval never touched
agent_sessions.status — only ask_operator did that, setting
awaiting_input. So a task blocked on an execution approval was
indistinguishable from one still genuinely working: the frontend's
"Needs input" bucket only checks status===awaiting_input (never lit
up for these), and the idle-sweep safety net only excludes
awaiting_input from its stale-task query, so after ~30 minutes idle
it would nudge the agent and then auto-close the task with
outcome=partial while the approval was still sitting there undecided.
classifyAndGate now flips the session into awaiting_input the moment
an execution is queued (internal/mcp/server.go), and DecideApproval
flips it back to executing once the approval is approved, denied, or
revoked (internal/httpapi/approvals.go) — mirroring askOperator /
answerQuestion's existing pattern for session_questions. Both emit
task.status so the board updates live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sticky top-0 on each <th> (not the <thead> itself — more consistent
sticky support across browsers for table headers) plus a background so
scrolled rows don't show through underneath it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
text-primary instead of text-success — keeps the "done" state on-brand
with the rest of the UI (buttons, focus rings) rather than introducing
a separate green that only really worked well on the dark theme.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
text-success/50 and /60 washed out to almost nothing against the light
theme's cream card background — full-opacity text-success still reads
as a calm, muted green (not alarming) but is actually visible on both
themes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The launcher's textarea inherited the base Textarea component's default
field-sizing:content (auto-grow to fit typed content) — ChatThread's
input already overrides this with field-sizing-fixed, but the desktop
launcher never did, so the box would jump taller the moment you started
typing. Also bumps the floating-window frosted-glass opacity from 70%
to 85%: backdrop-filter's blur strength isn't consistent across
engines, and Firefox blurs noticeably less than Chromium at the same
radius, making the Chromium-tuned opacity look far too see-through
there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pending operator-question card now renders inline in ChatThread (the
newest thing in the conversation) instead of in the context rail — it's
part of the chat, not a separate side panel, and the panel's hasContext
gate no longer needs to special-case it.
The desktop mascot's reactions are now entirely about whichever task
window has focus, not fleet-wide events: thinking/talking is a new
continuous `busy` behavior that tracks the focused session's own
streaming state (thinking before any text arrives, talking once it
does — using the previously-unwired peep/talk sprite), eureka fires with
the actual knowledge title that was recorded, happy fires with the
task's own completion summary, and alarmed now means "this task needs
your OK" (an operator question was raised) rather than a fleet-wide
critical/signal event.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New Task now opens directly as an empty ChatThread (NewTaskChat) instead of
a separate compose screen, sized like a real task window. The Scope/Activity
context rail in a task window no longer renders until there's actually
something to show (touched entities, activity, or an open question),
avoiding an empty-placeholder sidebar on every new task. Also fixes the
chat input defaulting to several lines tall on window open, centers the
empty-chat greeting vertically, and gives floating windows the same
frosted-glass look as the desktop's task launcher card.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Audits and fixes ground-teleport/flat-fall/toss-momentum physics bugs,
fixes drag getting stuck via missing pointercancel handling, replaces
sprite-based speech bubbles with real HTML text/emoji bubbles, adds
drag-onto-icon "investigate" reactions and idle chatter, merges the
name badge and reaction bubble into one floating element, and caps the
bubble to one line with a teleprompter-style auto-scroll instead of
ellipsizing overflow text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements plans/2026-07-20-desktop-mascot.md. New code under
web/src/lib/mascot/ (types/sprites/render/state/behavior/actions/
stimuli + Mascot/MascotLayer/RadialMenu/NameDialog components) plus
CC0 sprite sheets at web/public/mascot/ (chicken + Onocentaur egg pack
+ reaction bubbles). MascotLayer is inserted into Desktop.svelte after
WindowLayer; <2-line integration.
Tamagotchi: egg -> chick -> adult lifecycle persisted to
localStorage['oikos-mascot'] (debounced 300ms). Egg hatches on first
naming (no timed incubation per implementation deviation). Chick/adult
wander, peck, sleep, blink autonomously via a weighted-random FSM; the
chicken walks above windows (ground line = highest window top edge
beneath its x, recomputed each tick from wmState; rides the ground when
the window beneath is dragged).
Interaction: draggable with flutter-fall physics on release mid-air;
plain click = pet (heart bubble + happy anim); right-click opens a
rounded-button radial menu (Interact/Care/Identity/Debug nested groups)
mirroring the desktop's own right-click menu styling; auto-flips above/
left near screen edges.
Awareness: stimulus bus subscribes to chat.ts streaming, activity.ts
activityLog (knowledge-entry diff), events.ts liveEvents (critical/
signal -> alarmed, execution -> happy), with priority+cooldown gating.
Egg-stage reactions are suppressed. Reaction bubbles are anti-aliased.
Sprite loop runs at ~60fps via setTimeout (not rAF) per GraphBackground
convention, dt clamped to 100ms; position via transform: translate3d
+ will-change: transform for compositor-friendly motion. Z-index
ordering: WindowLayer z-40 < MascotLayer z-[45] < desktop context menu
z-50 < RadialMenu/NameDialog z-[60].
Docs: plan + docs/mascot/README.md (MBSE subsystem model) updated to
Implemented with a deviations note covering hatch-on-naming, PNG-sheet
art, button-column radial menu, 60fps loop, egg-reaction suppression,
and window-walking ground model. VERSION bumped 0.7.13 -> 0.8.0.
Classifier now unwraps pct exec / qm guest exec / bash -c / sh -c / sudo
and env-var assignments before classification, so read-only inspection
wrapped in pct exec no longer escalates to config_mutation. curl GET
(default method, no -d/-F/-T/-o/>) is read-only. Eliminates the three
duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) that bounced
off the classifier for the same goal.
New classify_command MCP tool: command-scoped preflight that returns the
exact risk class run would assign. Documented in SOUL.md with guidance
to pre-classify before run when the verdict is uncertain.
set_goal surfaces prior partial/failed sessions from the last 24h so the
agent picks up the thread instead of rediscovering it.
completeTask auto-closes in-flight plan steps (pending/running -> done
on success, skipped on partial/failure), so one-step plans no longer
need the per-step running->done dance right before completion.
Migration 021 adds blocker + closed_at to agent_sessions. completeTask
sets closed_at once and derives a structured blocker reason
(approval_timeout, user_abandoned, classifier_overreach, model_refusal,
tool_error, ...) from the last assistant message.
/sessions list now carries message_count, tool_call_count,
duration_seconds (server-side aggregates — no more N+1 transcript
fetches to audit a fleet). GET /sessions/{id} returns both metadata
and messages. New query params filter + paginate: outcome, status,
entity_id, blocker, since (RFC3339 or Go duration), cursor, limit.
Titles now prefer the goal when set; sessions without a goal fall back
to the first assistant text.
New GET /sessions/{id}/tool_calls flat view for audit scripts.
Plan: plans/2026-07-20-session-review-ten-sessions.md. VERSION 0.7.12 -> 0.7.13.
Design-only (no code yet): an MBSE subsystem model for a chicken mascot
that roams the desktop shell, is draggable, opens a Sims-style nested
radial menu, and has a tamagotchi lifecycle (egg -> chick -> adult) that
reacts to real app activity (chat streaming, knowledge-graph writes,
signals). Everything (animations, autonomous behaviors, menu actions,
environment reactions) is scoped as a data-driven registry for easy
extension.
- docs/mascot/README.md: subsystem Model conforming to docs/mbse's
Holt-based Framework — mission/boundary, requirements, structural view
(module registry map), behavioral view (behavior FSM + lifecycle state
machines + a stimulus sequence diagram), interfaces view (which web
stores it observes, read-only), extension guide, verification view.
- plans/2026-07-20-desktop-mascot.md: the concrete file-by-file
implementation plan for web/src/lib/mascot/ derived from the model,
with an ordered build sequence and a manual browser verification
checklist.
- Indexed both in docs/index.md and plans/index.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The taskbar's gear icon reopened the full-page "Connect to Oikos" screen
even once already connected. Split that: Config.svelte stays as the
first-run/unconfigured screen; a new Settings app (windowed, like Tasks or
Operations) now handles in-session changes, with a section list (Connection,
Appearance) built to grow — future settings are one more entry, not a new
screen.
- pages/Settings.svelte: Connection (server URL/token/Authentik, reusing
config.ts + oidc.ts) and Appearance (Terracotta/Carbon picker) sections.
- apps.ts: registered as a normal desktop app.
- Taskbar's gear button now opens the Settings window; removed the
onOpenConnection prop threaded through App -> Desktop -> Taskbar, since
Settings' "Forget saved connection" (clear config + reload) replaces it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace hand-rolled pointer-resize logic (TaskContextPanel's 3-way vertical
split, SessionChatWindow's rail, ChatThread's message/input split) with
svelte-splitpanes, themed onto the app's existing border/primary tokens.
- TaskContextPanel: Scope/Plan/Event-log sections collapse to a fixed header
height and restore their last size on reopen.
- ChatThread: input area is now a separate resizable pane, clamped to a
measured one-line minimum and a 45% max, instead of a fixed max-h textarea.
- Send button restyled to sit inside the input's corner (Claude-style),
swapping the up-arrow for a corner-down-left return icon.
- Adds a $app/environment shim + optimizeDeps exclude, since
svelte-splitpanes assumes SvelteKit and this is a plain Vite app.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the sidebar + hash-routed page shell with a desktop metaphor:
draggable app icons, apps opening as floating wmkit windows, a centered
"What should Nomos do?" task launcher, and a bottom taskbar showing all
open windows plus a system tray.
- New app registry ($lib/apps.ts) — adding an app is one entry, nothing
else to touch.
- New desktop shell components (Desktop, WindowLayer, DesktopIcon,
Taskbar, TaskLauncher) under $lib/components/desktop-shell/.
- Icon positions are a persisted, collision-avoiding grid ($lib/stores/icons.ts).
- Window layout persists across reloads (wmkit persist), with
drag-to-maximize, F6 window cycling, and now a right-click desktop menu
(cascade/tile/show desktop/reset icons) plus Cmd/Ctrl+Z undo/redo for
window moves, resizes, and closes.
- Taskbar buttons get a hover-close and self-correct their title once a
new task's real goal is known.
- New task windows (desktop launcher and the Tasks app's "New task"
button) open as a window, not a dialog, and hand off to the real
session window once the backend assigns an id.
- Fixed a real gap along the way: GET /sessions/{id} couldn't tell
"session deleted" from "session has no messages yet" (both returned
200 with an empty list) — cmd/nomos/main.go now checks existence and
404s, so a stale/persisted task window shows "Task not found" instead
of a misleadingly empty, live-looking chat.
- Test coverage for the new pure logic (icon placement/collision
avoidance, app registry id helpers) plus a vitest matchMedia polyfill
needed to import anything touching the theme store.
Deletes the now-superseded sidebar shell, MinimizedWindowsBar, and the
standalone Chat/EntityDetail pages (folded into the window layer).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Clicking a task now opens it as a wmkit floating window (like entity
windows already do) instead of navigating away from wherever you were.
Several task windows can be open and actively streaming at once, each
fully independent — no "which one's on screen" guard needed, since
each window owns its own store bundle:
- chat.ts: chatFor(sessionId)/loadSessionChat/sendSessionMessage give
each window its own messages/streaming/connectionState, alongside
the existing singleton path the main Chat page still uses unchanged.
- workspace.ts: same split for plan/questions/touched/health-diffs
(workspaceFor/startSessionWorkspace), each with its own live-event
watermark since several windows can watch the same event stream.
- activity.ts: activityLogFor(sessionId) mirrors the global derivation.
SessionGraph.svelte, OperatorQuestion.svelte, and ActivityTimeline.svelte
were converted from store-importing to prop-driven (matching the new
ChatThread.svelte, extracted from Chat.svelte's transcript/input so both
the main page and task windows share one implementation instead of
duplicating markup/styling) so each can render either the global
"current session" or a specific window's session.
Also: minimized-window taskbar chips now cap at a max width with
middle-ellipsis truncation instead of growing unbounded, and the
window header's title/action-button row is fixed to genuinely match
heights (not just share a center point) for more robust alignment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the Fleet/Network/Identity/Knowledge category tabs (which
scoped entity fetches server-side) with a single "Types" multiselect
shared by both the table and graph views — both now fetch the whole
entity set (paginated via the new fetchAllEntities) and filter
client-side, defaulting to fleet's types. Table and graph also share
one search/highlight field instead of two separately-labeled ones.
Along the way, fixed a real bug the wider entity set exposed: the
treegrid's parent/child grouping fired one fetchGraph call per
candidate root entity, fine for the old ~50-entity fleet scope but an
ERR_INSUFFICIENT_RESOURCES flood once scoped to the full ~1700-entity
set. Replaced with a single whole-graph fetch, deriving parent/child
pairs from its edges client-side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Quadratic-bezier edges instead of straight lines, and drop the
auto-refit-on-load that caused a jarring zoom/pan snap once the force
simulation settled. Also namespace each graph's dot-grid pattern id
with a per-instance uuid — multiple SessionGraph instances can now be
mounted at once (one per open task window), and duplicate SVG ids
silently blanked out every graph's background but the first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Session-review implementation for the three sessions audited in
plans/2026-07-18-session-review-three-sessions.md. v0.7.11 → v0.7.12.
P0.1 — retry cap + investigate-before-retry (cmd/nomos/retrycap.go,
agent.go): after 3 identical failing run calls in a single turn, refuse
to dispatch the call again and return a directive to investigate *why*
(ps/strace/lsof) or surface the blocker. Per-turn scope so a fresh turn
after the operator responds can retry once more. Session 1e9c7691's 20+
identical chown retries (knfsd held a kernel lock on the exported NFS
dir) is the direct motivation.
P0.2 + P1.8 + P2.10 — SOUL.md guidance: hung command is not a failed
command (investigate before retry); ask before proposing a multi-step
migration; multi-goal sessions summarize the arc not just the last goal.
P1.3 — two new runbook entities in seeds/knowledge.yaml:
- nfs-exported-dir-mutation-hang (the knfsd fchownat lock procedure:
killall → exportfs -u → mutate → exportfs -a → verify)
- netbird-mgmt-oidc-race-after-upgrade (docker restart netbird-mgmt
after ~30s for the traefik/authentik OIDC race)
P1.4 — setGoal emits task.superseded event when prior goal is overwritten
by a different goal (store.go, TestSetGoal_SupersededEvent). Session
55927f0a had two set_goal calls with the first silently abandoned.
P1.5 — inspect_path MCP tool: runs mount/df/ls/stat for one path across
up to 8 targets in one parallel call, replacing the 15+ run-call
fact-gathering fan-out sessions 1 and 2 each spent on cross-target path
tracing (tools.go, server.go: inspectPathAcrossTargets, inspectOneTarget).
P1.6 — vm: target support in run via qm guest exec (no more SSH-hop
with nested quoting). Extracted shared resolveProxmoxHostSlug for
LXC + VM, with hosts-relationship fallback when attributes.host is
absent (server.go, tools.go). Session 55927f0a's SSH-hop workarounds
for vm:zimaos are the direct motivation.
Deferred (documented in plan): P1.7 (approval window auto-extend on
timeout) and P2.9 (long-running command PENDING detection) — both
addressed at lower cost by the retry cap. Session 3's poll-after-timeout
pattern already works; the cap protects against the failure mode.
Every place that showed entity detail (Knowledge Base's right sidebar,
the EntitySheet drawer used by Knowledge and the chat session graph,
the standalone /entity/:slug page) now opens the entity in its own
floating, draggable, resizable window instead — several can be open
side by side, and clicking a relation inside one opens another,
building up a stack. Windows are managed by one global wmkit instance
(new $lib/stores/windows.ts + $lib/components/EntityDesktop.svelte,
mounted once in App.svelte), themed with the app's own card/border/ring
tokens rather than wmkit's bundled themes (app.css).
- Delete EntitySheet.svelte (redundant) and the KnowledgeBase resizable
detail pane; row/graph-node click handlers now call
openEntityWindow(slug) instead of setting local sidebar state.
- SessionGraph (chat's "Scope" mini-graph): clicking a node opens its
window directly instead of a click-through mini-detail panel with
its own resize handle and "Full detail" button — that whole
subsystem is now dead and removed. Node highlight ring is kept
(still useful to see what you last opened) and now clears itself via
an effect watching the shared window-manager store, so closing a
window drops the highlight instead of leaving it pointing at nothing
— same fix applied to Knowledge Base's row highlight.
- Compact the entity-detail panel's padding (container + each
DetailSection) now that it's typically viewed in a small window
rather than a full-height sidebar.
- Fix KnowledgeBase's browse pane losing its flex-1/min-w-0 (and thus
full width) when the wrapping single-child div around it was removed
along with the old detail-pane split.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
import.meta.env (used by main.ts's dev-token auto-config) was untyped
since that landed — vite-env.d.ts never referenced Vite's client types.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The clickable-button variant of relationRow was missing the truncate
class that the read-only span fallback already had — long slugs (task
UUIDs, exec IDs) rendered at their full pre-truncated length inside a
shrink-only flex item, overflowing the narrow detail sidebar and
wrapping to extra lines. Give both sides flex-1 + truncate so they
share the row's width evenly and always stay on one line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fetchGraph({root, depth:1}) is backed by blast_radius, which only
walks outgoing edges — so it could never surface an edge some other
entity points at this one (e.g. host:hubris —hosts→ lxc:sophia) unless
that other entity happened to also be reachable going forward from
here. The Outgoing/Incoming split was filtering correctly, but
"Incoming" was starved of data by construction.
Switch to GET /entities/{id}/relations?direction=both — a dedicated
endpoint that matches on source_id OR target_id directly — via a new
fetchEntityRelations(). Simplifies the incoming/outgoing derivation
too, since every relation returned is now actually incident to the
entity (no more sibling-edge filtering needed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Knowledge Base / Fleet browsing:
- EntityTable renders as a treegrid (arbitrary depth, expand/collapse,
ARIA row/level/expanded), grouped by parent-child relationships
derived entirely from the live ontology graph (cardinality ->
direction; typeDepth specificity for ties) rather than a hardcoded
relationship list — see loadFleetGrouping in KnowledgeBase.svelte.
- Fold Services and Storage categories into Fleet (services/pools/
volumes/datasets now nest under the compute entity or pool that
provides/contains them instead of having their own browsing tabs).
- Drop `cluster` entities from Fleet browsing so a host's `located-at`
(site) relationship wins the tree-parent slot without needing a
hardcoded priority override — member-of simply has no valid target
left to point at.
- Add a "show destroyed/inactive" Switch (default off) filtering on
entity.state, replacing an always-on checkbox.
Entity detail panel:
- Split the Relations section into Outgoing/Incoming groups (relative
to the viewed entity), and scope the section's count to edges
actually incident to it rather than the whole depth-1 neighborhood.
Dev experience:
- Auto-fill the SPA's token from the dev server's own OIKOS_API_TOKEN
(vite.config.ts define + main.ts, dev-only, only when unconfigured)
so the "Connect to Oikos" prompt doesn't reappear on every reload.
- .claude/launch.json: autoPort, since port 5173 is often already
claimed by another worktree's dev server.
Adds ui/checkbox and ui/switch (bits-ui primitives, following the
existing shadcn-svelte wrapper pattern) and fetchOntology()/
RelationshipTypeDef to api.ts. Also fixes a missing types.ts import
in api.ts (ChatEvent/MessageContent) that predates this branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
req.Params.RelType is *[]string; passing the nil pointer straight
through as a pgx query arg (both in the blast_radius() call and in
ListGraphEdges) panics because pgx can't infer the array element type
from a nil *[]string, only from a concrete (possibly nil) []string.
Dereference once up front instead. Also affected the sqlc-based
ListGraphEdges path added by the R3 refactor, which had the same bug.
Add a regression test for GET /api/v1/graph?root=X&depth=N with no
rel_type.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
R13 — on-client path reconciliation:
- AGENTS.md: 6 occurrences of /opt/homelab-context/ → /opt/homelab/
(sections 1, 2, 5, 7)
- .agents/NOMOS.md: 2 occurrences of /opt/homelab-context/ → /opt/homelab/
- CLIENTS.md already used /opt/homelab/ — now consistent across all docs.
The repo is still named 'homelab-context' (git remote), it just clones
to /opt/homelab/ on enrolled clients per CLIENTS.md.
R14 — golangci-lint/staticcheck/govulncheck tooling:
- .golangci.yml (new): config enabling govet, staticcheck, ineffassign,
unused, errcheck, gosimple, typecheck, misspell, revive. Excludes
generated code (internal/httpapi/gen/, internal/db/sqlcgen/) and
relaxes errcheck in test files.
- Makefile: split 'lint' target into vet, golangci, govulncheck subtargets.
Each checks if the tool is installed and prints install instructions
if not. 'make lint' runs all three.
- CI already had golangci-lint-action + govulncheck (both advisory);
the action auto-discovers .golangci.yml.
10 routes are registered manually on the chi router in server.go rather
than generated from openapi.yaml. Added a 'Non-OpenAPI routes' comment
block at the top of NewHandler listing each route with its structural
reason for the carve-out:
- Auth/infra: /healthz, /api/v1/auth/oidc-*, /oidc-callback — bypass
auth middleware or aren't JSON API
- SSE override: /api/v1/events/stream — re-registered for Flush()
- Ad-hoc aggregations: /knowledge/recent, /knowledge/content/{id},
/activity/recent, /activity/session/{id}, /learning/timeline,
/learning/trend — derived shapes with no schema type yet
Updated .agents/dev/CONTRIBUTING.md §OpenAPI codegen with the carve-out
policy: if an ad-hoc route stabilizes, promote it to openapi.yaml with a
proper schema and migrate the serve* function to a strict handler.
5 warnings → 0:
1. ActivityTimeline.svelte:103 — replaced deprecated <svelte:component
this={icon}> with direct dynamic component rendering ({@const IconComp
= icon}<IconComp />). In Svelte 5 runes mode, components are dynamic by
default; <svelte:component> is unnecessary.
2. DetailSection.svelte:18 — 'let open = (defaultOpen)' captured only
the initial value. Changed to (false) + to sync with
defaultOpen prop changes.
3. EntitySheet.svelte:10 — 'let currentSlug = (slug)' had the same
issue. Changed to <string|null>(null) + (the was
already there, now the initial value doesn't reference the prop).
4. theme.svelte.ts:23 — 'applyClass(current)' at module level referenced a
variable, capturing only the initial value. Changed to apply the
plain storedTheme() result for initialization; setTheme() already calls
applyClass() on changes.
5. Chat.svelte:326 — unused CSS selector '.prose-chat
:global(:first-child):is(h1,h2,h3)' replaced with explicit
:global(> h1:first-child) etc. (the :first-child pseudo wasn't matching
because the scoped wrapper div is the actual first child).
Build is now warning-free.
Rewrote .agents/domains/knowledge/schema.md and .agents/shared/llm-wiki.md
which described the deleted Python substrate (bin/homelab, oikos/cards/,
oikos/ledger.py, root inventory.yaml, knowledge/sources/, get_page/
search_docs MCP tools). Now reflect ADR 0003: Postgres DB is the single
source of truth for structured data and narrative knowledge; seeds/*.yaml
are bootstrap+DR manifests (content-hashed via seed_versions); archive/
knowledge/ is the frozen legacy wiki; MCP search_knowledge/get_entity_
knowledge replace get_page/search_docs.
Swept substrate refs in .agents/shared/{writing-style,page-templates}.md
and .agents/domains/operations/schema.md: bare inventory.yaml ->
seeds/inventory.yaml; knowledge/sources/ -> archive/knowledge/sources/
(historical); get_changelog/oikos/ledger.py -> DB audit trail / structured
document changelog field; HERMES -> Nomos.
Root inventory.yaml (618-line Python-era file superseded 2026-07-07 by
seeds/inventory.yaml) replaced with a deprecation stub pointing to the seed
and DB. Kept as a stub rather than deleted because AGENTS.md §1/§2 still
point clients at /opt/homelab-context/inventory.yaml; full on-client path
reconciliation deferred to R13.
Flagged export gap: oikos export regenerates seeds/{ontology,inventory,
policy}.yaml but NOT seeds/knowledge.yaml — API-added knowledge lives only
in the DB until hand-edited into the seed.
VERSION 0.7.7 -> 0.7.8. Plan R5 marked done.
Deleted 8 genuinely unused sqlc queries (no inline equivalent):
- UpsertCurrentRelationship, ListEntitiesCapped, ListEntityStatus,
UpdateSignalState, InsertClassification, InsertFeedback, InsertSkill,
UpsertCurrentRelationship — all had zero call sites.
Migrated 9 inline raw SQL sites to use sqlc queries:
- GetOntology (impl.go): ListEntityTypes, ListRelationshipTypes,
ListLifecycleDefs — replaces 3 raw pool.Query blocks with typed sqlcgen
calls, eliminating manual row scanning.
- EndRelationship (phase3.go): EndCurrentRelationship — replaces tx.Exec
with sqlcgen.New(tx).EndCurrentRelationship.
- checkPrecondition (impl.go): GetEntityStatus — replaces tx.QueryRow +
manual Scan with sqlcgen.New(tx).GetEntityStatus.
- GetEntityRelations (impl.go): ListEntityRelations — replaces raw pool.Query
+ scanRelationships helper (now deleted).
- GetGraph (impl.go): ListGraphEdges — replaces raw pool.Query +
scanRelationships.
- resolveEntityID (impl.go): GetEntityBySlug/GetEntityByID — replaces
raw pool.QueryRow + Scan.
- createApproval (mcp/server.go): InsertApproval — replaces raw pool.Exec
with sqlcgen.InsertApproval.
Deleted scanRelationships helper (was only used by the two migrated
graph queries above).
Regenerated sqlcgen — also picks up stale model updates (AgentSession,
SessionPlanStep, SessionQuestion, etc. from recent migrations).
Documented the carve-out in .agents/dev/CONTRIBUTING.md §SQL conventions:
sqlc is the default; raw pool.Query/Exec is reserved for LISTEN/NOTIFY,
dynamic WHERE builders, blast_radius(), and COPY.
go vet, build, httpapi/mcp/db tests all pass. -383/+170 lines.
Tool-renderer registry (21 files, ~1.5k lines):
- src/lib/tool-renderers.ts — registry + getToolRenderer (exported, never
imported anywhere)
- src/lib/renderers/index.ts + 10 .ts registrars + 10 .svelte components
- main.ts: removed the requestAnimationFrame(() => import('./lib/renderers'))
that was the only thing keeping the dead subsystem alive
Dead components (never imported):
- ToolCallGroup, PlanProgress, GoalHeader, InlineApproval, SessionDigest
Dead store exports (written, never read):
- context.ts: pendingApprovals writable (+ Approval type import)
- events.ts: connectionState writable (+ its .set() calls)
Dead API surface:
- api.ts: SessionDigest interface + fetchSessionDigest (only caller was the
dead SessionDigest.svelte)
Dead npm deps:
- mode-watcher (0 imports; superseded by stores/theme.svelte.ts)
- @internationalized/date (0 imports)
Also: fix stale comments referencing deleted symbols, update plan R1/R2
status. Build clean (4683 modules, down from 4706; one Svelte 5 warning
gone — the dead HealthSummary.svelte was emitting state_referenced_locally).
- internal/httpapi/stubs.go: delete — 5-line comment-only orphan file with
no declarations; its own comment said the stubs live in phase3.go.
- internal/notifier/notifier.go: delete VerifyApprovalToken — zero call
sites; phase3.go:DecideApproval reimplements the check inline (noted as
dead in docs/mbse). hashToken stays (used by generateApprovalToken).
- internal/checkdefaults/defaults.go: unexport ResolveHost, ForEntityType,
ShortSlug, DefaultInterval — only called within the package. Ensure stays
exported (called by internal/db/seed.go).
go vet, go build, and affected tests pass.
Four cross-linked documents under docs/mbse/, structured after Jon Holt's
Systems Engineering Demystified (2nd ed.): Framework = Ontology + Viewpoints,
producing a Model made of Views.
- framework.md — the Ontology (SE meta-concepts + Oikos's domain ontology)
and an 11-entry Viewpoint catalog (two repeating: Component, Ontology).
- README.md — the Model's 9 concern-based Views (mission, requirements,
functional/physical architecture, interfaces, behavior, V&V, risk, roadmap).
- components.md — 8 per-component Views going one layer deeper into each
running part of the system's own internal structure.
- ontology.md — 4 Views on the domain ontology itself: entity type
hierarchy (split into 9 digestible per-domain diagrams), full relationship
catalog, lifecycle state machines with their requires: gates, and concrete
population.
Grounded in direct verification against source (grep/read), not just
existing docs — every finding is graded verified vs. per-research-pass.
Surfaced several real, previously undocumented findings along the way:
the policy kill-switch (global.auto_act/never_auto_act) is checked only by
dead code and an unstarted actuator package, so it doesn't gate the live
run path; internal/actuator and internal/learning are compiled but never
started by any process; the relationship catalog grew from 34 to 47 types
since ADR-0014; and task has no registered lifecycle_defs entry despite
having a documented, code-enforced state machine.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The approval entry type was removed from ActivityEntry upstream; this
case/import were unreachable leftovers after merging that change in.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Plan/Activity panels: humanize step titles, richer icons, empty states
matching Scope's illustration style, pretty-printed expandable detail
- Activity log renamed to Event log; every step now expandable
- Chat: middle-truncate header title, remove redundant task-list rail and
header stat cluster (duplicated in the sidebar), simplify markdown styling
- Fix --font-mono actually being a monospace font (was aliased to DM Sans)
- Replace rotating loader-circle spinner with a smoother fading-blade Spinner
- SessionGraph entity detail panel: resizable and self-clamping against its
live container size (was overflowing into sibling sections), close button
- Dev launch config: fetch bearer token from the running api container so
`npm run dev` works against the local compose stack without a hardcoded
secret in a tracked file
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The auto-complete fired when the agent hit the P5 approval gate — it
queued a config_mutation run for approval, the P5 gate blocked further
runs, the turn ended, and auto-complete closed the session as 'partial'.
The operator's approval would then land on a dead task.
Fix: hasPendingApprovals check — if the session has any executions in
pending_approval state, skip auto-complete. The session stays in
'executing' until the operator approves (or denies).
VERSION 0.7.5 → 0.7.6
The len(pending)>0 path still injected a brief note saying 'execution(s)
are now running' — the model saw this, thought work was being done for
it, and no-op'd (finish_reason=stop, content_len=0). Same confusion as
the len(pending)==0 case, just from the other branch.
Fix: both assent paths are now fully silent. No system note at all. The
model sees 'go ahead' in the replayed history and responds naturally.
Also removed chat_assent tool_use/tool_result emit events. These were
persisted in the transcript and confused the model on replay — it saw
its own 'tool calls' (chat_assent) and thought it had already acted.
VERSION 0.7.4 → 0.7.5
The auto-complete safety net required hadEntityWriteback to be true,
which meant sessions where the agent did the work but forgot to call
update_entity_attributes stayed stuck in 'executing' forever.
Relax: auto-complete fires if the agent did discovery (ran run),
regardless of writeback. If writeback happened → success; if not →
partial (honest: work was done but knowledge graph not updated).
VERSION 0.7.3 → 0.7.4
When the chat handler approves a pending execution via chat-assent, the
execution completes in ~2s. The continuation worker detects the completed
execution and calls resumeSession — while the chat handler is still
processing 'go ahead'. Two concurrent LLM calls for the same session cause
empty responses (finish_reason=stop) and race conditions.
Fix: mark the execution as continued immediately after chat-assent grants
it, so the continuation worker skips it. The chat handler will drive the
continuation itself (the model sees 'go ahead' and executes the plan).
VERSION 0.7.2 → 0.7.3
The assent pre-processing injected verbose system notes ('the operator
approved... they are now running... you MUST continue...') on top of the
replayed user message ('go ahead'). The model saw both, latched onto
'now running', concluded the work was being done for it, and no-op'd
(finish_reason=stop, content_len=0) — leaving the session stuck in
'executing'.
Root cause: the model already sees 'go ahead' in the replayed history
(the user message is saved to the DB before chat() is called, and
getRecentMessages replays it). The system note was redundant AND
confusing — it told the model work was 'running' when it wasn't.
Fix:
- len(pending)==0 (plan-proposal approval): open assent window silently.
No system note. The model sees 'go ahead' and responds naturally.
- len(pending)>0 (actual pending executions): brief note naming the
specific execution IDs that were approved ('don't re-request those').
No 'continue the plan' directive — the model knows to continue.
VERSION 0.7.1 → 0.7.2
P1: add docker compose (logs|ps|top|config|images|port|cp) to read-only
allowlist. docker compose logs was classified as config_mutation, causing
individual approval cards for read-only inspection commands.
P2: remove approval entries from activityLog. They were always status=running
and never transitioned to done (the derived store builds from tool-call
text, not execution status), causing AgentIndicator to latch onto a stale
'Approval: ...' entry and never clear — even after the session completed.
P3: remove InlineApproval from Chat.svelte. The green 'Completed in 1s on
lxc:...' boxes were noise in the chat stream. Approval UX belongs in the
Operations page (already has it via Ops.svelte), not inline in the chat.
P4: stale execution cleanup. Startup sweep (mark >1hr non-terminal as
cancelled) + 5-min periodic sweep (mark >10min non-terminal as cancelled).
98 orphaned executions accumulated from eval testing (39 running from
apt_upgrade:audit timeouts, 19 pending_approval, 3 approved).
P5: refuse second config_mutation run when an approval is already pending
for the session. Without this, the agent queues N individual approvals
before the operator can respond — confirmed in session 20757eb9 (two
approval cards for what should have been one plan-level approval).
VERSION 0.7.0 → 0.7.1
The agent legitimately runs 20+ diagnostic commands for a config_mutation
task (reset service, re-run backup, verify, check logs). Cap of 8 was
too strict.
The agent often skips update_plan_step bookkeeping (leaving steps
pending/running) but still does the work + writeback. The strict
allPlanStepsTerminal check missed these cases.
Add path (b): if the agent did discovery (ran `run`) AND wrote back
(update_entity_attributes/create_relationship), auto-complete. D.1 already
enforces writeback before completion — if writeback happened, the work
is done.
The #1 remaining model reliability gap: the agent does the work (proposes
plan, executes all steps, writes back) but forgets to call complete_task,
leaving the session stuck in 'executing'. The eval showed 3/8 failures
with this pattern.
Fix: autoCompleteIfPlanDone — a structural safety net that fires at both
chat exit paths (normal completion + maxIterations). If the session has a
goal, the agent didn't call complete_task, and ALL plan steps are in a
terminal state (done/failed/replaced/skipped/blocked), auto-complete with
the agent's final text as the summary. Mirrors autoCompleteTrivialTask
but for structured tasks where the work is provably done.
Also: bump maxLLMRetries from 2 to 3 (complex multi-turn flows benefit
from one more retry on empty responses).
The assent system note said 'Do not re-request or call run again for
these' — the LLM interpreted this as 'don't call run at all' and produced
empty responses (finish_reason=stop, content_len=0) until retries were
exhausted, leaving the session stuck in 'executing'.
Fix: rewrite both assent notes (pending-approval path and pure-plan-approval
path) to be directive about WHAT TO DO NEXT: call update_plan_step(running)
then run for each remaining step. The 'don't re-request' guidance is now
scoped to 'THOSE SPECIFIC' executions, not all run calls.
Also bump maxLLMRetries from 2 to 3 — the empty-response flake on complex
multi-turn flows benefits from one more retry.
reopenSession was replacing plan steps on every follow-up message —
including approvals ('go ahead') — which destroyed the plan the operator
just approved, leaving the agent unable to track step progress and looping
run calls until maxIterations.
Fix: setGoal is the explicit signal for 'new sub-task' (the agent calls
it at the start of each follow-up direction). Step replacement now happens
there, not in reopenSession. An approval ('go ahead') does NOT call
set_goal, so the plan stays intact and the agent can execute + complete
it.
The check len(lastAssistantCalls) == 0 was too restrictive — it only
fired when the assistant had ZERO tool calls. But propose_plan + pre-plan
research are tool calls, so the assent window never opened when the
operator said 'go ahead' after a plan proposal. The agent then tried to
execute config_mutation run calls without the assent window, they queued
for approval, and the turn deadlocked.
Fix: check len(pending) == 0 (no pending APPROVALS) instead of
len(lastAssistantCalls) == 0 (no tool calls at all).
A follow-up on an executing session (first turn didn't complete_task) is
still a new direction — the old plan's steps must not block the new one.
Previously reopenSession was a no-op for executing sessions, leaving done
steps that caused errPlanInFlight on the next propose_plan call.
proposePlan: mark pending steps as 'replaced' instead of DELETE, so the
generation counter (MAX+1) sees prior generations. Without this, a first
plan that was proposed but never executed would be wiped, resetting the
counter — a follow-up's plan would look like generation 1 instead of 2.
plan-always-readonly: raise max_run_calls from 3 to 6 (agent inspects
thoroughly).
SOUL.md step 4: all-read-only plans skip the approval wait and execute
immediately. Only config_mutation/destructive steps need operator approval.
set_goal + propose_plan return text updated to match.
Fixes 3/4 eval failures where the agent proposed a plan then waited
for approval on a read-only task.
plan-always-readonly: prompt now demands live systemd timer inspection,
not just DB lookup. Added calls_tool: run assertion.
iteration-followup: added 'go ahead' as second followup so the
config_mutation plan gets approved and can execute.
iteration-readonly: replaced nonexistent lxc:prometheus with lxc:dns,
keep it read-only so no approval needed.
P1 plan-first: run handler refuses without propose_plan (structural gate,
not SOUL.md prose). Plan window decoupled from set_goal — config_mutation
auto-run only on operator approval (assent window). Closes the approval-free
config_mutation hole confirmed in session d0d562e0.
P2 iteration: reopenSession flips terminal→executing, marks prior plan steps
replaced, clears outcome. proposePlan excludes replaced from in-flight check,
bumps generation. A follow-up on a completed session starts a new sub-task
with a fresh plan — no more errPlanInFlight dead end.
P3 reasoning: accumulate per-iteration text into the persisted row instead
of overwriting with the last text event. Reload shows intermediate thinking,
not just the final summary.
P4 read-only allowlist: add find, tree, locate, systemctl list-timers/
list-unit-files/show, timedatectl, hostnamectl, systemd-analyze, rclone
ls/lsl/md5sum/check/cryptcheck. Fixes the find misclassification from
d0d562e0.
P5 eval harness: new assertion kinds (proposes_plan, plan_before_run,
plan_generations), multi-turn followups, fetch /sessions/{id}/plan. Four
manifests under evals/.
P6 SOUL.md: strip degenerate-case carve-out, add ITERATE step, update
set_goal guidance.
VERSION 0.6.0 → 0.7.0
- Terracotta (light) and Carbon (dark) themes with toggle
- Inknut Antiqua headings, DM Sans body
- Dot grid background on EntityGraph and GraphBackground
- Theme-adaptive graph colors on EntityGraph
- Art Nouveau chat styling (borders, underlines, blockquote quotes)
- Bullet point styles in chat prose
- Task goal in header, rename Overview→Tasks, New Task labels
- Logo uses var(--primary) for theme awareness
The unrooted graph endpoint caps at 500 entities with ORDER BY e.slug, which fills the cap with exec:* rows and excludes every host/lxc/service/vm entity. Since edges require both endpoints in the node set (ANY/ANY), 99.9% of edges were dropped — 500 nodes but only 1 edge survived.
Fix: select the 500 most-connected entities (by relationship count descending) so the topology is preserved. Result: 500 nodes, 900 edges across all relationship types.
The OIDC fix was committed in 3b98097 by a concurrent session. The plan's
status block was stale ('not yet committed/deployed') — updated to reflect
it's done. No remaining open items in this plan.
Status: In Progress → Done. All 18 fixes (A.1-A.3, B.1-B.6, C.1-C.2, D.1-D.2,
E.1-E.2, F.1-F.3) shipped in commits 337d577 + 3de359b + dd3076a, deployed
to oikos-nomos-1 (v0.5.3). The golden eval harness (cmd/nomos/eval/) passes
4/4 conversations, validating the structural gates + the SOUL.md
consolidation. Also fixed a pre-existing tool-call doubling bug found by
the eval harness.
Only remaining open item: the OIDC token-refresh fix (PM addition, web/src/
lib/{config,oidc,events}.ts) — implemented, not yet committed/deployed.
Ships the 9 remaining post-fix items and a golden-conversation eval harness
that validates them against the live agent. All 4 evals pass.
SOUL.md (F.1, C.2, E.1):
- Consolidated three overlapping task-flow sections (MANDATORY TASK FLOW,
'Every chat is a task', 'AFTER EVERY TASK: WRITE BACK') into one. ~50
lines shorter. The operator's 'be more crisp' feedback.
- Added anti-patterns: don't re-execute on UI/sidebar complaints (C.2);
don't re-run fleet-wide audits when same-day knowledge exists (E.1).
- Updated approval vocabulary in step 4 to match tasks.go (approved/yes/
go/proceed/continue/ok/go ahead).
Tool-result strings (F.2):
- set_goal: tightened to 'Goal set. NEXT: pre-plan (read-only tools only).
Then propose_plan. Do not call run.'
- update_plan_step: added '(Advance with update_plan_step + run; do not
re-propose.)'
C.1 — completeTask rejects re-completion of a terminal session:
- Returns errTaskAlreadyComplete when status is already done/failed.
- The tool result directs: 'Task is already complete. Do not call
complete_task again. If the operator pointed out a UI/sidebar
inconsistency, fix it with update_plan_step...'
B.4 — Surface real model error text:
- chatWith's error event now includes finish_reason + refusal text:
'Nomos returned an empty or unusable response (finish_reason=length).
Retry or rephrase.' instead of generic 'empty response'.
- The resume-failed note already carried errText (B.3), which now has
the real context.
B.5 — Back off between resume retries (4s, 8s):
- resumeSession now sleeps before attempts 1 and 2 (exponential backoff).
A transient provider issue gets time to clear instead of 3 identical
calls in 3 seconds.
B.6 — Don't persist the empty placeholder as a visible bubble:
- If a chat turn ends with no text and no tool calls (model empty-response'd
and all retries failed), delete the placeholder row instead of persisting
an empty bubble. The error was already streamed via done+error=true.
E.2 — list_lxcs last-audited hint:
- The list_lxcs result now includes last_audited_at — the most recent
knowledge entry (tagged audit/update, or titled audit/update) linked
via an 'about' edge. The agent can see 'nextcloud — last audited today'
and skip re-running it.
Tool-call doubling bug fix (found by the eval harness):
- main.go + continue.go: the tool_use and tool_result events were both
appending separate entries to the persisted tool_calls array, doubling
every tool call in the transcript. Confirmed pre-existing (d9cdcee1,
v0.3.x era). Fixed: tool_use creates the entry, tool_result merges the
result into the same entry (matched by id). One entry per tool call.
Golden eval harness (cmd/nomos/eval/):
- A standalone Go program that loads YAML manifests of golden conversations
+ assertions, sends prompts to the chat endpoint, drains the SSE stream
(keeping the agent's context alive), and scores structural assertions
against the persisted transcript.
- 4 golden conversations covering: trivial read-only (degenerate case),
plan + proceed (the original duplication bug), UI complaint (no re-exec),
fleet audit (knowledge preferred over re-execution).
- Structural assertions only (tool-call sequences, plan steps, writeback,
completion) — text quality is model-dependent and not scored.
- Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest
cmd/nomos/eval/evals/*.yaml (~$0.10/run in OpenRouter credits).
Eval results (4/4 passed):
trivial_readonly: 2 tool calls, no plan, no run
plan_advances_on_proceed: 13 tool calls, propose_plan x1, writes back
ui_complaint_no_rerun: 12 tool calls, propose_plan x1, writes back
knowledge_preferred_over_rerun: 7 tool calls, search_knowledge x1, 0 run
Version 0.5.2 -> 0.5.3 (minor: eval harness + structural hardening).
D.1 (complete_task refused without writeback) and D.2 (propose_plan
auto-appends writeback step) shipped in 3de359b (v0.5.1), e2e-validated
against the live agent. The knowledge loop is now structurally closed —
no blockers remain. Remaining items (F.1, F.2, C.1, C.2, B.4-B.6, E.1,
E.2) are all friction/cosmetic.
The overview background graph and the Knowledge Base graph both rendered
empty because the SPA's OIDC access token expired (~5 min TTL) and was
never refreshed. fetchWithAuth called getToken() synchronously (no refresh);
ensureToken returned the stale token without refreshing; storeTokens
discarded expires_in; the resulting 401 made fetchGraph return null and
both graphs drew nothing, with no error surfaced.
- oidc.ts: track expiresAt from expires_in; getToken() returns null within
30s of expiry; ensureToken/initOIDC refresh instead of returning stale
tokens; isOIDCConfigured no longer claims configured on expired-only state
- config.ts: fetchWithAuth awaits ensureToken (refresh on demand), falls
back to static token if OIDC can't yield one, flushes OIDC session on 401;
sseUrl is async + refreshes before constructing the EventSource
- stores/events.ts: connect() awaits the now-async sseUrl
D.1 — complete_task structural gate:
- hadDiscovery(ctx, session) reports whether the session ran `run` successfully
against a live target (NOT get_entity/list_lxcs — those are DB lookups, not
new facts). A trivial Q&A that only calls get_entity is a degenerate case
and must NOT be blocked.
- complete_task with outcome=success is REFUSED when hadDiscovery && !
hadEntityWriteback. The refusal fires BEFORE completeTask runs, so the
session stays in 'executing' state and the agent must call
update_entity_attributes/create_relationship then retry complete_task.
An explicit failure/partial is allowed through (the agent is acknowledging
it didn't finish — no reason to force writeback).
- Replaces the prior advisory warning (5.5) which the agent consistently
ignored. The agent saw the warning and ended the task anyway; this gate
makes the writeback a hard prerequisite for success.
D.2 — propose_plan auto-append writeback step:
- When the agent proposes a plan whose steps don't mention
update_entity_attributes or create_relationship, D.2 appends a final
'Write back: update_entity_attributes + create_relationship +
upsert_knowledge' step before persisting. The result string tells the
agent it was appended.
- With the seq-order enforcement (5.6) and D.1's complete_task gate, the
agent must complete the writeback step (and actually call the tools) to
finish. Neither relies on the agent reading SOUL.md.
- Removed the old advisory writeback nudge from propose_plan's result
string — D.2 makes it structural.
- Updated the propose_plan tool description to state both gates crisply.
Verification:
- TestHadDiscoveryAndWriteback: hadDiscovery true only after a successful
`run`; false after failed run, get_entity, or no calls. hadEntityWriteback
true only after update_entity_attributes/create_relationship.
- e2e against the live agent (oikos-nomos-1, v0.5.1):
- D.2: agent proposed 3 steps (no writeback); D.2 auto-appended step 4
'Write back: update_entity_attributes + ...'. Result string said
'(appended a writeback step — your plan didn't include one; step 4)'.
- D.1: agent ran `run` (uptime on lxc:gitea), called complete_task, was
REFUSED ('Refused: this session ran run against live targets (discovery)
but did not call update_entity_attributes...'). Agent self-corrected:
called update_entity_attributes, retried complete_task, succeeded.
Knowledge loop closed end-to-end.
Version 0.5.0 -> 0.5.1 (patch: structural enforcement of existing intent).
Status was 'shipped & e2e-validated'; now reflects the commit (337d577),
push to main, and deploy to oikos-nomos-1 (v0.5.0) that followed the
e2e validation. D.1 (refuse complete_task without writeback) is the next
blocker.
Operator-reported bug: on 'proceed with the rest' the agent re-proposed the
plan, duplicating it in the sidebar. Root cause was a three-bug chain, not
one bug:
1. Trigger — model empty-response on 'proceed' (approval vocabulary didn't
list 'proceed', so the agent wasn't sure it was approved and no-op'd).
2. Amplifier — chatWith emitted 'error' without 'done' on empty response
(agent.go:370). The frontend's onComplete saw !receivedDone and
misclassified the model failure as a network disconnect, calling
handleDisconnect -> resumeSession.
3. Divergence — the reconnect note was generic ('report your state'), so
the agent re-proposed + re-executed instead of advancing the plan.
Fixes (shipped, e2e-validated against the live agent on oikos-nomos-1):
- A.2: proposePlan refuses re-proposal once a step has started (returns
errPlanInFlight). Drops the append-mode safety net (commit 5384499) that
was the direct source of the sidebar duplication. The agent must advance
with update_plan_step + run; the tool result directs it.
- A.1: proposePlan sets the 'generation' column on INSERT (migration 020
added the column + frontend grouping, but the INSERT never wired it).
- A.3: propose_plan tool description restated as a crisp contract (ONCE,
STOP and wait, REFUSES once a step started, advance with update_plan_step).
- F.3: approval vocabulary expanded to approved/yes/go/proceed/continue/ok/
go ahead; propose_plan result string tightened to an imperative.
- B.1: chatWith emits 'done' after 'error' on every terminal path via a new
emitError helper. The frontend now treats model errors as ended (not
disconnected), so no auto-reconnect -> resumeSession fires.
- B.2: reconnect/resume note carries the operator's last message + an
explicit 'advance the plan, do NOT call propose_plan again' directive when
a plan is in flight. Wired into all 4 resume entry points (reconnect,
/resume, idle-sweep, question-answer) via enrichResumeNote.
- B.3: resumeSession escalates the recovery note across its 3 attempts (final
retry: 'pick the lowest-pending step, mark it running, call run — do that
now') instead of 3 identical notes -> 3 identical empties.
Verification: TestProposePlan_RefuseInFlight replaces TestProposePlan_
AppendVsReplace. e2e conversations against the rebuilt container:
conv2 ('proceed with the rest') -> 0 propose_plan calls, plan stayed at
3 steps (was 6+ before), update_plan_step x5 + run x2 + complete_task.
conv3 (full plan, 'go ahead') -> apt-get update on lxc:dns auto-ran under
the plan window, update_entity_attributes writeback, clean complete_task.
nomos logs show zero reconnect/resume entries for the plan-proposing
sessions (the three-bug chain is closed).
Remaining (not in this commit): D.1 refuse complete_task without writeback
(next blocker), C.1/C.2, F.1/F.2 SOUL.md consolidation, B.4-B.6, E.1/E.2.
See plans/2026-07-14-post-fix-session-remainders.md.
Also: re-audit 2026-07-10-general-gated-execution.md — request_execution enum
retirement (60effcb) closes item 9; only auto-act revival (item 10) remains.
Version 0.4.1 -> 0.5.0 (minor: new structural behavior, not a bugfix).
SOUL.md: mandatory 6-step task flow at TOP of file, unmissable.
Agent MUST: set_goal → pre-plan (research only) → propose_plan → STOP
and wait for approval → execute (auto-run under plan window).
Backend:
- set_goal now opens plan window immediately (config_mutation auto-runs)
- set_goal result tells agent to do pre-plan + propose_plan, not run
- propose_plan result tells agent to STOP and wait for approval
- plan window value unified to 'active' (set_goal + propose_plan)
This prevents 23 individual approval popups — one plan approval instead.
- activityLog now detects 'requires approval' in tool results
- Adds approval entries with shield icon + description + execution ID
- Works for both run and remaining approval paths
Backend:
- proposePlan sets plan window in autonomy_settings (nomos:plan:<session>)
- run handler checks plan window — auto-executes config_mutation commands
within plan without per-action approval
- planWindowActive function in server.go
- Plan window cleaned up on completeTask (already covered by LIKE '%:' || )
Frontend:
- Removed 'Session graph' header bar
- Cooler empty states: Plan shows animated dots + 'Awaiting plan…',
Activity shows pulsing dots + 'Waiting for activity…'
- AgentIndicator now shows only during stream or when running tools exist
(not on session status=executing which never cleared)
- Activity timeline: oldest-first ordering (reads top-to-bottom naturally)
- Removed unused liveStatus derivation and currentTask import from Chat
- Plan: Phase A+B+C for activity gaps + plan-approve-once policy
- Tool calls in Activity timeline are now tagged with current plan step
- Indented entries show which step they belong to
- Step tracking via update_plan_step(status=running) tool calls
- Removed inline tool renderers from chat (health summary, fleet snapshot, etc.)
— all tool output now visible only in sidebar Activity timeline
- TaskContextPanel restructured into 3 collapsible sections:
Scope (graph), Plan (goal + steps + progress), Activity (timeline)
- Collapsed headers show compact live status: 'Graph', 'Step X/N', 'N actions'
- Sections are vertically resizable via drag handles
- Plan section shows goal inline + step list + progress bar
- GoalHeader and PlanProgress no longer rendered separately
- ActivityTimeline header moved to TaskContextPanel
- New ActivityTimeline: unified timeline in sidebar showing all agent actions
(goal, plan steps, tool calls, knowledge, completion) in reverse chron order
- activityLog derived store merges messages + planSteps + currentTask
- AgentIndicator stays in chat (thinking/working indicator), simplified props
- ToolCallGroup removed from chat — tools visible only in sidebar timeline
- SessionDigest replaced by ActivityTimeline
- PlanProgress restored in sidebar (conceptual steps, separate from timeline)
- New AgentIndicator component: replaces 3 separate indicators
(loading dots, ToolCallGroup summary, activity bar) with one
- Positioned as last item in message list — scrolls naturally
- Shows current tool action: 'Researching lxc:nfs-export…' etc
- Spinner during work, check on completion, X on error
- Fades out 3s after turn completes
- Activity bar, loading dots, statusLabel removed from Chat
- Continue button moved to sidebar session panel
- SessionDigest now includes live tool timeline, plan steps, knowledge
- ToolCallGroup compact: single-line with collapsible names only (no JSON)
- Activity bar moved to bottom of messages, smart scroll respects user position
- setGoal now sets status=executing (removed stuck planning state)
- PlanProgress merged into SessionDigest, removed from TaskContextPanel
- New toolTimeline derived store in chat.ts
- Activity bar moved to bottom of message list (before messagesEnd)
- Smart scroll: auto-scroll only during streaming or when near bottom
- Scrolling up pauses auto-scroll until next send
- Removed duplicate $effect block
- Plan: tool timeline in sidebar (plans/2026-07-14-tool-timeline-sidebar.md)
- VERSION file at repo root (0.3.0)
- vite.config.ts reads VERSION at build time, injects __OIKOS_VERSION__
- App.svelte shows version in sidebar tooltip + subtle text below logo
- AGENTS.md §9: every commit to main MUST bump VERSION
(patch=bugfix, minor=new features, major=breaking changes)
RegisterHook + e.Cancel() prevents Wails from destroying the WebView
when the window is closed. The app now hides to the system tray. Left-
click on the tray icon correctly restores the window.
- Tray 'Check for Updates' now checks immediately and shows dialog
- Dialog has 'Install' and 'Later' buttons
- /update/check endpoint on local server for SPA to query
The apiUrl configured on the Config page was lost when the webview
navigated away to localhost and back. Now it's included in the return
URL as ?desktop=1&apiUrl=...&token=...
SPA navigates to 127.0.0.1:18901/oidc/start, passing ret URL.
Go opens browser, waits for callback, saves token, returns HTML with
<meta refresh> back to Wails app with ?desktop=1&token=TOKEN.
main.ts extracts token from URL on reload.
SPA fetches /oidc/open (returns session ID immediately), then polls
/oidc/result every 500ms. Go server opens browser in a goroutine.
Webview never leaves the Wails origin. Token is saved to keychain and
returned through the poll response.
The webview navigates to http://127.0.0.1:18901/oidc/open?apiUrl=...
The Go server opens the system browser to Authentik, waits for callback,
exchanges code for token, saves to keychain, then redirects the webview
back with ?desktop=1&token=TOKEN. main.ts extracts the token from URL.
The local HTTP server approach (fetch to 127.0.0.1) doesn't work in the
Wails webview. Simplify: use window.open() to launch OIDC in the real
browser. After authentication, the callback page at the server shows the
token. User copies and pastes into the Token tab.
Also fix: SetSize before app.Run() crashes with nil pointer — use
WebviewWindowOptions width/height directly from restored state.
The Wails runtime isn't reliably loading for IPC calls. Replace the
binding-based StartOIDCLogin with a local HTTP server on 127.0.0.1:18901:
- /oidc/login?apiUrl=... — opens system browser, waits for token
- /oidc/callback — Authentik redirect target, exchanges code
- /oidc/config?apiUrl=... — fetches OIDC provider config
- SPA detects desktop via ?desktop=1 URL param
- SPA calls localhost directly via fetch() instead of Wails IPC
- Remove custom asset handler — it broke Wails IPC routing
- Use application.AssetFileServerFS(distFS) so Wails serves its own runtime
- Add GetStoredConfig binding: SPA calls it on startup to retrieve keychain config
- main.ts: loadDesktopConfig() fetches stored creds before mounting
- Remove runtime.js embed (Wails serves it internally)
The SPA needs /wails/runtime.js for window.wails to be available.
Since we use a custom AssetOptions.Handler, Wails' internal routing
doesn't serve it. Embed the runtime and serve it explicitly.
ConfigService.StartOIDCLogin():
- Fetches OIDC config from the API
- Generates PKCE params
- Starts local HTTP server on 127.0.0.1:18901
- Opens system browser to Authentik
- Captures callback directly (no copy-paste)
- Exchanges code for token, saves to keychain
- Returns token to SPA → auto-connects
Config.svelte detects Wails environment and calls the binding.
The old icon.icns was copied from favicon.png which was actually
a dark-background .icns file. Regenerated from favicon.svg via
qlmanage → sips → iconutil to get white logo on transparent bg.
- Server: /oidc-callback HTML page exchanges Authentik code for token,
displays it for user to copy into the desktop app's Token tab
- oidc.ts: desktop mode uses apiUrl+/oidc-callback as redirect URI,
encodes PKCE verifier in state parameter
- Config.svelte: add Server URL field to OIDC tab for desktop UX
- Caddy: add /oidc-callback to enroll bypass (no Authentik gate)
- App: favicon.png as system tray icon, window title 'Oikos'
- web/index.html: title 'Oikos'
wails3 build v3 alpha delegates to Taskfile; the go build produces a raw
binary, not a .app. Package step now creates the bundle structure
(Contents/MacOS, Info.plist) and zips it.
12 of 33 MCP tools now render as rich inline cards instead of raw JSON:
EntityCard, HealthSummary, LXCList, EntityTable, KnowledgeResults,
BlastRadius, ChangeLog, FleetSnapshot, MetricChart.
Architecture:
- Server: annotateJSONResult() wraps queryRows with __renderer hints
- Registry: match/dispatch system maps tool names to Svelte components
- Chat: inline dispatch with 5-card limit, overflow to collapsed group
- ToolCallGroup: unmatched prop, hides when all matched, ARIA labels
Tests: 3 new Go tests for annotateJSONResult (wrap, no-op, multi-row).
Problem: the Oikos control room was browser-only — no native desktop
experience (system tray, notifications, keychain-persisted auth).
Change: add a Wails v3 thin-shell desktop app at cmd/desktop/ that embeds
the existing SPA in a webview. The Go side is ~380 lines — no bundled
server, no Postgres connection. It reads auth from the OS keychain,
injects it into the SPA on load, and the SPA talks HTTPS to the homelab
same as a browser.
Phase 1.0 — Scaffold + window:
- Embed web/dist/ into the Wails binary
- Inject window.__OIKOS_CONFIG__ with keychain-stored apiUrl + token
- 1400×900 window, min 1024×700
- System tray: Open/Quit, click toggles window
Phase 1.1 — Native shell:
- Poll /api/v1/dashboard/summary every 30s; osascript notification
when approvals or critical signals increase
- Save/restore window position to ~/.config/oikos/window.json
- EnableAutoStart/DisableAutoStart — macOS LaunchAgent plist
Phase 1.2 — Token management:
- Config.svelte calls window.wails.Call.ByName('SaveConfig') after
successful connection — persists to OS keychain
- ConfigService binds SaveConfig, ClearConfig, EnableAutoStart,
DisableAutoStart to the Wails runtime
Phase 1.3 — Auto-update:
- Poll Gitea releases API every 6h, compare semver, show dialog
- 'Check for Updates' tray menu item triggers immediate poll
Phase 1.4 — Distribution:
- macOS entitlements.plist: network client + keychain access
- .gitea/workflows/desktop.yml: CI builds macOS arm64 + Linux amd64
on 'desktop-*' / 'v*' tags, attaches artifacts to release
- Makefile: desktop (build), desktop-package (build + zip/tar.gz)
- CONTRIBUTING.md: documented desktop app + commands
Risk: low. Wails v3 alpha API may shift; the Go glue is ~380 lines and
trivially portable. The desktop app is additive — zero changes to the
existing server or SPA logic. No config mutation, no infrastructure
impact.
Verification: go build, go vet, go mod tidy all pass.
Fresh nodes (no prior x/y) get placed by d3-force's default init, which
spirals out from the ORIGIN — not (width/2, height/2) — while the
centering forces here are deliberately weak (0.04, so they don't fight
the link/collide layout) and alphaDecay stops the sim before a weak force
can always pull a far-off cluster back to center. Net effect: graphs could
settle visibly off-center on load, cramped in a corner of the pane.
Fixed by computing the actual node bounding box once the simulation's
'end' event fires and setting the view transform to fit it, instead of
relying on the force balance to land on center by itself. Gated behind a
`fit` flag so passive background reloads (live entity/relationship
events) don't yank the view out from under someone actively panning or
zoomed in on a specific area — only fresh loads (mount, root/depth
change, reset, re-root) reframe.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
search_knowledge and get_entity_knowledge only ever returned a ts_headline
snippet/short headline — enough to find a note, not enough to act on it.
Add get_knowledge_content(slug), mirroring the web UI's
/api/v1/knowledge/content/{id}, so the agent can read a document/
investigation/runbook's full markdown body once it knows which one it
needs. upsert_knowledge already covered the write side. Cross-referenced
all three tool descriptions so the agent discovers the full-read path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Executions were being created with no outgoing edges to what they acted
on or which task/session drove them, silently starving the graph of new
data going forward — found during this session's DB audit, which had to
backfill 245+25 missing targets/involves edges for existing executions.
This closes the gap at the source: every execution now gets a
target-->targets-->execution edge, and (when the caller supplies a
session/task) a task-->involves-->execution edge, both idempotent
(NOT EXISTS guards) so retries/backfills don't duplicate.
Two call sites: the deduped systemctl/apt_upgrade/pct_create fast path
and the general classifyAndGate path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two bugs found while verifying against real production data:
- Excluded activity types (execution/check/task etc., see categories.ts)
were falling through inCategory's "unknown type -> always visible"
fallback, since typeCategory only stored entries whose category was
defined. That fallback exists for types the ontology never returned at
all; it wrongly re-admitted types the ontology returned but categories.ts
deliberately excludes. Fixed by storing every type (including undefined
categories) and checking key presence, not value truthiness.
- Once that was fixed, the previous commit's 1-hop neighbor expansion
(dimmed cross-category context) turned out fine for a rooted view but
flooded an unrooted "browse the whole category" view: Fleet's ~49 focus
entities are hub-like enough that 1-hop pulled in 325+ of the system's
479 total entities. Neighbor expansion now only applies when a root is
set; the unscoped view goes back to same-category-only edges, which
measured at a clean 49 nodes for Fleet.
Verified against live production data (real bearer token, real DB) rather
than mocks: Fleet unrooted = 49 nodes matching the DB's compute+physical
count exactly; rooting on host:strong shows 33 nodes with both bright
same-category and dimmed cross-category neighbors, no isolated dots.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
chi.URLParam returns the raw, still-encoded path segment — unlike the
OpenAPI-generated routes, which decode via
runtime.BindStyledParameterWithOptions before the handler sees them. Slugs
like "document:containers/101-jellyfin" (encoded by the frontend's
encodeURIComponent) were arriving undecoded and matching no row. Found via
a standalone chi repro, not by patching the live deploy checkout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two fixes to the new category taxonomy:
- Knowledge Base couldn't show a document/investigation/runbook's own
markdown body — knowledge_entities.content was never exposed by any
endpoint (GetEntityKnowledge answers "what knowledge references this
entity", not "what is this entity's content"). Add GET
/api/v1/knowledge/content/{id} and render it with the existing
marked+DOMPurify pipeline in a new Content section.
- The graph hid any edge whose other endpoint wasn't in the active
category, so nodes with only cross-category neighbors rendered as
disconnected dots. Queried the real relationship table: ~70% of infra
edges cross Fleet/Network/Services/Storage lines (compute+network+
software+storage+physical used to be one "infrastructure" layer).
EntityGraph now keeps 1-hop neighbors visible but dimmed instead of
hiding them, so the edges — and what they connect to — stay visible.
- categories.ts: `cognition` domain conflated true knowledge (document/
investigation/runbook, 58 entities) with operational telemetry
(execution/check/task/signal/approval/pattern/skill/classification/
feedback, 300+ entities with their own Operations/Signals/Learning
pages). Mapping the whole domain to Knowledge pulled in 245 execution
entities fanning out from ~17 compute nodes via `targets` edges — the
single biggest source of graph clutter. Knowledge now maps by type
(document/investigation/runbook only); the rest of cognition is
excluded from Knowledge Base browsing entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the layer-based (Infrastructure/Governance/Cognition) browsing tabs
with a synthesized category taxonomy built from the ontology's finer-grained
`domain` field, since layer lumped unrelated entity types (an LXC and a DNS
record and a storage volume) into one bucket. Network and Fleet each span
two domains, so the table view now fans out per-domain fetches and merges,
while the graph view maps domain->category client-side. Also carries over
several detail-panel polish items (Tasks-not-raw-executions, slug URL
encoding, MultiSelectFilter) from earlier in this session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The SPA-from-binary split (0c0f35a) left `make deploy-ui` pointing at a
deploy path that was never actually wired up: scp to a "mac-mini" SSH
host that doesn't resolve from itself, a /var/www/oikos-ui/ that doesn't
exist, and `systemctl reload caddy` on a box with no Caddy installed at
all (not brew, not a container, nothing on 80/443).
Add a `web` service (compose/web/Dockerfile: node build -> caddy:2-alpine
static + SPA-fallback serving) to docker-compose.yml so the UI deploys
through the same push-to-main -> webhook -> docker compose build/up
pipeline the rest of the stack already uses, instead of a manual
scp/ssh step. Drop the broken `deploy-ui` Makefile target; `make ui`
stays as a local build sanity-check.
Update the reference Caddy config (compose/caddy/Caddyfile.oikos) to
reverse_proxy the new :8091 service instead of reading static files off
local disk, and fill in the <mac-mini-mesh-ip> placeholders with the
actual LAN IP (192.168.178.182 — the LXC and mac-mini subnets are
routed). This file is a reference only; the real caddy-conf repo change
is applied separately after review.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the separate Entities/Graph nav items with one Knowledge Base
page that browses all entities as either a table or a force-graph,
scoped by ontology layer (Infrastructure/Governance/Cognition), with a
resizable browse/detail split instead of a slide-over sheet.
- New KnowledgeBase.svelte: layer tabs, view toggle, resizable
browse/detail split (pattern from Chat.svelte's rail).
- EntityTable/EntityGraph extracted as presentational sub-components;
their search/filter/root/depth toolbars live in the shared page
toolbar (not the resizable pane) so they don't truncate when the
divider is dragged narrow, and both views start flush with the
detail pane for consistent height.
- EntityTable columns are sortable (slug/type/name/state/health).
- EntityDetailContent redesigned as a single-column list of
collapsible sections (DetailSection.svelte), collapsed by default
when empty; relation entries are clickable and select the entity in
the browse pane + detail pane (and drill in-place in EntitySheet
wherever it's used elsewhere in the app).
- api.ts: add layer filter to fetchEntities, add fetchEntityTypes for
client-side graph layer scoping (the graph endpoint has no layer
param).
Old hash routes (#/entities, #/graph) redirect to #/kb.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Documentation and repo-hygiene pass following the client/server split:
Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.
Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).
Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).
Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.
Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.
Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
GetClientContext handler) since the mechanism's introduction on
2026-06-02 — never matched any real filename, so no client has ever
picked up an auto-setup script via git-pull or the context-poller sync.
Fixed all three; the Go server-side fix is the one that actually matters
since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
(parsed, never consumed) left over from an earlier clone-based model.
Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 0 of plans/2026-07-12-wails-desktop-app.md. The control-room SPA
is no longer embedded (web/embed.go deleted); it's a standalone static
build served separately (make ui / make deploy-ui). The api process
adds CORS and drops the dev-open auth bypass — every route now needs a
real bearer token, including SSE (?token= query param, EventSource
can't set headers) and api's own /agent proxy to nomos (previously
unauthenticated by omission).
nomos was an unauthenticated client of api's /mcp and approval-decision
endpoints; closing dev-open would have broken it, so it now sends
Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN on every call back to api.
SPA gets a runtime config module (config.ts) and a Config.svelte
first-launch/reconfigure page, reachable afterwards via a "Connection"
entry in the sidebar footer. Every fetch() in api.ts routes through
fetchWithAuth so the same build works same-origin (browser prod, Vite
dev proxy) or cross-origin (future Wails webview, remote access).
Six gaps found against the plan and the live Caddy topology while
implementing — documented in the plan's "Plan review" section, most
notably: api's own /agent mount was never behind combinedAuth (fixed),
and production's Authentik forward-auth needs a bearer-token bypass for
API routes that this repo's Caddyfile.oikos reference copy now has, but
the real dtoro/caddy-conf deploy does not yet.
Verified live: cross-origin static SPA + API, CORS, bearer auth, SSE
query-token auth, and localStorage persistence all confirmed working
in-browser. Full Go test suite and npm run build pass with no
regressions against the pre-change baseline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gitea (LXC 104, 192.168.8.x) can't reach mac-mini (192.168.178.182) due to
ALLOWED_HOST_LIST. As a fallback, a 2-minute launchd poller checks if
origin/main has new commits and runs deploy.sh if so.
- cmd/webhook/main.go: HMAC-validated webhook receiver on :9797
- launchd plist: keeps webhook running, PATH includes docker
- Makefile: 'make webhook' target
- Registered as Gitea webhook id 15 on dtoro/oikos
Fixes: auto-deploy was not wired on mac-mini after the consolidation
Overview replaces Tasks as the default route: a centered new-task entry
with live fleet metrics, a scrollable/filterable task table, and an
ambient canvas rendering of the real entity graph (autonomous camera
drift + mouse parallax) behind it. Tasks sidebar entry is removed;
its status-bucketing logic moves to lib/tasks.ts for reuse.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Search hits and entity-knowledge hits never selected an id column, so
every KnowledgeHit.Id defaulted to the zero UUID. The frontend's keyed
{#each results as hit (hit.id)} then had all-duplicate keys, which
silently broke Svelte 5's if-block branch swap for the results panel —
search would set searched=true (Clear button appeared) but the view
never switched away from "Recently learned". Select e.id in both
queries and key the each block on hit.slug (guaranteed unique) instead.
Events, Agent, and Audit were standalone read-only pages that never
cross-referenced the entity they related to. Fold them into EntityDetail
as entity-scoped cards (Agent activity, Audit trail) alongside the
existing Signals/Executions/Knowledge cards, and give the Signals card
real Ack/Mute/Resolve actions. Signals stays a standalone page since
it's the only one with cross-entity triage value (badge count, actions).
Also fixes the underlying reason those new cards would've stayed empty:
agent_activity rows were never tagged with entity_id at insert time
(cmd/nomos/store.go, internal/mcp/server.go), even though the column
and the API filter both support it. Added a best-effort resolver that
checks common tool-arg keys (target, entity_slug, slug, ...) against
the entities table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes 1-3 deployed and verified live: fresh trivial Q&A sessions now reach
done immediately, and a goal-bearing session that stalled was correctly
nudged by the idle sweep. Fix 4 (backfill) was replaced with deletion after
the operator's call — verified against the DB first that zero knowledge
notes were linked to or written by any of the 53 removed sessions, so
nothing was lost. Documents the pagination gap in listSessions (hardcoded
LIMIT 50, no total count) that hid 6 of those sessions from the original
audit.
Also fixes relative links in this plan and in the UI-review plan that broke
when both moved from plans/ to plans/done/ (one directory level deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements fixes 1-3 of plans/2026-07-11-task-completion-safety-net.md.
Confirmed live that 50/50 production sessions never reached a terminal
status because the model almost never calls complete_task, even for
trivial single-tool Q&A turns SOUL.md explicitly calls out as needing it.
- Inline safety net (agent.go): a session that never called set_goal never
framed itself as a structured task, so its first plain-text turn-end IS
the task ending — auto-complete it there instead of leaving status stuck
at its creation default forever.
- Idle sweep (continue.go, new completion_nudges column): goal-bearing
sessions that stall get one nudge, then auto-close with outcome=partial
if the nudge goes unanswered, mirroring the pattern resumeSession already
uses for a different stuck-session failure mode.
Fix 4 (backfill of the 50 already-stuck live sessions) is deliberately
separate — deferred until this is deployed and verified live, per the
plan's implementation order.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Traced during UI-review verification: 50/50 live sessions are stuck
active/planning, never done/failed. Root cause confirmed against the
running DB — set_goal called once, propose_plan and complete_task
called zero times across all 50 sessions. The model consistently
skips the terminal complete_task call despite SOUL.md explicitly
instructing it to, especially for trivial single-tool Q&A turns.
Plan proposes an inline safety net for the common case plus an idle
sweep for structured goal/plan sessions that stall.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes the reviewed gaps: keyboard-inaccessible delete controls (SessionRail,
Entities row), case-sensitive entity filter, two competing entity-detail
navigation patterns (standardize on EntitySheet), non-clickable Overview KPI
cards, a bare button bypassing the shared Button component, inconsistent
blur-only vs live filtering, and an unenforced sanitization assumption on
search snippet HTML (now using the already-present dompurify dependency).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every finding from the review is now implemented and verified live:
A1 (3919ec3), B1+B2 (c5ffaec), A3 (926969a), D1-D3 (76f7630), A2 (c390164),
B3 (6d4f6de), F1 (11c18e8). C1 (nomos gateway has no authentication) remains
explicitly deferred per operator instruction. Kept in plans/ (not moved to
done/) since C1 is still open, matching how other partially-complete plans
in this index are tracked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix F1 of plans/2026-07-11-nomos-agent-code-review.md, the last item.
buildTools called listToolsFull (a tools/list MCP round-trip) at the start of
EVERY chat turn, including every auto-continuation resume — the tool list is
static for the lifetime of one MCP connection, changing only when the api
process re-registers tools (a restart, which this client already detects and
reacts to via reconnectLocked). Re-fetching it every single turn was
avoidable network+parsing work on the hot path.
mcpClient now caches the parsed tool list after its first fetch, guarded by
its own mutex (kept separate from the request-serializing mu so a cache
check never contends with an in-flight doRequest call). reconnectLocked
clears the cache — an api restart may have changed what's registered, so a
stale cache would be wrong, not just slow. fleetSnapshot's get_health_summary
call is deliberately left uncached — it's meant to be "as of now."
Since each session gets its own client (the per-session pool from the
concurrency work), this caches per-task-conversation rather than globally: a
task's FIRST turn still pays the round-trip, every turn after reuses the
cached list — which is exactly the case that mattered (long-running,
heavily-autonomous tasks with many auto-continuation resumes).
Verified live via the api's request log: a brand-new session's first turn
made 3 MCP calls (initialize, tools/list, get_health_summary); a second turn
on the SAME session made exactly 1 (only get_health_summary) — tools/list
correctly skipped.
This completes the implementation order in
plans/2026-07-11-nomos-agent-code-review.md — every A/B/D/E/F finding from
the review (excluding C1, explicitly deferred per operator instruction) is
now fixed, tested, and verified live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix B3 of plans/2026-07-11-nomos-agent-code-review.md. resumeSession's retry
loop (used by both auto-continuation and panel-answered questions) already
retried once on a transient LLM failure, but if BOTH attempts came back
empty/erroring, the code just logged and returned — the task was left at
whatever status it already had (typically 'executing' or 'awaiting_input')
with no outcome, no operator-visible signal beyond an inert error line
buried in the transcript, and no way to tell a genuinely stuck task apart
from one quietly still working.
On permanent failure, now calls store.completeTask(outcome='failure', a
summary built from the error) so the task board reflects reality instead of
showing a task that looks perpetually in-progress. Uses context.Background()
for that write, matching resumeSession's own persistence pattern, since the
context that led to the failure may itself be in a bad state. This doesn't
prevent the operator from continuing to work the task via a fresh chat
message afterward — it only replaces silent hanging with a real status.
A full live induction of a permanent LLM outage would require breaking the
model/API-key config for the whole nomos container — too invasive for this
fix's priority. Verified instead that the new branch stays correctly dormant
on the happy path: ran a real ask_operator → panel-answer → resume cycle
end-to-end and confirmed the task landed at status='executing' with no
outcome set, proving the failure-handling code doesn't false-positive on a
normal successful resume.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix A2 of plans/2026-07-11-nomos-agent-code-review.md. chatWith replayed a
session's ENTIRE message history into the LLM's context on EVERY turn, no
windowing, no token budget — confirmed against a documented production case
(a single turn with 70 tool calls, messages up to 106KB). Every subsequent
turn of a long-running or heavily-autonomous task re-sent that ever-growing
history in full — a real cost/latency/eventual-context-limit risk for
exactly the tasks this system runs longest (many auto-continuation cycles).
Design call (flagged in the review as needing one before implementation):
a fixed-size window for LLM replay specifically, not the UI's own transcript
view. Simplest option that still keeps roughly the current task's working
context; a token-aware trim or LLM-summarize-on-drop are documented as
stretch options if 30 proves insufficient in practice.
- store.go: new getRecentMessages(ctx, sessionID, limit) — last `limit`
messages in chronological order, plus whether older ones were omitted.
getMessages (used by the UI's GET /sessions/{id}) is untouched and stays
unbounded — the operator should still see a task's full history regardless
of length; only what gets sent to the model is bounded.
- agent.go: chatWith uses getRecentMessages(sessionID, historyWindowSize=30)
instead of the unbounded getMessages. When truncated, injects a system
note telling the model explicitly that older turns exist but aren't shown,
so it checks upsert_knowledge/search_knowledge rather than assuming
something wasn't done just because it isn't visible.
New cmd/nomos/store_test.go: real Postgres integration tests (mirroring
internal/db/integration_test.go's throwaway-database pattern, guarded by
OIKOS_TEST_DATABASE_URL). TestGetRecentMessages_Truncation is the direct
proof for this fix (35 messages → 30 returned, correctly ordered,
truncated=true; 5 messages → all 5, truncated=false) — both cases run
against a fully-migrated database, not mocked. Also added
TestProposePlan_AppendVsReplace, closing part of the review's test-coverage
finding (E) by permanently regression-testing the earlier append-vs-replace
plan fix (commit 5384499), which had only been verified manually until now.
Verified live: inflated a real session to 42 persisted messages via direct
SQL, then continued it with a real chat call — the turn proceeded normally
(multiple real tool-call iterations, no crash, no context-length error);
nomos stayed healthy throughout. A3's incremental persistence separately
confirmed to have caught the 7 real tool calls made before the client
connection was cut, cleanly closing out both fixes' interaction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes D1-D3 of plans/2026-07-11-nomos-agent-code-review.md:
- D1: deleted isTaskTool — defined, never called (dispatch already checks
handleTaskTool's own `handled` return value).
- D2: recordTouched issued one SELECT per entity slug found in a tool call's
args; batched into one `WHERE slug = ANY($1)` query. Verified live: a turn
naming three separate entities recorded involves edges for all three via
the single batched lookup.
- D3: complete_task's outcome had a declared enum (success|failure|partial)
in its tool schema but nothing validated it — an out-of-enum value (model
typo or a weaker model not respecting the schema) silently persisted as-is,
with only "failure" special-cased (anything else became status='done'
regardless of what the value actually said). Now validated in
handleTaskTool: empty defaults to "success" (unchanged), a recognized value
passes through, anything else defaults to "partial" (safer than silently
treating an unrecognized value as success) with a warning logged. Verified
live: instructed the agent to call complete_task with outcome="unclear" —
persisted as outcome='partial', not the literal invalid string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix A3 of plans/2026-07-11-nomos-agent-code-review.md. handleChat only ever
saved the assistant message ONCE, after a.chat(...) returned, using
ctx := r.Context() for that write — the same context that cancels the instant
the client disconnects (Stop button, tab close, network blip). A disconnect
mid-turn meant the final save ran with an already-cancelled context and its
error was never checked: the entire turn's tool-call history was silently
lost from the persisted transcript, even though real work (executions
launched, knowledge written) had already happened server-side.
Brought handleChat in line with resumeSession's existing pattern
(continue.go): insert a placeholder assistant row immediately, update the
SAME row after every tool call. The key fix is WHICH context the writes use —
a new pctx := context.Background() for every DB write in this handler
(session creation/touch, the user message, question auto-close, the
placeholder + incremental updates, the title update), while ctx/r.Context()
still gates the agent's own work (a.chat) and the SSE writes exactly as
before — a disconnect still correctly stops the agent from doing further
work, it just no longer also erases what it already did.
Verified live: sent a message requiring 6 tool calls (get_entity/
get_relations/get_blast_radius on two targets) and force-killed the client
connection mid-stream with curl -m 12 (confirmed via exit code 28). Before
this fix the persisted transcript would show 0 tool-call entries; after,
all 12 raw tool_use/tool_result entries (6 calls × 2) were present and
correctly attributed by tool name — proving both that progress survives an
abort and that the incremental writes aren't corrupting the data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together,
since the right granularity for B1 in the auto-continuation worker turned
out to require B2's restructuring anyway (see below).
B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned
nothing before this — every explicitly-spawned goroutine (continuation
worker, resumed chat turns, async execution dispatch, the SSE listener, two
duplicate sshExec implementations' output-collector goroutines) crashed the
whole process on an unhandled panic, not just that one goroutine. More
consequential post-concurrency: more simultaneous unattended background work
means more surface area for one bad input to end every running task.
New internal/safego package: Go(label, fn) launches fn in a goroutine with a
recover-and-log wrapper. Applied at every bare `go` spawn site across the
three packages. Two sites needed bespoke handling instead of the generic
helper because their callers block on a channel and a silent recover would
just make them hang until timeout: sshExec's output-collector goroutine (two
near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go)
and httpapi's ListenAndServe goroutine — both now recover AND send a
synthetic error result so the waiting select unblocks immediately instead of
waiting out the full timeout.
httpapi's sseListener got extra treatment: its per-notification handling was
extracted into handleNotification with its own recover, so a panic decoding
ONE malformed pg_notify payload can't kill the listener goroutine for every
connected SSE client — the outer goroutine spawn only needs to guard the
connection setup/reconnect code around it.
B2: cmd/nomos/continue.go's processContinuations used to run every pending
continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the
ticker — meaning (a) task B's continuation waited for task A's full (up to
10-minute) resumed turn to finish first, undercutting this session's earlier
concurrency work on exactly the path autonomous tasks depend on most, and
(b) an unrecovered panic anywhere in that call chain didn't just crash the
process (B1) — even WITH B1's recovery wrapped only at the top-level worker
spawn, the panic would still unwind the ENTIRE ticker-loop goroutine,
silently ending auto-continuation for every task until nomos restarted.
Fixed by spawning each pending item via safego.Go individually: real
parallelism, and a bad item can now only ever take down its own goroutine.
Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete
proof — a deliberate panic inside Go() that would otherwise crash the whole
test binary; reaching the assertion after it IS the evidence recovery works.
Verified live against the rebuilt containers: full chat turn round-tripped
correctly (hostname lookup, 2 iterations, normal completion) — no regression
from threading safego.Go through the tool-dispatch/continuation paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix A1 of plans/2026-07-11-nomos-agent-code-review.md. isAssent and
isTypedConfirmation used a space-padded word-boundary check for negation
words but a bare strings.Contains for assent/confirm words — confirmed live
via test probes: isAssent("...maybe yesterday's logs...") returned true
("yes" matched inside "yesterday"), and isTypedConfirmation("I haven't
confirmed anything yet") returned true ("confirm" matched inside "confirmed",
and "haven't" wasn't in negationWords — only "don't"/"do not" were).
isTypedConfirmation is the sole gate for DESTRUCTIVE actions, so the second
case meant a message merely stating something hadn't been confirmed could
read as an explicit confirmation.
- Replaced the ad-hoc space-padding/prefix-check negation logic with proper
tokenization (wordTokenRe) + containsPhrase, matching WHOLE tokens/phrases
only — never a mid-word substring. Handles curly apostrophes too (a
pre-existing gap: the old straight-quote-only check would have missed
"don't" typed with a smart quote).
- Added contracted negatives (haven't, hasn't, isn't, wasn't, aren't, can't,
cannot, won't, wouldn't, shouldn't, didn't, doesn't) to negationWords.
Deliberately did NOT add a bare "not" — too broad, would false-negative
ordinary assent like "go ahead, this is not risky".
- Added regression tests for both confirmed cases plus a couple of adjacent
ones (eyesight/isn't, can't confirm) so a future change can't silently
reintroduce either bug.
All existing assent/confirmation tests pass unchanged — this is a pure
robustness fix, not a behavior change for any previously-correct case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full read-through of cmd/nomos/ (agent.go, store.go, main.go, continue.go,
assent.go, tasks.go). Findings, ranked:
- A1 (confirmed via runnable probe): isAssent/isTypedConfirmation use
unpadded substring matching for assent/confirm words while negation uses
word-boundary checks — "yes" matches inside "yesterday", "confirm" matches
inside "confirmed" with no negation word covering contracted negatives
("haven't"). isTypedConfirmation gates DESTRUCTIVE actions specifically.
- A2: chatWith replays a session's ENTIRE message history every turn, no
windowing/token budget — confirmed unbounded against a documented
production case (70 tool calls, 106KB messages).
- A3: a live turn's tool-call history is lost entirely if the client
disconnects mid-stream (single end-of-turn save using the same
connection-tied, possibly-cancelled context) — resumeSession already has
the fix pattern (incremental placeholder+update), handleChat doesn't use it.
- B1: zero recover() anywhere in cmd/nomos/internal/mcp/internal/httpapi —
every explicitly-spawned goroutine (continuation worker, resumeSession,
executeApprovedViaAPI, sse listeners) crashes the whole process on panic.
- B2: auto-continuation processes its batch sequentially, one full LLM turn
at a time, undercutting this session's own concurrency work on exactly the
path autonomous tasks depend on most.
- B3: no terminal state for a permanently-failed auto-continuation.
- C1: nomos's own gateway (port 8092, directly published + mesh-reachable)
has ZERO authentication on any endpoint — chat, session read/delete,
chat-assent approval of gated executions, all open to anyone on the LAN.
- D1-D3: dead code (isTaskTool unused), N+1 query in recordTouched, no
validation on complete_task's outcome enum.
- E: zero automated tests for agent.go/store.go/main.go/tasks.go — including
today's new safety-critical logic (session-scoped windows, mcpClientPool,
proposePlan's append-vs-replace), verified only by live manual testing.
- F1: tool list + fleet snapshot re-fetched every turn (minor).
Prioritized implementation order in the doc: A1 → C1 → B1 → B2 → A3 → D1-3 →
A2 → B3/F1, tests landing alongside each fix rather than as a deferred pass.
Also archives the now-fully-shipped concurrent-task-execution plan to done/
(all 3 required fixes deployed this session; fix 4 explicitly deferred per
its own recommendation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 3 of plans/2026-07-11-concurrent-task-execution.md, the throughput one.
nomos held exactly one *mcpClient for the whole process, shared by every
/chat goroutine. Its mutex was held for the full duration of each tool
round-trip, and `run` executes its SSH command SYNCHRONOUSLY inside that
round-trip (capped at up to 10 minutes) — so while Task A was mid-`run`,
every other task's tool calls, even a trivial get_entity, queued behind that
single lock. Tasks could think (LLM calls) in parallel but never act in
parallel.
The MCP server has no per-connection state to protect (newServer returns one
shared *mcp.Server instance whose handlers close only over the DB
connection pool, already safe for concurrent use) — the mutex existed purely
because the client reused one stateful transport session. So the fix doesn't
touch the server at all:
- New mcpClientPool (cmd/nomos/main.go): one *mcpClient per session id,
created lazily (a real MCP initialize handshake) on first use and cached;
session-less traffic (the ephemeral no-DB-store path, the structured
/query endpoint) gets its own fixed, reused key instead of a fresh
connection per request. Idle clients (20 min past last use — long enough
to outlive a single slow `run`) are evicted on a 5-minute sweep ticker.
- agent.go: `client *mcpClient` → `clients *mcpClientPool`; every call site
(buildTools, fleetSnapshot, the tool-dispatch loop) now resolves its own
session's client via clients.get(sessionID) instead of reaching for one
shared field. A task's own tool calls stay sequential (already true — the
agent loop calls tools one at a time within a turn) but no longer block
anyone else's.
- main.go: handleQuery takes the pool instead of a client (keyed "query", a
fixed non-session slot); shutdown calls pool.closeAll().
Verified live against the deployed stack: fired a slow-but-ungated command
(`ping -c 15 127.0.0.1`, read-only per policy's allowlist, no approval
needed) as Task A, then — 2s into A's run — a trivial hostname lookup as
Task B, both through the real /chat endpoint. Task A's ping genuinely ran
~14.3s (confirmed via its own execution record and the agent's reported
output). Task B returned in 6s total, well before A finished — proving it
was never queued behind A's connection. Before this fix, B would have been
forced to wait out A's entire ~14.3s hold on the single shared client.
This completes plans/2026-07-11-concurrent-task-execution.md's required
scope — only the explicitly optional/deferred Fix 4 (a concurrency/cost cap,
pending real usage data) remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the known gap flagged in the previous commit (9131559). A single
module-level `activeController` meant cancelStream()/newChat() always
aborted whichever stream was MOST RECENTLY STARTED, regardless of what the
operator was currently viewing: start Task A, switch to an already-loaded
Task B, click "New task" — the click's cancelStream() would silently abort
Task A's still-running turn, even though the operator was never looking at
it and never asked to cancel it.
- Replaced the single controller with activeControllers (Map<sessionID,
AbortController>) plus pendingController for the brief pre-'session'-event
window of a brand-new task. Registered immediately in sendMessage (keyed by
the continuing session id right away, or held pending until the 'session'
event assigns a new one) and cleaned up on completion.
- cancelStream() now looks up by $currentSession (falling back to
pendingController when no session is assigned yet) — it can only ever
touch the stream belonging to the view being left, never an unrelated
background task's.
- newChat() unchanged in behavior (still calls cancelStream()), now correctly
scoped through the above.
Verified live, reproducing the exact bug: started Task A (slow, 5 tool
calls), switched to an existing Task B, clicked "New task" while viewing
B — Task A was NOT aborted, ran to completion server-side with a full,
correct final summary (previously this exact sequence would have killed it).
Confirmed the positive path is unaffected: started a task, clicked Stop
while actively viewing it — input re-enabled, stream genuinely aborted
("BodyStreamBuffer was aborted"), turn stopped mid-flight as expected.
This closes out Fix 2's scope from
plans/2026-07-11-concurrent-task-execution.md; only Fix 3 (per-session MCP
client pool, throughput) and the optional Fix 4 (concurrency cap) remain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 2 of plans/2026-07-11-concurrent-task-execution.md. sendMessage's SSE
callback mutated the global messages/currentSession stores unconditionally,
assuming only one task's turn is ever in flight. It isn't — the backend runs
every /chat request as its own goroutine with no serialization. Switching to
a different task while a previous one was still streaming let that
background stream's later events (tool_use, text_delta, ..., and worst of
all 'done''s currentSession.set) get applied to whatever the operator is now
looking at: corrupting another task's transcript, or yanking the view back
to the one they left.
- Captures the session a stream belongs to (openedFor at call time, updated
to the real id once the 'session' event assigns one) and checks
$currentSession still matches before every messages/error/streaming
mutation. The task keeps running server-side regardless — dropped events
just mean the live view isn't watching it; navigating back re-hydrates via
REST, same as already happens for auto-continuation.
- The 'session' event itself only claims currentSession if the operator
hasn't already navigated elsewhere since the call started (comparing
against openedFor, which is null for a brand-new task).
- loadSessionMessages/newChat now reset `streaming` to false unconditionally
on navigation — needed so the new guard can't leave a DIFFERENT task's view
stuck showing streaming=true (which would also silently stop startPolling's
loop from ever applying updates, since it bails while $streaming is true).
Known residual gap, not fixed here (matches the plan's "contained fix, not a
rearchitecture" scope): activeController is still a single global slot, so
starting a new task while another is mid-stream, then clicking "New task"
again, aborts whichever stream that slot last pointed at rather than only
the one being left. A genuine multi-session controller/store is the
plan's deferred "stretch" fix, not required for correctness here.
Verified live: started Task A with a deliberately slow 4-tool-call turn,
switched to an existing Task B mid-stream — Task B's transcript stayed
correct with zero A-originated entries and the input was NOT stuck disabled.
Task A kept running and completed normally server-side (status=done, full
6-tool transcript, 5-entity graph); navigating back loaded its complete,
uncorrupted result via REST.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 1 of plans/2026-07-11-concurrent-task-execution.md — the safety-critical
one. The assent window (and destructive window) were keyed purely by agent
id ("assent_window.agent:<uuid>"). With one agent:nomos entity serving every
concurrent task, this meant approving Task A's plan opened a window that ANY
concurrently-running task's config-mutation/destructive actions could also
ride, auto-executing without their own approval.
- store.go / agent.go: assentWindowActive/openAssentWindow and
destructiveWindowActive/openDestructiveWindow/destructiveWindowKey all gain
a sessionID parameter; keys become
"assent_window.agent:<id>.session:<sessionID>" and
"destructive_window.agent:<id>.target:<slug>.session:<sessionID>". Missing
session id fails closed (no window) rather than falling back to the old
agent-wide key.
- continue.go: the auto-continuation worker's window check moved from once-
per-batch to once-per-pending-item, scoped to that item's own session —
it was previously checking ONE agent-wide window for a batch that can span
multiple tasks.
- agent.go tool-dispatch: injects `_session_id` into a COPY of the wire args
sent to the MCP server (never into the args used for the emitted/logged/
persisted tool call, and never part of any tool's declared InputSchema —
invisible to the model) so the gating checks on the OTHER side of the
process boundary know which task is asking.
- internal/mcp/server.go: assentWindowActive/destructiveWindowActive/
classifyAndGate gain the same sessionID parameter, read from
args["_session_id"] at the three call sites (request_execution's
apt_upgrade/pct_create branches, and the shared classifyAndGate used by
restart/pct_exec/systemctl/run).
Verified against the live stack with the exact scenario from the plan: opened
an assent window for session A only, then called `run` with an identical
config-mutation command for session A (window open) and session B (same
agent, no window). A auto-ran (execution status completed); B correctly
queued for approval (pending_approval) instead of bleeding through — proven
at both the MCP response text and the executions table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New plan grounds three concurrency issues found by tracing the actual code
(not assumed): the assent/destructive windows are keyed by agent id only
(no session dimension), so an approved plan in one task can auto-run
unapproved actions in a concurrently-running task; nomos shares one
mutex-guarded MCP client across all sessions, so a single slow `run` call
serializes every other task's tool calls behind it; and chat.ts's SSE
callback has no session guard, so switching tasks mid-stream lets the
backgrounded task's events corrupt whatever's now displayed. Proposes
session-scoping the windows (critical/first), a frontend stream guard
(contained/second), a per-session MCP client pool (throughput/third), and
an optional concurrency cap (deferred pending real usage data).
Also archives the goal-oriented-chat-control-panel plan to done/ — all 7
phases shipped and are live in production (SHA e30813a) — fixing its
internal relative links for the new depth and pointing forward to the new
concurrency plan as follow-up hardening.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the gap that made the knowledge loop optional/implicit: every
non-trivial task now has an EXPLICIT first plan step (research) and last
plan step (write back), not just background behavior the model might skip.
New MCP tools (the agent had no way to do these before — only REST endpoints
existed, unexposed to it):
- update_entity_attributes(slug, attributes): shallow-merge new/changed facts
into an entity (an IP, a version, a discovered port) so a future task
doesn't have to rediscover them from scratch. No approval required — this
updates the knowledge graph, not live infra.
- create_relationship(source, target, type): record a discovered edge
(depends-on, hosts, provides, ...). Idempotent, FK-validated against the
ontology's relationship_types, no approval required.
SOUL.md: restructured the task loop so step 1 is explicitly "gather
knowledge, not just status" (get_entity_knowledge, search_knowledge,
get_relations, get_blast_radius, http_get) and the last step before
complete_task is explicitly "write back" (update_entity_attributes,
create_relationship, upsert_knowledge) — both called out as real plan
entries the operator should see in propose_plan, not silent side-work. This
is what prevents the graph drifting from reality and is the concrete
mechanism behind "tasks compound."
propose_plan's tool description reinforces the same first-step/last-step
convention at the call site.
Verified against the live stack: both tools registered and callable via MCP;
update_entity_attributes merged an attribute correctly; create_relationship
rejected an invalid type (FK violation, clear error) and succeeded with a
valid type+direction, confirmed idempotent (2 calls, 1 row).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause: proposePlan unconditionally deleted and replaced the whole
session_plan_steps list on every call. The model isn't strictly held to
"call propose_plan once with the full list" — nothing stopped it (and
production evidence + live testing showed it happening) from calling
propose_plan once per step as it worked. Each such call wiped every
already-completed step, so the operator only ever saw the model's latest
single step ("1/1") instead of the real, growing plan.
Fix, two layers:
- store.go: proposePlan now only does a destructive replace when no step
has left 'pending' yet (a genuine pre-execution revision). Once any step
has started, a new call APPENDS after the current max seq instead of
wiping — so the panel accumulates the full history regardless of how the
model chooses to call the tool. plan.proposed now carries `appended` so
the frontend knows whether to replace or append.
- workspace.ts: plan.proposed handler respects `appended` (update vs set).
- tasks.go / SOUL.md: strengthened the propose_plan description and task-
loop guidance to call it ONCE with the complete step list end-to-end,
using update_plan_step (not re-calling propose_plan) to advance — fixing
the root behavioral cause, with the store-side append as a safety net
that holds even if the model still calls it incrementally.
Verified: forced the exact incremental-call pattern (propose_plan with 1
step, mark it running, propose_plan again with 1 more step) — the second
call appended at seq 2 instead of erasing seq 1, and its plan.proposed
event carried appended=true.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the chat right rail's ad-hoc Digest+Graph stack with a single
TaskContextPanel that renders the task's live working state, driven by the
always-on events stream (not the per-turn chat SSE) so it keeps updating
during server-side auto-continuation/resume:
- GoalHeader: goal + status pill (planning/executing/awaiting_input/done/
failed), sourced from the sessions list.
- PlanProgress: ordered steps with live status icons + progress bar, hydrated
via new GET /sessions/{id}/plan; clicking a step with a target opens its
EntitySheet (no fake "jump to transcript" — bits-ui Collapsible content
isn't force-mounted, so a DOM-scroll jump would silently no-op for
collapsed tool groups).
- OperatorQuestion: the pinned structured question card (prompt/why/entity
chips/option buttons/free-text), hydrated via new GET /sessions/{id}/
questions; answering POSTs to the existing answer endpoint.
- SessionGraph upgraded to a live entity panel: entity.touched pulses the
node (animated ring) and shows "Now touching <slug>"; health.changed shows
a transient diff badge for touched entities.
- SessionDigest gains a success/failure/partial outcome banner and now also
refetches when the task's status changes, not just on session switch.
Two bugs found and fixed while wiring this up:
- workspace.ts's status-refresh trigger only covered goal.set/task.status;
question.raised/answered didn't refresh the sessions list, so GoalHeader's
pill went stale after answering via the panel (resumeSession runs entirely
server-side — no client 'done' event to piggyback a refresh on). Now every
status-affecting event triggers the (debounced) refetch.
- Forgot to rebuild the nomos container after adding the /plan and
/questions endpoints, so they silently fell through to the old default GET
handler — caught via a live curl diff against the running container,
not a code read.
Verified end-to-end against the live stack: goal/plan/question all update
without a reload as the agent works; answering a question via the panel
resumes the agent and the header pill correctly flips to Executing;
entity.touched pulses the live graph.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframes the chat surface as tasks:
- New Tasks.svelte: a card grid of tasks, each showing status (Running /
Needs input / Done / Failed), the goal as title, the outcome summary, and
relative time; filterable by status with live counts; delete on hover;
"New task" and per-card click open the conversation.
- Board updates LIVE off the events stream (goal.set / task.status /
question.*) via an explicit liveEvents.subscribe with a debounced refetch —
scanning all events newer than the last seen, since entity.touched bursts
bury task events below index 0.
- App shell: primary nav "Chat" → "Tasks" (board is now the home route),
"New chat" → "New task", conversation header gets a Tasks / Conversation
breadcrumb. Removed the superseded Sessions page.
- api.ts Session type carries the task fields (goal/status/outcome/summary).
Also fixes a pre-existing SSE bug that blocked ALL live updates app-wide:
writeSSE emitted `event: <type>`, which EventSource only delivers to
addEventListener(type) handlers — but stores/events.ts (and every page reading
liveEvents) consumes via onmessage, which never fires for named events. So the
live stream delivered nothing to the UI. Dropped the event-name line; the type
is already in the JSON payload, and new event types now need zero client
changes. SSE test still green (it parses data: lines).
Verified in the browser against the live stack: the board renders 50 tasks
with correct status buckets; a goal-driven task appears and flips to a Done
card with its summary in real time without a reload; Events page confirms the
stream now delivers to onmessage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The last backend piece: when the agent hits a decision only the operator can
make, it surfaces a structured question instead of guessing or stalling.
- ask_operator(prompt, why?, options?, context_entities?): nomos-local tool
that records a session_questions row, moves the task to awaiting_input, emits
question.raised, and ENDS the turn (the agent loop returns after it, so the
agent can't barrel past its own question). The prompt becomes the assistant's
visible message so the question also shows inline in the transcript.
- Two resume paths, both close the question + emit question.answered + return
the task to executing:
- Panel: POST /sessions/{id}/questions/{qid}/answer → resumes the agent in the
background with the answer injected (reusing the continuation machinery,
refactored continueSession → resumeSession). Returns 202; the reply lands via
message polling.
- Chat reply: the next chat message on a task with an open question IS the
answer — auto-closed in handleChat; the turn itself is the resume.
Verified end-to-end: forcing a decision paused the task at awaiting_input with
the structured question (prompt/why/options/entities); a panel answer resumed
the agent (it acknowledged host:strong and continued); a plain chat reply
auto-closed a second question. Cleanup + tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gives a task a legible, live-advancing plan via three more nomos-local tools:
- set_goal(goal): records the task goal, status → planning, emits goal.set.
- propose_plan(steps[]): persists ordered steps (clean replace for v1 — a
revision starts a new list), status → executing, emits plan.proposed with
the persisted steps (id+seq) so the panel can address them.
- update_plan_step(seq, status, execution_id?): advances a step, stamping
started_at/finished_at, emits plan.step.started/finished. Anchors the event
to the step's target entity when it has one.
Belt-and-suspenders: when an execution linked to a step reaches a terminal
state, the api auto-closes the step (closePlanStepForExecution in
emitExecutionEvent) and emits plan.step.finished — so the board stays honest
even if the agent forgets to close a step it started.
Verified end-to-end: a goal-driven task fired goal.set → plan.proposed →
2× step.started/finished → task.status on the SSE stream; both steps persisted
done with start/finish timestamps; status progressed planning→executing→done.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the compounding knowledge loop the task model is built around:
- complete_task(outcome, summary): a nomos-LOCAL, session-scoped tool (the
shared MCP server has no session id). Introduces the local-tool mechanism —
buildTools appends task tools, the agent loop routes them to handleTaskTool
instead of the MCP client. Sets the task's terminal status/outcome/summary,
mirrors it onto the task entity, and emits task.status.
- Knowledge → task linkage: after a successful upsert_knowledge in a task,
nomos links the note to the task entity (documents) and emits
knowledge.recorded, so the task's outcome view shows what it learned. The
note's about-link to the involved entity (written by upsert_knowledge) is the
retrieval path future tasks use.
- SOUL: every chat is a task loop — retrieve prior knowledge FIRST
(get_entity_knowledge on the target), plan, execute, record learnings, then
complete_task. Scales down for trivial read-only tasks.
- deleteSession now cleans up the task entity, its relationships, and its
task-scoped events (was orphaning them); the knowledge doc itself and its
about-links survive, as knowledge should outlive the task.
Verified end-to-end: a task recorded a note and completed; task.status +
knowledge.recorded hit the SSE stream; status=done/outcome=success persisted;
the note linked to both lxc:caddy (retrieval) and the task; a future
get_entity_knowledge(lxc:caddy) surfaces it; delete cleaned edges+events (0/0/0)
while the knowledge survived.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
As the agent runs a task, record which entities each tool call references:
write an idempotent task —involves→ entity relationship and publish one
entity.touched event per entity (correlation_id = session, data {slug,tool}).
Extracted from tool ARGS only — never results — so a bulk fleet query can't
drag every entity into the task graph; bulk/no-slug tools stay silent.
Emitted from the nomos agent loop rather than the shared MCP wrapper, which
has no session id. The involves edges make a task's graph neighborhood its
involved-entity set (queryable via get_relations) — the substrate for the
knowledge loop; the events are the live pulse the context panel consumes in
phase 6.
Verified end-to-end on the local stack: a chat referencing lxc:caddy/lxc:gitea
produced entity.touched on the browser SSE stream with slug+tool+correlation,
and exactly one involves edge per entity despite repeated touches.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Migration 018 adds goal/status/outcome/summary/entity_id to agent_sessions
and creates session_plan_steps + session_questions. Registers a 'task'
entity type and an 'involves' (task→entity) relationship in the ontology
so each session anchors its knowledge and involved-entity edges on the
existing relationships graph.
nomos createSession now mints a task:<session-id> entity (type task) and
links it via agent_sessions.entity_id — best-effort so chat never blocks on
it. listSessions/GET /sessions surface the new task fields.
No behaviour change yet; this is the data foundation for the task board and
live context panel. Verified end-to-end against the local stack: migration
applied, ontology ingested (60 types/47 rels), a new session mints a linked
task entity and the API returns status/goal/entity_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframes the chat surface as a board of tasks: each task carries a goal,
a plan approved once, a lifecycle status, an outcome, and a knowledge
loop that links learnings to the involved entities (and the task entity
itself) via relationships so future tasks compound. Supersedes the
sidebar-only framing and the free-form chat portion of the control-room
web UI plan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audited all 10 active plan docs against the codebase (not just commit
titles). 5 were fully shipped and stale-tagged "Planned"/"In Progress" —
moved to done/ with verification notes. The other 4 got corrected
Planned→In Progress status plus concrete remaining-gap notes so the next
pass doesn't re-derive what's already done.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The plan's "learning view" (runbook success-rate trends, promoted
skills, capability timeline) assumes the patterns/skills/feedback
pipeline is populated. It isn't: all three tables are empty in
production and nothing in the codebase ever writes to feedback, so
building the UI against them today would ship a permanently-empty
page. Scoped instead around data that's real and growing —
executions — while still wiring up /patterns and /skills so the page
needs no rework once that pipeline exists.
New /api/v1/learning/timeline (per-verb first-success date + success
rate, parsed via the existing splitAction helper) and
/api/v1/learning/trend (30-day daily success/fail counts), both
read-only queries against executions. Patterns and skills sections
call the existing (untouched) ListPatterns/ListSkills endpoints and
render an explanatory empty state instead of nothing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live: a chat request to restart caddy (the reverse proxy for the
whole fleet) executed instantly over SSH with zero approval. Root
cause was in request_execution's legacy handler — restart, pct_exec,
and systemctl (outside enable/disable) executed immediately with a
hardcoded risk_class='reversible_low' that was never actually checked
against anything, bypassing the classifier entirely. Only the `run`
tool's commands were ever gated.
Extracted the run tool's classify -> execute-or-queue logic into a
shared classifyAndGate() and route restart/pct_exec/systemctl through
it too, so every mutating path — regardless of which tool the model
reaches for — gets the same read-only/config-mutation/destructive
classification and approval gate. systemctl restart is already covered
by an existing classifier test (config_mutation), so no new test
needed; the gap was that request_execution never called the
classifier at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pending-approval cards showed target and risk but not what else the
action would affect — the operator approved config_mutation/destructive
commands blind to downstream impact, even though the graph-walk
(blast_radius() SQL, GetBlastRadius endpoint) already existed and was
just never wired into the approval path.
Fetch it once per pending approval and render "Affects N downstream: …"
on both the normal and destructive approval cards, reusing the existing
fetchBlastRadius() API client function which was already written but
unused anywhere.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ops "Executions" tab showed raw target UUIDs, alphabetical (not
recency) order, and a stale status vocabulary from an earlier schema
iteration — never actually usable as a live "what's happening" view.
Replaced with a new recency-ordered /api/v1/activity/recent endpoint
and matching table (human-readable action summaries, risk/status
badges, duration, inline error preview).
Also added /api/v1/activity/session/{id} + a collapsible SessionDigest
panel in the chat rail, answering "what did this session actually do"
(executions by status, entities touched, knowledge written) — the
missing piece for proactive outcome reporting to be visible in the UI,
not just in the chat transcript.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified live immediately after deploying: the endpoint returned 200 with
correct-looking stats (total=56, agent_authored=2) but items=[] always,
regardless of limit/source. Root cause: pgx v5 can't scan a timestamptz
column directly into a Go string — Scan() errored on every single row, and
that error was silently swallowed by a bare `continue`, so every row was
dropped with no trace in the logs. Fixed by casting updated_at::text in the
SQL (matching how every other handler in this codebase already returns
timestamps) and logging scan failures instead of swallowing them, so this
class of bug can't hide silently again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First slice of the observability/learning UI (the "see the system come alive
and learn" ask). The Knowledge page was search-only — blank until you typed —
so the knowledge Nomos now writes via upsert_knowledge was invisible unless
you knew to search for it. Now the page LEADS with what the system knows and
is learning:
- internal/httpapi/knowledge.go: GET /api/v1/knowledge/recent — recency-ordered
knowledge + a stats header (total, agent-authored, learned-this-week,
by-kind). Custom route (not OpenAPI-generated), same auth as the rest.
- web Knowledge page rewrite: stat cards up top (Total / Written by Nomos /
Learned this week / runbooks-investigations), then a "Recently learned" feed
with agent-authored notes highlighted and badged "learned by Nomos", tags,
and relative timestamps. A toggle filters to Nomos-only. Search still works,
now as a mode you enter/clear rather than the whole page.
This turns "the system is getting smarter" from a claim into something you
watch fill up: every gotcha the agent records shows here within seconds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
From the last (successful) TypeType deploy session, two gaps the operator hit:
1. Knowledge write-back — the missing half of the loop.
The agent could read the knowledge base (search_knowledge/get_entity_knowledge)
but had no way to WRITE it, so everything it learned (the Dragonfly memlock
rlimit gotcha, the NAT-hairpin DNS issue, etc.) lived only in an ephemeral
chat message and was lost — the system could never actually "get better."
This is the `upsert_knowledge` MCP tool the 2026-07-08 gaps plan called for.
- internal/mcp/server.go: upsert_knowledge(title, content, about?, tags?,
kind?) writes a document/investigation/runbook entity + knowledge_entities
row (search column is generated), upserts by slug so re-titling updates in
place, and optionally links it to the entity it's about so
get_entity_knowledge surfaces it there.
- SOUL.md: capture non-obvious findings/deploys/gotchas as part of finishing
work, not only when asked "what did we learn".
2. "I had to ask for status multiple times."
The clearest cause: a long working turn (64 tool calls) that exhausted the
iteration cap ended with a bare "max iterations reached without final
answer" — a dead end that forced the operator to ask what happened.
- cmd/nomos/agent.go: on exhaustion, make one final no-tools LLM call
(finalSummary) asking for a status report — what was accomplished, current
state, what remains — so the turn always ends with a real outcome.
- maxIterations 25 -> 40 (the decomposed per-step pct_create flow legitimately
needs more steps).
- SOUL.md: always end a turn with a clear outcome; never end silently or on a
bare tool call — the operator can't see the tools working and reads silence
as "nothing happened".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: "I'd like to be able to see in the chat what the agent is actually
running, right now I just wait while nothing happens." Two compounding gaps:
1. The auto-continuation worker (cmd/nomos/continue.go) had zero live push —
its result only appeared on a manual page reload, so approving a plan and
watching the chat looked completely dead even while the agent was actively
working.
2. Even with polling, continueSession only persisted ONE message at the very
end of a continuation — a continuation that runs several tool calls before
concluding would still show total silence for however long that took.
Fixed both:
- web/src/lib/stores/chat.ts: polls the current session's messages every 3s
between turns (never while a live stream owns the message list) and merges
in anything new. Started after a live turn ends and when a session loads;
stopped on new-chat/session-switch.
- cmd/nomos/store.go: insertMessageReturningID/updateMessage — lets a message
be created as a placeholder and updated in place.
- cmd/nomos/continue.go: continueSession now inserts a placeholder the
instant it starts (renders as the existing "thinking" dots — immediate
feedback that something is happening) and updates that SAME row after
EVERY tool call, not just at the end. A poll within ~3s of any tool call
landing shows it — individual `run` commands appear as the agent issues
them, not just the final rolled-up summary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit fixed a context-cancellation bug in the auto-approve path
and appeared to fix things, but re-testing end-to-end after deploy showed the
execution STILL never completed — just via a different symptom
("no pending execution found for approval" in the logs). Dug further and
found the real, deeper bug underneath: this whole mechanism has never
actually worked.
autoApprove() directly flipped BOTH approvals.status and executions.status to
'approved' via raw SQL, then called executeApprovedViaAPI to POST to the
decision endpoint. But DecideApproval's own logic specifically looks for the
execution still at status='pending_approval' to find and dispatch the real
SSH work (executeApprovedAction) — autoApprove's premature flip meant that
lookup always found zero rows. DecideApproval's UpdateApprovalStatus call
also silently no-ops the same way (sqlc :exec doesn't surface "0 rows
affected" as an error). Every assent-window auto-approved pct_create/
apt_upgrade has been sitting at 'approved' forever with the real work never
triggered — indistinguishable from "still running" until you check.
Fix: remove autoApprove() entirely. Call executeApprovedViaAPI directly
against the untouched pending_approval row from createApproval — identical
to the manual Approve-button path, just without the human click. DecideApproval
is now the single place that transitions status and dispatches, for both the
manual and auto-approved paths, closing the class of bug where two code paths
raced to do the same state transition.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live testing the new atomic pct_create: an assent-window
auto-approved pct_create appeared to "run" (logged "auto-approved... running
now") but the execution stayed stuck at 'approved' forever. Root cause:
`go executeApprovedViaAPI(ctx, ...)` passed the MCP tool-call's own context —
which is cancelled the instant the triggering /chat request's HTTP response
completes, i.e. on every normal turn. The spawned goroutine's POST to the
approval-decision endpoint died with "context canceled" before it could even
start the real work, and nothing surfaced this to the operator or the agent —
the execution just sat at 'approved' with no error, indistinguishable from
"still running."
This is exactly the context-lifetime bug class httpapi's own approval
goroutine (executeApprovedAction) already avoided by using
context.Background() — it had just been missed in these two call sites
(apt_upgrade and pct_create auto-approve). Fixed both to use
context.Background(), matching the correct pattern already in place
elsewhere. Audited for other goroutines spawned with a request-scoped ctx —
none found; the sshExec internal goroutines are synchronous/waited-on via
select and correctly scoped to the call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the two remaining open points from the auto-continuation work.
1. Atomic pct_create (observability, the bigger of the two):
pct_create used to bundle create + apt install + post_install script into
one black-box multi-minute SSH call — the agent got back a single opaque
success/fail with no way to see (or fix) which step actually broke.
Removed the whole post-create provisioning block (and the now-dead
provisionScript/sanitizePkgs helpers + their tests). pct_create is now
create + start + register ONLY — fast, and its result is fed back to the
agent via auto-continuation almost immediately. The agent installs
packages and runs setup as its OWN sequence of `run` calls against the new
lxc:<hostname>, observing each command's real output and able to diagnose
and retry exactly the step that failed — the same recovery loop already
proven for the general case, now applied to installs too, instead of
requiring a separate black-box mechanism.
- services/post_install removed from the pct_create params struct and
from the MCP tool schema/SOUL.md docs.
- SOUL.md: explains the new flow, moves the Docker CLI gotcha and DNS
troubleshooting guidance to be steps the agent runs itself.
2. Scoped destructive window (targeted autonomy for recovery):
Verified live in the previous session that a destructive recovery (a
failed destroy needing stop-then-destroy on the same container) required
TWO separate typed confirmations for what was clearly one recovery
action. Added a narrow, TARGET-scoped 15-minute grant
(destructive_window.agent:<id>.target:<slug> in autonomy_settings,
shared key format across cmd/nomos and internal/mcp) that opens only
after an EXPLICIT typed confirmation (never loose assent) or an explicit
button-approval of a destructive step, and only ever covers further
destructive commands against that SAME target. A different target always
needs its own fresh confirmation — this narrows risk instead of loosening
it globally, unlike broadening the general assent window to cover
destructive actions would have.
- cmd/nomos/store.go: openDestructiveWindow/destructiveWindowActive/
executionTarget.
- cmd/nomos/agent.go: opens the window when a typed confirmation grants a
destructive chat-assent execution.
- internal/mcp/server.go: `run` tool checks the window before gating a
destructive command; auto-runs if active.
- internal/httpapi/phase3.go: DecideApproval opens the same window when a
destructive execution is approved via the button/API, for parity with
the chat-assent path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live: the new auto-continuation worker (previous commit) worked end
to end for the happy path (provision -> auto-verify -> report success, zero
operator ticks). But testing a failure-recovery case (a destroy that failed
because the container was still running) surfaced a real bug: continueSession's
emit closure only captured "text" events, so when chatWith ended the turn on
an "error" event (LLM returned an empty/refusal response, internal retry also
empty), the worker persisted a completely blank, uninformative "auto" message
— no sign anything had gone wrong, undermining observability of the very
mechanism just built.
- Capture "error" events and, if the turn produced no text/tool_calls at all,
persist an explanatory placeholder instead of blank.
- Add one outer retry of the whole chatWith call when the first attempt
produces nothing — the principle behind this whole feature ("don't give up
on the first error") should apply to the continuation mechanism itself, not
just the homelab commands it's continuing.
Also verified live: recovery-from-failure works via the normal chat path once
prompted, and cleaned up the test container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three fixes for the session where the agent proposed a plan, waited
for 'proceed', then re-queued instead of being auto-approved:
1. Chat-assent fallback: when the operator says 'proceed' but the
preceding turn had NO pending approvals (agent proposed plan in text
without calling request_execution), inject a system note telling the
agent to execute the plan now. Opens the assent window so subsequent
config_mutation commands auto-run.
2. SOUL.md: instruct agent to ALWAYS call request_execution/run when
proposing a plan, not wait for 'proceed' first. This ensures a
pending approval exists for chat-assent to grant.
3. SOUL.md: stronger Docker instructions — Debian 13's docker.io package
installs the daemon but NOT the docker CLI binary. Must use
get.docker.com in post_install. Added 'handling errors' section:
diagnose, try alternatives, continue — don't stop after one failure.
When the operator has approved a plan via chat assent (assent window
active), pct_create and apt_upgrade now auto-approve and execute
instead of queuing for a separate approval round. The auto-approve
path updates the approval+execution status in the DB, then calls the
HTTP API's decision endpoint to trigger executeApprovedAction — same
code path as a manual Approve button, consistent audit trail.
Agent stopped after every approval step, forcing operator to type
'continue' 7× per deploy session. Root causes and fixes:
1. Compound read-only commands (e.g. 'systemctl status; journalctl')
defaulted to config_mutation — now splits on ;/&&/||/| and classifies
as read_only if all segments are inspection verbs. Added grep, wc,
sort, uniq, cut, tr, dpkg -l, apt list, docker stats to allowlist.
2. curl|sh was classified destructive, forcing typed confirmation for
legitimate installs (get.docker.com). Demoted to config_mutation —
loose assent grants it, no typed phrase needed.
3. SOUL.md said 'STOP after queuing' — replaced with 'continue working
on non-blocked steps'. Added assent window section instructing agent
to carry out the full plan after approval.
4. Assent window: when operator approves a plan via chat assent, a
30-minute window opens where config_mutation commands auto-run
without re-approval. Agent writes expiry to autonomy_settings; MCP
run tool checks it before gating. Destructive never auto-runs.
5. System note after approval now says 'CONTINUE executing the full
plan — do not stop and wait for continue.'
Verified live that after deploying the "fixed" bridge-bound pre-flight, it
still let a known-bad vmbr0+192.168.8.2 config straight through to a full
pct_create with no error. Root cause: the check used
`strings.Contains(pingOut, "REACHABLE")` against markers "REACHABLE" /
"UNREACHABLE" — but "UNREACHABLE" contains "REACHABLE" as a substring, so the
containment check was true for BOTH outcomes. The pre-flight was structurally
incapable of ever failing, regardless of the actual ping result.
Fixed with distinct, non-overlapping markers (PREFLIGHT_OK/PREFLIGHT_FAIL)
and exact-match comparison, pulled into a small gatewayPreflightPassed()
helper with a unit test asserting the exact historical bug case
("UNREACHABLE" must be false) so this bug class can't silently recur.
Re-verified live end-to-end: manually re-tested the exact ping command
(confirmed UNREACHABLE via vmbr0), and this was caught only by actually
running the check against production, not by reading the code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live that the pre-flight check added in the previous commit had a
real gap: a plain `ping <gateway>` from the Proxmox host succeeds via the
HOST's own routing table (which can have routes to a subnet through paths
the host alone knows about), even when the CONTAINER — attached via a plain
bridge with only a naive on-link default route — can never actually ARP that
gateway. Confirmed by creating a real test container on vmbr0 with
gw=192.168.8.2: the host-wide ping had said "reachable," but pinging from
inside the container showed 100% packet loss. Fixed by binding the pre-flight
ping to the specific requested bridge (`ping -I <bridge>`), which correctly
rejects vmbr0 for that gateway instead of false-positiving via the host's
broader routing table.
Also confirmed live: vmbr1 does exist and is up on strong (contrary to the
possibly-stale host doc), matching what romm/seanime's docs already said.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Investigated why the operator couldn't get past "no DNS/connectivity" across
multiple retries even after Nomos correctly diagnosed and fixed the gateway
(192.168.8.1 -> 192.168.8.2). It still failed. Root cause, confirmed from
strong's own documented network topology: on `strong`, vmbr0 physically
bridges only to 192.168.178.0/24 — the 192.168.8.0/24 service network is
reached via a Fritz!Box static route, not a local bridge. A container
attached to vmbr0 can never reach a 192.168.8.x gateway no matter which
address in that range is picked; ARP for it just gets silently dropped
(matching the earlier hang symptom). The gateway was never the problem — the
bridge was. 192.168.8.0/24 is also segmented into /28 blocks each with their
own gateway (192.168.8.2 is only the .0-.15 block's gateway), so even a
correct bridge with a copy-pasted gateway from a different block would still
fail.
No amount of retrying with a different gateway guess could have fixed this —
the missing fact (which bridge reaches which subnet, and the per-/28 gateway)
isn't inferable from the subnet alone.
- pct_create gets a `bridge` param (was hardcoded to vmbr0) so a correct
bridge can actually be requested once known.
- Fast pre-flight: for any static IP, ping the gateway from the target HOST
before creating anything. Was: a bad config took a multi-minute hang (or,
after last commit's timeout fix, ~2min) before failing. Now: ~2 seconds,
with a message that explicitly says not to guess a different gateway in
the same subnet — find a real neighbor's config or use DHCP.
- SOUL.md: DHCP is now framed as the default, not a fallback; static IP
requires finding an existing LXC on the same host in the same /28 and
copying its bridge+gateway verbatim — inventing one is explicitly called
out as the failure mode that caused this exact incident.
- MCP tool schema: pct_create's params description now documents `bridge`
and the neighbor-copy rule directly in what the model reads at call time,
not just in SOUL.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).
Two compounding bugs made this unrecoverable without manual intervention:
1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
NO execution timeout — `session.CombinedOutput()` blocks until the remote
command exits, with no deadline. A hung remote process blocks the Go
goroutine forever; the execution can never leave 'running', and the
operator has no way to make it stop. Fixed: both now race the SSH call
against a 10-minute hard timeout, closing the session/client and
returning a clear "timed out after 10m0s" error if exceeded. (The
mcp/server.go copy also still had the original "swallowed non-zero exit"
bug from before that fix was applied to httpapi's copy only — fixed here
too.)
2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
connectivity — it doesn't; a black-holed network can make each call hang
far past the resolver's nominal timeout, so the documented "~90s" budget
was never real. Wrapped every attempt in `timeout 3` so the wall-clock
budget is now actually enforced (~2min worst case), and the failure
message now suggests checking the net0 gateway.
Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):
- InlineApproval's track() polling loop had its own ~6min ceiling and simply
STOPPED polling after that — silently going stale before the backend (now
correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
with margin, and added a distinct 'stalled' state if that's ever exceeded
(explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
execution's created_at), the actual command being run, and the execution
ID — previously just a static "this can take a minute" with zero
information. Also added command display to the destructive pending-
approval card for full transparency before confirming.
Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "chat gave me no further feedback — had to go to Ops": two
compounding bugs, found by reading the actual production session transcript.
1. InlineApproval.svelte — all of last session's live-status/self-heal work —
was never imported or rendered anywhere. Chat.svelte had its own separate,
much dumber approval bar (no status tracking, no destructive handling, just
silently disappears after clicking) that WAS the one users actually saw.
Deleted the dead bar and its state; InlineApproval now renders per-message.
2. chat.ts's extractApprovals hardcoded `tool.name === 'request_execution'`,
so any approval raised by the newer `run` tool was invisible — no card, no
feedback, nothing to self-heal, forcing the operator to the Ops page with
zero acknowledgement in the conversation. This was the actual proximate
cause of last night's destroy-135 session. Fixed to match on response
shape, not tool name, so it doesn't silently break again for the next new
gated tool.
3. Nomos was telling operators "type something like 'I confirm destroy 135'"
for destructive actions (SOUL.md) but no backend path ever consumed that
phrase — chat-assent explicitly (and correctly) excludes destructive from
loose assent, but I never built the alternative. Added
isTypedConfirmation() (cmd/nomos/assent.go): stricter than loose assent,
requires an explicit "confirm" statement, only applies to destructive-
flagged pending approvals.
4. InlineApproval's completed-state hardcoded "Provisioned successfully" —
wrong/confusing for a destroy or arbitrary `run` command. Now says
"Completed on <target>" and shows the actual command output, verified live
against the real destroy-135 execution.
Verified live in a real browser against the production API/DB (dev server
proxying to :8090): the historical stuck session now retroactively renders
both executions as resolved with correct wording and real output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live testing hit `entities_slug_key` violations: exec slugs used an 8-char
prefix of a UUIDv7, whose leading bytes encode a millisecond timestamp — two
executions created seconds apart can share a prefix. Use the full UUID
(guaranteed unique) for the exec entity's slug/name in request_execution, the
new `run` tool, and the REST RequestExecution handler — all three had the
same pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found live: `run` against lxc:caddy failed with "missing pve_id" even though
pve_id=121 was present — caddy is an inventory-seeded LXC with no `host`
attribute at all (only pct_create-provisioned LXCs set one). The combined
query scanned attributes->>'host' (SQL NULL) into a plain Go string, which
errors the whole Scan — including the pve_id column that scanned fine.
COALESCE the host column to '' so a missing host attribute degrades to the
documented default instead of failing the whole resolution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the first slice of plans/2026-07-10-general-gated-execution.md:
Nomos gets one general execution tool instead of only a fixed action enum,
gated by an automatic risk classifier, and approval can be granted by the
operator just replying in chat instead of clicking a button.
- internal/policy/command.go: ClassifyCommand(cmd, declaredRisk) — rule-based
read-only allowlist + destructive denylist, default-escalate to
config_mutation for anything else. Classification can only ESCALATE the
caller's declared risk, never de-escalate it (destructive always wins even
if declared read_only). Compound commands (&&, ;, |, $()) never qualify for
the read-only fast path. Full test corpus.
- internal/mcp/server.go: new `run` MCP tool — target (host:/lxc:), command,
purpose, optional declared_risk. Read-only commands execute immediately;
everything else queues an approval exactly like pct_create today, executed
via httpapi's existing executeApprovedAction. Also fixes a real latent bug:
pct_exec resolved an LXC's host attribute without the "host:" prefix, so it
could never find the Proxmox host — new resolveExecTarget/resolveRunTarget
helpers (mcp + httpapi) fix this for both the new `run` action and existing
actions that route through the same execution path.
- internal/httpapi/phase3.go: "run" case in executeApprovedAction; fixes two
bugs found while wiring this up — (1) DecideApproval hardcoded risk_class to
'config_mutation' on every approve, silently corrupting the audit ledger for
every other risk class; (2) denying/revoking an approval never updated the
linked execution's status, so it stayed 'pending_approval' forever instead
of reflecting the decision.
- cmd/nomos/assent.go: deterministic (not LLM-judged) chat-assent detection.
Scoped to the immediately-preceding assistant turn's pending approvals only
— an old "yes" can't retroactively approve something new. Destructive-risk
actions are excluded from loose assent. Approves via the same HTTP decision
endpoint the UI button calls, so both paths share one audit trail.
- web/.../InlineApproval.svelte: self-healing poll — a pending approval card
now picks up being decided via ANY path (chat assent, Ops page, Matrix), not
just its own button. Previously the banner stayed stuck showing
Approve/Deny even after the action had already run elsewhere.
- nomos/SOUL.md: `run` is now the general capability ("no fixed menu, only a
risk gate"); documents chat-assent behavior and the destructive exception.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator directives: (1) the UI must show what's executing, its risk
classification, live status, and what knowledge the session created — the
system's growth should be visible, not just trusted. (2) approval should be
granted by chat assent ("go ahead"), not a separate button; destructive
actions still require a typed confirmation phrase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Evaluate the agent's action path against the OIKOS.md design. Finding: the
intended model (unlimited runbook-driven actions gated by a risk classifier)
already exists on paper and in scaffolding, but the live agent path regressed
to a hard-coded 5-action enum that bypasses the classifier. Plan a layered
realignment: (0) one general gated `run` primitive, (1) runbooks-as-data as the
reliable fast-path, (2) learning. Chosen v1 posture: approve-most (read-only
auto-runs, all state changes gate). Incremental, each step shippable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fresh Debian LXCs have no locale configured, spamming "apt-listchanges:
Can't set locale" / perl warnings across every install and breaking some
packages' post-install scripts. Pin LANG/LC_ALL=C.UTF-8 (and hoist
DEBIAN_FRONTEND) at the top of the in-container bootstrap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Production session provisioned the container but the service never installed:
apt failed with "Temporary failure resolving deb.debian.org" — a static-IP LXC
whose assigned nameserver couldn't resolve. The operator also got zero feedback:
the approval banner just sat there with no running/complete/failed status.
Backend robustness (provisionScript):
- Wait for real DNS/connectivity inside the container before apt, and self-heal
/etc/resolv.conf to a public resolver (1.1.1.1/8.8.8.8) if the assigned one
is dead. `set -e` after the gate so apt/post_install failures surface.
- apt-get update/install with Acquire::Retries=3.
Frontend feedback (InlineApproval):
- After approve, poll GET /executions/{id} and show live phase: submitting →
provisioning… → provisioned successfully / execution failed (with the error).
- add getExecution() to api.ts.
Agent guidance (SOUL.md):
- omit vmid (auto-assigned), prefer dhcp, docker-compose-plugin is not in Debian
(use docker.io + get.docker.com), end post_install with a health check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The final "UPDATE executions SET result=$::jsonb" built its payload with
fmt.Sprintf and only escaped newlines. apt/pct output contains quotes,
backslashes and control chars, so the payload was invalid JSON, the jsonb
cast failed, and the (unchecked) UPDATE was silently discarded — the
execution stayed 'approved' with a NULL result even though the LXC was fully
provisioned (verified live: vmid auto-assigned, container running, service
installed, post_install ran).
- executeApprovedAction: marshal result via json.Marshal; log UPDATE errors
- add jsonErr() helper; route all pct_create failure-path results through it
- mcp/server.go: add jsonOut() for restart/systemctl/pct_exec inline results
- regression test for JSON validity on quote/backslash/control-char output
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups found while verifying the approve→provision path end to end:
- vmid is now optional: the early required-field check rejected vmid:0
before the cluster VMID guard could auto-assign a free id. Only hostname
is required now; 0 (or a collision) resolves to `pvesh get /cluster/nextid`.
- net0: use ip=dhcp with no gateway when no static IP is given (Proxmox
rejects gw alongside dhcp); only attach gw for a static CIDR.
- bump post-create settle to 10s so a DHCP lease is up before apt runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real production failure when the operator clicked Approve in chat: nothing
provisioned, banner never cleared, execution marked completed.
Three root causes:
- sshExec swallowed non-zero exits when the command produced output, so a
`pct create` that printed "CT 132 already exists" and failed was reported
as success and a bogus lxc entity was registered. Now any non-zero exit
returns an error (with output) so the execution is correctly marked failed.
- The LLM reused VMID 132 (belongs to lxc:rclone; VMIDs are cluster-wide).
pct_create now checks in-use VMIDs via `pvesh get /cluster/resources` and
falls back to `pvesh get /cluster/nextid` when the requested id is taken.
- InlineApproval.svelte reset its state on every prop change (done was also
compared against the wrong string), so the banner never cleared and each
click re-POSTed /decision. Rewritten to track outcome per executionId,
clear on success, and block resubmits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "asks permission but never acts": the approved pct_create
execution failed to parse because the LLM emitted `"privileged":0` /
`"nesting":1` (numbers) into strict `bool` fields, so the container was
never created. Compounded by a hardcoded template name (debian-13.0-1)
that no longer exists on the host, and no way for the agent to read the web.
- flexBool: accept 0/1, "true", bool for privileged/nesting (the exact prod failure)
- pct_create template pre-flight: list host cache, validate/auto-pick newest debian
- pct_create services[] + post_install: one approval provisions a working service
- new http_get MCP tool (sanitized, size-capped, SSRF-guarded) — agent can read repos/sites
- request_execution description: target=host, full JSON schema + example
- SOUL.md: agent CAN fetch the web; prefer one-step provisioning
- default model deepseek-v4-flash -> v4-pro; maxIterations 15 -> 25
- unit tests for flexBool, template resolve, pkg sanitize, HTML sanitize + SSRF block
Verified live on host:strong with a throwaway VMID 999: template auto-resolved,
container created + booted, services installed, post_install ran, then destroyed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Execution entity name now includes UUID suffix: 'pct_create on host:strong (abc12345)'
so the (type,name) UNIQUE constraint doesn't block subsequent executions for the
same target+action. Dedup now uses JOIN + LIKE prefix match to find only
pending_approval executions.
- Move persistent approval bar from top of messages area to just above the
chat input box (bottom-fixed position, above the textarea form).
- Add dedup in request_execution: check entities(type,name) uniqueness before
creating duplicate executions. Returns 'already queued' message to the LLM,
preventing tool-calling loops.
- Fix createApproval JSON payload: use json.Marshal instead of fmt.Sprintf
to escape params (could contain unescaped double quotes from JSON config).
- Add ON CONFLICT DO NOTHING to entity/execution inserts for dedup race safety.
- Persistent approval bar at top of Chat.svelte: aggregates pendingApprovals
from all messages, fixed position (won't scroll away). Approve/deny/approve-all.
- Update SOUL.md: agent must STOP after queuing a gated action.
- Fix ToolCallGroup reactivity: wasActive = (active).
- Replace text-based regex parsing in InlineApproval with structured
pendingApprovals extracted from request_execution tool results. The tool
result text is deterministic (not LLM-generated), making UUID extraction
reliable regardless of how the LLM rephrases the response.
- Fix ToolCallGroup reactivity: wasActive = active captured initial
value. Now uses (active) so re-runs on prop changes.
- Extract approvals in both live streaming (done event) and history loading
for consistent behavior on resumed sessions.
| Narrative — synthesized wiki | `archive/knowledge/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
| Concern | Where it lives | How it gets there |
|---------|----------------|-------------------|
| Knowledge content — documents, investigations, runbooks | `knowledge_entities` table (rows linked to `entities` via `documents` / `about` edges) | Seeded from `seeds/knowledge.yaml` at deploy; mutated at runtime via the API |
| Seed manifest (bootstrap + DR) | `seeds/knowledge.yaml` | Hand-edited or regenerated; ingested idempotently (content-hashed via `seed_versions`) |
| Archived narrative wiki (read-only history) | `archive/knowledge/` (deleted 2026-08-16) | Frozen 2026-07-07 when the DB became source of truth; no longer in repo |
# Agent enrollment — bootstrap a client into the homelab context system
# Agent enrollment — operational notes
This walks through enrolling a new machine (workstation, LXC, or VM) so it
joins the cross-client context system: a `/opt/homelab-context/` clone of
this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
**For the actual enrollment flow, see [CLIENTS.md](../../CLIENTS.md#enrollment)
— it's the current, authoritative version.** This page used to duplicate
that flow in more detail, describing a `homelab` CLI-based two-step
ceremony (`homelab client add` reserves an inventory slot → client
bootstraps → operator finalizes the pubkey). That CLI and that flow don't
exist anymore — enrollment today is one shot: `bootstrap.sh` calls
`POST /api/v1/clients/enroll` directly and gets back an age keypair +
Infisical identity in the same response. What's left here is the handful
of things that are still true and weren't already covered elsewhere.
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
> See [nomos-agent.md](nomos-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-nomos` flag.
Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.
## Prerequisites the client must satisfy
## Prerequisites
| Requirement | Why | How to check |
| --- | --- | --- |
| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| Hostname matches an entry in `inventory.yaml` | `EnrollClient` looks up the entity by slug derived from hostname; it must exist in `planned`/`provisioning` state. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit `<bootstrap-tier1>`). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`.
**Path B — interactive OIDC (default; recommended):**
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../archive/knowledge/containers/106-auth-outpost.md). Only needed once per account.
**Path A — setup-key (headless/scripted onboarding):**
Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.
1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`.
3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver):
```bash
sudo netbird up --setup-key <KEY> \
--management-url https://netbird.hubris.network \
--ssh-jwt-cache-ttl 86400
```
**Why we can't OIDC-login from the public internet (still open as a follow-up):**
`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124.
### DNS prerequisite
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../../archive/knowledge/infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:
- **LAN client**: set DNS to 192.168.8.180 (per-interface or
`/etc/resolv.conf`).
- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script).
Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.
revocation). Likely maps to an entity lifecycle transition
(`.agents/skills/lifecycle-deprecate-node/` or `lifecycle-destroy-node/`)
but those skills reference the same dead CLI and need their own check.
- **Granting a secret to an already-enrolled client.** The old flow
hand-edited `.sops.yaml``creation_rules` + `sops updatekeys`. Given
Infisical is now the primary secrets backend (SOPS is the DR fallback),
the current mechanism is probably Infisical-side, not a `.sops.yaml` edit
— not confirmed.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git` |
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Enrollment 404s / entity not found | Hostname doesn't match a `planned`/`provisioning` inventory entry | See "Hostname mismatch" above |
| `gnutls_handshake() failed` / TLS errors reaching `*.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS instead of the LAN/mesh path | See the networking runbook (split-horizon DNS section) |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../archive/knowledge/containers/106-auth-outpost.md). |
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
Added a new "Post-bootstrap: SSH reachability" section covering SSH key
generation, pubkey publication, deployment to hosts, SSH config generation,
and LAN IP registration. New workstations enrolled via this doc will
automatically join the universal SSH mesh.
### 2026-05-31 — cross-link to nomos-agent.md
### 2026-07-12 — trimmed to current architecture
Removed everything describing the retired `homelab` CLI-based two-step
enrollment ceremony (now: `CLIENTS.md`'s one-shot flow), the Nous-Hermes/
Goose cross-link (that whole flow was removed the same day), and CLI-syntax
troubleshooting rows with no current equivalent. Migrated the still-true
Netbird/DNS/SSH-distribution content to a knowledge-base runbook rather
than duplicating it here. What's left is genuinely current or explicitly
flagged as unverified. Original ~365-line version is in git history
(`git log -- .agents/operations/agent-enrollment.md`) if any of the removed
detail turns out to still be needed.
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([nomos-agent.md](nomos-agent.md)) and noted it at the top of this page. The Nomos flow extends `bootstrap.sh` with `--with-nomos` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
Added three rows to the troubleshooting table covering issues surfaced
during the netbird vanilla migration. (Migrated 2026-07-12 to the
networking runbook.)
### 2026-05-20 — initial page
Captures the enrollment flow validated during Phase 2 of the homelab
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
Run from the hubris host as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
## Proxmox CLI
@@ -8,13 +8,13 @@ Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. Whe
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../../docs/infrastructure/media-permissions.md)) |
| `pct enter <id>` | Shell into a container |
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
| `pvesm status` | Storage pools status |
| `pvesh get /nodes --output-format json` | Node summary as JSON |
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Nomos cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Nomos cron — see [monitoring](../../../docs/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
@@ -22,22 +22,22 @@ Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. Whe
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../../docs/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
## Reverse proxy
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Caddyfile: `/etc/caddy/Caddyfile` on LXC 121.
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../../docs/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Split-horizon authority: [Technitium DNS](https://technitium.com) on dns (107) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
- See [DNS](../../../docs/infrastructure/dns.md).
## Web access
@@ -49,43 +49,47 @@ Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. Whe
-`ras-mc-ctl --errors` — full event log
-`cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
-`cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
-`ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/archive/2026-04-21-hubris-crash-loop.md))
-`ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see investigation)
## Fleet apt operations
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
| Command | What it does |
| --- | --- |
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
**No current CLI equivalent.**`homelab apt-audit`/`apt-upgrade` (dpkg-state
audit, fanned-out apt upgrade with pre-upgrade snapshots) were part of the
retired Python `homelab` CLI and don't have a ported replacement — apt
patching today is ad hoc `run` MCP tool calls per host, without the
audit/snapshot/status wrapping this used to provide. If that wrapping is
still wanted, it needs to be rebuilt (e.g. as a runbook driving `run`, or a
new MCP tool) — see
[runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) for
the dpkg-interrupted recovery procedure specifically.
## Oikos (agent OS layer)
See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
See [OIKOS.md](../OIKOS.md) for the operating model. The `homelab` CLI this
section used to document is retired; the actual current interface is the
MCP tool catalog in [AGENTS.md §3](../../AGENTS.md#3-the-mcp-server) plus
the REST API. Closest current equivalents for what used to live here:
| Command | What it does |
| Old `homelab` command | Current equivalent |
| --- | --- |
| `homelab service <name> explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history |
| `homelab node <name> relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius |
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Nomos, or the Oikos Console's `/approvals` page) |
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
| `homelab signal list\|ack\|resolve\|mute` | MCP `get_signal_history`, or REST `POST /api/v1/signals/{id}/ack\|resolve\|mute` (the control-room UI's Signals page wraps these) |
| `homelab approval request\|list\|reply\|check` | REST `GET/POST /api/v1/approvals*` (Matrix-delivered via the notifier, or the control-room UI's Operations page) |
| `homelab restart <service> --approval-id <id>` | MCP `run` (policy-gated — auto-executes if read-only/reversible_low, otherwise queues for the same Matrix/UI approval) |
| `homelab decide <action> <entity>` | No direct equivalent — classification now happens inline inside `run`, not as a separate dry-run call |
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../archive/oikos-cards/).
There is no separately-deployed "Oikos Console" anymore — the control-room
SPA is the operator dashboard. It lives in its own repo
(`dtoro/oikos-web`, local checkout `~/Projects/oikos-web`) with its own
deploy pipeline, publishing the same host port 8091 as before (Phase 1 of
| `nomos: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-nomos` from hubris |
| `nomos: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
| Goose hangs on first `nomos` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
| `homelab` extension fails to connect / no MCP tools listed | MCP server upgraded in Go rewrite (`internal/mcp/server.go`, Streamable HTTP via official MCP SDK). Old FastMCP SSE transport is deprecated. | Run `docker compose --profile full up` on mac-mini, or wait for the production cutover from apps/105. |
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-nomos`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
## Cross-references
- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the
Nomos flow assumes is done.
- [`NOMOS.md`](../NOMOS.md) — the persona the Nomos agent reads on every
**Content / narrative pages:** lowercase-with-dashes, date-prefixed as needed
- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from `inventory.yaml`.
- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from the entity's attributes in the DB (seeded via `seeds/inventory.yaml`).
- **Infrastructure / cross-cutting pages:** `<topic>.md` (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node.
@@ -36,7 +36,7 @@ Every doc-level page follows the same shape so a reader scans it in one pass.
1.**One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`.
2.**Opening definition.** First paragraph, 1–3 sentences, says what the thing is. No motivation, no marketing, no setup.
3.**Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md).
4.**`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (Go MCP `get_changelog` in `internal/mcp/server.go`); keep the `### YYYY-MM-DD — title` shape.
4.**`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is stored as a structured field on the `document` entity in the DB; keep the `### YYYY-MM-DD — title` shape so it parses cleanly.
5.**Related links** only at the bottom, only when a reference cannot be woven inline.
## Section indexes (folder READMEs)
@@ -53,12 +53,12 @@ duplicated prose, no narrative between the intro and the table.
- Prefer **tables** for enumerable items with internal structure (service/port maps, field lists, status grids). Reserve bullets for short non-structured lists.
- Use the **bold-leading-phrase pattern** for structured points: `**Read-only by construction.** The MCP server never mutates state.` — a bold noun phrase, a period, then the explanation.
- When enumerating across services or nodes, give each its own `###` sub-section or a table row, not one run-on paragraph.
- Use backticks for code, paths, hostnames, and file names (`inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology.
- Use backticks for code, paths, hostnames, and file names (`seeds/inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology.
- Use `>` blockquotes for caveats and gaps that interrupt the main flow: `> **Outstanding gap.** DNS-vs-inventory drift check not yet wired.` One thought per blockquote.
## Diagrams
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it. (Go DB-native topology generation planned.)
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` in the archived wiki was generated by the retired `oikos/gen-topology.py`; the DB-native equivalent is a future task — do not hand-edit the archived file expecting it to regenerate.
- ASCII box diagrams are fine for small shape diagrams; keep them to one screen.
description: "Examine a Nomos chat session, compare the user's objective with the actual outcome, identify causes of failure (missing tools, excessive tool calls, blocked actions, model behavior), and propose concrete fixes."
risk_class: reversible_low
inputs: [session_id]
---
# Session review
Analyze Nomos chat sessions from the live database, diff objectives
get_lxc_state(lxc_slug) — pct status from Proxmox host
ping_service(service_slug) — HTTP reachability + scheduler health state
list_lxcs(state) — all LXC containers with ID, host, IP, last-audited hint
restart_service(target, service) — restart a systemd service (config_mutation, requires approval)
docker_exec(lxc_slug, container, command, purpose) — run a command inside a Docker container on an LXC; handles escaping, resolves target from entity graph; read-only auto-runs, mutations require approval
push_file(target, source_path, dest_path, backup=true) — push a file into an LXC from the Proxmox host (config_mutation, requires approval)
ack_signal(signal_id) — acknowledge an open signal
resolve_signal(signal_id, resolution) — resolve a signal with optional note
mute_signal(signal_id, duration_s=3600) — temporarily mute a signal
cancel_execution(execution_id, reason) — cancel a queued/running execution
update_check(check_id, enabled) — enable or disable a health check
list_checks(entity_slug, enabled) — list health checks with verdict, probe kind
list_executions(entity_slug, status, limit=25) — cursor-paginated execution history
list_entity_sessions(entity_slug) — active Nomos sessions linked to an entity
Knowledge Tools — search, read, and maintain the knowledge base:
search_knowledge(query) — full-text search across docs (snippets, not full body)
get_entity_knowledge(entity_slug) — all docs/investigations/runbooks linked to a slug
get_knowledge_content(slug) — full markdown body of one knowledge entry
upsert_knowledge(title, content, about, tags, kind) — write what you learned
upsert_session_summary(session_id, summary, entities_touched, discoveries) — batch-write session findings into the graph; creates knowledge entries, links entities, records a session-audit entry
delete_knowledge(knowledge_slug) — soft-delete a knowledge entry
restore_knowledge(knowledge_slug) — restore a soft-deleted entry
merge_knowledge(target_slug, source_slugs) — fold entries into a target
rename_knowledge_tag(from, to) — bulk-rename tags across all entries
get_knowledge_revisions(knowledge_slug) — version history for a knowledge entry
get_knowledge_duplicates(threshold=0.6) — near-duplicate detection via trigram similarity
get_knowledge_orphans(stale_days=90) — unlinked, untagged, or stale entries
list_knowledge_tags() — all tags with usage counts and casing variants
list_my_secrets(caller_pubkey) — secrets accessible to a client by age public key
Analysis Tools — fleet health, metrics, and introspection:
get_health_summary(health) — fleet health per entity, optionally filtered
get_audit_trail(entity_id) — query the audit log
query_metrics(hours=24) — time-series with bucketed avg/min/max
get_signal_history(entity_slug, state, limit=50) — open and recent signals
@command -v golangci-lint >/dev/null 2>&1&& golangci-lint run ||echo"golangci-lint not installed, skipping"
golangci:
@command -v golangci-lint >/dev/null 2>&1&& golangci-lint run --config .golangci.yml ||echo"golangci-lint not installed — see https://golangci-lint.run/usage/install/"
govulncheck:
@command -v govulncheck >/dev/null 2>&1&& govulncheck ./... ||echo"govulncheck not installed — run: go install golang.org/x/vuln/cmd/govulncheck@latest"
.PHONY:lintvetgolangcigovulncheck
generate:
$(GO) run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.4.1 \
@@ -27,7 +40,7 @@ generate:
# CI drift guard: regenerate and fail if the committed output changed.
Full plan: [plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md).
Full plan: [plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md).
- 122 arriman — *arr stack, downloads write to library
**Stay on hubris, NFS-mount library from ludo-mini:**
- 103 paperless — documents, moderate I/O
- 104 gitea — git repos (small files, some I/O sensitivity but acceptable over NFS)
- 105 apps — Docker apps, mixed workloads
- 114 nextcloud — file sync
- 119 sophia — workshop, light use
- 126 plato — app, light use
- 100 zimaos — NAS frontend, already NFS-mounted
### 2. NFS architecture
Instead of changing every LXC's mount config, keep the bind-mount pattern on hubris:
```
ludo-mini: /mnt/library (ext4, local NVMe)
│
└── NFSv4 export to 192.168.8.0/24
│
└── hubris host: NFS-mount at /mnt/library
│
└── LXCs: bind-mount /mnt/library (unchanged!)
```
This is transparent to all hubris LXCs — no container config changes needed. Only the hubris host changes from ext4 local mount to NFS mount. The LXC bind-mounts "just work" because `/mnt/library` is still at the same path on the host.
### 3. Network — ludo-mini reachability from hubris LXCs
LXCs on `192.168.8.0/24` reach ludo-mini (`192.168.178.181`) through hubris routing:
**Alternative (cleaner):** Add a secondary IP `192.168.8.x` on ludo-mini's physical interface so it's directly on the homelab subnet. This avoids the router hop and keeps NFS traffic off kernel forwarding path. Worth considering but not required.
### 4. Gaming on ludo-mini with Proxmox
ludo-mini runs Sunshine (game streaming). Under Proxmox:
- **Option A:** Gaming VM with GPU passthrough — Sunshine + games in a VM, full GPU access
- **Option C:** Keep Sunshine on the Proxmox host itself (not recommended, but simplest)
Option A is the cleanest for isolation. Games need a full desktop environment and GPU drivers; a VM with GPU passthrough gives them that.
### 5. What about nfs-export (LXC 102)?
Currently exports `/mnt/library` to zimaos. After migration:
- If zimaos stays on hubris and accesses library via host NFS → bind-mount → LXC 102, that's triple-hop (ludo-mini → NFS → hubris → bind-mount → LXC 102 → NFS → zimaos). Terrible.
- Better: zimaos NFS-mounts directly from ludo-mini.
- So LXC 102 gets decommissioned (or repurposed).
- zimaos gets a new NFS mount pointing directly at ludo-mini.
## Migration phases
### Phase 1 — Preparation (no downtime)
1.**Document current state** on hubris:
-`pct list` — full container inventory
-`pct config <id>` for every library-mounting LXC
-`cat /etc/fstab` — capture the library mount line
-`df -h /mnt/library` — confirm space usage
-`lsblk -f` — UUID, filesystem
- Identify the exact NVMe device (`nvme1n1`)
2.**Pre-flight on ludo-mini:**
- Confirm hardware: CPU, RAM, available M.2 slots, GPU model
- Confirm it can take the Samsung 990 EVO Plus (M.2 NVMe, PCIe 4.0 x4)
- Verify BIOS supports virtualization (VT-d/AMD-Vi for PCIe passthrough)
- Check: does ludo-mini have a second drive for Proxmox OS? If not, we need to partition the library SSD for Proxmox boot + library LVM, which complicates things significantly
3.**Install Proxmox on ludo-mini:**
- Download Proxmox VE 9.x ISO
- Install to ludo-mini's system drive (NOT the library SSD)
- Configure networking: bridge for Proxmox, IP on 192.168.178.x
- Test: web UI accessible
4.**Prepare NFS server on ludo-mini Proxmox:**
- Create NFS-export LXC (or serve from host — simpler for now)
5. **Update DNS:** `nfs-export.hubris.network` → ludo-mini IP (or remove)
6. **Run `homelab sync`** to propagate changes
## Open questions / unknowns
1. **Does ludo-mini have a second drive for Proxmox OS?** If not, we'd need to repartition the library SSD — carve out ~100 GB for Proxmox, then the rest for library LVM. This is risky (data loss if partitioning goes wrong) and requires a full backup first. **Alternative:** Buy a small SSD for ludo-mini's OS.
2. **What GPU does ludo-mini have?** Proxmox GPU passthrough requires IOMMU support and a GPU that doesn't have the reset bug. Need to check the exact GPU model.
3. **NFS performance for git (gitea)?** Git operations over NFS can be problematic (locking, stat() storms). Gitea bare repos at `/mnt/library/repos/*.git` might need testing. Worst case: move gitea's repo storage to local disk and keep `/mnt/library` for large file/LFS storage only.
4. **Media permission drift.** NFS `all_squash,anonuid=33,anongid=10000` ensures all writes from hubris LXCs (over NFS) and ludo-mini LXCs (local) land as `www-data:media`. This is the same squash currently used by nfs-export (102). Should be fine.
5. **ludo-mini network — add 192.168.8.x address?** Adding a secondary IP on ludo-mini's interface directly on the homelab subnet avoids routing through hubris for NFS traffic. Cleaner, but requires Proxmox bridge setup. Worth doing during Proxmox install.
6. **Sunshine migration.** Currently runs on ludo-mini bare metal. After Proxmox install, it needs to run in a VM. What happens to existing Sunshine configs, game libraries, save files? Need to preserve these during the Proxmox install.
7. **Backup before moving.** The library SSD holds 3.7 TB of irreplaceable data (documents, photos, repos). Restic backups are currently disabled. **Before physically moving the drive, verify the data is readable and consider doing one backup** — or at minimum, `rsync` critical directories.
# Plan: Narrow Technitium DHCP Pool to Avoid Static-IP Conflicts
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
**Goal:** Eliminate the IP conflict risk created by the Technitium DHCP pool (`.100–.240`) overlapping with all static LXC/VM IPs (`.101–.239`).
**Architecture:** Shrink the DHCP pool range on Technitium so it only covers IPs that no static host uses. No LXC/VM IPs change. Single server-side change (Technitium API), plus documentation updates.
**Tech Stack:** Technitium DNS API (`/api/dhcp/scopes/set`), bash/curl, homelab-context repo for docs.
---
## Problem statement
The Technitium DHCP server on [CT 107](../../knowledge/wiki/containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
| Host | IP | Inside pool? |
|---|---|---|
| hubris (Proxmox) | .77 | No — below `.100` |
| haos (VM 108) | .101 | YES |
| gitea (104) | .121 | YES |
| paperless (103) | .130 | YES |
| arriman (122) | .132 | YES |
| mule-images (120) | .136 | YES |
| sophia (119) | .157 | YES |
| mac-mini | .174 | YES |
| caddy (121) | .175 | YES |
| authentik (124) | .180 | YES |
| plato (126) | .190 | YES |
| zimaos (VM 100) | .195 | YES |
| nfs-export (102) | .200 | YES |
| apps (105) | .205 | YES |
| jellyfin (101) | .206 | YES |
| nextcloud (114) | .224 | YES |
| claudio-bot (123) | .230 | YES |
| elementsynapse (118) | .239 | YES |
The docs claim "Static-IP LXCs (below `.100`) are unaffected" — this is **false**. Static IPs span `.101–.239`, the DHCP pool spans `.100–.240`. They overlap almost entirely.
If the DHCP server hands out `.121/.136/.224` (or any of the above) to a new dynamic client before the static LXC claims it on boot, the static service will fail to bind and the service goes dark.
---
## Proposed approach: Shrink the pool
**Move the DHCP pool start from `.100` to `.241`**, resulting in:
**Pitfall:** If the API returns `{"status":"error"}`, the scope name or parameter format may differ. Inspect the response body. Technitium's API might use `rangeStart`/`rangeEnd` instead of `startingAddress`/`endingAddress`. Adjust if needed (check the full scope object from Task 1 step 2 for exact key names).
---
### Task 3: Check for active DHCP leases in the old pool that would be stranded
**Objective:** Ensure no DHCP client is currently holding an IP in `.100–.240` that it will lose when its lease expires.
- If the only leases are from static LXCs that configured themselves before the DHCP move (e.g., old leases from before the 2026-06-02 static-IP migration), these leases are stale and harmless.
- If a *dynamic* client (e.g., a test laptop, transient VM) holds `.195` or similar, note it — it will lose its IP on next renew and should be moved to a static assignment or into the `.241+` pool.
- **ZimaOS (VM 100) at `.195` is a DHCP lease, not static** — this is the one host that needs attention. Either:
- Set a static IP inside ZimaOS (preferred), or
- Add a DHCP reservation for MAC in Technitium to pin `.195`
**Verification:** No "surprise" dynamic clients that would break on lease expiry.
---
### Task 4: Fix ZimaOS IP stability (if needed)
**Objective:** Ensure ZimaOS at `.195` won't float or break when the pool shrinks.
**If ZimaOS already has a static IP configured inside the VM:** Nothing to do.
**If ZimaOS is DHCP-only (likely — doc says "DHCP lease, not a reservation"):**
Option A (preferred): Set a static IP inside ZimaOS via its web UI at `http://192.168.8.195` → Settings → Network → Static IP → `192.168.8.195/24`, gateway `192.168.8.1`, DNS `192.168.8.2`.
Option B: Add a DHCP reservation in Technitium for ZimaOS's MAC address:
**Pitfall:** The `/api/dhcp/reservations/add` endpoint signature is unverified — confirm the exact endpoint name from Technitium's API docs or the web UI before running it. The web console at `http://192.168.8.2:5380` → DHCP → Reservations can be used as a manual fallback.
---
### Task 5: Update documentation in homelab-context
**Objective:** Fix the now-wrong claims about static IPs being "below .100".
**Files to edit:**
1.**`infrastructure/network.md`** — Line 53
- Old: `Most homelab LXCs use static IPs below \`.100\`. DHCP only covers new/transient containers.`
- New: `Static IPs span \`.101–.239\` (all LXCs + VMs + workstations). DHCP pool narrowed to \`.241–.254\` to avoid overlap.`
2.**`containers/107-dns.md`** — Lines 37, 42, 55
- Line 37: Update pool range: `192.168.8.241 – 192.168.8.254`
- Line 42: `Static-IP LXCs (below \`.100\`)` → `Static-IP LXCs (\`.101–.239\`) are excluded from the pool.`
- Line 55: Add changelog entry for the pool shrink
### 2026-06-03 — DHCP pool narrowed to `.241–.254` to exclude static IPs
Previous pool `.100–.240` overlapped with all static LXCs/VMs (\`.101–.239\`), creating IP conflict risk. Shrunk pool to `.241–.254`. No services re-IP'd. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
```
4. **`infrastructure/network.md`** — Line 51: Update pool range in the DHCP table row.
5. **`plans/2026-06-01-slate-ax-to-sodola-migration.md`** — Line 60: Optionally update the pool range in the config table (or add a post-migration note). This is the historical migration plan, so a footnote rather than an edit may be better.
| API call fails (wrong field names) | Medium | Low | Inspect live scope object first (Task 1); adjust payload |
| ZimaOS loses IP on next boot | Low | Medium | Task 4 makes ZimaOS static or reserved |
| Active DHCP client in `.100–.240` gets stranded | Low | Low | Task 3 surfaces this; client just requests a new IP from `.241+` |
| Technitium admin password file missing | Low | Medium | `/opt/technitium/admin_password.txt` was created during setup; verify existence |
## Open questions
1. Is zimaos (VM 100) currently DHCP or static? The doc says DHCP lease, but it's listed as `lan_ip: 192.168.8.195` in inventory. If it's actually DHCP, it's the one host that needs a static assignment before the pool shrinks.
2. Are there any transient DHCP clients (test laptops, phones) on the homelab subnet that hold `.100–.240` addresses? Check leases before cutting over.
3. Should we widen the pool slightly (e.g., `.230–.254`) for more headroom? Currently 14 IPs. If 3+ transient devices are expected, `.230–.254` = 25 IPs — still safe since the highest static is `.239` and `.230–.239` could be excluded.
## Execution preference
All changes are on the Technitium API + homelab-context repo. No LXC/VM restarts needed. The pool shrink takes effect immediately for NEW DHCP requests; existing leases in the old range continue until expiry (24h max).
# Plan: Prevent DHCP IP drift from breaking Caddy backends
**Date:** 2026-06-05
**Slug:** prevent-dhcp-ip-drift
---
## Goal
Eliminate the root cause of services becoming unreachable when DHCP lease renewals change backend IPs that Caddy's `reverse_proxy` directives hardcode.
**Triggering incident:** Paperless (LXC 103) and HAOS (VM 108) had DHCP-assigned IPs change from `.130→.243` and `.101→.241` respectively. Caddyfile still pointed at the old IPs → services unreachable from iPhone on Netbird.
## Current context
### DHCP vs static IP inventory
| Machine | Type | PVE ID | Current IP | Allocation method | Status |
| `/etc/caddy/Caddyfile` on CT 121 | Already fixed — only changes again if we re-assign paperless IP to `.130` |
| `scripts/check-caddy-backends.sh` | New validation script (new file in homelab-context) |
### Risks / Tradeoffs
- **Pinning paperless to `.130`** — if the LXC was reinstalled since then, `.130` may already be in use. Verify first with `arp-scan` or `nmap`.
- **HAOS static IP via Technitium reservation** vs **inside HAOS**: Technitium reservation is simpler (no HA config changes), but if HAOS's DHCP lease expires and the Technitium server is down, the reservation won't help. A static IP inside HAOS is more robust but requires poking the HA console.
- **Validation script false positives** — a service might be legitimately down for maintenance. The script should be a warning, not an alert.
- **Caddy reload** — each Caddyfile edit triggers an auto-reload via webhook. If the backend is down during reload, Caddy itself stays up (it's just a reverse_proxy target).
4. Confirm from iPhone: both `paperless.hubris.network` and `home.hubris.network` load
### Open questions
1. Should paperless go back to `.130` (its original), or stay at `.243` (current)? Going back to `.130` means updating the Caddyfile again, but keeps the static range allocation consistent.
2. HAOS: Technitium reservation or HAOS-internal static config? Reservation is easier; HAOS-internal is more robust.
3. Should the Caddyfile validation script run as a homelab cron job, or as a cron on the caddy LXC itself?
4. ZimaOS (VM 100) — should we also pin its IP while we're at it?
Stop requiring repeated login to Authentik (several times per day) by fixing session and cookie expiry settings so the user stays logged in for longer periods (e.g., 7–30 days, or until explicit logout).
## Current Context
Authentik runs on the VPS (`82.165.190.79`) in Docker Compose. Traffic flows:
Caddy's `forward_auth` uses the `(authentik)` snippet which proxies to `auth.hubris.network/outpost.goauthentik.io/auth/caddy`. The Authentik server version is **2026.5.2**.
The Authentik Django session (`authentik_session` cookie) is configured to **expire on browser close**. Every time the user closes and reopens their browser, the session cookie is cleared. The next visit to a service that requires OAuth2 authorization (Gitea, Jellyfin, etc.) will redirect to the Authentik login page.
Even with the browser left open continuously, the session expires after 24 hours. Combined with `SESSION_SAVE_EVERY_REQUEST = False`, activity does NOT extend the session.
- **Proxy cookie validity** — `hubris-forward-auth` has `access_token_validity = hours=24`, which is reasonable for the forward-auth token.
- **Server-side session duration** — The `user_login` stage has `session_duration = seconds=0` (indefinite).
- **Refresh tokens** — All OAuth2 providers have `refresh_token_validity = days=30`, which is fine.
- **Caddy configuration** — The forward-auth chain is correctly set up.
- **Outpost health** — All containers healthy, up for 6 days.
## Proposed Approach
Change two Django session settings via Authentik environment variables:
1.**`AUTHENTIK_SESSION_COOKIE_AGE` = 604800** (7 days) — extends session cookie lifetime from 24h to 7 days
2.**`AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` = false** — prevents session cookie from being cleared on browser close
This keeps users logged in for up to 7 days with normal browser use (close/reopen, daily usage). The session still expires after 7 days of inactivity (`SESSION_SAVE_EVERY_REQUEST` stays False).
## Step-by-step Plan
### Step 1: Add environment variables to Docker compose
Edit `/opt/docker-compose.yml` on the VPS to add these env vars to the `authentik-server` service:
```yaml
authentik-server:
environment:
# ... existing vars ...
AUTHENTIK_SESSION_COOKIE_AGE:"604800"# 7 days (was 86400 / 24h)
AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE:"false"# was true
```
Note: the Authentik config system uses `__` (double underscore) for nesting. The env vars map to the Django settings via the config YAML path. The correct Authentik env var for `SESSION_COOKIE_AGE` would be `AUTHENTIK_SESSION__COOKIE_AGE` if it goes through the config system, or just `SESSION_COOKIE_AGE` if it's passed directly. Need to verify the exact variable name Authentik expects.
### Step 2: Verify variable naming
Check the Authentik config YAML (`/authentik/lib/default.yml` inside the container) to confirm the exact env var name mapping. Authentik uses a custom config layer that maps env vars to settings.
**Alternative if env vars don't work:** Some Authentik settings need to be set via the admin UI (under System Settings or Tenant settings). The Django session settings might need to be configured differently in this version.
3.**24-hour test** — Check session is still alive after 24h of normal use
## Risks, Tradeoffs, and Open Questions
| Risk | Mitigation |
|------|------------|
| Env var names don't match Authentik's config schema | First verify in the container's `default.yml` config file |
| 7-day persistent cookie is a security concern (stolen cookie = 7 days of access) | This is the same risk as any "Remember Me" feature on any web app. The tradeoff is convenience vs. security. |
| The proxy cookie (`authentik_proxy_*`) may still have its own 24h limit | That's managed separately via the OAuth2 provider's `access_token_validity` setting. If we also want to extend that, we can update `hubris-forward-auth` provider's `access_token_validity` from `hours=24` to `days=7`. |
| `SESSION_COOKIE_SECURE = False` | Should be `True` since Authentik is served behind HTTPS. However, the forward-auth subrequest from Caddy to the outpost is HTTP internally (`http://127.0.0.1:8099`), so `False` may be intentional for the outpost check. |
## Open Questions
1.**What environment variable name does Authentik use for Django session settings?** Need to check `default.yml`. The config layer may use `AUTHENTIK_SESSION__COOKIE_AGE` (double underscore) or the raw Django setting name.
2.**Should we also extend the proxy token validity?** The `hubris-forward-auth` provider has `access_token_validity = hours=24`. If we want users to not need re-login for more than 24h, we should also bump this to match the session cookie age.
3.**Which specific service triggers the most login prompts?** The forward-auth (Caddy-gated) services use proxy cookies. OAuth2 services (Gitea, Jellyfin) use the Django session. Understanding which one the user is hitting most could narrow the fix scope.
Restore all `*.hubris.network` services that went offline when the Caddyfile on LXC 121 was truncated to only 3 photo-related site blocks, and implement automated safeguards to prevent this class of outage from recurring.
## Root cause
The Caddyfile at `/etc/caddy/Caddyfile` on LXC 121 was manually edited locally (not via the `dtoro/caddy-conf` git repo), overwriting ~260 lines (30+ site blocks + forward-auth infrastructure) with only 43 lines covering `photos.hubris.network`, `prism.hubris.network`, and a manually-added `photos2.hubris.network`.
**Evidence:**
-`git diff HEAD -- Caddyfile` shows `+3 / -159` lines diff — all other blocks deleted
- Git reflog shows HEAD at `32575ce` (`fix: sab... port 8081→8082`), but working tree diverges
- Deploy webhook log: Jun 06 12:39 — `deploy failed: git pull` (dirty tree blocks merge)
- Backup file `Caddyfile.bak.1780263919` (225 lines) confirms the full original was intact before truncation
-`origin/master` at `1b977aa` is the authoritative source — 260 lines, all blocks present
**Why "third time this week":**
| Incident | Date | Cause |
|---|---|---|
| 1 | Jun 02 | DHCP IP drift — paperless (130→243), HAOS (101→241) |
| 2 | Jun 05 | More DHCP drift — apps (205), mule-images (136 overridden by dhclient) |
| 3 | Jun 06 | **Caddyfile truncated** — unrelated to IPs, much worse |
The Caddyfile truncation is the most severe: it took down **all LAN services** except `photos.hubris.network` and `auth.hubris.network` (VPS-hosted).
## Immediate fix
### Step 1: Restore Caddyfile from origin/master and reload
On LXC 121:
```bash
cd /etc/caddy
# Stash any local changes
git stash
# Reset to origin/master
git checkout --force origin/master -- Caddyfile
# Caddyfile now has all 30+ sites
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy
```
This restores all service blocks including: media, git, paperless, books, home, cloud, matrix, proxmox, docker, jellyseerr, qbit, sab, blog, auth, artifacto, plato, zimaos, mcp, secrets, sso + authentik forward-auth infrastructure.
### Step 2: Add `photos2.hubris.network` via git (if still needed)
The `photos2.hubris.network` block was manually added locally and is NOT in origin/master. If the user wants to keep it, submit a PR/commit to the `dtoro/caddy-conf` repo.
### Step 3: Verify
- From any LAN/mesh client: `curl -sk https://media.hubris.network/` → 200
- Run `bash /opt/homelab-context/scripts/check-caddy-backends.sh` from hubris → all targets reachable
### Layer 5: DNS sync cron (fix already-deployed sync)
The `dns-sync.py` on LXC 107 at `/opt/dns-sync/sync.py` is installed but has **no crontab** — the sync never runs automatically. The NetBird managed DNS zone has drifted from Technitium. Add a systemd timer or crontab:
| `cronjob` in Hermes | Schedule `check-caddy-backends.sh` |
| `/etc/cron.d/dns-sync` on LXC 107 | New — add dns-sync cron |
## Verification
1. All `*.hubris.network` URLs load from mac-mini: `media`, `git`, `paperless`, `cloud`, `home`, `proxmox`, etc.
2.`check-caddy-backends.sh` exits 0 on hubris
3.`systemctl status caddy` shows active on LXC 121
4.`dns-sync` runs and writes to `/var/log/dns-sync.log`
## Risks / Tradeoffs
- **Restoring from origin/master overwrites photos2.hubris.network** — recreate it via proper git commit
- **Caddy staging ACME certs for prism/photos2**: The `tls dns ionos` directive uses staging env (`acme-staging-v02.api.letsencrypt.org`), which fails DNS propagation check (VPS port 53 unreachable from LXC). Once restored, these two subdomains will have the same issue. Move them to production IONOS DNS-01 by removing the staging CA directive or setting the correct `acme_issuer` in Caddyfile.
- **Dirty-tree stash could lose edits** — mitigated by `git stash push --message` + backup file creation before stash
## Open questions
1. Keep `photos2.hubris.network`? If yes, add via proper git push.
2.`prism.hubris.network` and `photos2` certs fail on staging ACME — set production `acme_issuer` in Caddyfile?
3. Should `check-caddy-backends.sh` run as a homelab cron job or as a regular cron on LXC 121?
strong becomes the media/library powerhouse. hubris becomes a lean core-infra
node (DNS, auth, git, docs, caddy, HA).
---
---
## Risk register
| Risk | Impact | Mitigation |
|------|--------|------------|
| NFS latency for library reads (jellyfin, arriman) | Media playback stutter, slow downloads | Test iperf between strong↔hubris first. If 2.5G link, NFS throughput is fine (~1 Gbit/s). |
| GPU passthrough on strong (680M vs 760M) | Transcode quality/compat differences | Both are AMD VAAPI — same driver stack. Test `vainfo` inside LXC before going live. |
| Caddy backend IP churn | Service outage if IP wrong | Update Caddyfile in git repo (caddy-conf), test each route before destroying old LXC. |
| vzdump/restore downtime | Service unavailable during migration | Schedule off-hours. Use rsync for large rootfs (120's 100G) to minimize freeze window. |
| 2-node quorum still fragile | If hubris goes down, strong /etc/pve goes read-only | Guests keep running. Add QDevice as follow-up. |
| Library data integrity during NFS transition | Permission drift | NFS `all_squash,anonuid=33,anongid=10000` matches existing LXC 102 config. Verify with `ls -la /mnt/library` after mount. |
---
## Open questions for operator
1. **Internal bridge on strong**: proceed with `vmbr1` on `192.168.8.3/24`
(Option A), or use `192.168.178.x` guest IPs (Option B)?
2. **Migration method**: `vzdump`/restore (clean, downtime) vs `rsync` rootfs
(faster for large disks, needs manual config copy)?
3. **Phase 1 priority**: move elementsynapse + house first (quick wins), or
go straight to Phase 2 (mule-images/jellyfin/arriman) for maximum relief?
4. **Should we add a QDevice now** before moving anything, to protect
management plane during the migration?
---
## Changelog
### 2026-07-05 — Phase 2d complete (grimmory migrated; media NFS to zimaos)
grimmory (130) → 192.168.8.247 on strong. Rsync'd /books (2.6G) to ludo-lvm.
LXC 102 (nfs-export) now mounts strong's NFS at /mnt/media and exports it as
a second share alongside /mnt/library. Zimaos mounts both: /media/library
(hubris user-generated) and /media/media (strong media+books).
Media server: serves the movies / TV / anime / music libraries from `/mnt/media_local` to LAN clients. Hardware transcoding via AMD Radeon 680M + RX 7600 VAAPI. Authentik SSO via OIDC.
## At a glance
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **IP:** `192.168.8.246`
- **Privilege:** privileged (recreated on strong as priv)
include `OidScopes` in the provider config (even if empty array).
- **`SchemeOverride: "https"` is required** — without it, the plugin generates
`http://` redirect URIs (from the internal HTTP listener). Authentik rejects
them with "Redirect URI Error".
- **SSO button JS is not served by the plugin** — the `__plugin/SSO-Auth.js`
endpoint returns 404 on Jellyfin 10.11.x when the plugin is installed
manually (not via Jellyfin's plugin manager). The `sso-inject.js` workaround
in `index.html` is the fallback.
- **No Caddy forward-auth gate** — the SSO plugin's OIDC redirect flow is
incompatible with Caddy's `import authentik` forward-auth. If both are
enabled, the forward-auth intercepts the OIDC callback and breaks the flow.
Use one or the other, not both. SSO plugin (OIDC redirect) is preferred.
- **API key for setup** — a temp API key can be inserted directly into the
`ApiKeys` SQLite table for automated configuration:
```sql
INSERT INTO ApiKeys VALUES (1, '2026-07-04', '2026-07-04', 'setup', 'jf-setup-key-...');
```
## Permissions
Member of the [media GID 10000](../infrastructure/media-permissions.md) standard. Service user `jellyfin` is in the `media` group inside the container; `/mnt/media_local` on strong's ludo-lvm is owned `root:media` with mode `2775`.
- **Database was wiped** during cache relocation attempt — no LVM snapshot
existed. All watch states, user accounts, and library configs lost.
Libraries re-added via setup wizard.
### 2026-04-28 — wiki entry created
Initial documentation. No config changes.
### 2026-04-20 — joined the `media` GID 10000 standard
Idmap block applied; in-container `media` group at GID 10000 mapped to host GID 10000. See [media permissions](../infrastructure/media-permissions.md). Config backup: `/root/101.conf.bak.*`.
Dedicated, single-purpose LXC that re-exports `/mnt/library` over NFSv4 to clients that can't use the host's PVE virtiofs path — currently only [100-zimaos](../vms/100-zimaos.md), which ships a kernel without virtiofs support.
## At a glance
- **Hostname:** `nfs-export`
- **IP:** `192.168.8.200` (static; LAN-only, no Caddy in front because NFS is L4)
- **LAN DNS:** `nfs-export.hubris.network` → `192.168.8.200` (direct, no Caddy)
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, apps)
## What it does
```
/mnt/library (host ext4 on nvme1n1)
│
├── bind-mounted into 7 other LXCs (jellyfin, paperless, …)
└── bind-mounted into LXC 102
│
└── nfs-kernel-server exports /mnt/library
│
└── consumed by VM 100 (ZimaOS)
```
Same inodes, same page cache. The NFS server is just one more access path on top of a tree that 8 other consumers already share — see [media permissions](../infrastructure/media-permissions.md).
Initially started as `ro`; promoted to `rw` on 2026-05-14 after the Files-UI evaluation confirmed (a) the library renders correctly as a folder under `/DATA`, (b) thumbnails are generated, (c) the squash works — a write from ZimaOS appears on `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention used by Nextcloud and mule-images.
### Guardrails (in order of importance)
1.**`all_squash,anonuid=33,anongid=10000`.** Every write from ZimaOS records on disk as `www-data:media` (uid 33, gid 10000), the same identity Nextcloud and mule-images use. Keeps the existing tree convention from drifting. See [`project_media_perms`](../infrastructure/media-permissions.md).
3.**Subnet restriction `192.168.8.0/24`.** No public/mesh access; LAN only.
4.**`no_subtree_check`** + **`sec=sys`** — standard performance/auth pair for a homelab.
5.**No `crossmnt`** because `/mnt/library` is a single ext4 filesystem on the host (no nested mounts to traverse).
### What we're *not* doing yet
- No per-subdir export. ZimaOS sees the whole tree; access is controlled by filesystem permissions (`drwxr-x---` private dirs like `documents/`, `notes/`, `heaper/` aren't readable by `media` group, so ZimaOS-as-squashed-uid won't see them either).
- No write-back. Until we promote to `rw`, ZimaOS can't write — eliminates the lock-domain split concern between NFS clients (NLM/v4) and local LXCs (POSIX flock).
- No Authentik / forward-auth. NFS doesn't sit behind HTTP, so the standard caddy+authentik path doesn't apply. Subnet ACL is the only auth.
## Why this LXC exists (vs serving from host)
We considered three options before building this:
| Option | Outcome |
|---|---|
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. |
| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
Rationale lives in [the install plan](https://git.hubris.network/dtoro/Homelab-Docs) `/root/.claude/plans/i-wannt-you-to-nifty-muffin.md` on hubris.
- **Restart cleanly:** `pct restart 102` (ZimaOS will retry the mount via `nofail`)
- **Destroy + rebuild:** `pct stop 102 && pct destroy 102 --purge` — reversible in seconds; only ZimaOS notices
## Open items
- Consider tightening the export to subdirs (e.g. `movies`, `tv`, `music`, `audiobooks`, `books`, `images`, `podcasts`, `roms`) if you don't want ZimaOS reachable into `documents/`, `notes/`, `heaper/`, etc. — though those private subdirs are already invisible to `media`-group perms.
- ZimaOS architecture finding: the Drives panel only enumerates physical/block devices via `GET /v2/local_storage/storages` (read-only API, no POST). Network shares **cannot** appear as Drives — they show up as folders in Files. This is intentional in CasaOS's design; don't try to work around it. Library-as-folder is the supported model.
- Consider adding Samba to this LXC if a future Mac/iOS client needs SMB on the same tree — same LXC, no host changes.
- No PBS backup (no PBS configured on hubris); the container is fully described in this page + `pct config 102`, rebuild from scratch in <2 min if lost.
## Related
- [100-zimaos](../vms/100-zimaos.md) — the only consumer today
- [media permissions](../infrastructure/media-permissions.md) — uid 33 / gid 10000 standard
- [DNS](../infrastructure/dns.md) — `nfs-export.hubris.network` entry (direct, no Caddy)
## Changelog
### 2026-05-14 — Promoted to rw; squash behaviour verified
After ZimaOS Files UI evaluation passed (library renders as folder under `/DATA`, thumbnails work, ZimaOS Drives panel ignores NFS by design), flipped export to `rw`. Tested: writing `/DATA/library/.zimaos-rw-test` from ZimaOS appears on hubris's `/mnt/library` owned `www-data:media` (uid 33, gid 10000), confirming `all_squash,anonuid=33,anongid=10000` works as designed. Also discovered the dead end: ZimaOS's `GET /v2/local_storage/storages` is the source of the Drives panel; it returns only physical storage and rejects POST/PUT — network shares cannot be promoted to Drives.
### 2026-05-14 — LXC built; NFS export live (read-only)
Privileged Debian 13 container created with bind-mount `/mnt/library`. `nfs-kernel-server` installed and enabled; export `/mnt/library` to `192.168.8.0/24` with `ro,all_squash,anonuid=33,anongid=10000`. Smoke-tested from hubris host (mounted, listed library tree, confirmed RO). `nfs-export.hubris.network` added to LXC 124 dnsmasq.
Behind [Authentik forward-auth](106-auth-outpost.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
## Storage
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
## Known issues
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- [Authentik](106-auth-outpost.md)
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-21 — wired into Authentik forward-auth
Cookie-domain-level proxy provider, embedded outpost binding. `/api/*` bypass added in Caddy site block to keep mobile clients working.
| `gitea-deploy-webhook` | `127.0.0.1:9797` | Receives push events for the customizations repo |
## Customizations
`/var/lib/gitea/custom` is a checkout of [`dtoro/gitea-customizations`](https://git.hubris.network/dtoro/gitea-customizations).
-`templates/custom/footer.tmpl` injects:
- **Online3DViewer** v0.18.0 (vendored under `public/assets/o3dv/`) for inline 3D preview of CAD raw-file links: 3dm, 3ds, 3mf, amf, bim, brep, dae, fbx, fcstd, glb, gltf, ifc, igs, iges, stp, step, stl, obj, off, ply, wrl.
- **PlantUML client JS** (vanilla, vendored from `gitea.com/davidsvantesson/plantuml-code-highlight`). Browser calls `https://git.hubris.network/_plantuml/img/<encoded>` — caddy `handle_path /_plantuml/*` rewrites and proxies to the PlantUML server on [apps (105)](105-apps.md) `:8079`.
-`home.tmpl` is the anonymous landing page (Homesauce hero + sign-in/explore buttons).
- O3DV binary assets are *not* tracked — `deploy.sh` fetches them on first run.
## Auto-deploy
Push to `dtoro/gitea-customizations` master → gitea webhook → `http://127.0.0.1:9797/deploy` (loopback) → `gitea-deploy-webhook.service` validates HMAC against `/etc/gitea-deploy/secret` → runs `/opt/gitea-customizations/deploy.sh`. Then `systemctl restart gitea` if templates changed.
`app.ini``[webhook] ALLOWED_HOST_LIST` includes `127.0.0.1`, `192.168.8.175`, `192.168.8.205`, `192.168.8.230`, `192.168.8.136`, `192.168.8.77` — every webhook target across the lab. **Don't strip these when editing app.ini.** See [auto-deploy](../infrastructure/auto-deploy.md) for the full list.
Git creds for the deploy service: `/etc/gitea-deploy/git-credentials` (mode 600), referenced via `credential.helper = store --file=/etc/gitea-deploy/git-credentials` in the repo's `.git/config` (because `ProtectHome=true` blocks `/root`).
## DNS workaround
LXC has `/etc/hosts` override mapping `auth.hubris.network → 192.168.8.175` (router DNS resolves the public IONOS A record, which is a dead end for LAN clients — see [DNS](../infrastructure/dns.md)). Belt-and-suspenders systemd unit `hubris-hosts-override.service` re-applies it idempotently. Override line is **outside** the `# --- BEGIN PVE ---` markers (Proxmox rewrites inside).
- [PlantUML server lives on apps (105)](105-apps.md)
## Changelog
### 2026-06-24 — terminalito deploy webhook (id 12)
Push webhook on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` ([trmnl (128)](128-trmnl.md)); `app.ini``ALLOWED_HOST_LIST` extended with `192.168.8.211`. See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-22 — `app.ini` ALLOWED_HOST_LIST extended for Artifacto webhook
Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md).
### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added
For OIDC integration with [authentik (124)](106-auth-outpost.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`.
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20. Booklore migrated to [grimmory (130)](130-grimmory.md) on 2026-06-29.
| `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). |
| `mcp.hubris.network` | homelab-mcp | `:9810` | MCP server over SSE. No SSO — agent-facing, gated at source-IP layer. See [homelab-context](../infrastructure/homelab-context.md). |
Self-hosted HTML-artifact share (Go + SQLite). Repo `dtoro/Artifacto` is the working tree itself — `git push origin main` triggers rebuild.
- Compose: tracked `docker-compose.yml` binds `./data:/data` and `127.0.0.1:3100:3000`. `docker-compose.override.yml` uses the `!override` YAML tag on `volumes:` and `ports:` to redirect `data` to `/mnt/library/documents/artifacts`.
- Data dir: `/mnt/library/documents/artifacts` — owner **UID 10001 : GID 10001** (matches the `app` user baked into the Dockerfile), mode 750. Parent `/mnt/library/documents` stays `www-data:www-data 750`; bind targets the child directly so no traversal needed.
-`.env` (untracked) holds `ADMIN_PASSWORD`, `SESSION_SECRET` (`openssl rand -hex 32`), `BASE_URL=https://artifacto.hubris.network`, `SSO_GATEWAY_SECRET` (must match `ARTIFACTO_GATEWAY_SECRET` in `/etc/caddy/caddy.env` on [caddy](121-caddy.md)). Don't `git checkout .env`.
- Auth: Authentik forward-auth on the admin surface (`/`, `/login`, `/a/*`, `/api/*`). `X-Authentik-Username` is honored only when the `X-Artifacto-Gateway` header matches `SSO_GATEWAY_SECRET` — prevents direct LAN connections from spoofing identity.
- Public exposure: VPS traefik gates path scope to share URLs; admin surface returns 404 at the edge. See [ingress](../infrastructure/ingress.md).
#### Auto-deploy pipeline (Artifacto)
Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webhook.py`, secret in `/etc/artifacto-deploy/secret`, systemd unit `artifacto-deploy-webhook.service`. Listens on `0.0.0.0:9798`, validates `X-Gitea-Signature` HMAC-SHA256, filters to `refs/heads/main`, runs deploy in a daemon thread (returns 202 immediately). Gitea webhook id 7 on `dtoro/Artifacto`.
- Slug length bumped 5 → 8 chars (in `internal/store/artifacts.go:41`) once the endpoint went public.
### PlantUML (`/opt/plantuml/`)
`plantuml/plantuml-server:jetty` listening on `192.168.8.205:8079`, context path `/plantuml` (`BASE_URL=plantuml`). Reached only via [gitea](104-gitea.md)'s caddy `handle_path /_plantuml/*` rewrite — no DNS record of its own. Same-origin from gitea so the embedded markdown previewer can call it without CORS.
### Portainer
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
(Tailscale), `192.168.8.0/24` (LAN). Tighten if the LAN gets
untrusted devices.
#### Auto-deploy pipeline (secrets-issuance)
Receiver at `/opt/secrets-issuance/secrets-issuance/deploy/webhook/`,
port `9821`, secret in `/etc/secrets-issuance-deploy/secret`, unit
`secrets-issuance-deploy.service`. Gitea webhook id 11.
### WriteFreely (`/opt/writefreely-docker/`)
Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.hubris.network`, scope `openid profile email`, `map_user_id = sub`. Redirect URI in Authentik: `https://blog.hubris.network/oauth/callback/generic`. Container has `extra_hosts: auth.hubris.network:192.168.8.175`. First-time users with a local WF account must connect SSO via Customize/Settings → Linked Accounts.
## Permissions
- LXC is privileged → in-container UID == host UID.
- Member of [media GID 10000](../infrastructure/media-permissions.md). Adding `media` to the LXC alone is *not* enough for inner Docker containers — each container needs `--group-add 10000` or `user: "<uid>:10000"` to write into `/mnt/library`.
## Related
- [Gitea (104)](104-gitea.md) — uses the PlantUML server
### 2026-06-29 — Booklore migrated to Grimmory on LXC 130
Booklore stack removed from Portainer. MariaDB dump taken first, then restored into [grimmory (130)](130-grimmory.md)'s fresh MariaDB. `books.hubris.network` Caddy backend updated to `192.168.8.213:6060`. Authentik OIDC provider updated to Public client type (PKCE) for Grimmory compatibility.
### 2026-05-20 — homelab-mcp + secrets-issuance live
Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md):
`homelab-mcp.service` on `:9810` (MCP read+management surface) and
`secrets-issuance.service` on `:9820` (per-client age-key provisioning).
Caddy fronts both with Let's Encrypt; new vhosts on
[caddy](121-caddy.md), split-horizon DNS entries on
Repo `dtoro/Artifacto`. Compose, data dir, `.env`, webhook (id 7), Caddyfile site block, gateway-secret pattern, public-internet share exposure on the VPS — all set up that day. Slug length bumped 5 → 8 for brute-force resistance.
### 2026-04-22 — Booklore mariadb data wiped (lesson, not a feature)
`docker compose up -d` from the host shell on a Portainer-managed stack recreated services and re-initialized the bind-mounted `./mariadb/config`. Recovered by reseeding from snapshot. Rule recorded: edit Portainer-managed stacks via the Portainer UI.
### 2026-04-22 — Portainer wired into Authentik (native OAuth2)
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](106-auth-outpost.md)
## Role
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
[Caddy (121)](121-caddy.md) `(authentik)` snippet `reverse_proxy`/`forward_auth` → `http://192.168.8.6:9000` (was `192.168.8.180:9000` on LXC 124). Tracked in `dtoro/caddy-conf`.
## `sso.hubris.network` — the callback domain (critical)
Domain-level (`forward_domain`) providers redirect the browser to `{external_host}/outpost.goauthentik.io/callback` after login. On LXC 124 that `external_host` was `auth.hubris.network`, which resolved (LAN) to Caddy → the 124 outpost — same box, so it worked. Post-migration `auth.hubris.network` points to the **VPS core**, so the callback hit the wrong outpost → **400 / redirect-uri mismatch**.
Fix: the LAN outpost gets its **own** domain.
- DNS: `sso.hubris.network → 192.168.8.175` (Caddy). *(in dnsmasq today — must be carried into Technitium in DNS Phase 2.)*
- Caddy: site `sso.hubris.network { tls dns ionos; reverse_proxy 192.168.8.6:9000 }` (in `caddy-conf`).
- Authentik: `hubris-forward-auth` and `Provider for Torrent` have `external_host=https://sso.hubris.network`; their `redirect_uris` must match (`set_oauth_defaults()` regenerates them from `external_host` — changing `external_host` alone does NOT update them).
-`Provider for Artifacto` is `forward_single` on its own domain, so it was unaffected.
**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
## Related
- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only)
### 2026-06-06 — Authentik session lifetime extended to 30 days
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md).
### 2026-06-01 — created; forward-auth cut over from LXC 124
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).
- **Created:** 2026-06-01, Debian 13. Its own resolver is `1.1.1.1` (no circular dependency on the DNS it serves).
## Role
Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recursive forwarding (`1.1.1.1`, `8.8.8.8`) for everything else. Technitium runs in Docker (`technitium/dns-server:latest`, `network_mode: host`), web console on `:5380`.
## The `hubris.network` zone
- Specific A overrides: app names → `192.168.8.175` (Caddy), `nfs-export → 192.168.8.200`, `auth/sso/...` as needed.
- **Wildcard** `*.hubris.network → 82.165.190.79` — mirrors the public IONOS wildcard so undefined names (e.g. `netbird`) resolve to the VPS, matching public behaviour.
- **MX / SPF-TXT / CAA replicated from public** so an authoritative zone doesn't shadow `hubris.network` email/cert records.
## Config / access
-`/opt/technitium/docker-compose.yml`; admin password in `/opt/technitium/admin_password.txt` (mode 600 — sops-encrypt in Phase 5).
- API: `http://192.168.8.2:5380/api/...` (token via `/api/user/login`). Zone was built via the API.
## Who points here
- **NetBird mesh peers:** resolve via the **NetBird managed DNS zone**, kept in sync *from* this Technitium (see dns-sync below). The `home-lab-dns` nameserver group (`→ 192.168.8.2`) is a thin fallback forwarder.
- **Homelab DHCP clients:** Technitium's own DHCP scope hands out `192.168.8.2` as the DNS server for `192.168.8.x` leases (see DHCP section below).
- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field.
## dns-sync (Technitium = authoring source)
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../../../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
## DHCP
Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02):
Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101–.239`) are excluded from the pool. Pool narrowed from `.100–.240` to `.241–.254` on 2026-06-03 to eliminate IP conflict risk.
## Related
- [124 — authentik](106-auth-outpost.md) — retired host of the old dnsmasq
- [DNS split-horizon](../infrastructure/dns.md)
- [Mesh](../infrastructure/mesh.md)
## Changelog
### 2026-06-24 — A record `trmnl.hubris.network → 192.168.8.175`
Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)); propagated to the NetBird managed zone by `dns-sync`.
### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
### 2026-06-03 — DHCP pool narrowed to `.241–.254`
Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
### 2026-06-02 — DHCP server enabled; replaces Slate AX DHCP
Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100–.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
### 2026-06-01 — created; replaced dnsmasq on 124
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](106-auth-outpost.md).
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](106-auth-outpost.md#per-app-username-override-pattern-authentik) for the full pattern.
Nextcloud's PHP HTTP client (`OC\Http\Client\DnsPinMiddleware`) calls `dns_get_record()` directly — it bypasses `/etc/hosts`. So the Authentik OIDC discovery URL needs **real DNS** answering with `192.168.8.175` for `auth.hubris.network`. Recipe applied:
- Local `dnsmasq` listening on `127.0.0.1:53`, `/etc/dnsmasq.d/hubris-internal.conf` with `address=/auth.hubris.network/192.168.8.175`, `server=192.168.8.1`, `server=1.1.1.1`, `interface=lo`, `bind-interfaces`, `no-hosts`, `no-resolv`.
-`pct set 114 --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"` so the LXC starts with the right resolver order.
-`/etc/hosts` override kept as belt-and-suspenders.
Also needs `allow_local_remote_servers=true` in `config.php`.
> Once internal DNS at the router level handles `*.hubris.network`, this LXC-local dnsmasq can go away. See [DNS](../infrastructure/dns.md).
## Storage
Files at `/mnt/library/homecloud`. Owned by Nextcloud's own permission model — **deliberately not on the `media` group**, NC manages it itself and would complain. See [media permissions](../infrastructure/media-permissions.md).
## Web stack (Apache event MPM + php-fpm)
Apache 2.4 with **`mpm_event`** as the MPM and PHP served via **php-fpm 8.4** over the Unix socket `/run/php/php8.4-fpm.sock` (mod_php disabled). Routing to FPM is via the distro's `/etc/apache2/conf-available/php8.4-fpm.conf` (gated by `<IfModule !mod_php8.c>`, so just `a2dismod php8.4` flips Apache onto FPM).
Why this matters: with mod_php every Apache worker is forced onto `mpm_prefork` (one heavy process per connection, full PHP interpreter loaded in each). On FPM + event, Apache workers are ~5–8 MB and async; PHP work is done by a separate FPM pool that keeps opcache hot. Concurrent upload throughput jumps significantly because Apache no longer blocks per-connection on a heavy PHP process.
Follow-up to the FPM/MPM/MariaDB round earlier the same day. After a WiFi-bound Linux client kept seeing ~2 MB/s downloads despite a 152 Mbps speed test (so the WiFi medium wasn't the cap), bumped Apache `KeepAliveTimeout 5 → 60` and `MaxKeepAliveRequests 100 → 1000` so the mirall desktop client reuses one TLS+TCP connection across many sequential file fetches instead of paying handshake + slow-start each time. Paired with TCP BBR + `fq` qdisc enabled on the [hubris host](../hosts/hubris.md#2026-05-13--tcp-bbr--fq-qdisc-enabled), which directly addresses single-stream throughput on jittery links. Server-side loopback baseline post-BBR is ~404 MB/s for a single 23 MB CR2 download via Caddy. Client-side diagnosis pending (curl single-stream + parallel tests) to determine if the remaining gap is mirall config (bandwidth limit / `parallelNetworkRequests`) or path shaping.
Investigated upload/download speed. Bottlenecks were inside this LXC — LAN (33.9 Gbit/s caddy→NC iperf3) and `/mnt/library` NVMe (714 MB/s sustained) were not the limit. Changes:
- **Apache mod_php → php-fpm 8.4** (`a2dismod php8.4`). Flips PHP from in-process to FPM-over-Unix-socket. Apache workers now ~5–8 MB instead of ~80 MB each.
- **MPM prefork → event** (`a2dismod mpm_prefork && a2enmod mpm_event`). Async I/O, far better concurrency for parallel uploads.
- **Apache mod_reqtimeout**: body trickle relaxed from `body=10,MinRate=500` → `body=20,MinRate=100` so slow-WAN uploads aren't killed.
- **PHP-FPM ini** synced to match what the apache2/cli inis already had (FPM was stuck at 2M/8M because FPM wasn't serving traffic before, but now it does).
Backups: `/etc/php/8.4/fpm/php.ini.bak.20260513`, `/etc/apache2/mods-available/reqtimeout.conf.bak.20260513`, `/etc/php/8.4/mods-available/opcache.ini.bak.20260513`. MariaDB tuning is a new file (rollback = `rm`).
Verified: `apache2ctl configtest` Syntax OK, `occ status` clean, `occ setupchecks` shows ✓ on memory_limit / output_buffering / DB indices, `status.php` returns 200 in <40 ms via Caddy with HTTP/2.
### 2026-05-10 — PHP upload limits raised
Apache + CLI php.ini bumped from distro defaults (`upload_max_filesize=2M`, `post_max_size=8M`, `memory_limit=128M`, `max_execution_time=30`) to NC-recommended (`16G/16G/512M/3600`). Backups at `/etc/php/8.4/{apache2,cli}/php.ini.bak-20260510-*`. Was causing slow/failing uploads via web UI.
[mulita (120)](120-mule-images.md) now reads photos from `/nextcloud-users/<nc_user>/files/...` and writes mutations back via WebDAV. Pre-migration DB dump kept at `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump`.
Local `dnsmasq` added on this LXC because Guzzle bypasses `/etc/hosts`. Resolver order set to `127.0.0.1 192.168.8.1 1.1.1.1`. Username override `dtoro → admin`.
- **`server_name`:** `avispero` (intentionally different from the public URL).
- **Database:** Synapse runs on **SQLite** (`/var/lib/matrix-synapse/homeserver.db`, ~90 MiB). PostgreSQL on `127.0.0.1:5432` is dedicated to the mautrix bridge databases (one per bridge), not Synapse itself.
All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name>/` — no systemd unit, `restart: unless-stopped`. Their appservice registration YAMLs live in `/etc/matrix-synapse/` and are listed in `homeserver.yaml` under `app_service_config_files`. If a bridge daemon is down, Synapse logs `push_bulk … ConnectionRefusedError` and retries every 512 s; restart the bridge with `cd /root/mautrix-<name> && docker compose up -d`.
## Operational notes
- Rootfs is on `pve` LVM-thin, grow with `pct resize 118 rootfs +<N>G` from the host — resize2fs runs automatically.
-`/var/lib/matrix-synapse/media_store` is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.
## Known issues
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- ~~[claudio-bot (123)](archive/123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
**Symptom:** Matrix was down. Caddy at `192.168.8.175` couldn't reach `192.168.8.239:8008` — the LXC was actually at `192.168.8.244` because the guest-side dhclient had overridden the PVE-assigned static IP.
**Root cause:** During the 2026-06-02 static-IP migration, `pct set 118 --net0 ... ip=192.168.8.239/24` was applied to the Proxmox config, but the internal `/etc/network/interfaces` still had `iface eth0 inet dhcp`. On every DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool.
- Verified: `curl http://192.168.8.239:8008` returns 302 from Caddy's LXC
**Prevention:** The `check-caddy-backends.sh` cron on hubris now runs every 10 minutes, which would have caught this drift within 10 minutes of occurrence.
### 2026-05-15 — phantom-notification cleanup for `@admin`
After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in `event_push_actions` that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:
1.**8 of 12 affected rooms** had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.
2.**The noisiest room** (a Signal-bridged DM) was flooded with `m.room.redaction` events from `@signalbot:avispero` (disappearing messages being deleted). Synapse counts redactions as notifications by default, but Element X doesn't render them — so they were invisible "ghosts".
Fix (after stopping Synapse and backing up `homeserver.db`):
-`DELETE FROM event_push_actions WHERE stream_ordering <= max(receipt_stream)` per (user, room) — clears receipt-superseded rows.
-`DELETE FROM event_push_actions WHERE event_id IN (SELECT event_id FROM events WHERE type='m.room.redaction')` — removes redaction-driven notifications.
- Rebuild `event_push_summary` from the remaining rows (preserve `thread_id='main'` to satisfy the NOT-NULL check constraint).
Result: 125 → 4 (legitimate WhatsApp unreads).
Then, to stop this recurring, an **override push-rule** was installed for `@admin` via the client API:
```
PUT /_matrix/client/v3/pushrules/global/override/suppress_redactions
With empty `actions`, all `m.room.redaction` events are silently ignored by the push system. Inspect/remove via the same endpoint. (Note: rule ids beginning with `.` are reserved for server defaults — custom ids cannot start with a dot.)
Element/Synapse was unreliable for ~2 days: messages dropping, sync requests stalling, mautrix bridges (slack/signal/meta/linkedin/whatsapp) all exited around 2026-05-13. Root cause: rootfs reached 100% (19 MB free of 7.8 G); Synapse logs showed `OSError: [Errno 28] No space left on device` during flush. Fix: `pct resize 118 rootfs +8G` (host `pve` VG had plenty of headroom), `journalctl --vacuum-size=50M` to recover ~200 M, `systemctl restart matrix-synapse`, then `docker compose up -d` in each `/root/mautrix-*` directory. Also expanded this wiki page to document the bridges, synapse-admin, and the actual DB backend (Postgres, not the SQLite leftover).
LXC has `www-data` in the [media GID 10000 group](../infrastructure/media-permissions.md). The `/mnt/library/sophia` subtree itself is single-container and not on the setgid shared group.
Hosts `mule-image` — the photos app at `photos.hubris.network`. PhotoPrism + Go sidecar + SvelteKit, replacing the legacy FastAPI/Celery stack as of 2026-05-22 (see Changelog). Auto-deploys from `dtoro/mule-image` on `git push origin main`.
`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose stack: base `docker-compose.yml` + LAN-port-rebind `docker-compose.override.yml` (untracked) + VA-API `docker-compose.gpu.yml`. The SvelteKit frontend is built on the host and served as a static bundle by nginx — no `vite dev` in prod.
| pp-sidecar | `0.0.0.0:8000` | Go service (file rename / folder ops / heap convert / dup scan / per-photo marks); also reconciles `USER_BASEPATHS` into PhotoPrism's `auth_users` every 60 s |
| nginx (host process) | `0.0.0.0:3000` | Serves the SvelteKit static bundle from `web/build/` |
**Per-user scoping** is driven by `USER_BASEPATHS=admin:admin/files, muli:muli/files` in `.env`. Sidecar applies it to PhotoPrism's `auth_users` table on boot + every 60s, `mkdir -p`s each target so PhotoPrism's ACL filter has somewhere to point.
`.env` is **untracked** — `git reset --hard` won't touch it, but `git clean -fdx` would. Holds PP_*, MariaDB passwords, SIDECAR_DB_PASSWORD, OIDC_* (existing `mule-image` Authentik app, redirect URI now `/api/v1/oidc/redirect`), `USER_BASEPATHS`, and `PP_FFMPEG_ENCODER=vaapi`.
`docker-compose.override.yml` is also untracked — it `!override`s the upstream `127.0.0.1:port` mappings to `0.0.0.0:port` so cross-host Caddy on [LXC 121](121-caddy.md) can reach pp-app + sidecar.
## Library access
PhotoPrism reads the library **directly off the bind-mounted filesystem** — no Nextcloud webhook integration in the new stack. The base path is `/mnt/library/homecloud` and per-user scoping comes from `USER_BASEPATHS` (see Stack section above).
-`dtoro` is mapped to NC user `admin` → `/mnt/library/homecloud/admin/files/`
-`muli` is mapped to NC user `muli` → `/mnt/library/homecloud/muli/files/`
PhotoPrism's container user is `33:10000` (`www-data:media`), matching the host ownership of the NC tree. The legacy `oc_webhook_listeners` rows + `NEXTCLOUD_WEBHOOK_SECRET` are gone — when NC writes via WebDAV (still its own primary surface), PhotoPrism picks up the new file on its next index pass.
## Authentication
OIDC via Authentik. App slug `mule-image`, redirect URI `https://photos.hubris.network/api/v1/oidc/redirect` (PhotoPrism's auto-derived path; PhotoPrism builds it from `PHOTOPRISM_SITE_URL`). `OIDC_REGISTER=true` auto-creates a PhotoPrism user at role `user` on first SSO login. The sidecar's basepath reconciler then assigns their scoped folder.
The `mule-image` Authentik app's redirect URI was migrated from the legacy FastAPI `/api/v1/auth/oidc/callback` path on 2026-05-22 — same client ID/secret were reused. The separate `mule-photos-new` Authentik app was deleted in the same operation.
## Auto-deploy
Push to `dtoro/mule-image``main` → gitea webhook → `http://192.168.8.136:9797/deploy` → `mule-deploy-webhook.service`:
- Validates HMAC against `/etc/mule-deploy/secret`
- Filters to `refs/heads/main`
- Runs `/opt/mule-deploy/deploy.sh` in a daemon thread (returns 202 immediately — docker builds exceed gitea's request timeout)
-`git fetch && git reset --hard origin/main`, force-recreates **only the sidecar** image (so PhotoPrism's `Session` HMAC key in `pp/storage/config/hub.yml` doesn't rotate and invalidate every in-flight OIDC state cookie), reconciles `pp-app` + `mariadb` in place, `cd web && npm ci && npm run build`, `systemctl reload nginx`.
- Conditionally layers `docker-compose.gpu.yml` when `/dev/dri/renderD128` exists, and always layers `docker-compose.override.yml` (the LAN-port rebind) when present.
Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`. Gitea webhook id 6. `app.ini``ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`.
For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials` (mode 600) — same token as `/etc/caddy-deploy/git-credentials` on [caddy](121-caddy.md).
## Related
- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries
- [Authentik (124)](106-auth-outpost.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Gitea (104)](104-gitea.md)
## Changelog
### 2026-05-22 — Cutover to PhotoPrism stack (Mulimage 2.0)
The `new` branch of `dtoro/mule-image` was merged into `main` as commit
`70dc1b6`. The merge replaces the legacy FastAPI + PostgreSQL + Celery
+ React stack with PhotoPrism + Go sidecar + MariaDB + SvelteKit, lifted
in from the M0 evaluation on (now-destroyed) [LXC 127](#).
Two big architectural shifts in one session, both aimed at killing work mule was doing that NC already does for the same source files.
**Phase 1 — thumbnail proxy (commits `576b0c2`, `28738ac`).** Photos table gains `nextcloud_fileid` (migration `0018`). `GET /api/v1/photos/{id}/thumb/{size}` proxies NC's `/index.php/core/preview` keyed by that fileid, authenticated with the owner's encrypted app password (small=240, medium=640, large=1280). Worker now only writes the `medium` size to `/data/thumbs` (the vision worker still loads it from disk); `small` + `large` no longer touch disk. Disk fallback stays wired for legacy rows and the rare 404 from NC (iPhone JPEGs mis-extensioned as `.DNG` — verified). Existing 17,591 photos backfilled via `backend/scripts/backfill_nextcloud_fileid.py`. Tested with real CR2, real DNG, fake DNG, JPEG, HEIC — all green.
**Range support for `/original` (commit `18dce33`).** Pre-existing bug surfaced by Phase 1 testing: `<video>` couldn't play `.mov` because `FileResponse` was returning 200 with the whole body and no `Accept-Ranges` header, so browsers reported "format not supported." Now parses `Range: bytes=START-END`, returns 206 with `Content-Range`, streams in 1 MB chunks.
**Fix: backfill_gps was the actual CPU drain, not the watcher (commit `d24c64e`).**`_scan_all_source_roots_async` auto-queued `backfill_gps` 30 s after every container boot, which then re-queued `extract_metadata` for every photo with `latitude IS NULL` — ~60k pointless tasks per deploy, pinning worker-light at 180+% CPU for ~30 min. Killed the auto-trigger; manual `POST /api/v1/library/backfill-gps` still works.
**Phase 2 — webhook receiver replaces watchfiles (commits `362fbc6`, `f657e2c`).** New endpoint `POST /api/v1/internal/nc-webhook` (auth via `Authorization: Bearer $NEXTCLOUD_WEBHOOK_SECRET`, `hmac.compare_digest`) handles `NodeCreated|NodeWritten|NodeDeleted|NodeRenamed`. Maps NC's `/admin/files/...` path to the bind-mounted `/nextcloud-users/admin/files/...` and dispatches the same `scan_folder` / `handle_file_deletion` machinery the watcher used. Registered against NC via `backend/scripts/register_nc_webhooks.py` (idempotent: deletes existing webhooks targeting the same URL first). `mulita-worker-watcher` container retired; celery `--beat` folded into `worker-light` so `discard_missing_photos_beat` still fires. End-to-end verified by uploading a test JPEG through WebDAV — both `NodeCreated` and `NodeWritten` fire, mule receives + dispatches, 200 OK.
**NC cron tightened to `*/1` so webhook delivery latency drops from up to 5 min to ~60 s** (`crontab -u www-data` on LXC 114). NC dispatches webhook calls through its background-job queue; the cron interval = the worst-case latency. `*/5` was the default; `*/1` is the upstream recommendation anyway.
**Post-Phase-2 delete-roundtrip patches (commit `9408825`).** End-to-end testing of the NC↔mule deletion paths surfaced two real gaps:
1.*Folder delete* — NC fires one `NodeDeletedEvent` for the folder, not one per child. The webhook handler bailed with "unsupported extension" and photos under the deleted folder kept `is_discarded=false` until the 30-min reconcile sweep. Fix: `handle_directory_deletion()` does a single `UPDATE photos SET is_discarded=true WHERE filepath LIKE 'dir/%'` when the deleted path has no supported image extension.
2.*Resurrect on rewrite* — PUT-overwrite of a previously-discarded file fired `NodeWrittenEvent` → `scan_folder`, but scan_folder's "photo exists, skip" branch left `is_discarded=true`. Fix: when the existing row is discarded, flip `is_discarded=false`, clear `discarded_at`, re-queue `extract_metadata`.
Verified end-to-end on `Photos/MuleTestFolder/{test1,test2,nested/test3}.jpg`: `DELETE Photos/MuleTestFolder/` discarded all 3 in one shot; `PUT test1.jpg` back resurrected only that one.
**Known remaining gap — trashbin restore.** Moving a file out of `/dav/trashbin/.../trash/foo.jpg.dXXXX` back to `/files/...` via WebDAV MOVE fires **no event mule subscribes to**. NC's trashbin app emits its own internal event class that isn't in the `OCP\Files\Events\Node\*` set we registered with `webhook_listeners`. Workaround: re-upload via PUT (covered by the resurrect-on-rewrite fix above) or wait for the 30-min reconcile sweep.
**Folder rename round-trip (commits `f4a03b6` + `f27f3cb`).** Both directions now work:
- *NC-side rename → mule:* webhook's "renamed" branch detects directory rename (neither path has a supported extension) and calls new `handle_directory_rename(old, new)` in `scan.py`. The helper iterates matching rows in Python and prefix-rewrites `Photo.filepath`, `Folder.path`, `SourceRoot.path` in one transaction. Cross-source-root case (rare) discards the old subtree and lets scan_folder add fresh rows under the new root.
- *Mule-side rename → NC:* the existing `PATCH /api/v1/folders/{id}` endpoint (`folders.py:202`) already does WebDAV MOVE via `nextcloud_dav.move_for_user` and rewrites mule's DB inline. The NodeRenamedEvent that bounces back through the webhook hits `handle_directory_rename`, which finds 0 rows under the old prefix and is a no-op — feedback loop is idempotent.
**asyncpg gotcha**: the original implementation of `handle_directory_rename` used raw SQL with `SUBSTRING(filepath FROM LENGTH(:old_prefix) + 1)` so the offset would be computed server-side. asyncpg's type inference miscategorises the `LENGTH()` result and rejects the parameter as "$2: int (expected str)". The fix iterates in Python (same pattern as the existing PATCH endpoint). Lesson: avoid passing `LENGTH(:x)+1` as an argument to `SUBSTRING(...)` via asyncpg + sqlalchemy `text()`.
**Phase 3 — Memories-backed `extract_metadata` (commit `2a5759c`).** Memories app re-enabled on NC. `extract_metadata` now tries `GET /index.php/apps/memories/api/image/info/{fileid}` (auth: Basic + `OCS-APIRequest: true` header to bypass CSRF) before falling back to ExifTool. Replaces ~80 ms of subprocess with ~1-2 ms HTTP for ongoing imports. New helpers: `nextcloud_dav.fetch_memories_info_async()` + `metadata._apply_memories_metadata()`. We kept mule's full date-fallback chain (SubSec → DateTimeOriginal → CreateDate → MediaCreateDate → TrackCreateDate → filename heuristic → mtime) because 35% of the library (`taken_at_source='path'`) depends on the filename heuristic, and Memories alone would silently mis-date those photos to mtime. PhotoInfoPanel reads `exif.Make/Model/ISO/FNumber` from `photos.exif_json` — Memories' `exif` blob uses those exact plain key names, so no frontend adapter was needed. ExifTool subprocess is still in place as the fallback for brand-new photos racing the NC scan, non-NC photos, and any NC HTTP failure.
**Phase 4 — retire `/data/thumbs` (commits `5a67ed7` + `7a1c6b6`).** Vision worker now fetches NC's 640px preview via a new sync helper `nextcloud_dav.get_preview_bytes()` instead of reading `/data/thumbs/{id}/medium.webp`. `thumbs.WORKER_THUMB_SIZES = set()` — `generate_thumbnails` still computes pHash on the original-res pixels (perceptual dedup is mule-only) but stops writing files. All ~22k `medium.webp` purged after verification; `/data/thumbs` shrank from 4.1 GB → 94 MB residual.
**SECRET_KEY bug found mid-deploy (commit `7a1c6b6`).**`docker-compose.yml` only set `SECRET_KEY` on the `backend` service, not on the workers. Workers' Fernet-based `decrypt(nextcloud_app_password_enc)` silently returned empty and `_credentials_for()` raised `NextcloudCredentialsMissing`. This meant Phase 3's `extract_metadata` in worker-light had been silently falling back to ExifTool *the entire time*, and Phase 4's vision worker couldn't fetch NC previews at all. Fix replicates `SECRET_KEY=${SECRET_KEY:-...}` to all worker services. After any compose edit, run `docker exec mulita-<svc> env | grep SECRET_KEY` to confirm propagation.
**Caveats worth knowing for Phase 4:**
- Vision is `disabled` in production (`redis GET mulita:flags:vision.enabled == "false"`). The Phase 4 vision-from-NC path is correct but unexercised by live traffic; it'll matter whenever vision is re-enabled.
- NC's preview generator hasn't covered the whole library at the 640px+ tier. A sample of 6 photos: 4 with proper sizes, 1 stuck at 160px (`IMG_4954.DNG`), 1 unfetchable (`IMG_0193.DNG`). If vision re-enables and quality matters, run `pct exec 114 -- sudo -u www-data php occ preview:generate-all` to backfill.
- The HTTP `/thumb/{size}` endpoint's inline-regen disk fallback still writes one WebP per NC-404 event (mis-extensioned RAW etc). `/data/thumbs` will grow back very slowly from that path.
`GET /api/v1/library/duplicates/groups` now includes `filepath` in each member payload. `DuplicatesView` renders a dark bottom-strip on every thumbnail showing the last two folder segments — `…/<grandparent>/<parent>` (e.g. `…/files/Phone Photos starting 01-26`) via a `duplicatePathLabel` helper. Going up two levels avoids the failure mode where two copies sit under matching parent names (e.g. duplicate `2023/` subfolders under different archives) and the label would collapse. Full filepath surfaces through the native `title` tooltip on hover. Dimensions chip moves from bottom-left to top-left so the path strip can run edge-to-edge. `memberToPhoto` finally stops faking `filepath = filename` — the synthetic Photo it hands to `PhotoThumbnail` now carries the real path.
**Infra side-effect during the deploy:** two consecutive `--force-recreate` cycles (deploy.sh and a manual `down/up`) raced and left **orphan duplicate worker containers** (`<id>_mulita-worker-light` + the named `mulita-worker-light`, same for vision) both pulling from the same Redis queue. Worker concurrency was effectively doubled, host load climbed past 120, and `pct exec` into LXC 120 hung for several minutes. Recovery: `docker rm -f` on both orphan IDs. The deploy-gotcha entry below covers the pattern; the new contribution from this session is "look for renamed `<id>_<service>` containers and remove them if you ever see load spike + pct hang after a deploy".
### 2026-05-11 — Drop Title field, add bulk Notes editor
- Removed the `Title` (user_title) row from `PhotoInfoPanel.tsx`: the form field, its `titleDraft`/`setTitleDraft` state, the `commitTitle` handler, and the `photo?.user_title` watcher in the draft-sync `useEffect`. Column stays on `photos` and on the backend model — only the UI affordance went away.
- New bulk action `set_notes` in `backend/app/routers/photos.py``bulk_action`: validates the value is a string (or null/empty to clear), then sets `user_notes` on every photo in the selection in one transaction.
- Frontend wiring: `bulkSetNotes` in `services/api.ts`, `bulkNotes` mutation in `hooks/useBulkPhotoMutations.ts` (optimistic patch with empty-string → null collapse, full rollback on error), surfaced in `RightSidebar.tsx`'s multi-photo bulk panel as a 2-row Textarea + Clear + Apply pair below the Tags section. Doesn't auto-fire on every keystroke — clicking Apply is the explicit commit (otherwise every keystroke would PATCH N rows).
- Shared `Photo` interface in `types/photo.ts` now includes `user_notes?: string | null` so the optimistic-patch typing accepts the field; previously it only existed on `PhotoInfoPanel`'s local `PhotoDetails` shape.
### 2026-05-10 — Right sidebar compact rebuild
Right sidebar previously had `<ActiveHeapCard />` + `<Header />` + a scroll region holding two parallel collapsibles ("Edit" + "Metadata"), with three nested `<Section>` sub-collapsibles (Basic Info / Camera / Location) inside Metadata. A lot of chrome for a per-photo form.
- **Removed** `frontend/src/components/heaps/ActiveHeapCard.tsx` entirely (no remaining importers; active-heap state stays in the store, Select/Discard buttons still consult it).
-`RightSidebar.tsx` single-photo branch also drops its `<Header />` strip — the new "METADATA" collapsible trigger inside `PhotoInfoPanel` is the visible title. Multi-photo branch keeps the Header for "N Photos Selected".
-`PhotoInfoPanel.tsx` is now two stacked collapsibles:
- **Metadata** (default expanded): readonly fields (Size / Dimensions grid, Path, GPS inlined when present), a thin `<hr>`, then editable form (Filename, Title, Date Taken, Notes, Tags, Rating + Color on one row, Flag).
- Compact density: Notes `rows=3 → 2`, stars/swatches `h-5 → h-4`, `space-y-2.5 → 2`, Flag buttons `text-sm → text-xs`, grid `gap-2 → gap-x-2 gap-y-1`, empty "No GPS data" chip dropped (now hidden when there are no coordinates).
- Local `Section` helper deleted from `PhotoInfoPanel.tsx` (no longer used).
`PreviewView` reuses `RightSidebar` inside its overlay aside, so the change applies in both grid and preview.
### 2026-05-10 — Timeline scroll-anchor on section switch
Clicking a folder in the left sidebar (or any other navigation that changed `currentSection` — All Photos, Rated, Discarded, etc.) didn't reset the timeline's scroll position. If the user was deep in All Photos and clicked a folder with fewer rows, the new section loaded with the previous scroll offset preserved, often landing on empty space below the last row.
`Timeline.tsx` already had a section-change effect that cleared selection and reset the auto-focus guard, and a separate auto-focus effect that selects `photos[0]` once the new query resolves. The latter has an "ensure visible" scrollTo, but it only fires if the cell is out of view AND only after the next render — by then the user has already seen the wrong scroll position. Added `parentRef.current.scrollTop = 0` synchronously inside the section-change effect so the first paint of the new section anchors at the top; the auto-focus selectPhoto then highlights photo[0] as before.
### 2026-05-10 — Filter bar 422-on-date-from
User reported "all filters broken, no photos shown" right after the metadata-extraction overhaul shipped. Tracing actual API traffic on a fresh session showed `GET /api/v1/photos?per_page=100&date_from=2026-04-10&sort=taken_at&order=desc` returning **422 Unprocessable Entity**: pydantic v2's `datetime` parser rejects bare-date strings (`"2026-04-10"`) for `Optional[datetime]` query params.
The frontend's `filtersToParams` in `store/filterStore.ts` had been padding `date_to` with `T23:59:59` for inclusive end-of-day, but `date_from` went out as a bare `YYYY-MM-DD` — so every date-range filter request 422'd, and TanStack Query's failure-state shows an empty grid. From the user's perspective it looked like "filters return nothing across the board"; from the backend it was a single endpoint signature problem.
Fix on both sides:
- **Frontend**: pad `date_from` with `T00:00:00` symmetric to `date_to`'s `T23:59:59`.
- **Backend**: switch `date_from` / `date_to` to `Optional[str]` and parse with `datetime.fromisoformat` inside the handler. `fromisoformat` accepts both bare dates (→ midnight) and full ISO strings, so any older client / curl that sends a date-only value still works. Tz-aware values get coerced to naive UTC, matching the `taken_at` column shape and the same fix applied to `PATCH /photos/{id}` earlier in the session. Bad input now returns 400 with a clear message instead of pydantic's 422.
### 2026-05-10 — Date extraction overhaul
User reported wrong "Date Taken" on stills (JPEG / HEIC / DNG). Tracing the pipeline showed the trusted-EXIF list at `services/metadata.py:230-244` ended in `EXIF:ModifyDate`, which is set every time a file is re-saved (Lightroom export, batch resize, EXIF strip), so any photo that lost its original capture metadata during editing was being labeled `taken_at_source='exif'` with the *edit* timestamp.
- Fall back to `guess_date_from_path()` (the same heuristic that already powers `has_date_warning`) when no trusted EXIF date is found. New `taken_at_source='path'` value with a "PATH" badge in the info panel; tooltip explains the date came from filename / folder rather than real EXIF.
- Skip the date-write block entirely when `photo.taken_at_source == 'manual'` so a rescan can't clobber a user correction. (Previous behavior overwrote manual edits.)
-`parse_exif_datetime` accepts tz-aware variants (`%z`), normalizes to naive UTC, and rejects the `0000:00:00` placeholder.
Backfill: new `backfill_taken_at` celery task + `POST /api/v1/library/maintenance/backfill-taken-at` endpoint. Re-enqueues `extract_metadata` for every non-manual, non-trashed photo so the new rules apply across the existing library. Snapshot before the sweep started: 11,223 `exif` + 6,039 `filesystem` + 10 `path`. Will reshape over the next ~45 min.
Side note: the default celery queue had ~209k pending tasks at the time we fired the backfill — the watcher's 5-minute restart loop (since fixed) had been re-enqueuing scans, and tasks for the 4,158 photos we hard-deleted earlier today were still sitting around. Most fail fast (`Photo not found`, ~24ms each); real work runs at ~100ms. Decided to let it drain instead of flushing — safer.
### 2026-05-10 — Two cascade-delete + datetime fixes after the hard-remove shipped
The first cut of `delete_nextcloud_source_root` blew up with `folders_parent_id_fkey` violations on the first real test (`Taco and Muli - 2024 onward`, 35 folders / 4,158 photos). Two iterations to get it right:
1.**NULL `parent_id` before deleting Folders.** Folders have a self-referential `parent_id` FK with no `ON DELETE` rule; postgres checks the constraint per row regardless of insertion order, so a flat `DELETE FROM folders WHERE id IN (...)` of the whole subtree fails on the parents whose children appear later in the same statement. Fixed with an `UPDATE folders SET parent_id = NULL WHERE id IN (folder_ids)` first.
2.**Widen the NULL UPDATE to cross-source-root children.** A "Leóns 1st Year" SourceRoot at `.../Taco and Muli - 2024 onward/Leóns 1st Year` had its own folder rows whose `parent_id` pointed into the Taco SourceRoot's hierarchy. The first patch only NULLed `parent_id` for folders whose `id` was in the delete set; the Leóns folders weren't in that set so they kept their references and the FK still tripped. Fix: `UPDATE folders SET parent_id = NULL WHERE parent_id IN (folder_ids)` — kills any incoming reference into the delete set, internal or external.
After both fixes: `DELETE /api/v1/nextcloud/source-roots/{id}` for `Taco and Muli - 2024 onward` cleared 4,158 photos and 35 folders in a single request and returned 200. UI swaps the trash icon for a `Loader2` spinner while the request is in flight (`removeRoot.isPending && removeRoot.variables === r.id`) so the multi-second cascade is visible.
Separate fix in the same session: `PATCH /api/v1/photos/{id}` returned 500 with `can't subtract offset-naive and offset-aware datetimes` when the frontend sent a tz-aware ISO string for `taken_at` (the datetime-local input is supposed to be naive but real-world locales / paste flows occasionally include `+02:00`). The DB column is `timestamp without time zone`, so asyncpg refused to bind. Normalize on the server with `astimezone(timezone.utc).replace(tzinfo=None)` if `tzinfo is not None`.
Deploy infra learning: the new `--force-recreate` in `deploy.sh` does NOT reliably recreate containers on every push — saw two consecutive deploys leave the backend at the previous `StartedAt` despite a fresh image. Manual `docker compose up -d --no-deps --force-recreate <svc>` after deploy still races with whatever compose state the auto-deploy left mid-flight, twice landing the stack in a half-broken state (orphaned `<id>_mulita-backend` rename containers, db / redis stopped). Recovery: `docker compose down && docker compose up -d`. Open question — `--force-recreate` may need to be replaced with something more explicit.
`DELETE /api/v1/nextcloud/source-roots/{id}` was a soft-deactivate (`is_active=false`) — the trash icon in Settings only hid the SourceRoot from active queries while leaving every Folder + Photo row in the DB forever. Re-adding the same path resurrected ghosts; `prune-missing` reported zero deletes for the soft-removed entry because the cleanup code skipped inactive source roots.
Endpoint now hard-deletes the SourceRoot, every Folder under it (chunked at 500), and every Photo in those folders. `photo_tags` and `heap_photos` cascade automatically via `ON DELETE CASCADE` on the join tables. `FolderShare` uses a stringly-typed `folder_id` (no FK) so cleaned manually for both `folder_type='folder'` and `folder_type='source_root'`. Returns `{deleted_photos, deleted_folders}` so the UI can toast a count. Files in Nextcloud are untouched.
Sync side: the `watch_folders` celery task in `backend/app/tasks/scan.py` already detected filesystem deletions and soft-marked photos `is_discarded=true`, but the global `task_soft_time_limit=300` in `app/tasks/celery.py` was killing the watcher every five minutes and dropping every FS event during the restart window. The `soft_time_limit=None` on the decorator was being interpreted as "use worker default" rather than "unlimited". Override to `soft_time_limit=0, time_limit=0` (Celery convention for unbounded). Backstop: a new `discard_missing_photos()` in `app/services/cleanup.py` — soft variant of `prune_missing_photos`, walks every `_sr_state == 'present'` source root and flips `is_discarded=true` on Photo rows whose file is gone. Wired as `discard_missing_photos_beat` celery task scheduled every 30 minutes via `beat_schedule` on the celery app. Beat runs in-process on `worker-watcher` (`--beat` flag in `docker-compose.yml`) — the watcher is already a Redis-locked singleton so no need for a separate beat container.
Manual `POST /api/v1/library/maintenance/prune-missing` remains the hard-delete path for when the user wants to permanently drop orphan rows; the new beat job only soft-discards (file gone → mule-image trash, restorable).
### 2026-05-10 — OIDC auto-redirect on LoginPage
`OIDC_ENABLED=true` was already set in `.env`, so the LoginPage rendered a "Sign in with Authentik" button next to the password form. With a single trusted IdP and a logged-in Authentik session, that extra click was friction without upside.
`LoginPage.tsx` now reads `/auth/config` on mount and, if OIDC is on, immediately navigates to the OIDC login URL. Authentik recognizes the existing session and bounces back through the callback with no user interaction. Two escape hatches: `?password=1` in the URL forces the password form, and a `skipAutoSso` sessionStorage flag (set by `AuthContext.logout` and by the OIDC callback's error branch) suppresses the next auto-redirect so logouts actually log out and OIDC failures surface their error instead of looping. While the redirect is in flight the page shows "Signing in with Authentik..." plus a "Use password instead" link.
### 2026-05-10 — right sidebar restructure (heap pinned, single scroll, collapsible Metadata)
The right sidepanel had three stacked flex regions: `<ActiveHeapCard />`, `<Header />`, and `<PhotoInfoPanel />` — with `PhotoInfoPanel` carrying its own internal scroll. That left the editable fields (filename, title, notes, rating, color, flag) stuck above the readonly metadata scroll, effectively two scroll boundaries on one sidebar.
- Moved the scroll boundary up to `RightSidebar.tsx`: only `ActiveHeapCard` and `Header` stay pinned now. Editable + readonly content scroll together in one `flex-1 overflow-y-auto` region beneath them.
-`PhotoInfoPanel.tsx` dropped its `h-full` / inner `flex-1 overflow-y-auto`.
- The four readonly sections (Tags / Basic Info / Camera / Location) are now wrapped in a single outer "Metadata" `Collapsible`. Default expanded, one click hides the whole block. Sub-sections stay individually collapsible.
- Second pass: the editable form (filename / title / notes / rating / color / flag) got the same treatment under an outer "Edit" `Collapsible` so the panel is now two equal collapsible groups below the title strip. Dropped the X (clear-selection) button from the Header; Esc and grid-empty-area-click still clear.
- Third pass: split editable vs read-only between the two groups consistently. `TagsEditor` and `TakenAtEditor` had been buried inside the readonly Metadata sub-sections — `Tags` as its own Section, taken-at wedged into `Basic Info` between size/dims and the filepath. Moved both into the Edit collapsible, ordered identification → description → categorization: Filename · Title · Date Taken · Notes · Tags · Rating · Color · Flag. Metadata now holds only readonly: `Basic Info` (size, dims, path), `Camera`, `Location`.
`PreviewView` reuses `RightSidebar` under an `<aside class="overflow-hidden">`, so the change applies in both the grid sidebar and the preview overlay.
User reported slow load. Five fixes shipped, in order:
1.**Backend out of dev mode.**`docker-compose.yml``command:` was running `uvicorn … --reload` (single worker, file-watcher). Page loads fan out ~15 parallel API calls; they all serialized on one event loop. → `--workers 2 --proxy-headers`. Two uvicorn worker procs now.
2.**iPhone Apple ProRAW / Linear DNG decode fixed.** rawpy 0.26.1 + LibRaw 0.22 rejects Apple Linear DNG (`Photometric Interpretation: Linear Raw`, 8-bit) as "Unsupported file format or not RAW file". Those files are TIFF containers with developed RGB inside and have no embedded preview to fall back to. Added PIL `Image.open(src_path)` as the last fallback in both `_generate_proxy_webp` (routers/photos.py) and `process_raw_image` (tasks/thumbs.py). ~1,300 iPhone DNGs were 415-ing on every detail view; now decode in <1s via PIL.
3.**Reconcile DB with disk (renamed-folder case).**`prune_missing_photos` was skipping all 4,154 orphaned photo rows under SourceRoot `Taco and Muli - 2024 onward` (renamed in Nextcloud to `Photo Archive 2004-2024`) because the leaf path didn't resolve and the code assumed "drive unmounted → must skip". Added `_sr_state()` to classify as `present` / `renamed` (parent mount fine, leaf gone) / `unmounted` (parent inaccessible). Only `unmounted` still skips. Two stale source roots logged with a clearer hint pointing at `POST /api/v1/library/maintenance/prune-missing`. User has not yet been asked to run that — endpoint is ready when they are.
4.**Frontend page size + idle polling.**`usePhotosQuery.ts` was first-fetching `per_page=500` (slow paint + 500 thumb requests at once). Split into `PER_PAGE_INITIAL=100` for first paint, `PER_PAGE_BACKGROUND=500` for the cursor-chain prefetch. Idle polling for `scan-status` and `worker-status` (`useScanActivity.ts`, `ScanProgress.tsx`) bumped from 10s/15s to 30s/30s while idle; active cadence (2s/3s) unchanged.
5.**Partial index on `photos`.** Default list query `WHERE NOT is_trashed AND NOT is_hidden ORDER BY taken_at DESC NULLS LAST, id DESC LIMIT N` was doing a seq-scan + top-N heapsort (~25ms standalone, worse under concurrency). Added migration `0017_photos_list_index` creating `ix_photos_list_visible` (partial index on the sort key, restricted to visible rows). EXPLAIN now shows an Index Only Scan → **24.7ms → 0.097ms**, ~250× speedup.
**Deploy gotcha — fixed (with caveat).** The original `docker compose up -d --build` in `deploy.sh` did not reliably recreate containers when only runtime config (`command:`, env-only) or migration files changed; image hash would change but compose treated the existing container as "current enough". Bit three times this session before `/opt/mule-deploy/deploy.sh` was updated to `docker compose up -d --build --force-recreate`. Trade-off accepted: an extra restart cycle on deploys where nothing user-visible changed.
**Caveat:** the first auto-deploy after the flag flip raced with my own earlier manual `docker compose up -d --no-deps --force-recreate frontend` and landed the stack in a half-broken state — `mulita-frontend` got stuck under a renamed temp container, several services dropped off `mule-image_mulita-network`, frontend nginx restarted in a loop with `host not found in upstream "backend"`. Fixed by `docker compose down && docker compose up -d`. Don't issue a manual `--force-recreate` on a single service while the auto-deploy webhook is also expected to fire — let the deploy own the lifecycle.
**Data drift still outstanding.** 4,154 photo rows + 1 unregistered folder (`Photo Archive 2004-2024`) on disk that's not a SourceRoot. The reconcile endpoints now work — user decides when to call them. The new folder needs to be added as a SourceRoot via the Settings UI before its files will be indexed.
**Proxy cache still empty** (`mule-image_proxies_data` volume is 4 KB). Pre-generating ~500–1,500 WebP proxies for non-web-safe formats would make first-open of every RAW/HEIC photo instant. Deferred — needs a one-shot script and the disk-space tradeoff isn't worth it until the data-drift reconcile happens first.
Bind `/mnt/library/homecloud` into the workers, reads via filesystem, writes via WebDAV. `users.nextcloud_username` override field added; `dtoro → admin` mapping. Surviving SourceRoots cleaned up to NC paths.
### 2026-04-22 — native OIDC via Authentik
Authlib-based code in `backend/app/auth_oidc.py`. `extra_hosts` override for `auth.hubris.network` in compose override (gitignored).
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
## Snippet: `(authentik)` forward-auth
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](106-auth-outpost.md#forward-auth-domain-level-setup).
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](106-auth-outpost.md).
## Caddy environment
Untracked secrets in `/etc/caddy/caddy.env`:
-`IONOS_AUTH_API_TOKEN` — DNS-01 ACME
-`ARTIFACTO_GATEWAY_SECRET` — must match `SSO_GATEWAY_SECRET` on [apps (105)](105-apps.md). Rotate together.
## Auto-deploy
Push to `dtoro/caddy-conf` master → gitea webhook → `http://192.168.8.175:9797/deploy` → `caddy-deploy-webhook.service` validates HMAC against `/etc/caddy-deploy/secret` → runs `/etc/caddy/scripts/deploy.sh`:
1.`git pull`
2.`caddy validate`
3.`systemctl reload caddy`
Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scripts, systemd unit all live in the repo under `scripts/`. The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`). Git credentials for the root-run service are at `/etc/caddy-deploy/git-credentials` (mode 600).
> **Systemd unit changes inside the repo do NOT auto-reinstall** — you have to re-run `scripts/webhook/install.sh` manually after editing the unit.
## Related
- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
- [Gitea (104)](104-gitea.md) — webhook source
- All container pages list their Caddy backend.
## Changelog
### 2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082
`sab.hubris.network` now uses `import authentik` inside a `handle` block. SABnzbd own auth disabled, `local_ranges` set for transparent proxy. Port bumped from `:8081` to `:8082` (fix from 2026-06-04) now documented.
### 2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added
**Symptom:** All `*.hubris.network` hosts except `photos` and `auth` (VPS-hosted) returned `tlsv1 alert internal error` or timeout. Only 3 site blocks (`photos`, `prism`, `photos2`) remained in the Caddyfile.
**Root cause:** The Caddyfile was manually edited directly on LXC 121 (not via the `dtoro/caddy-conf` git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.
**Permanent safeguards added to `/etc/caddy/scripts/deploy.sh`:**
- **Site-count guard:** refuses to reload if fewer than 20 `*.hubris.network` blocks detected
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
### 2026-06-02 — caddy.service unit missing; recreated
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
### 2026-04-28 — wiki entry created
Initial documentation. 16 active sites at this date.
Caddy site `artifacto.hubris.network`: forward-auth on admin paths, `header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}`, public matchers `/p/*`, `/static/*`, `/healthz`. `caddy.env` extended with the gateway secret.
### 2026-04-21 — `(authentik)` snippet added
Top-of-file snippet used by every gated site. Targets `http://192.168.8.180:9000` directly to dodge hairpin TLS round-trip.
### 2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132)
For the [arriman (122)](122-arriman.md) docker stack. Caddy + certs work; needed corresponding [DNS split-horizon](../infrastructure/dns.md) entries to be added to land cleanly.
### 2026-04-20 — auto-deploy pipeline shipped
Repo `dtoro/caddy-conf`. Webhook receiver at `:9797`, validate + reload on push to master.
Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr) compose. Replaced the old yunohost-based [LXC 100](index.md#recently-destroyed-kept-for-archaeology) on 2026-04-21.
## At a glance
- **Hostname:** `arriman`
- **IP:** `192.168.8.245`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below)
## Compose
`/home/arr/ezarr/docker-compose.yml`. Network `ezarr_default``172.18.0.0/16`. Service users all run with primary GID 10000 (host `media` group) inside their containers — see [media permissions](../infrastructure/media-permissions.md). Per-app `/config/<app>-config/` dirs bind to `/config` inside.
`.env` contents:
-`DOWNLOADS_DIR=/mnt/library/downloads/`
-`MEDIA_DIR=/mnt/library/`
-`MEDIACENTER_GID=10000` ← **must be 10000, not 13000.** s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. Without this, qBit errors every torrent with "Permission denied".
> **Stop the container before editing `qBittorrent.conf`.** qBit writes its in-memory config on graceful shutdown and clobbers any live edits. Recipe: `docker stop qbittorrent && sed -i ... && docker start qbittorrent`.
Mobile/desktop clients keep working via `/api/v2/*` path bypass on Caddy.
### SABnzbd
Gated with Authentik forward-auth (applied 2026-06-13). Caddy `sab.hubris.network` block uses `import authentik` inside a `handle` block. SABnzbd's own web auth is disabled:
-`html_login = 0` → no HTML login form
-`username` / `password` cleared → CherryPy Basic Auth not activated
-`local_ranges = 172.18.0.0/16, 192.168.8.0/24, 127.0.0.0/8` → proxied requests from Caddy (192.168.8.x) and Docker-proxy (172.18.x) pass without auth
**API key** (`67ef5a45e4e04157994e977005a33878`) still works for internal service-to-service calls (Sonarr/Radarr/Lidarr via Docker internal networking — they talk to SAB at `localhost:8082`, not through Caddy).
`host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host.
## Credentials
Backups of all migration-time secrets at `/mnt/library/backup/arr-migration-2026-04-21/state/`:
- Imported \*arr `config.xml`: must have `<BindAddress>*</BindAddress>`, `<Port>{ezarr-port}</Port>`, `<UrlBase></UrlBase>`. Otherwise app listens only on 127.0.0.1, wrong port, wrong path.
- Imported DB paths must be rewritten `/mnt/library/` → `/data/media/` (RootFolders / Series / Movies / Artists tables). Without this, scans and imports fail silently.
- Bazarr `config/config.yaml` (yaml, not ini) — `ip: 0.0.0.0`, `port: 6767`, empty top-level `base_url`; sonarr/radarr connection sections normalized to docker service names.
- Staging dir for migrations: `/mnt/library/backup/stage/<app>/` (visible to both old LXC 100 and arriman because both bind-mount the same host path).
## Permissions
Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has `www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr` in the `media` group. Each docker container also receives GID 10000 as its primary group via `MEDIACENTER_GID=10000`.
## Rollback material (kept until 2026-04-28+)
`/mnt/library/backup/arr-migration-2026-04-21/`:
-`tars/` — per-subtree tars of yunohost app data (granular restore)
-`snapshots/` — vzdumps of LXC 100 (pre-migration) and LXC 122 (pre-import), plus post-stop vzdumps of LXC 100 + 106
-`stage/<app>-preimport.tar.gz` — original empty ezarr `/config/<app>-config/` dirs
-`state/` — service dumps, listening ports, API keys, jellyseerr settings at migration time
### 2026-06-13 — SABnzbd gated with Authentik forward-auth
SABnzbd now uses Authentik forward-auth (same `import authentik` Caddy pattern as qBit). SABnzbd's own web auth disabled: `html_login=0`, credentials cleared, `local_ranges` extended to cover Docker bridge + homelab LAN. API key still works for internal *arr service calls. See [Auth section](#auth-reverse-proxy--authentik-forward-auth) above.
### 2026-06-04 — all arr services moved behind gluetun VPN; SAB port conflict fixed
- All services (sonarr, radarr, lidarr, bazarr, prowlarr, jellyseerr, homarr,
flaresolverr) now use `network_mode: service:gluetun` — whole stack routes
through the VPN
- Port mappings moved from individual services to gluetun container
- **Fixed SABnzbd port conflict**: was crashing in a restart loop because
qBittorrent held port 8080 inside the shared gluetun namespace. Changed
SAB internal port to 8082 (config at `/config/sabnzbd-config/sabnzbd.ini`)
- Caddy `sab.hubris.network` updated to point to `:8082`
- Jellyseerr's `extra_hosts` (auth.hubris.network) moved to gluetun since
`extra_hosts` conflicts with `network_mode`
### 2026-06-02 — ProtonVPN added (gluetun); LXC IP set static
- Added `gluetun` container to compose as a WireGuard VPN sidecar (ProtonVPN, server AL#57, located in Tirana, Albania)
- **qbittorrent** and **sabnzbd** now use `network_mode: service:gluetun` — all traffic routes through the VPN
qBit was erroring every torrent with "Permission denied" because s6-setuidgid only honors the primary PGID and `group_add:` doesn't propagate. Setting the primary GID to 10000 (the host `media` group) fixed it.
Old LXC 100 stopped; ezarr stack stood up here; data and DBs imported with path rewrites. flaresolverr folded in from old LXC 106. Three Caddy sites added (jellyseerr, qbit, sab) → see [caddy](121-caddy.md). Rollback artifacts under `/mnt/library/backup/arr-migration-2026-04-21/`.
Runs one FastAPI aggregator (`server.app:app`, port 9851) that mounts a router per plugin from the `dtoro/terminalito` repo. First consumer: `munich-home` (`/munich-home/dashboard`) — weather (Open-Meteo), MVG transit, Google Calendar, plus server-side Kita/quote logic. Talks out to the public internet for those APIs; TRMNL cloud polls it inbound every 15 min. Bearer-token gated (`TRMNL_POLL_TOKEN`); `/health` is open.
-`/etc/trmnl-plugins/env` — `TRMNL_POLL_TOKEN` (+ Google/MVG creds once enrolled)
-`/etc/systemd/system/trmnl-plugins.service`
## Auto-deploy
Wired — [auto-deploy](../infrastructure/auto-deploy.md) Shape B, webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` (`terminalito-deploy.service`). Push to `main` → `server/deploy/deploy.sh` (`git pull` + pip + reinstall units + restart `trmnl-plugins`). Secret `/etc/terminalito-deploy/secret`; git creds `/etc/terminalito-deploy/git-credentials` wired as a repo-local `credential.helper`. Manual: `pct exec 128 -- /opt/terminalito/server/deploy/deploy.sh`.
## Secrets
Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env`. Google Calendar + MVG creds are pending: enroll via `homelab client add trmnl` + bootstrap, add `secrets/trmnl-oauth.yaml`, then `server/deploy/render-env.sh` builds the env from `homelab secret trmnl-oauth`. Until then calendar/transit cards degrade to empty; weather works.
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`trmnl.hubris.network → 192.168.8.211:9851`)
Gitea Shape-B deploy pipeline (webhook id 12, `:9797`) — push to `dtoro/terminalito` redeploys; verified end-to-end. Technitium A record `trmnl.hubris.network → 192.168.8.175` added on [dns (107)](107-dns.md) (propagated to the NetBird managed zone via dns-sync), so LAN clients take the short path through [Caddy (121)](121-caddy.md). See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-06-24 — public path live
Verified end-to-end from the internet: `https://trmnl.hubris.network/munich-home/dashboard` → 200 with token, 401 without; `/health` 200. The provision-time outage was the netbird `home-lab-network` (192.168.8.0/24) route having no active routing peer — the **mac-mini routing peer's netbird daemon was down** (artifacto/blog were 504 too). Bringing netbird up on mac-mini restored the route; the edge recovered with no config change. See [ingress](../infrastructure/ingress.md) / [mesh](../infrastructure/mesh.md).
### 2026-06-24 — provisioned
LXC 128 created (Debian 13, unprivileged, `192.168.8.211`). Deployed `trmnl-plugins.service` on :9851 from `dtoro/terminalito`. Caddy block added (`dtoro/caddy-conf`) + LE cert via IONOS DNS-01; verified `/health` 200 and `/munich-home/dashboard` (live weather) through Caddy. Cert mirrored to VPS (`trmnl.fullchain.crt`/`trmnl.privkey.key`) + traefik router `trmnl-public` → `192.168.8.211:9851` added to `/opt/traefik-dynamic.yaml`. **Public path pending**: VPS↔home netbird route was down at provision time (`No networks available`, 3/6 peers — artifacto/blog also 504); resolves when the mesh route recovers. **LAN pending**: Technitium A record not yet added. Not SOPS-enrolled; Google/MVG creds pending.
- **Authentik SSO (OIDC):** Provider `Provider for Yuvomi` (PK 31) in Authentik on VPS. Env vars in `/opt/yuvomi/.env`: `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`. Redirect URI: `https://house.hubris.network/auth/oidc/callback`.
- **Paperless DMS connector (native):** Yuvomi connects directly to Paperless-ngx API at `http://192.168.8.130:8000/`. API token stored in SQLite `dms_accounts` table. Search, link, and upload documents from Yuvomi to Paperless via Settings → Documents → DMS.
- **Weather widget:** Open-Meteo (free, no API key). Munich coordinates set.
- **Google Calendar:** OAuth configured via env vars (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI`). Redirect URI: `https://house.hubris.network/api/v1/calendar/google/callback`. Authorize in Settings → Calendar → Connect Google Calendar.
## Config paths
-`/opt/yuvomi/docker-compose.yml` — downloaded from upstream
-`/opt/yuvomi/.env` — config including secrets (untracked)
-`/opt/yuvomi/data/` — SQLCipher SQLite DB (`oikos.db`)
-`/opt/yuvomi/backups/` — auto backups
-`/opt/yuvomi/modules/` — Yuvomi modules (empty for now)
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`house.hubris.network → 192.168.8.212:3000`)
### 2026-06-27 — Google Calendar OAuth env vars configured
`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI` set in `.env`. New OAuth client ID (`-bho4iq..`).
### 2026-06-26 — provisioned
LXC 129 created (Debian 13, unprivileged, `192.168.8.212`). Docker installed. Yuvomi container running on `:3000` from `ghcr.io/ulsklyc/yuvomi:latest`. Caddy block + DNS A record + VPS traefik router `house-public` for public access. Authentik OIDC provider created (PK 31). WebDAV document bridge on paperless LXC (103) at `:8088` for Paperless auto-import.
Self-hosted digital library (eBooks, comics, audiobooks). Community fork/successor of Booklore, with smart shelves, metadata enrichment, Kobo/KOReader sync, OPDS, and a built-in EPUB/PDF reader. Migrated from [apps (105)](105-apps.md) on 2026-06-29.
## At a glance
- **Hostname:** `grimmory`
- **IP:** `192.168.8.247`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged (UID = host UID for `/mnt/library` media GID)
Credentials live in `/opt/grimmory/.env` (untracked):
-`DATABASE_PASSWORD` / `MYSQL_PASSWORD` — MariaDB Grimmory user password
-`MYSQL_ROOT_PASSWORD` — MariaDB root password
## Authentik OIDC
Uses Confidential client (client secret stored in Grimmory's DB — migrated from Booklore). The OIDC config carried over in the database dump; no manual re-entry needed.
- **Authentik provider:** `Provider for Grimmory` (renamed from `Provider for Booklore` on migration)
LXC is privileged → in-container UID = host UID. Docker container gets media GID via `GROUP_ID=10000` env var (Grimmory/linuxserver pattern). The `/mnt/library/books` subtree is owned `:media` mode `2775` (setgid). See [media-permissions](../infrastructure/media-permissions.md).
LXC 130 created (Debian 13, privileged, `192.168.8.213`). Docker installed. Grimmory compose deployed at `/opt/grimmory/`. MariaDB dump from Booklore (LXC 105) restored — schema-compatible since Grimmory is a direct fork. Caddy `books.hubris.network` backend updated from `192.168.8.205:6060` to `192.168.8.213:6060`. Authentik provider updated: Booklore → Grimmory, Confidential → Public (PKCE). Booklore stack removed from Portainer on LXC 105.
| 127 | mule-photos-new | 2026-05-22 | PhotoPrism + sidecar + SvelteKit stack promoted to LXC 120 via Mulimage 2.0 merge (`70dc1b6`); M0 test LXC retired. Caddy + dnsmasq + gitea webhook + NC webhook listeners all cleaned up in the same cutover. |
| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose |
| 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
| 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology |
| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) |
| 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty |
| 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 110 | photoprism | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 111 | karakeep | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 112 | immich | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 115 | reticulum | between 2026-04-21 and 2026-04-28 | Decommissioned |
> Several `.conf.bak` files survive under `/etc/pve/lxc/` if you need to recover any of the configs.
## Conventions
- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](106-auth-outpost.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range.
-`onboot=1` on every container — the host brings them up after `pve-guests.service`.
- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library` on hubris, or `mp0: /mnt/media_local,mp=/mnt/library` on strong.
- Most containers are privileged. Unprivileged ones require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
`Homelab` cluster (see [Cluster](#cluster)); the second node is
[strong](strong.md), which hosts nothing yet.
## At a glance
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
- **Mesh:** Netbird `wt0``100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
See [containers/index](../containers/index.md). 10 active on hubris (101, 118, 122, 129, 130 migrated to [strong](strong.md) 2026-07-05).
## Boot-time tuning (load-bearing)
- **TCP BBR + `fq` qdisc** in `/etc/sysctl.d/99-bbr.conf`: `net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`, socket buffers up to 64 MiB (`rmem_max`/`wmem_max`/`tcp_rmem`/`tcp_wmem`). `tcp_bbr` module pinned to load at boot via `/etc/modules-load.d/bbr.conf`. Improves single-TCP-stream throughput on jittery / lossy links (notably WiFi clients downloading from [114 — nextcloud](../containers/114-nextcloud.md)). Container traffic crosses this kernel, so every guest inherits the change.
- **`cpu-epp.service`** (enabled) sets `scaling_governor=powersave` + EPP=`balance_power` at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. In `amd-pstate=active` mode the governor MUST be `powersave`, not `performance`, for EPP to apply. Unit ordering: `After=sysinit.target` + `Before=pve-guests.service` so it runs before the guest fleet starts (the original `After=multi-user.target` left the hottest boot window on `performance`). Fixed 2026-04-22.
- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
## Network performance baseline (2026-05-14)
| Path | Throughput | Notes |
| ---- | ---------- | ----- |
| WAN (Ookla, from host) | ↓ 113.5 Mbit/s · ↑ 19.9 Mbit/s · 24.6 ms | Asymmetric residential link. 20 Mbit upload is the cap for public ingress (artifacto, blog via VPS traefik). |
| `eno1` link | 1000 Mbit/s, Full duplex | Negotiated correctly to router. |
| Host → LXC over `vmbr0` | **34.7 Gbit/s** | Single TCP stream, iperf3 10 s, 0 retransmits. |
| LXC → LXC over `vmbr0` | **34.8 Gbit/s** | Single TCP stream, iperf3 5 s, 0 retransmits. |
Intra-host throughput is memory-bandwidth bound — not a bottleneck for any current workload.
**eno1 error counters** (uptime 14 d 6 h at measurement time): `tx_packets` 99.99 M / `rx_packets` 96.08 M; `rx_errors` 1,619,203 (~1.7 % of RX); `align_errors` 10,919; `tx_errors` 20. The 1.6 M rx_errors in 14 days is the noteworthy datapoint — likely cable / switch-port. Re-check this counter in ~1 week and confirm it is not still climbing at the same rate; if it is, swap the patch cable first.
See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/backups.md), [ingress](../infrastructure/ingress.md).
## Quirks
-`/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
- ZFS is **not** in use; storage is LVM-thin + ext4.
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
- Non-ECC RAM: silent memory faults are possible; suspect DIMMs if `cpu-epp` is on and crashes still happen.
- BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher).
-`root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
`pvecm add` could authenticate without a password prompt
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1``rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
### 2026-05-14 — LXC 116 (heaper) decommissioned
User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. Data subtree `/mnt/library/heaper` (224 MiB) retained for archaeology — see [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology) and [media-permissions](../infrastructure/media-permissions.md).
### 2026-05-13 — TCP BBR + fq qdisc enabled
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
### 2026-04-29 — relocated to better-ventilated spot
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
### 2026-04-28 — Phase 1 WiFi failover
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
### 2026-04-28 — wiki started
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
### 2026-04-22 — drive removal A/B test
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
### 2026-04-21 — `cpu-epp.service` deployed
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
## Cluster membership
Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on
2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full
cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice —
either node going down freezes management on the survivor).
## SSH
Root login via the same key set trusted on hubris (`root@hubris`,
`d.toro.v@pm.me`) — installed 2026-07-01 by appending to
`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys`
post cluster-join, so it's cluster-synced same as hubris). No password auth
needed going forward.
## Related
- [hubris — Proxmox host](hubris.md)
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
This is Phase 1a of the strong migration plan — see .hermes/plans/2026-07-05_strong-migration-assessment.md.
### 2026-07-01 — age key issued over LAN; 3 bugs found/fixed in bootstrap.sh
Re-ran bootstrap without `--no-secrets` to get a real age key. Hit three real bugs live, fixed all three in `bootstrap.sh` and re-ran clean:
1. The `mcp`-CLI pipx-install step and the (unused, `--with-hermes`-only) Goose installer both called `sudo -u <user>` unconditionally — fails with "sudo: command not found" on a minimal root-only image with no `sudo` binary at all. Added a `run_as()` helper that only shells out to `sudo` when there's a real distinct invoking user.
2.`sops` isn't an apt/dnf package (matches what `agent-enrollment.md`'s manual-install recipe already does) — the auto-installer tried `apt-get install sops` and failed outright. Added `install_sops_binary()`, fetching the GitHub release binary directly on both dnf and apt paths.
3. Bigger one: running without `--no-secrets` unconditionally tries to install + interactively connect Netbird (device-code SSO), even though the very next check already knows how to accept plain LAN reachability instead. Over SSH with nobody watching, this hangs forever — had to manually kill a stuck `netbird up` process. Added `--no-mesh`, which skips the Netbird install/connect step but keeps the LAN-fallback path for secrets issuance. This run used `bootstrap.sh --no-mesh` and completed cleanly: `mesh: lan`, age key installed, `mcp` CLI installed via pipx (proving fix #1 too).
Result: age key at `/etc/age/key.txt`, pubkey recorded in `inventory.yaml`. Not yet granted access to any actual secret file — see the note above.
### 2026-07-01 — enrolled as a homelab-context client
Ran `bootstrap.sh --no-secrets` (reused the operator's existing personal Gitea PAT for the initial clone rather than minting a fresh read-only one). Installed git, cloned `/opt/homelab-context`, installed the 5-min systemd sync timer, symlinked `homelab` CLI and `AGENTS.md`. Skipped age-key/secrets issuance and Netbird per operator choice — but bootstrap's own connectivity check reported `mesh: lan`, i.e. the secrets-issuance endpoint is already reachable over plain LAN, so re-running without `--no-secrets` later wouldn't require a Netbird join. Known gap: the `mcp` pipx CLI install step silently failed (`sudo: command not found` — bootstrap.sh's pipx step assumes a `sudo` binary even when already root; harmless, only affects the `homelab mcp <tool>` shell subcommand).
# 2026-04-21 — Hubris crash loop (thermal + USB drive)
## Summary
[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
## Timeline
### 2026-04-19 — drive attached
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
### 2026-04-19 → 2026-04-21 — first crashes
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
### 2026-04-21 — crash cadence escalates
23:02 (previous night) → 12:01 the next day. Cadence: 12 h → 43 m → 8 m → 5 m. Stable since the 12:03 boot once `cpu-epp.service` started applying `EPP=balance_power`.
### 2026-04-21 — first set of mitigations applied
- **`cpu-epp.service`** deployed. Sets `scaling_governor=powersave` + EPP=`balance_power` at boot. Drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle.
- **UAS blacklist** for the drive: `/etc/modprobe.d/usb-storage-quirks.conf` → `options usb-storage quirks=090c:2320:u`. Forces BOT instead of UAS for the SMI bridge.
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
### 2026-04-22 — recurrence after 30 h 37 m
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
### 2026-04-22 — drive removed (A/B test)
User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
#### 2026-04-23 thermal-pad verdict
Resolved (3 h self-paced watch). Steady-state across 17:03 → 19:56 stayed pinned at nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes on either drive. Controller sensor1 at 60-ish is normal idle for M.2 NVMe. **The load-bearing signal is the lifetime warning-time counter, not absolute sensor1** — watch that going forward, not sensor1 threshold.
### 2026-04-28 — A/B test passing so far
3+ days continuous uptime with the drive removed. If uptime now exceeds ~3 days without crash, the drive/PCIe-tunnel interaction is confirmed as the dominant root cause. Will revisit options (different drive, different USB port, different bridge) once stability is firmly established.
### 2026-04-29 — physical relocation
User moved the host to a spot with better airflow. Post-move idle baseline (45 min uptime, light load): Tctl **47.2 °C**, GPU 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C. Notably nvme0 sensor1 is **27 °C cooler** than the 2026-04-23 thermal-pad steady-state of 60–61 °C. Watch lifetime warning-time counters (`nvme smart-log` → "Warning Comp. Temperature Time") over the next week — flat trend confirms the move helped. Doesn't change the drive-removal A/B test, but reduces residual thermal headroom risk independently.
## Root cause
Two-contributor:
1.**Thermal**: Ryzen 5 7640HS (Phoenix APU) in a passively-cooled GMKtec NucBox sitting at ~95 °C idle under `performance` governor. Mitigated by `cpu-epp.service`. In `amd-pstate=active` mode the governor MUST be `powersave` for EPP values to take effect — `performance` silently ignores them.
2.**Storage**: external USB SSD's UAS keepalive interacts badly with the AMD USB4/Thunderbolt PCIe tunnel. UAS blacklist + mount-on-demand reduced but did not eliminate it. Removing the drive eliminates it (so far).
Background: BIOS 1.02 (latest publicly visible from GMKtec). Vendor not on LVFS so `fwupdmgr` can't update. Non-ECC RAM means memory faults are silent and undetectable except by memtest86+.
## Diagnostic notes
- **Pure hardware hangs leave no trace.** `pstore` stays empty across reboots despite `panic_on_oops/hardlockup/softlockup/rcu` + softdog `soft_panic=1` being configured. The CPU is gone before the kernel can react. Don't waste time on panic traces — use crash cadence + uptime pattern as the diagnostic.
- **First check** if hubris crash-loops again with the same silent-cutoff signature: `cpu-epp.service` is active and `/sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` is `balance_power`.
- **Non-ECC RAM means memory faults are silent.** If `cpu-epp` is on and crashes still happen, suspect DIMM next (run memtest86+) and/or try the commented-out `pcie_aspm=off pci=nomsi` workarounds in `/etc/default/grub`.
- **Grub escalation** (`pcie_aspm=off pcie_port_pm=off pci=nomsi`) **deliberately NOT applied** yet to keep the drive-removal test clean. Enable only if crashes continue without the drive.
-`mcelog` is not in Debian 13 repos; `rasdaemon` is the replacement.
## BIOS update path
Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the NucBox M6 Ultra. BIOS 1.02 (2025-08-06) is the latest publicly visible — GMKtec's download center only publishes Windows drivers, no BIOS listing or changelog. Update path is a `.exe` flasher that needs Windows (would require Windows-To-Go USB on hubris). Before committing to that, email `service@gmktec.com` with the S/N to confirm a newer BIOS / AGESA level actually exists. `fwupd` is now installed, so if GMKtec ever onboards to LVFS, updates appear automatically.
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
- If yes: which mitigation lets the drive come back? (different drive? different USB port bypassing the hub chain? rear motherboard USB 3 port? alternative bus path?)
- Newer BIOS / AGESA from GMKtec — answered email pending. Without LVFS we need explicit vendor confirmation.
- Memtest86+ pass on the DIMMs — not yet run; deferred until the drive case is closed.
# 2026-05-31 — Authentik migrated from LXC 124 to the VPS
## Summary
The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
OIDC configuration request returned status 504 with response: Gateway Timeout
```
The 504 came from VPS traefik trying to reach the old `auth-authentik` backend (`https://192.168.8.175`, home Caddy) over the down mesh.
### Fix — Phase A (restore the mesh)
1. Stood up Authentik on the VPS in `/opt/docker-compose.yml`: `authentik-postgres`, `authentik-redis`, `authentik-server`, `authentik-worker` (image `ghcr.io/goauthentik/server:2026.5.2`), on a dedicated `auth` Docker network (`172.30.1.0/24`). Traefik joined both `netbird` and `auth` networks.
2. Removed the `auth-authentik` → `192.168.8.175` backend from `/opt/traefik-dynamic.yaml`; Authentik now routed via Docker provider labels.
3. Added `depends_on: authentik-server: condition: service_healthy` to the mgmt service.
4. Created the NetBird OAuth2 provider/app manually (only manual step), then mgmt auto-recovered within 30 s.
### Fix — Phase B (full data migration)
Once the mesh was back, the homelab DB was reachable:
2. Stopped VPS authentik server/worker, dropped + recreated the VPS DB (connect to the `postgres` DB to drop `authentik`), restored the dump.
3.**Swapped `AUTHENTIK_SECRET_KEY` in `/opt/authentik.env` to the homelab's key** — Authentik encrypts OAuth2 client secrets in the DB with this key; a mismatch makes every client secret unreadable and silently breaks all SSO.
4. Restarted Authentik. All users / apps / passwords / groups present.
Scripts left on the VPS: `/opt/migrate-authentik.sh` (re-runnable), `/opt/backup-authentik.sh` (daily cron, 14-day retention).
### Fix — DNS cutover (the long tail)
The real reason the browser kept hitting the *old* Authentik even after the VPS one was healthy: split-horizon dnsmasq on LXC 124 still had `address=/auth.hubris.network/192.168.8.175`, and the NetBird per-client resolver (`100.122.255.254`) cached it. Symptom: the login page console logged `authentik(early): version 2026.2.2` (the old LXC) instead of `2026.5.2` (the VPS).
1.**Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
2.**Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
3.**"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
4.**Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
5.**WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
| DB engine | sqlite? (`StoreConfig` in mgmt) / Postgres for Authentik | Postgres, separate from netbird |
> **Security note:** the Authentik admin UI is now publicly reachable. Mitigated with a traefik `admin-allowlist` ipAllowList middleware on `/if/admin/` (currently `5.61.168.0/24`). Login/flow endpoints are public by necessity. The `auth` Docker network isolates Postgres/Redis from the netbird containers.
## Lessons learned
1.**Never put the IdP behind the VPN it authenticates.** The circular dependency is invisible until a restart, then it's total.
2.**`depends_on: condition: service_healthy` is the structural fix.** Docker refuses to start mgmt until Authentik is healthy, regardless of boot order.
3.**Migrating an Authentik DB requires carrying its `AUTHENTIK_SECRET_KEY`.** It encrypts client secrets at rest; a mismatch breaks SSO silently with no useful error.
4.**Default expression policies break across major Authentik versions.** Always run `ak apply_blueprints` after restoring an older DB into a newer Authentik.
5.**Split-horizon DNS + a VPN resolver cache = an invisible stale answer.** The NetBird resolver (`100.122.255.254`) outlives `netbird down/up`. Removing the dnsmasq override (fall through to public DNS) is the durable fix once a service is genuinely public.
6.**Redirect URIs: prefer `STRICT` over `REGEX`** in Authentik 2026.x.
7.**Frontend/backend version skew shows as "Unknown error."** Confirm which Authentik the browser actually loaded (`version` line in the console) before chasing backend logs.
8.**WebAuthn/passkeys do not survive host migrations** — delete and re-enroll.
## Forward-auth outpost — RESOLVED 2026-06-01
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
The default authentication flow was skipping the identification stage (showing only a password field with `pending_user: ""`), making login impossible. Root cause: a **"Trust me" reputation policy** (`threshold: -3`, `negate: true`) attached to the identification stage binding (#10). The `negate: true` + broken policy evaluation caused the stage binding to fail re-evaluation, and Authentik's `ReevaluateMarker` skipped the stage entirely.
The policy was orphaned (no matched type data or had incompatible evaluation). Removing it restores normal two-stage login (identification → password).
**Note:** This is different from the 2026-05-31 issue where NetBird's `login_hint` OAuth2 param intentionally pre-identifies users. That was expected behavior; this was a genuine bug.
## Outstanding
- **NetBird client version skew.** Mac client is `0.68.3` vs mgmt `0.71.3`; NetBird-SSH cert exchange fails between them. Upgrade all clients toward `0.71.x`.
- **republic-laptop** regular SSH publickey auth still rejected (correct `authorized_keys`/perms/SELinux context, yet denied — `sshd_config``AuthorizedKeysFile` suspect). DNS not yet flushed there either.
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential``netbird-dashboard` client. Verify which is live and correct the page.
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
3. **refresh-creds:** The credential is now managed by the remote URL.
If the PAT is rotated, `homelab refresh-creds` won't update the URL.
- **Workaround:** Run `git remote set-url origin` after `refresh-creds`
### Step 12 — `homelab` CLI hostname detection
- `homelab whoami` failed with `no hosts/Mac.yaml` because the CLI uses
`hostname -s` (returning `Mac`) while the bootstrap uses
`scutil --get LocalHostName` (which we fixed to `mac-mini`)
- **Fix:** Set both `LocalHostName` and `HostName` via `scutil`
- **Lesson:** The homelab CLI on macOS should prefer `scutil --get
LocalHostName` like the bootstrap does, or at least try both and pick the
one that matches an inventory key. This is a CLI bug.
## Improvement backlog
### High priority
| # | Issue | Fix |
|---|-------|-----|
| 1 | Homelab CLI uses `hostname -s` on macOS; should use `scutil --get LocalHostName` to match bootstrap | Patch `bin/homelab` to try `scutil` first, fall back to `hostname -s` |
| 2 | `homelab client add --finalize-pubkey` on hubris fails to push when remote is ahead (bootstrap pushes first) | `pull --rebase` before committing in `cmd_client_add` |
| 3 | Default Goose model (`nousresearch/hermes-4-405b`) doesn't support tool use on OpenRouter | Update `bootstrap.sh` default and `hermes-agent.md` to `deepseek/deepseek-v4-flash` |
| 4 | MCP server needs streamable_http migration | Follow-up #1 from `hermes-agent.md` — FastMCP `transport="sse"` → `transport="streamable_http"` |
| 5 | `homelab refresh-creds` should also update embedded remote URLs | In `cmd_refresh_creds`, after writing the credential file, update any remote URL that has an embedded credential |
### Medium priority
| # | Issue | Fix |
|---|-------|-----|
| 6 | macOS neovim with `clipboard+=unnamedplus` breaks sops editing | Add troubleshooting row to `agent-enrollment.md`; recommend `EDITOR=nano` |
| 7 | `/etc/homelab-context` directory with 0700 root permissions blocks non-root git | Change to 755 in `bootstrap.sh` on all OSes (or at least macOS) |
| 8 | Bootstrap `chown` at end gives `illegal group name` on macOS | Fix group detection in bootstrap.sh for macOS |
| 9 | `homelab secret` re-exec via sudo chain is fragile — needs passwordless sudo or TTY | Consider `SOPS_AGE_KEY` env-var fallback path in the CLI itself |
### Low priority
| # | Issue | Fix |
|---|-------|-----|
| 10 | Onboarding this machine revealed no `HERMES.md` file in the repo | Create the persona file for the Goose hints symlink |
| 11 | Git credential `store` helper lock mechanism is OS-sensitive and fragile | Document embedded-URL pattern as the preferred approach for workstations |
| 12 | AGENTS.md references `/opt/homelab-context/hosts/<hostname>.yaml` but the canonical path is now a symlink | Verify the path resolution edge cases |
## Changelog
### 2026-06-01 — initial post-mortem
Captured the full onboarding of mac-mini into the hubris homelab context
system, including the hostname mismatch, sudo/sops credential chain, model
selection, git credential gymnastics, and editor clipboard interference.
### 2026-06-02 — MCP server migrated to streamable-http
Item #4 resolved. FastMCP server now uses `transport="streamable-http"` (hyphen,
not underscore), serving at `https://mcp.hubris.network/mcp`. All doc references
and client configs (Goose, bootstrap.sh, agent-enrollment.md) updated. Service
# 2026-06-03 — Moonlight/Sunshine game streaming unstable over WiFi
## Summary
[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
2.**Dropouts** — 1-second UDP blackouts from radio retransmissions
3.**Jitter** — 28 ms stddev with spikes to 138 ms
Moonlight/Sunshine streams real-time video over UDP. Any jitter spike or dropout causes visual stutter or frame drops. The 57 Mbps Moonlight bitrate was too aggressive for the WiFi link's consistency, even though the average throughput is ~65 Mbps.
## Mitigations applied
### mac-mini — Moonlight (via `defaults write`)
| Setting | Before | After | Rationale |
||---------|--------|-------|-----------|
|| bitrate | 57 Mbps | 25 Mbps | Reduce to a level WiFi can sustain consistently |
|| framepacing | off (0) | on (1) | Smooths frame timing over variable latency |
|| fps | 30 | 60 | Restored to game-appropriate value |
Sunshine restarted after changes; confirmed active.
## Resolution
### 2026-06-04 — mac-mini wired to LAN
mac-mini plugged into Ethernet (en0, 192.168.178.182). This definitively resolves the WiFi jitter that caused streaming instability. All WiFi-era conservative settings reverted to appropriate wired values on both ends.
## Open questions
~~1. **Ethernet wiring** — mac-mini has an active en0 port. Plugging it into the same switch/router as ludo-mini would eliminate WiFi jitter entirely. This is the definitive fix.~~**RESOLVED** — mac-mini now wired via en0.
2.**WiFi alternatives** — If wiring is impractical, a WiFi 6 (802.11ax) USB adapter or Thunderbolt-to-Ethernet adapter would help, but wired Ethernet is the only reliable solution for game streaming.
3.**Netbird route** — `192.168.8.0/24` is routed over Netbird `utun100`. If Moonlight ever falls back to that subnet (ludo-mini's 192.168.8.133 LAN IP), traffic tunnels unnecessarily. Moonlight correctly uses 192.168.178.181, so this is not currently an issue.
## Changelog
### 2026-06-04 — mac-mini wired to LAN, configs reverted from WiFi mitigations
- mac-mini connected via en0 (wired Ethernet, 192.168.178.182), WiFi mitigations no longer needed
User needed to re-authenticate to Authentik several times per day. Root cause was the Django session being configured as a **session cookie** (cleared on browser close) with a short 24-hour lifetime. Fixed by extending both the session cookie lifetime and the user_login stage session duration to 30 days.
## Timeline
- **2026-06-06 ~23:00** — User reports "having to login to authentik several times per day"
- **~23:20** — Found `SESSION_EXPIRE_AT_BROWSER_CLOSE=True` — the `authentik_session` cookie has no Max-Age, cleared on browser close
- **~23:25** — Found the `user_login` stage has `session_duration=seconds=0`, which calls `session.set_expiry(0)` → `_session_expire_at_browser_close=True`
- **~23:30** — Applied DB fix: updated `authentik_stages_user_login_userloginstage` → `session_duration='days=30'`
- **~23:35** — Added `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` to `/opt/authentik.env` (maps to `sessions.unauthenticated_age` in Authentik config, setting `SESSION_COOKIE_AGE` to 30 days)
- **~23:40** — Recreated container with `docker compose up -d` (restart doesn't re-read env file)
- **~23:45** — Verified both changes took effect
## Root cause
Two independent but compounding issues:
1.**`SESSION_EXPIRE_AT_BROWSER_CLOSE=True`** — hardcoded in `/authentik/root/settings.py`. Makes the `authentik_session` cookie a session cookie (no `Max-Age`), cleared when the browser closes. Cannot be changed via env vars or config files (hardcoded in Python source).
2.**`user_login` stage `session_duration=seconds=0`** — the default-authentication-login stage called `session.set_expiry(0)`, which in Django sets `_session_expire_at_browser_close=True` on the session object, overriding any session cookie lifetime set via `SESSION_COOKIE_AGE`.
3.**`sessions.unauthenticated_age=days=1`** (default) — `SESSION_COOKIE_AGE` was 86,400 seconds (24 hours), so even with the browser left open, the server-side session data expired after 24 hours with no way to extend it (`SESSION_SAVE_EVERY_REQUEST=False`).
### Why forward-auth worked but OAuth2 didn't
- **Forward-auth (Caddy-gated services)** — The `authentik_proxy_*` cookie is a *persistent* cookie with 24-hour validity, set by the Authentik proxy outpost. It survives browser restart. Caddy's forward-auth validates this cookie directly with the outpost API — no Django session needed.
- **OAuth2 (Gitea, Jellyfin, etc.)** — These services redirect to `auth.hubris.network/application/o/authorize/`, which checks the Django session (`authentik_session` cookie). If the browser was closed, this cookie is gone → user is redirected to the login form.
Result: `SESSION_COOKIE_AGE` = 2,592,000 seconds (30 days). This is a fallback for sessions that don't go through the `user_login` stage or when the stage's explicit `set_expiry` doesn't apply.
-`SESSION_COOKIE_SECURE=False` — Should be `True` since Authentik is HTTPS-only, but the custom `SessionMiddleware.is_secure()` method dynamically sets the cookie's `secure` flag based on the request, so it's fine.
- 30 days is generous. Could be reduced to 7-14 days if desired. Change the DB value and env var accordingly.
# Investigation: Caddyfile truncation — all LAN services down (2026-06-06)
**Date:** 2026-06-06
**Status:** resolved
**Duration:** ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)
## Symptom
All `*.hubris.network` URLs except `photos.hubris.network` and `auth.hubris.network` returned `tlsv1 alert internal error` or TCP timeouts from LAN/mesh clients. `dig @192.168.8.2` and `dig @100.122.255.254` both resolved to `192.168.8.175` correctly — DNS was fine. The issue was at the Caddy level.
## Root cause
The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the `dtoro/caddy-conf` git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: `photos.hubris.network`, `prism.hubris.network`, and `photos2.hubris.network`.
### Timeline
| Time (UTC+2) | Event |
|---|---|
| Jun 04 23:43 | Last successful git-push deploy — full Caddyfile (260 lines) |
| Jun 06 ~12:00 | Caddyfile manually edited locally, truncating to 3 sites |
| Jun 06 12:39 | Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten" |
| Jun 06 14:13 | Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow) |
| Jun 06 22:34 | Investigation began |
| Jun 06 22:43 | Caddyfile restored from `origin/master`, `systemctl reload caddy` |
### Evidence
-`git diff HEAD -- Caddyfile` on LXC 121: `+3 / -159` lines
- Git reflog: HEAD at `32575ce` (fix: sab port 8081→8082), working tree diverged
- Backup file `Caddyfile.bak.1780263919`: 225 lines, full original config
-`git stash list` shows one auto-stash entry
-`origin/master` at `1b977aa`: 260 lines, all site blocks present
### Secondary root cause found during investigation
**elementsynapse (LXC 118)** had `iface eth0 inet dhcp` internally despite `pct set 118 --net0 ... ip=192.168.8.239/24`. On DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool. Caddy's `reverse_proxy 192.168.8.239:8008` was hitting a dead IP.
This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.
| Site-count guard | `/etc/caddy/scripts/deploy.sh` | Refuses reload if <20 `hubris.network` site blocks |
| Dirty-tree auto-stash | `/etc/caddy/scripts/deploy.sh` | Stashes local edits before git pull |
| Auto-backup | `/etc/caddy/scripts/deploy.sh` | Saves Caddyfile.bak.<timestamp> before any change, keeps 5 |
| Caddy backend health | `/etc/cron.d/caddy-backend-health` on hubris | Runs `check-caddy-backends.sh` every 10 min |
| DNS sync | `/etc/cron.d/dns-sync` on LXC 107 | Runs `dns-sync.py` every 10 min (was missing since 2026-06-04) |
## Related
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
Append-only record of documentation-maintenance operations on the knowledge wiki (restructures,
source ingests, lint sweeps). One line per operation, newest last. Infrastructure changes belong in
each page's `## Changelog` and the Oikos change ledger, not here.
## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output.
## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md.
## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference.
ZimaOS (IceWhale / CasaOS-family NAS distro), installed as a Proxmox VM to evaluate it as a potential primary NAS frontend in front of `/mnt/library` — alongside the existing fleet ([nextcloud (114)](../containers/114-nextcloud.md), [jellyfin (101)](../containers/101-jellyfin.md), [mule-images (120)](../containers/120-mule-images.md)).
- **EFI disk:** none — ZimaOS installer writes a removable-media fallback (`/EFI/BOOT/BOOTX64.EFI`), so a persistent EFI vars disk isn't required. PVE warns "no efidisk configured! Using temporary efivars disk" on start; harmless.
- **ISO:** `/var/lib/vz/template/iso/zimaos-x86_64-1.6.1_installer.iso` (1.48 GB, from `IceWhaleTech/ZimaOS` GitHub releases; upstream publishes SHA256 only for the `.img`/`.raucb` variants, not the `.iso`)
- **Netbird mesh (off-LAN):** same URL — works because hubris advertises `192.168.8.0/24` into the mesh, and once a peer resolves `zimaos.hubris.network → 192.168.8.175`, the request tunnels home. DNS resolution from a mesh peer needs **either** a netbird Management nameserver group (`hubris.network` match domain → `192.168.8.180`), or an `/etc/hosts` override on the peer (`192.168.8.175 zimaos.hubris.network`). See [mesh](../infrastructure/mesh.md).
- **Direct fallback:** <http://192.168.8.195> — backend ZimaOS bundles its own Caddy (`Server: Caddy`, `Via: ZimaOS-Gateway`), which the front-end Caddy on LXC 121 reverse-proxies to. Two Caddys in series.
## SSH
Disabled by default. To enable:
1. Toggle SSH on in the ZimaOS web UI ("Settings → SSH"), OR
2. From the VM console, switch to TTY2 with **Alt+F2** and run `passwd` to set a root password first.
## Storage strategy
**Initial assumption (didn't work).** PVE 9's native virtiofs share with the host's existing `library` dir mapping (used by [108-haos](108-haos.md)) was the obvious path. Attaching it took one line — but ZimaOS's kernel ships *without* the virtiofs module (`modinfo virtiofs` → `Module virtiofs not found`). The kvm-side device was attached fine; the guest just can't mount it. Detached on 2026-05-14.
**What we ended up with.** A dedicated NFS re-export LXC [102-nfs-export](../containers/102-nfs-export.md) bind-mounts `/mnt/library` from the host and serves it over NFSv4 to ZimaOS. Performance is within ~2% of host-served NFS (LXC adds namespace isolation only, not an IO hop), with the upside that no NFS/RPC daemons run on the bare-metal host. Read-only during evaluation; promote to rw once the UI test confirms behavior.
**Filesystem layout inside ZimaOS.**`/etc/fstab` lives on a writable overlay so persistent mounts via fstab DO survive reboots, but the *standard* path to add a network share is the ZimaOS Files UI (Connect Network Storage → NFS). UID/GID mapping is squashed at the server to `www-data:media` (33:10000) — the same identity Nextcloud and mule-images use — so any rw activity from ZimaOS records as a normal `media`-group write.
**Caveats discovered during install.**
- Root filesystem is **squashfs / read-only**; only `/DATA` is writable.
-`dtoro` identity is `uid 999, gid 1000(samba), groups samba+wheel`. Primary group is **`samba`** (not `dtoro` — that group doesn't exist). SSH key needs `chown dtoro:samba` to satisfy sshd StrictModes.
-`dtoro`'s home directory is literally `/DATA` (not `/home/dtoro`). Authorized keys go at `/DATA/.ssh/authorized_keys`.
-`/etc` is a writable overlay rooted at `/mnt/overlay/upper_etc`. Mounts added to `/etc/fstab` persist across reboots.
-`mount.nfs4` is a stripped busybox variant — rejects `_netdev`, `nofail`, `actimeo=N`. The working minimal fstab line: `192.168.8.200:/mnt/library /media/library nfs rw,vers=4 0 0`. **Mount path matters** — see the 2026-05-15 Changelog entry on why the NFS export is mounted at `/media/library` and not `/DATA/library`.
- No `qemu-guest-agent` is bundled — `qm guest cmd` calls fail. IP discovery via `qm monitor` + `screendump` instead.
- No package manager at runtime — no `apt`, no `opkg`. Apps come via ZimaOS's CasaOS-style appstore only.
The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. `qm set 100 --scsi1 library:1024`) was rejected because it would start empty and force content migration; the NFS path keeps `/mnt/library` as the single source of truth shared with the existing 8 LXCs.
## Open items
- **DHCP → static IP fixed (2026-06-03).** ZimaOS IP drifted from `.195` (Slate AX) → `.103` (Technitium) after the DHCP migration, causing Caddy 502s. Fixed by injecting a static systemd-networkd config and restarting the VM. IP now pinned at `192.168.8.195`. See [changelog](#2026-06-03--static-ip-set-to-195-dhcp-drift-fixed).
- **No Authentik wiring.** [authentik (124)](../containers/106-auth-outpost.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
- **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up.
- **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor` → `screendump`.
## Related
- [108 — `haos-16.3`](108-haos.md) — the existing VM, same q35/OVMF pattern; also a virtiofs `library` consumer
- [DNS](../infrastructure/dns.md) — split-horizon entry lives here
- [Caddy (121)](../containers/121-caddy.md) — would front this if/when promoted
- [Media permissions](../infrastructure/media-permissions.md) — relevant when wiring `/mnt/library`
## Changelog
### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed
ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix)
Symptom: deleting any file or folder inside the NFS-served `library` tree from the ZimaOS Files UI failed with `rename ... invalid cross-device link`. Root cause: ZimaOS's "Move to Trash" is implemented as `rename(2)` into a `.trash/` directory derived from the *drive root*, and icewhale-files identifies the drive root with the regex `^/media/([^/]+)`. With the NFS mounted at `/DATA/library`, the visible UI path was `/media/ZimaOS-HD/library/foo` → drive `ZimaOS-HD` → trash at `/media/ZimaOS-HD/.trash` (which lives on the local ext4 `/dev/sda8`). NFS → ext4 rename = `EXDEV`.
Fix: remount the NFS export at `/media/library` so it sits as its own `/media/<name>` segment. Now icewhale-files extracts drive=`library`, computes trash at `/media/library/.trash`, and the rename stays within the NFS — no cross-device. The local-storage daemon (LSBLK-based) doesn't surface the NFS in the Files-app *sidebar* (drives sidebar only lists block devices), but the icewhale-files daemon picks the mount up as its own "storage" tile in the Files view, and crucially exposes a direct **Delete** action there (no trash flow, since ZimaOS treats non-LSBLK mounts as foreign and skips trash for them). For a 4 TB NFS share, immediate delete is arguably the right behavior anyway.
Concrete change:
-`/etc/fstab` line changed from `… /DATA/library nfs rw,vers=4 0 0` to `… /media/library nfs rw,vers=4 0 0`.
- New mountpoint dir `/var/lib/casaos_data/.media/library` (= `/media/library` via the existing `/media` bind).
- Old `/DATA/library` mountpoint dir removed.
Side effect: navigating via `ZimaOS-HD → library` in the UI now shows an empty/missing folder. Users access the library via the separate `library` tile in the Files app instead.
### 2026-05-14 — Mesh-reachable from netbird peers
Verified `https://zimaos.hubris.network` reachable from `republic-laptop` (100.122.78.140) through the existing hubris→netbird subnet advertisement (`192.168.8.0/24`). No new infrastructure required — the path was already wired by [mesh](../infrastructure/mesh.md). Open: configure a netbird Management nameserver group for `hubris.network` so peers don't need per-device `/etc/hosts` overrides.
### 2026-05-14 — Phase 2 promoted to rw
NFS export on [nfs-export (102)](../containers/102-nfs-export.md) flipped from `ro` to `rw` after Files UI evaluation passed. Verified the squash: a write from ZimaOS appears on hubris's `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention.
### 2026-05-14 — Phase 2: NFS mount via new LXC 102 (read-only)
Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs module (verified by `modinfo virtiofs` returning "Module not found"). Pivoted to NFSv4: built [nfs-export (102)](../containers/102-nfs-export.md) as a dedicated 512-MiB privileged Debian LXC bind-mounting `/mnt/library` and serving it on `192.168.8.200:2049` with `ro,all_squash,anonuid=33,anongid=10000` to the LAN subnet. Detached the now-useless `virtiofs0` from VM 100. ZimaOS-side mount via Files UI (NFS protocol, server `192.168.8.200`, path `/mnt/library`). Promote export to `rw` after Files UI evaluation passes.
### 2026-05-14 — Wired through Caddy (HTTPS)
Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`.
### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1)
`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/106-auth-outpost.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.
Native OIDC via the HACS integration `christiaangoossens/hass-oidc-auth` (repo `https://github.com/christiaangoossens/hass-oidc-auth`).
Key gotchas:
- HAOS containers don't honor the Network-panel DNS. Set Supervisor DNS via:
```
ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1"
```
so OIDC discovery resolves internally to [authentik (124)](../containers/106-auth-outpost.md).
- Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen).
- YAML config:
- `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created).
- `features.default_redirect: true` — skip the welcome-splash so users land on the normal HA login page.
- Run `ha core restart` after each config change — HA caches DNS and OIDC discovery across frontend reloads.
## Telemetry
HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting.
- see the services this host runs for action-level risk classes
## Recent changes
- (none yet)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
