From bb5c0717a10d15ee657067b716940097d945e091 Mon Sep 17 00:00:00 2001 From: dtoro Date: Mon, 6 Jul 2026 14:22:09 +0200 Subject: [PATCH 1/8] docs: adopt wiki-hq conventions + separate agent instructions (phases 1-2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem: the narrative docs lacked an enforceable style standard, and agent-facing instruction (OIKOS/CAVEMAN/CONTRIBUTING) was interleaved with human content at the repo root. Change: - Add .agents/shared/{writing-style,llm-wiki}.md — a lint-checkable prose standard (with an imperative-voice exception for runbooks/recipes) and the sources/wiki/index/log layer model. - Move CAVEMAN.md -> .agents/shared/caveman.md, CONTRIBUTING.md -> .agents/shared/page-templates.md, OIKOS.md -> .agents/OIKOS.md; leave thin root stubs so old links resolve. - Add .agents/domains/{knowledge,operations}/schema.md; operations schema codifies "plans always live in plans/". - Repoint live references (AGENTS, README, GLOSSARY, OIKOS) and fix OIKOS.md's internal relative links for its new depth. Risk: none to the operational substrate — inventory.yaml, hosts/*.yaml, oikos/, mcp/, secrets/, bin/ untouched (verified via git status). Verification: relative-link check across .agents/ clean; substrate churn empty. Co-Authored-By: Claude Opus 4.8 --- .agents/OIKOS.md | 227 +++++++++++++++++ .agents/domains/knowledge/schema.md | 48 ++++ .agents/domains/operations/schema.md | 50 ++++ .agents/shared/caveman.md | 33 +++ .agents/shared/llm-wiki.md | 41 ++++ .agents/shared/page-templates.md | 152 ++++++++++++ .agents/shared/writing-style.md | 75 ++++++ AGENTS.md | 13 +- CAVEMAN.md | 34 +-- CONTRIBUTING.md | 150 +----------- GLOSSARY.md | 4 +- OIKOS.md | 228 +----------------- README.md | 3 +- ...26-07-06-adopt-wiki-hq-doc-architecture.md | 164 +++++++++++++ 14 files changed, 814 insertions(+), 408 deletions(-) create mode 100644 .agents/OIKOS.md create mode 100644 .agents/domains/knowledge/schema.md create mode 100644 .agents/domains/operations/schema.md create mode 100644 .agents/shared/caveman.md create mode 100644 .agents/shared/llm-wiki.md create mode 100644 .agents/shared/page-templates.md create mode 100644 .agents/shared/writing-style.md create mode 100644 plans/2026-07-06-adopt-wiki-hq-doc-architecture.md diff --git a/.agents/OIKOS.md b/.agents/OIKOS.md new file mode 100644 index 0000000..715f169 --- /dev/null +++ b/.agents/OIKOS.md @@ -0,0 +1,227 @@ +# Oikos — the operating model + +Oikos (Greek: *household*) is the agent operating system layered on this +repo. It is not new infrastructure: `inventory.yaml` is the kernel data +structure, the `homelab` CLI and MCP server are the syscall surface, and +this page defines the rules everything above them follows. + +Read this after [AGENTS.md](../AGENTS.md). Machine-readable companions: +[oikos/ontology.yaml](../oikos/ontology.yaml) (systems model), +[oikos/policy.yaml](../oikos/policy.yaml) (risk & approval). + +## The kernel loop: OODA + +Every Oikos activity — scheduled probe, agent task, operator request — is +one pass through **Observe → Orient → Decide → Act**: + +1. **Observe** — probes, drift detectors, and agent findings produce + **Signals** (structured records, not loose messages): pending updates, + high temperature, low disk, service down, cert expiry, stale backup, + inventory drift. +2. **Orient** — walk the ontology graph: what entity is affected, what + depends on it (blast radius), its lifecycle state, whether a runbook + matches, what the ledger says about past attempts. +3. **Decide** — the classifier scores **risk class × blast radius × + confidence** and routes: + - **auto-act**: within autonomy policy, high confidence, contained radius + - **escalate**: operator approval via Matrix (✅/❌ reaction) or the + Oikos Console's `/approvals` page (destructive actions additionally + need a typed confirmation phrase either way) + - **queue**: informational — console + reports + The classifier can only *lower* autonomy relative to policy, never raise + it. When in doubt, escalate. +4. **Act** — execute through `homelab` commands or runbooks (never ad-hoc + SSH), then **verify** with the action's verification command, write a + **ledger** entry, resolve the Signal, and update docs in the same session. + +## Primitives + +| Primitive | What it is | Lives in | +|---|---|---| +| Host / Service | topology entities | `inventory.yaml` (+ generated `hosts/*.yaml`) | +| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` | +| Runbook | executable workflow with risk class + verification | `runbooks/` (Week 2) | +| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) | +| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) | +| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) | +| Incident | investigation narrative | `investigations/` | +| Plan | design doc for non-trivial work | `plans/` | +| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` | + +## Risk classes (enforced, not advisory) + +From [oikos/policy.yaml](../oikos/policy.yaml): + +- **read_only** — status, logs, docs, inventory. Unattended. +- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger. +- **config_mutation** — tracked-config edits (commit+push, never local), + deploys, upgrades, DNS/ingress changes. Operator approval. +- **destructive** — destroy, format, wipe, rotate, revoke. Approval + + typed confirmation phrase. + +Lifecycle gates modify these: `provisioning` nodes are freely mutable +(nothing depends on them); `deprecated` nodes accept no new dependents; +anything touching a `destroyed` node is drift. + +## The systems model + +Eight domains — physical, compute, network, storage, software, +identity & access, operations, external — cover everything in the lab; +entities are connected by typed edges (`hosts`, `provides`, `mounts`, +`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …) +defined in [oikos/ontology.yaml](../oikos/ontology.yaml). Rule of +completeness: **if it can break, be changed, or hold data, it has an +entity and edges.** Blast-radius questions ("what breaks if strong goes +down?") are graph walks, not doc archaeology. + +Nodes move through an explicit lifecycle — +`planned → provisioning → active → migrating → deprecated → destroyed` — +stored as `state:` in inventory (absent = active). Destroyed nodes live in +the `archaeology:` section. Each transition is a runbook checklist; +deprecation completes only when inbound edges reach zero. + +Generated views: [infrastructure/topology.md](../infrastructure/topology.md) +(Mermaid, regenerated from inventory) and the live, clickable version at +`oikos.hubris.network/graph` once the Console is deployed. + +## Conventions carried forward + +- Inventory is the truth; live state wins over narrative docs. +- Prefer `homelab` CLI and MCP over ad-hoc SSH. +- Meaningful changes update docs in the same session. +- Secrets are decrypted locally via per-client keys; never into docs/comments. +- Tracked configs change by commit + push, not local edits. +- Netbird is the preferred mesh path for new traffic. +- Agents are terse ([caveman.md](shared/caveman.md)), verify claims, and fix + collateral drift when found. + +## Build status (30-day roadmap, started 2026-07-05) + +- **Week 1**: policy, ontology, service contract, archaeology, topology + generator, this brief. Shipped. +- **Week 2**: context cards, `homelab service …`, change ledger, + `node relations`, runbooks. Shipped. +- **Week 3**: ops scheduler + state cache (`homelab service health` + is cache-first, `--live` forces a probe), drift detectors, signal engine + (`homelab signal …`), decision classifier (`homelab decide …`), approval + engine (`homelab approval …` — shared-HMAC grants; Matrix delivery is + Hermes's existing `@dtoro:avispero` send path, not a new bot, see + `oikos/approve.py`), daily brief + weekly report (`oikos/report.py`). + Shipped, except: Prometheus is still `planned` (see + [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md)) — + trend signals (disk-full prediction, temp creep) wait on that LXC; the + scheduler's disk check today is point-in-time only, and CPU/NVMe + temperature isn't probed at all yet (no confirmed sensor path on + hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness + drift checks are also deferred (see `oikos/drift.py` docstring). +- **Week 4**: Oikos Console v0 shipped — signals landing page, service + grid + detail, node/blast-radius view, live Mermaid graph, drift view, + approvals queue (approve/deny, destructive confirmation-phrase + enforced), daily/weekly reports. Server-rendered FastAPI + Jinja2, no + SPA build chain, tested end-to-end against live production data (see + `oikos/console/`). Deploys as a third webhook on `dtoro/Homelab-Docs` + (`/opt/oikos-console`, port :9831) — see + [oikos/console/deploy/README.md](../oikos/console/deploy/README.md) for + the Caddy route and Gitea webhook registration this repo can't do for + itself. Approval grants are now single-use (a second `check_grant` call + for the same request fails even within the TTL) and already exact-bound + to request id + entity + action. + **Not shipped as originally planned:** per-agent *age-key-signed* + request authentication — age has no signing primitive (it's an + encryption-only keypair format), so "age-key-signed" wasn't + buildable as stated. The real alternative (SSH-key signing via + `ssh-keygen -Y sign`/`-Y verify`, using each host's already-provisioned + SSH key) is real and buildable, but needs SSH public keys recorded in + inventory first — not there today. Moved to the 60/90-day backlog. + Authentik step-up re-auth on the approve/deny route is documented but + needs a live Authentik instance to configure — also backlog. + Docs pass done (this file, AGENTS.md, operations/commands.md); found + and fixed two more stale references while at it (DNS section still + pointed at destroyed LXC 124/dnsmasq instead of Technitium on 107, and + a `claudio-monitor` reference that's been deprecated since 2026-06-04). + +### Real drift found while building Week 3 (unresolved, needs operator action) + +The drift detectors surfaced genuine, currently-true findings on first +run against production — recorded here rather than silently fixed, since +each is a `config_mutation`/`destructive`-class decision: + +- `republic-laptop` has no `age_pubkey:` in `inventory.yaml`, but its real + age key is granted on nearly every shared secret in `.sops.yaml` + (`age1vf8h7...`) — the enrollment write-back to inventory never + happened. Fix: `homelab client add republic-laptop --finalize-pubkey + age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6`. +- `grimmory` has an `age_pubkey` in inventory but is missing from + `secrets/hello.yaml`'s recipient list — incomplete enrollment the + other direction. Fix: re-run `homelab client add grimmory + --finalize-pubkey `. +- `pve_id 131` exists live on hubris (`pct list`) with no inventory entry + — investigate before assuming it's a stale ID (see the Prometheus LXC + plan doc above, which flags this explicitly). +- Three `lifecycle-pve-id-reuse` info findings (100, 106, 107 each shared + between an active host and an archaeology entry) — expected/benign ID + reuse after destroy, no action needed. + +## 60/90-day backlog + +Derived from gaps observed while building the 30-day roadmap, not +guesswork. Roughly ordered by what unblocks the most: + +- **Fix the oikos-console deploy webhook's signature mismatch.** Console + is live on apps (105) via a manual `deploy.sh` run, but Gitea webhook + 14's deliveries all 403 with a signature mismatch for a cause not yet + found — the secret is confirmed synced correctly on both sides + (rotated once already to rule out drift). Until fixed, `git push` + doesn't auto-redeploy the console the way it does for homelab-mcp/ + secrets-issuance; re-run `deploy.sh` on apps manually after changes. + See [oikos/console/deploy/README.md](../oikos/console/deploy/README.md). +- **SSH-key-signed approval requests.** Replaces the design note in + Week 4: age keys can't sign (encryption-only format), so per-agent + request authentication needs `ssh-keygen -Y sign`/`-Y verify` against + each host's existing SSH key. Blocked on a schema gap: inventory + doesn't record SSH public keys today, only ports/users. First step is + populating that field on enrollment, then wiring `oikos/approve.py` to + require and verify a signature over the request payload. +- **Authentik step-up re-auth** on the Console's `/approvals` POST route + — needs a live Authentik `PromptStage`/reauth flow scoped to that path; + not configurable without a running instance to test against. +- **Prometheus provisioning** (see + [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md)) + — unblocks trend signals (disk-full prediction, temp creep) and real + sparklines in the Console; investigate the undocumented `pve_id 131` + on hubris first. +- **CPU/NVMe temperature probing** in the scheduler — needs a confirmed + sensor path on hubris and strong (lm-sensors vs vendor tool) before a + real check can be written; guessing one risks a probe that silently + never fires. +- **DNS-vs-inventory drift check** — compare Technitium zone records + against `services.*.url`/`public_host`; not implemented (`oikos/drift.py` + has no Technitium API wiring yet). +- **Generic tracked-config-cleanliness drift check** — today only caddy's + `/etc/caddy` git-checkout path is hardcoded in `oikos/drift.py`; every + other service with a `config_repo` needs its local checkout path + recorded (a `mutation_path`-style field, same gap Week 1's service + contract flagged but didn't backfill) before this generalizes. +- **Per-service policy overrides** (`oikos/policy.yaml` + `service_overrides`) — schema is ready (caddy/dns already use it); + populate more as specific services turn out to need non-default risk + classes. +- **Incident timeline generator** — stitch ledger + signal history into + a single narrative for `investigations/` entries instead of writing + them by hand. +- **Secret access audit** — who-can-decrypt-what report from + `.sops.yaml` + inventory `age_pubkey`s, extending what + `oikos/drift.py`'s SOPS check already partially does. +- **Restore drills** — exercise `backs-up-to` (once populated) by + actually restoring from a backup target on a schedule, not just + checking freshness. +- **Multi-agent delegation model** — more than one agent acting + concurrently; needs the ledger's `agent` field to carry real identity + (age pubkey, not just hostname) consistently, which it mostly does + already but hasn't been stress-tested with concurrent writers. +- **Grafana** — only if the Console's own Prometheus-backed sparklines + turn out to be insufficient once Prometheus ships. +- **"Generalize later" extraction** — the original decision was personal- + first, generalize-later (see Week 1). Once patterns stabilize, extract + a config-driven Oikos core with no `hubris.network`/`hubris`/`strong` + hardcoding, so it's installable on a different homelab. diff --git a/.agents/domains/knowledge/schema.md b/.agents/domains/knowledge/schema.md new file mode 100644 index 0000000..e16071b --- /dev/null +++ b/.agents/domains/knowledge/schema.md @@ -0,0 +1,48 @@ +# Knowledge domain — schema + +The knowledge domain is the durable, authoritative current-state documentation of the homelab: one +page per node and per cross-cutting system, synthesized from live state and evidence. It answers +"what exists and how does it work right now." + +It follows the [LLM Wiki layer model](../../shared/llm-wiki.md) and the +[writing-style](../../shared/writing-style.md) and [page-templates](../../shared/page-templates.md) +rules. + +## The narrative / substrate split + +The knowledge wiki is **narrative**. It sits alongside a **machine-readable substrate** that it +describes but never contains. The split is load-bearing: several programs read the substrate at +fixed paths, so the wiki reorganization never moves it. + +| Layer | Location | Consumed by | +|-------|----------|-------------| +| Substrate — source of truth | `inventory.yaml` (root) | MCP server, `homelab` CLI, `oikos/` scheduler/drift/relations/gen-topology | +| Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` | +| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler | +| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` | +| Evidence — immutable sources | `knowledge/sources/`, `investigations/` | synthesis into wiki pages | + +## Wiki pages + +- **Node pages** (`knowledge/wiki/containers/-.md`, `.../vms/-.md`, + `.../hosts/.md`) follow the container/host template in + [page-templates.md](../../shared/page-templates.md): opening definition, `## At a glance`, + `## Role`, service/port map, storage, auto-deploy, `## Related`, `## Changelog`. +- **Cross-cutting pages** (`knowledge/wiki/infrastructure/.md`) follow the cross-cutting + template: `## Why`, `## Components`, `## How to apply`, `## Gotchas`, `## Related`, `## Changelog`. +- Each `inventory.yaml` host entry carries a `doc_page:` field pointing at its narrative page. + Changing where a page lives means updating that field (read by `bin/homelab`). + +## The two logs + +- The per-page **`## Changelog`** records infrastructure changes and is machine-parsed + (`get_changelog`, the Oikos ledger). Keep the `### YYYY-MM-DD — title` shape. +- **`knowledge/log.md`** is append-only and records *documentation-maintenance* operations only + (restructures, source ingests, lint sweeps): `## [YYYY-MM-DD] | `. It never + duplicates the Oikos change ledger (`oikos/ledger.py`). + +## Same-session update rule + +A change to a node updates every page that references it in the same session — the node page, the +section `README.md` table, the root `README.md`, the Caddy/DNS/ingress pages, the host page, and +`inventory.yaml`. See [page-templates.md](../../shared/page-templates.md#same-session-update-rule). diff --git a/.agents/domains/operations/schema.md b/.agents/domains/operations/schema.md new file mode 100644 index 0000000..ba8ca84 --- /dev/null +++ b/.agents/domains/operations/schema.md @@ -0,0 +1,50 @@ +# Operations domain — schema + +The operations domain holds the procedural and time-stamped documentation: runbooks (repeatable +procedures), investigations (incident evidence), and plans (design docs for non-trivial work). It +follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice +exception. + +## Plans always live in `plans/` + +**Any plan or design doc for the Homelab is written into the repo `plans/` folder as +`plans/YYYY-MM-DD-slug.md` — never a scratch path, an agent-private plan location, or a chat +message.** An agent drafting a plan: + +1. Writes the file under `plans/` using the plan template in [page-templates.md](../../shared/page-templates.md). +2. Lists it in `plans/index.md`. +3. On completion, moves it to `plans/done/` and updates the index status. + +This is the single source for homelab design intent; keeping it in-repo means the plan is +versioned, reviewable, and reachable by MCP `get_page`/`search_docs` like any other doc. + +## Runbooks + +Repeatable procedures live in `runbooks/.md` with YAML front-matter that the Oikos policy and +lifecycle machinery reads: + +```yaml +--- +name: +risk_class: read_only | reversible_low | config_mutation | destructive +inputs: [, ...] +verification: "" +docs_update_checklist: [] +transition: " -> " # only for lifecycle runbooks +--- +``` + +`risk_class` values and the lifecycle `transition` states must match +[`oikos/policy.yaml`](../../../oikos/policy.yaml) and [`oikos/ontology.yaml`](../../../oikos/ontology.yaml). + +## Investigations + +Incident records live in `investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written +once at incident time, then linked from the changelogs of the nodes they implicate. Sections: +`## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved +incidents move to `investigations/archive/`. + +## The operations log + +`plans/log.md` and `investigations/log.md` are append-only records of documentation operations on +those areas (`## [YYYY-MM-DD] | `), distinct from the Oikos change ledger. diff --git a/.agents/shared/caveman.md b/.agents/shared/caveman.md new file mode 100644 index 0000000..a6225d4 --- /dev/null +++ b/.agents/shared/caveman.md @@ -0,0 +1,33 @@ +# CAVEMAN.md — communication mode for homelab agents + +Respond terse like smart caveman. All technical substance stay. Only fluff die. + +## Rules + +Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact. + +Pattern: `[thing] [action] [reason]. [next step].` + +Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..." +Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:" + +## Levels + +- **lite** — no filler/hedging. Keep articles + full sentences. Professional but tight. +- **full** (default) — drop articles, fragments OK, short synonyms. Classic caveman. +- **ultra** — abbreviate prose words (DB/auth/config/req/res), strip conjunctions, arrows (X → Y). Code symbols/API names/errors: never abbreviate. + +Switch: `/caveman lite|full|ultra`. Stop: "normal mode". + +## Auto-Clarity + +Drop caveman for: security warnings, irreversible actions, multi-step sequences where fragments risk misread, user confused/repeating. Resume after clear part done. + +## Boundaries + +Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end. + +--- + +Source: https://github.com/JuliusBrussee/caveman +Copy to `~/.hermes/skills/` for Hermes Agent, or `~/.claude/projects//SKILL.md` for Claude Code. diff --git a/.agents/shared/llm-wiki.md b/.agents/shared/llm-wiki.md new file mode 100644 index 0000000..1d6f69f --- /dev/null +++ b/.agents/shared/llm-wiki.md @@ -0,0 +1,41 @@ +# LLM Wiki — the documentation contract + +How the narrative documentation in this repo is organized. The pattern is borrowed from the +`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top +of immutable evidence (`knowledge/sources/`, `investigations/`), with pure-listing indexes and an +append-only operations log. + +This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`, +generated `hosts/*.yaml`, `oikos/`, `mcp/`, `secrets/`, `bin/` — is not part of the wiki and never +moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for the split. + +## Layers + +- **Sources** are immutable raw material: incident records (`investigations/`), external reference + docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`). + Read them; do not rewrite them into other sources. +- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per + node (`containers/`, `vms/`, host narratives) and per cross-cutting system (`infrastructure/`). A + reader understands the topic from the wiki page without reading the sources. +- **Index** (`index.md` / folder `README.md`) is a pure listing — every page in scope with a + one-line summary, and nothing else. Anything the section wants to say up front goes into a page + the index lists, not into the index. +- **Log** (`log.md`) is append-only, recording *doc-maintenance operations* (restructures, source + ingests, lint sweeps) in single-line format: `## [YYYY-MM-DD] | `. + +## Two logs, kept distinct + +- **`## Changelog`** on each node/topic page records *infrastructure* changes to that node. It is + machine-parsed (`get_changelog`, the Oikos ledger) — keep the `### YYYY-MM-DD — title` shape. +- **`log.md`** per area records *documentation* operations only. It never duplicates the Oikos + change ledger (`oikos/ledger.py`), which stays authoritative for infra changes with + who/what/risk/approval/verification. + +## Rules + +- Wiki pages stay short and focused. A page past ~300 lines splits. +- Pages stay flat under `wiki/
/` until there are enough to warrant a sub-group. +- Every page follows [writing-style.md](writing-style.md). +- Plans and design docs always live in the repo `plans/` folder (`plans/YYYY-MM-DD-slug.md`), + listed in `plans/index.md`, moved to `plans/done/` on completion — never a scratch path or a chat + message. See [the operations schema](../domains/operations/schema.md). diff --git a/.agents/shared/page-templates.md b/.agents/shared/page-templates.md new file mode 100644 index 0000000..c848bc3 --- /dev/null +++ b/.agents/shared/page-templates.md @@ -0,0 +1,152 @@ +# Page templates for the Homelab Wiki + +The structural templates for each page type. Prose voice, vocabulary, and cross-reference rules live +in [writing-style.md](writing-style.md); the layer model (sources / wiki / index / log) lives in +[llm-wiki.md](llm-wiki.md). + +## Voice + +Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. Full rules in +[writing-style.md](writing-style.md). + +## Page templates + +### Container page (`containers/-.md`) + +```markdown +# — `` + +One-sentence purpose. + +## At a glance +- **Hostname:** `` +- **IP:** `192.168.8.x` +- **Privilege:** privileged | unprivileged +- **Resources:** N cores / M GiB RAM / D GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` (if any) +- **Public hostname:** `.hubris.network` (if proxied) + +## Role +What it does, what it talks to. + +## Service / port map +| Service | Listen | Notes | + +## Storage / config paths + +## Auto-deploy +(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md) + +## Related +- [Caddy](121-caddy.md) (if proxied) +- [DNS](../infrastructure/dns.md) (if has subdomain) +- [Authentik](124-authentik.md) (if SSO) +- ... + +## Changelog +### YYYY-MM-DD — short title +What changed, why, link to investigation if any. +``` + +### Cross-cutting page (`infrastructure/.md`) + +```markdown +# + +One-sentence summary. + +## Why +Design rationale — what it replaces, what it solves. + +## Components +Where it runs, what files matter. + +## How to apply / use +Recipes. + +## Gotchas + +## Related +Links to nodes that host or depend on this. + +## Changelog +``` + +### Plan (`plans/YYYY-MM-DD-slug.md`) + +```markdown +# YYYY-MM-DD — + +## Goal +What this change achieves and why. + +## Current topology / state +Diagram or description of what exists now. + +## Target topology / state +What it looks like after. + +## Pre-flight checklist + +## Step-by-step procedure + +## Verification + +## Post-migration +Changelog entries to write, index status to update. +``` + +### Investigation (`investigations/YYYY-MM-DD-slug.md`) + +```markdown +# YYYY-MM-DD — <title> + +## Summary +1-3 sentences. + +## Timeline + +## Root cause + +## Mitigations applied + +## Open questions +``` + +## Linking discipline + +- Every container page links to every cross-cutting page it participates in. +- Every cross-cutting page lists the nodes that participate. +- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog. +- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages. + +## Changelog hygiene + +- Reverse-chronological (newest first). +- One entry per discrete change, even if you make several in one day. +- If a change spans nodes, repeat the entry on each affected page (different perspective is fine). +- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them. + +## Same-session update rule + +When you make a change to a node — migrate an LXC, update an IP, change a +mount, deploy a new service — **update every relevant doc page in the same +session.** A change that touches a container page must also update: + +- The `containers/index.md` table (IPs, host, mounts, status) +- The `README.md` table (if the change affects listed columns) +- The Caddy page site list (if the change affects `*.hubris.network` routing) +- The DNS / ingress infrastructure pages (if the change affects routing) +- The `hosts/{hubris,strong}.md` host page (if container count changes) +- The `inventory.yaml` host entry (source of truth for the `hosts/*.yaml` generation) +- The `infrastructure/topology.md` (generated from inventory, but regen if needed) + +The pattern of updating only one page and leaving stale references on others +is a bug. If you're doing a multi-step migration, document the intermediate +state with a changelog entry that says "pending — will finalize after Phase +N." + +This rule is why Phase 2 of the strong migration (2026-07-05) caused +widespread stale data: individual container pages were updated in the +changelog but never had their At-a-glance sections, IPs, mount paths, or +host attribution updated. Don't repeat that. diff --git a/.agents/shared/writing-style.md b/.agents/shared/writing-style.md new file mode 100644 index 0000000..95da2d6 --- /dev/null +++ b/.agents/shared/writing-style.md @@ -0,0 +1,75 @@ +# Writing Style + +Write like a technical reference, not a marketing page. Every sentence conveys new information. +These rules govern **committed documentation** — wiki pages, READMEs, schemas, skills, `AGENTS.md`, +plans, investigations, and code comments. They are separate from [caveman.md](caveman.md), which +governs an agent's *chat responses*; the two do not conflict. + +New or rewritten pages follow these patterns from day one. Existing pages get updated the next time +they are touched. + +## Vocabulary — never use these + +- Significance puffers: "pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament", "paramount", "invaluable". +- Analytical verbs: "delve", "leverage", "utilize", "facilitate", "foster", "showcase", "underscore", "streamline", "harness". +- Poetic nouns: "tapestry", "landscape" (figurative), "realm", "paradigm", "ecosystem" (figurative), "journey" (figurative), "nexus", "cornerstone". +- Promotional adjectives: "robust", "seamless", "innovative", "cutting-edge", "meticulous", "holistic", "comprehensive". +- Opening crutches: "In today's world", "In the ever-evolving landscape of", "It's worth noting that", "It is important to note that". + +Use short, common words: "use" not "utilize", "help" not "facilitate", "show" not "demonstrate". + +## Voice + +Describe what systems do and how they work. + +- **Reference prose** (node pages, cross-cutting infrastructure descriptions, `## Role`, `## Why`, + `At a glance`) is third-person: state facts about the system, not instructions to a reader. +- **Recipes, runbooks, and skills** are the exception: second-person imperative is allowed and + preferred where it makes a procedure clearer ("Edit the Caddyfile, commit + push", "Verify with + `dig +short`"). This matches how the operator actually works. The vocabulary, structure, and + cross-reference rules below still apply. + +## Page shape + +Every doc-level page follows the same shape so a reader scans it in one pass. + +1. **One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`. +2. **Opening definition.** First paragraph, 1–3 sentences, says what the thing is. No motivation, no marketing, no setup. +3. **Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md). +4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (`get_changelog` in `mcp/server.py`); keep the `### YYYY-MM-DD — title` shape. +5. **Related links** only at the bottom, only when a reference cannot be woven inline. + +## Section indexes (folder READMEs) + +A folder's `README.md` opens with a 1–3 sentence prose intro that says what the section covers, then +a single navigation table — `| Document | What it covers |` — and nothing else. No stale counts, no +duplicated prose, no narrative between the intro and the table. + +## Structure rules + +- Make every sentence information-dense. Cut filler, qualifiers, and setup phrases. Lead with the concrete fact or action, not why it matters. +- No participial tack-ons (", highlighting the importance of…"). If the clause adds information, make it a separate sentence. +- **No meta-commentary about the content itself.** Do not narrate the page's own structure or linking strategy. +- Prefer **tables** for enumerable items with internal structure (service/port maps, field lists, status grids). Reserve bullets for short non-structured lists. +- Use the **bold-leading-phrase pattern** for structured points: `**Read-only by construction.** The MCP server never mutates state.` — a bold noun phrase, a period, then the explanation. +- When enumerating across services or nodes, give each its own `###` sub-section or a table row, not one run-on paragraph. +- Use backticks for code, paths, hostnames, and file names (`inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology. +- Use `>` blockquotes for caveats and gaps that interrupt the main flow: `> **Outstanding gap.** DNS-vs-inventory drift check not yet wired.` One thought per blockquote. + +## Diagrams + +- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it. +- ASCII box diagrams are fine for small shape diagrams; keep them to one screen. + +## Sourcing and cross-references + +- **Factual discipline.** Every claim is grounded in a cited source, an adjacent linked page, or a directly observable fact (`pct config`, `docker inspect`, running config). Do not write sentences that sound sourced but are inference. When docs disagree with live state, fix the doc and note it in the changelog. +- **One-sided cross-references.** When two pages relate, the link lives in the page where the connection makes organizational sense. Do not add a back-pointer unless that direction also carries content the reader needs. +- **Cross-references are content, not catalog.** Inline links arise from the surrounding prose; the linked page must be needed to understand the current sentence. A bottom-of-page "Related" list is the fallback, not the default. +- Pages link with standard relative markdown links (e.g. a container page links to `../infrastructure/dns.md`), forming a navigable graph. Orphans are a bug. + +## Code comments and commit/PR prose + +- Comments explain intent, trade-offs, or constraints the code cannot convey. No diff narration, no type restatement, no section-divider comments. +- Commit messages and PR descriptions are problem → change → risk → verification, not a file-by-file diff restatement. +- The banned vocabulary applies the same way in comments and commit messages. diff --git a/AGENTS.md b/AGENTS.md index c9bfd93..8349188 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,10 +5,19 @@ context is in this checkout at `/opt/homelab-context/`. This file is the entry point. Read it once at start, then keep working. The operating model — OODA loop, risk classes, approval rules, the ontology, -and node lifecycle — is defined in [OIKOS.md](OIKOS.md). Before any mutation, +and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation, classify the action against `oikos/policy.yaml`; when the class requires approval, stop and ask the operator. +Agent-facing instruction is separated from human content under `.agents/`: +`.agents/shared/` holds the conventions every agent applies +([writing-style](.agents/shared/writing-style.md), [caveman](.agents/shared/caveman.md), +[page-templates](.agents/shared/page-templates.md), [llm-wiki](.agents/shared/llm-wiki.md)), and +`.agents/domains/` holds the per-domain schemas +([knowledge](.agents/domains/knowledge/schema.md), [operations](.agents/domains/operations/schema.md)). +The narrative wiki lives under `knowledge/wiki/`; the machine-readable substrate +(`inventory.yaml`, `hosts/*.yaml`, `oikos/`) stays at the repo root. + ## 1. Who you are Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read: @@ -102,7 +111,7 @@ Grep is fine for browsing or when MCP is unreachable. ## 6. Communication mode -Read and apply `/opt/homelab-context/CAVEMAN.md` (if present). It defines the lab's +Read and apply `/opt/homelab-context/.agents/shared/caveman.md` (if present). It defines the lab's terse-communication standard — drop filler, keep substance, use fragments. ## 7. Auto-setup mechanism diff --git a/CAVEMAN.md b/CAVEMAN.md index a6225d4..1660494 100644 --- a/CAVEMAN.md +++ b/CAVEMAN.md @@ -1,33 +1,3 @@ -# CAVEMAN.md — communication mode for homelab agents +# CAVEMAN.md — moved -Respond terse like smart caveman. All technical substance stay. Only fluff die. - -## Rules - -Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact. - -Pattern: `[thing] [action] [reason]. [next step].` - -Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..." -Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:" - -## Levels - -- **lite** — no filler/hedging. Keep articles + full sentences. Professional but tight. -- **full** (default) — drop articles, fragments OK, short synonyms. Classic caveman. -- **ultra** — abbreviate prose words (DB/auth/config/req/res), strip conjunctions, arrows (X → Y). Code symbols/API names/errors: never abbreviate. - -Switch: `/caveman lite|full|ultra`. Stop: "normal mode". - -## Auto-Clarity - -Drop caveman for: security warnings, irreversible actions, multi-step sequences where fragments risk misread, user confused/repeating. Resume after clear part done. - -## Boundaries - -Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end. - ---- - -Source: https://github.com/JuliusBrussee/caveman -Copy to `~/.hermes/skills/` for Hermes Agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code. +Agent chat-mode rules now live at [`.agents/shared/caveman.md`](.agents/shared/caveman.md). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7706538..df5760e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,147 +1,7 @@ -# Contributing to the Homelab Wiki +# CONTRIBUTING.md — moved -## Voice +Documentation conventions are now split across `.agents/shared/`: -Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. - -## Page templates - -### Container page (`containers/<id>-<name>.md`) - -```markdown -# <id> — `<name>` - -One-sentence purpose. - -## At a glance -- **Hostname:** `<name>` -- **IP:** `192.168.8.x` -- **Privilege:** privileged | unprivileged -- **Resources:** N cores / M GiB RAM / D GiB rootfs -- **Mounts:** `/mnt/library` ↔ `/mnt/library` (if any) -- **Public hostname:** `<sub>.hubris.network` (if proxied) - -## Role -What it does, what it talks to. - -## Service / port map -| Service | Listen | Notes | - -## Storage / config paths - -## Auto-deploy -(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md) - -## Related -- [Caddy](121-caddy.md) (if proxied) -- [DNS](../infrastructure/dns.md) (if has subdomain) -- [Authentik](124-authentik.md) (if SSO) -- ... - -## Changelog -### YYYY-MM-DD — short title -What changed, why, link to investigation if any. -``` - -### Cross-cutting page (`infrastructure/<topic>.md`) - -```markdown -# <Topic> - -One-sentence summary. - -## Why -Design rationale — what it replaces, what it solves. - -## Components -Where it runs, what files matter. - -## How to apply / use -Recipes. - -## Gotchas - -## Related -Links to nodes that host or depend on this. - -## Changelog -``` - -### Plan (`plans/YYYY-MM-DD-slug.md`) - -```markdown -# YYYY-MM-DD — <title> - -## Goal -What this change achieves and why. - -## Current topology / state -Diagram or description of what exists now. - -## Target topology / state -What it looks like after. - -## Pre-flight checklist - -## Step-by-step procedure - -## Verification - -## Post-migration -Changelog entries to write, index status to update. -``` - -### Investigation (`investigations/YYYY-MM-DD-slug.md`) - -```markdown -# YYYY-MM-DD — <title> - -## Summary -1-3 sentences. - -## Timeline - -## Root cause - -## Mitigations applied - -## Open questions -``` - -## Linking discipline - -- Every container page links to every cross-cutting page it participates in. -- Every cross-cutting page lists the nodes that participate. -- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog. -- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages. - -## Changelog hygiene - -- Reverse-chronological (newest first). -- One entry per discrete change, even if you make several in one day. -- If a change spans nodes, repeat the entry on each affected page (different perspective is fine). -- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them. - -## Same-session update rule - -When you make a change to a node — migrate an LXC, update an IP, change a -mount, deploy a new service — **update every relevant doc page in the same -session.** A change that touches a container page must also update: - -- The `containers/index.md` table (IPs, host, mounts, status) -- The `README.md` table (if the change affects listed columns) -- The Caddy page site list (if the change affects `*.hubris.network` routing) -- The DNS / ingress infrastructure pages (if the change affects routing) -- The `hosts/{hubris,strong}.md` host page (if container count changes) -- The `inventory.yaml` host entry (source of truth for the `hosts/*.yaml` generation) -- The `infrastructure/topology.md` (generated from inventory, but regen if needed) - -The pattern of updating only one page and leaving stale references on others -is a bug. If you're doing a multi-step migration, document the intermediate -state with a changelog entry that says "pending — will finalize after Phase -N." - -This rule is why Phase 2 of the strong migration (2026-07-05) caused -widespread stale data: individual container pages were updated in the -changelog but never had their At-a-glance sections, IPs, mount paths, or -host attribution updated. Don't repeat that. +- [`page-templates.md`](.agents/shared/page-templates.md) — per-page-type structural templates, linking discipline, changelog hygiene, same-session update rule. +- [`writing-style.md`](.agents/shared/writing-style.md) — prose voice, banned vocabulary, page shape. +- [`llm-wiki.md`](.agents/shared/llm-wiki.md) — the sources / wiki / index / log layer model. diff --git a/GLOSSARY.md b/GLOSSARY.md index 624b77f..b88f94f 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -18,7 +18,7 @@ Terms and abbreviations used throughout the homelab wiki. | **Mesh** | Overlay VPN for off-LAN connectivity. Netbird is current; Tailscale is legacy | | **Netbird** | Preferred mesh VPN. VPS hosts the management plane; all homelab nodes are members | | **OIDC** | OpenID Connect. Protocol used by Authentik for SSO login flows | -| **Oikos** | Agent operating model (OIKOS.md). OODA loop, risk classes, policy, ontology | +| **Oikos** | Agent operating model ([.agents/OIKOS.md](.agents/OIKOS.md)). OODA loop, risk classes, policy, ontology | | **PVE** | Proxmox Virtual Environment — the hypervisor on both hubris and strong | | **SOPS** | `sops` — Mozilla SOPS. Encrypts secrets with age keys so they live in the git repo | | **Strong** | Secondary Proxmox VE node. Cluster member 2 (hostname `strong`, nickname ludo/ludo-mini) | @@ -30,4 +30,4 @@ Terms and abbreviations used throughout the homelab wiki. ## See also - [Infrastructure index](infrastructure/index.md) — cross-cutting systems each with their own doc page -- [OIKOS operating model](OIKOS.md) — agent policy, risk classes, lifecycle \ No newline at end of file +- [OIKOS operating model](.agents/OIKOS.md) — agent policy, risk classes, lifecycle \ No newline at end of file diff --git a/OIKOS.md b/OIKOS.md index 2746ecf..bc6bb92 100644 --- a/OIKOS.md +++ b/OIKOS.md @@ -1,227 +1,3 @@ -# Oikos — the operating model +# OIKOS.md — moved -Oikos (Greek: *household*) is the agent operating system layered on this -repo. It is not new infrastructure: `inventory.yaml` is the kernel data -structure, the `homelab` CLI and MCP server are the syscall surface, and -this page defines the rules everything above them follows. - -Read this after [AGENTS.md](AGENTS.md). Machine-readable companions: -[oikos/ontology.yaml](oikos/ontology.yaml) (systems model), -[oikos/policy.yaml](oikos/policy.yaml) (risk & approval). - -## The kernel loop: OODA - -Every Oikos activity — scheduled probe, agent task, operator request — is -one pass through **Observe → Orient → Decide → Act**: - -1. **Observe** — probes, drift detectors, and agent findings produce - **Signals** (structured records, not loose messages): pending updates, - high temperature, low disk, service down, cert expiry, stale backup, - inventory drift. -2. **Orient** — walk the ontology graph: what entity is affected, what - depends on it (blast radius), its lifecycle state, whether a runbook - matches, what the ledger says about past attempts. -3. **Decide** — the classifier scores **risk class × blast radius × - confidence** and routes: - - **auto-act**: within autonomy policy, high confidence, contained radius - - **escalate**: operator approval via Matrix (✅/❌ reaction) or the - Oikos Console's `/approvals` page (destructive actions additionally - need a typed confirmation phrase either way) - - **queue**: informational — console + reports - The classifier can only *lower* autonomy relative to policy, never raise - it. When in doubt, escalate. -4. **Act** — execute through `homelab` commands or runbooks (never ad-hoc - SSH), then **verify** with the action's verification command, write a - **ledger** entry, resolve the Signal, and update docs in the same session. - -## Primitives - -| Primitive | What it is | Lives in | -|---|---|---| -| Host / Service | topology entities | `inventory.yaml` (+ generated `hosts/*.yaml`) | -| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` | -| Runbook | executable workflow with risk class + verification | `runbooks/` (Week 2) | -| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) | -| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) | -| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) | -| Incident | investigation narrative | `investigations/` | -| Plan | design doc for non-trivial work | `plans/` | -| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` | - -## Risk classes (enforced, not advisory) - -From [oikos/policy.yaml](oikos/policy.yaml): - -- **read_only** — status, logs, docs, inventory. Unattended. -- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger. -- **config_mutation** — tracked-config edits (commit+push, never local), - deploys, upgrades, DNS/ingress changes. Operator approval. -- **destructive** — destroy, format, wipe, rotate, revoke. Approval + - typed confirmation phrase. - -Lifecycle gates modify these: `provisioning` nodes are freely mutable -(nothing depends on them); `deprecated` nodes accept no new dependents; -anything touching a `destroyed` node is drift. - -## The systems model - -Eight domains — physical, compute, network, storage, software, -identity & access, operations, external — cover everything in the lab; -entities are connected by typed edges (`hosts`, `provides`, `mounts`, -`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …) -defined in [oikos/ontology.yaml](oikos/ontology.yaml). Rule of -completeness: **if it can break, be changed, or hold data, it has an -entity and edges.** Blast-radius questions ("what breaks if strong goes -down?") are graph walks, not doc archaeology. - -Nodes move through an explicit lifecycle — -`planned → provisioning → active → migrating → deprecated → destroyed` — -stored as `state:` in inventory (absent = active). Destroyed nodes live in -the `archaeology:` section. Each transition is a runbook checklist; -deprecation completes only when inbound edges reach zero. - -Generated views: [infrastructure/topology.md](infrastructure/topology.md) -(Mermaid, regenerated from inventory) and the live, clickable version at -`oikos.hubris.network/graph` once the Console is deployed. - -## Conventions carried forward - -- Inventory is the truth; live state wins over narrative docs. -- Prefer `homelab` CLI and MCP over ad-hoc SSH. -- Meaningful changes update docs in the same session. -- Secrets are decrypted locally via per-client keys; never into docs/comments. -- Tracked configs change by commit + push, not local edits. -- Netbird is the preferred mesh path for new traffic. -- Agents are terse ([CAVEMAN.md](CAVEMAN.md)), verify claims, and fix - collateral drift when found. - -## Build status (30-day roadmap, started 2026-07-05) - -- **Week 1**: policy, ontology, service contract, archaeology, topology - generator, this brief. Shipped. -- **Week 2**: context cards, `homelab service <name> …`, change ledger, - `node relations`, runbooks. Shipped. -- **Week 3**: ops scheduler + state cache (`homelab service <name> health` - is cache-first, `--live` forces a probe), drift detectors, signal engine - (`homelab signal …`), decision classifier (`homelab decide …`), approval - engine (`homelab approval …` — shared-HMAC grants; Matrix delivery is - Hermes's existing `@dtoro:avispero` send path, not a new bot, see - `oikos/approve.py`), daily brief + weekly report (`oikos/report.py`). - Shipped, except: Prometheus is still `planned` (see - [plans/2026-07-05-oikos-prometheus-lxc.md](plans/2026-07-05-oikos-prometheus-lxc.md)) — - trend signals (disk-full prediction, temp creep) wait on that LXC; the - scheduler's disk check today is point-in-time only, and CPU/NVMe - temperature isn't probed at all yet (no confirmed sensor path on - hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness - drift checks are also deferred (see `oikos/drift.py` docstring). -- **Week 4**: Oikos Console v0 shipped — signals landing page, service - grid + detail, node/blast-radius view, live Mermaid graph, drift view, - approvals queue (approve/deny, destructive confirmation-phrase - enforced), daily/weekly reports. Server-rendered FastAPI + Jinja2, no - SPA build chain, tested end-to-end against live production data (see - `oikos/console/`). Deploys as a third webhook on `dtoro/Homelab-Docs` - (`/opt/oikos-console`, port :9831) — see - [oikos/console/deploy/README.md](oikos/console/deploy/README.md) for - the Caddy route and Gitea webhook registration this repo can't do for - itself. Approval grants are now single-use (a second `check_grant` call - for the same request fails even within the TTL) and already exact-bound - to request id + entity + action. - **Not shipped as originally planned:** per-agent *age-key-signed* - request authentication — age has no signing primitive (it's an - encryption-only keypair format), so "age-key-signed" wasn't - buildable as stated. The real alternative (SSH-key signing via - `ssh-keygen -Y sign`/`-Y verify`, using each host's already-provisioned - SSH key) is real and buildable, but needs SSH public keys recorded in - inventory first — not there today. Moved to the 60/90-day backlog. - Authentik step-up re-auth on the approve/deny route is documented but - needs a live Authentik instance to configure — also backlog. - Docs pass done (this file, AGENTS.md, operations/commands.md); found - and fixed two more stale references while at it (DNS section still - pointed at destroyed LXC 124/dnsmasq instead of Technitium on 107, and - a `claudio-monitor` reference that's been deprecated since 2026-06-04). - -### Real drift found while building Week 3 (unresolved, needs operator action) - -The drift detectors surfaced genuine, currently-true findings on first -run against production — recorded here rather than silently fixed, since -each is a `config_mutation`/`destructive`-class decision: - -- `republic-laptop` has no `age_pubkey:` in `inventory.yaml`, but its real - age key is granted on nearly every shared secret in `.sops.yaml` - (`age1vf8h7...`) — the enrollment write-back to inventory never - happened. Fix: `homelab client add republic-laptop --finalize-pubkey - age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6`. -- `grimmory` has an `age_pubkey` in inventory but is missing from - `secrets/hello.yaml`'s recipient list — incomplete enrollment the - other direction. Fix: re-run `homelab client add grimmory - --finalize-pubkey <its key>`. -- `pve_id 131` exists live on hubris (`pct list`) with no inventory entry - — investigate before assuming it's a stale ID (see the Prometheus LXC - plan doc above, which flags this explicitly). -- Three `lifecycle-pve-id-reuse` info findings (100, 106, 107 each shared - between an active host and an archaeology entry) — expected/benign ID - reuse after destroy, no action needed. - -## 60/90-day backlog - -Derived from gaps observed while building the 30-day roadmap, not -guesswork. Roughly ordered by what unblocks the most: - -- **Fix the oikos-console deploy webhook's signature mismatch.** Console - is live on apps (105) via a manual `deploy.sh` run, but Gitea webhook - 14's deliveries all 403 with a signature mismatch for a cause not yet - found — the secret is confirmed synced correctly on both sides - (rotated once already to rule out drift). Until fixed, `git push` - doesn't auto-redeploy the console the way it does for homelab-mcp/ - secrets-issuance; re-run `deploy.sh` on apps manually after changes. - See [oikos/console/deploy/README.md](oikos/console/deploy/README.md). -- **SSH-key-signed approval requests.** Replaces the design note in - Week 4: age keys can't sign (encryption-only format), so per-agent - request authentication needs `ssh-keygen -Y sign`/`-Y verify` against - each host's existing SSH key. Blocked on a schema gap: inventory - doesn't record SSH public keys today, only ports/users. First step is - populating that field on enrollment, then wiring `oikos/approve.py` to - require and verify a signature over the request payload. -- **Authentik step-up re-auth** on the Console's `/approvals` POST route - — needs a live Authentik `PromptStage`/reauth flow scoped to that path; - not configurable without a running instance to test against. -- **Prometheus provisioning** (see - [plans/2026-07-05-oikos-prometheus-lxc.md](plans/2026-07-05-oikos-prometheus-lxc.md)) - — unblocks trend signals (disk-full prediction, temp creep) and real - sparklines in the Console; investigate the undocumented `pve_id 131` - on hubris first. -- **CPU/NVMe temperature probing** in the scheduler — needs a confirmed - sensor path on hubris and strong (lm-sensors vs vendor tool) before a - real check can be written; guessing one risks a probe that silently - never fires. -- **DNS-vs-inventory drift check** — compare Technitium zone records - against `services.*.url`/`public_host`; not implemented (`oikos/drift.py` - has no Technitium API wiring yet). -- **Generic tracked-config-cleanliness drift check** — today only caddy's - `/etc/caddy` git-checkout path is hardcoded in `oikos/drift.py`; every - other service with a `config_repo` needs its local checkout path - recorded (a `mutation_path`-style field, same gap Week 1's service - contract flagged but didn't backfill) before this generalizes. -- **Per-service policy overrides** (`oikos/policy.yaml` - `service_overrides`) — schema is ready (caddy/dns already use it); - populate more as specific services turn out to need non-default risk - classes. -- **Incident timeline generator** — stitch ledger + signal history into - a single narrative for `investigations/` entries instead of writing - them by hand. -- **Secret access audit** — who-can-decrypt-what report from - `.sops.yaml` + inventory `age_pubkey`s, extending what - `oikos/drift.py`'s SOPS check already partially does. -- **Restore drills** — exercise `backs-up-to` (once populated) by - actually restoring from a backup target on a schedule, not just - checking freshness. -- **Multi-agent delegation model** — more than one agent acting - concurrently; needs the ledger's `agent` field to carry real identity - (age pubkey, not just hostname) consistently, which it mostly does - already but hasn't been stress-tested with concurrent writers. -- **Grafana** — only if the Console's own Prometheus-backed sparklines - turn out to be insufficient once Prometheus ships. -- **"Generalize later" extraction** — the original decision was personal- - first, generalize-later (see Week 1). Once patterns stabilize, extract - a config-driven Oikos core with no `hubris.network`/`hubris`/`strong` - hardcoding, so it's installable on a different homelab. +The Oikos operating model now lives at [`.agents/OIKOS.md`](.agents/OIKOS.md). diff --git a/README.md b/README.md index 2c0ab21..1e92e05 100644 --- a/README.md +++ b/README.md @@ -84,4 +84,5 @@ When you change a node: ## See also -- [`CONTRIBUTING.md`](CONTRIBUTING.md) — page templates and tone +- [`.agents/shared/page-templates.md`](.agents/shared/page-templates.md) — page templates and tone +- [`.agents/shared/writing-style.md`](.agents/shared/writing-style.md) — prose style, banned vocabulary diff --git a/plans/2026-07-06-adopt-wiki-hq-doc-architecture.md b/plans/2026-07-06-adopt-wiki-hq-doc-architecture.md new file mode 100644 index 0000000..c87b2a0 --- /dev/null +++ b/plans/2026-07-06-adopt-wiki-hq-doc-architecture.md @@ -0,0 +1,164 @@ +# 2026-07-06 — Adopt the wiki-hq documentation architecture + +## Goal + +Reorganize the Homelab-Docs **narrative layer** into the `wiki-hq` +(`/Users/dtoro/Downloads/wiki-hq-main`) documentation model — `sources / wiki / index / log` +plus an `.agents/` separation and a single lint-checkable writing-style standard — **without +breaking** the Oikos machine-readable substrate that reads fixed paths. Keep the existing Mermaid +topology generator (no LikeC4). + +## Context + +Homelab-Docs is already a mature operational-docs system: `inventory.yaml` as source of truth, +generated `hosts/*.yaml`, an MCP server, a `homelab` CLI, and the Oikos OODA kernel (scheduler, +drift, decide, ledger, approvals). Its weakness is on the *narrative* side — the prose layer grew +organically and lacks the discipline wiki-hq shows: + +- No enforceable **writing-style** standard (CONTRIBUTING.md has a one-line "voice" note; CAVEMAN.md governs agent *chat*, not docs). +- **Index/README files are inconsistent** — some pure listings, some prose+tables, some with stale counts. +- **No append-only operations log** for doc maintenance — doc changes are visible only in git. +- **Agent instructions and human content are interleaved** at the repo root (AGENTS.md, OIKOS.md, CAVEMAN.md, CONTRIBUTING.md, GLOSSARY.md alongside `containers/`, `infrastructure/`, …). + +Decision (confirmed with operator): **full structural adoption**, **keep Mermaid**, adopt all four +borrows — writing-style guide, section-index/README pattern, append-only per-area logs, and +agent-instruction separation. + +## Hard constraint: protect the operational substrate + +These paths are read programmatically and **must not move** (see `mcp/server.py`, `bin/homelab`, `oikos/`): + +- `inventory.yaml` (root) — MCP (`mcp/server.py:36`), CLI (`bin/homelab:36`), scheduler, drift, relations, gen-topology. +- `hosts/*.yaml` (root, generated) — `HOSTS_DIR` (`mcp/server.py:37`); written by `mcp/build_host_files.py`; read/written by `bin/homelab`. +- `oikos/` — kernel code, `oikos/cards/` (`explain` tool, `mcp/server.py:38`), `oikos/state.json`. +- `secrets/`, `secrets-issuance/`, `ssh/`, `scripts/`, `tools/`, `vps/`, `bin/`, `bootstrap.sh`. + +Two MCP tools are path-agnostic and survive any narrative reorg: `search_docs` (ripgreps all +`*.md`) and `get_page(path)` (agent supplies a repo-relative path). The **`## Changelog` convention +must stay** on node pages — `get_changelog` (`mcp/server.py:214`) and the ledger parse it. New +`log.md` files are **additive**, not a replacement for per-page changelogs. + +## Current → target structure + +Restructure the **narrative layer only**; leave the substrate in place. + +``` +Homelab-Docs/ + README.md # human landing page (kept; refreshed to new nav) + AGENTS.md # kept at root (conventional discovery path) + + .agents/ # NEW — agent-facing instruction, separated from content + shared/ + writing-style.md # NEW — adapted from wiki-hq (homelab voice) + llm-wiki.md # NEW — the sources/wiki/index/log contract for THIS repo + caveman.md # moved from CAVEMAN.md (agent chat mode) + page-templates.md # moved from CONTRIBUTING.md (page templates) + domains/ + knowledge/schema.md # contract for the current-state wiki + operations/schema.md # contract for runbooks / investigations / plans + skills/ # runbooks reshaped as SKILL.md (Phase 4) + lifecycle-provision-node/SKILL.md + service-health-check/SKILL.md + ... + OIKOS.md # operating-model doc (moved; agent-facing) + + knowledge/ # durable current-state wiki (Feedback/authoritative layer) + index.md # pure listing → section indexes only + log.md # append-only doc-maintenance operations log + GLOSSARY.md # moved from root + sources/ + index.md # flat catalog table + references/ # external docs (from infrastructure/references/) + wiki/ + hosts/ README.md + hubris.md + strong.md + containers/ README.md (= today's index.md) + <id>-<name>.md + vms/ README.md + <id>-<name>.md + infrastructure/ README.md + grouped: network/ identity/ storage/ ingress/ operations/ + + operations/ # operator runbook narrative index + index.md commands.md agent-enrollment.md hermes-agent.md + + investigations/ # incident evidence (Observe sources) + README.md + log.md + YYYY-MM-DD-*.md + archive/ + plans/ # projects / design docs (Act) + README.md + YYYY-MM-DD-*.md + done/ + + # substrate — UNCHANGED (see Hard constraint) + inventory.yaml hosts/*.yaml oikos/ mcp/ secrets/ secrets-issuance/ + ssh/ scripts/ tools/ vps/ bin/ bootstrap.sh +``` + +**hosts/ split:** generated `hosts/*.yaml` stay at root (substrate); only the two *narrative* pages +`hosts/hubris.md` and `hosts/strong.md` move to `knowledge/wiki/hosts/`. This is the one directory +where machine and narrative content currently mix. + +## Consumers to update when narrative paths move + +Same phase as the move: + +- **`inventory.yaml` `doc_page` fields** — per-host pointer; consumed by `bin/homelab` (`die("no doc_page recorded…")` near `bin/homelab:1288`). Repoint to `knowledge/wiki/...`. +- **`oikos/gen-topology.py` / `oikos/gen_topology_lib.py`** — write `infrastructure/topology.md` and link `oikos/cards/` to doc pages. Move `topology.md` under `knowledge/wiki/infrastructure/` and update the output-path constant (`oikos/gen-topology.py:3`, `:49`, `:195`). +- **`oikos/cards/` templates** and `oikos/drift.py` docstring at `oikos/drift.py:218` (`containers/121-caddy.md`) — cosmetic, update for accuracy. +- **Internal cross-links** — repo-wide relative-link rewrite + link-check pass. +- **Same-session update rule** (`CONTRIBUTING.md:125`) — rewrite its path checklist to the new layout. + +`search_docs`, `get_page`, `get_changelog` need **no** code change. + +## Writing-style standard (adapted, not copied) + +Create `.agents/shared/writing-style.md` from +`/Users/dtoro/Downloads/wiki-hq-main/.agents/shared/writing-style/writing-style.md`: + +- **Keep:** banned-vocabulary list (puffers, "leverage/utilize/delve", poetic nouns, promotional adjectives), information-density rule, bold-leading-phrase pattern, tables-over-bullets, `>` blockquotes for caveats, one-sided cross-references, "cross-references are content not catalog". +- **Adjust voice:** wiki-hq mandates strict third-person "no you". Homelab docs are operator runbooks that already use imperative recipes. Allow **imperative/second-person in runbooks, recipes, and skills**; third-person reference voice for node/infrastructure descriptions. State this exception explicitly. +- **Clarify vs CAVEMAN.md:** Caveman governs agent *chat responses*; writing-style governs *committed docs*. No conflict. +- **Make it lint-checkable:** the Oikos lint surface (or a new `.agents/skills/docs-lint`) greps banned words + structural violations. + +## Reconciling "append-only per-area logs" with the Oikos ledger + +- The **Oikos change ledger** (`oikos/ledger.py`) stays authoritative for *infrastructure changes* (who/what/risk/approval/verification). Do not duplicate it. +- **Per-page `## Changelog`** stays (parsed by `get_changelog`). +- New **per-area `log.md`** (`knowledge/log.md`, `investigations/log.md`, `plans/log.md`) records *doc-maintenance operations only* — restructures, source ingests, lint sweeps — in wiki-hq's single-line format `## [YYYY-MM-DD] <op> | <summary>`. + +## Convention: plans always live in `plans/` + +The restructure must codify — in `.agents/domains/operations/schema.md`, `AGENTS.md`, and +`.agents/shared/page-templates.md` — that **any plan or design doc for the Homelab is always +written into the repo `plans/` folder** (`plans/YYYY-MM-DD-slug.md`), never into a scratch/agent +plan path, an ad-hoc location, or a chat message. Agents drafting a plan create the file under +`plans/`, list it in `plans/index.md`, and move it to `plans/done/` on completion. This rule is +stated once in the operations schema and cross-referenced from `AGENTS.md` so every agent sees it +at orientation. + +## Section-index / README pattern + +Every folder gets a `README.md`: 1–3 sentence prose intro, then a single two-column nav table +`| Document | What it covers |`, nothing else — no stale counts, no duplicated prose. Pure-listing +`index.md` files (e.g. `containers/index.md`) become the folder `README.md`. Root `README.md` stays +the human landing page (may exceed the strict pattern). + +## Phased rollout + +Each phase independently valuable, independently verifiable, committed separately. + +1. **Conventions first (no moves).** Add `.agents/shared/writing-style.md`, `llm-wiki.md`; move CAVEMAN.md → `.agents/shared/caveman.md`, CONTRIBUTING.md → `.agents/shared/page-templates.md` (leave thin root stubs pointing to the new locations). Verify: links resolve; MCP untouched. +2. **Agent-instruction separation.** Move OIKOS.md → `.agents/OIKOS.md`; author `.agents/domains/{knowledge,operations}/schema.md`. Keep AGENTS.md at root, pointing at `.agents/`. Verify: AGENTS.md still the discovery entry; `tools/setup-hermes-soul.sh:51` references resolve or are updated. +3. **Narrative move into `knowledge/wiki/`.** Move `containers/`, `vms/`, `infrastructure/`, host narratives; add `knowledge/index.md`, `knowledge/log.md`, `knowledge/sources/`. Same commit: update `inventory.yaml` `doc_page` fields, `oikos/gen-topology.py` output path, all internal links. Verify: link-check clean; CLI `doc_page` lookups work; `gen-topology.py` writes to the new path; MCP `get_page`/`search_docs` return moved pages. +4. **Runbooks → skills.** Reshape `runbooks/*.md` into `.agents/skills/<name>/SKILL.md` (they already carry `risk_class`, `inputs`, `verification` frontmatter — near-compatible). Update runbook-path references. Verify: frontmatter parses; lifecycle transitions still map to `oikos/ontology.yaml`. +5. **Style + README pass.** Apply the writing-style standard and README pattern across narrative pages; add `log.md` entries recording the restructure. Verify: docs-lint clean. + +## Verification (end to end) + +- **Link integrity:** markdown link checker (or `rg` over `](` targets) after each move phase; zero broken relative links. +- **MCP resolves docs:** `HOMELAB_CONTEXT_DIR=<repo>` — `get_page("knowledge/wiki/containers/104-gitea.md")` returns content; `search_docs("gitea")` hits new paths; `get_changelog(<moved page>)` parses. +- **CLI doc_page lookups:** exercise `bin/homelab`'s `doc_page` read against a moved page — no "no doc_page recorded" error. +- **Topology generation:** run `oikos/gen-topology.py` — writes the new `topology.md` path; `oikos/cards/` links point at moved pages. +- **Substrate untouched:** `mcp/build_host_files.py` regenerates `hosts/*.yaml` identically; `git diff` shows no substrate churn. +- **Style lint:** banned-vocabulary grep over `knowledge/` returns no hits; every folder `README.md` matches the intro-plus-nav-table shape. + +## Post-migration + +Update `plans/index.md` to list this plan; on completion move it to `plans/done/`. Record the +restructure in the new `knowledge/log.md`. Update `README.md` and `AGENTS.md` navigation to the new +layout. +``` From 8a6422bd7dc3eaaa012d2e088bb010183d250ae3 Mon Sep 17 00:00:00 2001 From: dtoro <d.toro.v@pm.me> Date: Mon, 6 Jul 2026 14:35:23 +0200 Subject: [PATCH 2/8] docs: move narrative wiki under knowledge/wiki/ (phase 3) Problem: node and cross-cutting narratives lived at the repo root (containers/, vms/, infrastructure/, host .md files), interleaved with the machine-readable substrate. Change: - Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/, infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md -> knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/, GLOSSARY.md -> knowledge/GLOSSARY.md. - Add knowledge/{index.md,log.md,sources/index.md} scaffolding. - Rewrite all relative links repo-wide via a path-resolving mapper (inbound + outbound + between-moved-files), including .hermes/, runbooks, operations, investigations, plans, README, AGENTS. - Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which embed doc_page); update oikos/gen-topology.py output path, candidate doc paths, and footer links; update code-comment doc paths. Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated, idempotent), oikos/ code, mcp/, secrets/, bin/. Verification: - Logical broken-link set identical to pre-move baseline (net 128 -> 127; the topology regen fixed one, introduced none). Remaining are pre-existing refs to destroyed/archived nodes, out of scope for this move. - gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths. - build_host_files.py idempotent; all inventory doc_page targets resolve. - MCP contract verified: get_page/search_docs/get_changelog resolve moved pages. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- .agents/OIKOS.md | 2 +- AGENTS.md | 13 +++--- README.md | 42 +++++++++---------- hosts/apps.yaml | 6 +-- hosts/arriman.yaml | 4 +- hosts/auth-outpost.yaml | 2 - hosts/caddy.yaml | 4 +- hosts/dns.yaml | 4 +- hosts/elementsynapse.yaml | 4 +- hosts/gitea.yaml | 4 +- hosts/grimmory.yaml | 2 - hosts/haos.yaml | 4 +- hosts/house.yaml | 2 - hosts/hubris.yaml | 4 +- hosts/jellyfin.yaml | 4 +- hosts/mule-images.yaml | 4 +- hosts/netbird-vps.yaml | 2 +- hosts/nextcloud.yaml | 4 +- hosts/nfs-export.yaml | 2 - hosts/paperless.yaml | 4 +- hosts/romm.yaml | 2 - hosts/seanime.yaml | 2 - hosts/sophia.yaml | 2 - hosts/strong.yaml | 2 - hosts/teddycloud.yaml | 4 +- hosts/trmnl.yaml | 4 +- hosts/zimaos.yaml | 4 +- inventory.yaml | 34 +++++++-------- .../2026-06-06-authentik-session-lifetime.md | 4 +- .../2026-06-06-caddyfile-truncation.md | 4 +- GLOSSARY.md => knowledge/GLOSSARY.md | 6 +-- knowledge/index.md | 14 +++++++ knowledge/log.md | 7 ++++ knowledge/sources/index.md | 9 ++++ .../cert-sync-and-traefik-config.md | 0 .../wiki/containers}/101-jellyfin.md | 4 +- .../wiki/containers}/102-nfs-export.md | 2 +- .../wiki/containers}/103-paperless.md | 0 .../wiki/containers}/104-gitea.md | 0 .../wiki/containers}/105-apps.md | 0 .../wiki/containers}/106-auth-outpost.md | 10 ++--- .../wiki/containers}/107-dns.md | 4 +- .../wiki/containers}/114-nextcloud.md | 0 .../wiki/containers}/118-elementsynapse.md | 0 .../wiki/containers}/119-sophia.md | 0 .../wiki/containers}/120-mule-images.md | 0 .../wiki/containers}/121-caddy.md | 2 +- .../wiki/containers}/122-arriman.md | 0 .../wiki/containers}/128-trmnl.md | 2 +- .../wiki/containers}/129-house.md | 2 +- .../wiki/containers}/130-grimmory.md | 0 .../wiki/containers}/131-teddycloud.md | 4 +- .../wiki/containers}/132-rclone.md | 2 +- .../wiki/containers}/133-seanime.md | 0 .../wiki/containers}/134-romm.md | 0 .../containers}/archive/123-claudio-bot.md | 0 .../archive/127-mule-photos-new.md | 0 .../wiki/containers}/index.md | 2 +- {hosts => knowledge/wiki/hosts}/hubris.md | 22 +++++----- {hosts => knowledge/wiki/hosts}/strong.md | 10 ++--- .../wiki/infrastructure}/auto-deploy.md | 6 +-- .../wiki/infrastructure}/backups.md | 10 ++--- .../wiki/infrastructure}/dns.md | 8 ++-- .../wiki/infrastructure}/homelab-context.md | 6 +-- .../wiki/infrastructure}/index.md | 6 +-- .../wiki/infrastructure}/ingress.md | 4 +- .../wiki/infrastructure}/media-permissions.md | 0 .../wiki/infrastructure}/mesh.md | 2 +- .../wiki/infrastructure}/monitoring.md | 0 .../wiki/infrastructure}/network.md | 8 ++-- .../wiki/infrastructure}/ssh-access.md | 4 +- .../wiki/infrastructure}/topology.md | 4 +- .../wiki/infrastructure}/vps-hardening.md | 0 {vms => knowledge/wiki/vms}/100-zimaos.md | 2 +- {vms => knowledge/wiki/vms}/108-haos.md | 0 {vms => knowledge/wiki/vms}/index.md | 2 +- oikos/approve.py | 2 +- oikos/cards/host-apps.md | 4 +- oikos/cards/host-arriman.md | 2 +- oikos/cards/host-auth-outpost.md | 2 +- oikos/cards/host-caddy.md | 2 +- oikos/cards/host-dns.md | 4 +- oikos/cards/host-elementsynapse.md | 2 +- oikos/cards/host-gitea.md | 2 +- oikos/cards/host-grimmory.md | 2 +- oikos/cards/host-haos.md | 2 +- oikos/cards/host-house.md | 2 +- oikos/cards/host-hubris.md | 2 +- oikos/cards/host-jellyfin.md | 2 +- oikos/cards/host-mule-images.md | 2 +- oikos/cards/host-nextcloud.md | 2 +- oikos/cards/host-nfs-export.md | 2 +- oikos/cards/host-paperless.md | 2 +- oikos/cards/host-romm.md | 2 +- oikos/cards/host-seanime.md | 2 +- oikos/cards/host-sophia.md | 2 +- oikos/cards/host-strong.md | 2 +- oikos/cards/host-teddycloud.md | 4 +- oikos/cards/host-trmnl.md | 2 +- oikos/cards/host-zimaos.md | 2 +- oikos/cards/service-arr_stack.md | 2 +- oikos/cards/service-artifacto.md | 2 +- oikos/cards/service-authentik.md | 2 +- oikos/cards/service-caddy.md | 4 +- oikos/cards/service-dns.md | 2 +- oikos/cards/service-gitea.md | 2 +- oikos/cards/service-haos.md | 2 +- oikos/cards/service-homelab_mcp.md | 2 +- oikos/cards/service-jellyfin.md | 2 +- oikos/cards/service-matrix.md | 2 +- oikos/cards/service-nextcloud.md | 2 +- oikos/cards/service-paperless.md | 2 +- oikos/cards/service-photos.md | 2 +- oikos/cards/service-proxmox_ui.md | 2 +- oikos/cards/service-teddycloud.md | 2 +- oikos/cards/service-trmnl.md | 2 +- oikos/cards/service-zimaos.md | 2 +- oikos/console/deploy/README.md | 2 +- oikos/drift.py | 2 +- oikos/gen-topology.py | 16 +++---- oikos/gen_topology_lib.py | 2 +- oikos/report.py | 2 +- oikos/scheduler.py | 2 +- operations/agent-enrollment.md | 10 ++--- operations/commands.md | 26 ++++++------ operations/hermes-agent.md | 6 +-- plans/2026-06-24-trmnl-plugins-lxc.md | 6 +-- runbooks/config-change-deploy.md | 2 +- runbooks/runbook-dpkg-interrupted.md | 4 +- 129 files changed, 249 insertions(+), 264 deletions(-) rename GLOSSARY.md => knowledge/GLOSSARY.md (87%) create mode 100644 knowledge/index.md create mode 100644 knowledge/log.md create mode 100644 knowledge/sources/index.md rename {infrastructure => knowledge/sources}/references/cert-sync-and-traefik-config.md (100%) rename {containers => knowledge/wiki/containers}/101-jellyfin.md (97%) rename {containers => knowledge/wiki/containers}/102-nfs-export.md (98%) rename {containers => knowledge/wiki/containers}/103-paperless.md (100%) rename {containers => knowledge/wiki/containers}/104-gitea.md (100%) rename {containers => knowledge/wiki/containers}/105-apps.md (100%) rename {containers => knowledge/wiki/containers}/106-auth-outpost.md (81%) rename {containers => knowledge/wiki/containers}/107-dns.md (93%) rename {containers => knowledge/wiki/containers}/114-nextcloud.md (100%) rename {containers => knowledge/wiki/containers}/118-elementsynapse.md (100%) rename {containers => knowledge/wiki/containers}/119-sophia.md (100%) rename {containers => knowledge/wiki/containers}/120-mule-images.md (100%) rename {containers => knowledge/wiki/containers}/121-caddy.md (97%) rename {containers => knowledge/wiki/containers}/122-arriman.md (100%) rename {containers => knowledge/wiki/containers}/128-trmnl.md (98%) rename {containers => knowledge/wiki/containers}/129-house.md (97%) rename {containers => knowledge/wiki/containers}/130-grimmory.md (100%) rename {containers => knowledge/wiki/containers}/131-teddycloud.md (94%) rename {containers => knowledge/wiki/containers}/132-rclone.md (99%) rename {containers => knowledge/wiki/containers}/133-seanime.md (100%) rename {containers => knowledge/wiki/containers}/134-romm.md (100%) rename {containers => knowledge/wiki/containers}/archive/123-claudio-bot.md (100%) rename {containers => knowledge/wiki/containers}/archive/127-mule-photos-new.md (100%) rename {containers => knowledge/wiki/containers}/index.md (98%) rename {hosts => knowledge/wiki/hosts}/hubris.md (93%) rename {hosts => knowledge/wiki/hosts}/strong.md (95%) rename {infrastructure => knowledge/wiki/infrastructure}/auto-deploy.md (97%) rename {infrastructure => knowledge/wiki/infrastructure}/backups.md (94%) rename {infrastructure => knowledge/wiki/infrastructure}/dns.md (89%) rename {infrastructure => knowledge/wiki/infrastructure}/homelab-context.md (97%) rename {infrastructure => knowledge/wiki/infrastructure}/index.md (94%) rename {infrastructure => knowledge/wiki/infrastructure}/ingress.md (90%) rename {infrastructure => knowledge/wiki/infrastructure}/media-permissions.md (100%) rename {infrastructure => knowledge/wiki/infrastructure}/mesh.md (99%) rename {infrastructure => knowledge/wiki/infrastructure}/monitoring.md (100%) rename {infrastructure => knowledge/wiki/infrastructure}/network.md (94%) rename {infrastructure => knowledge/wiki/infrastructure}/ssh-access.md (98%) rename {infrastructure => knowledge/wiki/infrastructure}/topology.md (95%) rename {infrastructure => knowledge/wiki/infrastructure}/vps-hardening.md (100%) rename {vms => knowledge/wiki/vms}/100-zimaos.md (99%) rename {vms => knowledge/wiki/vms}/108-haos.md (100%) rename {vms => knowledge/wiki/vms}/index.md (94%) diff --git a/.agents/OIKOS.md b/.agents/OIKOS.md index 715f169..fe1fa17 100644 --- a/.agents/OIKOS.md +++ b/.agents/OIKOS.md @@ -80,7 +80,7 @@ stored as `state:` in inventory (absent = active). Destroyed nodes live in the `archaeology:` section. Each transition is a runbook checklist; deprecation completes only when inbound edges reach zero. -Generated views: [infrastructure/topology.md](../infrastructure/topology.md) +Generated views: [infrastructure/topology.md](../knowledge/wiki/infrastructure/topology.md) (Mermaid, regenerated from inventory) and the live, clickable version at `oikos.hubris.network/graph` once the Console is deployed. diff --git a/AGENTS.md b/AGENTS.md index 8349188..ea72a46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,10 +33,10 @@ the operator to run `homelab client add <hostname>` from an existing client. - `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation with their mesh addresses, roles, and service mappings. Treat this file as authoritative; anything you read in narrative pages should agree with it. -- `/opt/homelab-context/infrastructure/mesh.md` — Tailscale → Netbird state. +- `/opt/homelab-context/knowledge/wiki/infrastructure/mesh.md` — Tailscale → Netbird state. Both meshes are accepted today; Netbird is preferred for new traffic. -- `/opt/homelab-context/infrastructure/dns.md` — split-horizon DNS via - Technitium on [dns (107)](../containers/107-dns.md). `*.hubris.network` +- `/opt/homelab-context/knowledge/wiki/infrastructure/dns.md` — split-horizon DNS via + Technitium on [dns (107)](knowledge/wiki/containers/107-dns.md). `*.hubris.network` resolves to 192.168.x.x on the LAN and to mesh addresses off-LAN. - `/opt/homelab-context/operations/commands.md` — the operator's cheatsheet for pct, caddy, DNS, and the Oikos command surface. Use these verbs when @@ -75,8 +75,9 @@ Grep is fine for browsing or when MCP is unreachable. ## 4. Wiki conventions -- Pages live under `containers/`, `hosts/`, `vms/`, `infrastructure/`, - `investigations/`, `operations/`. Cross-link liberally; orphans are bugs. +- Narrative pages live under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`; + procedural pages under `investigations/`, `operations/`, `runbooks/`, `plans/`. + Cross-link liberally; orphans are bugs. - Every page ends with a `## Changelog` section, entries in reverse-chrono order: @@ -105,7 +106,7 @@ Grep is fine for browsing or when MCP is unreachable. a valid `--approval-id` from `homelab approval request` — see OIKOS.md. For ad-hoc work, SSH and edit directly — but commit changes that touch tracked configs (caddy, gitea custom, artifacto, mule-image, etc.; see - `infrastructure/auto-deploy.md`). + `knowledge/wiki/infrastructure/auto-deploy.md`). - **Wiki updates**: same-session rule applies to any meaningful state change this client makes. diff --git a/README.md b/README.md index 1e92e05..6515cc0 100644 --- a/README.md +++ b/README.md @@ -12,23 +12,23 @@ Living documentation for the **hubris** Proxmox homelab. Every node, every cross - [Cross-cutting infrastructure](#cross-cutting-infrastructure) — DNS, ingress, mesh, storage, auth - [Investigations](#investigations) — incident timelines - [Operations](#operations) — cheatsheet, enrollment, runbooks -- [Glossary](GLOSSARY.md) — term definitions +- [Glossary](knowledge/GLOSSARY.md) — term definitions - [Conventions](#conventions) — wiki style, changelog hygiene, same-session update rule ## Map ### Proxmox hosts -- [`hubris`](hosts/hubris.md) — PVE node, GMKtec NucBox M6 Ultra, `192.168.8.77` — primary host, runs [8 LXCs](containers/index.md) + 2 VMs -- [`strong`](hosts/strong.md) — PVE node (cluster hostname `strong`), `192.168.178.181` — 2nd member of `Homelab` cluster. Hosts [7 LXCs](containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05) +- [`hubris`](knowledge/wiki/hosts/hubris.md) — PVE node, GMKtec NucBox M6 Ultra, `192.168.8.77` — primary host, runs [8 LXCs](knowledge/wiki/containers/index.md) + 2 VMs +- [`strong`](knowledge/wiki/hosts/strong.md) — PVE node (cluster hostname `strong`), `192.168.178.181` — 2nd member of `Homelab` cluster. Hosts [7 LXCs](knowledge/wiki/containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05) ### VMs -- [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation) -- [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS -- See [vms/index.md](vms/index.md) for the full table +- [100 — `zimaos`](knowledge/wiki/vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation) +- [108 — `haos-16.3`](knowledge/wiki/vms/108-haos.md) — Home Assistant OS +- See [vms/index.md](knowledge/wiki/vms/index.md) for the full table ### LXC containers See the full table with IPs, hosts, mounts, and status in -[`containers/index.md`](containers/index.md). Quick summary: +[`containers/index.md`](knowledge/wiki/containers/index.md). Quick summary: - **hubris** (10 active): 102 nfs-export, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 119 sophia, 120 mule-images, 121 caddy, 124 authentik (outpost), @@ -38,21 +38,21 @@ See the full table with IPs, hosts, mounts, and status in - **Destroyed (archaeology)**: 100 arr, 106 flaresolverr, 107 marimo, 109 syncthing, 110 photoprism, 111 karakeep, 112 immich, 115 reticulum, 123 claudio-bot, 125 seafile, 126 plato, 127 mule-photos-new — see - [containers/index.md](containers/index.md#recently-destroyed-kept-for-archaeology) + [containers/index.md](knowledge/wiki/containers/index.md#recently-destroyed-kept-for-archaeology) ### Cross-cutting infrastructure -- [Infrastructure index](infrastructure/index.md) — map of every cross-cutting system -- [Glossary](GLOSSARY.md) — term definitions -- [DNS — split-horizon](infrastructure/dns.md) -- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md) -- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md) -- [Monitoring — Hermes health watchdog](infrastructure/monitoring.md) -- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md) -- [SSH access](infrastructure/ssh-access.md) -- [Backups — rclone → Proton Drive (LXC 132); restic-on-USB deprecated](infrastructure/backups.md) -- [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md) -- [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md) -- [Homelab context distribution](infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance +- [Infrastructure index](knowledge/wiki/infrastructure/index.md) — map of every cross-cutting system +- [Glossary](knowledge/GLOSSARY.md) — term definitions +- [DNS — split-horizon](knowledge/wiki/infrastructure/dns.md) +- [Ingress — Caddy + VPS traefik](knowledge/wiki/infrastructure/ingress.md) +- [Mesh — Tailscale → Netbird migration](knowledge/wiki/infrastructure/mesh.md) +- [Monitoring — Hermes health watchdog](knowledge/wiki/infrastructure/monitoring.md) +- [Media permissions — `media` GID 10000](knowledge/wiki/infrastructure/media-permissions.md) +- [SSH access](knowledge/wiki/infrastructure/ssh-access.md) +- [Backups — rclone → Proton Drive (LXC 132); restic-on-USB deprecated](knowledge/wiki/infrastructure/backups.md) +- [Auto-deploy — gitea-webhook pipelines](knowledge/wiki/infrastructure/auto-deploy.md) +- [VPS hardening — IONOS / netbird control plane](knowledge/wiki/infrastructure/vps-hardening.md) +- [Homelab context distribution](knowledge/wiki/infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance ### Investigations Time-stamped incident notes / experiments in [`investigations/index.md`](investigations/index.md). @@ -71,7 +71,7 @@ Resolved cases move to [`investigations/archive/`](investigations/archive/). ``` - **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug. - **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog. -- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local. +- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](knowledge/wiki/infrastructure/auto-deploy.md). Edits there must be pushed, not left local. - **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not. ## Maintaining this wiki diff --git a/hosts/apps.yaml b/hosts/apps.yaml index ad5a2cb..48cbb8d 100644 --- a/hosts/apps.yaml +++ b/hosts/apps.yaml @@ -34,7 +34,7 @@ services_hosted: - name: artifacto backend: apps url: https://artifacto.hubris.network - doc_page: containers/105-apps.md + doc_page: knowledge/wiki/containers/105-apps.md config_repo: dtoro/Artifacto - name: homelab_mcp backend: apps @@ -42,7 +42,7 @@ services_hosted: systemd_unit: homelab-mcp public_host: mcp.hubris.network endpoint: https://mcp.hubris.network/mcp - doc_page: infrastructure/homelab-context.md + doc_page: knowledge/wiki/infrastructure/homelab-context.md config_repo: dtoro/Homelab-Docs note: MCP server. Read-only context + management. Reachable on the LAN via Caddy and from off-LAN via Netbird (192.168.8.0/24 is a network resource routed through hubris). @@ -58,7 +58,5 @@ services_hosted: note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS). risk_notes: "identity issuance \u2014 any change is security-sensitive; key operations are destructive-class" age_pubkey: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0 -see_also: -- containers/105-apps.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/arriman.yaml b/hosts/arriman.yaml index 27c0933..58e7cf3 100644 --- a/hosts/arriman.yaml +++ b/hosts/arriman.yaml @@ -29,12 +29,10 @@ services_hosted: - name: arr_stack backend: arriman note: jellyseerr / qbit / sab on docker compose - doc_page: containers/122-arriman.md + doc_page: knowledge/wiki/containers/122-arriman.md notes: - Migrated from hubris to strong 2026-07-05 (Phase 2). Library on ludo-lvm. - Contains homarr, radarr, sonarr, lidarr, sabnzbd, qbittorrent, bazarr, flaresolverr, prowlarr, jellyseerr - qBittorrent auth subnet whitelist expanded to 192.168.8.0/24 (for seanime + Caddy access) -see_also: -- containers/122-arriman.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/auth-outpost.yaml b/hosts/auth-outpost.yaml index e849205..29a0fae 100644 --- a/hosts/auth-outpost.yaml +++ b/hosts/auth-outpost.yaml @@ -16,7 +16,5 @@ mesh_globals: - tailscale notes: - Runs Authentik outpost (reverse-proxy/SSO enforcement) for protected services -see_also: -- containers/106-auth-outpost.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/caddy.yaml b/hosts/caddy.yaml index 02c0b8a..c533e36 100644 --- a/hosts/caddy.yaml +++ b/hosts/caddy.yaml @@ -24,14 +24,12 @@ services_hosted: backend: caddy role: reverse-proxy note: terminates all *.hubris.network - doc_page: containers/121-caddy.md + doc_page: knowledge/wiki/containers/121-caddy.md config_repo: dtoro/caddy-conf risk_notes: "wide blast radius \u2014 every *.hubris.network route rides on it (see oikos/policy.yaml\ \ service_overrides)" notes: - Terminates all *.hubris.network - /etc/caddy is a git checkout of dtoro/caddy-conf -see_also: -- containers/121-caddy.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/dns.yaml b/hosts/dns.yaml index 8a966d2..67bc722 100644 --- a/hosts/dns.yaml +++ b/hosts/dns.yaml @@ -20,12 +20,10 @@ services_hosted: - name: dns backend: dns note: Technitium DNS, split-horizon zone - doc_page: containers/107-dns.md + doc_page: knowledge/wiki/containers/107-dns.md risk_notes: "LAN-wide resolver \u2014 misconfig breaks name resolution for every client" notes: - Technitium DNS, split-horizon zone for *.hubris.network - Primary DNS for 192.168.8.0/24 LAN (inventory.services.dns references this) -see_also: -- containers/107-dns.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/elementsynapse.yaml b/hosts/elementsynapse.yaml index 2f2dbad..535b6fa 100644 --- a/hosts/elementsynapse.yaml +++ b/hosts/elementsynapse.yaml @@ -23,11 +23,9 @@ services_hosted: - name: matrix url: https://matrix.hubris.network backend: elementsynapse - doc_page: containers/118-elementsynapse.md + doc_page: knowledge/wiki/containers/118-elementsynapse.md risk_notes: "alert/approval channel for Oikos \u2014 outage silences agent escalation" notes: - Migrated from hubris to strong 2026-07-05 (Phase 1 of strong migration plan). -see_also: -- containers/118-elementsynapse.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/gitea.yaml b/hosts/gitea.yaml index 01e8963..1aa4297 100644 --- a/hosts/gitea.yaml +++ b/hosts/gitea.yaml @@ -27,12 +27,10 @@ services_hosted: url: https://git.hubris.network backend: gitea backend_url: http://192.168.8.121:3000 - doc_page: containers/104-gitea.md + doc_page: knowledge/wiki/containers/104-gitea.md config_repo: dtoro/gitea-customizations risk_notes: hosts all config repos + deploy webhooks; outage blocks auto-deploy and sync notes: - Bare repos live at /mnt/library/repos/dtoro/*.git -see_also: -- containers/104-gitea.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/grimmory.yaml b/hosts/grimmory.yaml index d04d322..740ea4c 100644 --- a/hosts/grimmory.yaml +++ b/hosts/grimmory.yaml @@ -21,7 +21,5 @@ notes: - Docker host for Grimmory (community fork of Booklore). Created 2026-06-29. - Migrated from hubris to strong 2026-07-05 (Phase 2d). Books on ludo-lvm. age_pubkey: age1uellsemnjrzgfg9fxw4jefpy05laxzggwnwhh6ny3wl7alyp6v8q0muxet -see_also: -- containers/130-grimmory.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/haos.yaml b/hosts/haos.yaml index 909198b..467fd4a 100644 --- a/hosts/haos.yaml +++ b/hosts/haos.yaml @@ -22,8 +22,6 @@ runs: services_hosted: - name: haos backend: haos - doc_page: vms/108-haos.md -see_also: -- vms/108-haos.md + doc_page: knowledge/wiki/vms/108-haos.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/house.yaml b/hosts/house.yaml index edcf89c..34f6c72 100644 --- a/hosts/house.yaml +++ b/hosts/house.yaml @@ -22,7 +22,5 @@ notes: - 192.168.8.212 was the hubris IP before migration (briefly picked up by teddycloud via DHCP; teddycloud has since been given a static IP, see hosts.teddycloud) age_pubkey: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h -see_also: -- containers/129-house.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/hubris.yaml b/hosts/hubris.yaml index 0f3d065..2c14eab 100644 --- a/hosts/hubris.yaml +++ b/hosts/hubris.yaml @@ -29,10 +29,8 @@ services_hosted: url: https://proxmox.hubris.network backend: hubris port: 8006 - doc_page: hosts/hubris.md + doc_page: knowledge/wiki/hosts/hubris.md risk_notes: "hypervisor UI \u2014 changes here affect every guest on the node" age_pubkey: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6 -see_also: -- hosts/hubris.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/jellyfin.yaml b/hosts/jellyfin.yaml index 982a6df..67a871d 100644 --- a/hosts/jellyfin.yaml +++ b/hosts/jellyfin.yaml @@ -26,7 +26,7 @@ services_hosted: - name: jellyfin url: https://media.hubris.network backend: jellyfin - doc_page: containers/101-jellyfin.md + doc_page: knowledge/wiki/containers/101-jellyfin.md risk_notes: native Authentik OIDC via SSO-Auth plugin, no Caddy forward-auth gate; VAAPI transcode depends on GPU passthrough on strong notes: @@ -35,7 +35,5 @@ notes: - SSO-Auth plugin v4.0.0.4 with Authentik OIDC (no Caddy forward-auth gate) - GPU passed via dev0+dev1: /dev/dri/renderD128 + card0 - Migrated from hubris to strong 2026-07-05 (Phase 2). Library on ludo-lvm. -see_also: -- containers/101-jellyfin.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/mule-images.yaml b/hosts/mule-images.yaml index 26d4c12..ea596d1 100644 --- a/hosts/mule-images.yaml +++ b/hosts/mule-images.yaml @@ -26,9 +26,7 @@ services_hosted: - name: photos url: https://photos.hubris.network backend: mule-images - doc_page: containers/120-mule-images.md + doc_page: knowledge/wiki/containers/120-mule-images.md config_repo: dtoro/mule-image -see_also: -- containers/120-mule-images.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/netbird-vps.yaml b/hosts/netbird-vps.yaml index c3be6b5..fc468d1 100644 --- a/hosts/netbird-vps.yaml +++ b/hosts/netbird-vps.yaml @@ -23,7 +23,7 @@ services_hosted: - name: authentik url: https://auth.hubris.network backend: netbird-vps - doc_page: containers/106-auth-outpost.md + doc_page: knowledge/wiki/containers/106-auth-outpost.md note: core runs on the VPS since 2026-05-31; LAN forward-auth outpost is auth-outpost (LXC 106) at 192.168.8.6:9000. Previous backend value "authentik" referenced the retired embedded-outpost host (LXC 124). risk_notes: "SSO provider \u2014 outage locks login to OIDC/forward-auth services" diff --git a/hosts/nextcloud.yaml b/hosts/nextcloud.yaml index f2611cc..b81b3a4 100644 --- a/hosts/nextcloud.yaml +++ b/hosts/nextcloud.yaml @@ -26,8 +26,6 @@ services_hosted: - name: nextcloud url: https://cloud.hubris.network backend: nextcloud - doc_page: containers/114-nextcloud.md -see_also: -- containers/114-nextcloud.md + doc_page: knowledge/wiki/containers/114-nextcloud.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/nfs-export.yaml b/hosts/nfs-export.yaml index ab2e332..abcff5c 100644 --- a/hosts/nfs-export.yaml +++ b/hosts/nfs-export.yaml @@ -14,7 +14,5 @@ mesh_globals: accepted: - netbird - tailscale -see_also: -- containers/102-nfs-export.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/paperless.yaml b/hosts/paperless.yaml index 17bb657..b836978 100644 --- a/hosts/paperless.yaml +++ b/hosts/paperless.yaml @@ -26,9 +26,7 @@ services_hosted: - name: paperless url: https://paperless.hubris.network backend: paperless - doc_page: containers/103-paperless.md + doc_page: knowledge/wiki/containers/103-paperless.md risk_notes: "document archive \u2014 treat data as irreplaceable; DB operations are destructive-class" -see_also: -- containers/103-paperless.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/romm.yaml b/hosts/romm.yaml index 03b9a91..91c2674 100644 --- a/hosts/romm.yaml +++ b/hosts/romm.yaml @@ -22,7 +22,5 @@ notes: - MariaDB sidecar at /opt/romm/docker-compose.yml. - ROMs on ludo-lvm media volume at /mnt/media_local/roms. - 1 core / 2 GiB RAM / 16 GiB rootfs (ludo-lvm). -see_also: -- containers/134-romm.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/seanime.yaml b/hosts/seanime.yaml index 1857d2a..8344a10 100644 --- a/hosts/seanime.yaml +++ b/hosts/seanime.yaml @@ -25,7 +25,5 @@ notes: - /anime mounted from strong ludo-lvm (/mnt/media_local/anime) - Caddy: "https://seanime.hubris.network \u2192 192.168.8.248:43211" - qBittorrent auth subnet whitelist expanded to 192.168.8.0/24 for seanime access -see_also: -- containers/133-seanime.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/sophia.yaml b/hosts/sophia.yaml index c564bb6..87e3efe 100644 --- a/hosts/sophia.yaml +++ b/hosts/sophia.yaml @@ -19,7 +19,5 @@ mesh_globals: - tailscale mounts: - /mnt/library -see_also: -- containers/119-sophia.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/strong.yaml b/hosts/strong.yaml index 463a0f4..b0d2fab 100644 --- a/hosts/strong.yaml +++ b/hosts/strong.yaml @@ -28,7 +28,5 @@ notes: \ (filename kept as-is, it's a historical planning doc). Only Phase 1 (Proxmox install + cluster join)\ \ is done; no physical drive move, service migration, or GPU passthrough has happened yet." age_pubkey: age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4 -see_also: -- hosts/strong.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/teddycloud.yaml b/hosts/teddycloud.yaml index f39c622..15bb9a3 100644 --- a/hosts/teddycloud.yaml +++ b/hosts/teddycloud.yaml @@ -23,7 +23,7 @@ services_hosted: - name: teddycloud url: https://teddy.hubris.network backend: teddycloud - doc_page: containers/131-teddycloud.md + doc_page: knowledge/wiki/containers/131-teddycloud.md note: self-hosted TeddyCloud (Toniebox cloud reimplementation), docker compose risk_notes: "no Caddy forward-auth gate (unlike sab.hubris.network on the same Caddyfile) \u2014 reachable\ \ to anyone on the LAN/mesh who can resolve teddy.hubris.network; undocumented in inventory.yaml until\ @@ -38,7 +38,5 @@ notes: \ DHCP before that \u2014 see hosts/strong.md's 2026-07-05 changelog)." - "No age_pubkey / homelab-context enrollment \u2014 not a homelab CLI client, just a docker-compose app\ \ container. Not a required follow-up unless it needs secrets." -see_also: -- containers/131-teddycloud.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/trmnl.yaml b/hosts/trmnl.yaml index 2e80ecc..937dea7 100644 --- a/hosts/trmnl.yaml +++ b/hosts/trmnl.yaml @@ -22,9 +22,7 @@ services_hosted: backend: trmnl url: https://trmnl.hubris.network note: self-hosted middleware for TRMNL e-ink plugins (polled by TRMNL cloud) - doc_page: containers/128-trmnl.md + doc_page: knowledge/wiki/containers/128-trmnl.md config_repo: dtoro/terminalito -see_also: -- containers/128-trmnl.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/zimaos.yaml b/hosts/zimaos.yaml index 3fd4ac6..a384967 100644 --- a/hosts/zimaos.yaml +++ b/hosts/zimaos.yaml @@ -21,8 +21,6 @@ services_hosted: - name: zimaos url: https://zimaos.hubris.network backend: zimaos - doc_page: vms/100-zimaos.md -see_also: -- vms/100-zimaos.md + doc_page: knowledge/wiki/vms/100-zimaos.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/inventory.yaml b/inventory.yaml index 47d2398..db8bedb 100644 --- a/inventory.yaml +++ b/inventory.yaml @@ -48,26 +48,26 @@ services: url: https://proxmox.hubris.network backend: hubris port: 8006 - doc_page: hosts/hubris.md + doc_page: knowledge/wiki/hosts/hubris.md risk_notes: hypervisor UI — changes here affect every guest on the node gitea: url: https://git.hubris.network backend: gitea backend_url: http://192.168.8.121:3000 - doc_page: containers/104-gitea.md + doc_page: knowledge/wiki/containers/104-gitea.md config_repo: dtoro/gitea-customizations risk_notes: hosts all config repos + deploy webhooks; outage blocks auto-deploy and sync caddy: backend: caddy role: reverse-proxy note: terminates all *.hubris.network - doc_page: containers/121-caddy.md + doc_page: knowledge/wiki/containers/121-caddy.md config_repo: dtoro/caddy-conf risk_notes: wide blast radius — every *.hubris.network route rides on it (see oikos/policy.yaml service_overrides) authentik: url: https://auth.hubris.network backend: netbird-vps - doc_page: containers/106-auth-outpost.md + doc_page: knowledge/wiki/containers/106-auth-outpost.md note: >- core runs on the VPS since 2026-05-31; LAN forward-auth outpost is auth-outpost (LXC 106) at 192.168.8.6:9000. Previous backend value @@ -76,58 +76,58 @@ services: dns: backend: dns note: Technitium DNS, split-horizon zone - doc_page: containers/107-dns.md + doc_page: knowledge/wiki/containers/107-dns.md risk_notes: LAN-wide resolver — misconfig breaks name resolution for every client jellyfin: url: https://media.hubris.network backend: jellyfin - doc_page: containers/101-jellyfin.md + doc_page: knowledge/wiki/containers/101-jellyfin.md risk_notes: native Authentik OIDC via SSO-Auth plugin, no Caddy forward-auth gate; VAAPI transcode depends on GPU passthrough on strong nextcloud: url: https://cloud.hubris.network backend: nextcloud - doc_page: containers/114-nextcloud.md + doc_page: knowledge/wiki/containers/114-nextcloud.md paperless: url: https://paperless.hubris.network backend: paperless - doc_page: containers/103-paperless.md + doc_page: knowledge/wiki/containers/103-paperless.md risk_notes: document archive — treat data as irreplaceable; DB operations are destructive-class matrix: url: https://matrix.hubris.network backend: elementsynapse - doc_page: containers/118-elementsynapse.md + doc_page: knowledge/wiki/containers/118-elementsynapse.md risk_notes: alert/approval channel for Oikos — outage silences agent escalation photos: url: https://photos.hubris.network backend: mule-images - doc_page: containers/120-mule-images.md + doc_page: knowledge/wiki/containers/120-mule-images.md config_repo: dtoro/mule-image arr_stack: backend: arriman note: jellyseerr / qbit / sab on docker compose - doc_page: containers/122-arriman.md + doc_page: knowledge/wiki/containers/122-arriman.md artifacto: backend: apps url: https://artifacto.hubris.network - doc_page: containers/105-apps.md + doc_page: knowledge/wiki/containers/105-apps.md config_repo: dtoro/Artifacto trmnl: backend: trmnl url: https://trmnl.hubris.network note: self-hosted middleware for TRMNL e-ink plugins (polled by TRMNL cloud) - doc_page: containers/128-trmnl.md + doc_page: knowledge/wiki/containers/128-trmnl.md config_repo: dtoro/terminalito zimaos: url: https://zimaos.hubris.network backend: zimaos - doc_page: vms/100-zimaos.md + doc_page: knowledge/wiki/vms/100-zimaos.md haos: backend: haos - doc_page: vms/108-haos.md + doc_page: knowledge/wiki/vms/108-haos.md teddycloud: url: https://teddy.hubris.network backend: teddycloud - doc_page: containers/131-teddycloud.md + doc_page: knowledge/wiki/containers/131-teddycloud.md note: self-hosted TeddyCloud (Toniebox cloud reimplementation), docker compose risk_notes: no Caddy forward-auth gate (unlike sab.hubris.network on the same Caddyfile) — reachable to anyone on the LAN/mesh who can resolve teddy.hubris.network; undocumented @@ -138,7 +138,7 @@ services: systemd_unit: homelab-mcp public_host: mcp.hubris.network endpoint: https://mcp.hubris.network/mcp - doc_page: infrastructure/homelab-context.md + doc_page: knowledge/wiki/infrastructure/homelab-context.md config_repo: dtoro/Homelab-Docs note: MCP server. Read-only context + management. Reachable on the LAN via Caddy and from off-LAN via Netbird (192.168.8.0/24 is a network resource routed through diff --git a/investigations/2026-06-06-authentik-session-lifetime.md b/investigations/2026-06-06-authentik-session-lifetime.md index 0de009e..faf9613 100644 --- a/investigations/2026-06-06-authentik-session-lifetime.md +++ b/investigations/2026-06-06-authentik-session-lifetime.md @@ -90,9 +90,9 @@ print("session_duration:", stage.session_duration) # → "days=30" ## Related -- [Container 106 — auth-outpost](../containers/106-auth-outpost.md) +- [Container 106 — auth-outpost](../knowledge/wiki/containers/106-auth-outpost.md) - [Authentik VPS migration](2026-05-31-authentik-vps-migration.md) -- [Ingress (VPS Traefik)](../infrastructure/ingress.md) +- [Ingress (VPS Traefik)](../knowledge/wiki/infrastructure/ingress.md) - `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan ## Changelog diff --git a/investigations/2026-06-06-caddyfile-truncation.md b/investigations/2026-06-06-caddyfile-truncation.md index 856b074..cf3203a 100644 --- a/investigations/2026-06-06-caddyfile-truncation.md +++ b/investigations/2026-06-06-caddyfile-truncation.md @@ -55,7 +55,7 @@ This is the same class of drift as the June 5th incidents (paperless, HAOS, apps ## Related - [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md) -- [Caddy (121)](../containers/121-caddy.md) -- [elementsynapse (118)](../containers/118-elementsynapse.md) +- [Caddy (121)](../knowledge/wiki/containers/121-caddy.md) +- [elementsynapse (118)](../knowledge/wiki/containers/118-elementsynapse.md) - [dns-sync script](../scripts/dns-sync.py) - [check-caddy-backends script](../scripts/check-caddy-backends.sh) \ No newline at end of file diff --git a/GLOSSARY.md b/knowledge/GLOSSARY.md similarity index 87% rename from GLOSSARY.md rename to knowledge/GLOSSARY.md index b88f94f..9b68dc0 100644 --- a/GLOSSARY.md +++ b/knowledge/GLOSSARY.md @@ -18,7 +18,7 @@ Terms and abbreviations used throughout the homelab wiki. | **Mesh** | Overlay VPN for off-LAN connectivity. Netbird is current; Tailscale is legacy | | **Netbird** | Preferred mesh VPN. VPS hosts the management plane; all homelab nodes are members | | **OIDC** | OpenID Connect. Protocol used by Authentik for SSO login flows | -| **Oikos** | Agent operating model ([.agents/OIKOS.md](.agents/OIKOS.md)). OODA loop, risk classes, policy, ontology | +| **Oikos** | Agent operating model ([.agents/OIKOS.md](../.agents/OIKOS.md)). OODA loop, risk classes, policy, ontology | | **PVE** | Proxmox Virtual Environment — the hypervisor on both hubris and strong | | **SOPS** | `sops` — Mozilla SOPS. Encrypts secrets with age keys so they live in the git repo | | **Strong** | Secondary Proxmox VE node. Cluster member 2 (hostname `strong`, nickname ludo/ludo-mini) | @@ -29,5 +29,5 @@ Terms and abbreviations used throughout the homelab wiki. ## See also -- [Infrastructure index](infrastructure/index.md) — cross-cutting systems each with their own doc page -- [OIKOS operating model](.agents/OIKOS.md) — agent policy, risk classes, lifecycle \ No newline at end of file +- [Infrastructure index](wiki/infrastructure/index.md) — cross-cutting systems each with their own doc page +- [OIKOS operating model](../.agents/OIKOS.md) — agent policy, risk classes, lifecycle \ No newline at end of file diff --git a/knowledge/index.md b/knowledge/index.md new file mode 100644 index 0000000..0067f7f --- /dev/null +++ b/knowledge/index.md @@ -0,0 +1,14 @@ +# Knowledge + +The durable, authoritative current-state documentation of the homelab: one page per node and per +cross-cutting system, synthesized from live state and evidence. Structure and rules are in +[the knowledge schema](../.agents/domains/knowledge/schema.md). + +| Section | What it covers | +|---------|----------------| +| [wiki/hosts/](wiki/hosts/) | Proxmox host narratives — `hubris`, `strong`. | +| [wiki/containers/](wiki/containers/index.md) | LXC fleet — one page per container, plus the master table and archaeology. | +| [wiki/vms/](wiki/vms/index.md) | Virtual machines — ZimaOS, Home Assistant OS. | +| [wiki/infrastructure/](wiki/infrastructure/index.md) | Cross-cutting systems — DNS, ingress, mesh, storage, auth, monitoring, generated topology. | +| [sources/](sources/index.md) | External reference docs and the pointer to incident evidence. | +| [GLOSSARY.md](GLOSSARY.md) | Term definitions. | diff --git a/knowledge/log.md b/knowledge/log.md new file mode 100644 index 0000000..4cb195a --- /dev/null +++ b/knowledge/log.md @@ -0,0 +1,7 @@ +# Knowledge — operations log + +Append-only record of documentation-maintenance operations on the knowledge wiki (restructures, +source ingests, lint sweeps). One line per operation, newest last. Infrastructure changes belong in +each page's `## Changelog` and the Oikos change ledger, not here. + +## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output. diff --git a/knowledge/sources/index.md b/knowledge/sources/index.md new file mode 100644 index 0000000..f4559f3 --- /dev/null +++ b/knowledge/sources/index.md @@ -0,0 +1,9 @@ +# Sources + +Immutable evidence the wiki synthesizes from. External reference docs live under `references/`; +incident evidence lives in [`investigations/`](../../investigations/index.md) (written once at +incident time, then linked from the changelogs of the nodes they implicate). + +| Slug | Reference | Summary | +|------|-----------|---------| +| cert-sync-and-traefik-config | [references/cert-sync-and-traefik-config.md](references/cert-sync-and-traefik-config.md) | VPS traefik config and the LAN↔VPS certificate mirror. | diff --git a/infrastructure/references/cert-sync-and-traefik-config.md b/knowledge/sources/references/cert-sync-and-traefik-config.md similarity index 100% rename from infrastructure/references/cert-sync-and-traefik-config.md rename to knowledge/sources/references/cert-sync-and-traefik-config.md diff --git a/containers/101-jellyfin.md b/knowledge/wiki/containers/101-jellyfin.md similarity index 97% rename from containers/101-jellyfin.md rename to knowledge/wiki/containers/101-jellyfin.md index 597e231..28c68ac 100644 --- a/containers/101-jellyfin.md +++ b/knowledge/wiki/containers/101-jellyfin.md @@ -78,7 +78,7 @@ User → media.hubris.network → Caddy (TLS, no forward-auth) → Jellyfin :809 `http://` redirect URIs that Authentik rejects) - `EnableAuthorization`: `false` (prevents plugin from overwriting admin permissions on each SSO login — see - [jellyfin-sso-plugin](../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md)) + [jellyfin-sso-plugin](../../../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md)) - `OidScopes`: `["email"]` (openid+profile added by default by the plugin; must be non-null or `OidChallenge()` throws `ArgumentNullException`) @@ -133,7 +133,7 @@ Member of the [media GID 10000](../infrastructure/media-permissions.md) standard - [Media permissions](../infrastructure/media-permissions.md) - [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads - [DNS split-horizon](../infrastructure/dns.md) -- [Authentik admin](../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config +- [Authentik admin](../../../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config ## Changelog diff --git a/containers/102-nfs-export.md b/knowledge/wiki/containers/102-nfs-export.md similarity index 98% rename from containers/102-nfs-export.md rename to knowledge/wiki/containers/102-nfs-export.md index c8efe5a..15c69d9 100644 --- a/containers/102-nfs-export.md +++ b/knowledge/wiki/containers/102-nfs-export.md @@ -54,7 +54,7 @@ We considered three options before building this: | Option | Outcome | |---|---| -| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../investigations/index.md)). Rejected. | +| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../../investigations/index.md)). Rejected. | | **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. | | **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. | diff --git a/containers/103-paperless.md b/knowledge/wiki/containers/103-paperless.md similarity index 100% rename from containers/103-paperless.md rename to knowledge/wiki/containers/103-paperless.md diff --git a/containers/104-gitea.md b/knowledge/wiki/containers/104-gitea.md similarity index 100% rename from containers/104-gitea.md rename to knowledge/wiki/containers/104-gitea.md diff --git a/containers/105-apps.md b/knowledge/wiki/containers/105-apps.md similarity index 100% rename from containers/105-apps.md rename to knowledge/wiki/containers/105-apps.md diff --git a/containers/106-auth-outpost.md b/knowledge/wiki/containers/106-auth-outpost.md similarity index 81% rename from containers/106-auth-outpost.md rename to knowledge/wiki/containers/106-auth-outpost.md index c1293a4..19c16b1 100644 --- a/containers/106-auth-outpost.md +++ b/knowledge/wiki/containers/106-auth-outpost.md @@ -1,6 +1,6 @@ # 106 — `auth-outpost` -Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik. +Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik. ## At a glance - **Hostname:** `auth-outpost` @@ -12,7 +12,7 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co ## Role -Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../investigations/2026-05-31-authentik-vps-migration.md). +Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../../investigations/2026-05-31-authentik-vps-migration.md). ## Service / port map | Service | Listen | Notes | @@ -45,12 +45,12 @@ Fix: the LAN outpost gets its **own** domain. - [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only) - [Caddy (121)](121-caddy.md) — forward-auth consumer - [Ingress (VPS traefik)](../infrastructure/ingress.md) -- [Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md) +- [Authentik VPS migration](../../../investigations/2026-05-31-authentik-vps-migration.md) ## Changelog ### 2026-06-06 — Authentik session lifetime extended to 30 days -VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../investigations/2026-06-06-authentik-session-lifetime.md). +VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../../investigations/2026-06-06-authentik-session-lifetime.md). ### 2026-06-01 — created; forward-auth cut over from LXC 124 -New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md). +New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md). diff --git a/containers/107-dns.md b/knowledge/wiki/containers/107-dns.md similarity index 93% rename from containers/107-dns.md rename to knowledge/wiki/containers/107-dns.md index a07e2e7..74a99a6 100644 --- a/containers/107-dns.md +++ b/knowledge/wiki/containers/107-dns.md @@ -29,7 +29,7 @@ Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recur - **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field. ## dns-sync (Technitium = authoring source) -`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md). +`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../../../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md). ## DHCP @@ -55,7 +55,7 @@ Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)) Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`. ### 2026-06-03 — DHCP pool narrowed to `.241–.254` -Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../../../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). ### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone) This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md). diff --git a/containers/114-nextcloud.md b/knowledge/wiki/containers/114-nextcloud.md similarity index 100% rename from containers/114-nextcloud.md rename to knowledge/wiki/containers/114-nextcloud.md diff --git a/containers/118-elementsynapse.md b/knowledge/wiki/containers/118-elementsynapse.md similarity index 100% rename from containers/118-elementsynapse.md rename to knowledge/wiki/containers/118-elementsynapse.md diff --git a/containers/119-sophia.md b/knowledge/wiki/containers/119-sophia.md similarity index 100% rename from containers/119-sophia.md rename to knowledge/wiki/containers/119-sophia.md diff --git a/containers/120-mule-images.md b/knowledge/wiki/containers/120-mule-images.md similarity index 100% rename from containers/120-mule-images.md rename to knowledge/wiki/containers/120-mule-images.md diff --git a/containers/121-caddy.md b/knowledge/wiki/containers/121-caddy.md similarity index 97% rename from containers/121-caddy.md rename to knowledge/wiki/containers/121-caddy.md index 3e7ff19..a607a60 100644 --- a/containers/121-caddy.md +++ b/knowledge/wiki/containers/121-caddy.md @@ -85,7 +85,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip - **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits - **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5 -Also: [elementsynapse LXC 118](../containers/118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed. +Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed. ### 2026-06-02 — caddy.service unit missing; recreated After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it. diff --git a/containers/122-arriman.md b/knowledge/wiki/containers/122-arriman.md similarity index 100% rename from containers/122-arriman.md rename to knowledge/wiki/containers/122-arriman.md diff --git a/containers/128-trmnl.md b/knowledge/wiki/containers/128-trmnl.md similarity index 98% rename from containers/128-trmnl.md rename to knowledge/wiki/containers/128-trmnl.md index e15e569..c56f466 100644 --- a/containers/128-trmnl.md +++ b/knowledge/wiki/containers/128-trmnl.md @@ -35,7 +35,7 @@ Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env - [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router) - [DNS (107)](107-dns.md) — Technitium A record `trmnl → 192.168.8.175` (LAN path via Caddy) - [Gitea (104)](104-gitea.md) — source repo `dtoro/terminalito` -- [Plan: 2026-06-24 TRMNL plugins LXC](../plans/2026-06-24-trmnl-plugins-lxc.md) +- [Plan: 2026-06-24 TRMNL plugins LXC](../../../plans/2026-06-24-trmnl-plugins-lxc.md) ## Changelog ### 2026-06-24 — auto-deploy + LAN DNS wired diff --git a/containers/129-house.md b/knowledge/wiki/containers/129-house.md similarity index 97% rename from containers/129-house.md rename to knowledge/wiki/containers/129-house.md index 333fabb..30f3b44 100644 --- a/containers/129-house.md +++ b/knowledge/wiki/containers/129-house.md @@ -40,7 +40,7 @@ Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modul - [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy) - [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`) - [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source -- [Deployment plan](../plans/2026-06-25-yuvomi-deployment.md) +- [Deployment plan](../../../plans/2026-06-25-yuvomi-deployment.md) ## Changelog diff --git a/containers/130-grimmory.md b/knowledge/wiki/containers/130-grimmory.md similarity index 100% rename from containers/130-grimmory.md rename to knowledge/wiki/containers/130-grimmory.md diff --git a/containers/131-teddycloud.md b/knowledge/wiki/containers/131-teddycloud.md similarity index 94% rename from containers/131-teddycloud.md rename to knowledge/wiki/containers/131-teddycloud.md index 3531cab..518e9cf 100644 --- a/containers/131-teddycloud.md +++ b/knowledge/wiki/containers/131-teddycloud.md @@ -7,7 +7,7 @@ audio content against a local server instead of the official cloud. Predates the client-enrollment convention entirely; nobody wrote it down. Found and documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged `pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see -[OIKOS.md](../OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already +[OIKOS.md](../../../OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud` container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict for it — but it never got its own inventory entry or doc page until now. @@ -50,7 +50,7 @@ for the first time. - [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already taken by this container - [Containers index](index.md) -- [OIKOS.md](../OIKOS.md) — drift detector that caught this +- [OIKOS.md](../../../OIKOS.md) — drift detector that caught this ## Changelog diff --git a/containers/132-rclone.md b/knowledge/wiki/containers/132-rclone.md similarity index 99% rename from containers/132-rclone.md rename to knowledge/wiki/containers/132-rclone.md index 800adda..c322a6e 100644 --- a/containers/132-rclone.md +++ b/knowledge/wiki/containers/132-rclone.md @@ -95,7 +95,7 @@ after this set. Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH), rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to -[agent-enrollment.md troubleshooting](../operations/agent-enrollment.md#troubleshooting) if it recurs +[agent-enrollment.md troubleshooting](../../../operations/agent-enrollment.md#troubleshooting) if it recurs on future LXC bootstraps. ## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved) diff --git a/containers/133-seanime.md b/knowledge/wiki/containers/133-seanime.md similarity index 100% rename from containers/133-seanime.md rename to knowledge/wiki/containers/133-seanime.md diff --git a/containers/134-romm.md b/knowledge/wiki/containers/134-romm.md similarity index 100% rename from containers/134-romm.md rename to knowledge/wiki/containers/134-romm.md diff --git a/containers/archive/123-claudio-bot.md b/knowledge/wiki/containers/archive/123-claudio-bot.md similarity index 100% rename from containers/archive/123-claudio-bot.md rename to knowledge/wiki/containers/archive/123-claudio-bot.md diff --git a/containers/archive/127-mule-photos-new.md b/knowledge/wiki/containers/archive/127-mule-photos-new.md similarity index 100% rename from containers/archive/127-mule-photos-new.md rename to knowledge/wiki/containers/archive/127-mule-photos-new.md diff --git a/containers/index.md b/knowledge/wiki/containers/index.md similarity index 98% rename from containers/index.md rename to knowledge/wiki/containers/index.md index ca8486d..9119132 100644 --- a/containers/index.md +++ b/knowledge/wiki/containers/index.md @@ -32,7 +32,7 @@ Most containers live on [`hubris`](../hosts/hubris.md). Some have been | 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose | | 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained | | 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology | -| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) | +| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/2026-06-04_130000-deprecate-claudio-bot.md) | | 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty | | 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq | | 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned | diff --git a/hosts/hubris.md b/knowledge/wiki/hosts/hubris.md similarity index 93% rename from hosts/hubris.md rename to knowledge/wiki/hosts/hubris.md index d14e10a..d5155b2 100644 --- a/hosts/hubris.md +++ b/knowledge/wiki/hosts/hubris.md @@ -8,7 +8,7 @@ workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node ## At a glance - **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`) - **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`). -- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md). +- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../../investigations/2026-04-21-hubris-crash-loop.md). - **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`. - **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`. - **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP. @@ -46,7 +46,7 @@ Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md) it's a physical thinpool that only exists on this host's hardware. - strong currently hosts no LXCs/VMs — it exists solely as a cluster member so far. See [strong.md](strong.md) and the [library-SSD - migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) + migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) for what comes next (physical drive move, service migration — not started). ## Tenants @@ -113,20 +113,20 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220 - [Media permissions](../infrastructure/media-permissions.md) - [Monitoring](../infrastructure/monitoring.md) - [Backups (disabled)](../infrastructure/backups.md) -- [Operations cheatsheet](../operations/commands.md) -- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md) +- [Operations cheatsheet](../../../operations/commands.md) +- [Investigation: 2026-04-21 crash loop](../../../investigations/2026-04-21-hubris-crash-loop.md) - [strong — Proxmox host](strong.md) ## Changelog ### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab") -User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed. +User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed. ### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured -Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/2026-06-01-slate-ax-to-sodola-migration.md). ### 2026-05-14 — LXC 109 (syncthing) decommissioned -User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md). +User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md). ### 2026-05-14 — network performance baseline captured First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above. @@ -138,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. `/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel. ### 2026-04-29 — relocated to better-ventilated spot -User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation). +User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation). ### 2026-04-28 — Phase 1 WiFi failover Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them. @@ -147,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since. ### 2026-04-23 — SSD cooling + thermal pads installed -Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). +Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). ### 2026-04-22 — drive removal A/B test -Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). +Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). ### 2026-04-22 — `cpu-epp.service` ordering bug fixed Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`. @@ -159,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind ` `60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else. ### 2026-04-21 — `cpu-epp.service` deployed -Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md). +Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../../investigations/2026-04-21-hubris-crash-loop.md). diff --git a/hosts/strong.md b/knowledge/wiki/hosts/strong.md similarity index 95% rename from hosts/strong.md rename to knowledge/wiki/hosts/strong.md index 1778a4b..e3df409 100644 --- a/hosts/strong.md +++ b/knowledge/wiki/hosts/strong.md @@ -28,7 +28,7 @@ Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet. was actually created live; not worth renaming), restricted `nodes strong` in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool. Empty so far; this is separate from the [library-SSD migration - plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s + plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s planned drive move from hubris (that hasn't happened) — this is general-purpose VM/CT capacity. - **Network:** `vmbr0` is bridged straight onto the household LAN — @@ -50,13 +50,13 @@ Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet. `bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the 5-min sync timer are live; `homelab whoami` resolves correctly. See - [agent-enrollment.md](../operations/agent-enrollment.md). + [agent-enrollment.md](../../../operations/agent-enrollment.md). - **Age key / secrets:** issued the same day over plain LAN (no Netbird needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at `/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4` recorded in `inventory.yaml`. Not yet a recipient on any actual secret (`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see - ["Granting a secret to a new client"](../operations/agent-enrollment.md#granting-a-secret-to-a-new-client). + ["Granting a secret to a new client"](../../../operations/agent-enrollment.md#granting-a-secret-to-a-new-client). ## Cluster membership @@ -76,10 +76,10 @@ needed going forward. ## Related - [hubris — Proxmox host](hubris.md) -- [Library SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical) +- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical) - [Network](../infrastructure/network.md) - [SSH access](../infrastructure/ssh-access.md) -- [Agent enrollment](../operations/agent-enrollment.md) +- [Agent enrollment](../../../operations/agent-enrollment.md) ## Changelog diff --git a/infrastructure/auto-deploy.md b/knowledge/wiki/infrastructure/auto-deploy.md similarity index 97% rename from infrastructure/auto-deploy.md rename to knowledge/wiki/infrastructure/auto-deploy.md index f8fa9e6..f998201 100644 --- a/infrastructure/auto-deploy.md +++ b/knowledge/wiki/infrastructure/auto-deploy.md @@ -44,13 +44,13 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web | `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart | | `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart | | `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` | -| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../oikos/console/deploy/README.md) | +| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) | > Note: `dtoro/Homelab-Docs` has **three webhooks** firing on the same push. > Each owns its own clone on LXC 105. They don't conflict because each > deploy.sh only touches its own service unit + venv. -> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md). +> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md). ## When you change a tracked config @@ -117,7 +117,7 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l - [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these - [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets - [Backups (disabled)](backups.md) -- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference +- [Operations cheatsheet](../../../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference ## Changelog diff --git a/infrastructure/backups.md b/knowledge/wiki/infrastructure/backups.md similarity index 94% rename from infrastructure/backups.md rename to knowledge/wiki/infrastructure/backups.md index eba2dd7..99d9bd9 100644 --- a/infrastructure/backups.md +++ b/knowledge/wiki/infrastructure/backups.md @@ -24,7 +24,7 @@ See [132-rclone](../containers/132-rclone.md) for the full design. ## Legacy — restic on external drive (DISABLED 2026-04-22) -Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md). +Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../../investigations/2026-04-21-hubris-crash-loop.md). ## Status @@ -36,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver). -**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). +**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). **To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive. @@ -111,7 +111,7 @@ Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again. -After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). +After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). ## Thermal monitoring @@ -122,7 +122,7 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213 - ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) - [Monitoring](monitoring.md) - [Auto-deploy](auto-deploy.md) -- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md) +- [Investigation: 2026-04-21 crash loop](../../../investigations/2026-04-21-hubris-crash-loop.md) ## Changelog @@ -133,7 +133,7 @@ Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclon Initial documentation. Status remains DISABLED. ### 2026-04-22 — DISABLED -Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. +Drive removed as the A/B test in the [crash investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. ### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups. diff --git a/infrastructure/dns.md b/knowledge/wiki/infrastructure/dns.md similarity index 89% rename from infrastructure/dns.md rename to knowledge/wiki/infrastructure/dns.md index 011f7ee..c64eccd 100644 --- a/infrastructure/dns.md +++ b/knowledge/wiki/infrastructure/dns.md @@ -7,7 +7,7 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr ## Components - **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22). -- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04). +- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04). - **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns. - **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where. @@ -53,7 +53,7 @@ Creating a new Caddyfile site block is necessary but **not sufficient**. Without 3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`. 4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`. -> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync). +> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../../../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync). ## Public path — what does and doesn't follow the LAN map @@ -94,7 +94,7 @@ The "delete NetBird managed zone → forward everything to Technitium" plan was **Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group. -> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped). +> Reference: [scripts/dns-sync.py](../../../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped). ### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment) The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging. @@ -110,7 +110,7 @@ All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float o Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients. ### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`) -Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS). +Authentik migrated off LXC 124 onto the VPS (see [investigation](../../../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS). ### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP) NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful. diff --git a/infrastructure/homelab-context.md b/knowledge/wiki/infrastructure/homelab-context.md similarity index 97% rename from infrastructure/homelab-context.md rename to knowledge/wiki/infrastructure/homelab-context.md index fb58481..a0b76c9 100644 --- a/infrastructure/homelab-context.md +++ b/knowledge/wiki/infrastructure/homelab-context.md @@ -5,7 +5,7 @@ Code, Hermes Agent, future MCP-capable clients) on every machine in the lab self-locating and able to read the same source of truth. Operational walkthrough for enrolling a new client lives in -[operations/agent-enrollment.md](../operations/agent-enrollment.md); this +[operations/agent-enrollment.md](../../../operations/agent-enrollment.md); this page is the architecture reference. ## What's where @@ -121,7 +121,7 @@ The MCP server and secrets-issuance each have their own clone ## Related -- [Operations: agent enrollment](../operations/agent-enrollment.md) — the +- [Operations: agent enrollment](../../../operations/agent-enrollment.md) — the step-by-step for adding a new client - [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance` pipelines (and the rest of the lab's webhook pipelines) @@ -133,7 +133,7 @@ The MCP server and secrets-issuance each have their own clone ## Changelog ### 2026-05-20 — system live across hubris, apps, republic-laptop -Phase 1 of the [cross-client context plan](../README.md) merged. Three +Phase 1 of the [cross-client context plan](../../../README.md) merged. Three clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS decrypt verified on each. Webhook auto-deploy for both LXC 105 services wired (hook ids 10 + 11). `homelab refresh-creds` + atomic diff --git a/infrastructure/index.md b/knowledge/wiki/infrastructure/index.md similarity index 94% rename from infrastructure/index.md rename to knowledge/wiki/infrastructure/index.md index 3b86737..bb451f1 100644 --- a/infrastructure/index.md +++ b/knowledge/wiki/infrastructure/index.md @@ -60,7 +60,7 @@ are documented in their own pages. Each system below links to its full doc. ## Related -- [README](../README.md) — entry point +- [README](../../../README.md) — entry point - [Containers index](../containers/index.md) -- [Operations cheatsheet](../operations/commands.md) -- [OIKOS operating model](../OIKOS.md) \ No newline at end of file +- [Operations cheatsheet](../../../operations/commands.md) +- [OIKOS operating model](../../../OIKOS.md) \ No newline at end of file diff --git a/infrastructure/ingress.md b/knowledge/wiki/infrastructure/ingress.md similarity index 90% rename from infrastructure/ingress.md rename to knowledge/wiki/infrastructure/ingress.md index cb06518..75e6c5f 100644 --- a/infrastructure/ingress.md +++ b/knowledge/wiki/infrastructure/ingress.md @@ -49,7 +49,7 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8 ### `auth.hubris.network` — different pattern (local container, not cert-mirror) -Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above: +Since 2026-05-31 [Authentik runs on the VPS itself](../../../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above: - Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`. - TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough). @@ -91,7 +91,7 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for ` TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed. ### 2026-05-31 — `auth.hubris.network` now served locally on the VPS -Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack. +Authentik migrated onto the VPS ([investigation](../../../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack. ### 2026-04-28 — wiki entry created Initial documentation. diff --git a/infrastructure/media-permissions.md b/knowledge/wiki/infrastructure/media-permissions.md similarity index 100% rename from infrastructure/media-permissions.md rename to knowledge/wiki/infrastructure/media-permissions.md diff --git a/infrastructure/mesh.md b/knowledge/wiki/infrastructure/mesh.md similarity index 99% rename from infrastructure/mesh.md rename to knowledge/wiki/infrastructure/mesh.md index 1d7d78b..52711cc 100644 --- a/infrastructure/mesh.md +++ b/knowledge/wiki/infrastructure/mesh.md @@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man ## Changelog ### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below) -The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md). +The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../../investigations/2026-05-31-authentik-vps-migration.md). ### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik diff --git a/infrastructure/monitoring.md b/knowledge/wiki/infrastructure/monitoring.md similarity index 100% rename from infrastructure/monitoring.md rename to knowledge/wiki/infrastructure/monitoring.md diff --git a/infrastructure/network.md b/knowledge/wiki/infrastructure/network.md similarity index 94% rename from infrastructure/network.md rename to knowledge/wiki/infrastructure/network.md index 54ded7b..fb94a5d 100644 --- a/infrastructure/network.md +++ b/knowledge/wiki/infrastructure/network.md @@ -64,7 +64,7 @@ No NAT on Proxmox — traffic flows without double-NAT. ## Remote access -- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../vps/). +- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../../../vps/). - **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md). ## Related @@ -79,10 +79,10 @@ No NAT on Proxmox — traffic flows without double-NAT. ### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2) Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed. -Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). ### 2026-06-02 — Executed migration; Proxmox as subnet router -Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/2026-06-01-slate-ax-to-sodola-migration.md). ### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added -Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/2026-06-01-slate-ax-to-sodola-migration.md). diff --git a/infrastructure/ssh-access.md b/knowledge/wiki/infrastructure/ssh-access.md similarity index 98% rename from infrastructure/ssh-access.md rename to knowledge/wiki/infrastructure/ssh-access.md index 44c05eb..fa8ea73 100644 --- a/infrastructure/ssh-access.md +++ b/knowledge/wiki/infrastructure/ssh-access.md @@ -178,8 +178,8 @@ done - [Mesh migration](mesh.md) - [VPS hardening](vps-hardening.md) -- [Agent enrollment](../operations/agent-enrollment.md) -- [Homelab CLI](../bin/homelab) +- [Agent enrollment](../../../operations/agent-enrollment.md) +- [Homelab CLI](../../../bin/homelab) ## Changelog diff --git a/infrastructure/topology.md b/knowledge/wiki/infrastructure/topology.md similarity index 95% rename from infrastructure/topology.md rename to knowledge/wiki/infrastructure/topology.md index ee1764d..5bb1a9d 100644 --- a/infrastructure/topology.md +++ b/knowledge/wiki/infrastructure/topology.md @@ -3,8 +3,8 @@ # Topology (generated) -Source: [inventory.yaml](../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services. -Edge semantics: [oikos/ontology.yaml](../oikos/ontology.yaml). Operating model: [OIKOS.md](../OIKOS.md). +Source: [inventory.yaml](../../../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services. +Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). Operating model: [OIKOS.md](../../../.agents/OIKOS.md). ## Compute & ingress diff --git a/infrastructure/vps-hardening.md b/knowledge/wiki/infrastructure/vps-hardening.md similarity index 100% rename from infrastructure/vps-hardening.md rename to knowledge/wiki/infrastructure/vps-hardening.md diff --git a/vms/100-zimaos.md b/knowledge/wiki/vms/100-zimaos.md similarity index 99% rename from vms/100-zimaos.md rename to knowledge/wiki/vms/100-zimaos.md index 54a7959..ac58ac6 100644 --- a/vms/100-zimaos.md +++ b/knowledge/wiki/vms/100-zimaos.md @@ -59,7 +59,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. ## Changelog ### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed -ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). ### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix) diff --git a/vms/108-haos.md b/knowledge/wiki/vms/108-haos.md similarity index 100% rename from vms/108-haos.md rename to knowledge/wiki/vms/108-haos.md diff --git a/vms/index.md b/knowledge/wiki/vms/index.md similarity index 94% rename from vms/index.md rename to knowledge/wiki/vms/index.md index ad649c1..f61daae 100644 --- a/vms/index.md +++ b/knowledge/wiki/vms/index.md @@ -11,4 +11,4 @@ Two QEMU VMs running on [hubris](../hosts/hubris.md): - [Hubris host](../hosts/hubris.md) — both VMs run here - [Containers index](../containers/index.md) — LXCs on both nodes -- [README](../README.md) \ No newline at end of file +- [README](../../../README.md) \ No newline at end of file diff --git a/oikos/approve.py b/oikos/approve.py index a10fd47..89449b5 100644 --- a/oikos/approve.py +++ b/oikos/approve.py @@ -5,7 +5,7 @@ Repo-side half of the Week-3 approval flow. This module owns the request/ grant lifecycle and the HMAC signing; it does NOT talk to Matrix directly. There is no dedicated Matrix bot in this homelab — alerts already go out as the operator's own Hermes agent posting to @dtoro:avispero (see -infrastructure/monitoring.md's homelab-health-watchdog). The integration +knowledge/wiki/infrastructure/monitoring.md's homelab-health-watchdog). The integration contract is: 1. An agent or the Week-3 scheduler calls `request()` (or the CLI diff --git a/oikos/cards/host-apps.md b/oikos/cards/host-apps.md index 038fa80..4afe8ac 100644 --- a/oikos/cards/host-apps.md +++ b/oikos/cards/host-apps.md @@ -6,7 +6,7 @@ - role: docker-apps - address: 192.168.8.205 (mesh: tailscale:apps) - mounts: /mnt/library -- doc: containers/105-apps.md +- doc: knowledge/wiki/containers/105-apps.md - secrets: enrolled (age key present) ## Blast radius @@ -18,4 +18,4 @@ - see the services this host runs for action-level risk classes ## Recent changes -- (none yet) +- 2026-07-06T11:57:21+00:00 deploy-oikos-console (config_mutation) — ok diff --git a/oikos/cards/host-arriman.md b/oikos/cards/host-arriman.md index f641fa5..5db5481 100644 --- a/oikos/cards/host-arriman.md +++ b/oikos/cards/host-arriman.md @@ -6,7 +6,7 @@ - role: arr-stack - address: 192.168.8.245 (mesh: tailscale:arr) - mounts: /mnt/media_local -- doc: containers/122-arriman.md +- doc: knowledge/wiki/containers/122-arriman.md ## Blast radius - impacts: service:arr_stack diff --git a/oikos/cards/host-auth-outpost.md b/oikos/cards/host-auth-outpost.md index b450032..525c654 100644 --- a/oikos/cards/host-auth-outpost.md +++ b/oikos/cards/host-auth-outpost.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: authentik-gateway - address: 192.168.8.6 -- doc: containers/106-auth-outpost.md +- doc: knowledge/wiki/containers/106-auth-outpost.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-caddy.md b/oikos/cards/host-caddy.md index 73a53a3..ee80bca 100644 --- a/oikos/cards/host-caddy.md +++ b/oikos/cards/host-caddy.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: reverse-proxy - address: 192.168.8.175 -- doc: containers/121-caddy.md +- doc: knowledge/wiki/containers/121-caddy.md ## Blast radius - impacts: service:caddy diff --git a/oikos/cards/host-dns.md b/oikos/cards/host-dns.md index 7157d47..0b76c3f 100644 --- a/oikos/cards/host-dns.md +++ b/oikos/cards/host-dns.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: dns-server - address: 192.168.8.2 -- doc: containers/107-dns.md +- doc: knowledge/wiki/containers/107-dns.md ## Blast radius - impacts: service:dns @@ -16,4 +16,4 @@ - see the services this host runs for action-level risk classes ## Recent changes -- (none yet) +- 2026-07-06T11:40:28+00:00 add-record (config_mutation) — ok diff --git a/oikos/cards/host-elementsynapse.md b/oikos/cards/host-elementsynapse.md index 461ace3..6509c96 100644 --- a/oikos/cards/host-elementsynapse.md +++ b/oikos/cards/host-elementsynapse.md @@ -5,7 +5,7 @@ - runs-on: host:strong - role: matrix-server - address: 192.168.8.242 -- doc: containers/118-elementsynapse.md +- doc: knowledge/wiki/containers/118-elementsynapse.md ## Blast radius - impacts: service:matrix diff --git a/oikos/cards/host-gitea.md b/oikos/cards/host-gitea.md index ad600a6..a77eb7f 100644 --- a/oikos/cards/host-gitea.md +++ b/oikos/cards/host-gitea.md @@ -6,7 +6,7 @@ - role: git-server - address: 192.168.8.121 (mesh: tailscale:gitea) - mounts: /mnt/library -- doc: containers/104-gitea.md +- doc: knowledge/wiki/containers/104-gitea.md ## Blast radius - impacts: service:gitea diff --git a/oikos/cards/host-grimmory.md b/oikos/cards/host-grimmory.md index d6a0c72..706377c 100644 --- a/oikos/cards/host-grimmory.md +++ b/oikos/cards/host-grimmory.md @@ -6,7 +6,7 @@ - role: book-library - address: 192.168.8.247 - mounts: /mnt/media_local -- doc: containers/130-grimmory.md +- doc: knowledge/wiki/containers/130-grimmory.md - secrets: enrolled (age key present) ## Blast radius diff --git a/oikos/cards/host-haos.md b/oikos/cards/host-haos.md index ada4e76..45f2aba 100644 --- a/oikos/cards/host-haos.md +++ b/oikos/cards/host-haos.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: home-automation - address: 192.168.8.101 (mesh: tailscale:homeassistant) -- doc: vms/108-haos.md +- doc: knowledge/wiki/vms/108-haos.md ## Blast radius - impacts: service:haos diff --git a/oikos/cards/host-house.md b/oikos/cards/host-house.md index 3ad239e..5f3808c 100644 --- a/oikos/cards/host-house.md +++ b/oikos/cards/host-house.md @@ -5,7 +5,7 @@ - runs-on: host:strong - role: family-planner - address: 192.168.8.244 -- doc: containers/129-house.md +- doc: knowledge/wiki/containers/129-house.md - secrets: enrolled (age key present) ## Blast radius diff --git a/oikos/cards/host-hubris.md b/oikos/cards/host-hubris.md index 9ae7e5d..57236b6 100644 --- a/oikos/cards/host-hubris.md +++ b/oikos/cards/host-hubris.md @@ -5,7 +5,7 @@ - role: hypervisor - address: 192.168.8.77 (mesh: netbird:proxmox-server.netbird.selfhosted) - mounts: /mnt/library -- doc: hosts/hubris.md +- doc: knowledge/wiki/hosts/hubris.md - secrets: enrolled (age key present) ## Blast radius diff --git a/oikos/cards/host-jellyfin.md b/oikos/cards/host-jellyfin.md index 162b908..178c58d 100644 --- a/oikos/cards/host-jellyfin.md +++ b/oikos/cards/host-jellyfin.md @@ -6,7 +6,7 @@ - role: media-server - address: 192.168.8.246 (mesh: tailscale:jellyfin) - mounts: /mnt/media_local -- doc: containers/101-jellyfin.md +- doc: knowledge/wiki/containers/101-jellyfin.md ## Blast radius - impacts: service:jellyfin diff --git a/oikos/cards/host-mule-images.md b/oikos/cards/host-mule-images.md index 242f193..c1c98ca 100644 --- a/oikos/cards/host-mule-images.md +++ b/oikos/cards/host-mule-images.md @@ -6,7 +6,7 @@ - role: photo-management - address: 192.168.8.136 (mesh: tailscale:muleimage) - mounts: /mnt/library -- doc: containers/120-mule-images.md +- doc: knowledge/wiki/containers/120-mule-images.md ## Blast radius - impacts: service:photos diff --git a/oikos/cards/host-nextcloud.md b/oikos/cards/host-nextcloud.md index f3de0e1..2cd8e3b 100644 --- a/oikos/cards/host-nextcloud.md +++ b/oikos/cards/host-nextcloud.md @@ -6,7 +6,7 @@ - role: file-sync - address: 192.168.8.224 (mesh: tailscale:nextcloud) - mounts: /mnt/library -- doc: containers/114-nextcloud.md +- doc: knowledge/wiki/containers/114-nextcloud.md ## Blast radius - impacts: service:nextcloud diff --git a/oikos/cards/host-nfs-export.md b/oikos/cards/host-nfs-export.md index 53102e9..3731f13 100644 --- a/oikos/cards/host-nfs-export.md +++ b/oikos/cards/host-nfs-export.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: storage-export - address: 192.168.8.200 -- doc: containers/102-nfs-export.md +- doc: knowledge/wiki/containers/102-nfs-export.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-paperless.md b/oikos/cards/host-paperless.md index f21f1ab..f993551 100644 --- a/oikos/cards/host-paperless.md +++ b/oikos/cards/host-paperless.md @@ -6,7 +6,7 @@ - role: document-archive - address: 192.168.8.130 (mesh: tailscale:paperless) - mounts: /mnt/library -- doc: containers/103-paperless.md +- doc: knowledge/wiki/containers/103-paperless.md ## Blast radius - impacts: service:paperless diff --git a/oikos/cards/host-romm.md b/oikos/cards/host-romm.md index 6e9d58b..31fd618 100644 --- a/oikos/cards/host-romm.md +++ b/oikos/cards/host-romm.md @@ -6,7 +6,7 @@ - role: rom-manager - address: 192.168.8.249 - mounts: /mnt/media_local -- doc: containers/134-romm.md +- doc: knowledge/wiki/containers/134-romm.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-seanime.md b/oikos/cards/host-seanime.md index d2d58eb..155d991 100644 --- a/oikos/cards/host-seanime.md +++ b/oikos/cards/host-seanime.md @@ -6,7 +6,7 @@ - role: anime-media-server - address: 192.168.8.248 - mounts: /mnt/media_local/anime -- doc: containers/133-seanime.md +- doc: knowledge/wiki/containers/133-seanime.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-sophia.md b/oikos/cards/host-sophia.md index 505ebef..196efef 100644 --- a/oikos/cards/host-sophia.md +++ b/oikos/cards/host-sophia.md @@ -6,7 +6,7 @@ - role: workshop - address: 192.168.8.109 (mesh: tailscale:sophia) - mounts: /mnt/library -- doc: containers/119-sophia.md +- doc: knowledge/wiki/containers/119-sophia.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-strong.md b/oikos/cards/host-strong.md index 2ada6c4..9b921c1 100644 --- a/oikos/cards/host-strong.md +++ b/oikos/cards/host-strong.md @@ -4,7 +4,7 @@ - state: active - role: hypervisor - address: 192.168.178.181 -- doc: hosts/strong.md +- doc: knowledge/wiki/hosts/strong.md - secrets: enrolled (age key present) ## Blast radius diff --git a/oikos/cards/host-teddycloud.md b/oikos/cards/host-teddycloud.md index f496a1e..57bc992 100644 --- a/oikos/cards/host-teddycloud.md +++ b/oikos/cards/host-teddycloud.md @@ -6,7 +6,7 @@ - role: teddycloud - address: 192.168.8.150 - mounts: /mnt/library -- doc: containers/131-teddycloud.md +- doc: knowledge/wiki/containers/131-teddycloud.md ## Blast radius - impacts: service:teddycloud @@ -17,4 +17,4 @@ - see the services this host runs for action-level risk classes ## Recent changes -- (none yet) +- 2026-07-06T11:05:35+00:00 activate (config_mutation) — ok diff --git a/oikos/cards/host-trmnl.md b/oikos/cards/host-trmnl.md index d5d8d33..741b975 100644 --- a/oikos/cards/host-trmnl.md +++ b/oikos/cards/host-trmnl.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: trmnl-middleware - address: 192.168.8.211 -- doc: containers/128-trmnl.md +- doc: knowledge/wiki/containers/128-trmnl.md ## Blast radius - impacts: service:trmnl diff --git a/oikos/cards/host-zimaos.md b/oikos/cards/host-zimaos.md index 877b8f1..b249fa4 100644 --- a/oikos/cards/host-zimaos.md +++ b/oikos/cards/host-zimaos.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: nas-frontend-eval - address: 192.168.8.195 -- doc: vms/100-zimaos.md +- doc: knowledge/wiki/vms/100-zimaos.md ## Blast radius - impacts: service:zimaos diff --git a/oikos/cards/service-arr_stack.md b/oikos/cards/service-arr_stack.md index 335f5c5..2c09e3b 100644 --- a/oikos/cards/service-arr_stack.md +++ b/oikos/cards/service-arr_stack.md @@ -1,7 +1,7 @@ # arr_stack (service:arr_stack) - backend: host:arriman -- doc: containers/122-arriman.md +- doc: knowledge/wiki/containers/122-arriman.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/service-artifacto.md b/oikos/cards/service-artifacto.md index 756c158..fc857a3 100644 --- a/oikos/cards/service-artifacto.md +++ b/oikos/cards/service-artifacto.md @@ -2,7 +2,7 @@ - backend: host:apps - url: https://artifacto.hubris.network -- doc: containers/105-apps.md +- doc: knowledge/wiki/containers/105-apps.md - config repo: dtoro/Artifacto ## Blast radius diff --git a/oikos/cards/service-authentik.md b/oikos/cards/service-authentik.md index 252ddea..93ec04b 100644 --- a/oikos/cards/service-authentik.md +++ b/oikos/cards/service-authentik.md @@ -2,7 +2,7 @@ - backend: host:netbird-vps - url: https://auth.hubris.network -- doc: containers/106-auth-outpost.md +- doc: knowledge/wiki/containers/106-auth-outpost.md - risk notes: SSO provider — outage locks login to OIDC/forward-auth services ## Blast radius diff --git a/oikos/cards/service-caddy.md b/oikos/cards/service-caddy.md index 69e013d..a26093d 100644 --- a/oikos/cards/service-caddy.md +++ b/oikos/cards/service-caddy.md @@ -1,7 +1,7 @@ # caddy (service:caddy) - backend: host:caddy -- doc: containers/121-caddy.md +- doc: knowledge/wiki/containers/121-caddy.md - config repo: dtoro/caddy-conf - risk notes: wide blast radius — every *.hubris.network route rides on it (see oikos/policy.yaml service_overrides) @@ -17,4 +17,4 @@ - edit-config-and-deploy — config_mutation (approval: operator) ## Recent changes -- (none yet) +- 2026-07-06T11:29:56+00:00 add-site-block (config_mutation) — ok diff --git a/oikos/cards/service-dns.md b/oikos/cards/service-dns.md index 1b3af2e..a7a3d9e 100644 --- a/oikos/cards/service-dns.md +++ b/oikos/cards/service-dns.md @@ -1,7 +1,7 @@ # dns (service:dns) - backend: host:dns -- doc: containers/107-dns.md +- doc: knowledge/wiki/containers/107-dns.md - risk notes: LAN-wide resolver — misconfig breaks name resolution for every client ## Blast radius diff --git a/oikos/cards/service-gitea.md b/oikos/cards/service-gitea.md index ab6e9bf..89dca93 100644 --- a/oikos/cards/service-gitea.md +++ b/oikos/cards/service-gitea.md @@ -2,7 +2,7 @@ - backend: host:gitea - url: https://git.hubris.network -- doc: containers/104-gitea.md +- doc: knowledge/wiki/containers/104-gitea.md - config repo: dtoro/gitea-customizations - risk notes: hosts all config repos + deploy webhooks; outage blocks auto-deploy and sync diff --git a/oikos/cards/service-haos.md b/oikos/cards/service-haos.md index 60e6473..4c3401b 100644 --- a/oikos/cards/service-haos.md +++ b/oikos/cards/service-haos.md @@ -1,7 +1,7 @@ # haos (service:haos) - backend: host:haos -- doc: vms/108-haos.md +- doc: knowledge/wiki/vms/108-haos.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/service-homelab_mcp.md b/oikos/cards/service-homelab_mcp.md index 29462c6..fb5a156 100644 --- a/oikos/cards/service-homelab_mcp.md +++ b/oikos/cards/service-homelab_mcp.md @@ -2,7 +2,7 @@ - backend: host:apps - url: https://mcp.hubris.network/mcp -- doc: infrastructure/homelab-context.md +- doc: knowledge/wiki/infrastructure/homelab-context.md - config repo: dtoro/Homelab-Docs - risk notes: agents' primary read surface — outage degrades every agent to grepping the clone diff --git a/oikos/cards/service-jellyfin.md b/oikos/cards/service-jellyfin.md index 75085ca..e12e850 100644 --- a/oikos/cards/service-jellyfin.md +++ b/oikos/cards/service-jellyfin.md @@ -2,7 +2,7 @@ - backend: host:jellyfin - url: https://media.hubris.network -- doc: containers/101-jellyfin.md +- doc: knowledge/wiki/containers/101-jellyfin.md - risk notes: native Authentik OIDC via SSO-Auth plugin, no Caddy forward-auth gate; VAAPI transcode depends on GPU passthrough on strong ## Blast radius diff --git a/oikos/cards/service-matrix.md b/oikos/cards/service-matrix.md index 6d109a4..d321ee2 100644 --- a/oikos/cards/service-matrix.md +++ b/oikos/cards/service-matrix.md @@ -2,7 +2,7 @@ - backend: host:elementsynapse - url: https://matrix.hubris.network -- doc: containers/118-elementsynapse.md +- doc: knowledge/wiki/containers/118-elementsynapse.md - risk notes: alert/approval channel for Oikos — outage silences agent escalation ## Blast radius diff --git a/oikos/cards/service-nextcloud.md b/oikos/cards/service-nextcloud.md index 88d1df8..995644b 100644 --- a/oikos/cards/service-nextcloud.md +++ b/oikos/cards/service-nextcloud.md @@ -2,7 +2,7 @@ - backend: host:nextcloud - url: https://cloud.hubris.network -- doc: containers/114-nextcloud.md +- doc: knowledge/wiki/containers/114-nextcloud.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/service-paperless.md b/oikos/cards/service-paperless.md index 7f0c516..cc02065 100644 --- a/oikos/cards/service-paperless.md +++ b/oikos/cards/service-paperless.md @@ -2,7 +2,7 @@ - backend: host:paperless - url: https://paperless.hubris.network -- doc: containers/103-paperless.md +- doc: knowledge/wiki/containers/103-paperless.md - risk notes: document archive — treat data as irreplaceable; DB operations are destructive-class ## Blast radius diff --git a/oikos/cards/service-photos.md b/oikos/cards/service-photos.md index 76b64d8..a8f9442 100644 --- a/oikos/cards/service-photos.md +++ b/oikos/cards/service-photos.md @@ -2,7 +2,7 @@ - backend: host:mule-images - url: https://photos.hubris.network -- doc: containers/120-mule-images.md +- doc: knowledge/wiki/containers/120-mule-images.md - config repo: dtoro/mule-image ## Blast radius diff --git a/oikos/cards/service-proxmox_ui.md b/oikos/cards/service-proxmox_ui.md index 4c394ee..a83de8c 100644 --- a/oikos/cards/service-proxmox_ui.md +++ b/oikos/cards/service-proxmox_ui.md @@ -2,7 +2,7 @@ - backend: host:hubris - url: https://proxmox.hubris.network -- doc: hosts/hubris.md +- doc: knowledge/wiki/hosts/hubris.md - risk notes: hypervisor UI — changes here affect every guest on the node ## Blast radius diff --git a/oikos/cards/service-teddycloud.md b/oikos/cards/service-teddycloud.md index 23eb88d..1533be8 100644 --- a/oikos/cards/service-teddycloud.md +++ b/oikos/cards/service-teddycloud.md @@ -2,7 +2,7 @@ - backend: host:teddycloud - url: https://teddy.hubris.network -- doc: containers/131-teddycloud.md +- doc: knowledge/wiki/containers/131-teddycloud.md - risk notes: no Caddy forward-auth gate (unlike sab.hubris.network on the same Caddyfile) — reachable to anyone on the LAN/mesh who can resolve teddy.hubris.network; undocumented in inventory.yaml until 2026-07-06 (drift-caught) ## Blast radius diff --git a/oikos/cards/service-trmnl.md b/oikos/cards/service-trmnl.md index 690d35e..29e9540 100644 --- a/oikos/cards/service-trmnl.md +++ b/oikos/cards/service-trmnl.md @@ -2,7 +2,7 @@ - backend: host:trmnl - url: https://trmnl.hubris.network -- doc: containers/128-trmnl.md +- doc: knowledge/wiki/containers/128-trmnl.md - config repo: dtoro/terminalito ## Blast radius diff --git a/oikos/cards/service-zimaos.md b/oikos/cards/service-zimaos.md index f834747..6c8f048 100644 --- a/oikos/cards/service-zimaos.md +++ b/oikos/cards/service-zimaos.md @@ -2,7 +2,7 @@ - backend: host:zimaos - url: https://zimaos.hubris.network -- doc: vms/100-zimaos.md +- doc: knowledge/wiki/vms/100-zimaos.md ## Blast radius - impacts: (none) diff --git a/oikos/console/deploy/README.md b/oikos/console/deploy/README.md index ce69fc5..f3e8283 100644 --- a/oikos/console/deploy/README.md +++ b/oikos/console/deploy/README.md @@ -3,7 +3,7 @@ Deploys the same way `homelab-mcp` and `secrets-issuance` already do: Shape B webhook (own checkout, own systemd units, own deploy secret) on LXC 105 (apps), reading `HOMELAB_CONTEXT_DIR=/opt/homelab-context` for -all data. See [infrastructure/auto-deploy.md](../../../infrastructure/auto-deploy.md) +all data. See [infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for the general pattern; webhook ids 10 (homelab-mcp, :9811) and 11 (secrets-issuance, :9821) are the direct precedent — this is a third webhook on `dtoro/Homelab-Docs`, port :9831. diff --git a/oikos/drift.py b/oikos/drift.py index 513ed9d..8e598f9 100644 --- a/oikos/drift.py +++ b/oikos/drift.py @@ -215,7 +215,7 @@ def check_pct_list(inv: dict | None = None) -> list[dict]: def check_caddy_backends(inv: dict | None = None) -> list[dict]: """Caddy's /etc/caddy (a git checkout of dtoro/caddy-conf, per - containers/121-caddy.md) vs inventory service backend IPs. Best-effort + knowledge/wiki/containers/121-caddy.md) vs inventory service backend IPs. Best-effort grep for reverse_proxy targets; skips services whose Caddyfile snippet doesn't use a bare IP (e.g. references a Caddy snippet/import).""" inv = inv or _load_inventory() diff --git a/oikos/gen-topology.py b/oikos/gen-topology.py index 87505d8..ca68d46 100644 --- a/oikos/gen-topology.py +++ b/oikos/gen-topology.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 """ -Generate infrastructure/topology.md (Mermaid views) and per-entity context +Generate knowledge/wiki/infrastructure/topology.md (Mermaid views) and per-entity context cards from inventory.yaml. Views: @@ -42,7 +42,7 @@ from oikos import policy as oikos_policy # noqa: E402 from oikos import relations as oikos_relations # noqa: E402 INVENTORY = REPO / "inventory.yaml" -OUTPUT = REPO / "infrastructure" / "topology.md" +OUTPUT = REPO / "knowledge" / "wiki" / "infrastructure" / "topology.md" CARDS_DIR = REPO / "oikos" / "cards" BANNER = ( @@ -82,15 +82,15 @@ def _host_card(name: str, entry: dict, inv: dict) -> str: lines.append(f"- mounts: {', '.join(entry['mounts'])}") doc = None if entry.get("kind") == "lxc" and pve: - cand = REPO / "containers" / f"{pve}-{name}.md" + cand = REPO / "knowledge" / "wiki" / "containers" / f"{pve}-{name}.md" if cand.exists(): doc = str(cand.relative_to(REPO)) elif entry.get("kind") == "vm" and pve: - cand = REPO / "vms" / f"{pve}-{name}.md" + cand = REPO / "knowledge" / "wiki" / "vms" / f"{pve}-{name}.md" if cand.exists(): doc = str(cand.relative_to(REPO)) elif entry.get("kind") == "proxmox-host": - cand = REPO / "hosts" / f"{name}.md" + cand = REPO / "knowledge" / "wiki" / "hosts" / f"{name}.md" if cand.exists(): doc = str(cand.relative_to(REPO)) if doc: @@ -192,9 +192,9 @@ def render(inv: dict) -> str: parts = [ BANNER, "# Topology (generated)\n", - f"Source: [inventory.yaml](../inventory.yaml) — {counts}.", - "Edge semantics: [oikos/ontology.yaml](../oikos/ontology.yaml). " - "Operating model: [OIKOS.md](../OIKOS.md).\n", + f"Source: [inventory.yaml](../../../inventory.yaml) — {counts}.", + "Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). " + "Operating model: [OIKOS.md](../../../.agents/OIKOS.md).\n", "## Compute & ingress\n", "\n".join(compute_view(inv)) + "\n", "## Storage (mounts)\n", diff --git a/oikos/gen_topology_lib.py b/oikos/gen_topology_lib.py index 84c3e12..13c8548 100644 --- a/oikos/gen_topology_lib.py +++ b/oikos/gen_topology_lib.py @@ -2,7 +2,7 @@ Split out of oikos/gen-topology.py so it's importable (a hyphenated filename can't be `import`ed as a module). oikos/gen-topology.py is the -CLI entrypoint that writes infrastructure/topology.md + oikos/cards/; +CLI entrypoint that writes knowledge/wiki/infrastructure/topology.md + oikos/cards/; oikos/console/app.py imports this module directly to render the live /graph page without shelling out. """ diff --git a/oikos/report.py b/oikos/report.py index ef75a68..5d7e1a9 100644 --- a/oikos/report.py +++ b/oikos/report.py @@ -4,7 +4,7 @@ Both are generated text, not sent directly to Matrix — same integration contract as oikos/approve.py: there's no dedicated Matrix bot in this homelab, so Hermes (already posting alerts as @dtoro:avispero, see -infrastructure/monitoring.md) is the one that actually delivers this text. +knowledge/wiki/infrastructure/monitoring.md) is the one that actually delivers this text. The daily brief is meant to run once a day (e.g. chained after an early oikos-scheduler.service run, or its own systemd timer); the weekly report is a deeper markdown review. diff --git a/oikos/scheduler.py b/oikos/scheduler.py index fe0853e..b1e05ed 100644 --- a/oikos/scheduler.py +++ b/oikos/scheduler.py @@ -75,7 +75,7 @@ _HEALTH_OVERRIDES = { "url_transform": lambda url: url.rstrip("/").rsplit("/", 1)[0] + "/health", }, # Token-gated at the app level (401 without a token is correct, not - # down) — see containers/128-trmnl.md, which documents a dedicated + # down) — see knowledge/wiki/containers/128-trmnl.md, which documents a dedicated # /health endpoint returning 200 unauthenticated. "trmnl": {"url_transform": lambda url: url.rstrip("/") + "/health"}, } diff --git a/operations/agent-enrollment.md b/operations/agent-enrollment.md index a615c00..dfd41a6 100644 --- a/operations/agent-enrollment.md +++ b/operations/agent-enrollment.md @@ -6,7 +6,7 @@ this repo that auto-syncs every 5 min, a per-client age key for SOPS decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config. > Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment? -> See [hermes-agent.md](./hermes-agent.md). It uses the same `bootstrap.sh` +> See [hermes-agent.md](hermes-agent.md). It uses the same `bootstrap.sh` > with an additional `--with-hermes` flag. Architecture in [project_homelab_context_plan](https://… memory link); the @@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight. -Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../containers/124-authentik.md). Only needed once per account. +Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../knowledge/wiki/containers/124-authentik.md). Only needed once per account. **Path A — setup-key (headless/scripted onboarding):** @@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr ### DNS prerequisite `*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124 -([dns.md](../infrastructure/dns.md)) for LAN clients, **but only if the +([dns.md](../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the client uses 192.168.8.180 as its resolver**. Most LXCs and roaming workstations don't by default. Options: @@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually: | `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. | | `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. | | Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). | -| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). | +| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../knowledge/wiki/containers/124-authentik.md). | | `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. | | `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). | | `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. | @@ -356,7 +356,7 @@ and LAN IP registration. New workstations enrolled via this doc will automatically join the universal SSH mesh. ### 2026-05-31 — cross-link to hermes-agent.md -Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](./hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here. +Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here. ### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper. diff --git a/operations/commands.md b/operations/commands.md index d2821d3..bdc1cd7 100644 --- a/operations/commands.md +++ b/operations/commands.md @@ -1,6 +1,6 @@ # Operations cheatsheet -Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`. +Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`. ## Proxmox CLI @@ -8,13 +8,13 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root | --- | --- | | `pct list` / `qm list` | List LXC containers / VMs | | `pct config <id>` / `qm config <id>` | Container / VM config | -| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../infrastructure/media-permissions.md)) | +| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../knowledge/wiki/infrastructure/media-permissions.md)) | | `pct enter <id>` | Shell into a container | | `pct start <id>` / `pct stop <id>` | Boot / halt a container | | `pvesm status` | Storage pools status | | `pvesh get /nodes --output-format json` | Node summary as JSON | | `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status | -| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) | +| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) | | `pveversion` | PVE version | | `journalctl -u pve-cluster -n 100` | PVE service logs | @@ -22,22 +22,22 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root - Shared mount: `/mnt/library` (ext4 on lvmthin `library`). - Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data` -- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../infrastructure/media-permissions.md) for the GID-10000 onboarding recipe. +- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe. ## Reverse proxy -- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md). -- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it. +- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../knowledge/wiki/containers/121-caddy.md). +- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it. - Hot reload: `pct exec 121 -- systemctl reload caddy`. - Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`. - Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`. ## DNS -- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.) +- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.) - Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes. - Verify: `dig @192.168.8.2 +short <host>.hubris.network`. -- See [DNS](../infrastructure/dns.md). +- See [DNS](../knowledge/wiki/infrastructure/dns.md). ## Web access @@ -83,9 +83,9 @@ See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference: Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../oikos/console/deploy/README.md). ## Related -- [Hubris host](../hosts/hubris.md) -- [Containers index](../containers/index.md) -- [DNS](../infrastructure/dns.md) -- [Monitoring](../infrastructure/monitoring.md) -- [Auto-deploy](../infrastructure/auto-deploy.md) +- [Hubris host](../knowledge/wiki/hosts/hubris.md) +- [Containers index](../knowledge/wiki/containers/index.md) +- [DNS](../knowledge/wiki/infrastructure/dns.md) +- [Monitoring](../knowledge/wiki/infrastructure/monitoring.md) +- [Auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md) - [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction diff --git a/operations/hermes-agent.md b/operations/hermes-agent.md index 2f87838..7227f09 100644 --- a/operations/hermes-agent.md +++ b/operations/hermes-agent.md @@ -2,7 +2,7 @@ Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned Llama variant) as a working terminal agent on a homelab client. Builds on top -of standard client enrollment (see [agent-enrollment.md](./agent-enrollment.md)) +of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md)) — this page covers only the Hermes-specific additions. The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides: @@ -21,7 +21,7 @@ The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global | Requirement | How | | --- | --- | -| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](./agent-enrollment.md) | +| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) | | `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below | | The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` | @@ -165,7 +165,7 @@ every tool call, use `approve`. See ## Cross-references -- [agent-enrollment.md](./agent-enrollment.md) — base client onboarding the +- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the Hermes flow assumes is done. - [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every session start (via `~/.config/goose/.goosehints`). diff --git a/plans/2026-06-24-trmnl-plugins-lxc.md b/plans/2026-06-24-trmnl-plugins-lxc.md index c38f9f3..294219d 100644 --- a/plans/2026-06-24-trmnl-plugins-lxc.md +++ b/plans/2026-06-24-trmnl-plugins-lxc.md @@ -15,9 +15,9 @@ wiring, same split as Artifacto/Plato. - No TRMNL middleware in the lab. Highest LXC id is 127 (see `containers/index.md`). - Public hostnames terminate at the [VPS netbird traefik](../hosts/netbird-vps.md) → netbird - mesh → [caddy (121)](../containers/121-caddy.md) → backend LXC. Cert obtained by Caddy + mesh → [caddy (121)](../knowledge/wiki/containers/121-caddy.md) → backend LXC. Cert obtained by Caddy (IONOS DNS-01) and mirrored to the VPS by the daily cert-sync timer on the host. -- Auto-deploy pipelines are gitea-webhook driven, two shapes (see [auto-deploy](../infrastructure/auto-deploy.md)). +- Auto-deploy pipelines are gitea-webhook driven, two shapes (see [auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md)). ## Target state @@ -74,7 +74,7 @@ TRMNL cloud --GET 15m, Bearer token--> https://trmnl.hubris.network/munich-hom `/etc/terminalito-deploy/git-credentials` (mode 600). Register a gitea webhook on `dtoro/terminalito`; add `192.168.8.<128-ip>` to gitea `app.ini` `ALLOWED_HOST_LIST`. -5. **DNS**: add `trmnl.hubris.network` A → `192.168.8.175` (caddy) on [Technitium (107)](../containers/107-dns.md). +5. **DNS**: add `trmnl.hubris.network` A → `192.168.8.175` (caddy) on [Technitium (107)](../knowledge/wiki/containers/107-dns.md). 6. **Caddy** (`dtoro/caddy-conf`, commit+push auto-deploys): ``` diff --git a/runbooks/config-change-deploy.md b/runbooks/config-change-deploy.md index 8767f47..e86d8d6 100644 --- a/runbooks/config-change-deploy.md +++ b/runbooks/config-change-deploy.md @@ -21,7 +21,7 @@ own repo) and get it live, safely. [OIKOS.md](../OIKOS.md) conventions). 3. Make the change, commit, push to `main`. 4. The Gitea webhook fires the deploy pipeline for that repo (see - [infrastructure/auto-deploy.md](../infrastructure/auto-deploy.md) for + [infrastructure/auto-deploy.md](../knowledge/wiki/infrastructure/auto-deploy.md) for the exact receiver/reload for this service). 5. Run the preflight's verification command. If it fails, check `homelab service <name> log` for the reload/restart error. diff --git a/runbooks/runbook-dpkg-interrupted.md b/runbooks/runbook-dpkg-interrupted.md index 3e71f15..c619142 100644 --- a/runbooks/runbook-dpkg-interrupted.md +++ b/runbooks/runbook-dpkg-interrupted.md @@ -103,8 +103,8 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress. ## Related - [Operations cheatsheet](commands.md) -- [Auto-deploy pipelines](../infrastructure/auto-deploy.md) -- [Hubris host page](../hosts/hubris.md) +- [Auto-deploy pipelines](../knowledge/wiki/infrastructure/auto-deploy.md) +- [Hubris host page](../knowledge/wiki/hosts/hubris.md) ## Changelog From 5c5016b3c77ec8ed78e39c4d267a392c9dfa768f Mon Sep 17 00:00:00 2001 From: dtoro <d.toro.v@pm.me> Date: Mon, 6 Jul 2026 14:39:31 +0200 Subject: [PATCH 3/8] docs: reshape runbooks into .agents/skills/<name>/SKILL.md (phase 4) Problem: runbooks are agent-executable procedures but lived at the repo root, separate from the other agent instruction now under .agents/. Change: - Move runbooks/<name>.md -> .agents/skills/<name>/SKILL.md (folder per skill, matching the wiki-hq skills layout). Frontmatter (name, risk_class, inputs, verification, docs_update_checklist, transition) preserved. - Rewrite links (inbound from plans; between-skill siblings) via the move map. - Update prose references in AGENTS.md, HERMES.md, .agents/OIKOS.md, and the operations schema; fix a pre-existing stale link to operations/commands.md. No code consumed runbooks/ by path, so nothing else changes. Verification: all SKILL.md frontmatter parses with valid risk_class; every lifecycle transition resolves to an oikos/ontology.yaml state; broken-link count 127 -> 126 (fixed one, introduced none). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- .agents/OIKOS.md | 2 +- .agents/domains/operations/schema.md | 4 ++-- .../skills/client-enrollment/SKILL.md | 4 ++-- .../skills/config-change-deploy/SKILL.md | 4 ++-- .../skills/incident-investigation/SKILL.md | 0 .../skills/lifecycle-activate-node/SKILL.md | 4 ++-- .../skills/lifecycle-deprecate-node/SKILL.md | 4 ++-- .../skills/lifecycle-destroy-node/SKILL.md | 0 .../skills/lifecycle-migrate-node/SKILL.md | 2 +- .../skills/lifecycle-provision-node/SKILL.md | 4 ++-- .../skills/runbook-budget-from-csv/SKILL.md | 0 .../skills/runbook-dpkg-interrupted/SKILL.md | 6 +++--- .../skills/service-health-check/SKILL.md | 0 AGENTS.md | 3 ++- HERMES.md | 4 ++-- plans/2026-07-05-oikos-prometheus-lxc.md | 4 ++-- 16 files changed, 23 insertions(+), 22 deletions(-) rename runbooks/client-enrollment.md => .agents/skills/client-enrollment/SKILL.md (92%) rename runbooks/config-change-deploy.md => .agents/skills/config-change-deploy/SKILL.md (91%) rename runbooks/incident-investigation.md => .agents/skills/incident-investigation/SKILL.md (100%) rename runbooks/lifecycle-activate-node.md => .agents/skills/lifecycle-activate-node/SKILL.md (90%) rename runbooks/lifecycle-deprecate-node.md => .agents/skills/lifecycle-deprecate-node/SKILL.md (91%) rename runbooks/lifecycle-destroy-node.md => .agents/skills/lifecycle-destroy-node/SKILL.md (100%) rename runbooks/lifecycle-migrate-node.md => .agents/skills/lifecycle-migrate-node/SKILL.md (96%) rename runbooks/lifecycle-provision-node.md => .agents/skills/lifecycle-provision-node/SKILL.md (89%) rename runbooks/runbook-budget-from-csv.md => .agents/skills/runbook-budget-from-csv/SKILL.md (100%) rename runbooks/runbook-dpkg-interrupted.md => .agents/skills/runbook-dpkg-interrupted/SKILL.md (95%) rename runbooks/service-health-check.md => .agents/skills/service-health-check/SKILL.md (100%) diff --git a/.agents/OIKOS.md b/.agents/OIKOS.md index fe1fa17..af30e59 100644 --- a/.agents/OIKOS.md +++ b/.agents/OIKOS.md @@ -40,7 +40,7 @@ one pass through **Observe → Orient → Decide → Act**: |---|---|---| | Host / Service | topology entities | `inventory.yaml` (+ generated `hosts/*.yaml`) | | Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` | -| Runbook | executable workflow with risk class + verification | `runbooks/` (Week 2) | +| Runbook | executable workflow with risk class + verification | `.agents/skills/<name>/SKILL.md` | | Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) | | Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) | | Approval | short-TTL signed grant for a gated action | approval engine (Week 3) | diff --git a/.agents/domains/operations/schema.md b/.agents/domains/operations/schema.md index ba8ca84..aadef2b 100644 --- a/.agents/domains/operations/schema.md +++ b/.agents/domains/operations/schema.md @@ -20,8 +20,8 @@ versioned, reviewable, and reachable by MCP `get_page`/`search_docs` like any ot ## Runbooks -Repeatable procedures live in `runbooks/<name>.md` with YAML front-matter that the Oikos policy and -lifecycle machinery reads: +Repeatable procedures are skills — one folder per skill at `.agents/skills/<name>/SKILL.md`, with +YAML front-matter that the Oikos policy and lifecycle machinery reads: ```yaml --- diff --git a/runbooks/client-enrollment.md b/.agents/skills/client-enrollment/SKILL.md similarity index 92% rename from runbooks/client-enrollment.md rename to .agents/skills/client-enrollment/SKILL.md index 7de5e4a..6939e96 100644 --- a/runbooks/client-enrollment.md +++ b/.agents/skills/client-enrollment/SKILL.md @@ -11,12 +11,12 @@ docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm] Goal: bring a new host (workstation, LXC, VM) into inventory and the secrets model, with mesh membership only where it's actually needed. This wraps the existing `homelab client add` flow — see -[operations/agent-enrollment.md](../operations/agent-enrollment.md) for +[operations/agent-enrollment.md](../../../operations/agent-enrollment.md) for the full walkthrough; this runbook is the risk/lifecycle framing. 1. On any enrolled client: `homelab client add <hostname>` — appends a `hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned` - → `provisioning`, per [oikos/ontology.yaml](../oikos/ontology.yaml)), + → `provisioning`, per [oikos/ontology.yaml](../../../oikos/ontology.yaml)), commits + pushes. 2. Netbird join is **optional, not a required step** — only needed for hosts that must be reachable off-LAN (workstations that roam, e.g. diff --git a/runbooks/config-change-deploy.md b/.agents/skills/config-change-deploy/SKILL.md similarity index 91% rename from runbooks/config-change-deploy.md rename to .agents/skills/config-change-deploy/SKILL.md index e86d8d6..3b4668f 100644 --- a/runbooks/config-change-deploy.md +++ b/.agents/skills/config-change-deploy/SKILL.md @@ -18,10 +18,10 @@ own repo) and get it live, safely. `oikos/policy.yaml`. 2. Clone/pull the `config_repo` (never edit the backend's working tree directly — tracked configs change by commit + push, per - [OIKOS.md](../OIKOS.md) conventions). + [OIKOS.md](../../../OIKOS.md) conventions). 3. Make the change, commit, push to `main`. 4. The Gitea webhook fires the deploy pipeline for that repo (see - [infrastructure/auto-deploy.md](../knowledge/wiki/infrastructure/auto-deploy.md) for + [infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for the exact receiver/reload for this service). 5. Run the preflight's verification command. If it fails, check `homelab service <name> log` for the reload/restart error. diff --git a/runbooks/incident-investigation.md b/.agents/skills/incident-investigation/SKILL.md similarity index 100% rename from runbooks/incident-investigation.md rename to .agents/skills/incident-investigation/SKILL.md diff --git a/runbooks/lifecycle-activate-node.md b/.agents/skills/lifecycle-activate-node/SKILL.md similarity index 90% rename from runbooks/lifecycle-activate-node.md rename to .agents/skills/lifecycle-activate-node/SKILL.md index dd3772b..bc04a0d 100644 --- a/runbooks/lifecycle-activate-node.md +++ b/.agents/skills/lifecycle-activate-node/SKILL.md @@ -9,13 +9,13 @@ transition: "provisioning -> active" # Lifecycle: activate a node -Per [oikos/ontology.yaml](../oikos/ontology.yaml). Requires: age key +Per [oikos/ontology.yaml](../../../oikos/ontology.yaml). Requires: age key enrolled if it needs secrets, mesh joined if it needs off-LAN reach, ingress live if public, health check answering, doc page complete, ledger entry. 1. If the node is a `homelab` client: finish enrollment per - [client-enrollment.md](client-enrollment.md) (`--finalize-pubkey`, + [client-enrollment.md](../client-enrollment/SKILL.md) (`--finalize-pubkey`, mesh join, `homelab doctor` green). 2. If it hosts a public service: add the `services:` entry in `inventory.yaml` (backend, url, doc_page, config_repo, risk_notes — diff --git a/runbooks/lifecycle-deprecate-node.md b/.agents/skills/lifecycle-deprecate-node/SKILL.md similarity index 91% rename from runbooks/lifecycle-deprecate-node.md rename to .agents/skills/lifecycle-deprecate-node/SKILL.md index b667494..af4c15c 100644 --- a/runbooks/lifecycle-deprecate-node.md +++ b/.agents/skills/lifecycle-deprecate-node/SKILL.md @@ -9,7 +9,7 @@ transition: "active -> deprecated" # Lifecycle: deprecate a node -Per [oikos/ontology.yaml](../oikos/ontology.yaml): a node keeps running +Per [oikos/ontology.yaml](../../../oikos/ontology.yaml): a node keeps running but takes no new dependents. **Completion condition: zero remaining inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a suggestion; `oikos/policy.yaml` `lifecycle_overrides.deprecated.refuse` @@ -32,4 +32,4 @@ before proceeding — that's exactly the kind of drift the Week-3 detector will catch automatically, but until then this manual check is the gate. Next (once `affected_by` is empty): -[lifecycle-destroy-node.md](lifecycle-destroy-node.md). +[lifecycle-destroy-node.md](../lifecycle-destroy-node/SKILL.md). diff --git a/runbooks/lifecycle-destroy-node.md b/.agents/skills/lifecycle-destroy-node/SKILL.md similarity index 100% rename from runbooks/lifecycle-destroy-node.md rename to .agents/skills/lifecycle-destroy-node/SKILL.md diff --git a/runbooks/lifecycle-migrate-node.md b/.agents/skills/lifecycle-migrate-node/SKILL.md similarity index 96% rename from runbooks/lifecycle-migrate-node.md rename to .agents/skills/lifecycle-migrate-node/SKILL.md index da1c82a..e8a6e52 100644 --- a/runbooks/lifecycle-migrate-node.md +++ b/.agents/skills/lifecycle-migrate-node/SKILL.md @@ -10,7 +10,7 @@ transition: "active -> migrating -> active" # Lifecycle: migrate a node Modeled on the strong Phase 1+2 migration -([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)). +([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)). Requires (ontology): preflight + backup-verified before migrating; post-verify + Caddy backends checked + mounts checked + docs updated before returning to `active`. diff --git a/runbooks/lifecycle-provision-node.md b/.agents/skills/lifecycle-provision-node/SKILL.md similarity index 89% rename from runbooks/lifecycle-provision-node.md rename to .agents/skills/lifecycle-provision-node/SKILL.md index 1f0fb6b..d12ecdb 100644 --- a/runbooks/lifecycle-provision-node.md +++ b/.agents/skills/lifecycle-provision-node/SKILL.md @@ -9,7 +9,7 @@ transition: "planned -> provisioning" # Lifecycle: provision a node -Per [oikos/ontology.yaml](../oikos/ontology.yaml) `lifecycle.transitions`. +Per [oikos/ontology.yaml](../../../oikos/ontology.yaml) `lifecycle.transitions`. Policy note: `provisioning` nodes get a lifecycle override — `config_mutation` actions downgrade to `reversible_low` because nothing depends on the node yet (see `oikos/policy.yaml` `lifecycle_overrides`). @@ -30,4 +30,4 @@ chosen, doc page stub. is enough to satisfy the transition requirement. 4. Reserve the IP in DNS/DHCP notes if it's a fixed LAN address. -Next: [lifecycle-activate-node.md](lifecycle-activate-node.md). +Next: [lifecycle-activate-node.md](../lifecycle-activate-node/SKILL.md). diff --git a/runbooks/runbook-budget-from-csv.md b/.agents/skills/runbook-budget-from-csv/SKILL.md similarity index 100% rename from runbooks/runbook-budget-from-csv.md rename to .agents/skills/runbook-budget-from-csv/SKILL.md diff --git a/runbooks/runbook-dpkg-interrupted.md b/.agents/skills/runbook-dpkg-interrupted/SKILL.md similarity index 95% rename from runbooks/runbook-dpkg-interrupted.md rename to .agents/skills/runbook-dpkg-interrupted/SKILL.md index c619142..53ab143 100644 --- a/runbooks/runbook-dpkg-interrupted.md +++ b/.agents/skills/runbook-dpkg-interrupted/SKILL.md @@ -102,9 +102,9 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress. ## Related -- [Operations cheatsheet](commands.md) -- [Auto-deploy pipelines](../knowledge/wiki/infrastructure/auto-deploy.md) -- [Hubris host page](../knowledge/wiki/hosts/hubris.md) +- [Operations cheatsheet](../../../operations/commands.md) +- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md) +- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md) ## Changelog diff --git a/runbooks/service-health-check.md b/.agents/skills/service-health-check/SKILL.md similarity index 100% rename from runbooks/service-health-check.md rename to .agents/skills/service-health-check/SKILL.md diff --git a/AGENTS.md b/AGENTS.md index ea72a46..f330689 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,7 +76,8 @@ Grep is fine for browsing or when MCP is unreachable. ## 4. Wiki conventions - Narrative pages live under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`; - procedural pages under `investigations/`, `operations/`, `runbooks/`, `plans/`. + runbook procedures under `.agents/skills/<name>/SKILL.md`; other procedural + pages under `investigations/`, `operations/`, `plans/`. Cross-link liberally; orphans are bugs. - Every page ends with a `## Changelog` section, entries in reverse-chrono order: diff --git a/HERMES.md b/HERMES.md index 03fe761..927dd18 100644 --- a/HERMES.md +++ b/HERMES.md @@ -19,8 +19,8 @@ When in doubt, check `/opt/homelab-context/` first. For the canonical workflows (service health check, config change + deploy, client enrollment, incident investigation, and each node -lifecycle transition), read the matching file in `runbooks/*.md` before -acting. Each runbook carries its risk class, required inputs, the +lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before +acting. Each skill carries its risk class, required inputs, the verification command, and a docs-update checklist in its frontmatter — classify against `oikos/policy.yaml` using that risk class before any mutation. Don't re-derive topology or the mutation path by grepping the diff --git a/plans/2026-07-05-oikos-prometheus-lxc.md b/plans/2026-07-05-oikos-prometheus-lxc.md index b474f12..d2922ef 100644 --- a/plans/2026-07-05-oikos-prometheus-lxc.md +++ b/plans/2026-07-05-oikos-prometheus-lxc.md @@ -42,7 +42,7 @@ is actually unused end-to-end. ## Provisioning steps (once pve_id is assigned) -Follow [runbooks/lifecycle-provision-node.md](../runbooks/lifecycle-provision-node.md): +Follow [lifecycle-provision-node](../.agents/skills/lifecycle-provision-node/SKILL.md): 1. `pct create <new-id> ...` on hubris — confirm the assigned ID doesn't collide with 131 or anything else live. @@ -53,7 +53,7 @@ Follow [runbooks/lifecycle-provision-node.md](../runbooks/lifecycle-provision-no 4. Point node_exporter at hubris + strong (either install locally on each, or scrape via SSH-tunneled metrics — install locally is simpler and is the standard approach). -5. Follow [runbooks/lifecycle-activate-node.md](../runbooks/lifecycle-activate-node.md) +5. Follow [lifecycle-activate-node](../.agents/skills/lifecycle-activate-node/SKILL.md) to flip to `active`, complete the doc page, regenerate `hosts/*.yaml` + `infrastructure/topology.md`. 6. Extend `oikos/scheduler.py`'s disk/temp probes to query Prometheus From 1019a1cb5202d420c69375cce99260ad0d49dfa3 Mon Sep 17 00:00:00 2001 From: dtoro <d.toro.v@pm.me> Date: Mon, 6 Jul 2026 14:42:23 +0200 Subject: [PATCH 4/8] docs: section-index pass + docs-lint skill (phase 5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add knowledge/wiki/hosts/index.md (the one missing section index) and point knowledge/index.md at it. - Add .agents/skills/docs-lint/ (SKILL.md + lint.py) enforcing the mechanical parts of writing-style.md: banned vocabulary and broken relative links. The style guide and this skill are exempt from the banned-word check since they enumerate the list. - Record the restructure + lint in knowledge/log.md. Verification: banned-vocabulary scan of knowledge/ is clean (the few remaining repo-wide hits are false positives — the literal '_' character — or historical append-only plans quoting the vocabulary, which the standard does not restyle). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- .agents/skills/docs-lint/SKILL.md | 25 +++++++++++ .agents/skills/docs-lint/lint.py | 69 +++++++++++++++++++++++++++++++ knowledge/index.md | 2 +- knowledge/log.md | 1 + knowledge/wiki/hosts/index.md | 9 ++++ 5 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 .agents/skills/docs-lint/SKILL.md create mode 100644 .agents/skills/docs-lint/lint.py create mode 100644 knowledge/wiki/hosts/index.md diff --git a/.agents/skills/docs-lint/SKILL.md b/.agents/skills/docs-lint/SKILL.md new file mode 100644 index 0000000..e6a8a9a --- /dev/null +++ b/.agents/skills/docs-lint/SKILL.md @@ -0,0 +1,25 @@ +--- +name: docs-lint +risk_class: read_only +inputs: [paths] +verification: "python3 .agents/skills/docs-lint/lint.py" +docs_update_checklist: [] +--- + +# Docs lint + +Check committed documentation against the mechanical rules in +[writing-style.md](../../shared/writing-style.md): banned vocabulary and broken relative markdown +links. Prose-voice rules are not machine-checkable — those stay a review responsibility. + +Run from the repo root: + + python3 .agents/skills/docs-lint/lint.py # default: knowledge/ .agents/ operations/ investigations/ plans/ + python3 .agents/skills/docs-lint/lint.py knowledge/wiki/containers/104-gitea.md + +Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary +list mirrors `writing-style.md`; update both together if the standard changes. + +> **Known baseline.** The lab carries pre-existing broken links to destroyed/archived nodes (e.g. +> `124-authentik.md`, now `106-auth-outpost`). Clean those opportunistically; do not treat the +> current count as a regression from this skill. diff --git a/.agents/skills/docs-lint/lint.py b/.agents/skills/docs-lint/lint.py new file mode 100644 index 0000000..dc1261d --- /dev/null +++ b/.agents/skills/docs-lint/lint.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +"""Lint committed docs against .agents/shared/writing-style.md. + +Checks two mechanical rules: + 1. Banned vocabulary (significance puffers, analytical verbs, poetic nouns, + promotional adjectives, opening crutches). + 2. Broken relative markdown links. + +Prose-voice rules are not machine-checkable; this covers the parts that are. +Run from the repo root: python3 .agents/skills/docs-lint/lint.py [paths...] +Exit 1 if any violation is found. +""" +import os, re, sys + +BANNED = [ + "pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament", + "paramount", "invaluable", "delve", "leverage", "utilize", "facilitate", "foster", + "showcase", "underscore", "streamline", "harness", "tapestry", "realm", "paradigm", + "nexus", "cornerstone", "robust", "seamless", "innovative", "cutting-edge", + "meticulous", "holistic", "comprehensive", "in today's world", + "it's worth noting", "it is important to note", +] +BAN_RE = re.compile(r'(?<![\w-])(' + "|".join(re.escape(w) for w in BANNED) + r')(?![\w-])', re.I) +LINK = re.compile(r'\]\(([^)]+)\)') + +def iter_md(paths): + for p in paths: + if os.path.isfile(p) and p.endswith(".md"): + yield p + for root, dirs, files in os.walk(p): + dirs[:] = [d for d in dirs if d not in (".git", "node_modules")] + for f in files: + if f.endswith(".md"): + yield os.path.join(root, f) + +def main(argv): + paths = argv or ["knowledge", ".agents", "operations", "investigations", "plans"] + violations = 0 + # The style guide and this skill enumerate the banned words by definition. + ban_exempt = ("shared/writing-style.md", "skills/docs-lint/") + for f in sorted(set(iter_md(paths))): + check_banned = not any(x in f for x in ban_exempt) + fence = False + with open(f) as fh: + for ln, line in enumerate(fh, 1): + if line.lstrip().startswith("```"): + fence = not fence; continue + if fence: + continue + if check_banned: + for m in BAN_RE.finditer(line): + print(f"{f}:{ln}: banned word '{m.group(1)}'") + violations += 1 + for m in LINK.finditer(line): + link = m.group(1) + if re.match(r'^(https?:|mailto:|#|/)', link): + continue + path = re.split(r'[#?]', link)[0] + if not path: + continue + tgt = os.path.normpath(os.path.join(os.path.dirname(f), path)) + if not os.path.exists(tgt): + print(f"{f}:{ln}: broken link -> {link}") + violations += 1 + print(f"\n{violations} violation(s)") + return 1 if violations else 0 + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/knowledge/index.md b/knowledge/index.md index 0067f7f..8252ad3 100644 --- a/knowledge/index.md +++ b/knowledge/index.md @@ -6,7 +6,7 @@ cross-cutting system, synthesized from live state and evidence. Structure and ru | Section | What it covers | |---------|----------------| -| [wiki/hosts/](wiki/hosts/) | Proxmox host narratives — `hubris`, `strong`. | +| [wiki/hosts/](wiki/hosts/index.md) | Proxmox host narratives — `hubris`, `strong`. | | [wiki/containers/](wiki/containers/index.md) | LXC fleet — one page per container, plus the master table and archaeology. | | [wiki/vms/](wiki/vms/index.md) | Virtual machines — ZimaOS, Home Assistant OS. | | [wiki/infrastructure/](wiki/infrastructure/index.md) | Cross-cutting systems — DNS, ingress, mesh, storage, auth, monitoring, generated topology. | diff --git a/knowledge/log.md b/knowledge/log.md index 4cb195a..508908b 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -5,3 +5,4 @@ source ingests, lint sweeps). One line per operation, newest last. Infrastructur each page's `## Changelog` and the Oikos change ledger, not here. ## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output. +## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md. diff --git a/knowledge/wiki/hosts/index.md b/knowledge/wiki/hosts/index.md new file mode 100644 index 0000000..1c7ab00 --- /dev/null +++ b/knowledge/wiki/hosts/index.md @@ -0,0 +1,9 @@ +# Hosts + +The two Proxmox VE nodes that run the fleet. Narrative pages; the machine-readable host records are +the generated `hosts/*.yaml` files at the repo root. + +| Document | What it covers | +|----------|----------------| +| [hubris.md](hubris.md) | Primary PVE node (GMKtec NucBox M6 Ultra, `192.168.8.77`) — runs most LXCs plus the VMs. | +| [strong.md](strong.md) | Secondary PVE node / cluster member (`192.168.178.181`) — hosts the LXCs migrated from hubris. | From 4efddb8bedf39df54e8abc0866ff7be2e683380a Mon Sep 17 00:00:00 2001 From: dtoro <d.toro.v@pm.me> Date: Mon, 6 Jul 2026 17:53:35 +0200 Subject: [PATCH 5/8] docs: fix pre-existing broken links surfaced by docs-lint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem: docs-lint (added in the wiki-hq reorg) surfaced 126 broken relative links that predated this session — a container rename, incident/plan docs that moved into archive/done subfolders without their inbound links being updated, and a handful of relative-depth bugs in files nested under containers/archive/ and plans/done/. Fixes applied, by category: - 124-authentik.md -> 106-auth-outpost.md (container was renamed; ~40 refs). - investigations/{2026-04-21-hubris-crash-loop,2026-05-31-authentik-vps-migration}.md -> archive/ prefix (both moved to investigations/archive/ previously). - plans/{2026-06-01-slate-ax-to-sodola-migration,2026-06-04_130000-deprecate-claudio-bot, 2026-06-25-yuvomi-deployment}.md -> plans/done/ prefix. - Depth bugs in files nested one level deeper than their siblings assumed (investigations/archive/*, knowledge/wiki/containers/archive/*, plans/done/*) — corrected relative-path depth. - Destroyed containers with no surviving page (126-plato) delinked to the containers/index.md archaeology row instead of a 404. - ludo-mini.yaml -> strong.yaml (host was renamed, same physical machine). - netbird-vps.md (no narrative page exists) -> netbird-vps.yaml (substrate record, matching the existing convention for hosts without a wiki page). - runbook-dpkg-interrupted.md refs -> .agents/skills/runbook-dpkg-interrupted/SKILL.md (missed in the phase-4 runbook move because the referencing files used a bare filename, not a runbooks/ prefix). - One dangling forward-reference to a never-written investigation delinked to the actual incident record it was describing. Left alone: two links in knowledge/wiki/containers/101-jellyfin.md into devops/homelab-authentik-admin/ — an intentional reference to a sibling repo, not present in this checkout. Verification: broken-link count 126 -> 2 (real remainder is the cross-repo reference above); gen-topology.py --check still exit 0; build_host_files.py still idempotent; all inventory.yaml doc_page targets still resolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- .agents/skills/docs-lint/SKILL.md | 6 +++--- .../skills/lifecycle-migrate-node/SKILL.md | 2 +- ...-03_223218-dhcp-pool-exclude-static-ips.md | 2 +- ...26-06-03-moonlight-sunshine-wifi-jitter.md | 2 +- .../2026-06-06-authentik-session-lifetime.md | 2 +- .../2026-06-06-caddyfile-truncation.md | 2 +- .../archive/2026-04-21-hubris-crash-loop.md | 14 ++++++------- .../2026-05-31-authentik-vps-migration.md | 20 +++++++++---------- knowledge/log.md | 1 + knowledge/wiki/containers/103-paperless.md | 4 ++-- knowledge/wiki/containers/104-gitea.md | 2 +- knowledge/wiki/containers/105-apps.md | 4 ++-- knowledge/wiki/containers/106-auth-outpost.md | 12 +++++------ knowledge/wiki/containers/107-dns.md | 8 ++++---- knowledge/wiki/containers/114-nextcloud.md | 4 ++-- .../wiki/containers/118-elementsynapse.md | 2 +- knowledge/wiki/containers/120-mule-images.md | 2 +- knowledge/wiki/containers/121-caddy.md | 6 +++--- knowledge/wiki/containers/122-arriman.md | 2 +- knowledge/wiki/containers/129-house.md | 2 +- knowledge/wiki/containers/130-grimmory.md | 2 +- .../containers/archive/123-claudio-bot.md | 20 +++++++++---------- .../containers/archive/127-mule-photos-new.md | 12 +++++------ knowledge/wiki/containers/index.md | 6 +++--- knowledge/wiki/hosts/hubris.md | 14 ++++++------- knowledge/wiki/infrastructure/auto-deploy.md | 10 +++++----- knowledge/wiki/infrastructure/backups.md | 12 +++++------ knowledge/wiki/infrastructure/dns.md | 8 ++++---- knowledge/wiki/infrastructure/index.md | 2 +- knowledge/wiki/infrastructure/ingress.md | 4 ++-- knowledge/wiki/infrastructure/mesh.md | 8 ++++---- knowledge/wiki/infrastructure/network.md | 6 +++--- knowledge/wiki/vms/100-zimaos.md | 6 +++--- knowledge/wiki/vms/108-haos.md | 8 ++++---- operations/agent-enrollment.md | 6 +++--- operations/commands.md | 4 ++-- plans/2026-06-24-trmnl-plugins-lxc.md | 4 ++-- ...2026-06-01-slate-ax-to-sodola-migration.md | 12 +++++------ plans/done/2026-06-29-grimmory-migration.md | 2 +- 39 files changed, 123 insertions(+), 122 deletions(-) diff --git a/.agents/skills/docs-lint/SKILL.md b/.agents/skills/docs-lint/SKILL.md index e6a8a9a..b1f296c 100644 --- a/.agents/skills/docs-lint/SKILL.md +++ b/.agents/skills/docs-lint/SKILL.md @@ -20,6 +20,6 @@ Run from the repo root: Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary list mirrors `writing-style.md`; update both together if the standard changes. -> **Known baseline.** The lab carries pre-existing broken links to destroyed/archived nodes (e.g. -> `124-authentik.md`, now `106-auth-outpost`). Clean those opportunistically; do not treat the -> current count as a regression from this skill. +> **Known baseline.** `knowledge/wiki/containers/101-jellyfin.md` links into a sibling repo +> (`devops/homelab-authentik-admin`) that this checkout does not contain — expected, not a bug. +> Any other broken link is a real regression; investigate before dismissing it as baseline noise. diff --git a/.agents/skills/lifecycle-migrate-node/SKILL.md b/.agents/skills/lifecycle-migrate-node/SKILL.md index e8a6e52..b4a29e6 100644 --- a/.agents/skills/lifecycle-migrate-node/SKILL.md +++ b/.agents/skills/lifecycle-migrate-node/SKILL.md @@ -10,7 +10,7 @@ transition: "active -> migrating -> active" # Lifecycle: migrate a node Modeled on the strong Phase 1+2 migration -([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)). +([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)). Requires (ontology): preflight + backup-verified before migrating; post-verify + Caddy backends checked + mounts checked + docs updated before returning to `active`. diff --git a/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md b/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md index 3d0d9b6..c6def9c 100644 --- a/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md +++ b/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md @@ -12,7 +12,7 @@ ## Problem statement -The Technitium DHCP server on [CT 107](containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:** +The Technitium DHCP server on [CT 107](../../knowledge/wiki/containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:** | Host | IP | Inside pool? | |---|---|---| diff --git a/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md b/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md index f5a6a06..256f81d 100644 --- a/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md +++ b/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md @@ -2,7 +2,7 @@ ## Summary -[`ludo-mini`](../hosts/ludo-mini.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming. +[`ludo-mini`](../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming. ## Timeline diff --git a/investigations/2026-06-06-authentik-session-lifetime.md b/investigations/2026-06-06-authentik-session-lifetime.md index faf9613..76adcad 100644 --- a/investigations/2026-06-06-authentik-session-lifetime.md +++ b/investigations/2026-06-06-authentik-session-lifetime.md @@ -91,7 +91,7 @@ print("session_duration:", stage.session_duration) # → "days=30" ## Related - [Container 106 — auth-outpost](../knowledge/wiki/containers/106-auth-outpost.md) -- [Authentik VPS migration](2026-05-31-authentik-vps-migration.md) +- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md) - [Ingress (VPS Traefik)](../knowledge/wiki/infrastructure/ingress.md) - `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan diff --git a/investigations/2026-06-06-caddyfile-truncation.md b/investigations/2026-06-06-caddyfile-truncation.md index cf3203a..516d87f 100644 --- a/investigations/2026-06-06-caddyfile-truncation.md +++ b/investigations/2026-06-06-caddyfile-truncation.md @@ -54,7 +54,7 @@ This is the same class of drift as the June 5th incidents (paperless, HAOS, apps ## Related -- [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md) +- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md) - [Caddy (121)](../knowledge/wiki/containers/121-caddy.md) - [elementsynapse (118)](../knowledge/wiki/containers/118-elementsynapse.md) - [dns-sync script](../scripts/dns-sync.py) diff --git a/investigations/archive/2026-04-21-hubris-crash-loop.md b/investigations/archive/2026-04-21-hubris-crash-loop.md index 7fbcbf3..4466649 100644 --- a/investigations/archive/2026-04-21-hubris-crash-loop.md +++ b/investigations/archive/2026-04-21-hubris-crash-loop.md @@ -2,12 +2,12 @@ ## Summary -[`hubris`](../hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders. +[`hubris`](../../knowledge/wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders. ## Timeline ### 2026-04-19 — drive attached -External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../infrastructure/backups.md). Pre-attach uptime had been 33 days stable. +External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../knowledge/wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable. ### 2026-04-19 → 2026-04-21 — first crashes Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. @@ -24,13 +24,13 @@ Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly - **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running. ### 2026-04-22 — recurrence after 30 h 37 m -Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty. +Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../knowledge/wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty. ### 2026-04-22 — `cpu-epp.service` design bug fixed Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`. ### 2026-04-22 — drive removed (A/B test) -User physically removed the external USB drive. [Backup timers disabled](../infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause. +User physically removed the external USB drive. [Backup timers disabled](../../knowledge/wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause. ### 2026-04-23 — SSD cooling + thermal pads installed Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically. @@ -78,9 +78,9 @@ Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the Nu | `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive | ## Affected nodes -- [Hubris host](../hosts/hubris.md) -- [Backups (disabled)](../infrastructure/backups.md) -- [Monitoring](../infrastructure/monitoring.md) +- [Hubris host](../../knowledge/wiki/hosts/hubris.md) +- [Backups (disabled)](../../knowledge/wiki/infrastructure/backups.md) +- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md) ## Open questions - Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.) diff --git a/investigations/archive/2026-05-31-authentik-vps-migration.md b/investigations/archive/2026-05-31-authentik-vps-migration.md index 972e98d..ba91978 100644 --- a/investigations/archive/2026-05-31-authentik-vps-migration.md +++ b/investigations/archive/2026-05-31-authentik-vps-migration.md @@ -2,9 +2,9 @@ ## Summary -The NetBird management server (on the [VPS](../infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.** +The NetBird management server (on the [VPS](../../knowledge/wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.** -Resolved by moving Authentik off [LXC 124](../containers/124-authentik.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur. +Resolved by moving Authentik off [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur. The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration. @@ -43,7 +43,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS 1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`. 2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug. -3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../containers/124-authentik.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults. +3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../knowledge/wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults. 4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue. 5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA. @@ -51,7 +51,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS | | Before | After | |---|---|---| -| Authentik host | [LXC 124](../containers/124-authentik.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` | +| Authentik host | [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` | | Version | `2026.2.2` | `2026.5.2` | | `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) | | `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container | @@ -76,7 +76,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted). -Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy). +Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../knowledge/wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy). ### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy @@ -99,11 +99,11 @@ The policy was orphaned (no matched type data or had incompatible evaluation). R - **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`. - **VPS port 22** opened for this repair; close once remote access is otherwise stable. - **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service). -- **Reconcile [124-authentik](../containers/124-authentik.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page. +- **Reconcile [124-authentik](../../knowledge/wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page. - **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree. ## Related -- [124 — authentik](../containers/124-authentik.md) -- [DNS split-horizon](../infrastructure/dns.md) -- [Public ingress (VPS traefik)](../infrastructure/ingress.md) -- [Mesh migration](../infrastructure/mesh.md) +- [124 — authentik](../../knowledge/wiki/containers/106-auth-outpost.md) +- [DNS split-horizon](../../knowledge/wiki/infrastructure/dns.md) +- [Public ingress (VPS traefik)](../../knowledge/wiki/infrastructure/ingress.md) +- [Mesh migration](../../knowledge/wiki/infrastructure/mesh.md) diff --git a/knowledge/log.md b/knowledge/log.md index 508908b..c52f3d4 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -6,3 +6,4 @@ each page's `## Changelog` and the Oikos change ledger, not here. ## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output. ## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md. +## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference. diff --git a/knowledge/wiki/containers/103-paperless.md b/knowledge/wiki/containers/103-paperless.md index 20b6702..4f7c127 100644 --- a/knowledge/wiki/containers/103-paperless.md +++ b/knowledge/wiki/containers/103-paperless.md @@ -18,7 +18,7 @@ Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/d | paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers | ## Auth -Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login. +Behind [Authentik forward-auth](106-auth-outpost.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login. ## Storage - Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design). @@ -27,7 +27,7 @@ Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses fo - ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15. ## Related -- [Authentik](124-authentik.md) +- [Authentik](106-auth-outpost.md) - [Caddy](121-caddy.md) - [DNS](../infrastructure/dns.md) - [Monitoring](../infrastructure/monitoring.md) diff --git a/knowledge/wiki/containers/104-gitea.md b/knowledge/wiki/containers/104-gitea.md index 95e40c9..2415d8e 100644 --- a/knowledge/wiki/containers/104-gitea.md +++ b/knowledge/wiki/containers/104-gitea.md @@ -55,7 +55,7 @@ Initial documentation. Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md). ### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added -For OIDC integration with [authentik (124)](124-authentik.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency. +For OIDC integration with [authentik (124)](106-auth-outpost.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency. ### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped `dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`. diff --git a/knowledge/wiki/containers/105-apps.md b/knowledge/wiki/containers/105-apps.md index 3cb2130..f67ed94 100644 --- a/knowledge/wiki/containers/105-apps.md +++ b/knowledge/wiki/containers/105-apps.md @@ -100,7 +100,7 @@ Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.h ## Related - [Gitea (104)](104-gitea.md) — uses the PlantUML server - [Caddy (121)](121-caddy.md) -- [Authentik (124)](124-authentik.md) +- [Authentik (124)](106-auth-outpost.md) - [DNS](../infrastructure/dns.md) - [Auto-deploy](../infrastructure/auto-deploy.md) - [Public ingress (Artifacto + blog)](../infrastructure/ingress.md) @@ -116,7 +116,7 @@ Two new services from the [homelab-context distribution plan](../infrastructure/ `secrets-issuance.service` on `:9820` (per-client age-key provisioning). Caddy fronts both with Let's Encrypt; new vhosts on [caddy](121-caddy.md), split-horizon DNS entries on -[authentik (124)](124-authentik.md). Gitea webhook ids 10 + 11 wire +[authentik (124)](106-auth-outpost.md). Gitea webhook ids 10 + 11 wire auto-deploy. LXC is itself an enrolled context client (`/opt/homelab-context/`). diff --git a/knowledge/wiki/containers/106-auth-outpost.md b/knowledge/wiki/containers/106-auth-outpost.md index 19c16b1..27ae131 100644 --- a/knowledge/wiki/containers/106-auth-outpost.md +++ b/knowledge/wiki/containers/106-auth-outpost.md @@ -1,6 +1,6 @@ # 106 — `auth-outpost` -Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik. +Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../../investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik. ## At a glance - **Hostname:** `auth-outpost` @@ -8,11 +8,11 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co - **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`) - **Resources:** 1 core / 512 MiB / 4 GiB rootfs - **Mounts:** none -- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](124-authentik.md) +- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](106-auth-outpost.md) ## Role -Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../../investigations/2026-05-31-authentik-vps-migration.md). +Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md). ## Service / port map | Service | Listen | Notes | @@ -42,10 +42,10 @@ Fix: the LAN outpost gets its **own** domain. **Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`. ## Related -- [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only) +- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only) - [Caddy (121)](121-caddy.md) — forward-auth consumer - [Ingress (VPS traefik)](../infrastructure/ingress.md) -- [Authentik VPS migration](../../../investigations/2026-05-31-authentik-vps-migration.md) +- [Authentik VPS migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md) ## Changelog @@ -53,4 +53,4 @@ Fix: the LAN outpost gets its **own** domain. VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../../investigations/2026-06-06-authentik-session-lifetime.md). ### 2026-06-01 — created; forward-auth cut over from LXC 124 -New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md). +New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md). diff --git a/knowledge/wiki/containers/107-dns.md b/knowledge/wiki/containers/107-dns.md index 74a99a6..02d09a3 100644 --- a/knowledge/wiki/containers/107-dns.md +++ b/knowledge/wiki/containers/107-dns.md @@ -1,6 +1,6 @@ # 107 — `dns` -Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](124-authentik.md); single-purpose, one job. +Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](106-auth-outpost.md); single-purpose, one job. ## At a glance - **Hostname:** `dns` @@ -42,7 +42,7 @@ Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02): Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101–.239`) are excluded from the pool. Pool narrowed from `.100–.240` to `.241–.254` on 2026-06-03 to eliminate IP conflict risk. ## Related -- [124 — authentik](124-authentik.md) — retired host of the old dnsmasq +- [124 — authentik](106-auth-outpost.md) — retired host of the old dnsmasq - [DNS split-horizon](../infrastructure/dns.md) - [Mesh](../infrastructure/mesh.md) @@ -55,7 +55,7 @@ Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)) Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`. ### 2026-06-03 — DHCP pool narrowed to `.241–.254` -Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../../../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md). ### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone) This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md). @@ -64,4 +64,4 @@ This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100–.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium. ### 2026-06-01 — created; replaced dnsmasq on 124 -Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](124-authentik.md). +Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](106-auth-outpost.md). diff --git a/knowledge/wiki/containers/114-nextcloud.md b/knowledge/wiki/containers/114-nextcloud.md index 3792fc3..2150431 100644 --- a/knowledge/wiki/containers/114-nextcloud.md +++ b/knowledge/wiki/containers/114-nextcloud.md @@ -11,7 +11,7 @@ Personal cloud / file collaboration. Source-of-truth for the photo libraries sur - **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) ## Auth -Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](124-authentik.md#per-app-username-override-pattern-authentik) for the full pattern. +Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](106-auth-outpost.md#per-app-username-override-pattern-authentik) for the full pattern. Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on). @@ -74,7 +74,7 @@ Apply with `systemctl restart mariadb` (not reload — `innodb_log_file_size` ne ## Related - [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV -- [Authentik (124)](124-authentik.md) +- [Authentik (124)](106-auth-outpost.md) - [Caddy (121)](121-caddy.md) - [DNS](../infrastructure/dns.md) - [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md) diff --git a/knowledge/wiki/containers/118-elementsynapse.md b/knowledge/wiki/containers/118-elementsynapse.md index e552a08..17e4334 100644 --- a/knowledge/wiki/containers/118-elementsynapse.md +++ b/knowledge/wiki/containers/118-elementsynapse.md @@ -37,7 +37,7 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name - ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15. ## Related -- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent +- ~~[claudio-bot (123)](archive/123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent - [Caddy](121-caddy.md) - [DNS](../infrastructure/dns.md) - [Monitoring](../infrastructure/monitoring.md) diff --git a/knowledge/wiki/containers/120-mule-images.md b/knowledge/wiki/containers/120-mule-images.md index 69de592..893570c 100644 --- a/knowledge/wiki/containers/120-mule-images.md +++ b/knowledge/wiki/containers/120-mule-images.md @@ -60,7 +60,7 @@ For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials ## Related - [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries -- [Authentik (124)](124-authentik.md) +- [Authentik (124)](106-auth-outpost.md) - [Caddy (121)](121-caddy.md) - [DNS](../infrastructure/dns.md) - [Auto-deploy](../infrastructure/auto-deploy.md) diff --git a/knowledge/wiki/containers/121-caddy.md b/knowledge/wiki/containers/121-caddy.md index a607a60..6f0d335 100644 --- a/knowledge/wiki/containers/121-caddy.md +++ b/knowledge/wiki/containers/121-caddy.md @@ -37,9 +37,9 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L ## Snippet: `(authentik)` forward-auth -A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup). +A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](106-auth-outpost.md#forward-auth-domain-level-setup). -For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md). +For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](106-auth-outpost.md). ## Caddy environment @@ -60,7 +60,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip ## Related - [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain -- [Authentik (124)](124-authentik.md) — forward-auth + IdP +- [Authentik (124)](106-auth-outpost.md) — forward-auth + IdP - [Auto-deploy](../infrastructure/auto-deploy.md) - [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure - [Gitea (104)](104-gitea.md) — webhook source diff --git a/knowledge/wiki/containers/122-arriman.md b/knowledge/wiki/containers/122-arriman.md index 1e5a0d4..0566b42 100644 --- a/knowledge/wiki/containers/122-arriman.md +++ b/knowledge/wiki/containers/122-arriman.md @@ -115,7 +115,7 @@ Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has ## Related - [Caddy (121)](121-caddy.md) -- [Authentik (124)](124-authentik.md) — forward-auth wiring + per-app `/api/*` bypass +- [Authentik (124)](106-auth-outpost.md) — forward-auth wiring + per-app `/api/*` bypass - [DNS](../infrastructure/dns.md) - [Media permissions](../infrastructure/media-permissions.md) - [Hubris host](../hosts/hubris.md) diff --git a/knowledge/wiki/containers/129-house.md b/knowledge/wiki/containers/129-house.md index 30f3b44..e2d112a 100644 --- a/knowledge/wiki/containers/129-house.md +++ b/knowledge/wiki/containers/129-house.md @@ -40,7 +40,7 @@ Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modul - [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy) - [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`) - [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source -- [Deployment plan](../../../plans/2026-06-25-yuvomi-deployment.md) +- [Deployment plan](../../../plans/done/2026-06-25-yuvomi-deployment.md) ## Changelog diff --git a/knowledge/wiki/containers/130-grimmory.md b/knowledge/wiki/containers/130-grimmory.md index 8800200..e9e7267 100644 --- a/knowledge/wiki/containers/130-grimmory.md +++ b/knowledge/wiki/containers/130-grimmory.md @@ -55,7 +55,7 @@ LXC is privileged → in-container UID = host UID. Docker container gets media G - [apps (105)](105-apps.md) — previous host (Booklore) - [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.247:6060` -- [Authentik (124)](124-authentik.md) — OIDC provider `Grimmory` +- [Authentik (124)](106-auth-outpost.md) — OIDC provider `Grimmory` - [DNS (107)](107-dns.md) — `books.hubris.network → 192.168.8.175` (unchanged from Booklore) - [Media permissions](../infrastructure/media-permissions.md) diff --git a/knowledge/wiki/containers/archive/123-claudio-bot.md b/knowledge/wiki/containers/archive/123-claudio-bot.md index daf53bc..384bff3 100644 --- a/knowledge/wiki/containers/archive/123-claudio-bot.md +++ b/knowledge/wiki/containers/archive/123-claudio-bot.md @@ -3,7 +3,7 @@ > **This LXC was destroyed on 2026-06-04.** Replaced by Hermes Agent on mac-mini. > Monitoring migrated to `homelab-hardware-health` skill + 15-min Hermes cronjob. > Repos `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived (read-only) on Gitea. -> See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) for full details. +> See [deprecation plan](../../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) for full details. Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications. @@ -17,7 +17,7 @@ Matrix-resident control plane. Bot account `@claudio:avispero` joined to a priva ## Stack -Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`). +Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](../118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`). Room: `!dEUVJArVKPorxHHJZK:avispero` (invite-only, `@dtoro:avispero` allowed). @@ -43,8 +43,8 @@ Currently set to `lmstudio` → `google/gemma-4-e4b` on the Mac mini at `192.168 ## IPC `http://192.168.8.230:9090/{notify,propose,status}`. Header `X-Bot-Token` must match `/etc/claudio-bot/ipc.token`. Used by: -- [claudio-monitor on hubris](../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`) -- The (currently disabled) [restic backup wrapper](../infrastructure/backups.md) (token in `/etc/restic/bot.token`) +- [claudio-monitor on hubris](../../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`) +- The (currently disabled) [restic backup wrapper](../../infrastructure/backups.md) (token in `/etc/restic/bot.token`) > Token files at the source side **must hold the same value as `/etc/claudio-bot/ipc.token`** — rotate together. @@ -61,13 +61,13 @@ Active plugins: Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/deploy` → pull + `pip install` + restart. Same shape as caddy-conf. -`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.230`. +`app.ini` `ALLOWED_HOST_LIST` on [gitea](../104-gitea.md) includes `192.168.8.230`. ## Related -- [elementsynapse (118)](118-elementsynapse.md) -- [Monitoring (claudio-monitor)](../infrastructure/monitoring.md) -- [Backups (disabled)](../infrastructure/backups.md) -- [Auto-deploy](../infrastructure/auto-deploy.md) +- [elementsynapse (118)](../118-elementsynapse.md) +- [Monitoring (claudio-monitor)](../../infrastructure/monitoring.md) +- [Backups (disabled)](../../infrastructure/backups.md) +- [Auto-deploy](../../infrastructure/auto-deploy.md) ## Changelog @@ -84,7 +84,7 @@ Initial documentation. `backend: lmstudio` → `google/gemma-4-e4b` on the Mac mini. Anthropic key still present so the swap is reversible by flipping the config field. ### 2026-04-21 — `monitor` plugin added -Receives events from [claudio-monitor](../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`. +Receives events from [claudio-monitor](../../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`. ### 2026-04-20 — claudio-bot deployed LXC 123 provisioned. Repo, systemd unit, Matrix wiring, `system` + `backup` plugins, IPC server. diff --git a/knowledge/wiki/containers/archive/127-mule-photos-new.md b/knowledge/wiki/containers/archive/127-mule-photos-new.md index bbf1574..0966251 100644 --- a/knowledge/wiki/containers/archive/127-mule-photos-new.md +++ b/knowledge/wiki/containers/archive/127-mule-photos-new.md @@ -1,7 +1,7 @@ # 127 — `mule-photos-new` Side-by-side **PhotoPrism M0 test** of the `dtoro/mule-image` `new` branch -at `photos-new.hubris.network`. Production [LXC 120](120-mule-images.md) keeps +at `photos-new.hubris.network`. Production [LXC 120](../120-mule-images.md) keeps running on the legacy stack at `photos.hubris.network` until M5 cutover. ## At a glance @@ -11,7 +11,7 @@ running on the legacy stack at `photos.hubris.network` until M5 cutover. - **Resources:** 6 cores / 8 GiB RAM / 40 GiB rootfs / 1 GiB swap - **Features:** `nesting=1,fuse=1,keyctl=1` - **Mounts:** *(none — see scratch copy below)* -- **Public hostname:** [`photos-new.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → split (PhotoPrism `:2342`, sidecar `:8000`, Vite `:5173`) +- **Public hostname:** [`photos-new.hubris.network`](../../infrastructure/dns.md) → [caddy (121)](../121-caddy.md) → split (PhotoPrism `:2342`, sidecar `:8000`, Vite `:5173`) ## Stack (`/opt/mule-image`) @@ -65,7 +65,7 @@ unprivileged LXCs can't see through. ## Auth — Authentik OIDC -PhotoPrism's "Sign in with OIDC" button delegates to [Authentik (124)](124-authentik.md). +PhotoPrism's "Sign in with OIDC" button delegates to [Authentik (124)](../106-auth-outpost.md). - **Provider/Application slug:** `mule-photos-new` - **Issuer:** `https://auth.hubris.network/application/o/mule-photos-new/` @@ -111,7 +111,7 @@ Mirrors the LXC 120 pattern. Push to the `new` branch on [git.hubris.network/dtoro/mule-image](http://git.hubris.network/dtoro/mule-image) → webhook fires → rebuild. The legacy LXC 120 watches `main` and is unaffected. **Gitea gotcha:** the receiver IP must be in `[webhook] ALLOWED_HOST_LIST` -in `/etc/gitea/app.ini` on [LXC 104](104-gitea.md). LXC 127's +in `/etc/gitea/app.ini` on [LXC 104](../104-gitea.md). LXC 127's `192.168.8.181` was missing on first bring-up; every push delivered status 0 with the message `webhook can only call allowed HTTP servers`. Adding the IP and `systemctl restart gitea` is enough — same list is @@ -145,10 +145,10 @@ curl -sk --resolve photos-new.hubris.network:443:192.168.8.175 \ ``` > **Decommissioned 2026-05-22.** The PhotoPrism + sidecar + SvelteKit stack -> validated here was promoted into production on [LXC 120](120-mule-images.md) +> validated here was promoted into production on [LXC 120](../120-mule-images.md) > via the `Mulimage 2.0` merge (`dtoro/mule-image` commit `70dc1b6`). This > page is retained for archaeology; everything below is historic. See the -> 2026-05-22 entry in [120-mule-images.md](120-mule-images.md#changelog) for +> 2026-05-22 entry in [120-mule-images.md](../120-mule-images.md#changelog) for > the cutover detail. ## Changelog diff --git a/knowledge/wiki/containers/index.md b/knowledge/wiki/containers/index.md index 9119132..9d034ac 100644 --- a/knowledge/wiki/containers/index.md +++ b/knowledge/wiki/containers/index.md @@ -15,7 +15,7 @@ Most containers live on [`hubris`](../hosts/hubris.md). Some have been | 120 | [mule-images](120-mule-images.md) | hubris | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU) | `photos.hubris.network` | running | | 121 | [caddy](121-caddy.md) | hubris | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running | | 122 | [arriman](122-arriman.md) | **strong** | 192.168.8.245 | priv | 4 | 8 GiB | 24 GiB | `/mnt/media_local` (via mp0) | `jellyseerr` / `qbit` / `sab` | running | -| 124 | [authentik](124-authentik.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running | +| 124 | [authentik](106-auth-outpost.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running | | 128 | [trmnl](128-trmnl.md) | hubris | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running | | 129 | [house](129-house.md) | **strong** | 192.168.8.244 | unpriv | 2 | 3 GiB | 8 GiB | — | `house.hubris.network` | running | | 130 | [grimmory](130-grimmory.md) | **strong** | 192.168.8.247 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `books.hubris.network` | running | @@ -32,7 +32,7 @@ Most containers live on [`hubris`](../hosts/hubris.md). Some have been | 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose | | 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained | | 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology | -| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/2026-06-04_130000-deprecate-claudio-bot.md) | +| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) | | 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty | | 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq | | 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned | @@ -45,7 +45,7 @@ Most containers live on [`hubris`](../hosts/hubris.md). Some have been ## Conventions -- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](124-authentik.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range. +- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](106-auth-outpost.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range. - `onboot=1` on every container — the host brings them up after `pve-guests.service`. - Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library` on hubris, or `mp0: /mnt/media_local,mp=/mnt/library` on strong. - Most containers are privileged. Unprivileged ones require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard. diff --git a/knowledge/wiki/hosts/hubris.md b/knowledge/wiki/hosts/hubris.md index d5155b2..48d91ba 100644 --- a/knowledge/wiki/hosts/hubris.md +++ b/knowledge/wiki/hosts/hubris.md @@ -8,7 +8,7 @@ workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node ## At a glance - **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`) - **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`). -- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../../investigations/2026-04-21-hubris-crash-loop.md). +- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). - **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`. - **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`. - **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP. @@ -114,7 +114,7 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220 - [Monitoring](../infrastructure/monitoring.md) - [Backups (disabled)](../infrastructure/backups.md) - [Operations cheatsheet](../../../operations/commands.md) -- [Investigation: 2026-04-21 crash loop](../../../investigations/2026-04-21-hubris-crash-loop.md) +- [Investigation: 2026-04-21 crash loop](../../../investigations/archive/2026-04-21-hubris-crash-loop.md) - [strong — Proxmox host](strong.md) ## Changelog @@ -123,7 +123,7 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220 User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed. ### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured -Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md). ### 2026-05-14 — LXC 109 (syncthing) decommissioned User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md). @@ -138,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. `/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel. ### 2026-04-29 — relocated to better-ventilated spot -User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation). +User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation). ### 2026-04-28 — Phase 1 WiFi failover Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them. @@ -147,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since. ### 2026-04-23 — SSD cooling + thermal pads installed -Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). +Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). ### 2026-04-22 — drive removal A/B test -Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). +Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). ### 2026-04-22 — `cpu-epp.service` ordering bug fixed Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`. @@ -159,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind ` `60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else. ### 2026-04-21 — `cpu-epp.service` deployed -Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../../investigations/2026-04-21-hubris-crash-loop.md). +Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). diff --git a/knowledge/wiki/infrastructure/auto-deploy.md b/knowledge/wiki/infrastructure/auto-deploy.md index f998201..5774362 100644 --- a/knowledge/wiki/infrastructure/auto-deploy.md +++ b/knowledge/wiki/infrastructure/auto-deploy.md @@ -23,7 +23,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web - ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~ - `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md)) - `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library) - - ~~`192.168.8.190` ([plato (126)](../containers/126-plato.md))~~ (destroyed 2026-06-28) + - ~~`192.168.8.190` ([plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology))~~ (destroyed 2026-06-28) - `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito) **Don't strip these when editing app.ini.** @@ -38,8 +38,8 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web | `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed | | `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` | | `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` | -| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/126-plato.md) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed | -| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed | +| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed | +| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed | | `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) | | `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart | | `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart | @@ -50,7 +50,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web > Each owns its own clone on LXC 105. They don't conflict because each > deploy.sh only touches its own service unit + venv. -> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md). +> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.yaml). DNS moved to [Technitium on dns (107)](../containers/107-dns.md). ## When you change a tracked config @@ -131,7 +131,7 @@ Webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` on [ Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo. ### 2026-05-13 — Plato pipeline added -Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`. +Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`. ### 2026-04-28 — wiki entry created Initial documentation. Six active pipelines. diff --git a/knowledge/wiki/infrastructure/backups.md b/knowledge/wiki/infrastructure/backups.md index 99d9bd9..2d93237 100644 --- a/knowledge/wiki/infrastructure/backups.md +++ b/knowledge/wiki/infrastructure/backups.md @@ -24,7 +24,7 @@ See [132-rclone](../containers/132-rclone.md) for the full design. ## Legacy — restic on external drive (DISABLED 2026-04-22) -Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../../investigations/2026-04-21-hubris-crash-loop.md). +Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). ## Status @@ -36,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver). -**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). +**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). **To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive. @@ -111,7 +111,7 @@ Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again. -After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). +After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). ## Thermal monitoring @@ -119,10 +119,10 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213 ## Related - [Hubris host](../hosts/hubris.md) -- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) +- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) - [Monitoring](monitoring.md) - [Auto-deploy](auto-deploy.md) -- [Investigation: 2026-04-21 crash loop](../../../investigations/2026-04-21-hubris-crash-loop.md) +- [Investigation: 2026-04-21 crash loop](../../../investigations/archive/2026-04-21-hubris-crash-loop.md) ## Changelog @@ -133,7 +133,7 @@ Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclon Initial documentation. Status remains DISABLED. ### 2026-04-22 — DISABLED -Drive removed as the A/B test in the [crash investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. +Drive removed as the A/B test in the [crash investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. ### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups. diff --git a/knowledge/wiki/infrastructure/dns.md b/knowledge/wiki/infrastructure/dns.md index c64eccd..af3bb52 100644 --- a/knowledge/wiki/infrastructure/dns.md +++ b/knowledge/wiki/infrastructure/dns.md @@ -7,7 +7,7 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr ## Components - **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22). -- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04). +- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/106-auth-outpost.md) (decommissioned 2026-06-04). - **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns. - **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where. @@ -107,10 +107,10 @@ Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backe All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog). ### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired -Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients. +Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients. ### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`) -Authentik migrated off LXC 124 onto the VPS (see [investigation](../../../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS). +Authentik migrated off LXC 124 onto the VPS (see [investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS). ### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP) NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful. @@ -119,7 +119,7 @@ NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192. New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS). ### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed -New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend). +New LAN-only entry for [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend). ### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13) Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later. diff --git a/knowledge/wiki/infrastructure/index.md b/knowledge/wiki/infrastructure/index.md index bb451f1..5e21264 100644 --- a/knowledge/wiki/infrastructure/index.md +++ b/knowledge/wiki/infrastructure/index.md @@ -36,7 +36,7 @@ are documented in their own pages. Each system below links to its full doc. ## Identity & access -- **[Authentik SSO](../containers/124-authentik.md)** — identity provider. +- **[Authentik SSO](../containers/106-auth-outpost.md)** — identity provider. Core server runs on the VPS; LAN forward-auth outpost at LXC 106. OIDC providers configured for Jellyfin, Jellyseerr, Sabnzbd, qBittorrent, Yuvomi, and more. diff --git a/knowledge/wiki/infrastructure/ingress.md b/knowledge/wiki/infrastructure/ingress.md index 75e6c5f..ebc359d 100644 --- a/knowledge/wiki/infrastructure/ingress.md +++ b/knowledge/wiki/infrastructure/ingress.md @@ -49,7 +49,7 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8 ### `auth.hubris.network` — different pattern (local container, not cert-mirror) -Since 2026-05-31 [Authentik runs on the VPS itself](../../../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above: +Since 2026-05-31 [Authentik runs on the VPS itself](../../../investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above: - Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`. - TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough). @@ -91,7 +91,7 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for ` TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed. ### 2026-05-31 — `auth.hubris.network` now served locally on the VPS -Authentik migrated onto the VPS ([investigation](../../../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack. +Authentik migrated onto the VPS ([investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack. ### 2026-04-28 — wiki entry created Initial documentation. diff --git a/knowledge/wiki/infrastructure/mesh.md b/knowledge/wiki/infrastructure/mesh.md index 52711cc..f39c016 100644 --- a/knowledge/wiki/infrastructure/mesh.md +++ b/knowledge/wiki/infrastructure/mesh.md @@ -109,7 +109,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man ## Related - [DNS split-horizon](dns.md) -- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides +- [Authentik (124)](../containers/106-auth-outpost.md) — the IdP that triggers most of these overrides - [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B - [Gitea (104)](../containers/104-gitea.md) — example of Technique A - [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport @@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man ## Changelog ### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below) -The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../../investigations/2026-05-31-authentik-vps-migration.md). +The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md). ### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik @@ -139,11 +139,11 @@ The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Ca ### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN. -Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d. +Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/106-auth-outpost.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d. Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe). -The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/124-authentik.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers. +The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/106-auth-outpost.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers. **Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21): diff --git a/knowledge/wiki/infrastructure/network.md b/knowledge/wiki/infrastructure/network.md index fb94a5d..99f4c08 100644 --- a/knowledge/wiki/infrastructure/network.md +++ b/knowledge/wiki/infrastructure/network.md @@ -79,10 +79,10 @@ No NAT on Proxmox — traffic flows without double-NAT. ### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2) Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed. -Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md). ### 2026-06-02 — Executed migration; Proxmox as subnet router -Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md). ### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added -Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md). diff --git a/knowledge/wiki/vms/100-zimaos.md b/knowledge/wiki/vms/100-zimaos.md index ac58ac6..ac68f42 100644 --- a/knowledge/wiki/vms/100-zimaos.md +++ b/knowledge/wiki/vms/100-zimaos.md @@ -46,7 +46,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. ## Open items - **DHCP → static IP fixed (2026-06-03).** ZimaOS IP drifted from `.195` (Slate AX) → `.103` (Technitium) after the DHCP migration, causing Caddy 502s. Fixed by injecting a static systemd-networkd config and restarting the VM. IP now pinned at `192.168.8.195`. See [changelog](#2026-06-03--static-ip-set-to-195-dhcp-drift-fixed). -- **No Authentik wiring.** [authentik (124)](../containers/124-authentik.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists. +- **No Authentik wiring.** [authentik (124)](../containers/106-auth-outpost.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists. - **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up. - **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor` → `screendump`. @@ -59,7 +59,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. ## Changelog ### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed -ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md). ### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix) @@ -87,4 +87,4 @@ Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs mod Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`. ### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1) -`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/124-authentik.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation. +`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/106-auth-outpost.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation. diff --git a/knowledge/wiki/vms/108-haos.md b/knowledge/wiki/vms/108-haos.md index 6ac7ca6..90c78a7 100644 --- a/knowledge/wiki/vms/108-haos.md +++ b/knowledge/wiki/vms/108-haos.md @@ -7,7 +7,7 @@ Home Assistant OS — the only VM on hubris (HAOS doesn't run cleanly in an LXC, - **HAOS version:** 16.3 (last verified) - **IP:** `192.168.8.101` - **Resources:** 4 GiB RAM, 32 GiB boot disk -- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `192.168.8.101:8123` +- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.101:8123` ## Auth @@ -18,7 +18,7 @@ Key gotchas: ``` ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1" ``` - so OIDC discovery resolves internally to [authentik (124)](124-authentik.md). + so OIDC discovery resolves internally to [authentik (124)](../containers/106-auth-outpost.md). - Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen). - YAML config: - `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created). @@ -30,8 +30,8 @@ Key gotchas: HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting. ## Related -- [Authentik (124)](124-authentik.md) -- [Caddy (121)](121-caddy.md) +- [Authentik (124)](../containers/106-auth-outpost.md) +- [Caddy (121)](../containers/121-caddy.md) - [DNS](../infrastructure/dns.md) - [Monitoring](../infrastructure/monitoring.md) diff --git a/operations/agent-enrollment.md b/operations/agent-enrollment.md index dfd41a6..be0b15b 100644 --- a/operations/agent-enrollment.md +++ b/operations/agent-enrollment.md @@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight. -Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../knowledge/wiki/containers/124-authentik.md). Only needed once per account. +Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account. **Path A — setup-key (headless/scripted onboarding):** @@ -261,7 +261,7 @@ arguments. If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be visible to the classifier (it's not actually durable in 0.71.2, but the -ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md) +ControlMaster block is — see [runbook-dpkg-interrupted](../.agents/skills/runbook-dpkg-interrupted/SKILL.md) for context), drop a free-text rule into `autoMode.allow` describing the authorization. Optional. @@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually: | `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. | | `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. | | Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). | -| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../knowledge/wiki/containers/124-authentik.md). | +| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../knowledge/wiki/containers/106-auth-outpost.md). | | `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. | | `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). | | `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. | diff --git a/operations/commands.md b/operations/commands.md index bdc1cd7..797953b 100644 --- a/operations/commands.md +++ b/operations/commands.md @@ -49,7 +49,7 @@ Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When work - `ras-mc-ctl --errors` — full event log - `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power` - `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave` -- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md)) +- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/archive/2026-04-21-hubris-crash-loop.md)) ## Fleet apt operations @@ -88,4 +88,4 @@ Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — - [DNS](../knowledge/wiki/infrastructure/dns.md) - [Monitoring](../knowledge/wiki/infrastructure/monitoring.md) - [Auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md) -- [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction +- [Runbook: dpkg-interrupted recovery](../.agents/skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction diff --git a/plans/2026-06-24-trmnl-plugins-lxc.md b/plans/2026-06-24-trmnl-plugins-lxc.md index 294219d..78c88ba 100644 --- a/plans/2026-06-24-trmnl-plugins-lxc.md +++ b/plans/2026-06-24-trmnl-plugins-lxc.md @@ -14,7 +14,7 @@ wiring, same split as Artifacto/Plato. ## Current state - No TRMNL middleware in the lab. Highest LXC id is 127 (see `containers/index.md`). -- Public hostnames terminate at the [VPS netbird traefik](../hosts/netbird-vps.md) → netbird +- Public hostnames terminate at the [VPS netbird traefik](../hosts/netbird-vps.yaml) → netbird mesh → [caddy (121)](../knowledge/wiki/containers/121-caddy.md) → backend LXC. Cert obtained by Caddy (IONOS DNS-01) and mirrored to the VPS by the daily cert-sync timer on the host. - Auto-deploy pipelines are gitea-webhook driven, two shapes (see [auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md)). @@ -81,7 +81,7 @@ TRMNL cloud --GET 15m, Bearer token--> https://trmnl.hubris.network/munich-hom trmnl.hubris.network { reverse_proxy 192.168.8.<128-ip>:9851 } ``` -7. **Public exposure** on the [VPS](../hosts/netbird-vps.md): add traefik router+service for +7. **Public exposure** on the [VPS](../hosts/netbird-vps.yaml): add traefik router+service for `Host(\`trmnl.hubris.network\`)` → `http://192.168.8.<128-ip>:9851`; add the host to the cert-sync map so the LE cert mirrors over. diff --git a/plans/done/2026-06-01-slate-ax-to-sodola-migration.md b/plans/done/2026-06-01-slate-ax-to-sodola-migration.md index 8f67eae..62fdc67 100644 --- a/plans/done/2026-06-01-slate-ax-to-sodola-migration.md +++ b/plans/done/2026-06-01-slate-ax-to-sodola-migration.md @@ -37,7 +37,7 @@ ISP Fritz!Box takes over `192.168.8.1` — the same gateway IP the Slate AX used. No static IPs or gateway entries change on any LXC or VM. -See [network architecture](../infrastructure/network.md) for the permanent topology reference. +See [network architecture](../../knowledge/wiki/infrastructure/network.md) for the permanent topology reference. ## Pre-flight checklist @@ -60,7 +60,7 @@ See [network architecture](../infrastructure/network.md) for the permanent topol | DHCP range | `192.168.8.100 – 192.168.8.240` | | Assign to | LAN port that connects to SODOLA | | Network isolation | Enabled (blocks main LAN from initiating into homelab) | -| DNS for DHCP clients | `192.168.8.2` (Technitium on [CT 107](../containers/107-dns.md)) | +| DNS for DHCP clients | `192.168.8.2` (Technitium on [CT 107](../../knowledge/wiki/containers/107-dns.md)) | After creating the network, move any port forwards from the Slate AX into Fritz!Box → Internet → Permits (target IPs are now directly reachable on `192.168.8.x`). @@ -92,7 +92,7 @@ pct set <id> --net0 name=eth0,bridge=vmbr0,ip=<ip>/24,gw=192.168.8.1 ## DNS after migration -Technitium ([CT 107](../containers/107-dns.md)) at `192.168.8.2` continues to serve split-horizon DNS for `hubris.network`. The Fritz!Box DHCP server for VLAN 10 hands out `192.168.8.2` as the DNS server. This fixes the "update router DHCP DNS from dead .180 → .2" outstanding item in [dns.md](../infrastructure/dns.md). +Technitium ([CT 107](../../knowledge/wiki/containers/107-dns.md)) at `192.168.8.2` continues to serve split-horizon DNS for `hubris.network`. The Fritz!Box DHCP server for VLAN 10 hands out `192.168.8.2` as the DNS server. This fixes the "update router DHCP DNS from dead .180 → .2" outstanding item in [dns.md](../../knowledge/wiki/infrastructure/dns.md). ## Cutover procedure @@ -125,7 +125,7 @@ curl -sk https://auth.hubris.network/if/flow/default-authentication-flow/ | head ## Post-migration -- Update [network.md](../infrastructure/network.md) topology to reflect new state. -- Add changelog entries to [hosts/hubris.md](../hosts/hubris.md) and any affected container pages. -- Update status in [plans/index.md](index.md) to `Done`. +- Update [network.md](../../knowledge/wiki/infrastructure/network.md) topology to reflect new state. +- Add changelog entries to [hosts/hubris.md](../../knowledge/wiki/hosts/hubris.md) and any affected container pages. +- Update status in [plans/index.md](../index.md) to `Done`. - If anything went sideways, open an investigation in `investigations/`. diff --git a/plans/done/2026-06-29-grimmory-migration.md b/plans/done/2026-06-29-grimmory-migration.md index 2f89f23..89526d9 100644 --- a/plans/done/2026-06-29-grimmory-migration.md +++ b/plans/done/2026-06-29-grimmory-migration.md @@ -204,7 +204,7 @@ books.hubris.network { } ``` -Git push → Caddy webhook auto-reloads (see [caddy (121)](../containers/121-caddy.md)). +Git push → Caddy webhook auto-reloads (see [caddy (121)](../../knowledge/wiki/containers/121-caddy.md)). Test: ```bash From b5c124709383c32904b36d9d752254cfac92371a Mon Sep 17 00:00:00 2001 From: dtoro <d.toro.v@pm.me> Date: Mon, 6 Jul 2026 18:12:14 +0200 Subject: [PATCH 6/8] docs: streamline & consolidate the tree (phase 6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem: after the wiki-hq reorg, agent-instruction and human-doc domains were still scattered across the repo root, with three now-redundant stub files cluttering it. The organizing principle wasn't visible in the layout. Change — enforce three clear buckets: - .agents/ = how agents operate: OIKOS.md, HERMES.md (moved from root), shared/ conventions, domains/ schemas, skills/, and operations/ (operator cheatsheet + enrollment + hermes-agent, moved from root). - knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now including investigations/ (incident records are evidence/sources). - root = substrate + two entry points (AGENTS.md, README.md), plus plans/ as its own design-intent domain. Moves: - investigations/ -> knowledge/sources/investigations/ (incl. archive/, index). - operations/ -> .agents/operations/. - HERMES.md -> .agents/HERMES.md. - Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md (its 7 remaining linkers repointed to .agents/OIKOS.md). Consumers updated: - inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards. - tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md. - bin/homelab help string -> .agents/operations/hermes-agent.md. - knowledge/operations schemas, llm-wiki, page-templates, incident-investigation skill, AGENTS.md/README nav -> new investigations/operations paths. - All markdown links rewritten via the path-resolving mapper. Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/, plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml. Verification: docs-lint at baseline (2 intentional cross-repo refs, no new breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all doc_page targets resolve; Hermes provisioning scripts point at the new path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- HERMES.md => .agents/HERMES.md | 0 .agents/OIKOS.md | 4 +-- .agents/domains/knowledge/schema.md | 2 +- .agents/domains/operations/schema.md | 11 +++++-- .../operations}/agent-enrollment.md | 8 ++--- .../operations}/commands.md | 32 +++++++++---------- .../operations}/hermes-agent.md | 4 +-- .agents/shared/llm-wiki.md | 4 +-- .agents/shared/page-templates.md | 2 +- .agents/skills/client-enrollment/SKILL.md | 2 +- .agents/skills/config-change-deploy/SKILL.md | 2 +- .../skills/incident-investigation/SKILL.md | 2 +- .../skills/runbook-dpkg-interrupted/SKILL.md | 2 +- AGENTS.md | 8 ++--- CAVEMAN.md | 3 -- CONTRIBUTING.md | 7 ---- OIKOS.md | 3 -- README.md | 10 +++--- bin/homelab | 2 +- bootstrap.sh | 4 +-- hosts/apps.yaml | 2 +- inventory.yaml | 2 +- knowledge/log.md | 1 + knowledge/sources/index.md | 2 +- .../2026-06-01-mac-mini-onboarding.md | 0 ...26-06-03-moonlight-sunshine-wifi-jitter.md | 2 +- .../2026-06-06-authentik-session-lifetime.md | 4 +-- .../2026-06-06-caddyfile-truncation.md | 10 +++--- .../archive/2026-04-21-hubris-crash-loop.md | 14 ++++---- .../2026-05-31-authentik-vps-migration.md | 20 ++++++------ .../sources/investigations}/index.md | 0 knowledge/wiki/containers/102-nfs-export.md | 2 +- knowledge/wiki/containers/106-auth-outpost.md | 10 +++--- knowledge/wiki/containers/131-teddycloud.md | 4 +-- knowledge/wiki/containers/132-rclone.md | 2 +- knowledge/wiki/hosts/hubris.md | 14 ++++---- knowledge/wiki/hosts/strong.md | 6 ++-- knowledge/wiki/infrastructure/auto-deploy.md | 2 +- knowledge/wiki/infrastructure/backups.md | 10 +++--- knowledge/wiki/infrastructure/dns.md | 2 +- .../wiki/infrastructure/homelab-context.md | 4 +-- knowledge/wiki/infrastructure/index.md | 4 +-- knowledge/wiki/infrastructure/ingress.md | 4 +-- knowledge/wiki/infrastructure/mesh.md | 2 +- knowledge/wiki/infrastructure/ssh-access.md | 2 +- oikos/cards/service-secrets_issuance.md | 2 +- plans/2026-07-05-oikos-prometheus-lxc.md | 2 +- tools/setup-hermes-soul.sh | 2 +- 48 files changed, 118 insertions(+), 125 deletions(-) rename HERMES.md => .agents/HERMES.md (100%) rename {operations => .agents/operations}/agent-enrollment.md (98%) rename {operations => .agents/operations}/commands.md (74%) rename {operations => .agents/operations}/hermes-agent.md (98%) delete mode 100644 CAVEMAN.md delete mode 100644 CONTRIBUTING.md delete mode 100644 OIKOS.md rename {investigations => knowledge/sources/investigations}/2026-06-01-mac-mini-onboarding.md (100%) rename {investigations => knowledge/sources/investigations}/2026-06-03-moonlight-sunshine-wifi-jitter.md (89%) rename {investigations => knowledge/sources/investigations}/2026-06-06-authentik-session-lifetime.md (97%) rename {investigations => knowledge/sources/investigations}/2026-06-06-caddyfile-truncation.md (89%) rename {investigations => knowledge/sources/investigations}/archive/2026-04-21-hubris-crash-loop.md (86%) rename {investigations => knowledge/sources/investigations}/archive/2026-05-31-authentik-vps-migration.md (79%) rename {investigations => knowledge/sources/investigations}/index.md (100%) diff --git a/HERMES.md b/.agents/HERMES.md similarity index 100% rename from HERMES.md rename to .agents/HERMES.md diff --git a/.agents/OIKOS.md b/.agents/OIKOS.md index af30e59..5933f3d 100644 --- a/.agents/OIKOS.md +++ b/.agents/OIKOS.md @@ -44,7 +44,7 @@ one pass through **Observe → Orient → Decide → Act**: | Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) | | Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) | | Approval | short-TTL signed grant for a gated action | approval engine (Week 3) | -| Incident | investigation narrative | `investigations/` | +| Incident | investigation narrative | `knowledge/sources/investigations/` | | Plan | design doc for non-trivial work | `plans/` | | Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` | @@ -207,7 +207,7 @@ guesswork. Roughly ordered by what unblocks the most: populate more as specific services turn out to need non-default risk classes. - **Incident timeline generator** — stitch ledger + signal history into - a single narrative for `investigations/` entries instead of writing + a single narrative for `knowledge/sources/investigations/` entries instead of writing them by hand. - **Secret access audit** — who-can-decrypt-what report from `.sops.yaml` + inventory `age_pubkey`s, extending what diff --git a/.agents/domains/knowledge/schema.md b/.agents/domains/knowledge/schema.md index e16071b..da74682 100644 --- a/.agents/domains/knowledge/schema.md +++ b/.agents/domains/knowledge/schema.md @@ -20,7 +20,7 @@ fixed paths, so the wiki reorganization never moves it. | Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` | | Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler | | Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` | -| Evidence — immutable sources | `knowledge/sources/`, `investigations/` | synthesis into wiki pages | +| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages | ## Wiki pages diff --git a/.agents/domains/operations/schema.md b/.agents/domains/operations/schema.md index aadef2b..05c60e5 100644 --- a/.agents/domains/operations/schema.md +++ b/.agents/domains/operations/schema.md @@ -5,6 +5,11 @@ procedures), investigations (incident evidence), and plans (design docs for non- follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice exception. +Where each kind lives: runbooks are skills under [`.agents/skills/`](../../skills/); operator +reference (command cheatsheet, enrollment, Hermes agent) lives in +[`.agents/operations/`](../../operations/); investigations are sources under +`knowledge/sources/investigations/`; plans stay in the repo-root `plans/` folder (below). + ## Plans always live in `plans/` **Any plan or design doc for the Homelab is written into the repo `plans/` folder as @@ -39,12 +44,12 @@ transition: "<from> -> <to>" # only for lifecycle runbooks ## Investigations -Incident records live in `investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written +Incident records live in `knowledge/sources/investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written once at incident time, then linked from the changelogs of the nodes they implicate. Sections: `## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved -incidents move to `investigations/archive/`. +incidents move to `knowledge/sources/investigations/archive/`. ## The operations log -`plans/log.md` and `investigations/log.md` are append-only records of documentation operations on +`plans/log.md` and `knowledge/log.md` are append-only records of documentation operations on those areas (`## [YYYY-MM-DD] <op> | <summary>`), distinct from the Oikos change ledger. diff --git a/operations/agent-enrollment.md b/.agents/operations/agent-enrollment.md similarity index 98% rename from operations/agent-enrollment.md rename to .agents/operations/agent-enrollment.md index be0b15b..d4174ef 100644 --- a/operations/agent-enrollment.md +++ b/.agents/operations/agent-enrollment.md @@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight. -Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account. +Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account. **Path A — setup-key (headless/scripted onboarding):** @@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr ### DNS prerequisite `*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124 -([dns.md](../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the +([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the client uses 192.168.8.180 as its resolver**. Most LXCs and roaming workstations don't by default. Options: @@ -261,7 +261,7 @@ arguments. If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be visible to the classifier (it's not actually durable in 0.71.2, but the -ControlMaster block is — see [runbook-dpkg-interrupted](../.agents/skills/runbook-dpkg-interrupted/SKILL.md) +ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) for context), drop a free-text rule into `autoMode.allow` describing the authorization. Optional. @@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually: | `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. | | `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. | | Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). | -| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../knowledge/wiki/containers/106-auth-outpost.md). | +| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). | | `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. | | `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). | | `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. | diff --git a/operations/commands.md b/.agents/operations/commands.md similarity index 74% rename from operations/commands.md rename to .agents/operations/commands.md index 797953b..1dbc137 100644 --- a/operations/commands.md +++ b/.agents/operations/commands.md @@ -1,6 +1,6 @@ # Operations cheatsheet -Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`. +Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`. ## Proxmox CLI @@ -8,13 +8,13 @@ Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When work | --- | --- | | `pct list` / `qm list` | List LXC containers / VMs | | `pct config <id>` / `qm config <id>` | Container / VM config | -| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../knowledge/wiki/infrastructure/media-permissions.md)) | +| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) | | `pct enter <id>` | Shell into a container | | `pct start <id>` / `pct stop <id>` | Boot / halt a container | | `pvesm status` | Storage pools status | | `pvesh get /nodes --output-format json` | Node summary as JSON | | `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status | -| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) | +| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) | | `pveversion` | PVE version | | `journalctl -u pve-cluster -n 100` | PVE service logs | @@ -22,22 +22,22 @@ Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When work - Shared mount: `/mnt/library` (ext4 on lvmthin `library`). - Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data` -- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe. +- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe. ## Reverse proxy -- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../knowledge/wiki/containers/121-caddy.md). -- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it. +- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md). +- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it. - Hot reload: `pct exec 121 -- systemctl reload caddy`. - Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`. - Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`. ## DNS -- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.) +- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.) - Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes. - Verify: `dig @192.168.8.2 +short <host>.hubris.network`. -- See [DNS](../knowledge/wiki/infrastructure/dns.md). +- See [DNS](../../knowledge/wiki/infrastructure/dns.md). ## Web access @@ -49,7 +49,7 @@ Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When work - `ras-mc-ctl --errors` — full event log - `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power` - `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave` -- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/archive/2026-04-21-hubris-crash-loop.md)) +- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md)) ## Fleet apt operations @@ -80,12 +80,12 @@ See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference: | `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) | | `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant | -Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../oikos/console/deploy/README.md). +Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md). ## Related -- [Hubris host](../knowledge/wiki/hosts/hubris.md) -- [Containers index](../knowledge/wiki/containers/index.md) -- [DNS](../knowledge/wiki/infrastructure/dns.md) -- [Monitoring](../knowledge/wiki/infrastructure/monitoring.md) -- [Auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md) -- [Runbook: dpkg-interrupted recovery](../.agents/skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction +- [Hubris host](../../knowledge/wiki/hosts/hubris.md) +- [Containers index](../../knowledge/wiki/containers/index.md) +- [DNS](../../knowledge/wiki/infrastructure/dns.md) +- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md) +- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md) +- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction diff --git a/operations/hermes-agent.md b/.agents/operations/hermes-agent.md similarity index 98% rename from operations/hermes-agent.md rename to .agents/operations/hermes-agent.md index 7227f09..5c3736d 100644 --- a/operations/hermes-agent.md +++ b/.agents/operations/hermes-agent.md @@ -169,9 +169,9 @@ every tool call, use `approve`. See Hermes flow assumes is done. - [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every session start (via `~/.config/goose/.goosehints`). -- [`bin/hermes`](../bin/hermes) — the wrapper that decrypts the OpenRouter key +- [`bin/hermes`](../../bin/hermes) — the wrapper that decrypts the OpenRouter key and execs `goose session`. -- [`bootstrap.sh`](../bootstrap.sh) — the `--with-hermes` flag's install block. +- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-hermes` flag's install block. ## Follow-ups diff --git a/.agents/shared/llm-wiki.md b/.agents/shared/llm-wiki.md index 1d6f69f..6101caa 100644 --- a/.agents/shared/llm-wiki.md +++ b/.agents/shared/llm-wiki.md @@ -2,7 +2,7 @@ How the narrative documentation in this repo is organized. The pattern is borrowed from the `sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top -of immutable evidence (`knowledge/sources/`, `investigations/`), with pure-listing indexes and an +of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an append-only operations log. This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`, @@ -11,7 +11,7 @@ moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for t ## Layers -- **Sources** are immutable raw material: incident records (`investigations/`), external reference +- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`). Read them; do not rewrite them into other sources. - **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per diff --git a/.agents/shared/page-templates.md b/.agents/shared/page-templates.md index c848bc3..247a444 100644 --- a/.agents/shared/page-templates.md +++ b/.agents/shared/page-templates.md @@ -96,7 +96,7 @@ What it looks like after. Changelog entries to write, index status to update. ``` -### Investigation (`investigations/YYYY-MM-DD-slug.md`) +### Investigation (`knowledge/sources/investigations/YYYY-MM-DD-slug.md`) ```markdown # YYYY-MM-DD — <title> diff --git a/.agents/skills/client-enrollment/SKILL.md b/.agents/skills/client-enrollment/SKILL.md index 6939e96..043f233 100644 --- a/.agents/skills/client-enrollment/SKILL.md +++ b/.agents/skills/client-enrollment/SKILL.md @@ -11,7 +11,7 @@ docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm] Goal: bring a new host (workstation, LXC, VM) into inventory and the secrets model, with mesh membership only where it's actually needed. This wraps the existing `homelab client add` flow — see -[operations/agent-enrollment.md](../../../operations/agent-enrollment.md) for +[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for the full walkthrough; this runbook is the risk/lifecycle framing. 1. On any enrolled client: `homelab client add <hostname>` — appends a diff --git a/.agents/skills/config-change-deploy/SKILL.md b/.agents/skills/config-change-deploy/SKILL.md index 3b4668f..0f83139 100644 --- a/.agents/skills/config-change-deploy/SKILL.md +++ b/.agents/skills/config-change-deploy/SKILL.md @@ -18,7 +18,7 @@ own repo) and get it live, safely. `oikos/policy.yaml`. 2. Clone/pull the `config_repo` (never edit the backend's working tree directly — tracked configs change by commit + push, per - [OIKOS.md](../../../OIKOS.md) conventions). + [OIKOS.md](../../OIKOS.md) conventions). 3. Make the change, commit, push to `main`. 4. The Gitea webhook fires the deploy pipeline for that repo (see [infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for diff --git a/.agents/skills/incident-investigation/SKILL.md b/.agents/skills/incident-investigation/SKILL.md index 3d8a36f..2858192 100644 --- a/.agents/skills/incident-investigation/SKILL.md +++ b/.agents/skills/incident-investigation/SKILL.md @@ -23,7 +23,7 @@ Goal: understand what broke and why, before touching anything. 5. Check the change ledger for recent mutations to the affected entity or anything upstream of it: `homelab service <name> history` (once populated) or grep `ledger/*.jsonl`. -6. Write findings to a new `investigations/<date>-<slug>.md` — symptom, +6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom, timeline, root cause, fix applied, prevention. This is the durable record; don't rely on chat history. diff --git a/.agents/skills/runbook-dpkg-interrupted/SKILL.md b/.agents/skills/runbook-dpkg-interrupted/SKILL.md index 53ab143..071c58e 100644 --- a/.agents/skills/runbook-dpkg-interrupted/SKILL.md +++ b/.agents/skills/runbook-dpkg-interrupted/SKILL.md @@ -102,7 +102,7 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress. ## Related -- [Operations cheatsheet](../../../operations/commands.md) +- [Operations cheatsheet](../../operations/commands.md) - [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md) - [Hubris host page](../../../knowledge/wiki/hosts/hubris.md) diff --git a/AGENTS.md b/AGENTS.md index f330689..d29e474 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,7 +38,7 @@ the operator to run `homelab client add <hostname>` from an existing client. - `/opt/homelab-context/knowledge/wiki/infrastructure/dns.md` — split-horizon DNS via Technitium on [dns (107)](knowledge/wiki/containers/107-dns.md). `*.hubris.network` resolves to 192.168.x.x on the LAN and to mesh addresses off-LAN. -- `/opt/homelab-context/operations/commands.md` — the operator's cheatsheet +- `/opt/homelab-context/.agents/operations/commands.md` — the operator's cheatsheet for pct, caddy, DNS, and the Oikos command surface. Use these verbs when you take actions. @@ -76,9 +76,9 @@ Grep is fine for browsing or when MCP is unreachable. ## 4. Wiki conventions - Narrative pages live under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`; - runbook procedures under `.agents/skills/<name>/SKILL.md`; other procedural - pages under `investigations/`, `operations/`, `plans/`. - Cross-link liberally; orphans are bugs. + incident records under `knowledge/sources/investigations/`; runbook procedures under + `.agents/skills/<name>/SKILL.md`; operator reference under `.agents/operations/`; + design docs under `plans/`. Cross-link liberally; orphans are bugs. - Every page ends with a `## Changelog` section, entries in reverse-chrono order: diff --git a/CAVEMAN.md b/CAVEMAN.md deleted file mode 100644 index 1660494..0000000 --- a/CAVEMAN.md +++ /dev/null @@ -1,3 +0,0 @@ -# CAVEMAN.md — moved - -Agent chat-mode rules now live at [`.agents/shared/caveman.md`](.agents/shared/caveman.md). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index df5760e..0000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,7 +0,0 @@ -# CONTRIBUTING.md — moved - -Documentation conventions are now split across `.agents/shared/`: - -- [`page-templates.md`](.agents/shared/page-templates.md) — per-page-type structural templates, linking discipline, changelog hygiene, same-session update rule. -- [`writing-style.md`](.agents/shared/writing-style.md) — prose voice, banned vocabulary, page shape. -- [`llm-wiki.md`](.agents/shared/llm-wiki.md) — the sources / wiki / index / log layer model. diff --git a/OIKOS.md b/OIKOS.md deleted file mode 100644 index bc6bb92..0000000 --- a/OIKOS.md +++ /dev/null @@ -1,3 +0,0 @@ -# OIKOS.md — moved - -The Oikos operating model now lives at [`.agents/OIKOS.md`](.agents/OIKOS.md). diff --git a/README.md b/README.md index 6515cc0..2735ff9 100644 --- a/README.md +++ b/README.md @@ -55,12 +55,12 @@ See the full table with IPs, hosts, mounts, and status in - [Homelab context distribution](knowledge/wiki/infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance ### Investigations -Time-stamped incident notes / experiments in [`investigations/index.md`](investigations/index.md). -Resolved cases move to [`investigations/archive/`](investigations/archive/). +Time-stamped incident notes / experiments in [`investigations/index.md`](knowledge/sources/investigations/index.md). +Resolved cases move to [`investigations/archive/`](knowledge/sources/investigations/archive/). ### Operations -- [Command cheatsheet](operations/commands.md) -- [Agent enrollment](operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system +- [Command cheatsheet](.agents/operations/commands.md) +- [Agent enrollment](.agents/operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system ## Conventions @@ -80,7 +80,7 @@ When you change a node: 1. Update the relevant page (config snapshot, ports, mounts). 2. Add a changelog entry at the bottom of that page. 3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry. -4. If it's an incident, add an entry to [`investigations/`](investigations/index.md). +4. If it's an incident, add an entry to [`investigations/`](knowledge/sources/investigations/index.md). ## See also diff --git a/bin/homelab b/bin/homelab index 3a6d002..8f79cda 100755 --- a/bin/homelab +++ b/bin/homelab @@ -1929,7 +1929,7 @@ def main() -> int: csub_add.add_argument("--with-hermes", action="store_true", help="also grant secrets/openrouter-api-key.yaml so this " "host can run the Hermes agent (see " - "operations/hermes-agent.md). Combine with --finalize-pubkey.") + ".agents/operations/hermes-agent.md). Combine with --finalize-pubkey.") csub_add.set_defaults(func=cmd_client_add) csub_rm = csub.add_parser("remove") csub_rm.add_argument("name") diff --git a/bootstrap.sh b/bootstrap.sh index 70608bb..4431c4f 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -507,7 +507,7 @@ if [ "$WITH_HERMES" -eq 1 ]; then Linux) HERMES_LINK=/root/HERMES.md ;; Darwin) HERMES_LINK=/etc/HERMES.md ;; esac - run "ln -sfn '$CLONE_DIR/HERMES.md' '$HERMES_LINK'" + run "ln -sfn '$CLONE_DIR/.agents/HERMES.md' '$HERMES_LINK'" echo "[bootstrap] linked HERMES.md → $HERMES_LINK" # 4. Drop the Goose config. Idempotent YAML merge — preserves any keys the @@ -563,7 +563,7 @@ PYEOF # 5. Symlink HERMES.md as the global .goosehints — Goose injects it into # the system prompt on every session start. - run "ln -sfn '$CLONE_DIR/HERMES.md' '$GOOSEHINTS'" + run "ln -sfn '$CLONE_DIR/.agents/HERMES.md' '$GOOSEHINTS'" if [ "$DRY_RUN" -eq 0 ]; then chown -h "$H_USER" "$GOOSEHINTS" 2>/dev/null || true fi diff --git a/hosts/apps.yaml b/hosts/apps.yaml index 48cbb8d..375be02 100644 --- a/hosts/apps.yaml +++ b/hosts/apps.yaml @@ -53,7 +53,7 @@ services_hosted: systemd_unit: secrets-issuance public_host: secrets.hubris.network endpoint: https://secrets.hubris.network/issue - doc_page: operations/agent-enrollment.md + doc_page: .agents/operations/agent-enrollment.md config_repo: dtoro/Homelab-Docs note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS). risk_notes: "identity issuance \u2014 any change is security-sensitive; key operations are destructive-class" diff --git a/inventory.yaml b/inventory.yaml index db8bedb..3f5f087 100644 --- a/inventory.yaml +++ b/inventory.yaml @@ -150,7 +150,7 @@ services: systemd_unit: secrets-issuance public_host: secrets.hubris.network endpoint: https://secrets.hubris.network/issue - doc_page: operations/agent-enrollment.md + doc_page: .agents/operations/agent-enrollment.md config_repo: dtoro/Homelab-Docs note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS). diff --git a/knowledge/log.md b/knowledge/log.md index c52f3d4..e01f239 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -7,3 +7,4 @@ each page's `## Changelog` and the Oikos change ledger, not here. ## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output. ## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md. ## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference. +## [2026-07-06] restructure | Phase 6 consolidation: investigations/ -> knowledge/sources/investigations/; operations/ -> .agents/operations/; HERMES.md -> .agents/; deleted root OIKOS/CAVEMAN/CONTRIBUTING stubs. diff --git a/knowledge/sources/index.md b/knowledge/sources/index.md index f4559f3..e8243a6 100644 --- a/knowledge/sources/index.md +++ b/knowledge/sources/index.md @@ -1,7 +1,7 @@ # Sources Immutable evidence the wiki synthesizes from. External reference docs live under `references/`; -incident evidence lives in [`investigations/`](../../investigations/index.md) (written once at +incident evidence lives in [`investigations/`](investigations/index.md) (written once at incident time, then linked from the changelogs of the nodes they implicate). | Slug | Reference | Summary | diff --git a/investigations/2026-06-01-mac-mini-onboarding.md b/knowledge/sources/investigations/2026-06-01-mac-mini-onboarding.md similarity index 100% rename from investigations/2026-06-01-mac-mini-onboarding.md rename to knowledge/sources/investigations/2026-06-01-mac-mini-onboarding.md diff --git a/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md b/knowledge/sources/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md similarity index 89% rename from investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md rename to knowledge/sources/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md index 256f81d..1230b1d 100644 --- a/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md +++ b/knowledge/sources/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md @@ -2,7 +2,7 @@ ## Summary -[`ludo-mini`](../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming. +[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming. ## Timeline diff --git a/investigations/2026-06-06-authentik-session-lifetime.md b/knowledge/sources/investigations/2026-06-06-authentik-session-lifetime.md similarity index 97% rename from investigations/2026-06-06-authentik-session-lifetime.md rename to knowledge/sources/investigations/2026-06-06-authentik-session-lifetime.md index 76adcad..2a00d1c 100644 --- a/investigations/2026-06-06-authentik-session-lifetime.md +++ b/knowledge/sources/investigations/2026-06-06-authentik-session-lifetime.md @@ -90,9 +90,9 @@ print("session_duration:", stage.session_duration) # → "days=30" ## Related -- [Container 106 — auth-outpost](../knowledge/wiki/containers/106-auth-outpost.md) +- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md) - [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md) -- [Ingress (VPS Traefik)](../knowledge/wiki/infrastructure/ingress.md) +- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md) - `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan ## Changelog diff --git a/investigations/2026-06-06-caddyfile-truncation.md b/knowledge/sources/investigations/2026-06-06-caddyfile-truncation.md similarity index 89% rename from investigations/2026-06-06-caddyfile-truncation.md rename to knowledge/sources/investigations/2026-06-06-caddyfile-truncation.md index 516d87f..7b7f3c7 100644 --- a/investigations/2026-06-06-caddyfile-truncation.md +++ b/knowledge/sources/investigations/2026-06-06-caddyfile-truncation.md @@ -54,8 +54,8 @@ This is the same class of drift as the June 5th incidents (paperless, HAOS, apps ## Related -- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md) -- [Caddy (121)](../knowledge/wiki/containers/121-caddy.md) -- [elementsynapse (118)](../knowledge/wiki/containers/118-elementsynapse.md) -- [dns-sync script](../scripts/dns-sync.py) -- [check-caddy-backends script](../scripts/check-caddy-backends.sh) \ No newline at end of file +- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md) +- [Caddy (121)](../../wiki/containers/121-caddy.md) +- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md) +- [dns-sync script](../../../scripts/dns-sync.py) +- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh) \ No newline at end of file diff --git a/investigations/archive/2026-04-21-hubris-crash-loop.md b/knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md similarity index 86% rename from investigations/archive/2026-04-21-hubris-crash-loop.md rename to knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md index 4466649..4192b0d 100644 --- a/investigations/archive/2026-04-21-hubris-crash-loop.md +++ b/knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md @@ -2,12 +2,12 @@ ## Summary -[`hubris`](../../knowledge/wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders. +[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders. ## Timeline ### 2026-04-19 — drive attached -External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../knowledge/wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable. +External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable. ### 2026-04-19 → 2026-04-21 — first crashes Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. @@ -24,13 +24,13 @@ Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly - **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running. ### 2026-04-22 — recurrence after 30 h 37 m -Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../knowledge/wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty. +Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty. ### 2026-04-22 — `cpu-epp.service` design bug fixed Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`. ### 2026-04-22 — drive removed (A/B test) -User physically removed the external USB drive. [Backup timers disabled](../../knowledge/wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause. +User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause. ### 2026-04-23 — SSD cooling + thermal pads installed Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically. @@ -78,9 +78,9 @@ Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the Nu | `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive | ## Affected nodes -- [Hubris host](../../knowledge/wiki/hosts/hubris.md) -- [Backups (disabled)](../../knowledge/wiki/infrastructure/backups.md) -- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md) +- [Hubris host](../../../wiki/hosts/hubris.md) +- [Backups (disabled)](../../../wiki/infrastructure/backups.md) +- [Monitoring](../../../wiki/infrastructure/monitoring.md) ## Open questions - Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.) diff --git a/investigations/archive/2026-05-31-authentik-vps-migration.md b/knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md similarity index 79% rename from investigations/archive/2026-05-31-authentik-vps-migration.md rename to knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md index ba91978..5ca79c4 100644 --- a/investigations/archive/2026-05-31-authentik-vps-migration.md +++ b/knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md @@ -2,9 +2,9 @@ ## Summary -The NetBird management server (on the [VPS](../../knowledge/wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.** +The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.** -Resolved by moving Authentik off [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur. +Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur. The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration. @@ -43,7 +43,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS 1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`. 2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug. -3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../knowledge/wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults. +3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults. 4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue. 5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA. @@ -51,7 +51,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS | | Before | After | |---|---|---| -| Authentik host | [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` | +| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` | | Version | `2026.2.2` | `2026.5.2` | | `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) | | `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container | @@ -76,7 +76,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted). -Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../knowledge/wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy). +Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy). ### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy @@ -99,11 +99,11 @@ The policy was orphaned (no matched type data or had incompatible evaluation). R - **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`. - **VPS port 22** opened for this repair; close once remote access is otherwise stable. - **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service). -- **Reconcile [124-authentik](../../knowledge/wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page. +- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page. - **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree. ## Related -- [124 — authentik](../../knowledge/wiki/containers/106-auth-outpost.md) -- [DNS split-horizon](../../knowledge/wiki/infrastructure/dns.md) -- [Public ingress (VPS traefik)](../../knowledge/wiki/infrastructure/ingress.md) -- [Mesh migration](../../knowledge/wiki/infrastructure/mesh.md) +- [124 — authentik](../../../wiki/containers/106-auth-outpost.md) +- [DNS split-horizon](../../../wiki/infrastructure/dns.md) +- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md) +- [Mesh migration](../../../wiki/infrastructure/mesh.md) diff --git a/investigations/index.md b/knowledge/sources/investigations/index.md similarity index 100% rename from investigations/index.md rename to knowledge/sources/investigations/index.md diff --git a/knowledge/wiki/containers/102-nfs-export.md b/knowledge/wiki/containers/102-nfs-export.md index 15c69d9..02266e3 100644 --- a/knowledge/wiki/containers/102-nfs-export.md +++ b/knowledge/wiki/containers/102-nfs-export.md @@ -54,7 +54,7 @@ We considered three options before building this: | Option | Outcome | |---|---| -| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../../investigations/index.md)). Rejected. | +| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. | | **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. | | **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. | diff --git a/knowledge/wiki/containers/106-auth-outpost.md b/knowledge/wiki/containers/106-auth-outpost.md index 27ae131..478044c 100644 --- a/knowledge/wiki/containers/106-auth-outpost.md +++ b/knowledge/wiki/containers/106-auth-outpost.md @@ -1,6 +1,6 @@ # 106 — `auth-outpost` -Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../../investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik. +Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik. ## At a glance - **Hostname:** `auth-outpost` @@ -12,7 +12,7 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co ## Role -Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md). +Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md). ## Service / port map | Service | Listen | Notes | @@ -45,12 +45,12 @@ Fix: the LAN outpost gets its **own** domain. - [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only) - [Caddy (121)](121-caddy.md) — forward-auth consumer - [Ingress (VPS traefik)](../infrastructure/ingress.md) -- [Authentik VPS migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md) +- [Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) ## Changelog ### 2026-06-06 — Authentik session lifetime extended to 30 days -VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../../investigations/2026-06-06-authentik-session-lifetime.md). +VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md). ### 2026-06-01 — created; forward-auth cut over from LXC 124 -New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md). +New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md). diff --git a/knowledge/wiki/containers/131-teddycloud.md b/knowledge/wiki/containers/131-teddycloud.md index 518e9cf..e360403 100644 --- a/knowledge/wiki/containers/131-teddycloud.md +++ b/knowledge/wiki/containers/131-teddycloud.md @@ -7,7 +7,7 @@ audio content against a local server instead of the official cloud. Predates the client-enrollment convention entirely; nobody wrote it down. Found and documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged `pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see -[OIKOS.md](../../../OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already +[OIKOS.md](../../../.agents/OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud` container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict for it — but it never got its own inventory entry or doc page until now. @@ -50,7 +50,7 @@ for the first time. - [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already taken by this container - [Containers index](index.md) -- [OIKOS.md](../../../OIKOS.md) — drift detector that caught this +- [OIKOS.md](../../../.agents/OIKOS.md) — drift detector that caught this ## Changelog diff --git a/knowledge/wiki/containers/132-rclone.md b/knowledge/wiki/containers/132-rclone.md index c322a6e..55b89f8 100644 --- a/knowledge/wiki/containers/132-rclone.md +++ b/knowledge/wiki/containers/132-rclone.md @@ -95,7 +95,7 @@ after this set. Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH), rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to -[agent-enrollment.md troubleshooting](../../../operations/agent-enrollment.md#troubleshooting) if it recurs +[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs on future LXC bootstraps. ## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved) diff --git a/knowledge/wiki/hosts/hubris.md b/knowledge/wiki/hosts/hubris.md index 48d91ba..5a063cc 100644 --- a/knowledge/wiki/hosts/hubris.md +++ b/knowledge/wiki/hosts/hubris.md @@ -8,7 +8,7 @@ workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node ## At a glance - **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`) - **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`). -- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). +- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). - **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`. - **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`. - **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP. @@ -113,8 +113,8 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220 - [Media permissions](../infrastructure/media-permissions.md) - [Monitoring](../infrastructure/monitoring.md) - [Backups (disabled)](../infrastructure/backups.md) -- [Operations cheatsheet](../../../operations/commands.md) -- [Investigation: 2026-04-21 crash loop](../../../investigations/archive/2026-04-21-hubris-crash-loop.md) +- [Operations cheatsheet](../../../.agents/operations/commands.md) +- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md) - [strong — Proxmox host](strong.md) ## Changelog @@ -138,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. `/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel. ### 2026-04-29 — relocated to better-ventilated spot -User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation). +User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation). ### 2026-04-28 — Phase 1 WiFi failover Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them. @@ -147,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since. ### 2026-04-23 — SSD cooling + thermal pads installed -Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). +Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). ### 2026-04-22 — drive removal A/B test -Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). +Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). ### 2026-04-22 — `cpu-epp.service` ordering bug fixed Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`. @@ -159,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind ` `60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else. ### 2026-04-21 — `cpu-epp.service` deployed -Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). +Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). diff --git a/knowledge/wiki/hosts/strong.md b/knowledge/wiki/hosts/strong.md index e3df409..f1ce4c2 100644 --- a/knowledge/wiki/hosts/strong.md +++ b/knowledge/wiki/hosts/strong.md @@ -50,13 +50,13 @@ Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet. `bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the 5-min sync timer are live; `homelab whoami` resolves correctly. See - [agent-enrollment.md](../../../operations/agent-enrollment.md). + [agent-enrollment.md](../../../.agents/operations/agent-enrollment.md). - **Age key / secrets:** issued the same day over plain LAN (no Netbird needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at `/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4` recorded in `inventory.yaml`. Not yet a recipient on any actual secret (`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see - ["Granting a secret to a new client"](../../../operations/agent-enrollment.md#granting-a-secret-to-a-new-client). + ["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client). ## Cluster membership @@ -79,7 +79,7 @@ needed going forward. - [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical) - [Network](../infrastructure/network.md) - [SSH access](../infrastructure/ssh-access.md) -- [Agent enrollment](../../../operations/agent-enrollment.md) +- [Agent enrollment](../../../.agents/operations/agent-enrollment.md) ## Changelog diff --git a/knowledge/wiki/infrastructure/auto-deploy.md b/knowledge/wiki/infrastructure/auto-deploy.md index 5774362..00220e7 100644 --- a/knowledge/wiki/infrastructure/auto-deploy.md +++ b/knowledge/wiki/infrastructure/auto-deploy.md @@ -117,7 +117,7 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l - [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these - [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets - [Backups (disabled)](backups.md) -- [Operations cheatsheet](../../../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference +- [Operations cheatsheet](../../../.agents/operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference ## Changelog diff --git a/knowledge/wiki/infrastructure/backups.md b/knowledge/wiki/infrastructure/backups.md index 2d93237..e343fc8 100644 --- a/knowledge/wiki/infrastructure/backups.md +++ b/knowledge/wiki/infrastructure/backups.md @@ -24,7 +24,7 @@ See [132-rclone](../containers/132-rclone.md) for the full design. ## Legacy — restic on external drive (DISABLED 2026-04-22) -Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). +Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). ## Status @@ -36,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver). -**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). +**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). **To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive. @@ -111,7 +111,7 @@ Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again. -After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). +After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). ## Thermal monitoring @@ -122,7 +122,7 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213 - ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) - [Monitoring](monitoring.md) - [Auto-deploy](auto-deploy.md) -- [Investigation: 2026-04-21 crash loop](../../../investigations/archive/2026-04-21-hubris-crash-loop.md) +- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md) ## Changelog @@ -133,7 +133,7 @@ Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclon Initial documentation. Status remains DISABLED. ### 2026-04-22 — DISABLED -Drive removed as the A/B test in the [crash investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. +Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. ### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups. diff --git a/knowledge/wiki/infrastructure/dns.md b/knowledge/wiki/infrastructure/dns.md index af3bb52..b14567e 100644 --- a/knowledge/wiki/infrastructure/dns.md +++ b/knowledge/wiki/infrastructure/dns.md @@ -110,7 +110,7 @@ All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float o Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients. ### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`) -Authentik migrated off LXC 124 onto the VPS (see [investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS). +Authentik migrated off LXC 124 onto the VPS (see [investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS). ### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP) NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful. diff --git a/knowledge/wiki/infrastructure/homelab-context.md b/knowledge/wiki/infrastructure/homelab-context.md index a0b76c9..cafcce5 100644 --- a/knowledge/wiki/infrastructure/homelab-context.md +++ b/knowledge/wiki/infrastructure/homelab-context.md @@ -5,7 +5,7 @@ Code, Hermes Agent, future MCP-capable clients) on every machine in the lab self-locating and able to read the same source of truth. Operational walkthrough for enrolling a new client lives in -[operations/agent-enrollment.md](../../../operations/agent-enrollment.md); this +[operations/agent-enrollment.md](../../../.agents/operations/agent-enrollment.md); this page is the architecture reference. ## What's where @@ -121,7 +121,7 @@ The MCP server and secrets-issuance each have their own clone ## Related -- [Operations: agent enrollment](../../../operations/agent-enrollment.md) — the +- [Operations: agent enrollment](../../../.agents/operations/agent-enrollment.md) — the step-by-step for adding a new client - [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance` pipelines (and the rest of the lab's webhook pipelines) diff --git a/knowledge/wiki/infrastructure/index.md b/knowledge/wiki/infrastructure/index.md index 5e21264..f97ddf3 100644 --- a/knowledge/wiki/infrastructure/index.md +++ b/knowledge/wiki/infrastructure/index.md @@ -62,5 +62,5 @@ are documented in their own pages. Each system below links to its full doc. - [README](../../../README.md) — entry point - [Containers index](../containers/index.md) -- [Operations cheatsheet](../../../operations/commands.md) -- [OIKOS operating model](../../../OIKOS.md) \ No newline at end of file +- [Operations cheatsheet](../../../.agents/operations/commands.md) +- [OIKOS operating model](../../../.agents/OIKOS.md) \ No newline at end of file diff --git a/knowledge/wiki/infrastructure/ingress.md b/knowledge/wiki/infrastructure/ingress.md index ebc359d..08ff186 100644 --- a/knowledge/wiki/infrastructure/ingress.md +++ b/knowledge/wiki/infrastructure/ingress.md @@ -49,7 +49,7 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8 ### `auth.hubris.network` — different pattern (local container, not cert-mirror) -Since 2026-05-31 [Authentik runs on the VPS itself](../../../investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above: +Since 2026-05-31 [Authentik runs on the VPS itself](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above: - Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`. - TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough). @@ -91,7 +91,7 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for ` TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed. ### 2026-05-31 — `auth.hubris.network` now served locally on the VPS -Authentik migrated onto the VPS ([investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack. +Authentik migrated onto the VPS ([investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack. ### 2026-04-28 — wiki entry created Initial documentation. diff --git a/knowledge/wiki/infrastructure/mesh.md b/knowledge/wiki/infrastructure/mesh.md index f39c016..c855bf6 100644 --- a/knowledge/wiki/infrastructure/mesh.md +++ b/knowledge/wiki/infrastructure/mesh.md @@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man ## Changelog ### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below) -The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md). +The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md). ### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik diff --git a/knowledge/wiki/infrastructure/ssh-access.md b/knowledge/wiki/infrastructure/ssh-access.md index fa8ea73..f412a0a 100644 --- a/knowledge/wiki/infrastructure/ssh-access.md +++ b/knowledge/wiki/infrastructure/ssh-access.md @@ -178,7 +178,7 @@ done - [Mesh migration](mesh.md) - [VPS hardening](vps-hardening.md) -- [Agent enrollment](../../../operations/agent-enrollment.md) +- [Agent enrollment](../../../.agents/operations/agent-enrollment.md) - [Homelab CLI](../../../bin/homelab) ## Changelog diff --git a/oikos/cards/service-secrets_issuance.md b/oikos/cards/service-secrets_issuance.md index 6ceb33e..1178cbe 100644 --- a/oikos/cards/service-secrets_issuance.md +++ b/oikos/cards/service-secrets_issuance.md @@ -2,7 +2,7 @@ - backend: host:apps - url: https://secrets.hubris.network/issue -- doc: operations/agent-enrollment.md +- doc: .agents/operations/agent-enrollment.md - config repo: dtoro/Homelab-Docs - risk notes: identity issuance — any change is security-sensitive; key operations are destructive-class diff --git a/plans/2026-07-05-oikos-prometheus-lxc.md b/plans/2026-07-05-oikos-prometheus-lxc.md index d2922ef..baf2385 100644 --- a/plans/2026-07-05-oikos-prometheus-lxc.md +++ b/plans/2026-07-05-oikos-prometheus-lxc.md @@ -8,7 +8,7 @@ at `pct create` time. ## Why -Week-3 reliability layer (see [OIKOS.md](../OIKOS.md)) wants trend +Week-3 reliability layer (see [OIKOS.md](../.agents/OIKOS.md)) wants trend signals — "disk full in ~9 days at current rate", temperature creep — which need a real time-series store. The scheduler ([oikos/scheduler.py](../oikos/scheduler.py)) currently does point-in-time diff --git a/tools/setup-hermes-soul.sh b/tools/setup-hermes-soul.sh index ab2eb44..1d65588 100755 --- a/tools/setup-hermes-soul.sh +++ b/tools/setup-hermes-soul.sh @@ -14,7 +14,7 @@ set -euo pipefail CONTEXT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -HERMES_MD="$CONTEXT_DIR/HERMES.md" +HERMES_MD="$CONTEXT_DIR/.agents/HERMES.md" SOUL_MD="${HOME}/.hermes/SOUL.md" # Colors for output (only when connected to a terminal) From 658dc0f8b2d182fdadb1e3c833c6dffabf912bfd Mon Sep 17 00:00:00 2001 From: dtoro <d.toro.v@pm.me> Date: Mon, 6 Jul 2026 18:16:58 +0200 Subject: [PATCH 7/8] docs: document file naming conventions and clarify structure guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added explicit file-naming rules to page-templates.md so agents know: - Root entry-points: ALL-CAPS (AGENTS.md, README.md) - Containers: <id>-<name>.md (e.g., 101-jellyfin.md) - Infrastructure: lowercase-with-dashes (dns.md, auto-deploy.md) - Plans/investigations: YYYY-MM-DD-slug.md - Skills: lowercase-with-dashes/ folder containing SKILL.md Updated AGENTS.md section 4 (Wiki conventions) to link to page-templates.md and provided quick reference for file naming, page locations, and changelog format. Updated README.md conventions section to mention file naming and link to page-templates.md for the full rules. All agents now have a clear reference chain: 1. AGENTS.md (entry point) → points to conventions 2. page-templates.md (structure) → has file naming + page templates 3. writing-style.md (prose) → has voice, vocabulary, linking rules 4. llm-wiki.md (organization) → has sources/wiki/index/log model Verified: no broken links, all conventions documented, consistency check passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- .agents/shared/page-templates.md | 11 +++++++++++ AGENTS.md | 18 ++++++++++++------ README.md | 1 + 3 files changed, 24 insertions(+), 6 deletions(-) diff --git a/.agents/shared/page-templates.md b/.agents/shared/page-templates.md index 247a444..d02efde 100644 --- a/.agents/shared/page-templates.md +++ b/.agents/shared/page-templates.md @@ -4,6 +4,17 @@ The structural templates for each page type. Prose voice, vocabulary, and cross- in [writing-style.md](writing-style.md); the layer model (sources / wiki / index / log) lives in [llm-wiki.md](llm-wiki.md). +## File naming + +- **Root entry-point files:** ALL-CAPS (e.g. `AGENTS.md`, `README.md`). These are discovery paths — agents and humans find them first. +- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from `inventory.yaml`. +- **Infrastructure / cross-cutting pages:** lowercase-with-dashes (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node. +- **Plans / investigations / time-stamped docs:** `YYYY-MM-DD-<slug>.md` (e.g. `2026-07-05-oikos-prometheus-lxc.md`). Date first, then lowercase slug. +- **Skills / runbooks:** folder name is lowercase-with-dashes (e.g. `client-enrollment/`), contains `SKILL.md` (uppercase). Emphasizes the skill-name for navigation. +- **Indices:** `README.md` or `index.md` (both lowercase). Prefer `README.md` for folders; use `index.md` only when both an intro page and a listing coexist. + +All paths use lowercase letters, numbers, and hyphens (no underscores). This keeps them readable, shell-friendly, and consistent across the repo. + ## Voice Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. Full rules in diff --git a/AGENTS.md b/AGENTS.md index d29e474..07cae3a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,19 +75,25 @@ Grep is fine for browsing or when MCP is unreachable. ## 4. Wiki conventions -- Narrative pages live under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`; +See [page-templates.md](.agents/shared/page-templates.md) for file naming, page +structure, and the tone standard. Quick reference: + +- **File naming:** Root entry-points are ALL-CAPS (`AGENTS.md`); containers use + `<id>-<name>.md`; infrastructure uses lowercase-with-dashes; plans and investigations + use `YYYY-MM-DD-slug.md`; skills are folders with `SKILL.md` inside. +- **Where pages live:** Narrative under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`; incident records under `knowledge/sources/investigations/`; runbook procedures under `.agents/skills/<name>/SKILL.md`; operator reference under `.agents/operations/`; design docs under `plans/`. Cross-link liberally; orphans are bugs. -- Every page ends with a `## Changelog` section, entries in reverse-chrono - order: +- **Changelog format:** Every page ends with a `## Changelog` section, entries in + reverse-chronological order: ### YYYY-MM-DD — short title one or two lines describing what changed and why. -- Investigation files are dated and slugged: `YYYY-MM-DD-slug.md`. -- Live state takes precedence over docs. If you observe a discrepancy, update - the docs *in the same session* (per the same-session update rule). +- **Live state precedence.** If you observe a discrepancy between the docs and + running state, update the docs *in the same session* (per the same-session update + rule in [page-templates.md](.agents/shared/page-templates.md#same-session-update-rule)). ## 5. Acting on the homelab diff --git a/README.md b/README.md index 2735ff9..53cc9b7 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,7 @@ Resolved cases move to [`investigations/archive/`](knowledge/sources/investigati ## Conventions +- **File naming.** Root entry-points are ALL-CAPS; containers use `<id>-<name>.md`; infrastructure pages use lowercase-with-dashes; plans and incident records use `YYYY-MM-DD-slug.md`. See [page-templates.md](.agents/shared/page-templates.md) for the full rules. - **Each node page** ends with a `## Changelog` section. Reverse-chronological. Entry format: ``` ### YYYY-MM-DD — short title From 2ddc1eaa18cccb54d90247f21ad73057fd795f0c Mon Sep 17 00:00:00 2001 From: dtoro <d.toro.v@pm.me> Date: Mon, 6 Jul 2026 18:19:59 +0200 Subject: [PATCH 8/8] =?UTF-8?q?docs:=20correct=20file=20naming=20conventio?= =?UTF-8?q?n=20=E2=80=94=20foundational=20docs=20are=20ALL-CAPS?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous naming guide was incomplete. The actual convention is: **Foundational docs:** ALL-CAPS - Root entry-points: AGENTS.md, README.md (discovery paths) - Agent instruction: .agents/OIKOS.md, .agents/HERMES.md (docs agents read first) - Reference docs: GLOSSARY.md (like classic repo files: LICENSE, CHANGELOG) **Content pages:** lowercase-with-dashes - Containers: <id>-<name>.md (ID from inventory) - Infrastructure: <topic>.md (system description) - Plans/investigations: YYYY-MM-DD-slug.md (date-sorted) - Section indices: README.md (conventional) **Skills:** special pattern - <name>/SKILL.md where <name> is lowercase-with-dashes - SKILL.md filename is always uppercase — signpost for tools and humans Uppercase is reserved for foundational/signpost docs; all paths otherwise use lowercase with hyphens (no underscores). Updated page-templates.md with expanded explanation, and updated AGENTS.md + README.md to reference the corrected convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- .agents/shared/page-templates.md | 25 ++++++++++++++++++------- AGENTS.md | 6 +++--- README.md | 2 +- 3 files changed, 22 insertions(+), 11 deletions(-) diff --git a/.agents/shared/page-templates.md b/.agents/shared/page-templates.md index d02efde..28d6440 100644 --- a/.agents/shared/page-templates.md +++ b/.agents/shared/page-templates.md @@ -6,14 +6,25 @@ in [writing-style.md](writing-style.md); the layer model (sources / wiki / index ## File naming -- **Root entry-point files:** ALL-CAPS (e.g. `AGENTS.md`, `README.md`). These are discovery paths — agents and humans find them first. -- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from `inventory.yaml`. -- **Infrastructure / cross-cutting pages:** lowercase-with-dashes (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node. -- **Plans / investigations / time-stamped docs:** `YYYY-MM-DD-<slug>.md` (e.g. `2026-07-05-oikos-prometheus-lxc.md`). Date first, then lowercase slug. -- **Skills / runbooks:** folder name is lowercase-with-dashes (e.g. `client-enrollment/`), contains `SKILL.md` (uppercase). Emphasizes the skill-name for navigation. -- **Indices:** `README.md` or `index.md` (both lowercase). Prefer `README.md` for folders; use `index.md` only when both an intro page and a listing coexist. +**Foundational / entry-point files:** ALL-CAPS -All paths use lowercase letters, numbers, and hyphens (no underscores). This keeps them readable, shell-friendly, and consistent across the repo. +- **Root level:** `AGENTS.md`, `README.md` — discovery paths for agents and humans. +- **Agent instruction** (under `.agents/`): `OIKOS.md`, `HERMES.md` — foundational docs agents read before acting. +- **Reference docs:** `GLOSSARY.md` — lookup reference (like classic repo conventions: LICENSE, CHANGELOG, GLOSSARY). + +**Content / narrative pages:** lowercase-with-dashes, date-prefixed as needed + +- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from `inventory.yaml`. +- **Infrastructure / cross-cutting pages:** `<topic>.md` (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node. +- **Plans / investigations:** `YYYY-MM-DD-<slug>.md` (e.g. `2026-07-05-oikos-prometheus-lxc.md`). Date-sorted; slug is lowercase. +- **Section indices:** `README.md` (lowercase, conventional). Prefer in folders; `index.md` only if both intro prose and listing coexist. + +**Skills / runbooks:** special case + +- **Folder structure:** `<name>/SKILL.md` where `<name>` is lowercase-with-dashes (e.g. `client-enrollment/SKILL.md`). +- **The filename SKILL.md is always uppercase** — it acts as a signpost so tools and humans instantly recognize it as a skill. + +**General rules:** All paths use lowercase letters, numbers, and hyphens (no underscores). Uppercase is reserved for foundational docs (entry points + instruction) and filenames that signify document type (SKILL.md, GLOSSARY.md, etc.). ## Voice diff --git a/AGENTS.md b/AGENTS.md index 07cae3a..6ad008b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -78,9 +78,9 @@ Grep is fine for browsing or when MCP is unreachable. See [page-templates.md](.agents/shared/page-templates.md) for file naming, page structure, and the tone standard. Quick reference: -- **File naming:** Root entry-points are ALL-CAPS (`AGENTS.md`); containers use - `<id>-<name>.md`; infrastructure uses lowercase-with-dashes; plans and investigations - use `YYYY-MM-DD-slug.md`; skills are folders with `SKILL.md` inside. +- **File naming:** Foundational docs are ALL-CAPS (AGENTS.md, OIKOS.md, GLOSSARY.md); + containers use `<id>-<name>.md`; infrastructure uses lowercase-with-dashes; plans + and investigations use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`. - **Where pages live:** Narrative under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`; incident records under `knowledge/sources/investigations/`; runbook procedures under `.agents/skills/<name>/SKILL.md`; operator reference under `.agents/operations/`; diff --git a/README.md b/README.md index 53cc9b7..9bd425b 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ Resolved cases move to [`investigations/archive/`](knowledge/sources/investigati ## Conventions -- **File naming.** Root entry-points are ALL-CAPS; containers use `<id>-<name>.md`; infrastructure pages use lowercase-with-dashes; plans and incident records use `YYYY-MM-DD-slug.md`. See [page-templates.md](.agents/shared/page-templates.md) for the full rules. +- **File naming.** Foundational docs (entry-points, agent instruction, references) are ALL-CAPS (`AGENTS.md`, `OIKOS.md`, `GLOSSARY.md`); containers use `<id>-<name>.md`; infrastructure pages use lowercase-with-dashes; plans and incidents use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`. See [page-templates.md](.agents/shared/page-templates.md) for the full rules. - **Each node page** ends with a `## Changelog` section. Reverse-chronological. Entry format: ``` ### YYYY-MM-DD — short title