diff --git a/HERMES.md b/.agents/HERMES.md similarity index 96% rename from HERMES.md rename to .agents/HERMES.md index 03fe761..927dd18 100644 --- a/HERMES.md +++ b/.agents/HERMES.md @@ -19,8 +19,8 @@ When in doubt, check `/opt/homelab-context/` first. For the canonical workflows (service health check, config change + deploy, client enrollment, incident investigation, and each node -lifecycle transition), read the matching file in `runbooks/*.md` before -acting. Each runbook carries its risk class, required inputs, the +lifecycle transition), read the matching `.agents/skills//SKILL.md` before +acting. Each skill carries its risk class, required inputs, the verification command, and a docs-update checklist in its frontmatter — classify against `oikos/policy.yaml` using that risk class before any mutation. Don't re-derive topology or the mutation path by grepping the diff --git a/OIKOS.md b/.agents/OIKOS.md similarity index 92% rename from OIKOS.md rename to .agents/OIKOS.md index 2746ecf..5933f3d 100644 --- a/OIKOS.md +++ b/.agents/OIKOS.md @@ -5,9 +5,9 @@ repo. It is not new infrastructure: `inventory.yaml` is the kernel data structure, the `homelab` CLI and MCP server are the syscall surface, and this page defines the rules everything above them follows. -Read this after [AGENTS.md](AGENTS.md). Machine-readable companions: -[oikos/ontology.yaml](oikos/ontology.yaml) (systems model), -[oikos/policy.yaml](oikos/policy.yaml) (risk & approval). +Read this after [AGENTS.md](../AGENTS.md). Machine-readable companions: +[oikos/ontology.yaml](../oikos/ontology.yaml) (systems model), +[oikos/policy.yaml](../oikos/policy.yaml) (risk & approval). ## The kernel loop: OODA @@ -40,17 +40,17 @@ one pass through **Observe → Orient → Decide → Act**: |---|---|---| | Host / Service | topology entities | `inventory.yaml` (+ generated `hosts/*.yaml`) | | Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` | -| Runbook | executable workflow with risk class + verification | `runbooks/` (Week 2) | +| Runbook | executable workflow with risk class + verification | `.agents/skills//SKILL.md` | | Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) | | Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) | | Approval | short-TTL signed grant for a gated action | approval engine (Week 3) | -| Incident | investigation narrative | `investigations/` | +| Incident | investigation narrative | `knowledge/sources/investigations/` | | Plan | design doc for non-trivial work | `plans/` | | Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` | ## Risk classes (enforced, not advisory) -From [oikos/policy.yaml](oikos/policy.yaml): +From [oikos/policy.yaml](../oikos/policy.yaml): - **read_only** — status, logs, docs, inventory. Unattended. - **reversible_low** — restart, cache clear, sync pull. Unattended + ledger. @@ -69,7 +69,7 @@ Eight domains — physical, compute, network, storage, software, identity & access, operations, external — cover everything in the lab; entities are connected by typed edges (`hosts`, `provides`, `mounts`, `stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …) -defined in [oikos/ontology.yaml](oikos/ontology.yaml). Rule of +defined in [oikos/ontology.yaml](../oikos/ontology.yaml). Rule of completeness: **if it can break, be changed, or hold data, it has an entity and edges.** Blast-radius questions ("what breaks if strong goes down?") are graph walks, not doc archaeology. @@ -80,7 +80,7 @@ stored as `state:` in inventory (absent = active). Destroyed nodes live in the `archaeology:` section. Each transition is a runbook checklist; deprecation completes only when inbound edges reach zero. -Generated views: [infrastructure/topology.md](infrastructure/topology.md) +Generated views: [infrastructure/topology.md](../knowledge/wiki/infrastructure/topology.md) (Mermaid, regenerated from inventory) and the live, clickable version at `oikos.hubris.network/graph` once the Console is deployed. @@ -92,7 +92,7 @@ Generated views: [infrastructure/topology.md](infrastructure/topology.md) - Secrets are decrypted locally via per-client keys; never into docs/comments. - Tracked configs change by commit + push, not local edits. - Netbird is the preferred mesh path for new traffic. -- Agents are terse ([CAVEMAN.md](CAVEMAN.md)), verify claims, and fix +- Agents are terse ([caveman.md](shared/caveman.md)), verify claims, and fix collateral drift when found. ## Build status (30-day roadmap, started 2026-07-05) @@ -108,7 +108,7 @@ Generated views: [infrastructure/topology.md](infrastructure/topology.md) Hermes's existing `@dtoro:avispero` send path, not a new bot, see `oikos/approve.py`), daily brief + weekly report (`oikos/report.py`). Shipped, except: Prometheus is still `planned` (see - [plans/2026-07-05-oikos-prometheus-lxc.md](plans/2026-07-05-oikos-prometheus-lxc.md)) — + [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md)) — trend signals (disk-full prediction, temp creep) wait on that LXC; the scheduler's disk check today is point-in-time only, and CPU/NVMe temperature isn't probed at all yet (no confirmed sensor path on @@ -121,7 +121,7 @@ Generated views: [infrastructure/topology.md](infrastructure/topology.md) SPA build chain, tested end-to-end against live production data (see `oikos/console/`). Deploys as a third webhook on `dtoro/Homelab-Docs` (`/opt/oikos-console`, port :9831) — see - [oikos/console/deploy/README.md](oikos/console/deploy/README.md) for + [oikos/console/deploy/README.md](../oikos/console/deploy/README.md) for the Caddy route and Gitea webhook registration this repo can't do for itself. Approval grants are now single-use (a second `check_grant` call for the same request fails even within the TTL) and already exact-bound @@ -174,7 +174,7 @@ guesswork. Roughly ordered by what unblocks the most: (rotated once already to rule out drift). Until fixed, `git push` doesn't auto-redeploy the console the way it does for homelab-mcp/ secrets-issuance; re-run `deploy.sh` on apps manually after changes. - See [oikos/console/deploy/README.md](oikos/console/deploy/README.md). + See [oikos/console/deploy/README.md](../oikos/console/deploy/README.md). - **SSH-key-signed approval requests.** Replaces the design note in Week 4: age keys can't sign (encryption-only format), so per-agent request authentication needs `ssh-keygen -Y sign`/`-Y verify` against @@ -186,7 +186,7 @@ guesswork. Roughly ordered by what unblocks the most: — needs a live Authentik `PromptStage`/reauth flow scoped to that path; not configurable without a running instance to test against. - **Prometheus provisioning** (see - [plans/2026-07-05-oikos-prometheus-lxc.md](plans/2026-07-05-oikos-prometheus-lxc.md)) + [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md)) — unblocks trend signals (disk-full prediction, temp creep) and real sparklines in the Console; investigate the undocumented `pve_id 131` on hubris first. @@ -207,7 +207,7 @@ guesswork. Roughly ordered by what unblocks the most: populate more as specific services turn out to need non-default risk classes. - **Incident timeline generator** — stitch ledger + signal history into - a single narrative for `investigations/` entries instead of writing + a single narrative for `knowledge/sources/investigations/` entries instead of writing them by hand. - **Secret access audit** — who-can-decrypt-what report from `.sops.yaml` + inventory `age_pubkey`s, extending what diff --git a/.agents/domains/knowledge/schema.md b/.agents/domains/knowledge/schema.md new file mode 100644 index 0000000..da74682 --- /dev/null +++ b/.agents/domains/knowledge/schema.md @@ -0,0 +1,48 @@ +# Knowledge domain — schema + +The knowledge domain is the durable, authoritative current-state documentation of the homelab: one +page per node and per cross-cutting system, synthesized from live state and evidence. It answers +"what exists and how does it work right now." + +It follows the [LLM Wiki layer model](../../shared/llm-wiki.md) and the +[writing-style](../../shared/writing-style.md) and [page-templates](../../shared/page-templates.md) +rules. + +## The narrative / substrate split + +The knowledge wiki is **narrative**. It sits alongside a **machine-readable substrate** that it +describes but never contains. The split is load-bearing: several programs read the substrate at +fixed paths, so the wiki reorganization never moves it. + +| Layer | Location | Consumed by | +|-------|----------|-------------| +| Substrate — source of truth | `inventory.yaml` (root) | MCP server, `homelab` CLI, `oikos/` scheduler/drift/relations/gen-topology | +| Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` | +| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler | +| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` | +| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages | + +## Wiki pages + +- **Node pages** (`knowledge/wiki/containers/-.md`, `.../vms/-.md`, + `.../hosts/.md`) follow the container/host template in + [page-templates.md](../../shared/page-templates.md): opening definition, `## At a glance`, + `## Role`, service/port map, storage, auto-deploy, `## Related`, `## Changelog`. +- **Cross-cutting pages** (`knowledge/wiki/infrastructure/.md`) follow the cross-cutting + template: `## Why`, `## Components`, `## How to apply`, `## Gotchas`, `## Related`, `## Changelog`. +- Each `inventory.yaml` host entry carries a `doc_page:` field pointing at its narrative page. + Changing where a page lives means updating that field (read by `bin/homelab`). + +## The two logs + +- The per-page **`## Changelog`** records infrastructure changes and is machine-parsed + (`get_changelog`, the Oikos ledger). Keep the `### YYYY-MM-DD — title` shape. +- **`knowledge/log.md`** is append-only and records *documentation-maintenance* operations only + (restructures, source ingests, lint sweeps): `## [YYYY-MM-DD] | `. It never + duplicates the Oikos change ledger (`oikos/ledger.py`). + +## Same-session update rule + +A change to a node updates every page that references it in the same session — the node page, the +section `README.md` table, the root `README.md`, the Caddy/DNS/ingress pages, the host page, and +`inventory.yaml`. See [page-templates.md](../../shared/page-templates.md#same-session-update-rule). diff --git a/.agents/domains/operations/schema.md b/.agents/domains/operations/schema.md new file mode 100644 index 0000000..05c60e5 --- /dev/null +++ b/.agents/domains/operations/schema.md @@ -0,0 +1,55 @@ +# Operations domain — schema + +The operations domain holds the procedural and time-stamped documentation: runbooks (repeatable +procedures), investigations (incident evidence), and plans (design docs for non-trivial work). It +follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice +exception. + +Where each kind lives: runbooks are skills under [`.agents/skills/`](../../skills/); operator +reference (command cheatsheet, enrollment, Hermes agent) lives in +[`.agents/operations/`](../../operations/); investigations are sources under +`knowledge/sources/investigations/`; plans stay in the repo-root `plans/` folder (below). + +## Plans always live in `plans/` + +**Any plan or design doc for the Homelab is written into the repo `plans/` folder as +`plans/YYYY-MM-DD-slug.md` — never a scratch path, an agent-private plan location, or a chat +message.** An agent drafting a plan: + +1. Writes the file under `plans/` using the plan template in [page-templates.md](../../shared/page-templates.md). +2. Lists it in `plans/index.md`. +3. On completion, moves it to `plans/done/` and updates the index status. + +This is the single source for homelab design intent; keeping it in-repo means the plan is +versioned, reviewable, and reachable by MCP `get_page`/`search_docs` like any other doc. + +## Runbooks + +Repeatable procedures are skills — one folder per skill at `.agents/skills//SKILL.md`, with +YAML front-matter that the Oikos policy and lifecycle machinery reads: + +```yaml +--- +name: +risk_class: read_only | reversible_low | config_mutation | destructive +inputs: [, ...] +verification: "" +docs_update_checklist: [] +transition: " -> " # only for lifecycle runbooks +--- +``` + +`risk_class` values and the lifecycle `transition` states must match +[`oikos/policy.yaml`](../../../oikos/policy.yaml) and [`oikos/ontology.yaml`](../../../oikos/ontology.yaml). + +## Investigations + +Incident records live in `knowledge/sources/investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written +once at incident time, then linked from the changelogs of the nodes they implicate. Sections: +`## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved +incidents move to `knowledge/sources/investigations/archive/`. + +## The operations log + +`plans/log.md` and `knowledge/log.md` are append-only records of documentation operations on +those areas (`## [YYYY-MM-DD] | `), distinct from the Oikos change ledger. diff --git a/operations/agent-enrollment.md b/.agents/operations/agent-enrollment.md similarity index 96% rename from operations/agent-enrollment.md rename to .agents/operations/agent-enrollment.md index a615c00..d4174ef 100644 --- a/operations/agent-enrollment.md +++ b/.agents/operations/agent-enrollment.md @@ -6,7 +6,7 @@ this repo that auto-syncs every 5 min, a per-client age key for SOPS decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config. > Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment? -> See [hermes-agent.md](./hermes-agent.md). It uses the same `bootstrap.sh` +> See [hermes-agent.md](hermes-agent.md). It uses the same `bootstrap.sh` > with an additional `--with-hermes` flag. Architecture in [project_homelab_context_plan](https://… memory link); the @@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight. -Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../containers/124-authentik.md). Only needed once per account. +Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account. **Path A — setup-key (headless/scripted onboarding):** @@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr ### DNS prerequisite `*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124 -([dns.md](../infrastructure/dns.md)) for LAN clients, **but only if the +([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the client uses 192.168.8.180 as its resolver**. Most LXCs and roaming workstations don't by default. Options: @@ -261,7 +261,7 @@ arguments. If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be visible to the classifier (it's not actually durable in 0.71.2, but the -ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md) +ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) for context), drop a free-text rule into `autoMode.allow` describing the authorization. Optional. @@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually: | `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. | | `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. | | Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with ` ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). | -| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). | +| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). | | `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. | | `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh ` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). | | `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. | @@ -356,7 +356,7 @@ and LAN IP registration. New workstations enrolled via this doc will automatically join the universal SSH mesh. ### 2026-05-31 — cross-link to hermes-agent.md -Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](./hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here. +Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here. ### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper. diff --git a/operations/commands.md b/.agents/operations/commands.md similarity index 74% rename from operations/commands.md rename to .agents/operations/commands.md index d2821d3..1dbc137 100644 --- a/operations/commands.md +++ b/.agents/operations/commands.md @@ -1,6 +1,6 @@ # Operations cheatsheet -Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`. +Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`. ## Proxmox CLI @@ -8,13 +8,13 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root | --- | --- | | `pct list` / `qm list` | List LXC containers / VMs | | `pct config ` / `qm config ` | Container / VM config | -| `pct exec -- ` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../infrastructure/media-permissions.md)) | +| `pct exec -- ` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) | | `pct enter ` | Shell into a container | | `pct start ` / `pct stop ` | Boot / halt a container | | `pvesm status` | Storage pools status | | `pvesh get /nodes --output-format json` | Node summary as JSON | | `pvesh get /nodes/hubris/lxc//status/current` | Live container status | -| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) | +| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) | | `pveversion` | PVE version | | `journalctl -u pve-cluster -n 100` | PVE service logs | @@ -22,22 +22,22 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root - Shared mount: `/mnt/library` (ext4 on lvmthin `library`). - Bind into a container: `pct set -mp /mnt/library/,mp=/data` -- For the standard whole-tree mount: `pct set -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../infrastructure/media-permissions.md) for the GID-10000 onboarding recipe. +- For the standard whole-tree mount: `pct set -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe. ## Reverse proxy -- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md). -- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it. +- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md). +- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it. - Hot reload: `pct exec 121 -- systemctl reload caddy`. - Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`. - Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`. ## DNS -- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.) +- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.) - Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes. - Verify: `dig @192.168.8.2 +short .hubris.network`. -- See [DNS](../infrastructure/dns.md). +- See [DNS](../../knowledge/wiki/infrastructure/dns.md). ## Web access @@ -49,7 +49,7 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root - `ras-mc-ctl --errors` — full event log - `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power` - `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave` -- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md)) +- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md)) ## Fleet apt operations @@ -80,12 +80,12 @@ See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference: | `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) | | `homelab restart [--approval-id ]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant | -Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../oikos/console/deploy/README.md). +Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md). ## Related -- [Hubris host](../hosts/hubris.md) -- [Containers index](../containers/index.md) -- [DNS](../infrastructure/dns.md) -- [Monitoring](../infrastructure/monitoring.md) -- [Auto-deploy](../infrastructure/auto-deploy.md) -- [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction +- [Hubris host](../../knowledge/wiki/hosts/hubris.md) +- [Containers index](../../knowledge/wiki/containers/index.md) +- [DNS](../../knowledge/wiki/infrastructure/dns.md) +- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md) +- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md) +- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction diff --git a/operations/hermes-agent.md b/.agents/operations/hermes-agent.md similarity index 96% rename from operations/hermes-agent.md rename to .agents/operations/hermes-agent.md index 2f87838..5c3736d 100644 --- a/operations/hermes-agent.md +++ b/.agents/operations/hermes-agent.md @@ -2,7 +2,7 @@ Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned Llama variant) as a working terminal agent on a homelab client. Builds on top -of standard client enrollment (see [agent-enrollment.md](./agent-enrollment.md)) +of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md)) — this page covers only the Hermes-specific additions. The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides: @@ -21,7 +21,7 @@ The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global | Requirement | How | | --- | --- | -| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](./agent-enrollment.md) | +| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) | | `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below | | The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add --finalize-pubkey --with-hermes` | @@ -165,13 +165,13 @@ every tool call, use `approve`. See ## Cross-references -- [agent-enrollment.md](./agent-enrollment.md) — base client onboarding the +- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the Hermes flow assumes is done. - [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every session start (via `~/.config/goose/.goosehints`). -- [`bin/hermes`](../bin/hermes) — the wrapper that decrypts the OpenRouter key +- [`bin/hermes`](../../bin/hermes) — the wrapper that decrypts the OpenRouter key and execs `goose session`. -- [`bootstrap.sh`](../bootstrap.sh) — the `--with-hermes` flag's install block. +- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-hermes` flag's install block. ## Follow-ups diff --git a/CAVEMAN.md b/.agents/shared/caveman.md similarity index 100% rename from CAVEMAN.md rename to .agents/shared/caveman.md diff --git a/.agents/shared/llm-wiki.md b/.agents/shared/llm-wiki.md new file mode 100644 index 0000000..6101caa --- /dev/null +++ b/.agents/shared/llm-wiki.md @@ -0,0 +1,41 @@ +# LLM Wiki — the documentation contract + +How the narrative documentation in this repo is organized. The pattern is borrowed from the +`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top +of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an +append-only operations log. + +This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`, +generated `hosts/*.yaml`, `oikos/`, `mcp/`, `secrets/`, `bin/` — is not part of the wiki and never +moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for the split. + +## Layers + +- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference + docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`). + Read them; do not rewrite them into other sources. +- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per + node (`containers/`, `vms/`, host narratives) and per cross-cutting system (`infrastructure/`). A + reader understands the topic from the wiki page without reading the sources. +- **Index** (`index.md` / folder `README.md`) is a pure listing — every page in scope with a + one-line summary, and nothing else. Anything the section wants to say up front goes into a page + the index lists, not into the index. +- **Log** (`log.md`) is append-only, recording *doc-maintenance operations* (restructures, source + ingests, lint sweeps) in single-line format: `## [YYYY-MM-DD] | `. + +## Two logs, kept distinct + +- **`## Changelog`** on each node/topic page records *infrastructure* changes to that node. It is + machine-parsed (`get_changelog`, the Oikos ledger) — keep the `### YYYY-MM-DD — title` shape. +- **`log.md`** per area records *documentation* operations only. It never duplicates the Oikos + change ledger (`oikos/ledger.py`), which stays authoritative for infra changes with + who/what/risk/approval/verification. + +## Rules + +- Wiki pages stay short and focused. A page past ~300 lines splits. +- Pages stay flat under `wiki/
/` until there are enough to warrant a sub-group. +- Every page follows [writing-style.md](writing-style.md). +- Plans and design docs always live in the repo `plans/` folder (`plans/YYYY-MM-DD-slug.md`), + listed in `plans/index.md`, moved to `plans/done/` on completion — never a scratch path or a chat + message. See [the operations schema](../domains/operations/schema.md). diff --git a/CONTRIBUTING.md b/.agents/shared/page-templates.md similarity index 66% rename from CONTRIBUTING.md rename to .agents/shared/page-templates.md index 7706538..28d6440 100644 --- a/CONTRIBUTING.md +++ b/.agents/shared/page-templates.md @@ -1,8 +1,35 @@ -# Contributing to the Homelab Wiki +# Page templates for the Homelab Wiki + +The structural templates for each page type. Prose voice, vocabulary, and cross-reference rules live +in [writing-style.md](writing-style.md); the layer model (sources / wiki / index / log) lives in +[llm-wiki.md](llm-wiki.md). + +## File naming + +**Foundational / entry-point files:** ALL-CAPS + +- **Root level:** `AGENTS.md`, `README.md` — discovery paths for agents and humans. +- **Agent instruction** (under `.agents/`): `OIKOS.md`, `HERMES.md` — foundational docs agents read before acting. +- **Reference docs:** `GLOSSARY.md` — lookup reference (like classic repo conventions: LICENSE, CHANGELOG, GLOSSARY). + +**Content / narrative pages:** lowercase-with-dashes, date-prefixed as needed + +- **Container pages:** `-.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `` is the LXC/VM ordinal from `inventory.yaml`. +- **Infrastructure / cross-cutting pages:** `.md` (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node. +- **Plans / investigations:** `YYYY-MM-DD-.md` (e.g. `2026-07-05-oikos-prometheus-lxc.md`). Date-sorted; slug is lowercase. +- **Section indices:** `README.md` (lowercase, conventional). Prefer in folders; `index.md` only if both intro prose and listing coexist. + +**Skills / runbooks:** special case + +- **Folder structure:** `/SKILL.md` where `` is lowercase-with-dashes (e.g. `client-enrollment/SKILL.md`). +- **The filename SKILL.md is always uppercase** — it acts as a signpost so tools and humans instantly recognize it as a skill. + +**General rules:** All paths use lowercase letters, numbers, and hyphens (no underscores). Uppercase is reserved for foundational docs (entry points + instruction) and filenames that signify document type (SKILL.md, GLOSSARY.md, etc.). ## Voice -Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. +Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. Full rules in +[writing-style.md](writing-style.md). ## Page templates @@ -91,7 +118,7 @@ What it looks like after. Changelog entries to write, index status to update. ``` -### Investigation (`investigations/YYYY-MM-DD-slug.md`) +### Investigation (`knowledge/sources/investigations/YYYY-MM-DD-slug.md`) ```markdown # YYYY-MM-DD — diff --git a/.agents/shared/writing-style.md b/.agents/shared/writing-style.md new file mode 100644 index 0000000..95da2d6 --- /dev/null +++ b/.agents/shared/writing-style.md @@ -0,0 +1,75 @@ +# Writing Style + +Write like a technical reference, not a marketing page. Every sentence conveys new information. +These rules govern **committed documentation** — wiki pages, READMEs, schemas, skills, `AGENTS.md`, +plans, investigations, and code comments. They are separate from [caveman.md](caveman.md), which +governs an agent's *chat responses*; the two do not conflict. + +New or rewritten pages follow these patterns from day one. Existing pages get updated the next time +they are touched. + +## Vocabulary — never use these + +- Significance puffers: "pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament", "paramount", "invaluable". +- Analytical verbs: "delve", "leverage", "utilize", "facilitate", "foster", "showcase", "underscore", "streamline", "harness". +- Poetic nouns: "tapestry", "landscape" (figurative), "realm", "paradigm", "ecosystem" (figurative), "journey" (figurative), "nexus", "cornerstone". +- Promotional adjectives: "robust", "seamless", "innovative", "cutting-edge", "meticulous", "holistic", "comprehensive". +- Opening crutches: "In today's world", "In the ever-evolving landscape of", "It's worth noting that", "It is important to note that". + +Use short, common words: "use" not "utilize", "help" not "facilitate", "show" not "demonstrate". + +## Voice + +Describe what systems do and how they work. + +- **Reference prose** (node pages, cross-cutting infrastructure descriptions, `## Role`, `## Why`, + `At a glance`) is third-person: state facts about the system, not instructions to a reader. +- **Recipes, runbooks, and skills** are the exception: second-person imperative is allowed and + preferred where it makes a procedure clearer ("Edit the Caddyfile, commit + push", "Verify with + `dig +short`"). This matches how the operator actually works. The vocabulary, structure, and + cross-reference rules below still apply. + +## Page shape + +Every doc-level page follows the same shape so a reader scans it in one pass. + +1. **One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`. +2. **Opening definition.** First paragraph, 1–3 sentences, says what the thing is. No motivation, no marketing, no setup. +3. **Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md). +4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (`get_changelog` in `mcp/server.py`); keep the `### YYYY-MM-DD — title` shape. +5. **Related links** only at the bottom, only when a reference cannot be woven inline. + +## Section indexes (folder READMEs) + +A folder's `README.md` opens with a 1–3 sentence prose intro that says what the section covers, then +a single navigation table — `| Document | What it covers |` — and nothing else. No stale counts, no +duplicated prose, no narrative between the intro and the table. + +## Structure rules + +- Make every sentence information-dense. Cut filler, qualifiers, and setup phrases. Lead with the concrete fact or action, not why it matters. +- No participial tack-ons (", highlighting the importance of…"). If the clause adds information, make it a separate sentence. +- **No meta-commentary about the content itself.** Do not narrate the page's own structure or linking strategy. +- Prefer **tables** for enumerable items with internal structure (service/port maps, field lists, status grids). Reserve bullets for short non-structured lists. +- Use the **bold-leading-phrase pattern** for structured points: `**Read-only by construction.** The MCP server never mutates state.` — a bold noun phrase, a period, then the explanation. +- When enumerating across services or nodes, give each its own `###` sub-section or a table row, not one run-on paragraph. +- Use backticks for code, paths, hostnames, and file names (`inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology. +- Use `>` blockquotes for caveats and gaps that interrupt the main flow: `> **Outstanding gap.** DNS-vs-inventory drift check not yet wired.` One thought per blockquote. + +## Diagrams + +- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it. +- ASCII box diagrams are fine for small shape diagrams; keep them to one screen. + +## Sourcing and cross-references + +- **Factual discipline.** Every claim is grounded in a cited source, an adjacent linked page, or a directly observable fact (`pct config`, `docker inspect`, running config). Do not write sentences that sound sourced but are inference. When docs disagree with live state, fix the doc and note it in the changelog. +- **One-sided cross-references.** When two pages relate, the link lives in the page where the connection makes organizational sense. Do not add a back-pointer unless that direction also carries content the reader needs. +- **Cross-references are content, not catalog.** Inline links arise from the surrounding prose; the linked page must be needed to understand the current sentence. A bottom-of-page "Related" list is the fallback, not the default. +- Pages link with standard relative markdown links (e.g. a container page links to `../infrastructure/dns.md`), forming a navigable graph. Orphans are a bug. + +## Code comments and commit/PR prose + +- Comments explain intent, trade-offs, or constraints the code cannot convey. No diff narration, no type restatement, no section-divider comments. +- Commit messages and PR descriptions are problem → change → risk → verification, not a file-by-file diff restatement. +- The banned vocabulary applies the same way in comments and commit messages. diff --git a/runbooks/client-enrollment.md b/.agents/skills/client-enrollment/SKILL.md similarity index 92% rename from runbooks/client-enrollment.md rename to .agents/skills/client-enrollment/SKILL.md index 7de5e4a..043f233 100644 --- a/runbooks/client-enrollment.md +++ b/.agents/skills/client-enrollment/SKILL.md @@ -11,12 +11,12 @@ docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm] Goal: bring a new host (workstation, LXC, VM) into inventory and the secrets model, with mesh membership only where it's actually needed. This wraps the existing `homelab client add` flow — see -[operations/agent-enrollment.md](../operations/agent-enrollment.md) for +[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for the full walkthrough; this runbook is the risk/lifecycle framing. 1. On any enrolled client: `homelab client add <hostname>` — appends a `hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned` - → `provisioning`, per [oikos/ontology.yaml](../oikos/ontology.yaml)), + → `provisioning`, per [oikos/ontology.yaml](../../../oikos/ontology.yaml)), commits + pushes. 2. Netbird join is **optional, not a required step** — only needed for hosts that must be reachable off-LAN (workstations that roam, e.g. diff --git a/runbooks/config-change-deploy.md b/.agents/skills/config-change-deploy/SKILL.md similarity index 91% rename from runbooks/config-change-deploy.md rename to .agents/skills/config-change-deploy/SKILL.md index 8767f47..0f83139 100644 --- a/runbooks/config-change-deploy.md +++ b/.agents/skills/config-change-deploy/SKILL.md @@ -18,10 +18,10 @@ own repo) and get it live, safely. `oikos/policy.yaml`. 2. Clone/pull the `config_repo` (never edit the backend's working tree directly — tracked configs change by commit + push, per - [OIKOS.md](../OIKOS.md) conventions). + [OIKOS.md](../../OIKOS.md) conventions). 3. Make the change, commit, push to `main`. 4. The Gitea webhook fires the deploy pipeline for that repo (see - [infrastructure/auto-deploy.md](../infrastructure/auto-deploy.md) for + [infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for the exact receiver/reload for this service). 5. Run the preflight's verification command. If it fails, check `homelab service <name> log` for the reload/restart error. diff --git a/.agents/skills/docs-lint/SKILL.md b/.agents/skills/docs-lint/SKILL.md new file mode 100644 index 0000000..b1f296c --- /dev/null +++ b/.agents/skills/docs-lint/SKILL.md @@ -0,0 +1,25 @@ +--- +name: docs-lint +risk_class: read_only +inputs: [paths] +verification: "python3 .agents/skills/docs-lint/lint.py" +docs_update_checklist: [] +--- + +# Docs lint + +Check committed documentation against the mechanical rules in +[writing-style.md](../../shared/writing-style.md): banned vocabulary and broken relative markdown +links. Prose-voice rules are not machine-checkable — those stay a review responsibility. + +Run from the repo root: + + python3 .agents/skills/docs-lint/lint.py # default: knowledge/ .agents/ operations/ investigations/ plans/ + python3 .agents/skills/docs-lint/lint.py knowledge/wiki/containers/104-gitea.md + +Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary +list mirrors `writing-style.md`; update both together if the standard changes. + +> **Known baseline.** `knowledge/wiki/containers/101-jellyfin.md` links into a sibling repo +> (`devops/homelab-authentik-admin`) that this checkout does not contain — expected, not a bug. +> Any other broken link is a real regression; investigate before dismissing it as baseline noise. diff --git a/.agents/skills/docs-lint/lint.py b/.agents/skills/docs-lint/lint.py new file mode 100644 index 0000000..dc1261d --- /dev/null +++ b/.agents/skills/docs-lint/lint.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +"""Lint committed docs against .agents/shared/writing-style.md. + +Checks two mechanical rules: + 1. Banned vocabulary (significance puffers, analytical verbs, poetic nouns, + promotional adjectives, opening crutches). + 2. Broken relative markdown links. + +Prose-voice rules are not machine-checkable; this covers the parts that are. +Run from the repo root: python3 .agents/skills/docs-lint/lint.py [paths...] +Exit 1 if any violation is found. +""" +import os, re, sys + +BANNED = [ + "pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament", + "paramount", "invaluable", "delve", "leverage", "utilize", "facilitate", "foster", + "showcase", "underscore", "streamline", "harness", "tapestry", "realm", "paradigm", + "nexus", "cornerstone", "robust", "seamless", "innovative", "cutting-edge", + "meticulous", "holistic", "comprehensive", "in today's world", + "it's worth noting", "it is important to note", +] +BAN_RE = re.compile(r'(?<![\w-])(' + "|".join(re.escape(w) for w in BANNED) + r')(?![\w-])', re.I) +LINK = re.compile(r'\]\(([^)]+)\)') + +def iter_md(paths): + for p in paths: + if os.path.isfile(p) and p.endswith(".md"): + yield p + for root, dirs, files in os.walk(p): + dirs[:] = [d for d in dirs if d not in (".git", "node_modules")] + for f in files: + if f.endswith(".md"): + yield os.path.join(root, f) + +def main(argv): + paths = argv or ["knowledge", ".agents", "operations", "investigations", "plans"] + violations = 0 + # The style guide and this skill enumerate the banned words by definition. + ban_exempt = ("shared/writing-style.md", "skills/docs-lint/") + for f in sorted(set(iter_md(paths))): + check_banned = not any(x in f for x in ban_exempt) + fence = False + with open(f) as fh: + for ln, line in enumerate(fh, 1): + if line.lstrip().startswith("```"): + fence = not fence; continue + if fence: + continue + if check_banned: + for m in BAN_RE.finditer(line): + print(f"{f}:{ln}: banned word '{m.group(1)}'") + violations += 1 + for m in LINK.finditer(line): + link = m.group(1) + if re.match(r'^(https?:|mailto:|#|/)', link): + continue + path = re.split(r'[#?]', link)[0] + if not path: + continue + tgt = os.path.normpath(os.path.join(os.path.dirname(f), path)) + if not os.path.exists(tgt): + print(f"{f}:{ln}: broken link -> {link}") + violations += 1 + print(f"\n{violations} violation(s)") + return 1 if violations else 0 + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/runbooks/incident-investigation.md b/.agents/skills/incident-investigation/SKILL.md similarity index 94% rename from runbooks/incident-investigation.md rename to .agents/skills/incident-investigation/SKILL.md index 3d8a36f..2858192 100644 --- a/runbooks/incident-investigation.md +++ b/.agents/skills/incident-investigation/SKILL.md @@ -23,7 +23,7 @@ Goal: understand what broke and why, before touching anything. 5. Check the change ledger for recent mutations to the affected entity or anything upstream of it: `homelab service <name> history` (once populated) or grep `ledger/*.jsonl`. -6. Write findings to a new `investigations/<date>-<slug>.md` — symptom, +6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom, timeline, root cause, fix applied, prevention. This is the durable record; don't rely on chat history. diff --git a/runbooks/lifecycle-activate-node.md b/.agents/skills/lifecycle-activate-node/SKILL.md similarity index 90% rename from runbooks/lifecycle-activate-node.md rename to .agents/skills/lifecycle-activate-node/SKILL.md index dd3772b..bc04a0d 100644 --- a/runbooks/lifecycle-activate-node.md +++ b/.agents/skills/lifecycle-activate-node/SKILL.md @@ -9,13 +9,13 @@ transition: "provisioning -> active" # Lifecycle: activate a node -Per [oikos/ontology.yaml](../oikos/ontology.yaml). Requires: age key +Per [oikos/ontology.yaml](../../../oikos/ontology.yaml). Requires: age key enrolled if it needs secrets, mesh joined if it needs off-LAN reach, ingress live if public, health check answering, doc page complete, ledger entry. 1. If the node is a `homelab` client: finish enrollment per - [client-enrollment.md](client-enrollment.md) (`--finalize-pubkey`, + [client-enrollment.md](../client-enrollment/SKILL.md) (`--finalize-pubkey`, mesh join, `homelab doctor` green). 2. If it hosts a public service: add the `services:` entry in `inventory.yaml` (backend, url, doc_page, config_repo, risk_notes — diff --git a/runbooks/lifecycle-deprecate-node.md b/.agents/skills/lifecycle-deprecate-node/SKILL.md similarity index 91% rename from runbooks/lifecycle-deprecate-node.md rename to .agents/skills/lifecycle-deprecate-node/SKILL.md index b667494..af4c15c 100644 --- a/runbooks/lifecycle-deprecate-node.md +++ b/.agents/skills/lifecycle-deprecate-node/SKILL.md @@ -9,7 +9,7 @@ transition: "active -> deprecated" # Lifecycle: deprecate a node -Per [oikos/ontology.yaml](../oikos/ontology.yaml): a node keeps running +Per [oikos/ontology.yaml](../../../oikos/ontology.yaml): a node keeps running but takes no new dependents. **Completion condition: zero remaining inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a suggestion; `oikos/policy.yaml` `lifecycle_overrides.deprecated.refuse` @@ -32,4 +32,4 @@ before proceeding — that's exactly the kind of drift the Week-3 detector will catch automatically, but until then this manual check is the gate. Next (once `affected_by` is empty): -[lifecycle-destroy-node.md](lifecycle-destroy-node.md). +[lifecycle-destroy-node.md](../lifecycle-destroy-node/SKILL.md). diff --git a/runbooks/lifecycle-destroy-node.md b/.agents/skills/lifecycle-destroy-node/SKILL.md similarity index 100% rename from runbooks/lifecycle-destroy-node.md rename to .agents/skills/lifecycle-destroy-node/SKILL.md diff --git a/runbooks/lifecycle-migrate-node.md b/.agents/skills/lifecycle-migrate-node/SKILL.md similarity index 95% rename from runbooks/lifecycle-migrate-node.md rename to .agents/skills/lifecycle-migrate-node/SKILL.md index da1c82a..b4a29e6 100644 --- a/runbooks/lifecycle-migrate-node.md +++ b/.agents/skills/lifecycle-migrate-node/SKILL.md @@ -10,7 +10,7 @@ transition: "active -> migrating -> active" # Lifecycle: migrate a node Modeled on the strong Phase 1+2 migration -([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)). +([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)). Requires (ontology): preflight + backup-verified before migrating; post-verify + Caddy backends checked + mounts checked + docs updated before returning to `active`. diff --git a/runbooks/lifecycle-provision-node.md b/.agents/skills/lifecycle-provision-node/SKILL.md similarity index 89% rename from runbooks/lifecycle-provision-node.md rename to .agents/skills/lifecycle-provision-node/SKILL.md index 1f0fb6b..d12ecdb 100644 --- a/runbooks/lifecycle-provision-node.md +++ b/.agents/skills/lifecycle-provision-node/SKILL.md @@ -9,7 +9,7 @@ transition: "planned -> provisioning" # Lifecycle: provision a node -Per [oikos/ontology.yaml](../oikos/ontology.yaml) `lifecycle.transitions`. +Per [oikos/ontology.yaml](../../../oikos/ontology.yaml) `lifecycle.transitions`. Policy note: `provisioning` nodes get a lifecycle override — `config_mutation` actions downgrade to `reversible_low` because nothing depends on the node yet (see `oikos/policy.yaml` `lifecycle_overrides`). @@ -30,4 +30,4 @@ chosen, doc page stub. is enough to satisfy the transition requirement. 4. Reserve the IP in DNS/DHCP notes if it's a fixed LAN address. -Next: [lifecycle-activate-node.md](lifecycle-activate-node.md). +Next: [lifecycle-activate-node.md](../lifecycle-activate-node/SKILL.md). diff --git a/runbooks/runbook-budget-from-csv.md b/.agents/skills/runbook-budget-from-csv/SKILL.md similarity index 100% rename from runbooks/runbook-budget-from-csv.md rename to .agents/skills/runbook-budget-from-csv/SKILL.md diff --git a/runbooks/runbook-dpkg-interrupted.md b/.agents/skills/runbook-dpkg-interrupted/SKILL.md similarity index 95% rename from runbooks/runbook-dpkg-interrupted.md rename to .agents/skills/runbook-dpkg-interrupted/SKILL.md index 3e71f15..071c58e 100644 --- a/runbooks/runbook-dpkg-interrupted.md +++ b/.agents/skills/runbook-dpkg-interrupted/SKILL.md @@ -102,9 +102,9 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress. ## Related -- [Operations cheatsheet](commands.md) -- [Auto-deploy pipelines](../infrastructure/auto-deploy.md) -- [Hubris host page](../hosts/hubris.md) +- [Operations cheatsheet](../../operations/commands.md) +- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md) +- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md) ## Changelog diff --git a/runbooks/service-health-check.md b/.agents/skills/service-health-check/SKILL.md similarity index 100% rename from runbooks/service-health-check.md rename to .agents/skills/service-health-check/SKILL.md diff --git a/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md b/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md index 3d0d9b6..c6def9c 100644 --- a/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md +++ b/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md @@ -12,7 +12,7 @@ ## Problem statement -The Technitium DHCP server on [CT 107](containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:** +The Technitium DHCP server on [CT 107](../../knowledge/wiki/containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:** | Host | IP | Inside pool? | |---|---|---| diff --git a/AGENTS.md b/AGENTS.md index c9bfd93..6ad008b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,10 +5,19 @@ context is in this checkout at `/opt/homelab-context/`. This file is the entry point. Read it once at start, then keep working. The operating model — OODA loop, risk classes, approval rules, the ontology, -and node lifecycle — is defined in [OIKOS.md](OIKOS.md). Before any mutation, +and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation, classify the action against `oikos/policy.yaml`; when the class requires approval, stop and ask the operator. +Agent-facing instruction is separated from human content under `.agents/`: +`.agents/shared/` holds the conventions every agent applies +([writing-style](.agents/shared/writing-style.md), [caveman](.agents/shared/caveman.md), +[page-templates](.agents/shared/page-templates.md), [llm-wiki](.agents/shared/llm-wiki.md)), and +`.agents/domains/` holds the per-domain schemas +([knowledge](.agents/domains/knowledge/schema.md), [operations](.agents/domains/operations/schema.md)). +The narrative wiki lives under `knowledge/wiki/`; the machine-readable substrate +(`inventory.yaml`, `hosts/*.yaml`, `oikos/`) stays at the repo root. + ## 1. Who you are Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read: @@ -24,12 +33,12 @@ the operator to run `homelab client add <hostname>` from an existing client. - `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation with their mesh addresses, roles, and service mappings. Treat this file as authoritative; anything you read in narrative pages should agree with it. -- `/opt/homelab-context/infrastructure/mesh.md` — Tailscale → Netbird state. +- `/opt/homelab-context/knowledge/wiki/infrastructure/mesh.md` — Tailscale → Netbird state. Both meshes are accepted today; Netbird is preferred for new traffic. -- `/opt/homelab-context/infrastructure/dns.md` — split-horizon DNS via - Technitium on [dns (107)](../containers/107-dns.md). `*.hubris.network` +- `/opt/homelab-context/knowledge/wiki/infrastructure/dns.md` — split-horizon DNS via + Technitium on [dns (107)](knowledge/wiki/containers/107-dns.md). `*.hubris.network` resolves to 192.168.x.x on the LAN and to mesh addresses off-LAN. -- `/opt/homelab-context/operations/commands.md` — the operator's cheatsheet +- `/opt/homelab-context/.agents/operations/commands.md` — the operator's cheatsheet for pct, caddy, DNS, and the Oikos command surface. Use these verbs when you take actions. @@ -66,17 +75,25 @@ Grep is fine for browsing or when MCP is unreachable. ## 4. Wiki conventions -- Pages live under `containers/`, `hosts/`, `vms/`, `infrastructure/`, - `investigations/`, `operations/`. Cross-link liberally; orphans are bugs. -- Every page ends with a `## Changelog` section, entries in reverse-chrono - order: +See [page-templates.md](.agents/shared/page-templates.md) for file naming, page +structure, and the tone standard. Quick reference: + +- **File naming:** Foundational docs are ALL-CAPS (AGENTS.md, OIKOS.md, GLOSSARY.md); + containers use `<id>-<name>.md`; infrastructure uses lowercase-with-dashes; plans + and investigations use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`. +- **Where pages live:** Narrative under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`; + incident records under `knowledge/sources/investigations/`; runbook procedures under + `.agents/skills/<name>/SKILL.md`; operator reference under `.agents/operations/`; + design docs under `plans/`. Cross-link liberally; orphans are bugs. +- **Changelog format:** Every page ends with a `## Changelog` section, entries in + reverse-chronological order: ### YYYY-MM-DD — short title one or two lines describing what changed and why. -- Investigation files are dated and slugged: `YYYY-MM-DD-slug.md`. -- Live state takes precedence over docs. If you observe a discrepancy, update - the docs *in the same session* (per the same-session update rule). +- **Live state precedence.** If you observe a discrepancy between the docs and + running state, update the docs *in the same session* (per the same-session update + rule in [page-templates.md](.agents/shared/page-templates.md#same-session-update-rule)). ## 5. Acting on the homelab @@ -96,13 +113,13 @@ Grep is fine for browsing or when MCP is unreachable. a valid `--approval-id` from `homelab approval request` — see OIKOS.md. For ad-hoc work, SSH and edit directly — but commit changes that touch tracked configs (caddy, gitea custom, artifacto, mule-image, etc.; see - `infrastructure/auto-deploy.md`). + `knowledge/wiki/infrastructure/auto-deploy.md`). - **Wiki updates**: same-session rule applies to any meaningful state change this client makes. ## 6. Communication mode -Read and apply `/opt/homelab-context/CAVEMAN.md` (if present). It defines the lab's +Read and apply `/opt/homelab-context/.agents/shared/caveman.md` (if present). It defines the lab's terse-communication standard — drop filler, keep substance, use fragments. ## 7. Auto-setup mechanism diff --git a/README.md b/README.md index 2c0ab21..9bd425b 100644 --- a/README.md +++ b/README.md @@ -12,23 +12,23 @@ Living documentation for the **hubris** Proxmox homelab. Every node, every cross - [Cross-cutting infrastructure](#cross-cutting-infrastructure) — DNS, ingress, mesh, storage, auth - [Investigations](#investigations) — incident timelines - [Operations](#operations) — cheatsheet, enrollment, runbooks -- [Glossary](GLOSSARY.md) — term definitions +- [Glossary](knowledge/GLOSSARY.md) — term definitions - [Conventions](#conventions) — wiki style, changelog hygiene, same-session update rule ## Map ### Proxmox hosts -- [`hubris`](hosts/hubris.md) — PVE node, GMKtec NucBox M6 Ultra, `192.168.8.77` — primary host, runs [8 LXCs](containers/index.md) + 2 VMs -- [`strong`](hosts/strong.md) — PVE node (cluster hostname `strong`), `192.168.178.181` — 2nd member of `Homelab` cluster. Hosts [7 LXCs](containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05) +- [`hubris`](knowledge/wiki/hosts/hubris.md) — PVE node, GMKtec NucBox M6 Ultra, `192.168.8.77` — primary host, runs [8 LXCs](knowledge/wiki/containers/index.md) + 2 VMs +- [`strong`](knowledge/wiki/hosts/strong.md) — PVE node (cluster hostname `strong`), `192.168.178.181` — 2nd member of `Homelab` cluster. Hosts [7 LXCs](knowledge/wiki/containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05) ### VMs -- [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation) -- [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS -- See [vms/index.md](vms/index.md) for the full table +- [100 — `zimaos`](knowledge/wiki/vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation) +- [108 — `haos-16.3`](knowledge/wiki/vms/108-haos.md) — Home Assistant OS +- See [vms/index.md](knowledge/wiki/vms/index.md) for the full table ### LXC containers See the full table with IPs, hosts, mounts, and status in -[`containers/index.md`](containers/index.md). Quick summary: +[`containers/index.md`](knowledge/wiki/containers/index.md). Quick summary: - **hubris** (10 active): 102 nfs-export, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 119 sophia, 120 mule-images, 121 caddy, 124 authentik (outpost), @@ -38,32 +38,33 @@ See the full table with IPs, hosts, mounts, and status in - **Destroyed (archaeology)**: 100 arr, 106 flaresolverr, 107 marimo, 109 syncthing, 110 photoprism, 111 karakeep, 112 immich, 115 reticulum, 123 claudio-bot, 125 seafile, 126 plato, 127 mule-photos-new — see - [containers/index.md](containers/index.md#recently-destroyed-kept-for-archaeology) + [containers/index.md](knowledge/wiki/containers/index.md#recently-destroyed-kept-for-archaeology) ### Cross-cutting infrastructure -- [Infrastructure index](infrastructure/index.md) — map of every cross-cutting system -- [Glossary](GLOSSARY.md) — term definitions -- [DNS — split-horizon](infrastructure/dns.md) -- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md) -- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md) -- [Monitoring — Hermes health watchdog](infrastructure/monitoring.md) -- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md) -- [SSH access](infrastructure/ssh-access.md) -- [Backups — rclone → Proton Drive (LXC 132); restic-on-USB deprecated](infrastructure/backups.md) -- [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md) -- [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md) -- [Homelab context distribution](infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance +- [Infrastructure index](knowledge/wiki/infrastructure/index.md) — map of every cross-cutting system +- [Glossary](knowledge/GLOSSARY.md) — term definitions +- [DNS — split-horizon](knowledge/wiki/infrastructure/dns.md) +- [Ingress — Caddy + VPS traefik](knowledge/wiki/infrastructure/ingress.md) +- [Mesh — Tailscale → Netbird migration](knowledge/wiki/infrastructure/mesh.md) +- [Monitoring — Hermes health watchdog](knowledge/wiki/infrastructure/monitoring.md) +- [Media permissions — `media` GID 10000](knowledge/wiki/infrastructure/media-permissions.md) +- [SSH access](knowledge/wiki/infrastructure/ssh-access.md) +- [Backups — rclone → Proton Drive (LXC 132); restic-on-USB deprecated](knowledge/wiki/infrastructure/backups.md) +- [Auto-deploy — gitea-webhook pipelines](knowledge/wiki/infrastructure/auto-deploy.md) +- [VPS hardening — IONOS / netbird control plane](knowledge/wiki/infrastructure/vps-hardening.md) +- [Homelab context distribution](knowledge/wiki/infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance ### Investigations -Time-stamped incident notes / experiments in [`investigations/index.md`](investigations/index.md). -Resolved cases move to [`investigations/archive/`](investigations/archive/). +Time-stamped incident notes / experiments in [`investigations/index.md`](knowledge/sources/investigations/index.md). +Resolved cases move to [`investigations/archive/`](knowledge/sources/investigations/archive/). ### Operations -- [Command cheatsheet](operations/commands.md) -- [Agent enrollment](operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system +- [Command cheatsheet](.agents/operations/commands.md) +- [Agent enrollment](.agents/operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system ## Conventions +- **File naming.** Foundational docs (entry-points, agent instruction, references) are ALL-CAPS (`AGENTS.md`, `OIKOS.md`, `GLOSSARY.md`); containers use `<id>-<name>.md`; infrastructure pages use lowercase-with-dashes; plans and incidents use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`. See [page-templates.md](.agents/shared/page-templates.md) for the full rules. - **Each node page** ends with a `## Changelog` section. Reverse-chronological. Entry format: ``` ### YYYY-MM-DD — short title @@ -71,7 +72,7 @@ Resolved cases move to [`investigations/archive/`](investigations/archive/). ``` - **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug. - **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog. -- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local. +- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](knowledge/wiki/infrastructure/auto-deploy.md). Edits there must be pushed, not left local. - **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not. ## Maintaining this wiki @@ -80,8 +81,9 @@ When you change a node: 1. Update the relevant page (config snapshot, ports, mounts). 2. Add a changelog entry at the bottom of that page. 3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry. -4. If it's an incident, add an entry to [`investigations/`](investigations/index.md). +4. If it's an incident, add an entry to [`investigations/`](knowledge/sources/investigations/index.md). ## See also -- [`CONTRIBUTING.md`](CONTRIBUTING.md) — page templates and tone +- [`.agents/shared/page-templates.md`](.agents/shared/page-templates.md) — page templates and tone +- [`.agents/shared/writing-style.md`](.agents/shared/writing-style.md) — prose style, banned vocabulary diff --git a/bin/homelab b/bin/homelab index 3a6d002..8f79cda 100755 --- a/bin/homelab +++ b/bin/homelab @@ -1929,7 +1929,7 @@ def main() -> int: csub_add.add_argument("--with-hermes", action="store_true", help="also grant secrets/openrouter-api-key.yaml so this " "host can run the Hermes agent (see " - "operations/hermes-agent.md). Combine with --finalize-pubkey.") + ".agents/operations/hermes-agent.md). Combine with --finalize-pubkey.") csub_add.set_defaults(func=cmd_client_add) csub_rm = csub.add_parser("remove") csub_rm.add_argument("name") diff --git a/bootstrap.sh b/bootstrap.sh index 70608bb..4431c4f 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -507,7 +507,7 @@ if [ "$WITH_HERMES" -eq 1 ]; then Linux) HERMES_LINK=/root/HERMES.md ;; Darwin) HERMES_LINK=/etc/HERMES.md ;; esac - run "ln -sfn '$CLONE_DIR/HERMES.md' '$HERMES_LINK'" + run "ln -sfn '$CLONE_DIR/.agents/HERMES.md' '$HERMES_LINK'" echo "[bootstrap] linked HERMES.md → $HERMES_LINK" # 4. Drop the Goose config. Idempotent YAML merge — preserves any keys the @@ -563,7 +563,7 @@ PYEOF # 5. Symlink HERMES.md as the global .goosehints — Goose injects it into # the system prompt on every session start. - run "ln -sfn '$CLONE_DIR/HERMES.md' '$GOOSEHINTS'" + run "ln -sfn '$CLONE_DIR/.agents/HERMES.md' '$GOOSEHINTS'" if [ "$DRY_RUN" -eq 0 ]; then chown -h "$H_USER" "$GOOSEHINTS" 2>/dev/null || true fi diff --git a/hosts/apps.yaml b/hosts/apps.yaml index ad5a2cb..375be02 100644 --- a/hosts/apps.yaml +++ b/hosts/apps.yaml @@ -34,7 +34,7 @@ services_hosted: - name: artifacto backend: apps url: https://artifacto.hubris.network - doc_page: containers/105-apps.md + doc_page: knowledge/wiki/containers/105-apps.md config_repo: dtoro/Artifacto - name: homelab_mcp backend: apps @@ -42,7 +42,7 @@ services_hosted: systemd_unit: homelab-mcp public_host: mcp.hubris.network endpoint: https://mcp.hubris.network/mcp - doc_page: infrastructure/homelab-context.md + doc_page: knowledge/wiki/infrastructure/homelab-context.md config_repo: dtoro/Homelab-Docs note: MCP server. Read-only context + management. Reachable on the LAN via Caddy and from off-LAN via Netbird (192.168.8.0/24 is a network resource routed through hubris). @@ -53,12 +53,10 @@ services_hosted: systemd_unit: secrets-issuance public_host: secrets.hubris.network endpoint: https://secrets.hubris.network/issue - doc_page: operations/agent-enrollment.md + doc_page: .agents/operations/agent-enrollment.md config_repo: dtoro/Homelab-Docs note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS). risk_notes: "identity issuance \u2014 any change is security-sensitive; key operations are destructive-class" age_pubkey: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0 -see_also: -- containers/105-apps.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/arriman.yaml b/hosts/arriman.yaml index 27c0933..58e7cf3 100644 --- a/hosts/arriman.yaml +++ b/hosts/arriman.yaml @@ -29,12 +29,10 @@ services_hosted: - name: arr_stack backend: arriman note: jellyseerr / qbit / sab on docker compose - doc_page: containers/122-arriman.md + doc_page: knowledge/wiki/containers/122-arriman.md notes: - Migrated from hubris to strong 2026-07-05 (Phase 2). Library on ludo-lvm. - Contains homarr, radarr, sonarr, lidarr, sabnzbd, qbittorrent, bazarr, flaresolverr, prowlarr, jellyseerr - qBittorrent auth subnet whitelist expanded to 192.168.8.0/24 (for seanime + Caddy access) -see_also: -- containers/122-arriman.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/auth-outpost.yaml b/hosts/auth-outpost.yaml index e849205..29a0fae 100644 --- a/hosts/auth-outpost.yaml +++ b/hosts/auth-outpost.yaml @@ -16,7 +16,5 @@ mesh_globals: - tailscale notes: - Runs Authentik outpost (reverse-proxy/SSO enforcement) for protected services -see_also: -- containers/106-auth-outpost.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/caddy.yaml b/hosts/caddy.yaml index 02c0b8a..c533e36 100644 --- a/hosts/caddy.yaml +++ b/hosts/caddy.yaml @@ -24,14 +24,12 @@ services_hosted: backend: caddy role: reverse-proxy note: terminates all *.hubris.network - doc_page: containers/121-caddy.md + doc_page: knowledge/wiki/containers/121-caddy.md config_repo: dtoro/caddy-conf risk_notes: "wide blast radius \u2014 every *.hubris.network route rides on it (see oikos/policy.yaml\ \ service_overrides)" notes: - Terminates all *.hubris.network - /etc/caddy is a git checkout of dtoro/caddy-conf -see_also: -- containers/121-caddy.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/dns.yaml b/hosts/dns.yaml index 8a966d2..67bc722 100644 --- a/hosts/dns.yaml +++ b/hosts/dns.yaml @@ -20,12 +20,10 @@ services_hosted: - name: dns backend: dns note: Technitium DNS, split-horizon zone - doc_page: containers/107-dns.md + doc_page: knowledge/wiki/containers/107-dns.md risk_notes: "LAN-wide resolver \u2014 misconfig breaks name resolution for every client" notes: - Technitium DNS, split-horizon zone for *.hubris.network - Primary DNS for 192.168.8.0/24 LAN (inventory.services.dns references this) -see_also: -- containers/107-dns.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/elementsynapse.yaml b/hosts/elementsynapse.yaml index 2f2dbad..535b6fa 100644 --- a/hosts/elementsynapse.yaml +++ b/hosts/elementsynapse.yaml @@ -23,11 +23,9 @@ services_hosted: - name: matrix url: https://matrix.hubris.network backend: elementsynapse - doc_page: containers/118-elementsynapse.md + doc_page: knowledge/wiki/containers/118-elementsynapse.md risk_notes: "alert/approval channel for Oikos \u2014 outage silences agent escalation" notes: - Migrated from hubris to strong 2026-07-05 (Phase 1 of strong migration plan). -see_also: -- containers/118-elementsynapse.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/gitea.yaml b/hosts/gitea.yaml index 01e8963..1aa4297 100644 --- a/hosts/gitea.yaml +++ b/hosts/gitea.yaml @@ -27,12 +27,10 @@ services_hosted: url: https://git.hubris.network backend: gitea backend_url: http://192.168.8.121:3000 - doc_page: containers/104-gitea.md + doc_page: knowledge/wiki/containers/104-gitea.md config_repo: dtoro/gitea-customizations risk_notes: hosts all config repos + deploy webhooks; outage blocks auto-deploy and sync notes: - Bare repos live at /mnt/library/repos/dtoro/*.git -see_also: -- containers/104-gitea.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/grimmory.yaml b/hosts/grimmory.yaml index d04d322..740ea4c 100644 --- a/hosts/grimmory.yaml +++ b/hosts/grimmory.yaml @@ -21,7 +21,5 @@ notes: - Docker host for Grimmory (community fork of Booklore). Created 2026-06-29. - Migrated from hubris to strong 2026-07-05 (Phase 2d). Books on ludo-lvm. age_pubkey: age1uellsemnjrzgfg9fxw4jefpy05laxzggwnwhh6ny3wl7alyp6v8q0muxet -see_also: -- containers/130-grimmory.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/haos.yaml b/hosts/haos.yaml index 909198b..467fd4a 100644 --- a/hosts/haos.yaml +++ b/hosts/haos.yaml @@ -22,8 +22,6 @@ runs: services_hosted: - name: haos backend: haos - doc_page: vms/108-haos.md -see_also: -- vms/108-haos.md + doc_page: knowledge/wiki/vms/108-haos.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/house.yaml b/hosts/house.yaml index edcf89c..34f6c72 100644 --- a/hosts/house.yaml +++ b/hosts/house.yaml @@ -22,7 +22,5 @@ notes: - 192.168.8.212 was the hubris IP before migration (briefly picked up by teddycloud via DHCP; teddycloud has since been given a static IP, see hosts.teddycloud) age_pubkey: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h -see_also: -- containers/129-house.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/hubris.yaml b/hosts/hubris.yaml index 0f3d065..2c14eab 100644 --- a/hosts/hubris.yaml +++ b/hosts/hubris.yaml @@ -29,10 +29,8 @@ services_hosted: url: https://proxmox.hubris.network backend: hubris port: 8006 - doc_page: hosts/hubris.md + doc_page: knowledge/wiki/hosts/hubris.md risk_notes: "hypervisor UI \u2014 changes here affect every guest on the node" age_pubkey: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6 -see_also: -- hosts/hubris.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/jellyfin.yaml b/hosts/jellyfin.yaml index 982a6df..67a871d 100644 --- a/hosts/jellyfin.yaml +++ b/hosts/jellyfin.yaml @@ -26,7 +26,7 @@ services_hosted: - name: jellyfin url: https://media.hubris.network backend: jellyfin - doc_page: containers/101-jellyfin.md + doc_page: knowledge/wiki/containers/101-jellyfin.md risk_notes: native Authentik OIDC via SSO-Auth plugin, no Caddy forward-auth gate; VAAPI transcode depends on GPU passthrough on strong notes: @@ -35,7 +35,5 @@ notes: - SSO-Auth plugin v4.0.0.4 with Authentik OIDC (no Caddy forward-auth gate) - GPU passed via dev0+dev1: /dev/dri/renderD128 + card0 - Migrated from hubris to strong 2026-07-05 (Phase 2). Library on ludo-lvm. -see_also: -- containers/101-jellyfin.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/mule-images.yaml b/hosts/mule-images.yaml index 26d4c12..ea596d1 100644 --- a/hosts/mule-images.yaml +++ b/hosts/mule-images.yaml @@ -26,9 +26,7 @@ services_hosted: - name: photos url: https://photos.hubris.network backend: mule-images - doc_page: containers/120-mule-images.md + doc_page: knowledge/wiki/containers/120-mule-images.md config_repo: dtoro/mule-image -see_also: -- containers/120-mule-images.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/netbird-vps.yaml b/hosts/netbird-vps.yaml index c3be6b5..fc468d1 100644 --- a/hosts/netbird-vps.yaml +++ b/hosts/netbird-vps.yaml @@ -23,7 +23,7 @@ services_hosted: - name: authentik url: https://auth.hubris.network backend: netbird-vps - doc_page: containers/106-auth-outpost.md + doc_page: knowledge/wiki/containers/106-auth-outpost.md note: core runs on the VPS since 2026-05-31; LAN forward-auth outpost is auth-outpost (LXC 106) at 192.168.8.6:9000. Previous backend value "authentik" referenced the retired embedded-outpost host (LXC 124). risk_notes: "SSO provider \u2014 outage locks login to OIDC/forward-auth services" diff --git a/hosts/nextcloud.yaml b/hosts/nextcloud.yaml index f2611cc..b81b3a4 100644 --- a/hosts/nextcloud.yaml +++ b/hosts/nextcloud.yaml @@ -26,8 +26,6 @@ services_hosted: - name: nextcloud url: https://cloud.hubris.network backend: nextcloud - doc_page: containers/114-nextcloud.md -see_also: -- containers/114-nextcloud.md + doc_page: knowledge/wiki/containers/114-nextcloud.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/nfs-export.yaml b/hosts/nfs-export.yaml index ab2e332..abcff5c 100644 --- a/hosts/nfs-export.yaml +++ b/hosts/nfs-export.yaml @@ -14,7 +14,5 @@ mesh_globals: accepted: - netbird - tailscale -see_also: -- containers/102-nfs-export.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/paperless.yaml b/hosts/paperless.yaml index 17bb657..b836978 100644 --- a/hosts/paperless.yaml +++ b/hosts/paperless.yaml @@ -26,9 +26,7 @@ services_hosted: - name: paperless url: https://paperless.hubris.network backend: paperless - doc_page: containers/103-paperless.md + doc_page: knowledge/wiki/containers/103-paperless.md risk_notes: "document archive \u2014 treat data as irreplaceable; DB operations are destructive-class" -see_also: -- containers/103-paperless.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/romm.yaml b/hosts/romm.yaml index 03b9a91..91c2674 100644 --- a/hosts/romm.yaml +++ b/hosts/romm.yaml @@ -22,7 +22,5 @@ notes: - MariaDB sidecar at /opt/romm/docker-compose.yml. - ROMs on ludo-lvm media volume at /mnt/media_local/roms. - 1 core / 2 GiB RAM / 16 GiB rootfs (ludo-lvm). -see_also: -- containers/134-romm.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/seanime.yaml b/hosts/seanime.yaml index 1857d2a..8344a10 100644 --- a/hosts/seanime.yaml +++ b/hosts/seanime.yaml @@ -25,7 +25,5 @@ notes: - /anime mounted from strong ludo-lvm (/mnt/media_local/anime) - Caddy: "https://seanime.hubris.network \u2192 192.168.8.248:43211" - qBittorrent auth subnet whitelist expanded to 192.168.8.0/24 for seanime access -see_also: -- containers/133-seanime.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/sophia.yaml b/hosts/sophia.yaml index c564bb6..87e3efe 100644 --- a/hosts/sophia.yaml +++ b/hosts/sophia.yaml @@ -19,7 +19,5 @@ mesh_globals: - tailscale mounts: - /mnt/library -see_also: -- containers/119-sophia.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/strong.yaml b/hosts/strong.yaml index 463a0f4..b0d2fab 100644 --- a/hosts/strong.yaml +++ b/hosts/strong.yaml @@ -28,7 +28,5 @@ notes: \ (filename kept as-is, it's a historical planning doc). Only Phase 1 (Proxmox install + cluster join)\ \ is done; no physical drive move, service migration, or GPU passthrough has happened yet." age_pubkey: age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4 -see_also: -- hosts/strong.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/teddycloud.yaml b/hosts/teddycloud.yaml index f39c622..15bb9a3 100644 --- a/hosts/teddycloud.yaml +++ b/hosts/teddycloud.yaml @@ -23,7 +23,7 @@ services_hosted: - name: teddycloud url: https://teddy.hubris.network backend: teddycloud - doc_page: containers/131-teddycloud.md + doc_page: knowledge/wiki/containers/131-teddycloud.md note: self-hosted TeddyCloud (Toniebox cloud reimplementation), docker compose risk_notes: "no Caddy forward-auth gate (unlike sab.hubris.network on the same Caddyfile) \u2014 reachable\ \ to anyone on the LAN/mesh who can resolve teddy.hubris.network; undocumented in inventory.yaml until\ @@ -38,7 +38,5 @@ notes: \ DHCP before that \u2014 see hosts/strong.md's 2026-07-05 changelog)." - "No age_pubkey / homelab-context enrollment \u2014 not a homelab CLI client, just a docker-compose app\ \ container. Not a required follow-up unless it needs secrets." -see_also: -- containers/131-teddycloud.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/trmnl.yaml b/hosts/trmnl.yaml index 2e80ecc..937dea7 100644 --- a/hosts/trmnl.yaml +++ b/hosts/trmnl.yaml @@ -22,9 +22,7 @@ services_hosted: backend: trmnl url: https://trmnl.hubris.network note: self-hosted middleware for TRMNL e-ink plugins (polled by TRMNL cloud) - doc_page: containers/128-trmnl.md + doc_page: knowledge/wiki/containers/128-trmnl.md config_repo: dtoro/terminalito -see_also: -- containers/128-trmnl.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/zimaos.yaml b/hosts/zimaos.yaml index 3fd4ac6..a384967 100644 --- a/hosts/zimaos.yaml +++ b/hosts/zimaos.yaml @@ -21,8 +21,6 @@ services_hosted: - name: zimaos url: https://zimaos.hubris.network backend: zimaos - doc_page: vms/100-zimaos.md -see_also: -- vms/100-zimaos.md + doc_page: knowledge/wiki/vms/100-zimaos.md mcp_endpoint: https://mcp.hubris.network/mcp secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/inventory.yaml b/inventory.yaml index 47d2398..3f5f087 100644 --- a/inventory.yaml +++ b/inventory.yaml @@ -48,26 +48,26 @@ services: url: https://proxmox.hubris.network backend: hubris port: 8006 - doc_page: hosts/hubris.md + doc_page: knowledge/wiki/hosts/hubris.md risk_notes: hypervisor UI — changes here affect every guest on the node gitea: url: https://git.hubris.network backend: gitea backend_url: http://192.168.8.121:3000 - doc_page: containers/104-gitea.md + doc_page: knowledge/wiki/containers/104-gitea.md config_repo: dtoro/gitea-customizations risk_notes: hosts all config repos + deploy webhooks; outage blocks auto-deploy and sync caddy: backend: caddy role: reverse-proxy note: terminates all *.hubris.network - doc_page: containers/121-caddy.md + doc_page: knowledge/wiki/containers/121-caddy.md config_repo: dtoro/caddy-conf risk_notes: wide blast radius — every *.hubris.network route rides on it (see oikos/policy.yaml service_overrides) authentik: url: https://auth.hubris.network backend: netbird-vps - doc_page: containers/106-auth-outpost.md + doc_page: knowledge/wiki/containers/106-auth-outpost.md note: >- core runs on the VPS since 2026-05-31; LAN forward-auth outpost is auth-outpost (LXC 106) at 192.168.8.6:9000. Previous backend value @@ -76,58 +76,58 @@ services: dns: backend: dns note: Technitium DNS, split-horizon zone - doc_page: containers/107-dns.md + doc_page: knowledge/wiki/containers/107-dns.md risk_notes: LAN-wide resolver — misconfig breaks name resolution for every client jellyfin: url: https://media.hubris.network backend: jellyfin - doc_page: containers/101-jellyfin.md + doc_page: knowledge/wiki/containers/101-jellyfin.md risk_notes: native Authentik OIDC via SSO-Auth plugin, no Caddy forward-auth gate; VAAPI transcode depends on GPU passthrough on strong nextcloud: url: https://cloud.hubris.network backend: nextcloud - doc_page: containers/114-nextcloud.md + doc_page: knowledge/wiki/containers/114-nextcloud.md paperless: url: https://paperless.hubris.network backend: paperless - doc_page: containers/103-paperless.md + doc_page: knowledge/wiki/containers/103-paperless.md risk_notes: document archive — treat data as irreplaceable; DB operations are destructive-class matrix: url: https://matrix.hubris.network backend: elementsynapse - doc_page: containers/118-elementsynapse.md + doc_page: knowledge/wiki/containers/118-elementsynapse.md risk_notes: alert/approval channel for Oikos — outage silences agent escalation photos: url: https://photos.hubris.network backend: mule-images - doc_page: containers/120-mule-images.md + doc_page: knowledge/wiki/containers/120-mule-images.md config_repo: dtoro/mule-image arr_stack: backend: arriman note: jellyseerr / qbit / sab on docker compose - doc_page: containers/122-arriman.md + doc_page: knowledge/wiki/containers/122-arriman.md artifacto: backend: apps url: https://artifacto.hubris.network - doc_page: containers/105-apps.md + doc_page: knowledge/wiki/containers/105-apps.md config_repo: dtoro/Artifacto trmnl: backend: trmnl url: https://trmnl.hubris.network note: self-hosted middleware for TRMNL e-ink plugins (polled by TRMNL cloud) - doc_page: containers/128-trmnl.md + doc_page: knowledge/wiki/containers/128-trmnl.md config_repo: dtoro/terminalito zimaos: url: https://zimaos.hubris.network backend: zimaos - doc_page: vms/100-zimaos.md + doc_page: knowledge/wiki/vms/100-zimaos.md haos: backend: haos - doc_page: vms/108-haos.md + doc_page: knowledge/wiki/vms/108-haos.md teddycloud: url: https://teddy.hubris.network backend: teddycloud - doc_page: containers/131-teddycloud.md + doc_page: knowledge/wiki/containers/131-teddycloud.md note: self-hosted TeddyCloud (Toniebox cloud reimplementation), docker compose risk_notes: no Caddy forward-auth gate (unlike sab.hubris.network on the same Caddyfile) — reachable to anyone on the LAN/mesh who can resolve teddy.hubris.network; undocumented @@ -138,7 +138,7 @@ services: systemd_unit: homelab-mcp public_host: mcp.hubris.network endpoint: https://mcp.hubris.network/mcp - doc_page: infrastructure/homelab-context.md + doc_page: knowledge/wiki/infrastructure/homelab-context.md config_repo: dtoro/Homelab-Docs note: MCP server. Read-only context + management. Reachable on the LAN via Caddy and from off-LAN via Netbird (192.168.8.0/24 is a network resource routed through @@ -150,7 +150,7 @@ services: systemd_unit: secrets-issuance public_host: secrets.hubris.network endpoint: https://secrets.hubris.network/issue - doc_page: operations/agent-enrollment.md + doc_page: .agents/operations/agent-enrollment.md config_repo: dtoro/Homelab-Docs note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS). diff --git a/GLOSSARY.md b/knowledge/GLOSSARY.md similarity index 87% rename from GLOSSARY.md rename to knowledge/GLOSSARY.md index 624b77f..9b68dc0 100644 --- a/GLOSSARY.md +++ b/knowledge/GLOSSARY.md @@ -18,7 +18,7 @@ Terms and abbreviations used throughout the homelab wiki. | **Mesh** | Overlay VPN for off-LAN connectivity. Netbird is current; Tailscale is legacy | | **Netbird** | Preferred mesh VPN. VPS hosts the management plane; all homelab nodes are members | | **OIDC** | OpenID Connect. Protocol used by Authentik for SSO login flows | -| **Oikos** | Agent operating model (OIKOS.md). OODA loop, risk classes, policy, ontology | +| **Oikos** | Agent operating model ([.agents/OIKOS.md](../.agents/OIKOS.md)). OODA loop, risk classes, policy, ontology | | **PVE** | Proxmox Virtual Environment — the hypervisor on both hubris and strong | | **SOPS** | `sops` — Mozilla SOPS. Encrypts secrets with age keys so they live in the git repo | | **Strong** | Secondary Proxmox VE node. Cluster member 2 (hostname `strong`, nickname ludo/ludo-mini) | @@ -29,5 +29,5 @@ Terms and abbreviations used throughout the homelab wiki. ## See also -- [Infrastructure index](infrastructure/index.md) — cross-cutting systems each with their own doc page -- [OIKOS operating model](OIKOS.md) — agent policy, risk classes, lifecycle \ No newline at end of file +- [Infrastructure index](wiki/infrastructure/index.md) — cross-cutting systems each with their own doc page +- [OIKOS operating model](../.agents/OIKOS.md) — agent policy, risk classes, lifecycle \ No newline at end of file diff --git a/knowledge/index.md b/knowledge/index.md new file mode 100644 index 0000000..8252ad3 --- /dev/null +++ b/knowledge/index.md @@ -0,0 +1,14 @@ +# Knowledge + +The durable, authoritative current-state documentation of the homelab: one page per node and per +cross-cutting system, synthesized from live state and evidence. Structure and rules are in +[the knowledge schema](../.agents/domains/knowledge/schema.md). + +| Section | What it covers | +|---------|----------------| +| [wiki/hosts/](wiki/hosts/index.md) | Proxmox host narratives — `hubris`, `strong`. | +| [wiki/containers/](wiki/containers/index.md) | LXC fleet — one page per container, plus the master table and archaeology. | +| [wiki/vms/](wiki/vms/index.md) | Virtual machines — ZimaOS, Home Assistant OS. | +| [wiki/infrastructure/](wiki/infrastructure/index.md) | Cross-cutting systems — DNS, ingress, mesh, storage, auth, monitoring, generated topology. | +| [sources/](sources/index.md) | External reference docs and the pointer to incident evidence. | +| [GLOSSARY.md](GLOSSARY.md) | Term definitions. | diff --git a/knowledge/log.md b/knowledge/log.md new file mode 100644 index 0000000..e01f239 --- /dev/null +++ b/knowledge/log.md @@ -0,0 +1,10 @@ +# Knowledge — operations log + +Append-only record of documentation-maintenance operations on the knowledge wiki (restructures, +source ingests, lint sweeps). One line per operation, newest last. Infrastructure changes belong in +each page's `## Changelog` and the Oikos change ledger, not here. + +## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output. +## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md. +## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference. +## [2026-07-06] restructure | Phase 6 consolidation: investigations/ -> knowledge/sources/investigations/; operations/ -> .agents/operations/; HERMES.md -> .agents/; deleted root OIKOS/CAVEMAN/CONTRIBUTING stubs. diff --git a/knowledge/sources/index.md b/knowledge/sources/index.md new file mode 100644 index 0000000..e8243a6 --- /dev/null +++ b/knowledge/sources/index.md @@ -0,0 +1,9 @@ +# Sources + +Immutable evidence the wiki synthesizes from. External reference docs live under `references/`; +incident evidence lives in [`investigations/`](investigations/index.md) (written once at +incident time, then linked from the changelogs of the nodes they implicate). + +| Slug | Reference | Summary | +|------|-----------|---------| +| cert-sync-and-traefik-config | [references/cert-sync-and-traefik-config.md](references/cert-sync-and-traefik-config.md) | VPS traefik config and the LAN↔VPS certificate mirror. | diff --git a/investigations/2026-06-01-mac-mini-onboarding.md b/knowledge/sources/investigations/2026-06-01-mac-mini-onboarding.md similarity index 100% rename from investigations/2026-06-01-mac-mini-onboarding.md rename to knowledge/sources/investigations/2026-06-01-mac-mini-onboarding.md diff --git a/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md b/knowledge/sources/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md similarity index 89% rename from investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md rename to knowledge/sources/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md index f5a6a06..1230b1d 100644 --- a/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md +++ b/knowledge/sources/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md @@ -2,7 +2,7 @@ ## Summary -[`ludo-mini`](../hosts/ludo-mini.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming. +[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming. ## Timeline diff --git a/investigations/2026-06-06-authentik-session-lifetime.md b/knowledge/sources/investigations/2026-06-06-authentik-session-lifetime.md similarity index 95% rename from investigations/2026-06-06-authentik-session-lifetime.md rename to knowledge/sources/investigations/2026-06-06-authentik-session-lifetime.md index 0de009e..2a00d1c 100644 --- a/investigations/2026-06-06-authentik-session-lifetime.md +++ b/knowledge/sources/investigations/2026-06-06-authentik-session-lifetime.md @@ -90,9 +90,9 @@ print("session_duration:", stage.session_duration) # → "days=30" ## Related -- [Container 106 — auth-outpost](../containers/106-auth-outpost.md) -- [Authentik VPS migration](2026-05-31-authentik-vps-migration.md) -- [Ingress (VPS Traefik)](../infrastructure/ingress.md) +- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md) +- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md) +- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md) - `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan ## Changelog diff --git a/investigations/2026-06-06-caddyfile-truncation.md b/knowledge/sources/investigations/2026-06-06-caddyfile-truncation.md similarity index 88% rename from investigations/2026-06-06-caddyfile-truncation.md rename to knowledge/sources/investigations/2026-06-06-caddyfile-truncation.md index 856b074..7b7f3c7 100644 --- a/investigations/2026-06-06-caddyfile-truncation.md +++ b/knowledge/sources/investigations/2026-06-06-caddyfile-truncation.md @@ -54,8 +54,8 @@ This is the same class of drift as the June 5th incidents (paperless, HAOS, apps ## Related -- [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md) -- [Caddy (121)](../containers/121-caddy.md) -- [elementsynapse (118)](../containers/118-elementsynapse.md) -- [dns-sync script](../scripts/dns-sync.py) -- [check-caddy-backends script](../scripts/check-caddy-backends.sh) \ No newline at end of file +- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md) +- [Caddy (121)](../../wiki/containers/121-caddy.md) +- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md) +- [dns-sync script](../../../scripts/dns-sync.py) +- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh) \ No newline at end of file diff --git a/investigations/archive/2026-04-21-hubris-crash-loop.md b/knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md similarity index 86% rename from investigations/archive/2026-04-21-hubris-crash-loop.md rename to knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md index 7fbcbf3..4192b0d 100644 --- a/investigations/archive/2026-04-21-hubris-crash-loop.md +++ b/knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md @@ -2,12 +2,12 @@ ## Summary -[`hubris`](../hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders. +[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders. ## Timeline ### 2026-04-19 — drive attached -External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../infrastructure/backups.md). Pre-attach uptime had been 33 days stable. +External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable. ### 2026-04-19 → 2026-04-21 — first crashes Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. @@ -24,13 +24,13 @@ Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly - **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running. ### 2026-04-22 — recurrence after 30 h 37 m -Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty. +Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty. ### 2026-04-22 — `cpu-epp.service` design bug fixed Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`. ### 2026-04-22 — drive removed (A/B test) -User physically removed the external USB drive. [Backup timers disabled](../infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause. +User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause. ### 2026-04-23 — SSD cooling + thermal pads installed Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically. @@ -78,9 +78,9 @@ Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the Nu | `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive | ## Affected nodes -- [Hubris host](../hosts/hubris.md) -- [Backups (disabled)](../infrastructure/backups.md) -- [Monitoring](../infrastructure/monitoring.md) +- [Hubris host](../../../wiki/hosts/hubris.md) +- [Backups (disabled)](../../../wiki/infrastructure/backups.md) +- [Monitoring](../../../wiki/infrastructure/monitoring.md) ## Open questions - Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.) diff --git a/investigations/archive/2026-05-31-authentik-vps-migration.md b/knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md similarity index 80% rename from investigations/archive/2026-05-31-authentik-vps-migration.md rename to knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md index 972e98d..5ca79c4 100644 --- a/investigations/archive/2026-05-31-authentik-vps-migration.md +++ b/knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md @@ -2,9 +2,9 @@ ## Summary -The NetBird management server (on the [VPS](../infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.** +The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.** -Resolved by moving Authentik off [LXC 124](../containers/124-authentik.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur. +Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur. The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration. @@ -43,7 +43,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS 1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`. 2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug. -3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../containers/124-authentik.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults. +3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults. 4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue. 5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA. @@ -51,7 +51,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS | | Before | After | |---|---|---| -| Authentik host | [LXC 124](../containers/124-authentik.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` | +| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` | | Version | `2026.2.2` | `2026.5.2` | | `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) | | `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container | @@ -76,7 +76,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted). -Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy). +Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy). ### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy @@ -99,11 +99,11 @@ The policy was orphaned (no matched type data or had incompatible evaluation). R - **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`. - **VPS port 22** opened for this repair; close once remote access is otherwise stable. - **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service). -- **Reconcile [124-authentik](../containers/124-authentik.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page. +- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page. - **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree. ## Related -- [124 — authentik](../containers/124-authentik.md) -- [DNS split-horizon](../infrastructure/dns.md) -- [Public ingress (VPS traefik)](../infrastructure/ingress.md) -- [Mesh migration](../infrastructure/mesh.md) +- [124 — authentik](../../../wiki/containers/106-auth-outpost.md) +- [DNS split-horizon](../../../wiki/infrastructure/dns.md) +- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md) +- [Mesh migration](../../../wiki/infrastructure/mesh.md) diff --git a/investigations/index.md b/knowledge/sources/investigations/index.md similarity index 100% rename from investigations/index.md rename to knowledge/sources/investigations/index.md diff --git a/infrastructure/references/cert-sync-and-traefik-config.md b/knowledge/sources/references/cert-sync-and-traefik-config.md similarity index 100% rename from infrastructure/references/cert-sync-and-traefik-config.md rename to knowledge/sources/references/cert-sync-and-traefik-config.md diff --git a/containers/101-jellyfin.md b/knowledge/wiki/containers/101-jellyfin.md similarity index 97% rename from containers/101-jellyfin.md rename to knowledge/wiki/containers/101-jellyfin.md index 597e231..28c68ac 100644 --- a/containers/101-jellyfin.md +++ b/knowledge/wiki/containers/101-jellyfin.md @@ -78,7 +78,7 @@ User → media.hubris.network → Caddy (TLS, no forward-auth) → Jellyfin :809 `http://` redirect URIs that Authentik rejects) - `EnableAuthorization`: `false` (prevents plugin from overwriting admin permissions on each SSO login — see - [jellyfin-sso-plugin](../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md)) + [jellyfin-sso-plugin](../../../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md)) - `OidScopes`: `["email"]` (openid+profile added by default by the plugin; must be non-null or `OidChallenge()` throws `ArgumentNullException`) @@ -133,7 +133,7 @@ Member of the [media GID 10000](../infrastructure/media-permissions.md) standard - [Media permissions](../infrastructure/media-permissions.md) - [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads - [DNS split-horizon](../infrastructure/dns.md) -- [Authentik admin](../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config +- [Authentik admin](../../../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config ## Changelog diff --git a/containers/102-nfs-export.md b/knowledge/wiki/containers/102-nfs-export.md similarity index 98% rename from containers/102-nfs-export.md rename to knowledge/wiki/containers/102-nfs-export.md index c8efe5a..02266e3 100644 --- a/containers/102-nfs-export.md +++ b/knowledge/wiki/containers/102-nfs-export.md @@ -54,7 +54,7 @@ We considered three options before building this: | Option | Outcome | |---|---| -| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../investigations/index.md)). Rejected. | +| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. | | **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. | | **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. | diff --git a/containers/103-paperless.md b/knowledge/wiki/containers/103-paperless.md similarity index 76% rename from containers/103-paperless.md rename to knowledge/wiki/containers/103-paperless.md index 20b6702..4f7c127 100644 --- a/containers/103-paperless.md +++ b/knowledge/wiki/containers/103-paperless.md @@ -18,7 +18,7 @@ Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/d | paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers | ## Auth -Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login. +Behind [Authentik forward-auth](106-auth-outpost.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login. ## Storage - Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design). @@ -27,7 +27,7 @@ Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses fo - ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15. ## Related -- [Authentik](124-authentik.md) +- [Authentik](106-auth-outpost.md) - [Caddy](121-caddy.md) - [DNS](../infrastructure/dns.md) - [Monitoring](../infrastructure/monitoring.md) diff --git a/containers/104-gitea.md b/knowledge/wiki/containers/104-gitea.md similarity index 96% rename from containers/104-gitea.md rename to knowledge/wiki/containers/104-gitea.md index 95e40c9..2415d8e 100644 --- a/containers/104-gitea.md +++ b/knowledge/wiki/containers/104-gitea.md @@ -55,7 +55,7 @@ Initial documentation. Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md). ### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added -For OIDC integration with [authentik (124)](124-authentik.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency. +For OIDC integration with [authentik (124)](106-auth-outpost.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency. ### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped `dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`. diff --git a/containers/105-apps.md b/knowledge/wiki/containers/105-apps.md similarity index 98% rename from containers/105-apps.md rename to knowledge/wiki/containers/105-apps.md index 3cb2130..f67ed94 100644 --- a/containers/105-apps.md +++ b/knowledge/wiki/containers/105-apps.md @@ -100,7 +100,7 @@ Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.h ## Related - [Gitea (104)](104-gitea.md) — uses the PlantUML server - [Caddy (121)](121-caddy.md) -- [Authentik (124)](124-authentik.md) +- [Authentik (124)](106-auth-outpost.md) - [DNS](../infrastructure/dns.md) - [Auto-deploy](../infrastructure/auto-deploy.md) - [Public ingress (Artifacto + blog)](../infrastructure/ingress.md) @@ -116,7 +116,7 @@ Two new services from the [homelab-context distribution plan](../infrastructure/ `secrets-issuance.service` on `:9820` (per-client age-key provisioning). Caddy fronts both with Let's Encrypt; new vhosts on [caddy](121-caddy.md), split-horizon DNS entries on -[authentik (124)](124-authentik.md). Gitea webhook ids 10 + 11 wire +[authentik (124)](106-auth-outpost.md). Gitea webhook ids 10 + 11 wire auto-deploy. LXC is itself an enrolled context client (`/opt/homelab-context/`). diff --git a/containers/106-auth-outpost.md b/knowledge/wiki/containers/106-auth-outpost.md similarity index 77% rename from containers/106-auth-outpost.md rename to knowledge/wiki/containers/106-auth-outpost.md index c1293a4..478044c 100644 --- a/containers/106-auth-outpost.md +++ b/knowledge/wiki/containers/106-auth-outpost.md @@ -1,6 +1,6 @@ # 106 — `auth-outpost` -Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik. +Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik. ## At a glance - **Hostname:** `auth-outpost` @@ -8,11 +8,11 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co - **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`) - **Resources:** 1 core / 512 MiB / 4 GiB rootfs - **Mounts:** none -- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](124-authentik.md) +- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](106-auth-outpost.md) ## Role -Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../investigations/2026-05-31-authentik-vps-migration.md). +Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md). ## Service / port map | Service | Listen | Notes | @@ -42,15 +42,15 @@ Fix: the LAN outpost gets its **own** domain. **Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`. ## Related -- [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only) +- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only) - [Caddy (121)](121-caddy.md) — forward-auth consumer - [Ingress (VPS traefik)](../infrastructure/ingress.md) -- [Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md) +- [Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) ## Changelog ### 2026-06-06 — Authentik session lifetime extended to 30 days -VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../investigations/2026-06-06-authentik-session-lifetime.md). +VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md). ### 2026-06-01 — created; forward-auth cut over from LXC 124 -New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md). +New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md). diff --git a/containers/107-dns.md b/knowledge/wiki/containers/107-dns.md similarity index 89% rename from containers/107-dns.md rename to knowledge/wiki/containers/107-dns.md index a07e2e7..02d09a3 100644 --- a/containers/107-dns.md +++ b/knowledge/wiki/containers/107-dns.md @@ -1,6 +1,6 @@ # 107 — `dns` -Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](124-authentik.md); single-purpose, one job. +Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](106-auth-outpost.md); single-purpose, one job. ## At a glance - **Hostname:** `dns` @@ -29,7 +29,7 @@ Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recur - **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field. ## dns-sync (Technitium = authoring source) -`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md). +`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../../../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md). ## DHCP @@ -42,7 +42,7 @@ Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02): Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101–.239`) are excluded from the pool. Pool narrowed from `.100–.240` to `.241–.254` on 2026-06-03 to eliminate IP conflict risk. ## Related -- [124 — authentik](124-authentik.md) — retired host of the old dnsmasq +- [124 — authentik](106-auth-outpost.md) — retired host of the old dnsmasq - [DNS split-horizon](../infrastructure/dns.md) - [Mesh](../infrastructure/mesh.md) @@ -55,7 +55,7 @@ Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)) Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`. ### 2026-06-03 — DHCP pool narrowed to `.241–.254` -Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md). ### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone) This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md). @@ -64,4 +64,4 @@ This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100–.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium. ### 2026-06-01 — created; replaced dnsmasq on 124 -Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](124-authentik.md). +Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](106-auth-outpost.md). diff --git a/containers/114-nextcloud.md b/knowledge/wiki/containers/114-nextcloud.md similarity index 98% rename from containers/114-nextcloud.md rename to knowledge/wiki/containers/114-nextcloud.md index 3792fc3..2150431 100644 --- a/containers/114-nextcloud.md +++ b/knowledge/wiki/containers/114-nextcloud.md @@ -11,7 +11,7 @@ Personal cloud / file collaboration. Source-of-truth for the photo libraries sur - **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) ## Auth -Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](124-authentik.md#per-app-username-override-pattern-authentik) for the full pattern. +Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](106-auth-outpost.md#per-app-username-override-pattern-authentik) for the full pattern. Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on). @@ -74,7 +74,7 @@ Apply with `systemctl restart mariadb` (not reload — `innodb_log_file_size` ne ## Related - [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV -- [Authentik (124)](124-authentik.md) +- [Authentik (124)](106-auth-outpost.md) - [Caddy (121)](121-caddy.md) - [DNS](../infrastructure/dns.md) - [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md) diff --git a/containers/118-elementsynapse.md b/knowledge/wiki/containers/118-elementsynapse.md similarity index 98% rename from containers/118-elementsynapse.md rename to knowledge/wiki/containers/118-elementsynapse.md index e552a08..17e4334 100644 --- a/containers/118-elementsynapse.md +++ b/knowledge/wiki/containers/118-elementsynapse.md @@ -37,7 +37,7 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name - ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15. ## Related -- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent +- ~~[claudio-bot (123)](archive/123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent - [Caddy](121-caddy.md) - [DNS](../infrastructure/dns.md) - [Monitoring](../infrastructure/monitoring.md) diff --git a/containers/119-sophia.md b/knowledge/wiki/containers/119-sophia.md similarity index 100% rename from containers/119-sophia.md rename to knowledge/wiki/containers/119-sophia.md diff --git a/containers/120-mule-images.md b/knowledge/wiki/containers/120-mule-images.md similarity index 99% rename from containers/120-mule-images.md rename to knowledge/wiki/containers/120-mule-images.md index 69de592..893570c 100644 --- a/containers/120-mule-images.md +++ b/knowledge/wiki/containers/120-mule-images.md @@ -60,7 +60,7 @@ For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials ## Related - [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries -- [Authentik (124)](124-authentik.md) +- [Authentik (124)](106-auth-outpost.md) - [Caddy (121)](121-caddy.md) - [DNS](../infrastructure/dns.md) - [Auto-deploy](../infrastructure/auto-deploy.md) diff --git a/containers/121-caddy.md b/knowledge/wiki/containers/121-caddy.md similarity index 94% rename from containers/121-caddy.md rename to knowledge/wiki/containers/121-caddy.md index 3e7ff19..6f0d335 100644 --- a/containers/121-caddy.md +++ b/knowledge/wiki/containers/121-caddy.md @@ -37,9 +37,9 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L ## Snippet: `(authentik)` forward-auth -A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup). +A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](106-auth-outpost.md#forward-auth-domain-level-setup). -For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md). +For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](106-auth-outpost.md). ## Caddy environment @@ -60,7 +60,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip ## Related - [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain -- [Authentik (124)](124-authentik.md) — forward-auth + IdP +- [Authentik (124)](106-auth-outpost.md) — forward-auth + IdP - [Auto-deploy](../infrastructure/auto-deploy.md) - [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure - [Gitea (104)](104-gitea.md) — webhook source @@ -85,7 +85,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip - **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits - **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5 -Also: [elementsynapse LXC 118](../containers/118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed. +Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed. ### 2026-06-02 — caddy.service unit missing; recreated After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it. diff --git a/containers/122-arriman.md b/knowledge/wiki/containers/122-arriman.md similarity index 99% rename from containers/122-arriman.md rename to knowledge/wiki/containers/122-arriman.md index 1e5a0d4..0566b42 100644 --- a/containers/122-arriman.md +++ b/knowledge/wiki/containers/122-arriman.md @@ -115,7 +115,7 @@ Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has ## Related - [Caddy (121)](121-caddy.md) -- [Authentik (124)](124-authentik.md) — forward-auth wiring + per-app `/api/*` bypass +- [Authentik (124)](106-auth-outpost.md) — forward-auth wiring + per-app `/api/*` bypass - [DNS](../infrastructure/dns.md) - [Media permissions](../infrastructure/media-permissions.md) - [Hubris host](../hosts/hubris.md) diff --git a/containers/128-trmnl.md b/knowledge/wiki/containers/128-trmnl.md similarity index 98% rename from containers/128-trmnl.md rename to knowledge/wiki/containers/128-trmnl.md index e15e569..c56f466 100644 --- a/containers/128-trmnl.md +++ b/knowledge/wiki/containers/128-trmnl.md @@ -35,7 +35,7 @@ Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env - [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router) - [DNS (107)](107-dns.md) — Technitium A record `trmnl → 192.168.8.175` (LAN path via Caddy) - [Gitea (104)](104-gitea.md) — source repo `dtoro/terminalito` -- [Plan: 2026-06-24 TRMNL plugins LXC](../plans/2026-06-24-trmnl-plugins-lxc.md) +- [Plan: 2026-06-24 TRMNL plugins LXC](../../../plans/2026-06-24-trmnl-plugins-lxc.md) ## Changelog ### 2026-06-24 — auto-deploy + LAN DNS wired diff --git a/containers/129-house.md b/knowledge/wiki/containers/129-house.md similarity index 97% rename from containers/129-house.md rename to knowledge/wiki/containers/129-house.md index 333fabb..e2d112a 100644 --- a/containers/129-house.md +++ b/knowledge/wiki/containers/129-house.md @@ -40,7 +40,7 @@ Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modul - [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy) - [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`) - [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source -- [Deployment plan](../plans/2026-06-25-yuvomi-deployment.md) +- [Deployment plan](../../../plans/done/2026-06-25-yuvomi-deployment.md) ## Changelog diff --git a/containers/130-grimmory.md b/knowledge/wiki/containers/130-grimmory.md similarity index 98% rename from containers/130-grimmory.md rename to knowledge/wiki/containers/130-grimmory.md index 8800200..e9e7267 100644 --- a/containers/130-grimmory.md +++ b/knowledge/wiki/containers/130-grimmory.md @@ -55,7 +55,7 @@ LXC is privileged → in-container UID = host UID. Docker container gets media G - [apps (105)](105-apps.md) — previous host (Booklore) - [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.247:6060` -- [Authentik (124)](124-authentik.md) — OIDC provider `Grimmory` +- [Authentik (124)](106-auth-outpost.md) — OIDC provider `Grimmory` - [DNS (107)](107-dns.md) — `books.hubris.network → 192.168.8.175` (unchanged from Booklore) - [Media permissions](../infrastructure/media-permissions.md) diff --git a/containers/131-teddycloud.md b/knowledge/wiki/containers/131-teddycloud.md similarity index 94% rename from containers/131-teddycloud.md rename to knowledge/wiki/containers/131-teddycloud.md index 3531cab..e360403 100644 --- a/containers/131-teddycloud.md +++ b/knowledge/wiki/containers/131-teddycloud.md @@ -7,7 +7,7 @@ audio content against a local server instead of the official cloud. Predates the client-enrollment convention entirely; nobody wrote it down. Found and documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged `pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see -[OIKOS.md](../OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already +[OIKOS.md](../../../.agents/OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud` container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict for it — but it never got its own inventory entry or doc page until now. @@ -50,7 +50,7 @@ for the first time. - [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already taken by this container - [Containers index](index.md) -- [OIKOS.md](../OIKOS.md) — drift detector that caught this +- [OIKOS.md](../../../.agents/OIKOS.md) — drift detector that caught this ## Changelog diff --git a/containers/132-rclone.md b/knowledge/wiki/containers/132-rclone.md similarity index 99% rename from containers/132-rclone.md rename to knowledge/wiki/containers/132-rclone.md index 800adda..55b89f8 100644 --- a/containers/132-rclone.md +++ b/knowledge/wiki/containers/132-rclone.md @@ -95,7 +95,7 @@ after this set. Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH), rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to -[agent-enrollment.md troubleshooting](../operations/agent-enrollment.md#troubleshooting) if it recurs +[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs on future LXC bootstraps. ## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved) diff --git a/containers/133-seanime.md b/knowledge/wiki/containers/133-seanime.md similarity index 100% rename from containers/133-seanime.md rename to knowledge/wiki/containers/133-seanime.md diff --git a/containers/134-romm.md b/knowledge/wiki/containers/134-romm.md similarity index 100% rename from containers/134-romm.md rename to knowledge/wiki/containers/134-romm.md diff --git a/containers/archive/123-claudio-bot.md b/knowledge/wiki/containers/archive/123-claudio-bot.md similarity index 77% rename from containers/archive/123-claudio-bot.md rename to knowledge/wiki/containers/archive/123-claudio-bot.md index daf53bc..384bff3 100644 --- a/containers/archive/123-claudio-bot.md +++ b/knowledge/wiki/containers/archive/123-claudio-bot.md @@ -3,7 +3,7 @@ > **This LXC was destroyed on 2026-06-04.** Replaced by Hermes Agent on mac-mini. > Monitoring migrated to `homelab-hardware-health` skill + 15-min Hermes cronjob. > Repos `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived (read-only) on Gitea. -> See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) for full details. +> See [deprecation plan](../../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) for full details. Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications. @@ -17,7 +17,7 @@ Matrix-resident control plane. Bot account `@claudio:avispero` joined to a priva ## Stack -Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`). +Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](../118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`). Room: `!dEUVJArVKPorxHHJZK:avispero` (invite-only, `@dtoro:avispero` allowed). @@ -43,8 +43,8 @@ Currently set to `lmstudio` → `google/gemma-4-e4b` on the Mac mini at `192.168 ## IPC `http://192.168.8.230:9090/{notify,propose,status}`. Header `X-Bot-Token` must match `/etc/claudio-bot/ipc.token`. Used by: -- [claudio-monitor on hubris](../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`) -- The (currently disabled) [restic backup wrapper](../infrastructure/backups.md) (token in `/etc/restic/bot.token`) +- [claudio-monitor on hubris](../../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`) +- The (currently disabled) [restic backup wrapper](../../infrastructure/backups.md) (token in `/etc/restic/bot.token`) > Token files at the source side **must hold the same value as `/etc/claudio-bot/ipc.token`** — rotate together. @@ -61,13 +61,13 @@ Active plugins: Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/deploy` → pull + `pip install` + restart. Same shape as caddy-conf. -`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.230`. +`app.ini` `ALLOWED_HOST_LIST` on [gitea](../104-gitea.md) includes `192.168.8.230`. ## Related -- [elementsynapse (118)](118-elementsynapse.md) -- [Monitoring (claudio-monitor)](../infrastructure/monitoring.md) -- [Backups (disabled)](../infrastructure/backups.md) -- [Auto-deploy](../infrastructure/auto-deploy.md) +- [elementsynapse (118)](../118-elementsynapse.md) +- [Monitoring (claudio-monitor)](../../infrastructure/monitoring.md) +- [Backups (disabled)](../../infrastructure/backups.md) +- [Auto-deploy](../../infrastructure/auto-deploy.md) ## Changelog @@ -84,7 +84,7 @@ Initial documentation. `backend: lmstudio` → `google/gemma-4-e4b` on the Mac mini. Anthropic key still present so the swap is reversible by flipping the config field. ### 2026-04-21 — `monitor` plugin added -Receives events from [claudio-monitor](../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`. +Receives events from [claudio-monitor](../../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`. ### 2026-04-20 — claudio-bot deployed LXC 123 provisioned. Repo, systemd unit, Matrix wiring, `system` + `backup` plugins, IPC server. diff --git a/containers/archive/127-mule-photos-new.md b/knowledge/wiki/containers/archive/127-mule-photos-new.md similarity index 96% rename from containers/archive/127-mule-photos-new.md rename to knowledge/wiki/containers/archive/127-mule-photos-new.md index bbf1574..0966251 100644 --- a/containers/archive/127-mule-photos-new.md +++ b/knowledge/wiki/containers/archive/127-mule-photos-new.md @@ -1,7 +1,7 @@ # 127 — `mule-photos-new` Side-by-side **PhotoPrism M0 test** of the `dtoro/mule-image` `new` branch -at `photos-new.hubris.network`. Production [LXC 120](120-mule-images.md) keeps +at `photos-new.hubris.network`. Production [LXC 120](../120-mule-images.md) keeps running on the legacy stack at `photos.hubris.network` until M5 cutover. ## At a glance @@ -11,7 +11,7 @@ running on the legacy stack at `photos.hubris.network` until M5 cutover. - **Resources:** 6 cores / 8 GiB RAM / 40 GiB rootfs / 1 GiB swap - **Features:** `nesting=1,fuse=1,keyctl=1` - **Mounts:** *(none — see scratch copy below)* -- **Public hostname:** [`photos-new.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → split (PhotoPrism `:2342`, sidecar `:8000`, Vite `:5173`) +- **Public hostname:** [`photos-new.hubris.network`](../../infrastructure/dns.md) → [caddy (121)](../121-caddy.md) → split (PhotoPrism `:2342`, sidecar `:8000`, Vite `:5173`) ## Stack (`/opt/mule-image`) @@ -65,7 +65,7 @@ unprivileged LXCs can't see through. ## Auth — Authentik OIDC -PhotoPrism's "Sign in with OIDC" button delegates to [Authentik (124)](124-authentik.md). +PhotoPrism's "Sign in with OIDC" button delegates to [Authentik (124)](../106-auth-outpost.md). - **Provider/Application slug:** `mule-photos-new` - **Issuer:** `https://auth.hubris.network/application/o/mule-photos-new/` @@ -111,7 +111,7 @@ Mirrors the LXC 120 pattern. Push to the `new` branch on [git.hubris.network/dtoro/mule-image](http://git.hubris.network/dtoro/mule-image) → webhook fires → rebuild. The legacy LXC 120 watches `main` and is unaffected. **Gitea gotcha:** the receiver IP must be in `[webhook] ALLOWED_HOST_LIST` -in `/etc/gitea/app.ini` on [LXC 104](104-gitea.md). LXC 127's +in `/etc/gitea/app.ini` on [LXC 104](../104-gitea.md). LXC 127's `192.168.8.181` was missing on first bring-up; every push delivered status 0 with the message `webhook can only call allowed HTTP servers`. Adding the IP and `systemctl restart gitea` is enough — same list is @@ -145,10 +145,10 @@ curl -sk --resolve photos-new.hubris.network:443:192.168.8.175 \ ``` > **Decommissioned 2026-05-22.** The PhotoPrism + sidecar + SvelteKit stack -> validated here was promoted into production on [LXC 120](120-mule-images.md) +> validated here was promoted into production on [LXC 120](../120-mule-images.md) > via the `Mulimage 2.0` merge (`dtoro/mule-image` commit `70dc1b6`). This > page is retained for archaeology; everything below is historic. See the -> 2026-05-22 entry in [120-mule-images.md](120-mule-images.md#changelog) for +> 2026-05-22 entry in [120-mule-images.md](../120-mule-images.md#changelog) for > the cutover detail. ## Changelog diff --git a/containers/index.md b/knowledge/wiki/containers/index.md similarity index 93% rename from containers/index.md rename to knowledge/wiki/containers/index.md index ca8486d..9d034ac 100644 --- a/containers/index.md +++ b/knowledge/wiki/containers/index.md @@ -15,7 +15,7 @@ Most containers live on [`hubris`](../hosts/hubris.md). Some have been | 120 | [mule-images](120-mule-images.md) | hubris | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU) | `photos.hubris.network` | running | | 121 | [caddy](121-caddy.md) | hubris | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running | | 122 | [arriman](122-arriman.md) | **strong** | 192.168.8.245 | priv | 4 | 8 GiB | 24 GiB | `/mnt/media_local` (via mp0) | `jellyseerr` / `qbit` / `sab` | running | -| 124 | [authentik](124-authentik.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running | +| 124 | [authentik](106-auth-outpost.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running | | 128 | [trmnl](128-trmnl.md) | hubris | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running | | 129 | [house](129-house.md) | **strong** | 192.168.8.244 | unpriv | 2 | 3 GiB | 8 GiB | — | `house.hubris.network` | running | | 130 | [grimmory](130-grimmory.md) | **strong** | 192.168.8.247 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `books.hubris.network` | running | @@ -32,7 +32,7 @@ Most containers live on [`hubris`](../hosts/hubris.md). Some have been | 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose | | 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained | | 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology | -| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) | +| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) | | 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty | | 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq | | 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned | @@ -45,7 +45,7 @@ Most containers live on [`hubris`](../hosts/hubris.md). Some have been ## Conventions -- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](124-authentik.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range. +- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](106-auth-outpost.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range. - `onboot=1` on every container — the host brings them up after `pve-guests.service`. - Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library` on hubris, or `mp0: /mnt/media_local,mp=/mnt/library` on strong. - Most containers are privileged. Unprivileged ones require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard. diff --git a/hosts/hubris.md b/knowledge/wiki/hosts/hubris.md similarity index 92% rename from hosts/hubris.md rename to knowledge/wiki/hosts/hubris.md index d14e10a..5a063cc 100644 --- a/hosts/hubris.md +++ b/knowledge/wiki/hosts/hubris.md @@ -8,7 +8,7 @@ workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node ## At a glance - **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`) - **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`). -- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md). +- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). - **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`. - **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`. - **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP. @@ -46,7 +46,7 @@ Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md) it's a physical thinpool that only exists on this host's hardware. - strong currently hosts no LXCs/VMs — it exists solely as a cluster member so far. See [strong.md](strong.md) and the [library-SSD - migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) + migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) for what comes next (physical drive move, service migration — not started). ## Tenants @@ -113,20 +113,20 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220 - [Media permissions](../infrastructure/media-permissions.md) - [Monitoring](../infrastructure/monitoring.md) - [Backups (disabled)](../infrastructure/backups.md) -- [Operations cheatsheet](../operations/commands.md) -- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md) +- [Operations cheatsheet](../../../.agents/operations/commands.md) +- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md) - [strong — Proxmox host](strong.md) ## Changelog ### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab") -User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed. +User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed. ### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured -Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md). ### 2026-05-14 — LXC 109 (syncthing) decommissioned -User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md). +User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md). ### 2026-05-14 — network performance baseline captured First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above. @@ -138,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. `/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel. ### 2026-04-29 — relocated to better-ventilated spot -User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation). +User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation). ### 2026-04-28 — Phase 1 WiFi failover Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them. @@ -147,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since. ### 2026-04-23 — SSD cooling + thermal pads installed -Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). +Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). ### 2026-04-22 — drive removal A/B test -Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). +Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). ### 2026-04-22 — `cpu-epp.service` ordering bug fixed Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`. @@ -159,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind ` `60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else. ### 2026-04-21 — `cpu-epp.service` deployed -Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md). +Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). diff --git a/knowledge/wiki/hosts/index.md b/knowledge/wiki/hosts/index.md new file mode 100644 index 0000000..1c7ab00 --- /dev/null +++ b/knowledge/wiki/hosts/index.md @@ -0,0 +1,9 @@ +# Hosts + +The two Proxmox VE nodes that run the fleet. Narrative pages; the machine-readable host records are +the generated `hosts/*.yaml` files at the repo root. + +| Document | What it covers | +|----------|----------------| +| [hubris.md](hubris.md) | Primary PVE node (GMKtec NucBox M6 Ultra, `192.168.8.77`) — runs most LXCs plus the VMs. | +| [strong.md](strong.md) | Secondary PVE node / cluster member (`192.168.178.181`) — hosts the LXCs migrated from hubris. | diff --git a/hosts/strong.md b/knowledge/wiki/hosts/strong.md similarity index 95% rename from hosts/strong.md rename to knowledge/wiki/hosts/strong.md index 1778a4b..f1ce4c2 100644 --- a/hosts/strong.md +++ b/knowledge/wiki/hosts/strong.md @@ -28,7 +28,7 @@ Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet. was actually created live; not worth renaming), restricted `nodes strong` in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool. Empty so far; this is separate from the [library-SSD migration - plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s + plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s planned drive move from hubris (that hasn't happened) — this is general-purpose VM/CT capacity. - **Network:** `vmbr0` is bridged straight onto the household LAN — @@ -50,13 +50,13 @@ Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet. `bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the 5-min sync timer are live; `homelab whoami` resolves correctly. See - [agent-enrollment.md](../operations/agent-enrollment.md). + [agent-enrollment.md](../../../.agents/operations/agent-enrollment.md). - **Age key / secrets:** issued the same day over plain LAN (no Netbird needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at `/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4` recorded in `inventory.yaml`. Not yet a recipient on any actual secret (`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see - ["Granting a secret to a new client"](../operations/agent-enrollment.md#granting-a-secret-to-a-new-client). + ["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client). ## Cluster membership @@ -76,10 +76,10 @@ needed going forward. ## Related - [hubris — Proxmox host](hubris.md) -- [Library SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical) +- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical) - [Network](../infrastructure/network.md) - [SSH access](../infrastructure/ssh-access.md) -- [Agent enrollment](../operations/agent-enrollment.md) +- [Agent enrollment](../../../.agents/operations/agent-enrollment.md) ## Changelog diff --git a/infrastructure/auto-deploy.md b/knowledge/wiki/infrastructure/auto-deploy.md similarity index 92% rename from infrastructure/auto-deploy.md rename to knowledge/wiki/infrastructure/auto-deploy.md index f8fa9e6..00220e7 100644 --- a/infrastructure/auto-deploy.md +++ b/knowledge/wiki/infrastructure/auto-deploy.md @@ -23,7 +23,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web - ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~ - `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md)) - `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library) - - ~~`192.168.8.190` ([plato (126)](../containers/126-plato.md))~~ (destroyed 2026-06-28) + - ~~`192.168.8.190` ([plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology))~~ (destroyed 2026-06-28) - `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito) **Don't strip these when editing app.ini.** @@ -38,19 +38,19 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web | `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed | | `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` | | `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` | -| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/126-plato.md) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed | -| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed | +| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed | +| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed | | `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) | | `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart | | `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart | | `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` | -| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../oikos/console/deploy/README.md) | +| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) | > Note: `dtoro/Homelab-Docs` has **three webhooks** firing on the same push. > Each owns its own clone on LXC 105. They don't conflict because each > deploy.sh only touches its own service unit + venv. -> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md). +> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.yaml). DNS moved to [Technitium on dns (107)](../containers/107-dns.md). ## When you change a tracked config @@ -117,7 +117,7 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l - [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these - [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets - [Backups (disabled)](backups.md) -- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference +- [Operations cheatsheet](../../../.agents/operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference ## Changelog @@ -131,7 +131,7 @@ Webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` on [ Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo. ### 2026-05-13 — Plato pipeline added -Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`. +Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`. ### 2026-04-28 — wiki entry created Initial documentation. Six active pipelines. diff --git a/infrastructure/backups.md b/knowledge/wiki/infrastructure/backups.md similarity index 93% rename from infrastructure/backups.md rename to knowledge/wiki/infrastructure/backups.md index eba2dd7..e343fc8 100644 --- a/infrastructure/backups.md +++ b/knowledge/wiki/infrastructure/backups.md @@ -24,7 +24,7 @@ See [132-rclone](../containers/132-rclone.md) for the full design. ## Legacy — restic on external drive (DISABLED 2026-04-22) -Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md). +Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). ## Status @@ -36,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver). -**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). +**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). **To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive. @@ -111,7 +111,7 @@ Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again. -After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). +After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). ## Thermal monitoring @@ -119,10 +119,10 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213 ## Related - [Hubris host](../hosts/hubris.md) -- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) +- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) - [Monitoring](monitoring.md) - [Auto-deploy](auto-deploy.md) -- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md) +- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md) ## Changelog @@ -133,7 +133,7 @@ Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclon Initial documentation. Status remains DISABLED. ### 2026-04-22 — DISABLED -Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. +Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. ### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups. diff --git a/infrastructure/dns.md b/knowledge/wiki/infrastructure/dns.md similarity index 82% rename from infrastructure/dns.md rename to knowledge/wiki/infrastructure/dns.md index 011f7ee..b14567e 100644 --- a/infrastructure/dns.md +++ b/knowledge/wiki/infrastructure/dns.md @@ -7,7 +7,7 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr ## Components - **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22). -- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04). +- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/106-auth-outpost.md) (decommissioned 2026-06-04). - **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns. - **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where. @@ -53,7 +53,7 @@ Creating a new Caddyfile site block is necessary but **not sufficient**. Without 3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`. 4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`. -> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync). +> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../../../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync). ## Public path — what does and doesn't follow the LAN map @@ -94,7 +94,7 @@ The "delete NetBird managed zone → forward everything to Technitium" plan was **Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group. -> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped). +> Reference: [scripts/dns-sync.py](../../../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped). ### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment) The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging. @@ -107,10 +107,10 @@ Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backe All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog). ### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired -Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients. +Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients. ### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`) -Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS). +Authentik migrated off LXC 124 onto the VPS (see [investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS). ### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP) NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful. @@ -119,7 +119,7 @@ NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192. New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS). ### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed -New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend). +New LAN-only entry for [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend). ### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13) Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later. diff --git a/infrastructure/homelab-context.md b/knowledge/wiki/infrastructure/homelab-context.md similarity index 96% rename from infrastructure/homelab-context.md rename to knowledge/wiki/infrastructure/homelab-context.md index fb58481..cafcce5 100644 --- a/infrastructure/homelab-context.md +++ b/knowledge/wiki/infrastructure/homelab-context.md @@ -5,7 +5,7 @@ Code, Hermes Agent, future MCP-capable clients) on every machine in the lab self-locating and able to read the same source of truth. Operational walkthrough for enrolling a new client lives in -[operations/agent-enrollment.md](../operations/agent-enrollment.md); this +[operations/agent-enrollment.md](../../../.agents/operations/agent-enrollment.md); this page is the architecture reference. ## What's where @@ -121,7 +121,7 @@ The MCP server and secrets-issuance each have their own clone ## Related -- [Operations: agent enrollment](../operations/agent-enrollment.md) — the +- [Operations: agent enrollment](../../../.agents/operations/agent-enrollment.md) — the step-by-step for adding a new client - [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance` pipelines (and the rest of the lab's webhook pipelines) @@ -133,7 +133,7 @@ The MCP server and secrets-issuance each have their own clone ## Changelog ### 2026-05-20 — system live across hubris, apps, republic-laptop -Phase 1 of the [cross-client context plan](../README.md) merged. Three +Phase 1 of the [cross-client context plan](../../../README.md) merged. Three clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS decrypt verified on each. Webhook auto-deploy for both LXC 105 services wired (hook ids 10 + 11). `homelab refresh-creds` + atomic diff --git a/infrastructure/index.md b/knowledge/wiki/infrastructure/index.md similarity index 91% rename from infrastructure/index.md rename to knowledge/wiki/infrastructure/index.md index 3b86737..f97ddf3 100644 --- a/infrastructure/index.md +++ b/knowledge/wiki/infrastructure/index.md @@ -36,7 +36,7 @@ are documented in their own pages. Each system below links to its full doc. ## Identity & access -- **[Authentik SSO](../containers/124-authentik.md)** — identity provider. +- **[Authentik SSO](../containers/106-auth-outpost.md)** — identity provider. Core server runs on the VPS; LAN forward-auth outpost at LXC 106. OIDC providers configured for Jellyfin, Jellyseerr, Sabnzbd, qBittorrent, Yuvomi, and more. @@ -60,7 +60,7 @@ are documented in their own pages. Each system below links to its full doc. ## Related -- [README](../README.md) — entry point +- [README](../../../README.md) — entry point - [Containers index](../containers/index.md) -- [Operations cheatsheet](../operations/commands.md) -- [OIKOS operating model](../OIKOS.md) \ No newline at end of file +- [Operations cheatsheet](../../../.agents/operations/commands.md) +- [OIKOS operating model](../../../.agents/OIKOS.md) \ No newline at end of file diff --git a/infrastructure/ingress.md b/knowledge/wiki/infrastructure/ingress.md similarity index 90% rename from infrastructure/ingress.md rename to knowledge/wiki/infrastructure/ingress.md index cb06518..08ff186 100644 --- a/infrastructure/ingress.md +++ b/knowledge/wiki/infrastructure/ingress.md @@ -49,7 +49,7 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8 ### `auth.hubris.network` — different pattern (local container, not cert-mirror) -Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above: +Since 2026-05-31 [Authentik runs on the VPS itself](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above: - Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`. - TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough). @@ -91,7 +91,7 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for ` TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed. ### 2026-05-31 — `auth.hubris.network` now served locally on the VPS -Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack. +Authentik migrated onto the VPS ([investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack. ### 2026-04-28 — wiki entry created Initial documentation. diff --git a/infrastructure/media-permissions.md b/knowledge/wiki/infrastructure/media-permissions.md similarity index 100% rename from infrastructure/media-permissions.md rename to knowledge/wiki/infrastructure/media-permissions.md diff --git a/infrastructure/mesh.md b/knowledge/wiki/infrastructure/mesh.md similarity index 95% rename from infrastructure/mesh.md rename to knowledge/wiki/infrastructure/mesh.md index 1d7d78b..c855bf6 100644 --- a/infrastructure/mesh.md +++ b/knowledge/wiki/infrastructure/mesh.md @@ -109,7 +109,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man ## Related - [DNS split-horizon](dns.md) -- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides +- [Authentik (124)](../containers/106-auth-outpost.md) — the IdP that triggers most of these overrides - [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B - [Gitea (104)](../containers/104-gitea.md) — example of Technique A - [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport @@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man ## Changelog ### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below) -The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md). +The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md). ### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik @@ -139,11 +139,11 @@ The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Ca ### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN. -Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d. +Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/106-auth-outpost.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d. Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe). -The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/124-authentik.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers. +The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/106-auth-outpost.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers. **Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21): diff --git a/infrastructure/monitoring.md b/knowledge/wiki/infrastructure/monitoring.md similarity index 100% rename from infrastructure/monitoring.md rename to knowledge/wiki/infrastructure/monitoring.md diff --git a/infrastructure/network.md b/knowledge/wiki/infrastructure/network.md similarity index 93% rename from infrastructure/network.md rename to knowledge/wiki/infrastructure/network.md index 54ded7b..99f4c08 100644 --- a/infrastructure/network.md +++ b/knowledge/wiki/infrastructure/network.md @@ -64,7 +64,7 @@ No NAT on Proxmox — traffic flows without double-NAT. ## Remote access -- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../vps/). +- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../../../vps/). - **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md). ## Related @@ -79,10 +79,10 @@ No NAT on Proxmox — traffic flows without double-NAT. ### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2) Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed. -Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md). ### 2026-06-02 — Executed migration; Proxmox as subnet router -Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md). ### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added -Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md). +Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md). diff --git a/infrastructure/ssh-access.md b/knowledge/wiki/infrastructure/ssh-access.md similarity index 98% rename from infrastructure/ssh-access.md rename to knowledge/wiki/infrastructure/ssh-access.md index 44c05eb..f412a0a 100644 --- a/infrastructure/ssh-access.md +++ b/knowledge/wiki/infrastructure/ssh-access.md @@ -178,8 +178,8 @@ done - [Mesh migration](mesh.md) - [VPS hardening](vps-hardening.md) -- [Agent enrollment](../operations/agent-enrollment.md) -- [Homelab CLI](../bin/homelab) +- [Agent enrollment](../../../.agents/operations/agent-enrollment.md) +- [Homelab CLI](../../../bin/homelab) ## Changelog diff --git a/infrastructure/topology.md b/knowledge/wiki/infrastructure/topology.md similarity index 95% rename from infrastructure/topology.md rename to knowledge/wiki/infrastructure/topology.md index ee1764d..5bb1a9d 100644 --- a/infrastructure/topology.md +++ b/knowledge/wiki/infrastructure/topology.md @@ -3,8 +3,8 @@ # Topology (generated) -Source: [inventory.yaml](../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services. -Edge semantics: [oikos/ontology.yaml](../oikos/ontology.yaml). Operating model: [OIKOS.md](../OIKOS.md). +Source: [inventory.yaml](../../../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services. +Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). Operating model: [OIKOS.md](../../../.agents/OIKOS.md). ## Compute & ingress diff --git a/infrastructure/vps-hardening.md b/knowledge/wiki/infrastructure/vps-hardening.md similarity index 100% rename from infrastructure/vps-hardening.md rename to knowledge/wiki/infrastructure/vps-hardening.md diff --git a/vms/100-zimaos.md b/knowledge/wiki/vms/100-zimaos.md similarity index 94% rename from vms/100-zimaos.md rename to knowledge/wiki/vms/100-zimaos.md index 54a7959..ac68f42 100644 --- a/vms/100-zimaos.md +++ b/knowledge/wiki/vms/100-zimaos.md @@ -46,7 +46,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. ## Open items - **DHCP → static IP fixed (2026-06-03).** ZimaOS IP drifted from `.195` (Slate AX) → `.103` (Technitium) after the DHCP migration, causing Caddy 502s. Fixed by injecting a static systemd-networkd config and restarting the VM. IP now pinned at `192.168.8.195`. See [changelog](#2026-06-03--static-ip-set-to-195-dhcp-drift-fixed). -- **No Authentik wiring.** [authentik (124)](../containers/124-authentik.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists. +- **No Authentik wiring.** [authentik (124)](../containers/106-auth-outpost.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists. - **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up. - **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor` → `screendump`. @@ -59,7 +59,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. ## Changelog ### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed -ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md). +ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md). ### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix) @@ -87,4 +87,4 @@ Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs mod Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`. ### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1) -`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/124-authentik.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation. +`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/106-auth-outpost.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation. diff --git a/vms/108-haos.md b/knowledge/wiki/vms/108-haos.md similarity index 86% rename from vms/108-haos.md rename to knowledge/wiki/vms/108-haos.md index 6ac7ca6..90c78a7 100644 --- a/vms/108-haos.md +++ b/knowledge/wiki/vms/108-haos.md @@ -7,7 +7,7 @@ Home Assistant OS — the only VM on hubris (HAOS doesn't run cleanly in an LXC, - **HAOS version:** 16.3 (last verified) - **IP:** `192.168.8.101` - **Resources:** 4 GiB RAM, 32 GiB boot disk -- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `192.168.8.101:8123` +- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.101:8123` ## Auth @@ -18,7 +18,7 @@ Key gotchas: ``` ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1" ``` - so OIDC discovery resolves internally to [authentik (124)](124-authentik.md). + so OIDC discovery resolves internally to [authentik (124)](../containers/106-auth-outpost.md). - Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen). - YAML config: - `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created). @@ -30,8 +30,8 @@ Key gotchas: HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting. ## Related -- [Authentik (124)](124-authentik.md) -- [Caddy (121)](121-caddy.md) +- [Authentik (124)](../containers/106-auth-outpost.md) +- [Caddy (121)](../containers/121-caddy.md) - [DNS](../infrastructure/dns.md) - [Monitoring](../infrastructure/monitoring.md) diff --git a/vms/index.md b/knowledge/wiki/vms/index.md similarity index 94% rename from vms/index.md rename to knowledge/wiki/vms/index.md index ad649c1..f61daae 100644 --- a/vms/index.md +++ b/knowledge/wiki/vms/index.md @@ -11,4 +11,4 @@ Two QEMU VMs running on [hubris](../hosts/hubris.md): - [Hubris host](../hosts/hubris.md) — both VMs run here - [Containers index](../containers/index.md) — LXCs on both nodes -- [README](../README.md) \ No newline at end of file +- [README](../../../README.md) \ No newline at end of file diff --git a/oikos/approve.py b/oikos/approve.py index a10fd47..89449b5 100644 --- a/oikos/approve.py +++ b/oikos/approve.py @@ -5,7 +5,7 @@ Repo-side half of the Week-3 approval flow. This module owns the request/ grant lifecycle and the HMAC signing; it does NOT talk to Matrix directly. There is no dedicated Matrix bot in this homelab — alerts already go out as the operator's own Hermes agent posting to @dtoro:avispero (see -infrastructure/monitoring.md's homelab-health-watchdog). The integration +knowledge/wiki/infrastructure/monitoring.md's homelab-health-watchdog). The integration contract is: 1. An agent or the Week-3 scheduler calls `request()` (or the CLI diff --git a/oikos/cards/host-apps.md b/oikos/cards/host-apps.md index 038fa80..4afe8ac 100644 --- a/oikos/cards/host-apps.md +++ b/oikos/cards/host-apps.md @@ -6,7 +6,7 @@ - role: docker-apps - address: 192.168.8.205 (mesh: tailscale:apps) - mounts: /mnt/library -- doc: containers/105-apps.md +- doc: knowledge/wiki/containers/105-apps.md - secrets: enrolled (age key present) ## Blast radius @@ -18,4 +18,4 @@ - see the services this host runs for action-level risk classes ## Recent changes -- (none yet) +- 2026-07-06T11:57:21+00:00 deploy-oikos-console (config_mutation) — ok diff --git a/oikos/cards/host-arriman.md b/oikos/cards/host-arriman.md index f641fa5..5db5481 100644 --- a/oikos/cards/host-arriman.md +++ b/oikos/cards/host-arriman.md @@ -6,7 +6,7 @@ - role: arr-stack - address: 192.168.8.245 (mesh: tailscale:arr) - mounts: /mnt/media_local -- doc: containers/122-arriman.md +- doc: knowledge/wiki/containers/122-arriman.md ## Blast radius - impacts: service:arr_stack diff --git a/oikos/cards/host-auth-outpost.md b/oikos/cards/host-auth-outpost.md index b450032..525c654 100644 --- a/oikos/cards/host-auth-outpost.md +++ b/oikos/cards/host-auth-outpost.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: authentik-gateway - address: 192.168.8.6 -- doc: containers/106-auth-outpost.md +- doc: knowledge/wiki/containers/106-auth-outpost.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-caddy.md b/oikos/cards/host-caddy.md index 73a53a3..ee80bca 100644 --- a/oikos/cards/host-caddy.md +++ b/oikos/cards/host-caddy.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: reverse-proxy - address: 192.168.8.175 -- doc: containers/121-caddy.md +- doc: knowledge/wiki/containers/121-caddy.md ## Blast radius - impacts: service:caddy diff --git a/oikos/cards/host-dns.md b/oikos/cards/host-dns.md index 7157d47..0b76c3f 100644 --- a/oikos/cards/host-dns.md +++ b/oikos/cards/host-dns.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: dns-server - address: 192.168.8.2 -- doc: containers/107-dns.md +- doc: knowledge/wiki/containers/107-dns.md ## Blast radius - impacts: service:dns @@ -16,4 +16,4 @@ - see the services this host runs for action-level risk classes ## Recent changes -- (none yet) +- 2026-07-06T11:40:28+00:00 add-record (config_mutation) — ok diff --git a/oikos/cards/host-elementsynapse.md b/oikos/cards/host-elementsynapse.md index 461ace3..6509c96 100644 --- a/oikos/cards/host-elementsynapse.md +++ b/oikos/cards/host-elementsynapse.md @@ -5,7 +5,7 @@ - runs-on: host:strong - role: matrix-server - address: 192.168.8.242 -- doc: containers/118-elementsynapse.md +- doc: knowledge/wiki/containers/118-elementsynapse.md ## Blast radius - impacts: service:matrix diff --git a/oikos/cards/host-gitea.md b/oikos/cards/host-gitea.md index ad600a6..a77eb7f 100644 --- a/oikos/cards/host-gitea.md +++ b/oikos/cards/host-gitea.md @@ -6,7 +6,7 @@ - role: git-server - address: 192.168.8.121 (mesh: tailscale:gitea) - mounts: /mnt/library -- doc: containers/104-gitea.md +- doc: knowledge/wiki/containers/104-gitea.md ## Blast radius - impacts: service:gitea diff --git a/oikos/cards/host-grimmory.md b/oikos/cards/host-grimmory.md index d6a0c72..706377c 100644 --- a/oikos/cards/host-grimmory.md +++ b/oikos/cards/host-grimmory.md @@ -6,7 +6,7 @@ - role: book-library - address: 192.168.8.247 - mounts: /mnt/media_local -- doc: containers/130-grimmory.md +- doc: knowledge/wiki/containers/130-grimmory.md - secrets: enrolled (age key present) ## Blast radius diff --git a/oikos/cards/host-haos.md b/oikos/cards/host-haos.md index ada4e76..45f2aba 100644 --- a/oikos/cards/host-haos.md +++ b/oikos/cards/host-haos.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: home-automation - address: 192.168.8.101 (mesh: tailscale:homeassistant) -- doc: vms/108-haos.md +- doc: knowledge/wiki/vms/108-haos.md ## Blast radius - impacts: service:haos diff --git a/oikos/cards/host-house.md b/oikos/cards/host-house.md index 3ad239e..5f3808c 100644 --- a/oikos/cards/host-house.md +++ b/oikos/cards/host-house.md @@ -5,7 +5,7 @@ - runs-on: host:strong - role: family-planner - address: 192.168.8.244 -- doc: containers/129-house.md +- doc: knowledge/wiki/containers/129-house.md - secrets: enrolled (age key present) ## Blast radius diff --git a/oikos/cards/host-hubris.md b/oikos/cards/host-hubris.md index 9ae7e5d..57236b6 100644 --- a/oikos/cards/host-hubris.md +++ b/oikos/cards/host-hubris.md @@ -5,7 +5,7 @@ - role: hypervisor - address: 192.168.8.77 (mesh: netbird:proxmox-server.netbird.selfhosted) - mounts: /mnt/library -- doc: hosts/hubris.md +- doc: knowledge/wiki/hosts/hubris.md - secrets: enrolled (age key present) ## Blast radius diff --git a/oikos/cards/host-jellyfin.md b/oikos/cards/host-jellyfin.md index 162b908..178c58d 100644 --- a/oikos/cards/host-jellyfin.md +++ b/oikos/cards/host-jellyfin.md @@ -6,7 +6,7 @@ - role: media-server - address: 192.168.8.246 (mesh: tailscale:jellyfin) - mounts: /mnt/media_local -- doc: containers/101-jellyfin.md +- doc: knowledge/wiki/containers/101-jellyfin.md ## Blast radius - impacts: service:jellyfin diff --git a/oikos/cards/host-mule-images.md b/oikos/cards/host-mule-images.md index 242f193..c1c98ca 100644 --- a/oikos/cards/host-mule-images.md +++ b/oikos/cards/host-mule-images.md @@ -6,7 +6,7 @@ - role: photo-management - address: 192.168.8.136 (mesh: tailscale:muleimage) - mounts: /mnt/library -- doc: containers/120-mule-images.md +- doc: knowledge/wiki/containers/120-mule-images.md ## Blast radius - impacts: service:photos diff --git a/oikos/cards/host-nextcloud.md b/oikos/cards/host-nextcloud.md index f3de0e1..2cd8e3b 100644 --- a/oikos/cards/host-nextcloud.md +++ b/oikos/cards/host-nextcloud.md @@ -6,7 +6,7 @@ - role: file-sync - address: 192.168.8.224 (mesh: tailscale:nextcloud) - mounts: /mnt/library -- doc: containers/114-nextcloud.md +- doc: knowledge/wiki/containers/114-nextcloud.md ## Blast radius - impacts: service:nextcloud diff --git a/oikos/cards/host-nfs-export.md b/oikos/cards/host-nfs-export.md index 53102e9..3731f13 100644 --- a/oikos/cards/host-nfs-export.md +++ b/oikos/cards/host-nfs-export.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: storage-export - address: 192.168.8.200 -- doc: containers/102-nfs-export.md +- doc: knowledge/wiki/containers/102-nfs-export.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-paperless.md b/oikos/cards/host-paperless.md index f21f1ab..f993551 100644 --- a/oikos/cards/host-paperless.md +++ b/oikos/cards/host-paperless.md @@ -6,7 +6,7 @@ - role: document-archive - address: 192.168.8.130 (mesh: tailscale:paperless) - mounts: /mnt/library -- doc: containers/103-paperless.md +- doc: knowledge/wiki/containers/103-paperless.md ## Blast radius - impacts: service:paperless diff --git a/oikos/cards/host-romm.md b/oikos/cards/host-romm.md index 6e9d58b..31fd618 100644 --- a/oikos/cards/host-romm.md +++ b/oikos/cards/host-romm.md @@ -6,7 +6,7 @@ - role: rom-manager - address: 192.168.8.249 - mounts: /mnt/media_local -- doc: containers/134-romm.md +- doc: knowledge/wiki/containers/134-romm.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-seanime.md b/oikos/cards/host-seanime.md index d2d58eb..155d991 100644 --- a/oikos/cards/host-seanime.md +++ b/oikos/cards/host-seanime.md @@ -6,7 +6,7 @@ - role: anime-media-server - address: 192.168.8.248 - mounts: /mnt/media_local/anime -- doc: containers/133-seanime.md +- doc: knowledge/wiki/containers/133-seanime.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-sophia.md b/oikos/cards/host-sophia.md index 505ebef..196efef 100644 --- a/oikos/cards/host-sophia.md +++ b/oikos/cards/host-sophia.md @@ -6,7 +6,7 @@ - role: workshop - address: 192.168.8.109 (mesh: tailscale:sophia) - mounts: /mnt/library -- doc: containers/119-sophia.md +- doc: knowledge/wiki/containers/119-sophia.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/host-strong.md b/oikos/cards/host-strong.md index 2ada6c4..9b921c1 100644 --- a/oikos/cards/host-strong.md +++ b/oikos/cards/host-strong.md @@ -4,7 +4,7 @@ - state: active - role: hypervisor - address: 192.168.178.181 -- doc: hosts/strong.md +- doc: knowledge/wiki/hosts/strong.md - secrets: enrolled (age key present) ## Blast radius diff --git a/oikos/cards/host-teddycloud.md b/oikos/cards/host-teddycloud.md index f496a1e..57bc992 100644 --- a/oikos/cards/host-teddycloud.md +++ b/oikos/cards/host-teddycloud.md @@ -6,7 +6,7 @@ - role: teddycloud - address: 192.168.8.150 - mounts: /mnt/library -- doc: containers/131-teddycloud.md +- doc: knowledge/wiki/containers/131-teddycloud.md ## Blast radius - impacts: service:teddycloud @@ -17,4 +17,4 @@ - see the services this host runs for action-level risk classes ## Recent changes -- (none yet) +- 2026-07-06T11:05:35+00:00 activate (config_mutation) — ok diff --git a/oikos/cards/host-trmnl.md b/oikos/cards/host-trmnl.md index d5d8d33..741b975 100644 --- a/oikos/cards/host-trmnl.md +++ b/oikos/cards/host-trmnl.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: trmnl-middleware - address: 192.168.8.211 -- doc: containers/128-trmnl.md +- doc: knowledge/wiki/containers/128-trmnl.md ## Blast radius - impacts: service:trmnl diff --git a/oikos/cards/host-zimaos.md b/oikos/cards/host-zimaos.md index 877b8f1..b249fa4 100644 --- a/oikos/cards/host-zimaos.md +++ b/oikos/cards/host-zimaos.md @@ -5,7 +5,7 @@ - runs-on: host:hubris - role: nas-frontend-eval - address: 192.168.8.195 -- doc: vms/100-zimaos.md +- doc: knowledge/wiki/vms/100-zimaos.md ## Blast radius - impacts: service:zimaos diff --git a/oikos/cards/service-arr_stack.md b/oikos/cards/service-arr_stack.md index 335f5c5..2c09e3b 100644 --- a/oikos/cards/service-arr_stack.md +++ b/oikos/cards/service-arr_stack.md @@ -1,7 +1,7 @@ # arr_stack (service:arr_stack) - backend: host:arriman -- doc: containers/122-arriman.md +- doc: knowledge/wiki/containers/122-arriman.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/service-artifacto.md b/oikos/cards/service-artifacto.md index 756c158..fc857a3 100644 --- a/oikos/cards/service-artifacto.md +++ b/oikos/cards/service-artifacto.md @@ -2,7 +2,7 @@ - backend: host:apps - url: https://artifacto.hubris.network -- doc: containers/105-apps.md +- doc: knowledge/wiki/containers/105-apps.md - config repo: dtoro/Artifacto ## Blast radius diff --git a/oikos/cards/service-authentik.md b/oikos/cards/service-authentik.md index 252ddea..93ec04b 100644 --- a/oikos/cards/service-authentik.md +++ b/oikos/cards/service-authentik.md @@ -2,7 +2,7 @@ - backend: host:netbird-vps - url: https://auth.hubris.network -- doc: containers/106-auth-outpost.md +- doc: knowledge/wiki/containers/106-auth-outpost.md - risk notes: SSO provider — outage locks login to OIDC/forward-auth services ## Blast radius diff --git a/oikos/cards/service-caddy.md b/oikos/cards/service-caddy.md index 69e013d..a26093d 100644 --- a/oikos/cards/service-caddy.md +++ b/oikos/cards/service-caddy.md @@ -1,7 +1,7 @@ # caddy (service:caddy) - backend: host:caddy -- doc: containers/121-caddy.md +- doc: knowledge/wiki/containers/121-caddy.md - config repo: dtoro/caddy-conf - risk notes: wide blast radius — every *.hubris.network route rides on it (see oikos/policy.yaml service_overrides) @@ -17,4 +17,4 @@ - edit-config-and-deploy — config_mutation (approval: operator) ## Recent changes -- (none yet) +- 2026-07-06T11:29:56+00:00 add-site-block (config_mutation) — ok diff --git a/oikos/cards/service-dns.md b/oikos/cards/service-dns.md index 1b3af2e..a7a3d9e 100644 --- a/oikos/cards/service-dns.md +++ b/oikos/cards/service-dns.md @@ -1,7 +1,7 @@ # dns (service:dns) - backend: host:dns -- doc: containers/107-dns.md +- doc: knowledge/wiki/containers/107-dns.md - risk notes: LAN-wide resolver — misconfig breaks name resolution for every client ## Blast radius diff --git a/oikos/cards/service-gitea.md b/oikos/cards/service-gitea.md index ab6e9bf..89dca93 100644 --- a/oikos/cards/service-gitea.md +++ b/oikos/cards/service-gitea.md @@ -2,7 +2,7 @@ - backend: host:gitea - url: https://git.hubris.network -- doc: containers/104-gitea.md +- doc: knowledge/wiki/containers/104-gitea.md - config repo: dtoro/gitea-customizations - risk notes: hosts all config repos + deploy webhooks; outage blocks auto-deploy and sync diff --git a/oikos/cards/service-haos.md b/oikos/cards/service-haos.md index 60e6473..4c3401b 100644 --- a/oikos/cards/service-haos.md +++ b/oikos/cards/service-haos.md @@ -1,7 +1,7 @@ # haos (service:haos) - backend: host:haos -- doc: vms/108-haos.md +- doc: knowledge/wiki/vms/108-haos.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/service-homelab_mcp.md b/oikos/cards/service-homelab_mcp.md index 29462c6..fb5a156 100644 --- a/oikos/cards/service-homelab_mcp.md +++ b/oikos/cards/service-homelab_mcp.md @@ -2,7 +2,7 @@ - backend: host:apps - url: https://mcp.hubris.network/mcp -- doc: infrastructure/homelab-context.md +- doc: knowledge/wiki/infrastructure/homelab-context.md - config repo: dtoro/Homelab-Docs - risk notes: agents' primary read surface — outage degrades every agent to grepping the clone diff --git a/oikos/cards/service-jellyfin.md b/oikos/cards/service-jellyfin.md index 75085ca..e12e850 100644 --- a/oikos/cards/service-jellyfin.md +++ b/oikos/cards/service-jellyfin.md @@ -2,7 +2,7 @@ - backend: host:jellyfin - url: https://media.hubris.network -- doc: containers/101-jellyfin.md +- doc: knowledge/wiki/containers/101-jellyfin.md - risk notes: native Authentik OIDC via SSO-Auth plugin, no Caddy forward-auth gate; VAAPI transcode depends on GPU passthrough on strong ## Blast radius diff --git a/oikos/cards/service-matrix.md b/oikos/cards/service-matrix.md index 6d109a4..d321ee2 100644 --- a/oikos/cards/service-matrix.md +++ b/oikos/cards/service-matrix.md @@ -2,7 +2,7 @@ - backend: host:elementsynapse - url: https://matrix.hubris.network -- doc: containers/118-elementsynapse.md +- doc: knowledge/wiki/containers/118-elementsynapse.md - risk notes: alert/approval channel for Oikos — outage silences agent escalation ## Blast radius diff --git a/oikos/cards/service-nextcloud.md b/oikos/cards/service-nextcloud.md index 88d1df8..995644b 100644 --- a/oikos/cards/service-nextcloud.md +++ b/oikos/cards/service-nextcloud.md @@ -2,7 +2,7 @@ - backend: host:nextcloud - url: https://cloud.hubris.network -- doc: containers/114-nextcloud.md +- doc: knowledge/wiki/containers/114-nextcloud.md ## Blast radius - impacts: (none) diff --git a/oikos/cards/service-paperless.md b/oikos/cards/service-paperless.md index 7f0c516..cc02065 100644 --- a/oikos/cards/service-paperless.md +++ b/oikos/cards/service-paperless.md @@ -2,7 +2,7 @@ - backend: host:paperless - url: https://paperless.hubris.network -- doc: containers/103-paperless.md +- doc: knowledge/wiki/containers/103-paperless.md - risk notes: document archive — treat data as irreplaceable; DB operations are destructive-class ## Blast radius diff --git a/oikos/cards/service-photos.md b/oikos/cards/service-photos.md index 76b64d8..a8f9442 100644 --- a/oikos/cards/service-photos.md +++ b/oikos/cards/service-photos.md @@ -2,7 +2,7 @@ - backend: host:mule-images - url: https://photos.hubris.network -- doc: containers/120-mule-images.md +- doc: knowledge/wiki/containers/120-mule-images.md - config repo: dtoro/mule-image ## Blast radius diff --git a/oikos/cards/service-proxmox_ui.md b/oikos/cards/service-proxmox_ui.md index 4c394ee..a83de8c 100644 --- a/oikos/cards/service-proxmox_ui.md +++ b/oikos/cards/service-proxmox_ui.md @@ -2,7 +2,7 @@ - backend: host:hubris - url: https://proxmox.hubris.network -- doc: hosts/hubris.md +- doc: knowledge/wiki/hosts/hubris.md - risk notes: hypervisor UI — changes here affect every guest on the node ## Blast radius diff --git a/oikos/cards/service-secrets_issuance.md b/oikos/cards/service-secrets_issuance.md index 6ceb33e..1178cbe 100644 --- a/oikos/cards/service-secrets_issuance.md +++ b/oikos/cards/service-secrets_issuance.md @@ -2,7 +2,7 @@ - backend: host:apps - url: https://secrets.hubris.network/issue -- doc: operations/agent-enrollment.md +- doc: .agents/operations/agent-enrollment.md - config repo: dtoro/Homelab-Docs - risk notes: identity issuance — any change is security-sensitive; key operations are destructive-class diff --git a/oikos/cards/service-teddycloud.md b/oikos/cards/service-teddycloud.md index 23eb88d..1533be8 100644 --- a/oikos/cards/service-teddycloud.md +++ b/oikos/cards/service-teddycloud.md @@ -2,7 +2,7 @@ - backend: host:teddycloud - url: https://teddy.hubris.network -- doc: containers/131-teddycloud.md +- doc: knowledge/wiki/containers/131-teddycloud.md - risk notes: no Caddy forward-auth gate (unlike sab.hubris.network on the same Caddyfile) — reachable to anyone on the LAN/mesh who can resolve teddy.hubris.network; undocumented in inventory.yaml until 2026-07-06 (drift-caught) ## Blast radius diff --git a/oikos/cards/service-trmnl.md b/oikos/cards/service-trmnl.md index 690d35e..29e9540 100644 --- a/oikos/cards/service-trmnl.md +++ b/oikos/cards/service-trmnl.md @@ -2,7 +2,7 @@ - backend: host:trmnl - url: https://trmnl.hubris.network -- doc: containers/128-trmnl.md +- doc: knowledge/wiki/containers/128-trmnl.md - config repo: dtoro/terminalito ## Blast radius diff --git a/oikos/cards/service-zimaos.md b/oikos/cards/service-zimaos.md index f834747..6c8f048 100644 --- a/oikos/cards/service-zimaos.md +++ b/oikos/cards/service-zimaos.md @@ -2,7 +2,7 @@ - backend: host:zimaos - url: https://zimaos.hubris.network -- doc: vms/100-zimaos.md +- doc: knowledge/wiki/vms/100-zimaos.md ## Blast radius - impacts: (none) diff --git a/oikos/console/deploy/README.md b/oikos/console/deploy/README.md index ce69fc5..f3e8283 100644 --- a/oikos/console/deploy/README.md +++ b/oikos/console/deploy/README.md @@ -3,7 +3,7 @@ Deploys the same way `homelab-mcp` and `secrets-issuance` already do: Shape B webhook (own checkout, own systemd units, own deploy secret) on LXC 105 (apps), reading `HOMELAB_CONTEXT_DIR=/opt/homelab-context` for -all data. See [infrastructure/auto-deploy.md](../../../infrastructure/auto-deploy.md) +all data. See [infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for the general pattern; webhook ids 10 (homelab-mcp, :9811) and 11 (secrets-issuance, :9821) are the direct precedent — this is a third webhook on `dtoro/Homelab-Docs`, port :9831. diff --git a/oikos/drift.py b/oikos/drift.py index 513ed9d..8e598f9 100644 --- a/oikos/drift.py +++ b/oikos/drift.py @@ -215,7 +215,7 @@ def check_pct_list(inv: dict | None = None) -> list[dict]: def check_caddy_backends(inv: dict | None = None) -> list[dict]: """Caddy's /etc/caddy (a git checkout of dtoro/caddy-conf, per - containers/121-caddy.md) vs inventory service backend IPs. Best-effort + knowledge/wiki/containers/121-caddy.md) vs inventory service backend IPs. Best-effort grep for reverse_proxy targets; skips services whose Caddyfile snippet doesn't use a bare IP (e.g. references a Caddy snippet/import).""" inv = inv or _load_inventory() diff --git a/oikos/gen-topology.py b/oikos/gen-topology.py index 87505d8..ca68d46 100644 --- a/oikos/gen-topology.py +++ b/oikos/gen-topology.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 """ -Generate infrastructure/topology.md (Mermaid views) and per-entity context +Generate knowledge/wiki/infrastructure/topology.md (Mermaid views) and per-entity context cards from inventory.yaml. Views: @@ -42,7 +42,7 @@ from oikos import policy as oikos_policy # noqa: E402 from oikos import relations as oikos_relations # noqa: E402 INVENTORY = REPO / "inventory.yaml" -OUTPUT = REPO / "infrastructure" / "topology.md" +OUTPUT = REPO / "knowledge" / "wiki" / "infrastructure" / "topology.md" CARDS_DIR = REPO / "oikos" / "cards" BANNER = ( @@ -82,15 +82,15 @@ def _host_card(name: str, entry: dict, inv: dict) -> str: lines.append(f"- mounts: {', '.join(entry['mounts'])}") doc = None if entry.get("kind") == "lxc" and pve: - cand = REPO / "containers" / f"{pve}-{name}.md" + cand = REPO / "knowledge" / "wiki" / "containers" / f"{pve}-{name}.md" if cand.exists(): doc = str(cand.relative_to(REPO)) elif entry.get("kind") == "vm" and pve: - cand = REPO / "vms" / f"{pve}-{name}.md" + cand = REPO / "knowledge" / "wiki" / "vms" / f"{pve}-{name}.md" if cand.exists(): doc = str(cand.relative_to(REPO)) elif entry.get("kind") == "proxmox-host": - cand = REPO / "hosts" / f"{name}.md" + cand = REPO / "knowledge" / "wiki" / "hosts" / f"{name}.md" if cand.exists(): doc = str(cand.relative_to(REPO)) if doc: @@ -192,9 +192,9 @@ def render(inv: dict) -> str: parts = [ BANNER, "# Topology (generated)\n", - f"Source: [inventory.yaml](../inventory.yaml) — {counts}.", - "Edge semantics: [oikos/ontology.yaml](../oikos/ontology.yaml). " - "Operating model: [OIKOS.md](../OIKOS.md).\n", + f"Source: [inventory.yaml](../../../inventory.yaml) — {counts}.", + "Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). " + "Operating model: [OIKOS.md](../../../.agents/OIKOS.md).\n", "## Compute & ingress\n", "\n".join(compute_view(inv)) + "\n", "## Storage (mounts)\n", diff --git a/oikos/gen_topology_lib.py b/oikos/gen_topology_lib.py index 84c3e12..13c8548 100644 --- a/oikos/gen_topology_lib.py +++ b/oikos/gen_topology_lib.py @@ -2,7 +2,7 @@ Split out of oikos/gen-topology.py so it's importable (a hyphenated filename can't be `import`ed as a module). oikos/gen-topology.py is the -CLI entrypoint that writes infrastructure/topology.md + oikos/cards/; +CLI entrypoint that writes knowledge/wiki/infrastructure/topology.md + oikos/cards/; oikos/console/app.py imports this module directly to render the live /graph page without shelling out. """ diff --git a/oikos/report.py b/oikos/report.py index ef75a68..5d7e1a9 100644 --- a/oikos/report.py +++ b/oikos/report.py @@ -4,7 +4,7 @@ Both are generated text, not sent directly to Matrix — same integration contract as oikos/approve.py: there's no dedicated Matrix bot in this homelab, so Hermes (already posting alerts as @dtoro:avispero, see -infrastructure/monitoring.md) is the one that actually delivers this text. +knowledge/wiki/infrastructure/monitoring.md) is the one that actually delivers this text. The daily brief is meant to run once a day (e.g. chained after an early oikos-scheduler.service run, or its own systemd timer); the weekly report is a deeper markdown review. diff --git a/oikos/scheduler.py b/oikos/scheduler.py index fe0853e..b1e05ed 100644 --- a/oikos/scheduler.py +++ b/oikos/scheduler.py @@ -75,7 +75,7 @@ _HEALTH_OVERRIDES = { "url_transform": lambda url: url.rstrip("/").rsplit("/", 1)[0] + "/health", }, # Token-gated at the app level (401 without a token is correct, not - # down) — see containers/128-trmnl.md, which documents a dedicated + # down) — see knowledge/wiki/containers/128-trmnl.md, which documents a dedicated # /health endpoint returning 200 unauthenticated. "trmnl": {"url_transform": lambda url: url.rstrip("/") + "/health"}, } diff --git a/plans/2026-06-24-trmnl-plugins-lxc.md b/plans/2026-06-24-trmnl-plugins-lxc.md index c38f9f3..78c88ba 100644 --- a/plans/2026-06-24-trmnl-plugins-lxc.md +++ b/plans/2026-06-24-trmnl-plugins-lxc.md @@ -14,10 +14,10 @@ wiring, same split as Artifacto/Plato. ## Current state - No TRMNL middleware in the lab. Highest LXC id is 127 (see `containers/index.md`). -- Public hostnames terminate at the [VPS netbird traefik](../hosts/netbird-vps.md) → netbird - mesh → [caddy (121)](../containers/121-caddy.md) → backend LXC. Cert obtained by Caddy +- Public hostnames terminate at the [VPS netbird traefik](../hosts/netbird-vps.yaml) → netbird + mesh → [caddy (121)](../knowledge/wiki/containers/121-caddy.md) → backend LXC. Cert obtained by Caddy (IONOS DNS-01) and mirrored to the VPS by the daily cert-sync timer on the host. -- Auto-deploy pipelines are gitea-webhook driven, two shapes (see [auto-deploy](../infrastructure/auto-deploy.md)). +- Auto-deploy pipelines are gitea-webhook driven, two shapes (see [auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md)). ## Target state @@ -74,14 +74,14 @@ TRMNL cloud --GET 15m, Bearer token--> https://trmnl.hubris.network/munich-hom `/etc/terminalito-deploy/git-credentials` (mode 600). Register a gitea webhook on `dtoro/terminalito`; add `192.168.8.<128-ip>` to gitea `app.ini` `ALLOWED_HOST_LIST`. -5. **DNS**: add `trmnl.hubris.network` A → `192.168.8.175` (caddy) on [Technitium (107)](../containers/107-dns.md). +5. **DNS**: add `trmnl.hubris.network` A → `192.168.8.175` (caddy) on [Technitium (107)](../knowledge/wiki/containers/107-dns.md). 6. **Caddy** (`dtoro/caddy-conf`, commit+push auto-deploys): ``` trmnl.hubris.network { reverse_proxy 192.168.8.<128-ip>:9851 } ``` -7. **Public exposure** on the [VPS](../hosts/netbird-vps.md): add traefik router+service for +7. **Public exposure** on the [VPS](../hosts/netbird-vps.yaml): add traefik router+service for `Host(\`trmnl.hubris.network\`)` → `http://192.168.8.<128-ip>:9851`; add the host to the cert-sync map so the LE cert mirrors over. diff --git a/plans/2026-07-05-oikos-prometheus-lxc.md b/plans/2026-07-05-oikos-prometheus-lxc.md index b474f12..baf2385 100644 --- a/plans/2026-07-05-oikos-prometheus-lxc.md +++ b/plans/2026-07-05-oikos-prometheus-lxc.md @@ -8,7 +8,7 @@ at `pct create` time. ## Why -Week-3 reliability layer (see [OIKOS.md](../OIKOS.md)) wants trend +Week-3 reliability layer (see [OIKOS.md](../.agents/OIKOS.md)) wants trend signals — "disk full in ~9 days at current rate", temperature creep — which need a real time-series store. The scheduler ([oikos/scheduler.py](../oikos/scheduler.py)) currently does point-in-time @@ -42,7 +42,7 @@ is actually unused end-to-end. ## Provisioning steps (once pve_id is assigned) -Follow [runbooks/lifecycle-provision-node.md](../runbooks/lifecycle-provision-node.md): +Follow [lifecycle-provision-node](../.agents/skills/lifecycle-provision-node/SKILL.md): 1. `pct create <new-id> ...` on hubris — confirm the assigned ID doesn't collide with 131 or anything else live. @@ -53,7 +53,7 @@ Follow [runbooks/lifecycle-provision-node.md](../runbooks/lifecycle-provision-no 4. Point node_exporter at hubris + strong (either install locally on each, or scrape via SSH-tunneled metrics — install locally is simpler and is the standard approach). -5. Follow [runbooks/lifecycle-activate-node.md](../runbooks/lifecycle-activate-node.md) +5. Follow [lifecycle-activate-node](../.agents/skills/lifecycle-activate-node/SKILL.md) to flip to `active`, complete the doc page, regenerate `hosts/*.yaml` + `infrastructure/topology.md`. 6. Extend `oikos/scheduler.py`'s disk/temp probes to query Prometheus diff --git a/plans/2026-07-06-adopt-wiki-hq-doc-architecture.md b/plans/2026-07-06-adopt-wiki-hq-doc-architecture.md new file mode 100644 index 0000000..c87b2a0 --- /dev/null +++ b/plans/2026-07-06-adopt-wiki-hq-doc-architecture.md @@ -0,0 +1,164 @@ +# 2026-07-06 — Adopt the wiki-hq documentation architecture + +## Goal + +Reorganize the Homelab-Docs **narrative layer** into the `wiki-hq` +(`/Users/dtoro/Downloads/wiki-hq-main`) documentation model — `sources / wiki / index / log` +plus an `.agents/` separation and a single lint-checkable writing-style standard — **without +breaking** the Oikos machine-readable substrate that reads fixed paths. Keep the existing Mermaid +topology generator (no LikeC4). + +## Context + +Homelab-Docs is already a mature operational-docs system: `inventory.yaml` as source of truth, +generated `hosts/*.yaml`, an MCP server, a `homelab` CLI, and the Oikos OODA kernel (scheduler, +drift, decide, ledger, approvals). Its weakness is on the *narrative* side — the prose layer grew +organically and lacks the discipline wiki-hq shows: + +- No enforceable **writing-style** standard (CONTRIBUTING.md has a one-line "voice" note; CAVEMAN.md governs agent *chat*, not docs). +- **Index/README files are inconsistent** — some pure listings, some prose+tables, some with stale counts. +- **No append-only operations log** for doc maintenance — doc changes are visible only in git. +- **Agent instructions and human content are interleaved** at the repo root (AGENTS.md, OIKOS.md, CAVEMAN.md, CONTRIBUTING.md, GLOSSARY.md alongside `containers/`, `infrastructure/`, …). + +Decision (confirmed with operator): **full structural adoption**, **keep Mermaid**, adopt all four +borrows — writing-style guide, section-index/README pattern, append-only per-area logs, and +agent-instruction separation. + +## Hard constraint: protect the operational substrate + +These paths are read programmatically and **must not move** (see `mcp/server.py`, `bin/homelab`, `oikos/`): + +- `inventory.yaml` (root) — MCP (`mcp/server.py:36`), CLI (`bin/homelab:36`), scheduler, drift, relations, gen-topology. +- `hosts/*.yaml` (root, generated) — `HOSTS_DIR` (`mcp/server.py:37`); written by `mcp/build_host_files.py`; read/written by `bin/homelab`. +- `oikos/` — kernel code, `oikos/cards/` (`explain` tool, `mcp/server.py:38`), `oikos/state.json`. +- `secrets/`, `secrets-issuance/`, `ssh/`, `scripts/`, `tools/`, `vps/`, `bin/`, `bootstrap.sh`. + +Two MCP tools are path-agnostic and survive any narrative reorg: `search_docs` (ripgreps all +`*.md`) and `get_page(path)` (agent supplies a repo-relative path). The **`## Changelog` convention +must stay** on node pages — `get_changelog` (`mcp/server.py:214`) and the ledger parse it. New +`log.md` files are **additive**, not a replacement for per-page changelogs. + +## Current → target structure + +Restructure the **narrative layer only**; leave the substrate in place. + +``` +Homelab-Docs/ + README.md # human landing page (kept; refreshed to new nav) + AGENTS.md # kept at root (conventional discovery path) + + .agents/ # NEW — agent-facing instruction, separated from content + shared/ + writing-style.md # NEW — adapted from wiki-hq (homelab voice) + llm-wiki.md # NEW — the sources/wiki/index/log contract for THIS repo + caveman.md # moved from CAVEMAN.md (agent chat mode) + page-templates.md # moved from CONTRIBUTING.md (page templates) + domains/ + knowledge/schema.md # contract for the current-state wiki + operations/schema.md # contract for runbooks / investigations / plans + skills/ # runbooks reshaped as SKILL.md (Phase 4) + lifecycle-provision-node/SKILL.md + service-health-check/SKILL.md + ... + OIKOS.md # operating-model doc (moved; agent-facing) + + knowledge/ # durable current-state wiki (Feedback/authoritative layer) + index.md # pure listing → section indexes only + log.md # append-only doc-maintenance operations log + GLOSSARY.md # moved from root + sources/ + index.md # flat catalog table + references/ # external docs (from infrastructure/references/) + wiki/ + hosts/ README.md + hubris.md + strong.md + containers/ README.md (= today's index.md) + <id>-<name>.md + vms/ README.md + <id>-<name>.md + infrastructure/ README.md + grouped: network/ identity/ storage/ ingress/ operations/ + + operations/ # operator runbook narrative index + index.md commands.md agent-enrollment.md hermes-agent.md + + investigations/ # incident evidence (Observe sources) + README.md + log.md + YYYY-MM-DD-*.md + archive/ + plans/ # projects / design docs (Act) + README.md + YYYY-MM-DD-*.md + done/ + + # substrate — UNCHANGED (see Hard constraint) + inventory.yaml hosts/*.yaml oikos/ mcp/ secrets/ secrets-issuance/ + ssh/ scripts/ tools/ vps/ bin/ bootstrap.sh +``` + +**hosts/ split:** generated `hosts/*.yaml` stay at root (substrate); only the two *narrative* pages +`hosts/hubris.md` and `hosts/strong.md` move to `knowledge/wiki/hosts/`. This is the one directory +where machine and narrative content currently mix. + +## Consumers to update when narrative paths move + +Same phase as the move: + +- **`inventory.yaml` `doc_page` fields** — per-host pointer; consumed by `bin/homelab` (`die("no doc_page recorded…")` near `bin/homelab:1288`). Repoint to `knowledge/wiki/...`. +- **`oikos/gen-topology.py` / `oikos/gen_topology_lib.py`** — write `infrastructure/topology.md` and link `oikos/cards/` to doc pages. Move `topology.md` under `knowledge/wiki/infrastructure/` and update the output-path constant (`oikos/gen-topology.py:3`, `:49`, `:195`). +- **`oikos/cards/` templates** and `oikos/drift.py` docstring at `oikos/drift.py:218` (`containers/121-caddy.md`) — cosmetic, update for accuracy. +- **Internal cross-links** — repo-wide relative-link rewrite + link-check pass. +- **Same-session update rule** (`CONTRIBUTING.md:125`) — rewrite its path checklist to the new layout. + +`search_docs`, `get_page`, `get_changelog` need **no** code change. + +## Writing-style standard (adapted, not copied) + +Create `.agents/shared/writing-style.md` from +`/Users/dtoro/Downloads/wiki-hq-main/.agents/shared/writing-style/writing-style.md`: + +- **Keep:** banned-vocabulary list (puffers, "leverage/utilize/delve", poetic nouns, promotional adjectives), information-density rule, bold-leading-phrase pattern, tables-over-bullets, `>` blockquotes for caveats, one-sided cross-references, "cross-references are content not catalog". +- **Adjust voice:** wiki-hq mandates strict third-person "no you". Homelab docs are operator runbooks that already use imperative recipes. Allow **imperative/second-person in runbooks, recipes, and skills**; third-person reference voice for node/infrastructure descriptions. State this exception explicitly. +- **Clarify vs CAVEMAN.md:** Caveman governs agent *chat responses*; writing-style governs *committed docs*. No conflict. +- **Make it lint-checkable:** the Oikos lint surface (or a new `.agents/skills/docs-lint`) greps banned words + structural violations. + +## Reconciling "append-only per-area logs" with the Oikos ledger + +- The **Oikos change ledger** (`oikos/ledger.py`) stays authoritative for *infrastructure changes* (who/what/risk/approval/verification). Do not duplicate it. +- **Per-page `## Changelog`** stays (parsed by `get_changelog`). +- New **per-area `log.md`** (`knowledge/log.md`, `investigations/log.md`, `plans/log.md`) records *doc-maintenance operations only* — restructures, source ingests, lint sweeps — in wiki-hq's single-line format `## [YYYY-MM-DD] <op> | <summary>`. + +## Convention: plans always live in `plans/` + +The restructure must codify — in `.agents/domains/operations/schema.md`, `AGENTS.md`, and +`.agents/shared/page-templates.md` — that **any plan or design doc for the Homelab is always +written into the repo `plans/` folder** (`plans/YYYY-MM-DD-slug.md`), never into a scratch/agent +plan path, an ad-hoc location, or a chat message. Agents drafting a plan create the file under +`plans/`, list it in `plans/index.md`, and move it to `plans/done/` on completion. This rule is +stated once in the operations schema and cross-referenced from `AGENTS.md` so every agent sees it +at orientation. + +## Section-index / README pattern + +Every folder gets a `README.md`: 1–3 sentence prose intro, then a single two-column nav table +`| Document | What it covers |`, nothing else — no stale counts, no duplicated prose. Pure-listing +`index.md` files (e.g. `containers/index.md`) become the folder `README.md`. Root `README.md` stays +the human landing page (may exceed the strict pattern). + +## Phased rollout + +Each phase independently valuable, independently verifiable, committed separately. + +1. **Conventions first (no moves).** Add `.agents/shared/writing-style.md`, `llm-wiki.md`; move CAVEMAN.md → `.agents/shared/caveman.md`, CONTRIBUTING.md → `.agents/shared/page-templates.md` (leave thin root stubs pointing to the new locations). Verify: links resolve; MCP untouched. +2. **Agent-instruction separation.** Move OIKOS.md → `.agents/OIKOS.md`; author `.agents/domains/{knowledge,operations}/schema.md`. Keep AGENTS.md at root, pointing at `.agents/`. Verify: AGENTS.md still the discovery entry; `tools/setup-hermes-soul.sh:51` references resolve or are updated. +3. **Narrative move into `knowledge/wiki/`.** Move `containers/`, `vms/`, `infrastructure/`, host narratives; add `knowledge/index.md`, `knowledge/log.md`, `knowledge/sources/`. Same commit: update `inventory.yaml` `doc_page` fields, `oikos/gen-topology.py` output path, all internal links. Verify: link-check clean; CLI `doc_page` lookups work; `gen-topology.py` writes to the new path; MCP `get_page`/`search_docs` return moved pages. +4. **Runbooks → skills.** Reshape `runbooks/*.md` into `.agents/skills/<name>/SKILL.md` (they already carry `risk_class`, `inputs`, `verification` frontmatter — near-compatible). Update runbook-path references. Verify: frontmatter parses; lifecycle transitions still map to `oikos/ontology.yaml`. +5. **Style + README pass.** Apply the writing-style standard and README pattern across narrative pages; add `log.md` entries recording the restructure. Verify: docs-lint clean. + +## Verification (end to end) + +- **Link integrity:** markdown link checker (or `rg` over `](` targets) after each move phase; zero broken relative links. +- **MCP resolves docs:** `HOMELAB_CONTEXT_DIR=<repo>` — `get_page("knowledge/wiki/containers/104-gitea.md")` returns content; `search_docs("gitea")` hits new paths; `get_changelog(<moved page>)` parses. +- **CLI doc_page lookups:** exercise `bin/homelab`'s `doc_page` read against a moved page — no "no doc_page recorded" error. +- **Topology generation:** run `oikos/gen-topology.py` — writes the new `topology.md` path; `oikos/cards/` links point at moved pages. +- **Substrate untouched:** `mcp/build_host_files.py` regenerates `hosts/*.yaml` identically; `git diff` shows no substrate churn. +- **Style lint:** banned-vocabulary grep over `knowledge/` returns no hits; every folder `README.md` matches the intro-plus-nav-table shape. + +## Post-migration + +Update `plans/index.md` to list this plan; on completion move it to `plans/done/`. Record the +restructure in the new `knowledge/log.md`. Update `README.md` and `AGENTS.md` navigation to the new +layout. +``` diff --git a/plans/done/2026-06-01-slate-ax-to-sodola-migration.md b/plans/done/2026-06-01-slate-ax-to-sodola-migration.md index 8f67eae..62fdc67 100644 --- a/plans/done/2026-06-01-slate-ax-to-sodola-migration.md +++ b/plans/done/2026-06-01-slate-ax-to-sodola-migration.md @@ -37,7 +37,7 @@ ISP Fritz!Box takes over `192.168.8.1` — the same gateway IP the Slate AX used. No static IPs or gateway entries change on any LXC or VM. -See [network architecture](../infrastructure/network.md) for the permanent topology reference. +See [network architecture](../../knowledge/wiki/infrastructure/network.md) for the permanent topology reference. ## Pre-flight checklist @@ -60,7 +60,7 @@ See [network architecture](../infrastructure/network.md) for the permanent topol | DHCP range | `192.168.8.100 – 192.168.8.240` | | Assign to | LAN port that connects to SODOLA | | Network isolation | Enabled (blocks main LAN from initiating into homelab) | -| DNS for DHCP clients | `192.168.8.2` (Technitium on [CT 107](../containers/107-dns.md)) | +| DNS for DHCP clients | `192.168.8.2` (Technitium on [CT 107](../../knowledge/wiki/containers/107-dns.md)) | After creating the network, move any port forwards from the Slate AX into Fritz!Box → Internet → Permits (target IPs are now directly reachable on `192.168.8.x`). @@ -92,7 +92,7 @@ pct set <id> --net0 name=eth0,bridge=vmbr0,ip=<ip>/24,gw=192.168.8.1 ## DNS after migration -Technitium ([CT 107](../containers/107-dns.md)) at `192.168.8.2` continues to serve split-horizon DNS for `hubris.network`. The Fritz!Box DHCP server for VLAN 10 hands out `192.168.8.2` as the DNS server. This fixes the "update router DHCP DNS from dead .180 → .2" outstanding item in [dns.md](../infrastructure/dns.md). +Technitium ([CT 107](../../knowledge/wiki/containers/107-dns.md)) at `192.168.8.2` continues to serve split-horizon DNS for `hubris.network`. The Fritz!Box DHCP server for VLAN 10 hands out `192.168.8.2` as the DNS server. This fixes the "update router DHCP DNS from dead .180 → .2" outstanding item in [dns.md](../../knowledge/wiki/infrastructure/dns.md). ## Cutover procedure @@ -125,7 +125,7 @@ curl -sk https://auth.hubris.network/if/flow/default-authentication-flow/ | head ## Post-migration -- Update [network.md](../infrastructure/network.md) topology to reflect new state. -- Add changelog entries to [hosts/hubris.md](../hosts/hubris.md) and any affected container pages. -- Update status in [plans/index.md](index.md) to `Done`. +- Update [network.md](../../knowledge/wiki/infrastructure/network.md) topology to reflect new state. +- Add changelog entries to [hosts/hubris.md](../../knowledge/wiki/hosts/hubris.md) and any affected container pages. +- Update status in [plans/index.md](../index.md) to `Done`. - If anything went sideways, open an investigation in `investigations/`. diff --git a/plans/done/2026-06-29-grimmory-migration.md b/plans/done/2026-06-29-grimmory-migration.md index 2f89f23..89526d9 100644 --- a/plans/done/2026-06-29-grimmory-migration.md +++ b/plans/done/2026-06-29-grimmory-migration.md @@ -204,7 +204,7 @@ books.hubris.network { } ``` -Git push → Caddy webhook auto-reloads (see [caddy (121)](../containers/121-caddy.md)). +Git push → Caddy webhook auto-reloads (see [caddy (121)](../../knowledge/wiki/containers/121-caddy.md)). Test: ```bash diff --git a/tools/setup-hermes-soul.sh b/tools/setup-hermes-soul.sh index ab2eb44..1d65588 100755 --- a/tools/setup-hermes-soul.sh +++ b/tools/setup-hermes-soul.sh @@ -14,7 +14,7 @@ set -euo pipefail CONTEXT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -HERMES_MD="$CONTEXT_DIR/HERMES.md" +HERMES_MD="$CONTEXT_DIR/.agents/HERMES.md" SOUL_MD="${HOME}/.hermes/SOUL.md" # Colors for output (only when connected to a terminal)