From e8c2ccf7bb04965b62ffd13b3cbac675257e96ec Mon Sep 17 00:00:00 2001 From: root Date: Wed, 20 May 2026 18:34:45 +0200 Subject: [PATCH] wiki: document homelab-context distribution system Adds infrastructure/homelab-context.md as the architecture reference for the cross-client context + MCP + secrets-issuance system. Updates: - 105-apps.md: two new ## Stacks sections (homelab-mcp, secrets-issuance) with their deploy pipelines + a row each in the public-hostname table; changelog entry. - auto-deploy.md: both new pipelines added to the table (one repo, two webhooks, same push); per-pipeline notes covering the clone-per-service pattern and the deploy.sh self-restart caveat; changelog entry. - README.md: link to the new infrastructure page. Operational walkthrough already lives at operations/agent-enrollment.md; this commit is the architecture side of the same story. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 1 + containers/105-apps.md | 59 +++++++++++- infrastructure/auto-deploy.md | 25 ++++++ infrastructure/homelab-context.md | 143 ++++++++++++++++++++++++++++++ 4 files changed, 227 insertions(+), 1 deletion(-) create mode 100644 infrastructure/homelab-context.md diff --git a/README.md b/README.md index 62a611c..ec16d4e 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links: - [Backups — restic on external drive (disabled)](infrastructure/backups.md) - [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md) - [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md) +- [Homelab context distribution](infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance ### Investigations Time-stamped incident notes / experiments in [`investigations/`](investigations/index.md). diff --git a/containers/105-apps.md b/containers/105-apps.md index ef9b409..f7fc582 100644 --- a/containers/105-apps.md +++ b/containers/105-apps.md @@ -1,6 +1,6 @@ # 105 — `apps` -Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog). +Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20. ## At a glance - **Hostname:** `apps` @@ -19,6 +19,8 @@ Docker host for everything that doesn't justify its own LXC. Currently runs Arti | `artifacto.hubris.network` | Artifacto | `:3100` | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). | | `blog.hubris.network` | WriteFreely | `:8080` | Native OIDC via `[oauth.generic]`. | | `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). | +| `mcp.hubris.network` | homelab-mcp | `:9810` | MCP server over SSE. No SSO — agent-facing, gated at source-IP layer. See [homelab-context](../infrastructure/homelab-context.md). | +| `secrets.hubris.network` | secrets-issuance | `:9820` | Per-client age-key issuance. Mesh+LAN source-IP gated (`MESH_SUBNETS`). | ## Stacks @@ -49,6 +51,51 @@ Native OIDC via Authentik (Settings → OIDC). Redirect URI `/oauth2-callback` ( > ⚠️ **Never `docker compose up` Portainer-managed stacks from the host shell.** Portainer's compose state lives at `/var/lib/docker/volumes/portainer_data/_data/compose//`. Running `docker compose up -d ` from the host triggers recreates of OTHER services in the stack and silently destroys bind-mounted data. **This wiped Booklore's mariadb data on 2026-04-22.** Use the Portainer UI editor for compose changes. See [mesh migration](../infrastructure/mesh.md#critical-never-docker-compose-up-portainer-managed-stacks) for the full warning. +### homelab-mcp (`/opt/homelab-mcp/`) +FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from +`/opt/homelab-context/` (this LXC is itself an enrolled +[homelab-context](../infrastructure/homelab-context.md) client). Source +is `dtoro/Homelab-Docs/mcp/server.py`; service unit +`homelab-mcp.service` binds `0.0.0.0:9810`. DNS-rebinding protection +disabled at the FastMCP layer because mesh+LAN gating is the actual +trust boundary. + +- Endpoint: `https://mcp.hubris.network/sse` (Caddy → `:9810`). +- 14 tools registered: `get_host`, `list_services`, `find_service`, + `get_topology`, `search_docs`, `get_page`, `get_changelog`, `whoami`, + `list_my_secrets` (context); `get_service_status`, `tail_log`, + `list_lxcs`, `get_lxc_state`, `ping_service` (read-only management). +- Management tools shell out via SSH using a dedicated `mcp-reader` key + (planned; not provisioned yet — management tools fail until then). + +#### Auto-deploy pipeline (homelab-mcp) +Receiver at `/opt/homelab-mcp/mcp/deploy/webhook/` (in-tree alongside +the server code). Listens on `0.0.0.0:9811`, secret in +`/etc/homelab-mcp-deploy/secret`, systemd unit +`homelab-mcp-deploy.service`. Gitea webhook id 10 on +`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and +restarts on push. + +### secrets-issuance (`/opt/secrets-issuance/`) +Tiny HTTP service that issues per-client age keypairs the first time +each client calls `/issue`. Idempotent: subsequent calls return the +same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in +`secrets-issuance.service`. State (per-client keys + denylist) lives at +`/var/lib/secrets-issuance/` (NOT in the repo). + +- Endpoint: `https://secrets.hubris.network/issue` (Caddy → `:9820`). +- `/revoke` is admin-token-gated by `/etc/secrets-issuance/admin-token`; + shreds the local key file and adds the hostname to the denylist. + Called by `homelab client remove`. +- Trust subnets today: `100.122.0.0/16` (Netbird), `100.64.0.0/10` + (Tailscale), `192.168.8.0/24` (LAN). Tighten if the LAN gets + untrusted devices. + +#### Auto-deploy pipeline (secrets-issuance) +Receiver at `/opt/secrets-issuance/secrets-issuance/deploy/webhook/`, +port `9821`, secret in `/etc/secrets-issuance-deploy/secret`, unit +`secrets-issuance-deploy.service`. Gitea webhook id 11. + ### WriteFreely (`/opt/writefreely-docker/`) Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.hubris.network`, scope `openid profile email`, `map_user_id = sub`. Redirect URI in Authentik: `https://blog.hubris.network/oauth/callback/generic`. Container has `extra_hosts: auth.hubris.network:192.168.8.175`. First-time users with a local WF account must connect SSO via Customize/Settings → Linked Accounts. @@ -66,6 +113,16 @@ Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.h ## Changelog +### 2026-05-20 — homelab-mcp + secrets-issuance live +Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md): +`homelab-mcp.service` on `:9810` (MCP read+management surface) and +`secrets-issuance.service` on `:9820` (per-client age-key provisioning). +Caddy fronts both with Let's Encrypt; new vhosts on +[caddy](121-caddy.md), split-horizon DNS entries on +[authentik (124)](124-authentik.md). Gitea webhook ids 10 + 11 wire +auto-deploy. LXC is itself an enrolled context client +(`/opt/homelab-context/`). + ### 2026-04-28 — wiki entry created Initial documentation. diff --git a/infrastructure/auto-deploy.md b/infrastructure/auto-deploy.md index 08f53e7..942891f 100644 --- a/infrastructure/auto-deploy.md +++ b/infrastructure/auto-deploy.md @@ -40,6 +40,12 @@ The app repo at `/opt/` is the working tree, but the deploy tooling (`web | `dtoro/Plato` | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` | | `dtoro/claudio-bot` | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` | | `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) | +| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart | +| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart | + +> Note: `dtoro/Homelab-Docs` has **two webhooks** firing on the same push. +> Each owns its own clone on LXC 105. They don't conflict because each +> deploy.sh only touches its own service unit + venv. > **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern. @@ -54,6 +60,7 @@ Always commit + push. Local-only edits drift. Common ones: - `/opt/plato/app/` ↔ `dtoro/Plato` (auto-deploys) - `/opt/claudio-bot/` ↔ `dtoro/claudio-bot` (auto-deploys) - `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys) +- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md)) ## Per-pipeline notes / gotchas @@ -81,6 +88,21 @@ Always commit + push. Local-only edits drift. Common ones: - Currently the only deploy that targets the host directly (`192.168.8.77:9798`). - `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run. +### homelab-mcp / secrets-issuance +- Both ride a single push to `dtoro/Homelab-Docs`. Two clones on LXC 105 + (`/opt/homelab-mcp`, `/opt/secrets-issuance`) — each is an independent + Shape-B target with its own webhook receiver. +- The deploy script restarts the service it just updated. Because the + webhook receiver itself is a separate systemd unit (`*-deploy.service`), + it does NOT restart itself — but `deploy.sh` running `systemctl + restart homelab-mcp-deploy.service` (or the secrets-issuance one) + would create a kill-self loop. The current `deploy.sh` is careful + to only restart the main service. +- Both services consume `/opt/homelab-context` for their runtime data + (inventory, secret recipient lookup). That clone is **the same clone + every other client has** — kept fresh by `homelab-context-sync.timer`, + not by these webhooks. + ## Related - [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these - [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets @@ -88,6 +110,9 @@ Always commit + push. Local-only edits drift. Common ones: ## Changelog +### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added +Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo. + ### 2026-05-13 — Plato pipeline added Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`. diff --git a/infrastructure/homelab-context.md b/infrastructure/homelab-context.md new file mode 100644 index 0000000..8dc40ec --- /dev/null +++ b/infrastructure/homelab-context.md @@ -0,0 +1,143 @@ +# Homelab context distribution + +The cross-client context-and-secrets system that makes every agent (Claude +Code, claudio-bot, future MCP-capable clients) on every machine in the lab +self-locating and able to read the same source of truth. + +Operational walkthrough for enrolling a new client lives in +[operations/agent-enrollment.md](../operations/agent-enrollment.md); this +page is the architecture reference. + +## What's where + +| Piece | Host | Path | Role | +| --- | --- | --- | --- | +| Source of truth | [gitea (104)](../containers/104-gitea.md) | `dtoro/Homelab-Docs.git` | Inventory + wiki + service code | +| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, claudio-bot | +| `homelab` CLI | every enrolled client | `/usr/local/bin/homelab` → `/opt/homelab-context/bin/homelab` (symlink) | Operator surface for enroll/secret/ssh/pct | +| Per-client age key | every enrolled client | `/etc/age/key.txt` (0600 root) | Decrypts SOPS-encrypted secrets the client is a recipient on | +| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/sse) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets | +| Secrets-issuance | [apps (105)](../containers/105-apps.md) | `secrets-issuance.service` on port 9820 (https://secrets.hubris.network/issue) | Generates per-client age keypair on first bootstrap; idempotent; admin-token-gated `/revoke` | +| Sync timer | every enrolled client | `homelab-context-sync.timer` (Linux) / `network.hubris.homelab-context-sync.plist` (macOS) | `git pull --ff-only` every 5 min | +| Encrypted secrets | `dtoro/Homelab-Docs` | `secrets/*.yaml` (SOPS+age) | Recipients declared in `.sops.yaml` | +| Read-only context PAT | `dtoro` Gitea user, scope `read:repository` | given to operators out-of-band | Bootstrap-only — for the initial clone before SOPS works | +| Write-scoped PAT | `secrets/gitea-pat.yaml` (SOPS) | `homelab refresh-creds` swaps it into `/etc/homelab-context/git-credentials` | All post-bootstrap pushes (client lifecycle, wiki edits) | + +## Data flow + +``` + dtoro/Homelab-Docs (gitea) + │ + ┌────────── push ────────┤ ◀── git push (write PAT or SSH) + │ │ + │ ┌────── push ──────┘ + │ │ │ + │ │ ▼ webhook (push event) + │ │ ┌─── homelab-mcp-deploy ──── (LXC 105:9811) + │ │ └─── secrets-issuance-deploy (LXC 105:9821) + │ │ │ + │ │ ▼ + │ │ git pull → deploy.sh → restart service + │ │ + │ └── on every client: + │ timer (5 min) → git pull --ff-only into /opt/homelab-context + │ + ▼ + homelab CLI / MCP server reads /opt/homelab-context for everything +``` + +## Why two clones on LXC 105 + +The MCP server and secrets-issuance each have their own clone +(`/opt/homelab-mcp`, `/opt/secrets-issuance`) **in addition to** +`/opt/homelab-context`. Reasons: + +- The deploy webhook for each service updates its own clone, runs + `deploy.sh` from there, and re-installs the systemd unit. Mixing this + with the client-context clone would create a circular dependency + (deploy reinstalls the unit that pulled it). +- The MCP server reads its data from `/opt/homelab-context` (the same path + every client uses) so changes to inventory propagate identically. Code + changes live in `/opt/homelab-mcp` and trigger a service restart. + +## Mesh / network gates + +- Both services bind `0.0.0.0:`. The trust boundary is + `MESH_SUBNETS` in the service's environment + nftables (planned). Today + `MESH_SUBNETS=100.122.0.0/16,100.64.0.0/10,192.168.8.0/24` — Netbird + + Tailscale + the homelab LAN. Adjust if the LAN ever has untrusted + devices. +- Caddy fronts both with Let's Encrypt certs via the IONOS DNS challenge: + `mcp.hubris.network` → `192.168.8.205:9810`, + `secrets.hubris.network` → `192.168.8.205:9820`. Off-LAN clients on + Netbird reach them via the `192.168.8.0/24` network resource routed + through the PVE peer ([mesh.md](mesh.md)). +- Clients with default-public DNS (workstations not on Netbird, LXCs + using router DNS) need a `/etc/hosts` override pointing + `mcp.hubris.network` and `secrets.hubris.network` at the caddy LXC + (`192.168.8.175`) — same caveat as every other `*.hubris.network` + service, see [dns.md](dns.md). + +## Secrets model + +- Each enrolled client gets one **age private key** issued by + secrets-issuance on first bootstrap. The key file stays root-only on + the client; the public key is committed to `inventory.yaml` (and + becomes a recipient on SOPS-encrypted files via `.sops.yaml`). +- `secrets/*.yaml` are SOPS+age. Recipients per file are pinned in + `.sops.yaml` `creation_rules` by `path_regex`. Re-encrypting a file is + `sops updatekeys -y secrets/.yaml`. +- The MCP server's `list_my_secrets(caller_pubkey)` tool returns only + secret *names* a given pubkey can decrypt — the server never sees + plaintext. Decryption is local-on-client (`homelab secret ` + shells out to `sops -d` with the client's key). +- The "all-clients" secrets (`hello.yaml` for the bootstrap decrypt + test, `gitea-pat.yaml` for the write-scoped PAT) are auto-granted to + every newly enrolled client by `homelab client add --finalize-pubkey` + (which appends the pubkey to the matching `.sops.yaml` rule and runs + `sops updatekeys`). +- **Removal does not erase past disclosure.** Revoking a client via + `homelab client remove` shreds the issuance-side key, denylists the + hostname, removes them from the recipient list, and re-keys all + shared secrets — but anything they already decrypted to disk is out of + your control. Rotate the underlying credential if compromise is + suspected. + +## Why this design + +- **One source of truth** keeps inventory, code, secrets, and docs + versioned together. A `git log` of `inventory.yaml` is the history of + the homelab. +- **Per-client age keys** scale better than a shared admin secret — + removing a client is a real revocation (for new ciphertext), not just + removing them from a wiki page. +- **MCP layer over the same clone** gives MCP-capable agents structured + query (`find_service`, `search_docs`) without forcing non-MCP tools to + go without — anything can still `cat` the markdown. +- **Sync timer rather than push fan-out** keeps the failure mode + contained: one client's webhook outage doesn't block a push from + landing on the others. Sub-5-min staleness is fine for docs and rare + enough for secrets that we don't need lower latency. + +## Related + +- [Operations: agent enrollment](../operations/agent-enrollment.md) — the + step-by-step for adding a new client +- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance` + pipelines (and the rest of the lab's webhook pipelines) +- [Mesh](mesh.md) — Netbird / Tailscale paths and the `192.168.8.0/24` + network resource +- [Apps (105)](../containers/105-apps.md) — where both services run +- [Gitea (104)](../containers/104-gitea.md) — the source of truth + +## Changelog + +### 2026-05-20 — system live across hubris, apps, republic-laptop +Phase 1 of the [cross-client context plan](../README.md) merged. Three +clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS +decrypt verified on each. Webhook auto-deploy for both LXC 105 services +wired (hook ids 10 + 11). `homelab refresh-creds` + atomic +`client add --finalize-pubkey` grant flow live so new clients are one +ceremony instead of four manual steps. Outstanding: bootstrap mac-mini +(macOS, exercises launchd) + ludo-mini + the remaining LXCs; +claudio-bot integration so the matrix agent uses inventory at chat-time.