diff --git a/README.md b/README.md index 585033f..62a611c 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,7 @@ Time-stamped incident notes / experiments in [`investigations/`](investigations/ ### Operations - [Command cheatsheet](operations/commands.md) +- [Agent enrollment](operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system ## Conventions diff --git a/operations/agent-enrollment.md b/operations/agent-enrollment.md new file mode 100644 index 0000000..ca8cbe6 --- /dev/null +++ b/operations/agent-enrollment.md @@ -0,0 +1,213 @@ +# Agent enrollment — bootstrap a client into the homelab context system + +This walks through enrolling a new machine (workstation, LXC, or VM) so it +joins the cross-client context system: a `/opt/homelab-context/` clone of +this repo that auto-syncs every 5 min, a per-client age key for SOPS +decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config. + +Architecture in [project_homelab_context_plan](https://… memory link); the +operational reference is here. + +## Prerequisites the client must satisfy + +| Requirement | Why | How to check | +| --- | --- | --- | +| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) | +| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` | +| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets | `netbird status` / `tailscale status` | +| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below | +| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/sse` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) | + +### Hostname mismatch is the most common bootstrap failure + +If the bootstrap exits with `no hosts/.yaml in the repo`, the +hostname doesn't match any inventory entry. Two fixes: + +- **Rename the host**: `sudo hostnamectl set-hostname ` + (Linux) or System Preferences → Sharing (macOS), then re-run. +- **Rename the inventory entry**: edit `inventory.yaml` on hubris, + regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates. + +### DNS prerequisite + +`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124 +([dns.md](../infrastructure/dns.md)) for LAN clients, **but only if the +client uses 192.168.8.180 as its resolver**. Most LXCs and roaming +workstations don't by default. Options: + +- **LAN client**: set DNS to 192.168.8.180 (per-interface or + `/etc/resolv.conf`). +- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to + point `*.hubris.network` at LXC 124. +- **Hack-fix anywhere**: append to `/etc/hosts`: + ``` + 192.168.8.175 mcp.hubris.network secrets.hubris.network + 192.168.8.175 git.hubris.network + ``` + (192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.) + +If DNS isn't an option at all, override the URLs at bootstrap time: + +```bash +sudo HOMELAB_GITEA_TOKEN=... \ + HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \ + HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \ + HOMELAB_MCP_URL=http://192.168.8.205:9810/sse \ + bash /tmp/bootstrap.sh --with-mcp +``` + +## Install dependencies + +### Fedora / RHEL / Nobara + +```bash +sudo dnf install -y git python3-pyyaml age curl +SOPS_VERSION=v3.9.4 +sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \ + -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops +``` + +### Debian / Ubuntu + +```bash +sudo apt update && sudo apt install -y git python3-yaml age curl +SOPS_VERSION=v3.9.4 +sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \ + -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops +``` + +### macOS + +```bash +brew install git age sops +pip3 install pyyaml # if `python3 -c "import yaml"` fails +``` + +## Run the bootstrap + +You need a Gitea read-only personal access token for the initial clone +(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml` +but a new client can't decrypt it before bootstrap — chicken-and-egg). +Ask the operator (or generate in Gitea: Settings → Applications → Generate +New Token → scope `read:repository`). + +```bash +TOKEN=... # your Gitea PAT, scope read:repository + +# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy). +curl -fsSL -u "dtoro:$TOKEN" \ + https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \ + -o /tmp/bootstrap.sh + +# Run it. +sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp +``` + +Flags: + +| Flag | Effect | +| --- | --- | +| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user | +| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) | +| `--dry-run` | Prints actions without executing | + +The bootstrap is idempotent: re-running on an enrolled client just +verifies state, re-issues the age key only if it doesn't match the +inventory pubkey, and refreshes the sync timer + symlinks. + +## Verify + +```bash +homelab whoami # prints hosts/$(hostname).yaml +homelab list # shows the full topology +homelab status # ping + HTTP-check across hosts/services +homelab secret hello # decrypt the bootstrap-test secret +systemctl list-timers homelab-context-sync.timer + # next run within ≤5 min +``` + +For Claude Code: start a new session — the `homelab` MCP server appears +in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management, +1 secrets-metadata). + +## Adding a new client to inventory + +If the hostname you want isn't yet in inventory, enrollment is a two-step +ceremony driven from an existing enrolled client (e.g. hubris). The +`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3. + +```bash +# 1. On hubris (or any existing client): add the inventory entry. +homelab client add my-new-machine +# Prompts for kind, os, netbird FQDN, role. Commits + pushes. + +# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key). + +# 3. On the new machine: install deps + run bootstrap (above). +# Bootstrap calls /issue, receives a fresh age keypair, and prints the +# public key for the operator to commit back to inventory. + +# 4. On hubris: finalize the age public key. +homelab client add my-new-machine --finalize-pubkey age1... +# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates +# hosts/*.yaml, commits + pushes. The 5-min sync propagates. +``` + +## Granting a secret to a new client + +Adding a client doesn't grant them every secret. Recipients are explicit +per file via `.sops.yaml` glob rules. To grant a client access to (say) +`secrets/hello.yaml`: + +1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to + the matching `creation_rules` block. +2. Re-key the existing ciphertext for the new recipient list: + ```bash + sops updatekeys -y secrets/hello.yaml + ``` +3. Commit + push. On the next sync (≤5 min), the client can decrypt. + +## Removing a client + +```bash +# From any existing client: +homelab client remove my-old-machine +``` + +This: +1. Removes the inventory entry and `hosts/my-old-machine.yaml`. +2. Runs `sops updatekeys -y` against every file in `secrets/` (operator + must first remove the pubkey from `.sops.yaml` rules). +3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to + shred the key file and add the hostname to the denylist. +4. Commits + pushes. + +The CLI prints a follow-up checklist that the operator must do manually: + +- Revoke the peer in the Netbird console (denies future mesh access). +- **Rotate any credentials whose ciphertext the removed client already + has on disk.** The age key revocation only protects *future* + ciphertext; what's already been pulled is still decryptable until the + underlying credential changes. +- Optional: `homelab nuke my-old-machine` SSHes in, shreds + `/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync. + +## Troubleshooting + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `no hosts/.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) | +| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` | +| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` | +| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix | +| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy | +| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) | +| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) | + +## Changelog + +### 2026-05-20 — initial page +Captures the enrollment flow validated during Phase 2 of the homelab +context distribution rollout. hubris + LXC 105 (apps) enrolled; first +workstation (republic-laptop) blocked on hostname mismatch, documented +the resolution.