oikos/operations/agent-enrollment.md

# Agent enrollment — bootstrap a client into the homelab context system

This walks through enrolling a new machine (workstation, LXC, or VM) so it
joins the cross-client context system: a `/opt/homelab-context/` clone of
this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.

Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.

## Prerequisites the client must satisfy

| Requirement | Why | How to check |
| --- | --- | --- |
| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/sse` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |

### Hostname mismatch is the most common bootstrap failure

If the bootstrap exits with `no hosts/<name>.yaml in the repo`, the
hostname doesn't match any inventory entry. Two fixes:

- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
  (Linux) or System Preferences → Sharing (macOS), then re-run.
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
  regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.

### Getting onto Netbird

The bootstrap *requires* the new client to already be a connected mesh peer (its `netbird status` shows `Management: Connected`). Two paths to get there:

**Path A — setup-key (currently the only working path, as of 2026-05-21):**

1. From an **already-enrolled** machine (e.g. an existing workstation or hubris), log into the dashboard at `https://netbird.hubris.network/`. The dashboard auth flow goes via Authentik on [LXC 124](../containers/124-authentik.md), which is only reachable from inside the mesh — that's why this step has to be done from a mesh peer.
2. **Setup Keys** → Create → set reusable + expiry as appropriate → copy the key.
3. On the new client (after installing netbird per [netbird docs](https://docs.netbird.io/how-to/installation)):
   ```bash
   sudo netbird up --setup-key <KEY> \
                   --management-url https://netbird.hubris.network
   ```
   Optionally pass `--ssh-jwt-cache-ttl=86400` here too (one less SSO per day for ssh into mesh peers).
4. Confirm: `netbird status` shows `Management: Connected`, `Signal: Connected`, peer IP `100.122.x.x/16`. Then proceed to "Install dependencies" and "Run the bootstrap" below.

**Path B — interactive OIDC ("`netbird up` without setup-key"):**

Currently **broken** because Authentik's OAuth2 Device Code Stage isn't configured yet — the device-code URL renders a blank consent screen, and the code expires after 60s. See [124-authentik.md "KNOWN MISSING — Device Code Stage"](../containers/124-authentik.md) for the fix. Until then, stick with Path A.

**Why we can't OIDC-login from the public internet:**

`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A new client off-mesh hitting `auth.hubris.network` gets a Traefik default 404. Solving this needs a VPS-side Traefik route forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124. Tracked as a future-session improvement.

### DNS prerequisite

`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:

- **LAN client**: set DNS to 192.168.8.180 (per-interface or
  `/etc/resolv.conf`).
- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
  point `*.hubris.network` at LXC 124.
- **Hack-fix anywhere**: append to `/etc/hosts`:
  ```
  192.168.8.175  mcp.hubris.network secrets.hubris.network
  192.168.8.175  git.hubris.network
  ```
  (192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.)

If DNS isn't an option at all, override the URLs at bootstrap time:

```bash
sudo HOMELAB_GITEA_TOKEN=... \
     HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
     HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
     HOMELAB_MCP_URL=http://192.168.8.205:9810/sse \
     bash /tmp/bootstrap.sh --with-mcp
```

## Install dependencies

### Fedora / RHEL / Nobara

```bash
sudo dnf install -y git python3-pyyaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
  -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
```

### Debian / Ubuntu

```bash
sudo apt update && sudo apt install -y git python3-yaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
  -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
```

### macOS

```bash
brew install git age sops
pip3 install pyyaml   # if `python3 -c "import yaml"` fails
```

## Run the bootstrap

You need a Gitea read-only personal access token for the initial clone
(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml`
but a new client can't decrypt it before bootstrap — chicken-and-egg).
Ask the operator (or generate in Gitea: Settings → Applications → Generate
New Token → scope `read:repository`).

```bash
TOKEN=...   # your Gitea PAT, scope read:repository

# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
curl -fsSL -u "dtoro:$TOKEN" \
  https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
  -o /tmp/bootstrap.sh

# Run it.
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp
```

Flags:

| Flag | Effect |
| --- | --- |
| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user |
| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) |
| `--dry-run` | Prints actions without executing |

The bootstrap is idempotent: re-running on an enrolled client just
verifies state, re-issues the age key only if it doesn't match the
inventory pubkey, and refreshes the sync timer + symlinks.

## Verify

```bash
homelab whoami                    # prints hosts/$(hostname).yaml
homelab list                      # shows the full topology
homelab status                    # ping + HTTP-check across hosts/services
homelab secret hello              # decrypt the bootstrap-test secret
systemctl list-timers homelab-context-sync.timer
                                  # next run within ≤5 min
```

For Claude Code: start a new session — the `homelab` MCP server appears
in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
1 secrets-metadata).

## Claude Code permissions for fleet ops

By default Claude Code's auto-mode classifier asks for confirmation on every
ssh into the mesh. The bootstrap already installs the ssh ControlMaster block
so subsequent in-session sshes multiplex, but the *first* ssh of each session
still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns
by adding to `~/.claude/settings.json`:

```json
{
  "permissions": {
    "defaultMode": "auto",
    "allow": [
      "Bash(ssh -p 22022 *)",
      "Bash(homelab *)"
    ]
  }
}
```

The first rule covers any ssh to a mesh peer on the homelab netbird port; the
second covers all `homelab` CLI invocations. Both are scoped tight enough that
the classifier doesn't gate them but loose enough to handle the variety of
arguments.

If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
visible to the classifier (it's not actually durable in 0.71.2, but the
ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md)
for context), drop a free-text rule into `autoMode.allow` describing the
authorization. Optional.

## Adding a new client to inventory

If the hostname you want isn't yet in inventory, enrollment is a two-step
ceremony driven from an existing enrolled client (e.g. hubris). The
`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3.

```bash
# 1. On hubris (or any existing client): add the inventory entry.
homelab client add my-new-machine
# Prompts for kind, os, netbird FQDN, role. Commits + pushes.

# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key).

# 3. On the new machine: install deps + run bootstrap (above).
#    Bootstrap calls /issue, receives a fresh age keypair, and prints the
#    public key for the operator to commit back to inventory.

# 4. On hubris: finalize the age public key.
homelab client add my-new-machine --finalize-pubkey age1...
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
```

## Granting a secret to a new client

Adding a client doesn't grant them every secret. Recipients are explicit
per file via `.sops.yaml` glob rules. To grant a client access to (say)
`secrets/hello.yaml`:

1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to
   the matching `creation_rules` block.
2. Re-key the existing ciphertext for the new recipient list:
   ```bash
   sops updatekeys -y secrets/hello.yaml
   ```
3. Commit + push. On the next sync (≤5 min), the client can decrypt.

## Removing a client

```bash
# From any existing client:
homelab client remove my-old-machine
```

This:
1. Removes the inventory entry and `hosts/my-old-machine.yaml`.
2. Runs `sops updatekeys -y` against every file in `secrets/` (operator
   must first remove the pubkey from `.sops.yaml` rules).
3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to
   shred the key file and add the hostname to the denylist.
4. Commits + pushes.

The CLI prints a follow-up checklist that the operator must do manually:

- Revoke the peer in the Netbird console (denies future mesh access).
- **Rotate any credentials whose ciphertext the removed client already
  has on disk.** The age key revocation only protects *future*
  ciphertext; what's already been pulled is still decryptable until the
  underlying credential changes.
- Optional: `homelab nuke my-old-machine` SSHes in, shreds
  `/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). |

## Changelog

### 2026-05-20 — initial page
Captures the enrollment flow validated during Phase 2 of the homelab
context distribution rollout. hubris + LXC 105 (apps) enrolled; first
workstation (republic-laptop) blocked on hostname mismatch, documented
the resolution.