oikos/operations/agent-enrollment.md

# Agent enrollment — bootstrap a client into the homelab context system

This walks through enrolling a new machine (workstation, LXC, or VM) so it
joins the cross-client context system: a `/opt/homelab-context/` clone of
this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.

> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
> See [hermes-agent.md](./hermes-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-hermes` flag.

Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.

## Prerequisites the client must satisfy

| Requirement | Why | How to check |
| --- | --- | --- |
| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/sse` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |

### Hostname mismatch is the most common bootstrap failure

If the bootstrap exits with `no hosts/<name>.yaml in the repo`, the
hostname doesn't match any inventory entry. Two fixes:

- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
  (Linux) or System Preferences → Sharing (macOS), then re-run.
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
  regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.

### Getting onto Netbird

Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit `<bootstrap-tier1>`). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`.

**Path B — interactive OIDC (default; recommended):**

The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.

Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../containers/124-authentik.md). Only needed once per account.

**Path A — setup-key (headless/scripted onboarding):**

Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.

1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`.
2. **Setup Keys** → Create → set reusable + expiry → copy.
3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver):
   ```bash
   sudo netbird up --setup-key <KEY> \
                   --management-url https://netbird.hubris.network \
                   --ssh-jwt-cache-ttl 86400
   ```

**Why we can't OIDC-login from the public internet (still open as a follow-up):**

`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124.

### DNS prerequisite

`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:

- **LAN client**: set DNS to 192.168.8.180 (per-interface or
  `/etc/resolv.conf`).
- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
  point `*.hubris.network` at LXC 124.
- **Hack-fix anywhere**: append to `/etc/hosts`:
  ```
  192.168.8.175  mcp.hubris.network secrets.hubris.network
  192.168.8.175  git.hubris.network
  ```
  (192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.)

If DNS isn't an option at all, override the URLs at bootstrap time:

```bash
sudo HOMELAB_GITEA_TOKEN=... \
     HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
     HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
     HOMELAB_MCP_URL=http://192.168.8.205:9810/sse \
     bash /tmp/bootstrap.sh --with-mcp
```

## Install dependencies

Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script).

Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.

<details>
<summary>Manual recipes (Fedora / Debian / macOS)</summary>

```bash
# Fedora / RHEL / Nobara
sudo dnf install -y git python3-pyyaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
  -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops

# Debian / Ubuntu
sudo apt update && sudo apt install -y git python3-yaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
  -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops

# macOS
brew install git age sops
pip3 install pyyaml   # if `python3 -c "import yaml"` fails
```

</details>

## Run the bootstrap

You need a Gitea read-only personal access token for the initial clone
(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml`
but a new client can't decrypt it before bootstrap — chicken-and-egg).
Ask the operator (or generate in Gitea: Settings → Applications → Generate
New Token → scope `read:repository`).

```bash
TOKEN=...   # your Gitea PAT, scope read:repository

# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
curl -fsSL -u "dtoro:$TOKEN" \
  https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
  -o /tmp/bootstrap.sh

# Run it.
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp
```

Flags:

| Flag | Effect |
| --- | --- |
| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user |
| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) |
| `--dry-run` | Prints actions without executing |

The bootstrap is idempotent: re-running on an enrolled client just
verifies state, re-issues the age key only if it doesn't match the
inventory pubkey, and refreshes the sync timer + symlinks.

## Verify

```bash
homelab whoami                    # prints hosts/$(hostname).yaml
homelab list                      # shows the full topology
homelab status                    # ping + HTTP-check across hosts/services
homelab secret hello              # decrypt the bootstrap-test secret
systemctl list-timers homelab-context-sync.timer
                                  # next run within ≤5 min
```

For Claude Code: start a new session — the `homelab` MCP server appears
in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
1 secrets-metadata).

## Claude Code permissions for fleet ops

By default Claude Code's auto-mode classifier asks for confirmation on every
ssh into the mesh. The bootstrap already installs the ssh ControlMaster block
so subsequent in-session sshes multiplex, but the *first* ssh of each session
still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns
by adding to `~/.claude/settings.json`:

```json
{
  "permissions": {
    "defaultMode": "auto",
    "allow": [
      "Bash(ssh -p 22022 *)",
      "Bash(homelab *)"
    ]
  }
}
```

The first rule covers any ssh to a mesh peer on the homelab netbird port; the
second covers all `homelab` CLI invocations. Both are scoped tight enough that
the classifier doesn't gate them but loose enough to handle the variety of
arguments.

If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
visible to the classifier (it's not actually durable in 0.71.2, but the
ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md)
for context), drop a free-text rule into `autoMode.allow` describing the
authorization. Optional.

## Adding a new client to inventory

If the hostname you want isn't yet in inventory, enrollment is a two-step
ceremony driven from an existing enrolled client (e.g. hubris). The
`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3.

```bash
# 1. On hubris (or any existing client): add the inventory entry.
homelab client add my-new-machine
# Prompts for kind, os, netbird FQDN, role. Commits + pushes.

# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key).

# 3. On the new machine: install deps + run bootstrap (above).
#    Bootstrap calls /issue, receives a fresh age keypair, and prints the
#    public key for the operator to commit back to inventory.

# 4. On hubris: finalize the age public key.
homelab client add my-new-machine --finalize-pubkey age1...
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
```

## Granting a secret to a new client

Adding a client doesn't grant them every secret. Recipients are explicit
per file via `.sops.yaml` glob rules. To grant a client access to (say)
`secrets/hello.yaml`:

1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to
   the matching `creation_rules` block.
2. Re-key the existing ciphertext for the new recipient list:
   ```bash
   sops updatekeys -y secrets/hello.yaml
   ```
3. Commit + push. On the next sync (≤5 min), the client can decrypt.

## Removing a client

```bash
# From any existing client:
homelab client remove my-old-machine
```

This:
1. Removes the inventory entry and `hosts/my-old-machine.yaml`.
2. Runs `sops updatekeys -y` against every file in `secrets/` (operator
   must first remove the pubkey from `.sops.yaml` rules).
3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to
   shred the key file and add the hostname to the denylist.
4. Commits + pushes.

The CLI prints a follow-up checklist that the operator must do manually:

- Revoke the peer in the Netbird console (denies future mesh access).
- **Rotate any credentials whose ciphertext the removed client already
  has on disk.** The age key revocation only protects *future*
  ciphertext; what's already been pulled is still decryptable until the
  underlying credential changes.
- Optional: `homelab nuke my-old-machine` SSHes in, shreds
  `/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). |
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |

## Changelog

### 2026-05-31 — cross-link to hermes-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](./hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.

### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.

### 2026-05-20 — initial page
Captures the enrollment flow validated during Phase 2 of the homelab
context distribution rollout. hubris + LXC 105 (apps) enrolled; first
workstation (republic-laptop) blocked on hostname mismatch, documented
the resolution.