oikos/.agents/operations/agent-enrollment.md

Agent enrollment — bootstrap a client into the homelab context system

This walks through enrolling a new machine (workstation, LXC, or VM) so it joins the cross-client context system: a /opt/homelab-context/ clone of this repo that auto-syncs every 5 min, a per-client age key for SOPS decryption, the homelab CLI, and an MCP endpoint in Claude Code's config.

Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment? See hermes-agent.md. It uses the same bootstrap.sh with an additional --with-hermes flag.

Architecture in [project_homelab_context_plan](https://… memory link); the operational reference is here.

Prerequisites the client must satisfy

Requirement	Why	How to check
Hostname matches an entry in inventory.yaml	The bootstrap looks up hosts/$(hostname).yaml.	hostname (Linux) / scutil --get LocalHostName (macOS)
OS is Linux or macOS	bootstrap detects via uname -s	uname -s
On the mesh (Netbird or Tailscale) or on the LAN	issuance is gated to mesh + LAN subnets. For Netbird: use a setup-key, not interactive auth — see "Getting onto Netbird" below.	netbird status / tailscale status
git, python3, python3-yaml, age, sops	bootstrap preflight; homelab CLI imports yaml	See per-OS commands below
Can resolve *.hubris.network	bootstrap calls https://secrets.hubris.network/issue and writes https://mcp.hubris.network/mcp	dig +short mcp.hubris.network (should return 192.168.8.175)

Hostname mismatch is the most common bootstrap failure

If the bootstrap exits with no hosts/<name>.yaml in the repo, the hostname doesn't match any inventory entry. Two fixes:

• Rename the host: sudo hostnamectl set-hostname <inventory-name> (Linux) or System Preferences → Sharing (macOS), then re-run.
• Rename the inventory entry: edit inventory.yaml on hubris, regenerate hosts/*.yaml, push. The next sync (≤5 min) propagates.

Getting onto Netbird

Bootstrap auto-installs netbird and drives netbird up if the mesh isn't already connected (since commit <bootstrap-tier1>). Both paths below produce the same end state: netbird status shows Management: Connected, peer IP 100.122.x.x/16.

Path B — interactive OIDC (default; recommended):

The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns Connected. Bootstrap then proceeds with the rest of preflight.

Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as pending_approval=1 and needs an sqlite promotion to owner — see 124-authentik.md First-time owner promotion gotcha. Only needed once per account.

Path A — setup-key (headless/scripted onboarding):

Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.

1. From an already-enrolled machine, log into the dashboard at https://netbird.hubris.network/.
2. Setup Keys → Create → set reusable + expiry → copy.
3. On the new client (after installing netbird, OR let bootstrap install it and skip its netbird up driver):

   sudo netbird up --setup-key <KEY> \
                   --management-url https://netbird.hubris.network \
                   --ssh-jwt-cache-ttl 86400
   

Why we can't OIDC-login from the public internet (still open as a follow-up):

auth.hubris.network resolves publicly to the VPS (82.165.190.79), but Traefik on the VPS doesn't currently route that hostname — only netbird.hubris.network is exposed. A brand-new client off the mesh hitting auth.hubris.network directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding auth.hubris.network via the netbird-routed 192.168.8.0/24 to LXC 124.

DNS prerequisite

*.hubris.network resolves via the split-horizon dnsmasq on LXC 124 (dns.md) for LAN clients, but only if the client uses 192.168.8.180 as its resolver. Most LXCs and roaming workstations don't by default. Options:

• LAN client: set DNS to 192.168.8.180 (per-interface or /etc/resolv.conf).
• Off-LAN workstation on Netbird: configure Netbird DNS forwarder to point *.hubris.network at LXC 124.
• Hack-fix anywhere: append to /etc/hosts:

  192.168.8.175  mcp.hubris.network secrets.hubris.network
  192.168.8.175  git.hubris.network
  

  (192.168.8.175 = caddy on LXC 121, terminates all *.hubris.network.)

If DNS isn't an option at all, override the URLs at bootstrap time:

sudo HOMELAB_GITEA_TOKEN=... \
     HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
     HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
     HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
     bash /tmp/bootstrap.sh --with-mcp

Install dependencies

Bootstrap auto-installs missing prerequisites (git, python3 + PyYAML, age, sops, netbird) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual apt/dnf/brew needed before running it. The only thing you must have on hand BEFORE the curl ... | sudo bash line is curl itself (used to pipe the script).

Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.

Manual recipes (Fedora / Debian / macOS)

# Fedora / RHEL / Nobara
sudo dnf install -y git python3-pyyaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
  -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops

# Debian / Ubuntu
sudo apt update && sudo apt install -y git python3-yaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
  -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops

# macOS
brew install git age sops
pip3 install pyyaml   # if `python3 -c "import yaml"` fails

Run the bootstrap

You need a Gitea read-only personal access token for the initial clone (the in-cluster shared PAT is encrypted at secrets/gitea-readonly-pat.yaml but a new client can't decrypt it before bootstrap — chicken-and-egg). Ask the operator (or generate in Gitea: Settings → Applications → Generate New Token → scope read:repository).

TOKEN=...   # your Gitea PAT, scope read:repository

# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
curl -fsSL -u "dtoro:$TOKEN" \
  https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
  -o /tmp/bootstrap.sh

# Run it.
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp

Flags:

Flag	Effect
--with-mcp	Merges the homelab MCP server into ~/.claude/.mcp.json of the invoking user
--no-secrets	Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists)
--dry-run	Prints actions without executing

The bootstrap is idempotent: re-running on an enrolled client just verifies state, re-issues the age key only if it doesn't match the inventory pubkey, and refreshes the sync timer + symlinks.

Verify

homelab whoami                    # prints hosts/$(hostname).yaml
homelab list                      # shows the full topology
homelab status                    # ping + HTTP-check across hosts/services
homelab secret hello              # decrypt the bootstrap-test secret
systemctl list-timers homelab-context-sync.timer
                                  # next run within ≤5 min

For Claude Code: start a new session — the homelab MCP server appears in ~/.claude/.mcp.json and registers 14 tools (8 context, 5 management, 1 secrets-metadata).

Post-bootstrap: SSH reachability

A new workstation must be reachable from other workstations and must be able to reach every host by short hostname. Run these steps after the bootstrap verify passes:

1. Enable SSH server

# macOS:
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist

# Linux:
sudo systemctl enable --now sshd

2. Generate SSH key (if missing)

ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100

3. Publish pubkey to the repo

cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push

4. Deploy keys to all hosts

From any existing enrolled machine (hubris or another workstation):

ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"

This adds the new workstation's pubkey to hubris and every running LXC.

5. Generate SSH config

homelab ssh-config --install

Verify:

ssh hubris hostname     # should return "hubris" without password
ssh gitea hostname      # should return "gitea" without password
ssh mac-mini hostname   # should return "mac-mini" without password (workstation-to-workstation)

6. Add LAN IP to inventory (if on LAN)

If the workstation has a static or reserved LAN IP, add it to inventory.yaml:

hosts:
  your-hostname:
    lan_ip: 192.168.8.xxx

This gives it a primary LAN entry in the generated SSH config (faster than the Netbird fallback). Commit + push, then:

cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install

Claude Code permissions for fleet ops

By default Claude Code's auto-mode classifier asks for confirmation on every ssh into the mesh. The bootstrap already installs the ssh ControlMaster block so subsequent in-session sshes multiplex, but the first ssh of each session still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns by adding to ~/.claude/settings.json:

{
  "permissions": {
    "defaultMode": "auto",
    "allow": [
      "Bash(ssh -p 22022 *)",
      "Bash(homelab *)"
    ]
  }
}

The first rule covers any ssh to a mesh peer on the homelab netbird port; the second covers all homelab CLI invocations. Both are scoped tight enough that the classifier doesn't gate them but loose enough to handle the variety of arguments.

If you also want the netbird --ssh-jwt-cache-ttl flag rationale to be visible to the classifier (it's not actually durable in 0.71.2, but the ControlMaster block is — see runbook-dpkg-interrupted for context), drop a free-text rule into autoMode.allow describing the authorization. Optional.

Adding a new client to inventory

If the hostname you want isn't yet in inventory, enrollment is a two-step ceremony driven from an existing enrolled client (e.g. hubris). The homelab CLI handles steps 1 + 4; you provide steps 2 + 3.

# 1. On hubris (or any existing client): add the inventory entry.
homelab client add my-new-machine
# Prompts for kind, os, netbird FQDN, role. Commits + pushes.

# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key).

# 3. On the new machine: install deps + run bootstrap (above).
#    Bootstrap calls /issue, receives a fresh age keypair, and prints the
#    public key for the operator to commit back to inventory.

# 4. On hubris: finalize the age public key.
homelab client add my-new-machine --finalize-pubkey age1...
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.

Granting a secret to a new client

Adding a client doesn't grant them every secret. Recipients are explicit per file via .sops.yaml glob rules. To grant a client access to (say) secrets/hello.yaml:

1. Edit .sops.yaml at the repo root, add the client's age_pubkey to the matching creation_rules block.
2. Re-key the existing ciphertext for the new recipient list:

   sops updatekeys -y secrets/hello.yaml
   

3. Commit + push. On the next sync (≤5 min), the client can decrypt.

Removing a client

# From any existing client:
homelab client remove my-old-machine

This:

1. Removes the inventory entry and hosts/my-old-machine.yaml.
2. Runs sops updatekeys -y against every file in secrets/ (operator must first remove the pubkey from .sops.yaml rules).
3. Calls secrets-issuance /revoke (admin-token-gated, on LXC 105) to shred the key file and add the hostname to the denylist.
4. Commits + pushes.

The CLI prints a follow-up checklist that the operator must do manually:

• Revoke the peer in the Netbird console (denies future mesh access).
• Rotate any credentials whose ciphertext the removed client already has on disk. The age key revocation only protects future ciphertext; what's already been pulled is still decryptable until the underlying credential changes.
• Optional: homelab nuke my-old-machine SSHes in, shreds /etc/age/key.txt, removes /opt/homelab-context, disables sync.

Troubleshooting

Symptom	Cause	Fix
no hosts/<hostname>.yaml in the repo	Hostname doesn't match inventory entry	Rename either side (see above)
fatal: could not read Username for 'http://192.168.8.121:3000'	bootstrap.sh's credentials file has wrong scheme	Fixed in commit de6f8be; pull latest bootstrap.sh
gnutls_handshake() failed: TLS connection was non-properly terminated cloning git.hubris.network	Client DNS resolves *.hubris.network to the public VPS IP	Configure split-DNS (LXC 180 / Netbird forwarder) or /etc/hosts override; or use HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git
TLS/SSL connection has been closed (EOF) connecting MCP	Same — mcp.hubris.network resolves to public VPS without this vhost	Same DNS fix
Invalid Host header from MCP server	FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only)	Fixed in commit 6848640; pull latest mcp/server.py and redeploy
python3-yaml install fails on Fedora	Wrong package name	Use python3-pyyaml (Fedora) instead of python3-yaml (Debian)
address already in use for FastMCP	FastMCP defaults to 127.0.0.1:8000	Fixed: server now sets mcp.settings.host/port from env (default 0.0.0.0:9810)
homelab: no age key at /etc/age/key.txt even after bootstrap	/etc/age is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users	Fixed in commit df6aca8: the CLI re-execs sops -d via sudo when invoked as a non-root user. On older deployments, re-link the CLI with sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab after the 5-min sync.
homelab CLI doesn't pick up repo updates	Pre-02db… bootstrap copied the binary instead of symlinking	One-time migration: sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab. New bootstraps use the symlink, which auto-tracks the synced repo.
homelab-context-sync.service journal shows fatal: could not read Username for 'https://git.hubris.network'	Pre-fix bootstrap set the gitea credential helper via git config --global, which writes to /root/.gitconfig — invisible to the systemd timer's git process (no HOME set).	One-time migration: sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials". New bootstraps store the helper in /etc/gitconfig instead.
Chat-mode ! shell can't sudo (a terminal is required to read the password)	Claude Code's ! invocation doesn't allocate a tty, and standard sudo won't read its password from stdin or a non-tty pipe.	Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. /etc/sudoers.d/homelab-self with <user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *).
netbird status -d reports 192.168.8.180:53 ... is Unavailable but DNS actually works	netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic.	Ignore unless dig @192.168.8.180 git.hubris.network also fails — then check dnsmasq on LXC 124.
netbird ssh rejected with JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)	Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. systemctl restart netbird and netbird down/up don't clear it — client/internal/engine_ssh.go bails out of updateSSH() if the SSH server is already running.	Full daemon bounce: sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird. Verify with grep -iE "issuer|audience" /var/log/netbird/client.log | tail. Apply once per peer post-migration.
netbird ssh JWT passes but session closes with user privilege check failed: user dtoro not found: unknown user dtoro	netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have root.	Always use explicit root@ prefix manually: netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted. homelab ssh <host> does this automatically via inventory.yaml's per-host ssh.user field (defaults to root).
homelab ssh hubris (or any host on the LAN) fails with Connection refused or hangs, despite mesh routing being up	Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route.	Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your /usr/local/bin/homelab is a symlink to /opt/homelab-context/bin/homelab it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea.

Changelog

2026-06-02 — SSH reachability post-bootstrap steps

Added a new "Post-bootstrap: SSH reachability" section covering SSH key generation, pubkey publication, deployment to hosts, SSH config generation, and LAN IP registration. New workstations enrolled via this doc will automatically join the universal SSH mesh.

2026-05-31 — cross-link to hermes-agent.md

Added a sibling page covering Nous-Hermes-on-Goose enrollment (hermes-agent.md) and noted it at the top of this page. The Hermes flow extends bootstrap.sh with --with-hermes and homelab client add with the same flag; it does not change the underlying enrollment steps documented here.

2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows

Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full systemctl stop/start required, not restart), (2) user not found from netbird-ssh's local-username default (use explicit root@), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host ssh.user field in inventory.yaml + homelab CLI's ssh_target() helper.

2026-05-20 — initial page

Captures the enrollment flow validated during Phase 2 of the homelab context distribution rollout. hubris + LXC 105 (apps) enrolled; first workstation (republic-laptop) blocked on hostname mismatch, documented the resolution.