dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

docs: fix plan/repo drift, retire dead Goose+Nomos and Caveman tooling
Some checks failed
ci / build-test (push) Has been cancelled
Details
ci / docker-build (push) Has been cancelled
Details

Browse Source 
Documentation and repo-hygiene pass following the client/server split:

Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
  described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
  refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
  deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
  to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.

Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).

Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).

Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.

Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.

Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
  places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
  GetClientContext handler) since the mechanism's introduction on
  2026-06-02 — never matched any real filename, so no client has ever
  picked up an auto-setup script via git-pull or the context-poller sync.
  Fixed all three; the Go server-side fix is the one that actually matters
  since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
  (parsed, never consumed) left over from an earlier clone-based model.

Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-07-12 18:19:41 +02:00
parent 0c0f35a3a9
commit d80a394b7f
43 changed files with 609 additions and 1430 deletions
Download Patch File Download Diff File Expand all files Collapse all files

395
.agents/operations/agent-enrollment.md
View File

@@ -1,365 +1,120 @@
# Agent enrollment — bootstrap a client into the homelab context system
# Agent enrollment — operational notes
This walks through enrolling a new machine (workstation, LXC, or VM) so it
joins the cross-client context system: a `/opt/homelab-context/` clone of
this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
**For the actual enrollment flow, see [CLIENTS.md](../../CLIENTS.md#enrollment)
— it's the current, authoritative version.** This page used to duplicate
that flow in more detail, describing a `homelab` CLI-based two-step
ceremony (`homelab client add` reserves an inventory slot → client
bootstraps → operator finalizes the pubkey). That CLI and that flow don't
exist anymore — enrollment today is one shot: `bootstrap.sh` calls
`POST /api/v1/clients/enroll` directly and gets back an age keypair +
Infisical identity in the same response. What's left here is the handful
of things that are still true and weren't already covered elsewhere.
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
> See [nomos-agent.md](nomos-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-nomos` flag.
Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.
## Prerequisites the client must satisfy
## Prerequisites
| Requirement | Why | How to check |
| --- | --- | --- |
| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| Hostname matches an entry in `inventory.yaml` | `EnrollClient` looks up the entity by slug derived from hostname; it must exist in `planned`/`provisioning` state. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |
| On the mesh (Netbird) **or** on the LAN | enrollment validates mesh IP against expected subnets | `netbird status` |
| `curl`, `jq`, `age`, `python3` | bootstrap preflight (`bootstrap.sh:100`) — auto-installed on Fedora/RHEL/Debian/Ubuntu/macOS if missing | `command -v curl jq age python3` |
| Can resolve `*.hubris.network` | bootstrap calls the Oikos API and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` |
### Hostname mismatch is the most common bootstrap failure
If the bootstrap exits with `no hosts/<name>.yaml in the repo`, the
hostname doesn't match any inventory entry. Two fixes:
If the entity for your hostname doesn't exist yet (in `planned` or
`provisioning` state), enrollment 4xxs. Two fixes:
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
(Linux) or System Preferences → Sharing (macOS), then re-run.
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
update `inventory.yaml`, push. The next sync (≤5 min) propagates.
- **Rename the host** to match an existing planned entity:
`sudo hostnamectl set-hostname <inventory-name>` (Linux) or System
Preferences → Sharing (macOS), then re-run.
- **Add/rename the inventory entry**: edit `seeds/inventory.yaml`, ingest
via `oikos seed` (or the equivalent MCP/API entity-creation path), then
re-run bootstrap.
### Getting onto Netbird
### Networking prerequisites (Netbird, DNS, SSH key distribution)
Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit `<bootstrap-tier1>`). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`.
**Path B — interactive OIDC (default; recommended):**
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../archive/knowledge/containers/106-auth-outpost.md). Only needed once per account.
**Path A — setup-key (headless/scripted onboarding):**
Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.
1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`.
2. **Setup Keys** → Create → set reusable + expiry → copy.
3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver):
```bash
sudo netbird up --setup-key <KEY> \
--management-url https://netbird.hubris.network \
--ssh-jwt-cache-ttl 86400
```
**Why we can't OIDC-login from the public internet (still open as a follow-up):**
`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124.
### DNS prerequisite
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../../archive/knowledge/infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:
- **LAN client**: set DNS to 192.168.8.180 (per-interface or
`/etc/resolv.conf`).
- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
point `*.hubris.network` at LXC 124.
- **Hack-fix anywhere**: append to `/etc/hosts`:
```
192.168.8.175 mcp.hubris.network secrets.hubris.network
192.168.8.175 git.hubris.network
```
(192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.)
If DNS isn't an option at all, override the URLs at bootstrap time:
```bash
sudo HOMELAB_GITEA_TOKEN=... \
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git \
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
bash /tmp/bootstrap.sh --with-mcp
```
## Install dependencies
Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script).
Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.
<details>
<summary>Manual recipes (Fedora / Debian / macOS)</summary>
```bash
# Fedora / RHEL / Nobara
sudo dnf install -y git python3-pyyaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
-o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
# Debian / Ubuntu
sudo apt update && sudo apt install -y git python3-yaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
-o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
# macOS
brew install git age sops
pip3 install pyyaml # if `python3 -c "import yaml"` fails
```
</details>
## Run the bootstrap
You need a Gitea read-only personal access token for the initial clone
(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml`
but a new client can't decrypt it before bootstrap — chicken-and-egg).
Ask the operator (or generate in Gitea: Settings → Applications → Generate
New Token → scope `read:repository`).
```bash
TOKEN=... # your Gitea PAT, scope read:repository
# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
curl -fsSL -u "dtoro:$TOKEN" \
https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \
-o /tmp/bootstrap.sh
# Run it.
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp
```
Flags:
| Flag | Effect |
| --- | --- |
| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user |
| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) |
| `--dry-run` | Prints actions without executing |
The bootstrap is idempotent: re-running on an enrolled client just
verifies state, re-issues the age key only if it doesn't match the
inventory pubkey, and refreshes the sync timer + symlinks.
## Verify
```bash
homelab whoami # prints hosts/$(hostname).yaml
homelab list # shows the full topology
homelab status # ping + HTTP-check across hosts/services
homelab secret hello # decrypt the bootstrap-test secret
systemctl list-timers homelab-context-sync.timer
# next run within ≤5 min
```
For Claude Code: start a new session — the `homelab` MCP server appears
in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
1 secrets-metadata).
## Post-bootstrap: SSH reachability
A new workstation must be reachable from other workstations and must be
able to reach every host by short hostname. Run these steps after the
bootstrap verify passes:
### 1. Enable SSH server
```bash
# macOS:
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
# Linux:
sudo systemctl enable --now sshd
```
### 2. Generate SSH key (if missing)
```bash
ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100
```
### 3. Publish pubkey to the repo
```bash
cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push
```
### 4. Deploy keys to all hosts
From any existing enrolled machine (hubris or another workstation):
```bash
ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
```
This adds the new workstation's pubkey to hubris and every running LXC.
### 5. Generate SSH config
```bash
homelab ssh-config --install
```
Verify:
```bash
ssh hubris hostname # should return "hubris" without password
ssh gitea hostname # should return "gitea" without password
ssh mac-mini hostname # should return "mac-mini" without password (workstation-to-workstation)
```
### 6. Add LAN IP to inventory (if on LAN)
If the workstation has a static or reserved LAN IP, add it to
`inventory.yaml`:
```yaml
hosts:
your-hostname:
lan_ip: 192.168.8.xxx
```
This gives it a primary LAN entry in the generated SSH config (faster
than the Netbird fallback). Commit + push, then:
```bash
cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install
```
Migrated to a runbook in the knowledge base — query
`search_knowledge("netbird mesh dns")` or `get_entity_knowledge`, or ask
Nomos. Covers: getting onto the Netbird mesh (interactive OIDC vs.
setup-key), why OIDC login can fail from off-mesh, split-horizon DNS
options, and distributing a new workstation's SSH pubkey across the fleet
via `ssh/deploy-keys.sh`.
## Claude Code permissions for fleet ops
By default Claude Code's auto-mode classifier asks for confirmation on every
ssh into the mesh. The bootstrap already installs the ssh ControlMaster block
so subsequent in-session sshes multiplex, but the *first* ssh of each session
still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns
by adding to `~/.claude/settings.json`:
ssh into the mesh. Pre-authorize the common fleet ssh pattern by adding to
`~/.claude/settings.json`:
```json
{
"permissions": {
"defaultMode": "auto",
"allow": [
"Bash(ssh -p 22022 *)",
"Bash(homelab *)"
"Bash(ssh -p 22022 *)"
]
}
}
```
The first rule covers any ssh to a mesh peer on the homelab netbird port; the
second covers all `homelab` CLI invocations. Both are scoped tight enough that
the classifier doesn't gate them but loose enough to handle the variety of
arguments.
This covers any ssh to a mesh peer on the homelab netbird port, scoped tight
enough that the classifier doesn't gate it but loose enough to handle the
variety of arguments.
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
visible to the classifier (it's not actually durable in 0.71.2, but the
ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md)
for context), drop a free-text rule into `autoMode.allow` describing the
authorization. Optional.
## Open questions (not verified against current architecture — don't
guess these from the old flow)
## Adding a new client to inventory
The old two-step ceremony had answers for these; the current one-shot
`/api/v1/clients/enroll` flow may handle them differently and this hasn't
been re-verified:
If the hostname you want isn't yet in inventory, enrollment is a two-step
ceremony driven from an existing enrolled client (e.g. hubris). The
`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3.
```bash
# 1. On hubris (or any existing client): add the inventory entry.
homelab client add my-new-machine
# Prompts for kind, os, netbird FQDN, role. Commits + pushes.
# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key).
# 3. On the new machine: install deps + run bootstrap (above).
# Bootstrap calls /issue, receives a fresh age keypair, and prints the
# public key for the operator to commit back to inventory.
# 4. On hubris: finalize the age public key.
homelab client add my-new-machine --finalize-pubkey age1...
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
# inventory.yaml, commits + pushes. The 5-min sync propagates.
```
## Granting a secret to a new client
Adding a client doesn't grant them every secret. Recipients are explicit
per file via `.sops.yaml` glob rules. To grant a client access to (say)
`secrets/hello.yaml`:
1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to
the matching `creation_rules` block.
2. Re-key the existing ciphertext for the new recipient list:
```bash
sops updatekeys -y secrets/hello.yaml
```
3. Commit + push. On the next sync (≤5 min), the client can decrypt.
## Removing a client
```bash
# From any existing client:
homelab client remove my-old-machine
```
This:
1. Removes the inventory entry and `hosts/my-old-machine.yaml`.
2. Runs `sops updatekeys -y` against every file in `secrets/` (operator
must first remove the pubkey from `.sops.yaml` rules).
3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to
shred the key file and add the hostname to the denylist.
4. Commits + pushes.
The CLI prints a follow-up checklist that the operator must do manually:
- Revoke the peer in the Netbird console (denies future mesh access).
- **Rotate any credentials whose ciphertext the removed client already
has on disk.** The age key revocation only protects *future*
ciphertext; what's already been pulled is still decryptable until the
underlying credential changes.
- Optional: `homelab nuke my-old-machine` SSHes in, shreds
`/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync.
- **Removing a client.** No current equivalent confirmed for the old
`homelab client remove` (inventory removal + secret re-keying + key
revocation). Likely maps to an entity lifecycle transition
(`.agents/skills/lifecycle-deprecate-node/` or `lifecycle-destroy-node/`)
but those skills reference the same dead CLI and need their own check.
- **Granting a secret to an already-enrolled client.** The old flow
hand-edited `.sops.yaml` `creation_rules` + `sops updatekeys`. Given
Infisical is now the primary secrets backend (SOPS is the DR fallback),
the current mechanism is probably Infisical-side, not a `.sops.yaml` edit
— not confirmed.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git` |
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Enrollment 404s / entity not found | Hostname doesn't match a `planned`/`provisioning` inventory entry | See "Hostname mismatch" above |
| `gnutls_handshake() failed` / TLS errors reaching `*.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS instead of the LAN/mesh path | See the networking runbook (split-horizon DNS section) |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../archive/knowledge/containers/106-auth-outpost.md). |
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
## Changelog
### 2026-06-02 — SSH reachability post-bootstrap steps
Added a new "Post-bootstrap: SSH reachability" section covering SSH key
generation, pubkey publication, deployment to hosts, SSH config generation,
and LAN IP registration. New workstations enrolled via this doc will
automatically join the universal SSH mesh.
### 2026-05-31 — cross-link to nomos-agent.md
### 2026-07-12 — trimmed to current architecture
Removed everything describing the retired `homelab` CLI-based two-step
enrollment ceremony (now: `CLIENTS.md`'s one-shot flow), the Nous-Hermes/
Goose cross-link (that whole flow was removed the same day), and CLI-syntax
troubleshooting rows with no current equivalent. Migrated the still-true
Netbird/DNS/SSH-distribution content to a knowledge-base runbook rather
than duplicating it here. What's left is genuinely current or explicitly
flagged as unverified. Original ~365-line version is in git history
(`git log -- .agents/operations/agent-enrollment.md`) if any of the removed
detail turns out to still be needed.
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([nomos-agent.md](nomos-agent.md)) and noted it at the top of this page. The Nomos flow extends `bootstrap.sh` with `--with-nomos` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
### 2026-06-02 — SSH reachability post-bootstrap steps
Added a section covering SSH key generation, pubkey publication,
deployment to hosts, SSH config generation, and LAN IP registration. New
workstations enrolled via this doc automatically join the SSH mesh.
(Superseded 2026-07-12 — migrated to the networking runbook.)
### 2026-05-31 — cross-link to nomos-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment. (Removed
2026-07-12 along with the rest of that flow.)
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
Added three rows to the troubleshooting table covering issues surfaced
during the netbird vanilla migration. (Migrated 2026-07-12 to the
networking runbook.)
### 2026-05-20 — initial page
Captures the enrollment flow validated during Phase 2 of the homelab