From daf4747c5c4171d730bc4d03cacc09b558e7c2ec Mon Sep 17 00:00:00 2001 From: dtoro Date: Mon, 1 Jun 2026 23:45:22 +0200 Subject: [PATCH] =?UTF-8?q?post-mortem:=20mac-mini=20onboarding=20?= =?UTF-8?q?=E2=80=94=2012=20issues=20found?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-06-01-mac-mini-onboarding.md | 185 ++++++++++++++++++ 1 file changed, 185 insertions(+) create mode 100644 investigations/2026-06-01-mac-mini-onboarding.md diff --git a/investigations/2026-06-01-mac-mini-onboarding.md b/investigations/2026-06-01-mac-mini-onboarding.md new file mode 100644 index 0000000..99eb1ce --- /dev/null +++ b/investigations/2026-06-01-mac-mini-onboarding.md @@ -0,0 +1,185 @@ +# mac-mini onboarding — post-mortem & lessons learned + +Onboarded the `mac-mini` workstation (macOS Sequoia, arm64) into the hubris +homelab context system with the `--with-hermes` profile. What follows is a +chronological recap of every hitch, the fix, and the systemic improvements +needed so the next workstation takes 5 min instead of an hour. + +## Session log + +### Step 1 — clone + symlink +- Manually cloned `git.hubris.network/dtoro/Homelab-Docs` to `/Users/dtoro/Homelab-Docs`. +- Created `/opt/homelab-context` → `/Users/dtoro/Homelab-Docs` symlink. +- **Lesson:** bootstrap.sh was designed to do this from scratch, but we'd + already cloned by hand. The bootstrap's `clone exists; pulling` path handled + it gracefully. + +### Step 2 — hostname mismatch +- `scutil --get LocalHostName` → `Davids-Mac-mini` +- `hostname -s` → `Mac` +- Inventory file: `hosts/mac-mini.yaml` +- **Fix:** `sudo scutil --set LocalHostName mac-mini && sudo scutil --set HostName mac-mini` +- **Lesson:** The bootstrap and `homelab whoami` use different hostname + resolution. Bootstrap uses `scutil --get LocalHostName` (correct on macOS), + but the `homelab` CLI binary uses `hostname -s`. Both need to match the + inventory key. On a fresh macOS machine, neither does. + +### Step 3 — bootstrap dependencies +- pyyaml was missing → `pip install pyyaml` +- age and sops were missing → `brew install age sops` +- Netbird was already installed and connected ✓ +- **Lesson:** The bootstrap preflight handles these, but only if running + `bootstrap.sh` from the start. Since we ran it after manual setup, some + steps (netbird install) were correctly skipped as already-present. + +### Step 4 — full bootstrap with `--with-mcp --with-hermes` +- Ran `sudo HOMELAB_GITEA_TOKEN=... bash bootstrap.sh --with-mcp --with-hermes` +- Age key issued ✓ +- Launchd sync timer installed ✓ +- Goose binary installed ✓ +- Hermes CLI linked ✓ +- MCP config merged ✓ +- `refresh-creds` skipped (not yet a recipient) ⚠️ +- Cosmetics: `chown: dtoro: illegal group name` at the end (benign, macOS + group-naming quirk) + +### Step 5 — finalize from hubris +- Ran `homelab client add mac-mini --finalize-pubkey --with-hermes` on + hubris +- Push failed: `[rejected] main -> main (fetch first)` — hubris clone was + stale, bootstrap had already pushed from mac-mini +- **Fix:** `git pull --rebase && git push` on hubris +- **Lesson:** bootstrap pushes remote changes before hubris can finalize, + creating a race. The `homelab client add --finalize-pubkey` command should + pull before committing/pushing. + +### Step 6 — sops couldn't find the age key +- `homelab secret hello` failed because sops looks in + `/Users/dtoro/.ssh/id_rsa` etc. by default, not `/etc/age/key.txt` +- The `homelab` CLI re-execs via `sudo -E env SOPS_AGE_KEY_FILE=... sops ...`, + but this requires passwordless sudo and the correct env var passthrough +- **Fix:** + 1. Added NOPASSWD sudo rules + 2. Eventually `SOPS_AGE_KEY` env with the raw key content worked directly +- **Lesson:** Document the explicit `SOPS_AGE_KEY_FILE` incantation in + agent-enrollment troubleshooting. New clients can't assume `homelab secret` + works out of the gate — the sudo re-exec chain has permission pitfall. + +### Step 7 — OpenRouter key was a placeholder +- `secrets/openrouter-api-key.yaml` contained + `api_key: PLACEHOLDER_REPLACE_WITH_REAL_OPENROUTER_KEY` +- User ran `sops` on hubris, but got the same error (age key not found on + hubris either — `/root/.config/sops/age/keys.txt` didn't exist) +- **Fix:** `SOPS_AGE_KEY_FILE=/etc/age/key.txt sops ...` on hubris. + Later: the user pasted the real key, but the sops file showed + `sk-or-...5c55` — the literal content was truncated with ellipsis. + +### Step 8 — editor loaded the wrong data +- Neovim on the system is configured with `clipboard+=unnamedplus`, which + points `*` and `+` registers to the macOS clipboard manager rather than + X11. When editing SOPS files, this caused the **system clipboard** to be + pasted instead of the actual ciphertext. +- This wasn't diagnosed during the session — the sops file would load empty + or show the wrong content because the editor's idea of "paste" was + disconnected from what sops expected. +- **Fix:** Run `sops` with `EDITOR=nano` or another editor that doesn't + hijack OS clipboards: + ```bash + EDITOR=nano SOPS_AGE_KEY_FILE=/etc/age/key.txt sops secrets/openrouter-api-key.yaml + ``` +- **Lesson:** Add a strong warning to `hermes-agent.md` / `agent-enrollment.md`: + macOS neovim with `clipboard+=unnamedplus` silently breaks sops editing + because the paste register reads from the system clipboard instead of the + sops-managed buffer. Use `EDITOR=nano` or `EDITOR=vim` when running sops + interactively. Alternatively, override the clipboard option with + `EDITOR='nvim -c "set clipboard="'`. +- Also useful for the troubleshooting table in `agent-enrollment.md` under a + new row: "sops file loads empty / wrong content on macOS" + +### Step 9 — model doesn't support tool use +- Goose config pinned `nousresearch/hermes-4-405b` via OpenRouter +- Error: `No endpoints found that support tool use` +- **Fix:** Switched to `deepseek/deepseek-v4-flash` in + `~/.config/goose/config.yaml` +- Also updated `operations/hermes-agent.md` with the correct model +- **Lesson:** The default model in `bootstrap.sh` and `hermes-agent.md` was + never validated on OpenRouter for tool-use capability. Need to either: + (a) Pin a model known to work (`deepseek/deepseek-v4-flash`) + (b) Or have the bootstrap probe OpenRouter at install time + +### Step 10 — MCP homelab extension returns 404 +- Goose's homelab extension configured as `streamable_http` at + `https://mcp.hubris.network/mcp` returns HTTP 404 +- The actual MCP server runs on SSE (`/sse`), not streamable_http +- **Known follow-up** from `hermes-agent.md` — server needs migration +- **Workaround:** The agent works fine via the built-in `developer` extension + (shell + file) and the `homelab` CLI + +### Step 11 — git credential dance +Several layers of issues with git authentication: + +1. **Directory permissions:** `/etc/homelab-context` was `0700 root` — dtoro + couldn't access the credential file, and git's `store` helper couldn't + create its `.lock` file. + - **Fix 1a:** `sudo chmod 755 /etc/homelab-context` + - **Fix 1b:** `sudo chown -R dtoro:staff /etc/homelab-context` + +2. **macOS built-in osxkeychain:** Apple's git has `osxkeychain` compiled in + as a default credential helper. Even after configuring the `store` helper, + git calls osxkeychain after every successful `get`, which prompts for + macOS keychain access. + - `credential.helper = ""` does NOT disable the compiled-in default + - The `store` helper's `.lock` file in `/etc/homelab-context/` also failed + silently, causing fallback behaviour + - **Fix:** Embed the credential in the remote URL directly: + `https://dtoro:TOKEN@git.hubris.network/dtoro/Homelab-Docs.git` + - This bypasses all credential helpers entirely + +3. **refresh-creds:** The credential is now managed by the remote URL. + If the PAT is rotated, `homelab refresh-creds` won't update the URL. + - **Workaround:** Run `git remote set-url origin` after `refresh-creds` + +### Step 12 — `homelab` CLI hostname detection +- `homelab whoami` failed with `no hosts/Mac.yaml` because the CLI uses + `hostname -s` (returning `Mac`) while the bootstrap uses + `scutil --get LocalHostName` (which we fixed to `mac-mini`) +- **Fix:** Set both `LocalHostName` and `HostName` via `scutil` +- **Lesson:** The homelab CLI on macOS should prefer `scutil --get + LocalHostName` like the bootstrap does, or at least try both and pick the + one that matches an inventory key. This is a CLI bug. + +## Improvement backlog + +### High priority + +| # | Issue | Fix | +|---|-------|-----| +| 1 | Homelab CLI uses `hostname -s` on macOS; should use `scutil --get LocalHostName` to match bootstrap | Patch `bin/homelab` to try `scutil` first, fall back to `hostname -s` | +| 2 | `homelab client add --finalize-pubkey` on hubris fails to push when remote is ahead (bootstrap pushes first) | `pull --rebase` before committing in `cmd_client_add` | +| 3 | Default Goose model (`nousresearch/hermes-4-405b`) doesn't support tool use on OpenRouter | Update `bootstrap.sh` default and `hermes-agent.md` to `deepseek/deepseek-v4-flash` | +| 4 | MCP server needs streamable_http migration | Follow-up #1 from `hermes-agent.md` — FastMCP `transport="sse"` → `transport="streamable_http"` | +| 5 | `homelab refresh-creds` should also update embedded remote URLs | In `cmd_refresh_creds`, after writing the credential file, update any remote URL that has an embedded credential | + +### Medium priority + +| # | Issue | Fix | +|---|-------|-----| +| 6 | macOS neovim with `clipboard+=unnamedplus` breaks sops editing | Add troubleshooting row to `agent-enrollment.md`; recommend `EDITOR=nano` | +| 7 | `/etc/homelab-context` directory with 0700 root permissions blocks non-root git | Change to 755 in `bootstrap.sh` on all OSes (or at least macOS) | +| 8 | Bootstrap `chown` at end gives `illegal group name` on macOS | Fix group detection in bootstrap.sh for macOS | +| 9 | `homelab secret` re-exec via sudo chain is fragile — needs passwordless sudo or TTY | Consider `SOPS_AGE_KEY` env-var fallback path in the CLI itself | + +### Low priority + +| # | Issue | Fix | +|---|-------|-----| +| 10 | Onboarding this machine revealed no `HERMES.md` file in the repo | Create the persona file for the Goose hints symlink | +| 11 | Git credential `store` helper lock mechanism is OS-sensitive and fragile | Document embedded-URL pattern as the preferred approach for workstations | +| 12 | AGENTS.md references `/opt/homelab-context/hosts/.yaml` but the canonical path is now a symlink | Verify the path resolution edge cases | + +## Changelog + +### 2026-06-01 — initial post-mortem +Captured the full onboarding of mac-mini into the hubris homelab context +system, including the hostname mismatch, sudo/sops credential chain, model +selection, git credential gymnastics, and editor clipboard interference.