oikos/investigations/2026-06-01-mac-mini-onboarding.md

mac-mini onboarding — post-mortem & lessons learned

Onboarded the mac-mini workstation (macOS Sequoia, arm64) into the hubris homelab context system with the --with-hermes profile. What follows is a chronological recap of every hitch, the fix, and the systemic improvements needed so the next workstation takes 5 min instead of an hour.

Session log

Step 1 — clone + symlink

• Manually cloned git.hubris.network/dtoro/Homelab-Docs to /Users/dtoro/Homelab-Docs.
• Created /opt/homelab-context → /Users/dtoro/Homelab-Docs symlink.
• Lesson: bootstrap.sh was designed to do this from scratch, but we'd already cloned by hand. The bootstrap's clone exists; pulling path handled it gracefully.

Step 2 — hostname mismatch

• scutil --get LocalHostName → Davids-Mac-mini
• hostname -s → Mac
• Inventory file: hosts/mac-mini.yaml
• Fix: sudo scutil --set LocalHostName mac-mini && sudo scutil --set HostName mac-mini
• Lesson: The bootstrap and homelab whoami use different hostname resolution. Bootstrap uses scutil --get LocalHostName (correct on macOS), but the homelab CLI binary uses hostname -s. Both need to match the inventory key. On a fresh macOS machine, neither does.

Step 3 — bootstrap dependencies

• pyyaml was missing → pip install pyyaml
• age and sops were missing → brew install age sops
• Netbird was already installed and connected ✓
• Lesson: The bootstrap preflight handles these, but only if running bootstrap.sh from the start. Since we ran it after manual setup, some steps (netbird install) were correctly skipped as already-present.

Step 4 — full bootstrap with --with-mcp --with-hermes

• Ran sudo HOMELAB_GITEA_TOKEN=... bash bootstrap.sh --with-mcp --with-hermes
• Age key issued ✓
• Launchd sync timer installed ✓
• Goose binary installed ✓
• Hermes CLI linked ✓
• MCP config merged ✓
• refresh-creds skipped (not yet a recipient) ⚠️
• Cosmetics: chown: dtoro: illegal group name at the end (benign, macOS group-naming quirk)

Step 5 — finalize from hubris

• Ran homelab client add mac-mini --finalize-pubkey <age...> --with-hermes on hubris
• Push failed: [rejected] main -> main (fetch first) — hubris clone was stale, bootstrap had already pushed from mac-mini
• Fix: git pull --rebase && git push on hubris
• Lesson: bootstrap pushes remote changes before hubris can finalize, creating a race. The homelab client add --finalize-pubkey command should pull before committing/pushing.

Step 6 — sops couldn't find the age key

• homelab secret hello failed because sops looks in /Users/dtoro/.ssh/id_rsa etc. by default, not /etc/age/key.txt
• The homelab CLI re-execs via sudo -E env SOPS_AGE_KEY_FILE=... sops ..., but this requires passwordless sudo and the correct env var passthrough
• Fix:
  1. Added NOPASSWD sudo rules
  2. Eventually SOPS_AGE_KEY env with the raw key content worked directly
• Lesson: Document the explicit SOPS_AGE_KEY_FILE incantation in agent-enrollment troubleshooting. New clients can't assume homelab secret works out of the gate — the sudo re-exec chain has permission pitfall.

Step 7 — OpenRouter key was a placeholder

• secrets/openrouter-api-key.yaml contained api_key: PLACEHOLDER_REPLACE_WITH_REAL_OPENROUTER_KEY
• User ran sops on hubris, but got the same error (age key not found on hubris either — /root/.config/sops/age/keys.txt didn't exist)
• Fix: SOPS_AGE_KEY_FILE=/etc/age/key.txt sops ... on hubris. Later: the user pasted the real key, but the sops file showed sk-or-...5c55 — the literal content was truncated with ellipsis.

Step 8 — editor loaded the wrong data

• Neovim on the system is configured with clipboard+=unnamedplus, which points * and + registers to the macOS clipboard manager rather than X11. When editing SOPS files, this caused the system clipboard to be pasted instead of the actual ciphertext.
• This wasn't diagnosed during the session — the sops file would load empty or show the wrong content because the editor's idea of "paste" was disconnected from what sops expected.
• Fix: Run sops with EDITOR=nano or another editor that doesn't hijack OS clipboards:

  EDITOR=nano SOPS_AGE_KEY_FILE=/etc/age/key.txt sops secrets/openrouter-api-key.yaml
  

• Lesson: Add a strong warning to hermes-agent.md / agent-enrollment.md: macOS neovim with clipboard+=unnamedplus silently breaks sops editing because the paste register reads from the system clipboard instead of the sops-managed buffer. Use EDITOR=nano or EDITOR=vim when running sops interactively. Alternatively, override the clipboard option with EDITOR='nvim -c "set clipboard="'.
• Also useful for the troubleshooting table in agent-enrollment.md under a new row: "sops file loads empty / wrong content on macOS"

Step 9 — model doesn't support tool use

• Goose config pinned nousresearch/hermes-4-405b via OpenRouter
• Error: No endpoints found that support tool use
• Fix: Switched to deepseek/deepseek-v4-flash in ~/.config/goose/config.yaml
• Also updated operations/hermes-agent.md with the correct model
• Lesson: The default model in bootstrap.sh and hermes-agent.md was never validated on OpenRouter for tool-use capability. Need to either: (a) Pin a model known to work (deepseek/deepseek-v4-flash) (b) Or have the bootstrap probe OpenRouter at install time

Step 10 — MCP homelab extension returns 404

• Goose's homelab extension configured as streamable_http at https://mcp.hubris.network/mcp returns HTTP 404
• The actual MCP server runs on SSE (/sse), not streamable_http
• Known follow-up from hermes-agent.md — server needs migration
• Workaround: The agent works fine via the built-in developer extension (shell + file) and the homelab CLI

Step 11 — git credential dance

Several layers of issues with git authentication:

1. Directory permissions: /etc/homelab-context was 0700 root — dtoro couldn't access the credential file, and git's store helper couldn't create its .lock file.

   • Fix 1a: sudo chmod 755 /etc/homelab-context
   • Fix 1b: sudo chown -R dtoro:staff /etc/homelab-context

2. macOS built-in osxkeychain: Apple's git has osxkeychain compiled in as a default credential helper. Even after configuring the store helper, git calls osxkeychain after every successful get, which prompts for macOS keychain access.

   • credential.helper = "" does NOT disable the compiled-in default
   • The store helper's .lock file in /etc/homelab-context/ also failed silently, causing fallback behaviour
   • Fix: Embed the credential in the remote URL directly: https://dtoro:TOKEN@git.hubris.network/dtoro/Homelab-Docs.git
   • This bypasses all credential helpers entirely

3. refresh-creds: The credential is now managed by the remote URL. If the PAT is rotated, homelab refresh-creds won't update the URL.

   • Workaround: Run git remote set-url origin after refresh-creds

Step 12 — homelab CLI hostname detection

• homelab whoami failed with no hosts/Mac.yaml because the CLI uses hostname -s (returning Mac) while the bootstrap uses scutil --get LocalHostName (which we fixed to mac-mini)
• Fix: Set both LocalHostName and HostName via scutil
• Lesson: The homelab CLI on macOS should prefer scutil --get LocalHostName like the bootstrap does, or at least try both and pick the one that matches an inventory key. This is a CLI bug.

Improvement backlog

High priority

#	Issue	Fix
1	Homelab CLI uses hostname -s on macOS; should use scutil --get LocalHostName to match bootstrap	Patch bin/homelab to try scutil first, fall back to hostname -s
2	homelab client add --finalize-pubkey on hubris fails to push when remote is ahead (bootstrap pushes first)	pull --rebase before committing in cmd_client_add
3	Default Goose model (nousresearch/hermes-4-405b) doesn't support tool use on OpenRouter	Update bootstrap.sh default and hermes-agent.md to deepseek/deepseek-v4-flash
4	MCP server needs streamable_http migration	Follow-up #1 from hermes-agent.md — FastMCP transport="sse" → transport="streamable_http"
5	homelab refresh-creds should also update embedded remote URLs	In cmd_refresh_creds, after writing the credential file, update any remote URL that has an embedded credential

Medium priority

#	Issue	Fix
6	macOS neovim with clipboard+=unnamedplus breaks sops editing	Add troubleshooting row to agent-enrollment.md; recommend EDITOR=nano
7	/etc/homelab-context directory with 0700 root permissions blocks non-root git	Change to 755 in bootstrap.sh on all OSes (or at least macOS)
8	Bootstrap chown at end gives illegal group name on macOS	Fix group detection in bootstrap.sh for macOS
9	homelab secret re-exec via sudo chain is fragile — needs passwordless sudo or TTY	Consider SOPS_AGE_KEY env-var fallback path in the CLI itself

Low priority

#	Issue	Fix
10	Onboarding this machine revealed no HERMES.md file in the repo	Create the persona file for the Goose hints symlink
11	Git credential store helper lock mechanism is OS-sensitive and fragile	Document embedded-URL pattern as the preferred approach for workstations
12	AGENTS.md references /opt/homelab-context/hosts/<hostname>.yaml but the canonical path is now a symlink	Verify the path resolution edge cases

Changelog

2026-06-01 — initial post-mortem

Captured the full onboarding of mac-mini into the hubris homelab context system, including the hostname mismatch, sudo/sops credential chain, model selection, git credential gymnastics, and editor clipboard interference.