oikos/knowledge/sources/investigations/2026-06-01-mac-mini-onboarding.md

# mac-mini onboarding — post-mortem & lessons learned

Onboarded the `mac-mini` workstation (macOS Sequoia, arm64) into the hubris
homelab context system with the `--with-hermes` profile. What follows is a
chronological recap of every hitch, the fix, and the systemic improvements
needed so the next workstation takes 5 min instead of an hour.

## Session log

### Step 1 — clone + symlink
- Manually cloned `git.hubris.network/dtoro/Homelab-Docs` to `/Users/dtoro/Homelab-Docs`.
- Created `/opt/homelab-context` → `/Users/dtoro/Homelab-Docs` symlink.
- **Lesson:** bootstrap.sh was designed to do this from scratch, but we'd
  already cloned by hand. The bootstrap's `clone exists; pulling` path handled
  it gracefully.

### Step 2 — hostname mismatch
- `scutil --get LocalHostName` → `Davids-Mac-mini`
- `hostname -s` → `Mac`
- Inventory file: `hosts/mac-mini.yaml`
- **Fix:** `sudo scutil --set LocalHostName mac-mini && sudo scutil --set HostName mac-mini`
- **Lesson:** The bootstrap and `homelab whoami` use different hostname
  resolution. Bootstrap uses `scutil --get LocalHostName` (correct on macOS),
  but the `homelab` CLI binary uses `hostname -s`. Both need to match the
  inventory key. On a fresh macOS machine, neither does.

### Step 3 — bootstrap dependencies
- pyyaml was missing → `pip install pyyaml`
- age and sops were missing → `brew install age sops`
- Netbird was already installed and connected ✓
- **Lesson:** The bootstrap preflight handles these, but only if running
  `bootstrap.sh` from the start. Since we ran it after manual setup, some
  steps (netbird install) were correctly skipped as already-present.

### Step 4 — full bootstrap with `--with-mcp --with-hermes`
- Ran `sudo HOMELAB_GITEA_TOKEN=... bash bootstrap.sh --with-mcp --with-hermes`
- Age key issued ✓
- Launchd sync timer installed ✓
- Goose binary installed ✓
- Hermes CLI linked ✓
- MCP config merged ✓
- `refresh-creds` skipped (not yet a recipient) ⚠️
- Cosmetics: `chown: dtoro: illegal group name` at the end (benign, macOS
  group-naming quirk)

### Step 5 — finalize from hubris
- Ran `homelab client add mac-mini --finalize-pubkey <age...> --with-hermes` on
  hubris
- Push failed: `[rejected] main -> main (fetch first)` — hubris clone was
  stale, bootstrap had already pushed from mac-mini
- **Fix:** `git pull --rebase && git push` on hubris
- **Lesson:** bootstrap pushes remote changes before hubris can finalize,
  creating a race. The `homelab client add --finalize-pubkey` command should
  pull before committing/pushing.

### Step 6 — sops couldn't find the age key
- `homelab secret hello` failed because sops looks in
  `/Users/dtoro/.ssh/id_rsa` etc. by default, not `/etc/age/key.txt`
- The `homelab` CLI re-execs via `sudo -E env SOPS_AGE_KEY_FILE=... sops ...`,
  but this requires passwordless sudo and the correct env var passthrough
- **Fix:**
  1. Added NOPASSWD sudo rules
  2. Eventually `SOPS_AGE_KEY` env with the raw key content worked directly
- **Lesson:** Document the explicit `SOPS_AGE_KEY_FILE` incantation in
  agent-enrollment troubleshooting. New clients can't assume `homelab secret`
  works out of the gate — the sudo re-exec chain has permission pitfall.

### Step 7 — OpenRouter key was a placeholder
- `secrets/openrouter-api-key.yaml` contained
  `api_key: PLACEHOLDER_REPLACE_WITH_REAL_OPENROUTER_KEY`
- User ran `sops` on hubris, but got the same error (age key not found on
  hubris either — `/root/.config/sops/age/keys.txt` didn't exist)
- **Fix:** `SOPS_AGE_KEY_FILE=/etc/age/key.txt sops ...` on hubris.
  Later: the user pasted the real key, but the sops file showed
  `sk-or-...5c55` — the literal content was truncated with ellipsis.

### Step 8 — editor loaded the wrong data
- Neovim on the system is configured with `clipboard+=unnamedplus`, which
  points `*` and `+` registers to the macOS clipboard manager rather than
  X11. When editing SOPS files, this caused the **system clipboard** to be
  pasted instead of the actual ciphertext.
- This wasn't diagnosed during the session — the sops file would load empty
  or show the wrong content because the editor's idea of "paste" was
  disconnected from what sops expected.
- **Fix:** Run `sops` with `EDITOR=nano` or another editor that doesn't
  hijack OS clipboards:
  ```bash
  EDITOR=nano SOPS_AGE_KEY_FILE=/etc/age/key.txt sops secrets/openrouter-api-key.yaml
  ```
- **Lesson:** Add a strong warning to `hermes-agent.md` / `agent-enrollment.md`:
  macOS neovim with `clipboard+=unnamedplus` silently breaks sops editing
  because the paste register reads from the system clipboard instead of the
  sops-managed buffer. Use `EDITOR=nano` or `EDITOR=vim` when running sops
  interactively. Alternatively, override the clipboard option with
  `EDITOR='nvim -c "set clipboard="'`.
- Also useful for the troubleshooting table in `agent-enrollment.md` under a
  new row: "sops file loads empty / wrong content on macOS"

### Step 9 — model doesn't support tool use
- Goose config pinned `nousresearch/hermes-4-405b` via OpenRouter
- Error: `No endpoints found that support tool use`
- **Fix:** Switched to `deepseek/deepseek-v4-flash` in
  `~/.config/goose/config.yaml`
- Also updated `operations/hermes-agent.md` with the correct model
- **Lesson:** The default model in `bootstrap.sh` and `hermes-agent.md` was
  never validated on OpenRouter for tool-use capability. Need to either:
  (a) Pin a model known to work (`deepseek/deepseek-v4-flash`)
  (b) Or have the bootstrap probe OpenRouter at install time

### Step 10 — MCP homelab extension returns 404
- Goose's homelab extension configured as `streamable_http` at
  `https://mcp.hubris.network/mcp` returns HTTP 404
- The actual MCP server runs on SSE (`/sse`), not streamable_http
- **Known follow-up** from `hermes-agent.md` — server needs migration
- **Workaround:** The agent works fine via the built-in `developer` extension
  (shell + file) and the `homelab` CLI

### Step 11 — git credential dance
Several layers of issues with git authentication:

1. **Directory permissions:** `/etc/homelab-context` was `0700 root` — dtoro
   couldn't access the credential file, and git's `store` helper couldn't
   create its `.lock` file.
   - **Fix 1a:** `sudo chmod 755 /etc/homelab-context`
   - **Fix 1b:** `sudo chown -R dtoro:staff /etc/homelab-context`

2. **macOS built-in osxkeychain:** Apple's git has `osxkeychain` compiled in
   as a default credential helper. Even after configuring the `store` helper,
   git calls osxkeychain after every successful `get`, which prompts for
   macOS keychain access.
   - `credential.helper = ""` does NOT disable the compiled-in default
   - The `store` helper's `.lock` file in `/etc/homelab-context/` also failed
     silently, causing fallback behaviour
   - **Fix:** Embed the credential in the remote URL directly:
     `https://dtoro:TOKEN@git.hubris.network/dtoro/Homelab-Docs.git`
   - This bypasses all credential helpers entirely

3. **refresh-creds:** The credential is now managed by the remote URL.
   If the PAT is rotated, `homelab refresh-creds` won't update the URL.
   - **Workaround:** Run `git remote set-url origin` after `refresh-creds`

### Step 12 — `homelab` CLI hostname detection
- `homelab whoami` failed with `no hosts/Mac.yaml` because the CLI uses
  `hostname -s` (returning `Mac`) while the bootstrap uses
  `scutil --get LocalHostName` (which we fixed to `mac-mini`)
- **Fix:** Set both `LocalHostName` and `HostName` via `scutil`
- **Lesson:** The homelab CLI on macOS should prefer `scutil --get
  LocalHostName` like the bootstrap does, or at least try both and pick the
  one that matches an inventory key. This is a CLI bug.

## Improvement backlog

### High priority

| # | Issue | Fix |
|---|-------|-----|
| 1 | Homelab CLI uses `hostname -s` on macOS; should use `scutil --get LocalHostName` to match bootstrap | Patch `bin/homelab` to try `scutil` first, fall back to `hostname -s` |
| 2 | `homelab client add --finalize-pubkey` on hubris fails to push when remote is ahead (bootstrap pushes first) | `pull --rebase` before committing in `cmd_client_add` |
| 3 | Default Goose model (`nousresearch/hermes-4-405b`) doesn't support tool use on OpenRouter | Update `bootstrap.sh` default and `hermes-agent.md` to `deepseek/deepseek-v4-flash` |
| 4 | MCP server needs streamable_http migration | Follow-up #1 from `hermes-agent.md` — FastMCP `transport="sse"` → `transport="streamable_http"` |
| 5 | `homelab refresh-creds` should also update embedded remote URLs | In `cmd_refresh_creds`, after writing the credential file, update any remote URL that has an embedded credential |

### Medium priority

| # | Issue | Fix |
|---|-------|-----|
| 6 | macOS neovim with `clipboard+=unnamedplus` breaks sops editing | Add troubleshooting row to `agent-enrollment.md`; recommend `EDITOR=nano` |
| 7 | `/etc/homelab-context` directory with 0700 root permissions blocks non-root git | Change to 755 in `bootstrap.sh` on all OSes (or at least macOS) |
| 8 | Bootstrap `chown` at end gives `illegal group name` on macOS | Fix group detection in bootstrap.sh for macOS |
| 9 | `homelab secret` re-exec via sudo chain is fragile — needs passwordless sudo or TTY | Consider `SOPS_AGE_KEY` env-var fallback path in the CLI itself |

### Low priority

| # | Issue | Fix |
|---|-------|-----|
| 10 | Onboarding this machine revealed no `HERMES.md` file in the repo | Create the persona file for the Goose hints symlink |
| 11 | Git credential `store` helper lock mechanism is OS-sensitive and fragile | Document embedded-URL pattern as the preferred approach for workstations |
| 12 | AGENTS.md references `/opt/homelab-context/hosts/<hostname>.yaml` but the canonical path is now a symlink | Verify the path resolution edge cases |

## Changelog

### 2026-06-01 — initial post-mortem
Captured the full onboarding of mac-mini into the hubris homelab context
system, including the hostname mismatch, sudo/sops credential chain, model
selection, git credential gymnastics, and editor clipboard interference.

### 2026-06-02 — MCP server migrated to streamable-http
Item #4 resolved. FastMCP server now uses `transport="streamable-http"` (hyphen,
not underscore), serving at `https://mcp.hubris.network/mcp`. All doc references
and client configs (Goose, bootstrap.sh, agent-enrollment.md) updated. Service
re-deployed on apps LXC (105).