docs: streamline & consolidate the tree (phase 6)
Problem: after the wiki-hq reorg, agent-instruction and human-doc domains were still scattered across the repo root, with three now-redundant stub files cluttering it. The organizing principle wasn't visible in the layout. Change — enforce three clear buckets: - .agents/ = how agents operate: OIKOS.md, HERMES.md (moved from root), shared/ conventions, domains/ schemas, skills/, and operations/ (operator cheatsheet + enrollment + hermes-agent, moved from root). - knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now including investigations/ (incident records are evidence/sources). - root = substrate + two entry points (AGENTS.md, README.md), plus plans/ as its own design-intent domain. Moves: - investigations/ -> knowledge/sources/investigations/ (incl. archive/, index). - operations/ -> .agents/operations/. - HERMES.md -> .agents/HERMES.md. - Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md (its 7 remaining linkers repointed to .agents/OIKOS.md). Consumers updated: - inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards. - tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md. - bin/homelab help string -> .agents/operations/hermes-agent.md. - knowledge/operations schemas, llm-wiki, page-templates, incident-investigation skill, AGENTS.md/README nav -> new investigations/operations paths. - All markdown links rewritten via the path-resolving mapper. Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/, plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml. Verification: docs-lint at baseline (2 intentional cross-repo refs, no new breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all doc_page targets resolve; Hermes provisioning scripts point at the new path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
96
.agents/HERMES.md
Normal file
96
.agents/HERMES.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# HERMES.md — Agent persona for homelab clients
|
||||
|
||||
This file is the canonical agent persona for **all** AI agents running on
|
||||
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
|
||||
conventions, and the source-of-truth hierarchy.
|
||||
|
||||
## Source of truth
|
||||
|
||||
The homelab-context repo at `/opt/homelab-context/` is the single source of
|
||||
truth for:
|
||||
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
|
||||
- Service endpoints and credentials (via `homelab secret`)
|
||||
- Agent behaviour and conventions
|
||||
- Everything in this file
|
||||
|
||||
When in doubt, check `/opt/homelab-context/` first.
|
||||
|
||||
## Runbooks — load, don't rediscover
|
||||
|
||||
For the canonical workflows (service health check, config change +
|
||||
deploy, client enrollment, incident investigation, and each node
|
||||
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
|
||||
acting. Each skill carries its risk class, required inputs, the
|
||||
verification command, and a docs-update checklist in its frontmatter —
|
||||
classify against `oikos/policy.yaml` using that risk class before any
|
||||
mutation. Don't re-derive topology or the mutation path by grepping the
|
||||
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
|
||||
operating model these runbooks execute inside (OODA loop, risk classes,
|
||||
approval flow, ontology).
|
||||
|
||||
## Agent type — how this file gets loaded
|
||||
|
||||
| Agent | Loading mechanism |
|
||||
|-------|------------------|
|
||||
| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
|
||||
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints` → `/opt/homelab-context/HERMES.md` |
|
||||
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
|
||||
|
||||
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
|
||||
homelab-context repo instead. Changes propagate to all clients on the next
|
||||
sync (`sudo homelab sync`).
|
||||
|
||||
---
|
||||
|
||||
## Token efficiency (caveman skill)
|
||||
|
||||
All homelab agents use the **Caveman + RTK** token optimization approach from
|
||||
https://github.com/adityahimaone/hermes-agent-rtk-caveman.
|
||||
|
||||
### Before running any CLI command, ask:
|
||||
|
||||
1. **Is there a caveman wrapper equivalent?** Use the wrapper for token-efficient
|
||||
output. Available wrappers (installed at `~/bin/caveman_wrapper.sh`):
|
||||
- `~/bin/caveman_wrapper.sh git-status` — compact git status
|
||||
- `~/bin/caveman_wrapper.sh git-log [n]` — compact git log
|
||||
- `~/bin/caveman_wrapper.sh lint [target]` — compact lint results
|
||||
- `~/bin/caveman_wrapper.sh test-results [cmd]` — compact test results
|
||||
|
||||
2. **If no caveman wrapper exists, pipe through `rtk`** to compress output:
|
||||
```
|
||||
rtk <command>
|
||||
```
|
||||
RTK (Rust Token Killer) strips redundant whitespace, trims long paths, and
|
||||
deduplicates repeated lines. This reduces token usage by 60-90% on CLI
|
||||
operations.
|
||||
|
||||
3. **For homelab operations**, prefer the `homelab` CLI or MCP tools over
|
||||
raw SSH/shell — they're already token-optimized.
|
||||
|
||||
### Templates
|
||||
|
||||
Caveman templates live at `~/templates/`:
|
||||
- `git_status.txt` — compact git status format
|
||||
- `git_log.txt` — compact git log format
|
||||
- `lint_results.txt` — compact ESLint format
|
||||
- `test_results.txt` — compact vitest/jest format
|
||||
|
||||
### When to skip caveman/rtk
|
||||
|
||||
- Interactive commands (editors, prompts) — let human-readable output pass
|
||||
- Commands with no output — skip entirely
|
||||
- When you need the exact raw output for post-processing
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
|
||||
```
|
||||
|
||||
## Important note for Hermes agents
|
||||
|
||||
If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
|
||||
by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
|
||||
can verify the content matches or re-provision by running:
|
||||
|
||||
bash /opt/homelab-context/tools/setup-hermes-soul.sh
|
||||
@@ -44,7 +44,7 @@ one pass through **Observe → Orient → Decide → Act**:
|
||||
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
|
||||
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
|
||||
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
|
||||
| Incident | investigation narrative | `investigations/` |
|
||||
| Incident | investigation narrative | `knowledge/sources/investigations/` |
|
||||
| Plan | design doc for non-trivial work | `plans/` |
|
||||
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
|
||||
|
||||
@@ -207,7 +207,7 @@ guesswork. Roughly ordered by what unblocks the most:
|
||||
populate more as specific services turn out to need non-default risk
|
||||
classes.
|
||||
- **Incident timeline generator** — stitch ledger + signal history into
|
||||
a single narrative for `investigations/` entries instead of writing
|
||||
a single narrative for `knowledge/sources/investigations/` entries instead of writing
|
||||
them by hand.
|
||||
- **Secret access audit** — who-can-decrypt-what report from
|
||||
`.sops.yaml` + inventory `age_pubkey`s, extending what
|
||||
|
||||
@@ -20,7 +20,7 @@ fixed paths, so the wiki reorganization never moves it.
|
||||
| Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` |
|
||||
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
|
||||
| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
|
||||
| Evidence — immutable sources | `knowledge/sources/`, `investigations/` | synthesis into wiki pages |
|
||||
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
|
||||
|
||||
## Wiki pages
|
||||
|
||||
|
||||
@@ -5,6 +5,11 @@ procedures), investigations (incident evidence), and plans (design docs for non-
|
||||
follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice
|
||||
exception.
|
||||
|
||||
Where each kind lives: runbooks are skills under [`.agents/skills/`](../../skills/); operator
|
||||
reference (command cheatsheet, enrollment, Hermes agent) lives in
|
||||
[`.agents/operations/`](../../operations/); investigations are sources under
|
||||
`knowledge/sources/investigations/`; plans stay in the repo-root `plans/` folder (below).
|
||||
|
||||
## Plans always live in `plans/`
|
||||
|
||||
**Any plan or design doc for the Homelab is written into the repo `plans/` folder as
|
||||
@@ -39,12 +44,12 @@ transition: "<from> -> <to>" # only for lifecycle runbooks
|
||||
|
||||
## Investigations
|
||||
|
||||
Incident records live in `investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written
|
||||
Incident records live in `knowledge/sources/investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written
|
||||
once at incident time, then linked from the changelogs of the nodes they implicate. Sections:
|
||||
`## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved
|
||||
incidents move to `investigations/archive/`.
|
||||
incidents move to `knowledge/sources/investigations/archive/`.
|
||||
|
||||
## The operations log
|
||||
|
||||
`plans/log.md` and `investigations/log.md` are append-only records of documentation operations on
|
||||
`plans/log.md` and `knowledge/log.md` are append-only records of documentation operations on
|
||||
those areas (`## [YYYY-MM-DD] <op> | <summary>`), distinct from the Oikos change ledger.
|
||||
|
||||
368
.agents/operations/agent-enrollment.md
Normal file
368
.agents/operations/agent-enrollment.md
Normal file
@@ -0,0 +1,368 @@
|
||||
# Agent enrollment — bootstrap a client into the homelab context system
|
||||
|
||||
This walks through enrolling a new machine (workstation, LXC, or VM) so it
|
||||
joins the cross-client context system: a `/opt/homelab-context/` clone of
|
||||
this repo that auto-syncs every 5 min, a per-client age key for SOPS
|
||||
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
|
||||
|
||||
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
|
||||
> See [hermes-agent.md](hermes-agent.md). It uses the same `bootstrap.sh`
|
||||
> with an additional `--with-hermes` flag.
|
||||
|
||||
Architecture in [project_homelab_context_plan](https://… memory link); the
|
||||
operational reference is here.
|
||||
|
||||
## Prerequisites the client must satisfy
|
||||
|
||||
| Requirement | Why | How to check |
|
||||
| --- | --- | --- |
|
||||
| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
|
||||
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
|
||||
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
|
||||
| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
|
||||
| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |
|
||||
|
||||
### Hostname mismatch is the most common bootstrap failure
|
||||
|
||||
If the bootstrap exits with `no hosts/<name>.yaml in the repo`, the
|
||||
hostname doesn't match any inventory entry. Two fixes:
|
||||
|
||||
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
|
||||
(Linux) or System Preferences → Sharing (macOS), then re-run.
|
||||
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
|
||||
regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
|
||||
|
||||
### Getting onto Netbird
|
||||
|
||||
Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit `<bootstrap-tier1>`). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`.
|
||||
|
||||
**Path B — interactive OIDC (default; recommended):**
|
||||
|
||||
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
|
||||
|
||||
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
|
||||
|
||||
**Path A — setup-key (headless/scripted onboarding):**
|
||||
|
||||
Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.
|
||||
|
||||
1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`.
|
||||
2. **Setup Keys** → Create → set reusable + expiry → copy.
|
||||
3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver):
|
||||
```bash
|
||||
sudo netbird up --setup-key <KEY> \
|
||||
--management-url https://netbird.hubris.network \
|
||||
--ssh-jwt-cache-ttl 86400
|
||||
```
|
||||
|
||||
**Why we can't OIDC-login from the public internet (still open as a follow-up):**
|
||||
|
||||
`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124.
|
||||
|
||||
### DNS prerequisite
|
||||
|
||||
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
|
||||
([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
|
||||
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
|
||||
workstations don't by default. Options:
|
||||
|
||||
- **LAN client**: set DNS to 192.168.8.180 (per-interface or
|
||||
`/etc/resolv.conf`).
|
||||
- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
|
||||
point `*.hubris.network` at LXC 124.
|
||||
- **Hack-fix anywhere**: append to `/etc/hosts`:
|
||||
```
|
||||
192.168.8.175 mcp.hubris.network secrets.hubris.network
|
||||
192.168.8.175 git.hubris.network
|
||||
```
|
||||
(192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.)
|
||||
|
||||
If DNS isn't an option at all, override the URLs at bootstrap time:
|
||||
|
||||
```bash
|
||||
sudo HOMELAB_GITEA_TOKEN=... \
|
||||
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
|
||||
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
|
||||
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
|
||||
bash /tmp/bootstrap.sh --with-mcp
|
||||
```
|
||||
|
||||
## Install dependencies
|
||||
|
||||
Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script).
|
||||
|
||||
Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.
|
||||
|
||||
<details>
|
||||
<summary>Manual recipes (Fedora / Debian / macOS)</summary>
|
||||
|
||||
```bash
|
||||
# Fedora / RHEL / Nobara
|
||||
sudo dnf install -y git python3-pyyaml age curl
|
||||
SOPS_VERSION=v3.9.4
|
||||
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
|
||||
-o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
|
||||
|
||||
# Debian / Ubuntu
|
||||
sudo apt update && sudo apt install -y git python3-yaml age curl
|
||||
SOPS_VERSION=v3.9.4
|
||||
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
|
||||
-o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
|
||||
|
||||
# macOS
|
||||
brew install git age sops
|
||||
pip3 install pyyaml # if `python3 -c "import yaml"` fails
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Run the bootstrap
|
||||
|
||||
You need a Gitea read-only personal access token for the initial clone
|
||||
(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml`
|
||||
but a new client can't decrypt it before bootstrap — chicken-and-egg).
|
||||
Ask the operator (or generate in Gitea: Settings → Applications → Generate
|
||||
New Token → scope `read:repository`).
|
||||
|
||||
```bash
|
||||
TOKEN=... # your Gitea PAT, scope read:repository
|
||||
|
||||
# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
|
||||
curl -fsSL -u "dtoro:$TOKEN" \
|
||||
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
|
||||
-o /tmp/bootstrap.sh
|
||||
|
||||
# Run it.
|
||||
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp
|
||||
```
|
||||
|
||||
Flags:
|
||||
|
||||
| Flag | Effect |
|
||||
| --- | --- |
|
||||
| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user |
|
||||
| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) |
|
||||
| `--dry-run` | Prints actions without executing |
|
||||
|
||||
The bootstrap is idempotent: re-running on an enrolled client just
|
||||
verifies state, re-issues the age key only if it doesn't match the
|
||||
inventory pubkey, and refreshes the sync timer + symlinks.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
homelab whoami # prints hosts/$(hostname).yaml
|
||||
homelab list # shows the full topology
|
||||
homelab status # ping + HTTP-check across hosts/services
|
||||
homelab secret hello # decrypt the bootstrap-test secret
|
||||
systemctl list-timers homelab-context-sync.timer
|
||||
# next run within ≤5 min
|
||||
```
|
||||
|
||||
For Claude Code: start a new session — the `homelab` MCP server appears
|
||||
in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
|
||||
1 secrets-metadata).
|
||||
|
||||
## Post-bootstrap: SSH reachability
|
||||
|
||||
A new workstation must be reachable from other workstations and must be
|
||||
able to reach every host by short hostname. Run these steps after the
|
||||
bootstrap verify passes:
|
||||
|
||||
### 1. Enable SSH server
|
||||
|
||||
```bash
|
||||
# macOS:
|
||||
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
|
||||
|
||||
# Linux:
|
||||
sudo systemctl enable --now sshd
|
||||
```
|
||||
|
||||
### 2. Generate SSH key (if missing)
|
||||
|
||||
```bash
|
||||
ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100
|
||||
```
|
||||
|
||||
### 3. Publish pubkey to the repo
|
||||
|
||||
```bash
|
||||
cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub
|
||||
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push
|
||||
```
|
||||
|
||||
### 4. Deploy keys to all hosts
|
||||
|
||||
From any existing enrolled machine (hubris or another workstation):
|
||||
|
||||
```bash
|
||||
ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
|
||||
```
|
||||
|
||||
This adds the new workstation's pubkey to hubris and every running LXC.
|
||||
|
||||
### 5. Generate SSH config
|
||||
|
||||
```bash
|
||||
homelab ssh-config --install
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
ssh hubris hostname # should return "hubris" without password
|
||||
ssh gitea hostname # should return "gitea" without password
|
||||
ssh mac-mini hostname # should return "mac-mini" without password (workstation-to-workstation)
|
||||
```
|
||||
|
||||
### 6. Add LAN IP to inventory (if on LAN)
|
||||
|
||||
If the workstation has a static or reserved LAN IP, add it to
|
||||
`inventory.yaml`:
|
||||
|
||||
```yaml
|
||||
hosts:
|
||||
your-hostname:
|
||||
lan_ip: 192.168.8.xxx
|
||||
```
|
||||
|
||||
This gives it a primary LAN entry in the generated SSH config (faster
|
||||
than the Netbird fallback). Commit + push, then:
|
||||
|
||||
```bash
|
||||
cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install
|
||||
```
|
||||
|
||||
## Claude Code permissions for fleet ops
|
||||
|
||||
By default Claude Code's auto-mode classifier asks for confirmation on every
|
||||
ssh into the mesh. The bootstrap already installs the ssh ControlMaster block
|
||||
so subsequent in-session sshes multiplex, but the *first* ssh of each session
|
||||
still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns
|
||||
by adding to `~/.claude/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"permissions": {
|
||||
"defaultMode": "auto",
|
||||
"allow": [
|
||||
"Bash(ssh -p 22022 *)",
|
||||
"Bash(homelab *)"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The first rule covers any ssh to a mesh peer on the homelab netbird port; the
|
||||
second covers all `homelab` CLI invocations. Both are scoped tight enough that
|
||||
the classifier doesn't gate them but loose enough to handle the variety of
|
||||
arguments.
|
||||
|
||||
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
|
||||
visible to the classifier (it's not actually durable in 0.71.2, but the
|
||||
ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md)
|
||||
for context), drop a free-text rule into `autoMode.allow` describing the
|
||||
authorization. Optional.
|
||||
|
||||
## Adding a new client to inventory
|
||||
|
||||
If the hostname you want isn't yet in inventory, enrollment is a two-step
|
||||
ceremony driven from an existing enrolled client (e.g. hubris). The
|
||||
`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3.
|
||||
|
||||
```bash
|
||||
# 1. On hubris (or any existing client): add the inventory entry.
|
||||
homelab client add my-new-machine
|
||||
# Prompts for kind, os, netbird FQDN, role. Commits + pushes.
|
||||
|
||||
# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key).
|
||||
|
||||
# 3. On the new machine: install deps + run bootstrap (above).
|
||||
# Bootstrap calls /issue, receives a fresh age keypair, and prints the
|
||||
# public key for the operator to commit back to inventory.
|
||||
|
||||
# 4. On hubris: finalize the age public key.
|
||||
homelab client add my-new-machine --finalize-pubkey age1...
|
||||
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
|
||||
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
|
||||
```
|
||||
|
||||
## Granting a secret to a new client
|
||||
|
||||
Adding a client doesn't grant them every secret. Recipients are explicit
|
||||
per file via `.sops.yaml` glob rules. To grant a client access to (say)
|
||||
`secrets/hello.yaml`:
|
||||
|
||||
1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to
|
||||
the matching `creation_rules` block.
|
||||
2. Re-key the existing ciphertext for the new recipient list:
|
||||
```bash
|
||||
sops updatekeys -y secrets/hello.yaml
|
||||
```
|
||||
3. Commit + push. On the next sync (≤5 min), the client can decrypt.
|
||||
|
||||
## Removing a client
|
||||
|
||||
```bash
|
||||
# From any existing client:
|
||||
homelab client remove my-old-machine
|
||||
```
|
||||
|
||||
This:
|
||||
1. Removes the inventory entry and `hosts/my-old-machine.yaml`.
|
||||
2. Runs `sops updatekeys -y` against every file in `secrets/` (operator
|
||||
must first remove the pubkey from `.sops.yaml` rules).
|
||||
3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to
|
||||
shred the key file and add the hostname to the denylist.
|
||||
4. Commits + pushes.
|
||||
|
||||
The CLI prints a follow-up checklist that the operator must do manually:
|
||||
|
||||
- Revoke the peer in the Netbird console (denies future mesh access).
|
||||
- **Rotate any credentials whose ciphertext the removed client already
|
||||
has on disk.** The age key revocation only protects *future*
|
||||
ciphertext; what's already been pulled is still decryptable until the
|
||||
underlying credential changes.
|
||||
- Optional: `homelab nuke my-old-machine` SSHes in, shreds
|
||||
`/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
|
||||
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
|
||||
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
|
||||
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
|
||||
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
|
||||
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
|
||||
| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
|
||||
| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
|
||||
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
|
||||
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
|
||||
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
|
||||
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). |
|
||||
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
|
||||
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
|
||||
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-02 — SSH reachability post-bootstrap steps
|
||||
Added a new "Post-bootstrap: SSH reachability" section covering SSH key
|
||||
generation, pubkey publication, deployment to hosts, SSH config generation,
|
||||
and LAN IP registration. New workstations enrolled via this doc will
|
||||
automatically join the universal SSH mesh.
|
||||
|
||||
### 2026-05-31 — cross-link to hermes-agent.md
|
||||
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
|
||||
|
||||
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
|
||||
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
|
||||
|
||||
### 2026-05-20 — initial page
|
||||
Captures the enrollment flow validated during Phase 2 of the homelab
|
||||
context distribution rollout. hubris + LXC 105 (apps) enrolled; first
|
||||
workstation (republic-laptop) blocked on hostname mismatch, documented
|
||||
the resolution.
|
||||
91
.agents/operations/commands.md
Normal file
91
.agents/operations/commands.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Operations cheatsheet
|
||||
|
||||
Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
||||
|
||||
## Proxmox CLI
|
||||
|
||||
| Command | Use |
|
||||
| --- | --- |
|
||||
| `pct list` / `qm list` | List LXC containers / VMs |
|
||||
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
||||
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) |
|
||||
| `pct enter <id>` | Shell into a container |
|
||||
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
||||
| `pvesm status` | Storage pools status |
|
||||
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
||||
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
||||
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
||||
| `pveversion` | PVE version |
|
||||
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
||||
|
||||
## Storage
|
||||
|
||||
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
||||
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
||||
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
||||
|
||||
## Reverse proxy
|
||||
|
||||
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md).
|
||||
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
||||
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
||||
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
||||
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
|
||||
|
||||
## DNS
|
||||
|
||||
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
||||
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
|
||||
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
|
||||
- See [DNS](../../knowledge/wiki/infrastructure/dns.md).
|
||||
|
||||
## Web access
|
||||
|
||||
- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
|
||||
|
||||
## Telemetry quick checks
|
||||
|
||||
- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
|
||||
- `ras-mc-ctl --errors` — full event log
|
||||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
||||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
||||
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md))
|
||||
|
||||
## Fleet apt operations
|
||||
|
||||
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
|
||||
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
|
||||
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
|
||||
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
|
||||
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
|
||||
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
|
||||
|
||||
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
|
||||
|
||||
## Oikos (agent OS layer)
|
||||
|
||||
See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `homelab service <name> explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history |
|
||||
| `homelab node <name> relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius |
|
||||
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
|
||||
| `homelab decide <action> <entity>` | Decision classifier: risk × blast radius × confidence → auto-act or escalate |
|
||||
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
|
||||
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
|
||||
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
|
||||
|
||||
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md).
|
||||
|
||||
## Related
|
||||
- [Hubris host](../../knowledge/wiki/hosts/hubris.md)
|
||||
- [Containers index](../../knowledge/wiki/containers/index.md)
|
||||
- [DNS](../../knowledge/wiki/infrastructure/dns.md)
|
||||
- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md)
|
||||
- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)
|
||||
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction
|
||||
210
.agents/operations/hermes-agent.md
Normal file
210
.agents/operations/hermes-agent.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# Hermes agent — Nous-Hermes-powered Goose sessions on a homelab client
|
||||
|
||||
Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
|
||||
Llama variant) as a working terminal agent on a homelab client. Builds on top
|
||||
of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md))
|
||||
— this page covers only the Hermes-specific additions.
|
||||
|
||||
The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
|
||||
|
||||
- The chat loop, multi-turn history, and streaming
|
||||
- The OpenRouter provider that routes to Nous Hermes
|
||||
- The built-in `developer` extension (shell + file editor — same surface Claude
|
||||
Code has)
|
||||
- A remote MCP extension pointed at `mcp.hubris.network` for read-only
|
||||
homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
|
||||
|
||||
The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
|
||||
`.goosehints` so it's injected into the system prompt on every session.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Requirement | How |
|
||||
| --- | --- |
|
||||
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) |
|
||||
| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
|
||||
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` |
|
||||
|
||||
## Onboarding flow
|
||||
|
||||
```bash
|
||||
# 1. On hubris (or any enrolled client): reserve the inventory entry.
|
||||
homelab client add new-machine
|
||||
|
||||
# 2. Join new-machine to Netbird (setup-key or OIDC).
|
||||
|
||||
# 3. On new-machine: bootstrap with --with-hermes.
|
||||
TOKEN=... # gitea PAT, read:repository
|
||||
curl -fsSL -u "dtoro:$TOKEN" \
|
||||
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
|
||||
-o /tmp/bootstrap.sh
|
||||
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-hermes
|
||||
|
||||
# 4. Back on hubris: finalize the age pubkey AND grant the Hermes secret.
|
||||
homelab client add new-machine \
|
||||
--finalize-pubkey age1... \
|
||||
--with-hermes
|
||||
|
||||
# 5. Wait ≤5 min for sync, then on new-machine:
|
||||
hermes "what LXCs are running?"
|
||||
```
|
||||
|
||||
The bootstrap `--with-hermes` flag does five things, all idempotent:
|
||||
|
||||
1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
|
||||
(upstream installer) and symlinks `/usr/local/bin/goose` to it.
|
||||
2. Symlinks `/opt/homelab-context/bin/hermes` → `/usr/local/bin/hermes`.
|
||||
3. Symlinks `/opt/homelab-context/HERMES.md` → `/root/HERMES.md` (Linux) or
|
||||
`/etc/HERMES.md` (macOS) for `cat`-as-operator convenience.
|
||||
4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
|
||||
extensions (preserves any keys the operator added by hand).
|
||||
5. Symlinks `~/.config/goose/.goosehints` → HERMES.md, so the persona is
|
||||
injected as the system prompt on every session.
|
||||
|
||||
## Seeding the OpenRouter key
|
||||
|
||||
The first time anyone enrolls with `--with-hermes`, the encrypted file
|
||||
`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
|
||||
existing recipient):
|
||||
|
||||
```bash
|
||||
sops secrets/openrouter-api-key.yaml
|
||||
# editor opens; replace api_key value with the real sk-or-... key, save, close.
|
||||
git -C /opt/homelab-context add secrets/openrouter-api-key.yaml
|
||||
git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key'
|
||||
git -C /opt/homelab-context push
|
||||
```
|
||||
|
||||
Until this step happens, `hermes …` exits with `openrouter-api-key.yaml still
|
||||
contains the placeholder`. Subsequent enrollees get the real key automatically
|
||||
via `--with-hermes` (which adds them as a sops recipient on
|
||||
`secrets/openrouter-api-key.yaml`).
|
||||
|
||||
## Granting the OpenRouter key to an already-enrolled host
|
||||
|
||||
If a host was enrolled without `--with-hermes` and you want to add it later:
|
||||
|
||||
```bash
|
||||
# On hubris:
|
||||
PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
|
||||
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-hermes
|
||||
```
|
||||
|
||||
`--finalize-pubkey` is required by the existing flow even when the pubkey is
|
||||
unchanged — it's also the trigger that runs the sops grant.
|
||||
|
||||
After ≤5 min sync the host can decrypt the key. Bootstrap doesn't need to
|
||||
re-run; only the secret recipient list changed.
|
||||
|
||||
## Verifying
|
||||
|
||||
```bash
|
||||
homelab whoami # standard enrollment OK
|
||||
homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
|
||||
which goose && which hermes # binaries present
|
||||
goose info -v # provider/model wiring sane
|
||||
hermes "what LXCs are running?" # interactive Goose session
|
||||
|
||||
# Non-interactive smoke test:
|
||||
echo "List the homelab MCP tools you have available" | hermes
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
The bootstrap-managed keys in `~/.config/goose/config.yaml`:
|
||||
|
||||
```yaml
|
||||
GOOSE_PROVIDER: openrouter
|
||||
GOOSE_MODEL: deepseek/deepseek-v4-flash
|
||||
GOOSE_MODE: smart_approve # asks before destructive tool calls
|
||||
extensions:
|
||||
developer:
|
||||
type: builtin
|
||||
bundled: true
|
||||
enabled: true
|
||||
name: developer
|
||||
timeout: 300
|
||||
homelab:
|
||||
type: streamable_http
|
||||
enabled: true
|
||||
name: homelab
|
||||
uri: https://mcp.hubris.network/mcp
|
||||
timeout: 60
|
||||
```
|
||||
|
||||
Override via env on a single bootstrap run:
|
||||
|
||||
```bash
|
||||
HOMELAB_HERMES_MODEL=nousresearch/hermes-3-llama-3.1-405b \
|
||||
HOMELAB_HERMES_MCP_URI=https://mcp.hubris.network/mcp \
|
||||
sudo bash /tmp/bootstrap.sh --with-hermes
|
||||
```
|
||||
|
||||
Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
|
||||
preserved across re-bootstraps — the merge only overwrites the keys it manages.
|
||||
|
||||
## Tool permissions
|
||||
|
||||
`GOOSE_MODE: smart_approve` is the bootstrap default: Goose runs read-only
|
||||
shell commands without prompting and asks for confirmation before destructive
|
||||
ones. To make the agent fully unattended (e.g. for scheduled jobs), set
|
||||
`GOOSE_MODE: auto` in `~/.config/goose/config.yaml`. To require confirmation on
|
||||
every tool call, use `approve`. See
|
||||
[goose-permissions](https://goose-docs.ai/docs/guides/managing-tools/goose-permissions/).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
|
||||
| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
|
||||
| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
|
||||
| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py` — `mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
|
||||
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
|
||||
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
|
||||
|
||||
## Cross-references
|
||||
|
||||
- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the
|
||||
Hermes flow assumes is done.
|
||||
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
|
||||
session start (via `~/.config/goose/.goosehints`).
|
||||
- [`bin/hermes`](../../bin/hermes) — the wrapper that decrypts the OpenRouter key
|
||||
and execs `goose session`.
|
||||
- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-hermes` flag's install block.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
|
||||
(`"SSE transport is no longer supported - kept only for config file
|
||||
compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
|
||||
server at `mcp/server.py:336` still calls `mcp.run(transport="sse")`. Until
|
||||
that's changed, the `homelab` MCP extension in Goose will fail to connect.
|
||||
The developer extension (shell + edit) covers most ops without it; this is
|
||||
a polish item, not a blocker.
|
||||
2. **Per-host OpenRouter keys** for billing attribution. Today all Hermes
|
||||
hosts share one key.
|
||||
3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
|
||||
directly — OpenRouter periodically rotates the underlying weights.
|
||||
4. **Local-inference fallback** (ollama / vllm) once the homelab has a GPU
|
||||
node. The wrapper, persona, and MCP wiring stay unchanged; only
|
||||
`GOOSE_PROVIDER`/`GOOSE_MODEL` change.
|
||||
|
||||
7. **Caveman auto-setup via post-pull hook.** The sync timer now calls
|
||||
`tools/post-pull.sh`, which runs any `tools/*.setup.sh` after git pull.
|
||||
Currently this auto-installs the Caveman npm package, wrapper scripts, and
|
||||
compact output templates on all agent hosts (*token efficiency*).
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-01 — caveman + post-pull auto-setup
|
||||
Added `tools/post-pull.sh` sync hook that auto-runs `tools/*.setup.sh`
|
||||
after every git pull. First user: `tools/setup-caveman.sh` installed Caveman
|
||||
templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token-
|
||||
efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
|
||||
Also created `tools/caveman/` with the wrapper script, JS renderer, and
|
||||
templates — the canonical source for all agent hosts.
|
||||
Captures the Hermes-on-Goose onboarding flow added in the same commit as
|
||||
`bootstrap.sh --with-hermes`, `bin/hermes`, the sops rule for
|
||||
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-hermes`
|
||||
extension. MCP streamable_http migration is queued as follow-up #1.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
How the narrative documentation in this repo is organized. The pattern is borrowed from the
|
||||
`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top
|
||||
of immutable evidence (`knowledge/sources/`, `investigations/`), with pure-listing indexes and an
|
||||
of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an
|
||||
append-only operations log.
|
||||
|
||||
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
|
||||
@@ -11,7 +11,7 @@ moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for t
|
||||
|
||||
## Layers
|
||||
|
||||
- **Sources** are immutable raw material: incident records (`investigations/`), external reference
|
||||
- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference
|
||||
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
|
||||
Read them; do not rewrite them into other sources.
|
||||
- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per
|
||||
|
||||
@@ -96,7 +96,7 @@ What it looks like after.
|
||||
Changelog entries to write, index status to update.
|
||||
```
|
||||
|
||||
### Investigation (`investigations/YYYY-MM-DD-slug.md`)
|
||||
### Investigation (`knowledge/sources/investigations/YYYY-MM-DD-slug.md`)
|
||||
|
||||
```markdown
|
||||
# YYYY-MM-DD — <title>
|
||||
|
||||
@@ -11,7 +11,7 @@ docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
|
||||
Goal: bring a new host (workstation, LXC, VM) into inventory and the
|
||||
secrets model, with mesh membership only where it's actually needed.
|
||||
This wraps the existing `homelab client add` flow — see
|
||||
[operations/agent-enrollment.md](../../../operations/agent-enrollment.md) for
|
||||
[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for
|
||||
the full walkthrough; this runbook is the risk/lifecycle framing.
|
||||
|
||||
1. On any enrolled client: `homelab client add <hostname>` — appends a
|
||||
|
||||
@@ -18,7 +18,7 @@ own repo) and get it live, safely.
|
||||
`oikos/policy.yaml`.
|
||||
2. Clone/pull the `config_repo` (never edit the backend's working tree
|
||||
directly — tracked configs change by commit + push, per
|
||||
[OIKOS.md](../../../OIKOS.md) conventions).
|
||||
[OIKOS.md](../../OIKOS.md) conventions).
|
||||
3. Make the change, commit, push to `main`.
|
||||
4. The Gitea webhook fires the deploy pipeline for that repo (see
|
||||
[infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for
|
||||
|
||||
@@ -23,7 +23,7 @@ Goal: understand what broke and why, before touching anything.
|
||||
5. Check the change ledger for recent mutations to the affected entity
|
||||
or anything upstream of it: `homelab service <name> history` (once
|
||||
populated) or grep `ledger/*.jsonl`.
|
||||
6. Write findings to a new `investigations/<date>-<slug>.md` — symptom,
|
||||
6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom,
|
||||
timeline, root cause, fix applied, prevention. This is the durable
|
||||
record; don't rely on chat history.
|
||||
|
||||
|
||||
@@ -102,7 +102,7 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
|
||||
|
||||
## Related
|
||||
|
||||
- [Operations cheatsheet](../../../operations/commands.md)
|
||||
- [Operations cheatsheet](../../operations/commands.md)
|
||||
- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md)
|
||||
- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user