docs: streamline & consolidate the tree (phase 6)

Problem: after the wiki-hq reorg, agent-instruction and human-doc domains
were still scattered across the repo root, with three now-redundant stub
files cluttering it. The organizing principle wasn't visible in the layout.

Change — enforce three clear buckets:
- .agents/  = how agents operate: OIKOS.md, HERMES.md (moved from root),
  shared/ conventions, domains/ schemas, skills/, and operations/ (operator
  cheatsheet + enrollment + hermes-agent, moved from root).
- knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now
  including investigations/ (incident records are evidence/sources).
- root = substrate + two entry points (AGENTS.md, README.md), plus plans/
  as its own design-intent domain.

Moves:
- investigations/ -> knowledge/sources/investigations/ (incl. archive/, index).
- operations/ -> .agents/operations/.
- HERMES.md -> .agents/HERMES.md.
- Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md
  (its 7 remaining linkers repointed to .agents/OIKOS.md).

Consumers updated:
- inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards.
- tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md.
- bin/homelab help string -> .agents/operations/hermes-agent.md.
- knowledge/operations schemas, llm-wiki, page-templates, incident-investigation
  skill, AGENTS.md/README nav -> new investigations/operations paths.
- All markdown links rewritten via the path-resolving mapper.

Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/,
plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml.

Verification: docs-lint at baseline (2 intentional cross-repo refs, no new
breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all
doc_page targets resolve; Hermes provisioning scripts point at the new path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-06 18:12:14 +02:00
parent 4efddb8bed
commit b5c1247093
48 changed files with 118 additions and 125 deletions

96
.agents/HERMES.md Normal file
View File

@@ -0,0 +1,96 @@
# HERMES.md — Agent persona for homelab clients
This file is the canonical agent persona for **all** AI agents running on
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
conventions, and the source-of-truth hierarchy.
## Source of truth
The homelab-context repo at `/opt/homelab-context/` is the single source of
truth for:
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
- Service endpoints and credentials (via `homelab secret`)
- Agent behaviour and conventions
- Everything in this file
When in doubt, check `/opt/homelab-context/` first.
## Runbooks — load, don't rediscover
For the canonical workflows (service health check, config change +
deploy, client enrollment, incident investigation, and each node
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
acting. Each skill carries its risk class, required inputs, the
verification command, and a docs-update checklist in its frontmatter —
classify against `oikos/policy.yaml` using that risk class before any
mutation. Don't re-derive topology or the mutation path by grepping the
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
operating model these runbooks execute inside (OODA loop, risk classes,
approval flow, ontology).
## Agent type — how this file gets loaded
| Agent | Loading mechanism |
|-------|------------------|
| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints``/opt/homelab-context/HERMES.md` |
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
homelab-context repo instead. Changes propagate to all clients on the next
sync (`sudo homelab sync`).
---
## Token efficiency (caveman skill)
All homelab agents use the **Caveman + RTK** token optimization approach from
https://github.com/adityahimaone/hermes-agent-rtk-caveman.
### Before running any CLI command, ask:
1. **Is there a caveman wrapper equivalent?** Use the wrapper for token-efficient
output. Available wrappers (installed at `~/bin/caveman_wrapper.sh`):
- `~/bin/caveman_wrapper.sh git-status` — compact git status
- `~/bin/caveman_wrapper.sh git-log [n]` — compact git log
- `~/bin/caveman_wrapper.sh lint [target]` — compact lint results
- `~/bin/caveman_wrapper.sh test-results [cmd]` — compact test results
2. **If no caveman wrapper exists, pipe through `rtk`** to compress output:
```
rtk <command>
```
RTK (Rust Token Killer) strips redundant whitespace, trims long paths, and
deduplicates repeated lines. This reduces token usage by 60-90% on CLI
operations.
3. **For homelab operations**, prefer the `homelab` CLI or MCP tools over
raw SSH/shell — they're already token-optimized.
### Templates
Caveman templates live at `~/templates/`:
- `git_status.txt` — compact git status format
- `git_log.txt` — compact git log format
- `lint_results.txt` — compact ESLint format
- `test_results.txt` — compact vitest/jest format
### When to skip caveman/rtk
- Interactive commands (editors, prompts) — let human-readable output pass
- Commands with no output — skip entirely
- When you need the exact raw output for post-processing
### Verification
```bash
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
```
## Important note for Hermes agents
If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
can verify the content matches or re-provision by running:
bash /opt/homelab-context/tools/setup-hermes-soul.sh

View File

@@ -44,7 +44,7 @@ one pass through **Observe → Orient → Decide → Act**:
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
| Incident | investigation narrative | `investigations/` |
| Incident | investigation narrative | `knowledge/sources/investigations/` |
| Plan | design doc for non-trivial work | `plans/` |
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
@@ -207,7 +207,7 @@ guesswork. Roughly ordered by what unblocks the most:
populate more as specific services turn out to need non-default risk
classes.
- **Incident timeline generator** — stitch ledger + signal history into
a single narrative for `investigations/` entries instead of writing
a single narrative for `knowledge/sources/investigations/` entries instead of writing
them by hand.
- **Secret access audit** — who-can-decrypt-what report from
`.sops.yaml` + inventory `age_pubkey`s, extending what

View File

@@ -20,7 +20,7 @@ fixed paths, so the wiki reorganization never moves it.
| Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` |
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
| Evidence — immutable sources | `knowledge/sources/`, `investigations/` | synthesis into wiki pages |
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
## Wiki pages

View File

@@ -5,6 +5,11 @@ procedures), investigations (incident evidence), and plans (design docs for non-
follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice
exception.
Where each kind lives: runbooks are skills under [`.agents/skills/`](../../skills/); operator
reference (command cheatsheet, enrollment, Hermes agent) lives in
[`.agents/operations/`](../../operations/); investigations are sources under
`knowledge/sources/investigations/`; plans stay in the repo-root `plans/` folder (below).
## Plans always live in `plans/`
**Any plan or design doc for the Homelab is written into the repo `plans/` folder as
@@ -39,12 +44,12 @@ transition: "<from> -> <to>" # only for lifecycle runbooks
## Investigations
Incident records live in `investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written
Incident records live in `knowledge/sources/investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written
once at incident time, then linked from the changelogs of the nodes they implicate. Sections:
`## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved
incidents move to `investigations/archive/`.
incidents move to `knowledge/sources/investigations/archive/`.
## The operations log
`plans/log.md` and `investigations/log.md` are append-only records of documentation operations on
`plans/log.md` and `knowledge/log.md` are append-only records of documentation operations on
those areas (`## [YYYY-MM-DD] <op> | <summary>`), distinct from the Oikos change ledger.

View File

@@ -0,0 +1,368 @@
# Agent enrollment — bootstrap a client into the homelab context system
This walks through enrolling a new machine (workstation, LXC, or VM) so it
joins the cross-client context system: a `/opt/homelab-context/` clone of
this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
> See [hermes-agent.md](hermes-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-hermes` flag.
Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.
## Prerequisites the client must satisfy
| Requirement | Why | How to check |
| --- | --- | --- |
| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |
### Hostname mismatch is the most common bootstrap failure
If the bootstrap exits with `no hosts/<name>.yaml in the repo`, the
hostname doesn't match any inventory entry. Two fixes:
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
(Linux) or System Preferences → Sharing (macOS), then re-run.
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
### Getting onto Netbird
Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit `<bootstrap-tier1>`). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`.
**Path B — interactive OIDC (default; recommended):**
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
**Path A — setup-key (headless/scripted onboarding):**
Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.
1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`.
2. **Setup Keys** → Create → set reusable + expiry → copy.
3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver):
```bash
sudo netbird up --setup-key <KEY> \
--management-url https://netbird.hubris.network \
--ssh-jwt-cache-ttl 86400
```
**Why we can't OIDC-login from the public internet (still open as a follow-up):**
`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124.
### DNS prerequisite
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:
- **LAN client**: set DNS to 192.168.8.180 (per-interface or
`/etc/resolv.conf`).
- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
point `*.hubris.network` at LXC 124.
- **Hack-fix anywhere**: append to `/etc/hosts`:
```
192.168.8.175 mcp.hubris.network secrets.hubris.network
192.168.8.175 git.hubris.network
```
(192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.)
If DNS isn't an option at all, override the URLs at bootstrap time:
```bash
sudo HOMELAB_GITEA_TOKEN=... \
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
bash /tmp/bootstrap.sh --with-mcp
```
## Install dependencies
Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script).
Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.
<details>
<summary>Manual recipes (Fedora / Debian / macOS)</summary>
```bash
# Fedora / RHEL / Nobara
sudo dnf install -y git python3-pyyaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
-o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
# Debian / Ubuntu
sudo apt update && sudo apt install -y git python3-yaml age curl
SOPS_VERSION=v3.9.4
sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
-o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
# macOS
brew install git age sops
pip3 install pyyaml # if `python3 -c "import yaml"` fails
```
</details>
## Run the bootstrap
You need a Gitea read-only personal access token for the initial clone
(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml`
but a new client can't decrypt it before bootstrap — chicken-and-egg).
Ask the operator (or generate in Gitea: Settings → Applications → Generate
New Token → scope `read:repository`).
```bash
TOKEN=... # your Gitea PAT, scope read:repository
# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
curl -fsSL -u "dtoro:$TOKEN" \
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
-o /tmp/bootstrap.sh
# Run it.
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp
```
Flags:
| Flag | Effect |
| --- | --- |
| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user |
| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) |
| `--dry-run` | Prints actions without executing |
The bootstrap is idempotent: re-running on an enrolled client just
verifies state, re-issues the age key only if it doesn't match the
inventory pubkey, and refreshes the sync timer + symlinks.
## Verify
```bash
homelab whoami # prints hosts/$(hostname).yaml
homelab list # shows the full topology
homelab status # ping + HTTP-check across hosts/services
homelab secret hello # decrypt the bootstrap-test secret
systemctl list-timers homelab-context-sync.timer
# next run within ≤5 min
```
For Claude Code: start a new session — the `homelab` MCP server appears
in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
1 secrets-metadata).
## Post-bootstrap: SSH reachability
A new workstation must be reachable from other workstations and must be
able to reach every host by short hostname. Run these steps after the
bootstrap verify passes:
### 1. Enable SSH server
```bash
# macOS:
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
# Linux:
sudo systemctl enable --now sshd
```
### 2. Generate SSH key (if missing)
```bash
ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100
```
### 3. Publish pubkey to the repo
```bash
cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push
```
### 4. Deploy keys to all hosts
From any existing enrolled machine (hubris or another workstation):
```bash
ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
```
This adds the new workstation's pubkey to hubris and every running LXC.
### 5. Generate SSH config
```bash
homelab ssh-config --install
```
Verify:
```bash
ssh hubris hostname # should return "hubris" without password
ssh gitea hostname # should return "gitea" without password
ssh mac-mini hostname # should return "mac-mini" without password (workstation-to-workstation)
```
### 6. Add LAN IP to inventory (if on LAN)
If the workstation has a static or reserved LAN IP, add it to
`inventory.yaml`:
```yaml
hosts:
your-hostname:
lan_ip: 192.168.8.xxx
```
This gives it a primary LAN entry in the generated SSH config (faster
than the Netbird fallback). Commit + push, then:
```bash
cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install
```
## Claude Code permissions for fleet ops
By default Claude Code's auto-mode classifier asks for confirmation on every
ssh into the mesh. The bootstrap already installs the ssh ControlMaster block
so subsequent in-session sshes multiplex, but the *first* ssh of each session
still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns
by adding to `~/.claude/settings.json`:
```json
{
"permissions": {
"defaultMode": "auto",
"allow": [
"Bash(ssh -p 22022 *)",
"Bash(homelab *)"
]
}
}
```
The first rule covers any ssh to a mesh peer on the homelab netbird port; the
second covers all `homelab` CLI invocations. Both are scoped tight enough that
the classifier doesn't gate them but loose enough to handle the variety of
arguments.
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
visible to the classifier (it's not actually durable in 0.71.2, but the
ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md)
for context), drop a free-text rule into `autoMode.allow` describing the
authorization. Optional.
## Adding a new client to inventory
If the hostname you want isn't yet in inventory, enrollment is a two-step
ceremony driven from an existing enrolled client (e.g. hubris). The
`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3.
```bash
# 1. On hubris (or any existing client): add the inventory entry.
homelab client add my-new-machine
# Prompts for kind, os, netbird FQDN, role. Commits + pushes.
# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key).
# 3. On the new machine: install deps + run bootstrap (above).
# Bootstrap calls /issue, receives a fresh age keypair, and prints the
# public key for the operator to commit back to inventory.
# 4. On hubris: finalize the age public key.
homelab client add my-new-machine --finalize-pubkey age1...
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
```
## Granting a secret to a new client
Adding a client doesn't grant them every secret. Recipients are explicit
per file via `.sops.yaml` glob rules. To grant a client access to (say)
`secrets/hello.yaml`:
1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to
the matching `creation_rules` block.
2. Re-key the existing ciphertext for the new recipient list:
```bash
sops updatekeys -y secrets/hello.yaml
```
3. Commit + push. On the next sync (≤5 min), the client can decrypt.
## Removing a client
```bash
# From any existing client:
homelab client remove my-old-machine
```
This:
1. Removes the inventory entry and `hosts/my-old-machine.yaml`.
2. Runs `sops updatekeys -y` against every file in `secrets/` (operator
must first remove the pubkey from `.sops.yaml` rules).
3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to
shred the key file and add the hostname to the denylist.
4. Commits + pushes.
The CLI prints a follow-up checklist that the operator must do manually:
- Revoke the peer in the Netbird console (denies future mesh access).
- **Rotate any credentials whose ciphertext the removed client already
has on disk.** The age key revocation only protects *future*
ciphertext; what's already been pulled is still decryptable until the
underlying credential changes.
- Optional: `homelab nuke my-old-machine` SSHes in, shreds
`/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync.
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). |
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
## Changelog
### 2026-06-02 — SSH reachability post-bootstrap steps
Added a new "Post-bootstrap: SSH reachability" section covering SSH key
generation, pubkey publication, deployment to hosts, SSH config generation,
and LAN IP registration. New workstations enrolled via this doc will
automatically join the universal SSH mesh.
### 2026-05-31 — cross-link to hermes-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
### 2026-05-20 — initial page
Captures the enrollment flow validated during Phase 2 of the homelab
context distribution rollout. hubris + LXC 105 (apps) enrolled; first
workstation (republic-laptop) blocked on hostname mismatch, documented
the resolution.

View File

@@ -0,0 +1,91 @@
# Operations cheatsheet
Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
## Proxmox CLI
| Command | Use |
| --- | --- |
| `pct list` / `qm list` | List LXC containers / VMs |
| `pct config <id>` / `qm config <id>` | Container / VM config |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) |
| `pct enter <id>` | Shell into a container |
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
| `pvesm status` | Storage pools status |
| `pvesh get /nodes --output-format json` | Node summary as JSON |
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pveversion` | PVE version |
| `journalctl -u pve-cluster -n 100` | PVE service logs |
## Storage
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
## Reverse proxy
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
## DNS
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
- See [DNS](../../knowledge/wiki/infrastructure/dns.md).
## Web access
- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
## Telemetry quick checks
- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
- `ras-mc-ctl --errors` — full event log
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md))
## Fleet apt operations
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
| Command | What it does |
| --- | --- |
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot``vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
## Oikos (agent OS layer)
See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
| Command | What it does |
| --- | --- |
| `homelab service <name> explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history |
| `homelab node <name> relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius |
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
| `homelab decide <action> <entity>` | Decision classifier: risk × blast radius × confidence → auto-act or escalate |
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md).
## Related
- [Hubris host](../../knowledge/wiki/hosts/hubris.md)
- [Containers index](../../knowledge/wiki/containers/index.md)
- [DNS](../../knowledge/wiki/infrastructure/dns.md)
- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md)
- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction

View File

@@ -0,0 +1,210 @@
# Hermes agent — Nous-Hermes-powered Goose sessions on a homelab client
Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
Llama variant) as a working terminal agent on a homelab client. Builds on top
of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md))
— this page covers only the Hermes-specific additions.
The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
- The chat loop, multi-turn history, and streaming
- The OpenRouter provider that routes to Nous Hermes
- The built-in `developer` extension (shell + file editor — same surface Claude
Code has)
- A remote MCP extension pointed at `mcp.hubris.network` for read-only
homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
`.goosehints` so it's injected into the system prompt on every session.
## Prerequisites
| Requirement | How |
| --- | --- |
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) |
| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` |
## Onboarding flow
```bash
# 1. On hubris (or any enrolled client): reserve the inventory entry.
homelab client add new-machine
# 2. Join new-machine to Netbird (setup-key or OIDC).
# 3. On new-machine: bootstrap with --with-hermes.
TOKEN=... # gitea PAT, read:repository
curl -fsSL -u "dtoro:$TOKEN" \
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
-o /tmp/bootstrap.sh
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-hermes
# 4. Back on hubris: finalize the age pubkey AND grant the Hermes secret.
homelab client add new-machine \
--finalize-pubkey age1... \
--with-hermes
# 5. Wait ≤5 min for sync, then on new-machine:
hermes "what LXCs are running?"
```
The bootstrap `--with-hermes` flag does five things, all idempotent:
1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
(upstream installer) and symlinks `/usr/local/bin/goose` to it.
2. Symlinks `/opt/homelab-context/bin/hermes``/usr/local/bin/hermes`.
3. Symlinks `/opt/homelab-context/HERMES.md``/root/HERMES.md` (Linux) or
`/etc/HERMES.md` (macOS) for `cat`-as-operator convenience.
4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
extensions (preserves any keys the operator added by hand).
5. Symlinks `~/.config/goose/.goosehints` → HERMES.md, so the persona is
injected as the system prompt on every session.
## Seeding the OpenRouter key
The first time anyone enrolls with `--with-hermes`, the encrypted file
`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
existing recipient):
```bash
sops secrets/openrouter-api-key.yaml
# editor opens; replace api_key value with the real sk-or-... key, save, close.
git -C /opt/homelab-context add secrets/openrouter-api-key.yaml
git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key'
git -C /opt/homelab-context push
```
Until this step happens, `hermes …` exits with `openrouter-api-key.yaml still
contains the placeholder`. Subsequent enrollees get the real key automatically
via `--with-hermes` (which adds them as a sops recipient on
`secrets/openrouter-api-key.yaml`).
## Granting the OpenRouter key to an already-enrolled host
If a host was enrolled without `--with-hermes` and you want to add it later:
```bash
# On hubris:
PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-hermes
```
`--finalize-pubkey` is required by the existing flow even when the pubkey is
unchanged — it's also the trigger that runs the sops grant.
After ≤5 min sync the host can decrypt the key. Bootstrap doesn't need to
re-run; only the secret recipient list changed.
## Verifying
```bash
homelab whoami # standard enrollment OK
homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
which goose && which hermes # binaries present
goose info -v # provider/model wiring sane
hermes "what LXCs are running?" # interactive Goose session
# Non-interactive smoke test:
echo "List the homelab MCP tools you have available" | hermes
```
## Configuration
The bootstrap-managed keys in `~/.config/goose/config.yaml`:
```yaml
GOOSE_PROVIDER: openrouter
GOOSE_MODEL: deepseek/deepseek-v4-flash
GOOSE_MODE: smart_approve # asks before destructive tool calls
extensions:
developer:
type: builtin
bundled: true
enabled: true
name: developer
timeout: 300
homelab:
type: streamable_http
enabled: true
name: homelab
uri: https://mcp.hubris.network/mcp
timeout: 60
```
Override via env on a single bootstrap run:
```bash
HOMELAB_HERMES_MODEL=nousresearch/hermes-3-llama-3.1-405b \
HOMELAB_HERMES_MCP_URI=https://mcp.hubris.network/mcp \
sudo bash /tmp/bootstrap.sh --with-hermes
```
Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
preserved across re-bootstraps — the merge only overwrites the keys it manages.
## Tool permissions
`GOOSE_MODE: smart_approve` is the bootstrap default: Goose runs read-only
shell commands without prompting and asks for confirmation before destructive
ones. To make the agent fully unattended (e.g. for scheduled jobs), set
`GOOSE_MODE: auto` in `~/.config/goose/config.yaml`. To require confirmation on
every tool call, use `approve`. See
[goose-permissions](https://goose-docs.ai/docs/guides/managing-tools/goose-permissions/).
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py``mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
## Cross-references
- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the
Hermes flow assumes is done.
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
session start (via `~/.config/goose/.goosehints`).
- [`bin/hermes`](../../bin/hermes) — the wrapper that decrypts the OpenRouter key
and execs `goose session`.
- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-hermes` flag's install block.
## Follow-ups
1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
(`"SSE transport is no longer supported - kept only for config file
compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
server at `mcp/server.py:336` still calls `mcp.run(transport="sse")`. Until
that's changed, the `homelab` MCP extension in Goose will fail to connect.
The developer extension (shell + edit) covers most ops without it; this is
a polish item, not a blocker.
2. **Per-host OpenRouter keys** for billing attribution. Today all Hermes
hosts share one key.
3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
directly — OpenRouter periodically rotates the underlying weights.
4. **Local-inference fallback** (ollama / vllm) once the homelab has a GPU
node. The wrapper, persona, and MCP wiring stay unchanged; only
`GOOSE_PROVIDER`/`GOOSE_MODEL` change.
7. **Caveman auto-setup via post-pull hook.** The sync timer now calls
`tools/post-pull.sh`, which runs any `tools/*.setup.sh` after git pull.
Currently this auto-installs the Caveman npm package, wrapper scripts, and
compact output templates on all agent hosts (*token efficiency*).
## Changelog
### 2026-06-01 — caveman + post-pull auto-setup
Added `tools/post-pull.sh` sync hook that auto-runs `tools/*.setup.sh`
after every git pull. First user: `tools/setup-caveman.sh` installed Caveman
templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token-
efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
Also created `tools/caveman/` with the wrapper script, JS renderer, and
templates — the canonical source for all agent hosts.
Captures the Hermes-on-Goose onboarding flow added in the same commit as
`bootstrap.sh --with-hermes`, `bin/hermes`, the sops rule for
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-hermes`
extension. MCP streamable_http migration is queued as follow-up #1.

View File

@@ -2,7 +2,7 @@
How the narrative documentation in this repo is organized. The pattern is borrowed from the
`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top
of immutable evidence (`knowledge/sources/`, `investigations/`), with pure-listing indexes and an
of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an
append-only operations log.
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
@@ -11,7 +11,7 @@ moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for t
## Layers
- **Sources** are immutable raw material: incident records (`investigations/`), external reference
- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
Read them; do not rewrite them into other sources.
- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per

View File

@@ -96,7 +96,7 @@ What it looks like after.
Changelog entries to write, index status to update.
```
### Investigation (`investigations/YYYY-MM-DD-slug.md`)
### Investigation (`knowledge/sources/investigations/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>

View File

@@ -11,7 +11,7 @@ docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
Goal: bring a new host (workstation, LXC, VM) into inventory and the
secrets model, with mesh membership only where it's actually needed.
This wraps the existing `homelab client add` flow — see
[operations/agent-enrollment.md](../../../operations/agent-enrollment.md) for
[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for
the full walkthrough; this runbook is the risk/lifecycle framing.
1. On any enrolled client: `homelab client add <hostname>` — appends a

View File

@@ -18,7 +18,7 @@ own repo) and get it live, safely.
`oikos/policy.yaml`.
2. Clone/pull the `config_repo` (never edit the backend's working tree
directly — tracked configs change by commit + push, per
[OIKOS.md](../../../OIKOS.md) conventions).
[OIKOS.md](../../OIKOS.md) conventions).
3. Make the change, commit, push to `main`.
4. The Gitea webhook fires the deploy pipeline for that repo (see
[infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for

View File

@@ -23,7 +23,7 @@ Goal: understand what broke and why, before touching anything.
5. Check the change ledger for recent mutations to the affected entity
or anything upstream of it: `homelab service <name> history` (once
populated) or grep `ledger/*.jsonl`.
6. Write findings to a new `investigations/<date>-<slug>.md` — symptom,
6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom,
timeline, root cause, fix applied, prevention. This is the durable
record; don't rely on chat history.

View File

@@ -102,7 +102,7 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
## Related
- [Operations cheatsheet](../../../operations/commands.md)
- [Operations cheatsheet](../../operations/commands.md)
- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md)
- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md)