diff --git a/.agents/NOMOS.md b/.agents/NOMOS.md index 3da0ceb..1c8d1a1 100644 --- a/.agents/NOMOS.md +++ b/.agents/NOMOS.md @@ -1,19 +1,27 @@ # NOMOS.md — Agent persona for homelab clients -This file is the canonical agent persona for **all** AI agents running on -machines in the **hubris** homelab. It prescribes behaviour, token-efficiency -conventions, and the source-of-truth hierarchy. +This file is the canonical agent persona for AI agents running on machines +in the **hubris** homelab (Claude Code, Codex, or similar). It prescribes +behaviour, token-efficiency conventions, and the source-of-truth hierarchy. + +The *production* Nomos agent (`cmd/nomos`, the containerized MCP client +gateway everyone actually talks to) uses a separate, code-adjacent persona — +`nomos/SOUL.md`, baked into its Docker image at build time +(`compose/nomos/Dockerfile`). This file is unrelated to that one; it's for +AI coding agents working *on* a homelab client machine, not the Nomos +service itself. ## Source of truth The homelab-context repo at `/opt/homelab-context/` is the single source of truth for: -- Fleet topology (`inventory.yaml`, `inventory.yaml`) -- Service endpoints and credentials (via `homelab secret`) +- Fleet topology (`inventory.yaml`) - Agent behaviour and conventions - Everything in this file -When in doubt, check `/opt/homelab-context/` first. +When in doubt, check `/opt/homelab-context/` first, or query the Oikos API/MCP +server directly (see [AGENTS.md](../AGENTS.md) §3-4) — the database is +authoritative at runtime. ## Runbooks — load, don't rediscover @@ -28,69 +36,9 @@ wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the operating model these runbooks execute inside (OODA loop, risk classes, approval flow, ontology). -## Agent type — how this file gets loaded +## Token efficiency -| Agent | Loading mechanism | -|-------|------------------| -| **Nomos** | `tools/setup-nomos-soul.sh` (auto-setup) → provisions `~/.nomos/SOUL.md` from this file | -| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints` → `/opt/homelab-context/NOMOS.md` | -| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions | - -**Do not edit SOUL.md or .goosehints directly.** Edit this file in the -homelab-context repo instead. Changes propagate to all clients on the next -sync (`sudo homelab sync`). - ---- - -## Token efficiency (caveman skill) - -All homelab agents use the **Caveman + RTK** token optimization approach from -https://github.com/adityahimaone/hermes-agent-rtk-caveman. - -### Before running any CLI command, ask: - -1. **Is there a caveman wrapper equivalent?** Use the wrapper for token-efficient - output. Available wrappers (installed at `~/bin/caveman_wrapper.sh`): - - `~/bin/caveman_wrapper.sh git-status` — compact git status - - `~/bin/caveman_wrapper.sh git-log [n]` — compact git log - - `~/bin/caveman_wrapper.sh lint [target]` — compact lint results - - `~/bin/caveman_wrapper.sh test-results [cmd]` — compact test results - -2. **If no caveman wrapper exists, pipe through `rtk`** to compress output: - ``` - rtk - ``` - RTK (Rust Token Killer) strips redundant whitespace, trims long paths, and - deduplicates repeated lines. This reduces token usage by 60-90% on CLI - operations. - -3. **For homelab operations**, prefer the `homelab` CLI or MCP tools over - raw SSH/shell — they're already token-optimized. - -### Templates - -Caveman templates live at `~/templates/`: -- `git_status.txt` — compact git status format -- `git_log.txt` — compact git log format -- `lint_results.txt` — compact ESLint format -- `test_results.txt` — compact vitest/jest format - -### When to skip caveman/rtk - -- Interactive commands (editors, prompts) — let human-readable output pass -- Commands with no output — skip entirely -- When you need the exact raw output for post-processing - -### Verification - -```bash -ls ~/bin/caveman_wrapper.sh && echo "caveman ready" -``` - -## Important note for Nomos agents - -If you are reading this as a Nomos agent, your SOUL.md was auto-provisioned -by `tools/setup-nomos-soul.sh`. This file is the canonical original — you -can verify the content matches or re-provision by running: - - bash /opt/homelab-context/tools/setup-nomos-soul.sh +Apply [caveman.md](shared/caveman.md) — terse, fragment-heavy chat responses +(not committed documentation). There's no separate tool to install for +this; it's a response-style convention any agent follows by reading the +file. diff --git a/.agents/dev/CONTRIBUTING.md b/.agents/dev/CONTRIBUTING.md index c199adb..27a4ee8 100644 --- a/.agents/dev/CONTRIBUTING.md +++ b/.agents/dev/CONTRIBUTING.md @@ -10,6 +10,7 @@ see [CONTRIBUTING.md](../../CONTRIBUTING.md) for a human-friendly version. cmd/oikos/main.go Entry point. Subcommands: api, scheduler, notifier, migrate, seed, export, secret, all cmd/nomos/main.go Nomos MCP client gateway (standalone binary, formerly Hermes) +cmd/webhook/main.go Gitea deploy-webhook receiver (push-to-deploy on mac-mini) internal/httpapi/ REST + MCP server. Chi router. OpenAPI-generated types from internal/httpapi/gen/api.gen.go. Strict server in impl.go. internal/mcp/ MCP tool implementations (get_entity, search_knowledge, etc.) @@ -26,14 +27,19 @@ internal/domain/ Core types: entities, approvals, executions, signals internal/ontology/ Type hierarchy validation, relationship checks internal/knowledge/ Knowledge YAML seed ingestion internal/config/ Config loading from env vars +web/ Control-room SPA (Svelte 5) — standalone static build, not + embedded in the oikos binary (plans/2026-07-12-wails-desktop-app.md) api/openapi.yaml REST API contract. Source of truth for endpoints. api/codegen.yaml oapi-codegen config → generates internal/httpapi/gen/ migrations/ Forward-only SQL. Format: NNN_name.up.sql. No down migrations. seeds/ Bootstrap YAML. ontology.yaml, inventory.yaml, policy.yaml, knowledge.yaml. Regenerated from DB via oikos export. -compose/ Dockerfiles. oikos/ (multi-stage), nomos/ (distroless). +compose/ Dockerfiles. oikos/ (2-stage, Go only — SPA is built/deployed + separately), nomos/ (distroless). Caddy config at compose/caddy/Caddyfile.oikos. scripts/ Deploy, rollback, watchdog, verification, cutover checklist. +checks/ Host health-check scripts run over SSH by the scheduler. +tools/ Client auto-setup scripts (checks). nomos/ Nomos config.yaml, SOUL.md, skills. .agents/ Agent instruction files, domains, shared conventions, skills. plans/ Design documents. active/ + done/. diff --git a/.agents/operations/agent-enrollment.md b/.agents/operations/agent-enrollment.md index dce3790..74ee0e7 100644 --- a/.agents/operations/agent-enrollment.md +++ b/.agents/operations/agent-enrollment.md @@ -1,365 +1,120 @@ -# Agent enrollment — bootstrap a client into the homelab context system +# Agent enrollment — operational notes -This walks through enrolling a new machine (workstation, LXC, or VM) so it -joins the cross-client context system: a `/opt/homelab-context/` clone of -this repo that auto-syncs every 5 min, a per-client age key for SOPS -decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config. +**For the actual enrollment flow, see [CLIENTS.md](../../CLIENTS.md#enrollment) +— it's the current, authoritative version.** This page used to duplicate +that flow in more detail, describing a `homelab` CLI-based two-step +ceremony (`homelab client add` reserves an inventory slot → client +bootstraps → operator finalizes the pubkey). That CLI and that flow don't +exist anymore — enrollment today is one shot: `bootstrap.sh` calls +`POST /api/v1/clients/enroll` directly and gets back an age keypair + +Infisical identity in the same response. What's left here is the handful +of things that are still true and weren't already covered elsewhere. -> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment? -> See [nomos-agent.md](nomos-agent.md). It uses the same `bootstrap.sh` -> with an additional `--with-nomos` flag. - -Architecture in [project_homelab_context_plan](https://… memory link); the -operational reference is here. - -## Prerequisites the client must satisfy +## Prerequisites | Requirement | Why | How to check | | --- | --- | --- | -| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) | +| Hostname matches an entry in `inventory.yaml` | `EnrollClient` looks up the entity by slug derived from hostname; it must exist in `planned`/`provisioning` state. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) | | OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` | -| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` | -| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below | -| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) | +| On the mesh (Netbird) **or** on the LAN | enrollment validates mesh IP against expected subnets | `netbird status` | +| `curl`, `jq`, `age`, `python3` | bootstrap preflight (`bootstrap.sh:100`) — auto-installed on Fedora/RHEL/Debian/Ubuntu/macOS if missing | `command -v curl jq age python3` | +| Can resolve `*.hubris.network` | bootstrap calls the Oikos API and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` | ### Hostname mismatch is the most common bootstrap failure -If the bootstrap exits with `no hosts/.yaml in the repo`, the -hostname doesn't match any inventory entry. Two fixes: +If the entity for your hostname doesn't exist yet (in `planned` or +`provisioning` state), enrollment 4xxs. Two fixes: -- **Rename the host**: `sudo hostnamectl set-hostname ` - (Linux) or System Preferences → Sharing (macOS), then re-run. -- **Rename the inventory entry**: edit `inventory.yaml` on hubris, - update `inventory.yaml`, push. The next sync (≤5 min) propagates. +- **Rename the host** to match an existing planned entity: + `sudo hostnamectl set-hostname ` (Linux) or System + Preferences → Sharing (macOS), then re-run. +- **Add/rename the inventory entry**: edit `seeds/inventory.yaml`, ingest + via `oikos seed` (or the equivalent MCP/API entity-creation path), then + re-run bootstrap. -### Getting onto Netbird +### Networking prerequisites (Netbird, DNS, SSH key distribution) -Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit ``). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`. - -**Path B — interactive OIDC (default; recommended):** - -The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight. - -Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../archive/knowledge/containers/106-auth-outpost.md). Only needed once per account. - -**Path A — setup-key (headless/scripted onboarding):** - -Useful for headless servers (no browser at all) or unattended cloud-init bootstraps. - -1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`. -2. **Setup Keys** → Create → set reusable + expiry → copy. -3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver): - ```bash - sudo netbird up --setup-key \ - --management-url https://netbird.hubris.network \ - --ssh-jwt-cache-ttl 86400 - ``` - -**Why we can't OIDC-login from the public internet (still open as a follow-up):** - -`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124. - -### DNS prerequisite - -`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124 -([dns.md](../../archive/knowledge/infrastructure/dns.md)) for LAN clients, **but only if the -client uses 192.168.8.180 as its resolver**. Most LXCs and roaming -workstations don't by default. Options: - -- **LAN client**: set DNS to 192.168.8.180 (per-interface or - `/etc/resolv.conf`). -- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to - point `*.hubris.network` at LXC 124. -- **Hack-fix anywhere**: append to `/etc/hosts`: - ``` - 192.168.8.175 mcp.hubris.network secrets.hubris.network - 192.168.8.175 git.hubris.network - ``` - (192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.) - -If DNS isn't an option at all, override the URLs at bootstrap time: - -```bash -sudo HOMELAB_GITEA_TOKEN=... \ - HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git \ - HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \ - HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \ - bash /tmp/bootstrap.sh --with-mcp -``` - -## Install dependencies - -Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script). - -Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional. - -
-Manual recipes (Fedora / Debian / macOS) - -```bash -# Fedora / RHEL / Nobara -sudo dnf install -y git python3-pyyaml age curl -SOPS_VERSION=v3.9.4 -sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \ - -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops - -# Debian / Ubuntu -sudo apt update && sudo apt install -y git python3-yaml age curl -SOPS_VERSION=v3.9.4 -sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \ - -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops - -# macOS -brew install git age sops -pip3 install pyyaml # if `python3 -c "import yaml"` fails -``` - -
- -## Run the bootstrap - -You need a Gitea read-only personal access token for the initial clone -(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml` -but a new client can't decrypt it before bootstrap — chicken-and-egg). -Ask the operator (or generate in Gitea: Settings → Applications → Generate -New Token → scope `read:repository`). - -```bash -TOKEN=... # your Gitea PAT, scope read:repository - -# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy). -curl -fsSL -u "dtoro:$TOKEN" \ - https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \ - -o /tmp/bootstrap.sh - -# Run it. -sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp -``` - -Flags: - -| Flag | Effect | -| --- | --- | -| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user | -| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) | -| `--dry-run` | Prints actions without executing | - -The bootstrap is idempotent: re-running on an enrolled client just -verifies state, re-issues the age key only if it doesn't match the -inventory pubkey, and refreshes the sync timer + symlinks. - -## Verify - -```bash -homelab whoami # prints hosts/$(hostname).yaml -homelab list # shows the full topology -homelab status # ping + HTTP-check across hosts/services -homelab secret hello # decrypt the bootstrap-test secret -systemctl list-timers homelab-context-sync.timer - # next run within ≤5 min -``` - -For Claude Code: start a new session — the `homelab` MCP server appears -in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management, -1 secrets-metadata). - -## Post-bootstrap: SSH reachability - -A new workstation must be reachable from other workstations and must be -able to reach every host by short hostname. Run these steps after the -bootstrap verify passes: - -### 1. Enable SSH server - -```bash -# macOS: -sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist - -# Linux: -sudo systemctl enable --now sshd -``` - -### 2. Generate SSH key (if missing) - -```bash -ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100 -``` - -### 3. Publish pubkey to the repo - -```bash -cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub -cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push -``` - -### 4. Deploy keys to all hosts - -From any existing enrolled machine (hubris or another workstation): - -```bash -ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh" -``` - -This adds the new workstation's pubkey to hubris and every running LXC. - -### 5. Generate SSH config - -```bash -homelab ssh-config --install -``` - -Verify: - -```bash -ssh hubris hostname # should return "hubris" without password -ssh gitea hostname # should return "gitea" without password -ssh mac-mini hostname # should return "mac-mini" without password (workstation-to-workstation) -``` - -### 6. Add LAN IP to inventory (if on LAN) - -If the workstation has a static or reserved LAN IP, add it to -`inventory.yaml`: - -```yaml -hosts: - your-hostname: - lan_ip: 192.168.8.xxx -``` - -This gives it a primary LAN entry in the generated SSH config (faster -than the Netbird fallback). Commit + push, then: - -```bash -cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install -``` +Migrated to a runbook in the knowledge base — query +`search_knowledge("netbird mesh dns")` or `get_entity_knowledge`, or ask +Nomos. Covers: getting onto the Netbird mesh (interactive OIDC vs. +setup-key), why OIDC login can fail from off-mesh, split-horizon DNS +options, and distributing a new workstation's SSH pubkey across the fleet +via `ssh/deploy-keys.sh`. ## Claude Code permissions for fleet ops By default Claude Code's auto-mode classifier asks for confirmation on every -ssh into the mesh. The bootstrap already installs the ssh ControlMaster block -so subsequent in-session sshes multiplex, but the *first* ssh of each session -still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns -by adding to `~/.claude/settings.json`: +ssh into the mesh. Pre-authorize the common fleet ssh pattern by adding to +`~/.claude/settings.json`: ```json { "permissions": { "defaultMode": "auto", "allow": [ - "Bash(ssh -p 22022 *)", - "Bash(homelab *)" + "Bash(ssh -p 22022 *)" ] } } ``` -The first rule covers any ssh to a mesh peer on the homelab netbird port; the -second covers all `homelab` CLI invocations. Both are scoped tight enough that -the classifier doesn't gate them but loose enough to handle the variety of -arguments. +This covers any ssh to a mesh peer on the homelab netbird port, scoped tight +enough that the classifier doesn't gate it but loose enough to handle the +variety of arguments. -If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be -visible to the classifier (it's not actually durable in 0.71.2, but the -ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) -for context), drop a free-text rule into `autoMode.allow` describing the -authorization. Optional. +## Open questions (not verified against current architecture — don't +guess these from the old flow) -## Adding a new client to inventory +The old two-step ceremony had answers for these; the current one-shot +`/api/v1/clients/enroll` flow may handle them differently and this hasn't +been re-verified: -If the hostname you want isn't yet in inventory, enrollment is a two-step -ceremony driven from an existing enrolled client (e.g. hubris). The -`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3. - -```bash -# 1. On hubris (or any existing client): add the inventory entry. -homelab client add my-new-machine -# Prompts for kind, os, netbird FQDN, role. Commits + pushes. - -# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key). - -# 3. On the new machine: install deps + run bootstrap (above). -# Bootstrap calls /issue, receives a fresh age keypair, and prints the -# public key for the operator to commit back to inventory. - -# 4. On hubris: finalize the age public key. -homelab client add my-new-machine --finalize-pubkey age1... -# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates -# inventory.yaml, commits + pushes. The 5-min sync propagates. -``` - -## Granting a secret to a new client - -Adding a client doesn't grant them every secret. Recipients are explicit -per file via `.sops.yaml` glob rules. To grant a client access to (say) -`secrets/hello.yaml`: - -1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to - the matching `creation_rules` block. -2. Re-key the existing ciphertext for the new recipient list: - ```bash - sops updatekeys -y secrets/hello.yaml - ``` -3. Commit + push. On the next sync (≤5 min), the client can decrypt. - -## Removing a client - -```bash -# From any existing client: -homelab client remove my-old-machine -``` - -This: -1. Removes the inventory entry and `hosts/my-old-machine.yaml`. -2. Runs `sops updatekeys -y` against every file in `secrets/` (operator - must first remove the pubkey from `.sops.yaml` rules). -3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to - shred the key file and add the hostname to the denylist. -4. Commits + pushes. - -The CLI prints a follow-up checklist that the operator must do manually: - -- Revoke the peer in the Netbird console (denies future mesh access). -- **Rotate any credentials whose ciphertext the removed client already - has on disk.** The age key revocation only protects *future* - ciphertext; what's already been pulled is still decryptable until the - underlying credential changes. -- Optional: `homelab nuke my-old-machine` SSHes in, shreds - `/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync. +- **Removing a client.** No current equivalent confirmed for the old + `homelab client remove` (inventory removal + secret re-keying + key + revocation). Likely maps to an entity lifecycle transition + (`.agents/skills/lifecycle-deprecate-node/` or `lifecycle-destroy-node/`) + but those skills reference the same dead CLI and need their own check. +- **Granting a secret to an already-enrolled client.** The old flow + hand-edited `.sops.yaml` `creation_rules` + `sops updatekeys`. Given + Infisical is now the primary secrets backend (SOPS is the DR fallback), + the current mechanism is probably Infisical-side, not a `.sops.yaml` edit + — not confirmed. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | -| `no hosts/.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) | -| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` | -| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git` | -| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix | -| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy | -| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) | -| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) | -| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. | -| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. | -| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. | +| Enrollment 404s / entity not found | Hostname doesn't match a `planned`/`provisioning` inventory entry | See "Hostname mismatch" above | +| `gnutls_handshake() failed` / TLS errors reaching `*.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS instead of the LAN/mesh path | See the networking runbook (split-horizon DNS section) | | Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with ` ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). | -| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../archive/knowledge/containers/106-auth-outpost.md). | -| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. | -| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh ` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). | -| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. | ## Changelog -### 2026-06-02 — SSH reachability post-bootstrap steps -Added a new "Post-bootstrap: SSH reachability" section covering SSH key -generation, pubkey publication, deployment to hosts, SSH config generation, -and LAN IP registration. New workstations enrolled via this doc will -automatically join the universal SSH mesh. -### 2026-05-31 — cross-link to nomos-agent.md +### 2026-07-12 — trimmed to current architecture +Removed everything describing the retired `homelab` CLI-based two-step +enrollment ceremony (now: `CLIENTS.md`'s one-shot flow), the Nous-Hermes/ +Goose cross-link (that whole flow was removed the same day), and CLI-syntax +troubleshooting rows with no current equivalent. Migrated the still-true +Netbird/DNS/SSH-distribution content to a knowledge-base runbook rather +than duplicating it here. What's left is genuinely current or explicitly +flagged as unverified. Original ~365-line version is in git history +(`git log -- .agents/operations/agent-enrollment.md`) if any of the removed +detail turns out to still be needed. -Added a sibling page covering Nous-Hermes-on-Goose enrollment ([nomos-agent.md](nomos-agent.md)) and noted it at the top of this page. The Nomos flow extends `bootstrap.sh` with `--with-nomos` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here. +### 2026-06-02 — SSH reachability post-bootstrap steps +Added a section covering SSH key generation, pubkey publication, +deployment to hosts, SSH config generation, and LAN IP registration. New +workstations enrolled via this doc automatically join the SSH mesh. +(Superseded 2026-07-12 — migrated to the networking runbook.) + +### 2026-05-31 — cross-link to nomos-agent.md +Added a sibling page covering Nous-Hermes-on-Goose enrollment. (Removed +2026-07-12 along with the rest of that flow.) ### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows -Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper. +Added three rows to the troubleshooting table covering issues surfaced +during the netbird vanilla migration. (Migrated 2026-07-12 to the +networking runbook.) ### 2026-05-20 — initial page Captures the enrollment flow validated during Phase 2 of the homelab diff --git a/.agents/operations/commands.md b/.agents/operations/commands.md index fec5ff7..f276ccf 100644 --- a/.agents/operations/commands.md +++ b/.agents/operations/commands.md @@ -53,34 +53,36 @@ Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. Whe ## Fleet apt operations -Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC. - -| Command | What it does | -| --- | --- | -| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. | -| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. | -| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. | -| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. | -| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. | -| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. | - -PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm. +**No current CLI equivalent.** `homelab apt-audit`/`apt-upgrade` (dpkg-state +audit, fanned-out apt upgrade with pre-upgrade snapshots) were part of the +retired Python `homelab` CLI and don't have a ported replacement — apt +patching today is ad hoc `run` MCP tool calls per host, without the +audit/snapshot/status wrapping this used to provide. If that wrapping is +still wanted, it needs to be rebuilt (e.g. as a runbook driving `run`, or a +new MCP tool) — see +[runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) for +the dpkg-interrupted recovery procedure specifically. ## Oikos (agent OS layer) -See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference: +See [OIKOS.md](../OIKOS.md) for the operating model. The `homelab` CLI this +section used to document is retired; the actual current interface is the +33 MCP tools cataloged in [AGENTS.md](../../AGENTS.md#3-the-mcp-server) plus +the REST API. Closest current equivalents for what used to live here: -| Command | What it does | +| Old `homelab` command | Current equivalent | | --- | --- | -| `homelab service explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history | -| `homelab node relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius | -| `homelab change preflight ` | Dry-run report before mutating: risk class, current health, config repo, verification command | -| `homelab decide ` | Decision classifier: risk × blast radius × confidence → auto-act or escalate | -| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention | -| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Nomos, or the Oikos Console's `/approvals` page) | -| `homelab restart [--approval-id ]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant | +| `homelab service explain\|health\|docs\|log` | MCP `explain`, `get_service_status`, `tail_log`, `get_entity_knowledge` | +| `homelab node relations` | MCP `get_blast_radius` | +| `homelab change preflight ` | MCP `preflight` | +| `homelab signal list\|ack\|resolve\|mute` | MCP `get_signal_history`, or REST `POST /api/v1/signals/{id}/ack\|resolve\|mute` (the control-room UI's Signals page wraps these) | +| `homelab approval request\|list\|reply\|check` | REST `GET/POST /api/v1/approvals*` (Matrix-delivered via the notifier, or the control-room UI's Operations page) | +| `homelab restart --approval-id ` | MCP `run` (policy-gated — auto-executes if read-only/reversible_low, otherwise queues for the same Matrix/UI approval) | +| `homelab decide ` | No direct equivalent — classification now happens inline inside `run`/`request_execution`, not as a separate dry-run call | -Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../archive/oikos-cards/). +There is no separately-deployed "Oikos Console" anymore — the control-room +SPA (`web/`) is the operator dashboard, served standalone (see +[plans/2026-07-12-wails-desktop-app.md](../../plans/2026-07-12-wails-desktop-app.md)). ## Related - [Hubris host](../../archive/knowledge/hosts/hubris.md) diff --git a/.agents/operations/nomos-agent.md b/.agents/operations/nomos-agent.md deleted file mode 100644 index f3ac066..0000000 --- a/.agents/operations/nomos-agent.md +++ /dev/null @@ -1,210 +0,0 @@ -# Nomos agent — LLM-powered terminal sessions on a homelab client - -Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned -Llama variant) as a working terminal agent on a homelab client. Builds on top -of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md)) -— this page covers only the Hermes-specific additions. - -The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides: - -- The chat loop, multi-turn history, and streaming -- The OpenRouter provider that routes to the configured LLM -- The built-in `developer` extension (shell + file editor — same surface Claude - Code has) -- A remote MCP extension pointed at `mcp.hubris.network` for read-only - homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.) - -The persona is `/opt/homelab-context/NOMOS.md`, symlinked as Goose's global -`.goosehints` so it's injected into the system prompt on every session. - -## Prerequisites - -| Requirement | How | -| --- | --- | -| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) | -| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below | -| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add --finalize-pubkey --with-nomos` | - -## Onboarding flow - -```bash -# 1. On hubris (or any enrolled client): reserve the inventory entry. -homelab client add new-machine - -# 2. Join new-machine to Netbird (setup-key or OIDC). - -# 3. On new-machine: bootstrap with --with-nomos. -TOKEN=... # gitea PAT, read:repository -curl -fsSL -u "dtoro:$TOKEN" \ - https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \ - -o /tmp/bootstrap.sh -sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-nomos - -# 4. Back on hubris: finalize the age pubkey AND grant the Nomos secret. -homelab client add new-machine \ - --finalize-pubkey age1... \ - --with-nomos - -# 5. Wait ≤5 min for sync, then on new-machine: -nomos "what LXCs are running?" -``` - -The bootstrap `--with-nomos` flag does five things, all idempotent: - -1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose` - (upstream installer) and symlinks `/usr/local/bin/goose` to it. -2. Symlinks `/opt/homelab-context/bin/nomos` → `/usr/local/bin/nomos`. -3. Symlinks `/opt/homelab-context/NOMOS.md` → `/root/NOMOS.md` (Linux) or - `/etc/NOMOS.md` (macOS) for `cat`-as-operator convenience. -4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and - extensions (preserves any keys the operator added by hand). -5. Symlinks `~/.config/goose/.goosehints` → NOMOS.md, so the persona is - injected as the system prompt on every session. - -## Seeding the OpenRouter key - -The first time anyone enrolls with `--with-nomos`, the encrypted file -`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any -existing recipient): - -```bash -sops secrets/openrouter-api-key.yaml -# editor opens; replace api_key value with the real sk-or-... key, save, close. -git -C /opt/homelab-context add secrets/openrouter-api-key.yaml -git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key' -git -C /opt/homelab-context push -``` - -Until this step happens, `nomos …` exits with `openrouter-api-key.yaml still -contains the placeholder`. Subsequent enrollees get the real key automatically -via `--with-nomos` (which adds them as a sops recipient on -`secrets/openrouter-api-key.yaml`). - -## Granting the OpenRouter key to an already-enrolled host - -If a host was enrolled without `--with-nomos` and you want to add it later: - -```bash -# On hubris: -PUBKEY=$(homelab whoami --hostname | grep age_pubkey | awk '{print $2}') -homelab client add --finalize-pubkey "$PUBKEY" --with-nomos -``` - -`--finalize-pubkey` is required by the existing flow even when the pubkey is -unchanged — it's also the trigger that runs the sops grant. - -After ≤5 min sync the host can decrypt the key. Bootstrap doesn't need to -re-run; only the secret recipient list changed. - -## Verifying - -```bash -homelab whoami # standard enrollment OK -homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`) -which goose && which nomos # binaries present -goose info -v # provider/model wiring sane -nomos "what LXCs are running?" # interactive Goose session - -# Non-interactive smoke test: -echo "List the homelab MCP tools you have available" | nomos -``` - -## Configuration - -The bootstrap-managed keys in `~/.config/goose/config.yaml`: - -```yaml -GOOSE_PROVIDER: openrouter -GOOSE_MODEL: deepseek/deepseek-v4-flash -GOOSE_MODE: smart_approve # asks before destructive tool calls -extensions: - developer: - type: builtin - bundled: true - enabled: true - name: developer - timeout: 300 - homelab: - type: streamable_http - enabled: true - name: homelab - uri: https://mcp.hubris.network/mcp - timeout: 60 -``` - -Override via env on a single bootstrap run: - -```bash -HOMELAB_NOMOS_MODEL=nousresearch/hermes-3-llama-3.1-405b \ -HOMELAB_NOMOS_MCP_URI=https://mcp.hubris.network/mcp \ - sudo bash /tmp/bootstrap.sh --with-nomos -``` - -Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are -preserved across re-bootstraps — the merge only overwrites the keys it manages. - -## Tool permissions - -`GOOSE_MODE: smart_approve` is the bootstrap default: Goose runs read-only -shell commands without prompting and asks for confirmation before destructive -ones. To make the agent fully unattended (e.g. for scheduled jobs), set -`GOOSE_MODE: auto` in `~/.config/goose/config.yaml`. To require confirmation on -every tool call, use `approve`. See -[goose-permissions](https://goose-docs.ai/docs/guides/managing-tools/goose-permissions/). - -## Troubleshooting - -| Symptom | Cause | Fix | -| --- | --- | --- | -| `nomos: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add --finalize-pubkey --with-nomos` from hubris | -| `nomos: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above | -| Goose hangs on first `nomos` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. | -| `homelab` extension fails to connect / no MCP tools listed | MCP server upgraded in Go rewrite (`internal/mcp/server.go`, Streamable HTTP via official MCP SDK). Old FastMCP SSE transport is deprecated. | Run `docker compose --profile full up` on mac-mini, or wait for the production cutover from apps/105. | -| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-nomos`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. | -| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. | - -## Cross-references - -- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the - Nomos flow assumes is done. -- [`NOMOS.md`](../NOMOS.md) — the persona the Nomos agent reads on every - session start (via `~/.config/goose/.goosehints`). -- [`bin/nomos`](../../bin/nomos) — the wrapper that decrypts the OpenRouter key - and execs `goose session`. -- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-nomos` flag's install block. - -## Follow-ups - -1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE - (`"SSE transport is no longer supported - kept only for config file - compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP - server at `internal/mcp/server.go` uses Streamable HTTP (official MCP SDK). Until - that's changed, the `homelab` MCP extension in Goose will fail to connect. - The developer extension (shell + edit) covers most ops without it; this is - a polish item, not a blocker. -2. **Per-host OpenRouter keys** for billing attribution. Today all Nomos - hosts share one key. -3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b` - directly — OpenRouter periodically rotates the underlying weights. -4. **Local-inference fallback** (ollama / vllm) once the homelab has a GPU - node. The wrapper, persona, and MCP wiring stay unchanged; only - `GOOSE_PROVIDER`/`GOOSE_MODEL` change. - -7. **Caveman auto-setup via post-pull hook.** The sync timer now calls - `tools/post-pull.sh`, which runs any `tools/*.setup.sh` after git pull. - Currently this auto-installs the Caveman npm package, wrapper scripts, and - compact output templates on all agent hosts (*token efficiency*). - -## Changelog - -### 2026-06-01 — caveman + post-pull auto-setup -Added `tools/post-pull.sh` sync hook that auto-runs `tools/*.setup.sh` -after every git pull. First user: `tools/setup-caveman.sh` installed Caveman -templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token- -efficient CLI output. Replaces raw `git pull` in launchd/systemd timers. -Also created `tools/caveman/` with the wrapper script, JS renderer, and -templates — the canonical source for all agent hosts. -Captures the Nomos-on-Goose onboarding flow added in the same commit as -`bootstrap.sh --with-nomos`, `bin/nomos`, the sops rule for -`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-nomos` -extension. MCP streamable_http migration is queued as follow-up #1. diff --git a/.agents/skills/client-enrollment/SKILL.md b/.agents/skills/client-enrollment/SKILL.md index 3844999..7ba4ddc 100644 --- a/.agents/skills/client-enrollment/SKILL.md +++ b/.agents/skills/client-enrollment/SKILL.md @@ -2,39 +2,46 @@ name: client-enrollment risk_class: config_mutation inputs: [hostname, kind, role] -verification: "homelab doctor (on the new client)" +verification: "MCP whoami(hostname) shows the entity active" docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm] --- # Client enrollment Goal: bring a new host (workstation, LXC, VM) into inventory and the -secrets model, with mesh membership only where it's actually needed. -This wraps the existing `homelab client add` flow — see -[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for -the full walkthrough; this runbook is the risk/lifecycle framing. +secrets model, with mesh membership only where it's actually needed. See +[CLIENTS.md](../../../CLIENTS.md#enrollment) for the actual current +flow and [operations/agent-enrollment.md](../../operations/agent-enrollment.md) +for operational notes; this runbook is the risk/lifecycle framing. -1. On any enrolled client: `homelab client add ` — appends a - `hosts.:` block to `inventory.yaml` (lifecycle `state: planned` - → `provisioning`, per [seeds/ontology.yaml](../../../seeds/ontology.yaml)), - commits + pushes. +1. The entity must exist in `planned`/`provisioning` state before the new + host can self-enroll — add a `hosts.:` block to + `seeds/inventory.yaml` and `oikos seed` to ingest it (lifecycle + `planned` → `provisioning`, per + [seeds/ontology.yaml](../../../seeds/ontology.yaml)). 2. Netbird join is **optional, not a required step** — only needed for hosts that must be reachable off-LAN (workstations that roam, e.g. - `republic-laptop`, `mac-mini`). A node reachable on the household LAN - (192.168.8.0/24 — most LXCs/VMs) doesn't need it: it's already - reachable directly, and off-LAN clients reach it too via hubris's - routed `192.168.8.0/24` Netbird network resource. Skip this step for - LAN-only nodes; do it (out-of-band, console or setup key) only for - hosts that need independent off-LAN reachability. -3. On the new host: run `bootstrap.sh` (add `--with-nomos` to also - enroll the Hermes agent). This provisions `/etc/age/key.txt`, the - sync timer, and prints an age pubkey. -4. Back on an enrolled client: `homelab client add - --finalize-pubkey ` — sets `age_pubkey`, grants shared - secrets, re-keys SOPS, commits + pushes. This is the - `provisioning → active` transition. -5. Verify: `homelab doctor` on the new client should show all checks - green (clone, sync timer, age key, CLI symlink, MCP reachable). + `mac-mini`). A node reachable on the household LAN (192.168.8.0/24 — + most LXCs/VMs) doesn't need it. Skip for LAN-only nodes; do it + (out-of-band, console or setup key) only for hosts that need + independent off-LAN reachability. +3. On the new host: run `bootstrap.sh`. This calls + `POST /api/v1/clients/enroll`, which validates the entity exists and + the mesh IP is in an expected subnet, then returns an age keypair and + Infisical machine identity in one response — provisions + `/etc/age/key.txt`, `/etc/infisical/identity`, and the context poller. +4. **Known gap, confirmed 2026-07-12: `provisioning → active` has no + working path.** `EnrollClient` (`internal/httpapi/impl.go`) sets the + entity's state to `provisioning`, never `active`. `bootstrap.sh` prints + `POST /api/v1/clients/ws:$HNAME/activate` as the next step, but that + route doesn't exist — `api/openapi.yaml` only has `/clients/enroll`, + `/clients/{slug}/context`, `/clients/{slug}/secrets`. Until this is + fixed (add the route, or use the generic entity PATCH to flip `state`), + a freshly-enrolled client is stuck in `provisioning` — MCP `preflight` + and policy's `lifecycle_overrides` for `provisioning` still apply, but + nothing transitions it onward automatically. +5. Verify: MCP `whoami(hostname)` shows the entity in `active` state with + its peers and health. Docs-update checklist: if the new host is an LXC/VM, add its narrative page under `containers/` or `vms/` and set `doc_page` in its inventory diff --git a/.agents/skills/config-change-deploy/SKILL.md b/.agents/skills/config-change-deploy/SKILL.md index 3a4798c..f33fae2 100644 --- a/.agents/skills/config-change-deploy/SKILL.md +++ b/.agents/skills/config-change-deploy/SKILL.md @@ -2,7 +2,7 @@ name: config-change-deploy risk_class: config_mutation inputs: [service_name, change_description] -verification: "curl -sf (or homelab service health)" +verification: "curl -sf (or MCP get_service_status)" docs_update_checklist: [doc_page, changelog] --- @@ -11,11 +11,10 @@ docs_update_checklist: [doc_page, changelog] Goal: change a tracked config repo (Caddy, Gitea customizations, an app's own repo) and get it live, safely. -1. `homelab change preflight ` — current health, the service's - `config_repo`, its risk class, and the verification command to run - after. If risk class requires approval (`config_mutation` or - `destructive`), stop and get operator sign-off before editing — see - `seeds/policy.yaml`. +1. MCP `preflight` — current health, the service's `config_repo`, its + risk class, and the verification command to run after. If risk class + requires approval (`config_mutation` or `destructive`), stop and get + operator sign-off before editing — see `seeds/policy.yaml`. 2. Clone/pull the `config_repo` (never edit the backend's working tree directly — tracked configs change by commit + push, per [OIKOS.md](../../OIKOS.md) conventions). @@ -24,10 +23,10 @@ own repo) and get it live, safely. [infrastructure/auto-deploy.md](../../../archive/knowledge/infrastructure/auto-deploy.md) for the exact receiver/reload for this service). 5. Run the preflight's verification command. If it fails, check - `homelab service log` for the reload/restart error. -6. Record the change: once `oikos/ledger.py` is wired into deploy tooling - (Week 3), this is automatic; until then, note the change and outcome - in the relevant investigation/plan doc. + MCP `tail_log` for the reload/restart error. +6. No manual record-keeping step needed — mutations made through the API + (e.g. via the `run` MCP tool) are recorded automatically in the + `audit_log` table. Docs-update checklist: update the service's `doc_page` if the change alters its behavior, ingress route, or ownership; add a changelog entry diff --git a/.agents/skills/incident-investigation/SKILL.md b/.agents/skills/incident-investigation/SKILL.md index 2858192..e914863 100644 --- a/.agents/skills/incident-investigation/SKILL.md +++ b/.agents/skills/incident-investigation/SKILL.md @@ -10,22 +10,23 @@ docs_update_checklist: [investigations_entry] Goal: understand what broke and why, before touching anything. -1. `homelab service explain` (or `homelab node relations` - if the affected entity is a host) — get the blast radius and doc - pointer first. Don't start pulling logs blind. -2. `homelab service health` + `homelab service log` (or - MCP `get_service_status` / `tail_log`) for the affected service. +1. MCP `explain` (or `get_blast_radius` if the affected entity is a + host) — get the blast radius and doc pointer first. Don't start + pulling logs blind. +2. MCP `get_service_status` + `tail_log` for the affected service. 3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`, - `authentik`, or the backend host itself)? `homelab node - relations` shows "affected by" — check those first. -4. `homelab apt-audit` if the symptom looks like a dpkg/upgrade - interaction. + `authentik`, or the backend host itself)? MCP `get_blast_radius` + shows "affected by" — check those first. +4. If the symptom looks like a dpkg/upgrade interaction, see + [runbook-dpkg-interrupted](../runbook-dpkg-interrupted/SKILL.md) — + there's no fleet-wide apt-audit tool anymore, check the host directly. 5. Check the change ledger for recent mutations to the affected entity - or anything upstream of it: `homelab service history` (once - populated) or grep `ledger/*.jsonl`. -6. Write findings to a new `knowledge/sources/investigations/-.md` — symptom, - timeline, root cause, fix applied, prevention. This is the durable - record; don't rely on chat history. + or anything upstream of it: MCP `get_change_history` or `get_audit_trail`. +6. Write findings via MCP `upsert_knowledge` (`kind: investigation`) — + symptom, timeline, root cause, fix applied, prevention, `about` set to + the affected entity's slug. The DB is the durable record now, not a + markdown file — `search_knowledge`/`get_entity_knowledge` read it back; + a chat message alone is forgotten. Docs-update checklist: always create the investigation entry. If the root cause was stale/wrong inventory data (a `doc_page`, `config_repo`, diff --git a/.agents/skills/lifecycle-activate-node/SKILL.md b/.agents/skills/lifecycle-activate-node/SKILL.md index fa12296..e40853e 100644 --- a/.agents/skills/lifecycle-activate-node/SKILL.md +++ b/.agents/skills/lifecycle-activate-node/SKILL.md @@ -2,7 +2,7 @@ name: lifecycle-activate-node risk_class: config_mutation inputs: [node_name] -verification: "homelab service health (if it hosts a service); homelab doctor (if it's a client)" +verification: "MCP get_service_status (if it hosts a service); MCP whoami (if it's a client)" docs_update_checklist: [doc_page_complete] transition: "provisioning -> active" --- @@ -14,23 +14,22 @@ enrolled if it needs secrets, mesh joined if it needs off-LAN reach, ingress live if public, health check answering, doc page complete, ledger entry. -1. If the node is a `homelab` client: finish enrollment per - [client-enrollment.md](../client-enrollment/SKILL.md) (`--finalize-pubkey`, - mesh join, `homelab doctor` green). +1. If the node self-enrolls as a client: finish enrollment per + [CLIENTS.md](../../../CLIENTS.md#enrollment) (`bootstrap.sh` → + `/api/v1/clients/enroll`, mesh join, MCP `whoami` returns the entity). 2. If it hosts a public service: add the `services:` entry in - `inventory.yaml` (backend, url, doc_page, config_repo, risk_notes — - see the Week-1 service contract fields) and wire the Caddy route in - `dtoro/caddy-conf`. -3. Confirm the health check answers: `homelab service health` or - a direct `curl`. + `seeds/inventory.yaml` (backend, url, doc_page, config_repo, + risk_notes) and wire the Caddy route in `dtoro/caddy-conf`. +3. Confirm the health check answers: MCP `get_service_status` or a + direct `curl`. 4. Flip `state: provisioning` → `state: active` (or delete the `state:` - field — `active` is the default) in `inventory.yaml`. + field — `active` is the default) in `seeds/inventory.yaml`, then + `oikos seed` to ingest. 5. Complete the doc page (stub → full narrative: role, specs, how it's configured, dependencies). -6. Record the activation: `oikos/ledger.py append host: activate - config_mutation --result ok` (or let the CLI wrapper do this once - Week 3's runbook automation lands). +6. No manual record-keeping step needed — the activation (via whatever + API call flipped the state) is recorded automatically in `audit_log`. -Regenerate derived data: `python3 mcp/build_host_files.py && python3 -inventory.yaml` so `inventory.yaml`, the topology diagram, and -the context card all reflect the new state. +Regenerate: `oikos seed` re-ingests `seeds/inventory.yaml`; `oikos export` +writes DB state back out to the YAML if you mutated via the API/MCP +instead of editing the file directly. diff --git a/.agents/skills/lifecycle-deprecate-node/SKILL.md b/.agents/skills/lifecycle-deprecate-node/SKILL.md index d9d0857..a7cfa9d 100644 --- a/.agents/skills/lifecycle-deprecate-node/SKILL.md +++ b/.agents/skills/lifecycle-deprecate-node/SKILL.md @@ -2,7 +2,7 @@ name: lifecycle-deprecate-node risk_class: config_mutation inputs: [node_name, replacement_node_or_reason] -verification: "homelab node relations — 'affected by' must be empty before completing" +verification: "MCP get_blast_radius — 'affected by' must be empty before completing" docs_update_checklist: [doc_page_deprecation_note] transition: "active -> deprecated" --- @@ -16,14 +16,14 @@ suggestion; `seeds/policy.yaml` `lifecycle_overrides.deprecated.refuse` lists `new-inbound-edges` as refused going forward. 1. Set `state: deprecated` on the node. -2. `homelab node relations` — read `affected_by`. Every entry - there is something still relying on this node. +2. MCP `get_blast_radius` — read `affected_by`. Every entry there is + something still relying on this node. 3. Migrate or retire each dependent one at a time (point its `backend`/ `config_repo`/ingress route elsewhere, or deprecate it too if it's being retired alongside). -4. Re-run `homelab node relations` after each dependent is moved. - The transition to `destroyed` is only safe once `affected_by` is - empty — check this every time, don't assume from memory. +4. Re-run MCP `get_blast_radius` after each dependent is moved. The + transition to `destroyed` is only safe once `affected_by` is empty — + check this every time, don't assume from memory. 5. Note the deprecation on the doc page: reason, replacement (if any), date. diff --git a/.agents/skills/lifecycle-destroy-node/SKILL.md b/.agents/skills/lifecycle-destroy-node/SKILL.md index 5c9ad22..dc3d119 100644 --- a/.agents/skills/lifecycle-destroy-node/SKILL.md +++ b/.agents/skills/lifecycle-destroy-node/SKILL.md @@ -2,7 +2,7 @@ name: lifecycle-destroy-node risk_class: destructive inputs: [node_name] -verification: "homelab node relations returns unknown-entity; pct list on the backend no longer shows it" +verification: "MCP get_blast_radius returns unknown-entity; pct list on the backend no longer shows it" docs_update_checklist: [archaeology_entry, containers_index_update] transition: "deprecated -> destroyed" --- @@ -15,28 +15,35 @@ recipients removed + re-keyed, ingress/DNS removed, archaeology entry, ledger entry. 1. Confirm the node is `deprecated` with zero `affected_by` edges - (`homelab node relations`) — do not skip this even if the - deprecation runbook was followed recently; state can drift. -2. If it's an enrolled client: `homelab client remove ` — revokes - the age key, re-keys SOPS, removes the inventory entry. This is - already destructive-class and confirmed in the CLI. + (MCP `get_blast_radius`) — do not skip this even if the deprecation + runbook was followed recently; state can drift. +2. **If it's an enrolled client: no current tool for revoking its age key / + removing its Infisical identity.** The old `homelab client remove` + (age key revocation + SOPS re-key + inventory removal, all one + destructive-class CLI call) is retired along with the rest of that CLI + and hasn't been re-verified against the current enrollment + architecture (`POST /api/v1/clients/enroll` + Infisical machine + identities) — see the "Open questions" section in + [agent-enrollment.md](../../operations/agent-enrollment.md). Until + that's confirmed, treat key/identity revocation as a manual step: + at minimum remove the client's `age_pubkey` from any SOPS recipient + lists and rotate credentials whose ciphertext it already decrypted. 3. Remove any ingress route (Caddy config repo) and DNS record still pointing at it. 4. Verify backups of anything on it are retained per policy before the disk goes away (see `backs-up-to`). 5. Destroy the LXC/VM (`pct destroy` / `qm destroy`). -6. Move the `hosts.:` block (if any inventory remnant survives - `client remove`, e.g. infra-only LXCs with no age key) into - inventory.yaml's `archaeology:` section: `pve_id`, `destroyed` date, - `reason`. Add a row to `containers/index.md` "Recently destroyed" - table (kept for human-readable browsing alongside the structured - data). -7. `oikos/ledger.py append host: destroy destructive --result ok`. -8. Regenerate: `python3 mcp/build_host_files.py && python3 - inventory.yaml` — the node drops out of `inventory.yaml` and - appears in the topology doc's archaeology table. +6. Update the entity's `state` to `destroyed` in `seeds/inventory.yaml` + (or move it to an `archaeology:`-style section if the schema still has + one) — `pve_id`, `destroyed` date, `reason` — then `oikos seed` to + ingest. Add a row to `containers/index.md` "Recently destroyed" table + (kept for human-readable browsing alongside the structured data). +7. No manual ledger step — mutations through the API are recorded + automatically in the `audit_log` table (MCP `get_audit_trail`, + `get_change_history`). The old `oikos/ledger.py append` was retired + when this became automatic. -If the destroy fails partway (e.g. secrets revoked but pct destroy -errors), do not re-run step 2 — `client remove` is not idempotent -against a second revocation attempt on the issuance server. Finish the -remaining steps manually and note the partial state in an investigation. +If the destroy fails partway (e.g. secrets not fully revoked but pct +destroy errors), finish the remaining steps manually and note the +partial state in an investigation (MCP `upsert_knowledge`, +`kind: investigation`). diff --git a/.agents/skills/lifecycle-migrate-node/SKILL.md b/.agents/skills/lifecycle-migrate-node/SKILL.md index ac942ae..1d615ba 100644 --- a/.agents/skills/lifecycle-migrate-node/SKILL.md +++ b/.agents/skills/lifecycle-migrate-node/SKILL.md @@ -2,7 +2,7 @@ name: lifecycle-migrate-node risk_class: config_mutation inputs: [node_name, source_host, target_host] -verification: "homelab node relations (re-check blast radius); homelab service health for every hosted service" +verification: "MCP get_blast_radius (re-check blast radius); MCP get_service_status for every hosted service" docs_update_checklist: [doc_page_migration_note, inventory_host_and_lan_ip] transition: "active -> migrating -> active" --- @@ -10,12 +10,12 @@ transition: "active -> migrating -> active" # Lifecycle: migrate a node Modeled on the strong Phase 1+2 migration -([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)). +([archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)). Requires (ontology): preflight + backup-verified before migrating; post-verify + Caddy backends checked + mounts checked + docs updated before returning to `active`. -1. `homelab change preflight ` — capture +1. MCP `preflight` for every service the node hosts — capture current health as a baseline. 2. Verify backups are current for anything with data at rest on the node (see `backs-up-to` edges once populated). @@ -29,11 +29,12 @@ before returning to `active`. 6. Post-verify: re-run the Week-1 drift check by hand — confirm Caddy's backend IP for each affected service matches the new `lan_ip` (automatic in Week 3's drift detector), confirm mounts still resolve. -7. `homelab service health` for every service the node hosts. +7. MCP `get_service_status` for every service the node hosts. 8. Set `state: active`. Add a migration note to the node's doc page (old host/IP → new, date, phase reference) — this repo's convention for every past migration (see `archive/knowledge/containers/101-jellyfin.md`, `containers/129-house.md`). -Regenerate: `python3 mcp/build_host_files.py && python3 -inventory.yaml`. +Regenerate: `oikos seed` (re-ingests `seeds/inventory.yaml` into the DB — +the DB is authoritative at runtime, the YAML is the source of truth +on disk). diff --git a/.agents/skills/lifecycle-provision-node/SKILL.md b/.agents/skills/lifecycle-provision-node/SKILL.md index 46531a9..9b3dcba 100644 --- a/.agents/skills/lifecycle-provision-node/SKILL.md +++ b/.agents/skills/lifecycle-provision-node/SKILL.md @@ -21,10 +21,12 @@ chosen, doc page stub. `qm create`), choosing the storage pool deliberately — record it as the `storage:` field once populated (Week 1 schema; not yet backfilled for existing nodes). -2. Add the inventory entry: `homelab client add ` for anything that - will run the `homelab` CLI, or a direct `hosts.:` block with - `state: provisioning`, `kind`, `host`, `pve_id`, `lan_ip` for - infra-only LXCs that won't self-enroll. +2. Add the inventory entry: a `hosts.:` block in + `seeds/inventory.yaml` with `state: provisioning`, `kind`, `host`, + `pve_id`, `lan_ip`, then `oikos seed` to ingest it. For anything that + will self-enroll as a client afterward (see + [CLIENTS.md](../../../CLIENTS.md#enrollment)), the entity must exist in + `planned`/`provisioning` state before `bootstrap.sh` runs there. 3. Stub the doc page (`containers/-.md` or `vms/-.md`) — even a one-line "provisioning, see plan X" is enough to satisfy the transition requirement. diff --git a/.agents/skills/runbook-budget-from-csv/SKILL.md b/.agents/skills/runbook-budget-from-csv/SKILL.md index c0ccf8d..2bfcc69 100644 --- a/.agents/skills/runbook-budget-from-csv/SKILL.md +++ b/.agents/skills/runbook-budget-from-csv/SKILL.md @@ -18,7 +18,7 @@ summarised into targets and fixed costs. - `yuvomi-mcp` is running on LXC 129 and connected as an MCP server in Claude. - The CSV is an N26 export (columns: Booking Date, Value Date, Partner Name, Partner Iban, Type, Payment Reference, Account Name, Amount (EUR), …). -- API token: `homelab secret yuvomi-api-token` (decrypts on any enrolled client). +- API token: `yuvomi-api-token`, via Infisical (primary) or `oikos secret` (SOPS fallback). - Direct API base: `https://house.hubris.network/api/v1` --- diff --git a/.agents/skills/runbook-dpkg-interrupted/SKILL.md b/.agents/skills/runbook-dpkg-interrupted/SKILL.md index 22d0981..5d07f87 100644 --- a/.agents/skills/runbook-dpkg-interrupted/SKILL.md +++ b/.agents/skills/runbook-dpkg-interrupted/SKILL.md @@ -13,7 +13,8 @@ has packages that are **unpacked but not configured**. Symptoms: manually run 'dpkg --configure -a' to correct the problem.` - `dpkg --audit` lists packages with header `The following packages have been unpacked but not yet configured.` -- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host. +- `dpkg --audit` on the host directly shows unpacked-not-configured packages + (there's no fleet-wide audit tool anymore — check per-host). The system is still running the **old** binaries (still in memory), but the **new** binaries are unpacked and waiting for their postinst to run. Two @@ -33,19 +34,20 @@ config dirs, capabilities, etc.). The system might not come back up cleanly. ## Path A — target is still reachable over ssh (preferred) ``` -homelab ssh -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade' +ssh -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade' ``` -Or for an LXC by name: +Or for an LXC by name (via the MCP `run` tool, or directly on the Proxmox +host): ``` -homelab pct exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade' +pct exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade' ``` When that returns, confirm: ``` -homelab apt-audit --target +ssh -- dpkg --audit ``` Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally @@ -89,10 +91,11 @@ DEBIAN_FRONTEND=noninteractive dpkg --configure -a \ ## Prevention -The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect` -unit on the target, so it survives ssh teardown — the failure mode that put -hubris into this state in the first place is no longer reachable through the -standard tool. If you absolutely need to run apt manually over ssh, wrap it: +The old `homelab apt-upgrade` wrapper (retired along with the rest of the +`homelab` CLI) used to launch apt inside a `systemd-run --collect` unit on +the target so it survived ssh teardown — that's the failure mode that put +hubris into this state in the first place. There's no fleet-wide wrapper +anymore; if you run apt manually over ssh, wrap it yourself the same way: ``` ssh systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade' diff --git a/.agents/skills/service-health-check/SKILL.md b/.agents/skills/service-health-check/SKILL.md index a545b89..835b80b 100644 --- a/.agents/skills/service-health-check/SKILL.md +++ b/.agents/skills/service-health-check/SKILL.md @@ -2,7 +2,7 @@ name: service-health-check risk_class: read_only inputs: [service_name] -verification: "homelab service health" +verification: "MCP get_service_status" docs_update_checklist: [] --- @@ -10,15 +10,15 @@ docs_update_checklist: [] Goal: determine whether a service is actually healthy, without ad-hoc SSH. -1. `homelab service explain` — read the context card: backend, - blast radius, doc pointer, risk notes. -2. `homelab service health` — live health probe (HTTP code against - the service's `url`/`endpoint`). Once the Week-3 scheduler ships, this - reads a cached snapshot by default; pass `--live` to force a fresh probe. -3. If unhealthy, `homelab service log` (or MCP `tail_log`) for the - last 200 lines. -4. Cross-check blast radius: `homelab node relations` — is this - entity's own backend host healthy? A downstream failure (e.g. `strong` +1. MCP `explain` — read the context card: backend, blast radius, doc + pointer, risk notes. +2. MCP `get_service_status` — live health probe (HTTP code against the + service's `url`/`endpoint`); the scheduler also probes on its own + interval, so this may reflect a recent cached result, not necessarily + a fresh one. +3. If unhealthy, `tail_log` for the last 200 lines. +4. Cross-check blast radius: MCP `get_blast_radius` — is this entity's + own backend host healthy? A downstream failure (e.g. a Proxmox host down) will show up here before the service's own logs explain anything. 5. If the fix is a restart: classify first (`seeds/policy.yaml` — `service-restart` is `reversible_low` unless the service has a diff --git a/.gitignore b/.gitignore index e20dcf2..d4c9048 100644 --- a/.gitignore +++ b/.gitignore @@ -2,20 +2,12 @@ __pycache__/ *.pyc -# Regenerated every scheduler run; ephemeral health-probe cache. -oikos/state.json - # Compiled binaries (Go rewrite — bin/oikos, bin/nomos) bin/oikos bin/nomos -oikos/oikos oikos webhook -# Legacy Python oikos (superseded by cmd/oikos Go binary — Phase 1-6 rewrite). -# oikos/ kernel files are still imported by bin/homelab for operational CLI -# commands (ssh, pct, logs, restart, status, open, secret, client, sync, mcp). -# Remove oikos/* when bin/homelab is ported to Go. backups/ .env .infisical-credentials diff --git a/AGENTS.md b/AGENTS.md index 622a052..7bf7dae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,7 +34,8 @@ Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read: That file tells you your role, your peers, what's mounted, and what services you host. If it does not exist, this client was not enrolled — stop and tell -the operator to run `homelab client add ` from an existing client. +the operator; see [CLIENTS.md](CLIENTS.md#enrollment) for the enrollment flow +(the entity needs to exist in `planned`/`provisioning` state first). ## 2. The topology @@ -49,17 +50,24 @@ the operator to run `homelab client add ` from an existing client. ## 3. The MCP server The homelab exposes a Model Context Protocol server with structured tools. -Endpoint: `https://mcp.hubris.network/mcp`. +Endpoint: `https://mcp.hubris.network/mcp`. Every call needs +`Authorization: Bearer ` — the API has no unauthenticated path except +enrollment and `/healthz` (see "Authentication" below for where the token +comes from). -Available tools (21 total): +Available tools (33 total): Context — observe + orient: get_entity(slug), list_entities(type, limit, cursor), get_relations(entity), get_blast_radius(entity), search_knowledge(query) — ILIKE search over documents, investigations, runbooks in the knowledge_entities table + get_entity_knowledge(entity_slug) — every document, investigation, and + runbook linked to one entity, in one call get_patterns(status, entity_type, action) — learned action patterns get_skills(status) — available automation skills + http_get(url) — fetch a public page/raw file (e.g. researching how to + deploy something before provisioning it); HTTP/HTTPS only, ~16KB cap Management — live state: get_service_status(service_slug) — systemctl is-active on target host @@ -67,6 +75,8 @@ Available tools (21 total): list_lxcs() — all LXC containers with ID, host, IP, health get_lxc_state(lxc_slug) — pct status from Proxmox host ping_service(service_slug) — HTTP reachability from entity_status + list_my_secrets(caller_pubkey) — secrets accessible to this client by + age public key Oikos — decisions: explain(service_slug) — compact context card (type, state, health, relations) @@ -84,11 +94,27 @@ Available tools (21 total): get_trend(entity_id, days=7) — metric slope over time get_event_timeline(severity, entity_slug, limit) — recent events - Execution — the single mutation path: - request_execution(target, action, params) — policy-gated. - reversible_low (restart, reload, pct_exec, apt audit) runs immediately; - config_mutation (systemctl enable/disable, apt upgrade) queues for operator - approval via Matrix, then executes on ✅. + Knowledge — keep the graph current (none require approval; this updates + the knowledge graph, not live infrastructure): + upsert_knowledge(title, content) — record what you learned after solving + a non-obvious problem; the only way anything persists past a session + update_entity_attributes(slug, attributes) — merge a discovered fact + (IP, version, port, ...) into an entity so a future task doesn't + rediscover it from scratch + create_relationship(source, target, type) — record a discovered edge + (depends-on, hosts, routes-to, ...) between two entities + + Execution — mutating the live infrastructure: + run(target, command) — the general execution primitive. Run any shell + command against a host or LXC; every command is auto-classified — + read-only inspection runs immediately, anything state-changing needs + operator approval, and destructive patterns (rm -rf, dd, mkfs, + pct/qm destroy, DROP TABLE, reboot, curl-pipe-to-shell, ...) always + need approval regardless of what you declare. Prefer this over + request_execution for anything not already covered by its fixed enum. + request_execution(target, action, params) — the older, fixed-enum path + (restart, systemctl, pct_exec, apt_upgrade, pct_create). Still the + route for those specific actions; policy-gated the same way `run` is. get_execution_status(execution_id) — poll progress **When to prefer MCP over grepping the clone:** always for knowledge queries. @@ -97,7 +123,18 @@ the DB with entity links. `get_entity_knowledge("lxc:jellyfin")` returns documen runbooks, and investigations in one call. Grep the clone only when MCP is unreachable. -## 4. Knowledge conventions +## 4. Authentication + +Every API/MCP route requires `Authorization: Bearer ` except +`POST /api/v1/clients/enroll` and `/healthz`. Enrollment (see +[CLIENTS.md](CLIENTS.md#enrollment)) does not currently issue a per-client +API/MCP bearer token — there is one shared +secret (`OIKOS_MCP_BEARER_TOKEN`, validated in `internal/httpapi/server.go`'s +`combinedAuth`); get it from the operator until per-client token issuance +exists. The SPA has its own flow instead: a first-launch Config screen that +stores a token in `localStorage` (see `web/src/pages/Config.svelte`). + +## 5. Knowledge conventions All narrative knowledge (documents, investigations, runbooks) lives in the DB (`knowledge_entities` table) and is seeded from `seeds/knowledge.yaml`. Agents @@ -122,47 +159,52 @@ per the DB-as-source-of-truth plan. running state, update the DB *in the same session* via the API. The `oikos export` command regenerates `seeds/knowledge.yaml` for version control. -## 5. Acting on the homelab +## 6. Acting on the homelab - **Read state**: use MCP tools. Nomos (the AI agent) is the primary - operator interface — it has 21 MCP tools for observe/orient/decide/act. -- **Actions** (restart, logs, apt, pct exec): Nomos calls `request_execution` - via MCP. `reversible_low` actions execute immediately; `config_mutation` - and `destructive` actions are queued for operator approval via Matrix. -- **Secrets**: managed by Infisical (`oikos secret` subcommand for migration). - Never hardcode secrets — use env vars from `.env`. + operator interface — it has 33 MCP tools for observe/orient/decide/act + (§3). +- **Actions** (restart, logs, apt, pct exec, or anything else): Nomos calls + `run` (the general execution primitive) or `request_execution` (the older + fixed-enum path) via MCP. `reversible_low`/read-only actions execute + immediately; `config_mutation` and `destructive` actions are queued for + operator approval via Matrix or the control-room UI's Operations page. +- **Secrets**: managed by Infisical (`oikos secret` subcommand for + migration). Never hardcode secrets — use env vars from `.env`. - **Mutations** (restart, edit configs, etc.): classified against `seeds/policy.yaml`. `reversible_low` actions auto-execute; - `config_mutation`/`destructive` actions require approval. - a valid `--approval-id` from `homelab approval request` — see OIKOS.md. + `config_mutation`/`destructive` actions require approval — granted by + the operator via Matrix reply or the control-room UI, not a CLI flag. + See OIKOS.md. -## 6. Communication mode +## 7. Communication mode Read and apply `/opt/homelab-context/.agents/shared/caveman.md` (if present). It defines the lab's terse-communication standard — drop filler, keep substance, use fragments. -## 7. Auto-setup mechanism +## 8. Auto-setup mechanism The homelab-context repo ships tooling that gets automatically installed on every client after `git pull`. This is handled by `tools/post-pull.sh` (replaces the raw git pull in the sync timer) which runs any script matching -`tools/*.setup.sh` after pull. +`tools/setup-*.sh` after pull. Currently auto-setup: -- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm - package, wrapper scripts, and compact output templates for token-efficient - CLI output. Wrapper at `~/bin/caveman_wrapper.sh`. -- **Nomos agent persona** (`tools/setup-nomos-soul.sh`): Provisions - `~/.nomos/SOUL.md` from `NOMOS.md` on Nomos agents. This ensures every - Nomos agent follows the canonical homelab persona (token efficiency, source - of truth hierarchy). No-op on non-Nomos agents. +- **Host checks** (`tools/setup-checks.sh`): Deploys `checks/install.sh`'s + health-check scripts to `/opt/oikos/checks` on each host. The scheduler's + `ssh-script` check kind depends on these actually being there — 20 are + live in the DB as of 2026-07-12. -To add a new auto-setup, create `tools/.setup.sh` in the repo, +To add a new auto-setup, create `tools/setup-.sh` in the repo, commit and push. All enrolled clients pick it up within 5 minutes. -To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer. +To trigger sync manually: run `/opt/homelab/tools/context-poller.sh`, or +wait for the 5-min timer. (This mechanism — and the server-side +`tools_changed` detection behind it — only correctly recognized +`setup-*.sh` scripts as of 2026-07-12; before that it silently matched +nothing, so nothing auto-ran on any client via this path.) -## 8. When in doubt +## 9. When in doubt Use MCP tools: `search_knowledge ` for narrative context, `get_entity ` for structured data, `get_entity_knowledge ` for diff --git a/CLIENTS.md b/CLIENTS.md index 69c483b..baf5a3a 100644 --- a/CLIENTS.md +++ b/CLIENTS.md @@ -24,9 +24,22 @@ Docker stack on mac-mini and exposes an MCP server + REST API. | State snapshot (health, disk, drift) | MCP `get_state_snapshot` | | Secrets (Infisical) | REST API + `oikos secret` CLI | | Approval tokens | Matrix via notifier | +| Run a command on a host/LXC (policy-gated) | MCP `run` | +| Record a discovered fact/relationship | MCP `update_entity_attributes`, `create_relationship`, `upsert_knowledge` | -All MCP tools are read-only. Mutations use the `homelab` CLI with operator -approval. +Most MCP tools are read-only; a few mutate the knowledge graph (recording +what you learned) or the live infrastructure (`run`, `request_execution`), +gated by risk classification and — for `config_mutation`/`destructive` +actions — operator approval. See [AGENTS.md](AGENTS.md#3-the-mcp-server) for +the full tool catalog. + +## Authentication + +Every API/MCP call needs `Authorization: Bearer ` — there is no +unauthenticated path except `POST /api/v1/clients/enroll` and `/healthz`. +Enrollment (below) does not currently hand out a per-client bearer token; +get the shared `OIKOS_MCP_BEARER_TOKEN` from the operator until per-client +token issuance exists. ## Enrollment @@ -41,7 +54,6 @@ curl -fsSL https://git.hubris.network/dtoro/oikos/raw/main/bootstrap.sh | sudo b # Or with optional tooling: curl ... | sudo bash -s -- --with-mcp # wire Claude's MCP config -curl ... | sudo bash -s -- --with-nomos # install Goose + Nomos ``` This calls `POST /api/v1/clients/enroll` on the Oikos API, which: @@ -56,7 +68,7 @@ This calls `POST /api/v1/clients/enroll` on the Oikos API, which: ### What changes on your machine - `/opt/homelab/` — agent orientation files (CLIENTS.md, AGENTS.md, OIKOS.md) -- `/opt/homelab/tools/` — tooling scripts (caveman, nomos-soul) +- `/opt/homelab/tools/` — tooling scripts (checks) - `/etc/age/key.txt` — age private key for SOPS decryption (fallback) - `/etc/infisical/identity` — Infisical machine identity (primary secrets) - Context poller — launchd/systemd timer hits `GET /api/v1/clients/{slug}/context` every 5 minutes for agent file updates diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e573c53..65350c2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -35,6 +35,7 @@ cd web && OIKOS_API_TOKEN=dev-token npm run dev ``` cmd/oikos/ Single-binary entry point cmd/nomos/ Nomos MCP client gateway +cmd/webhook/ Gitea deploy-webhook receiver (push-to-deploy on mac-mini) internal/ All Go packages httpapi/ REST + MCP server (OpenAPI-generated) mcp/ MCP tool implementations @@ -55,10 +56,13 @@ migrations/ Forward-only SQL migrations (TimescaleDB) seeds/ Bootstrap YAML: ontology, inventory, policy, knowledge compose/ Dockerfiles + Caddy config scripts/ Deploy, watchdog, rollback +checks/ Host health-check scripts run over SSH by the scheduler +tools/ Client auto-setup scripts (checks) nomos/ Nomos config, persona, skills .agents/ Agent instruction files + skills plans/ Design documents docs/adr/ Architecture decision records +docs/operations/ Runbooks (rollback, etc.) ``` ## Commands @@ -78,6 +82,8 @@ docs/adr/ Architecture decision records | `make clean` | Remove binary + test cache | | `make ui` | Build the SPA (`web/dist/`) | | `make deploy-ui` | Build + deploy the SPA to the Caddy host | +| `make webhook` | Build `cmd/webhook` (deploy-webhook receiver) | +| `make tidy` | `go mod tidy` | ## Conventions diff --git a/README.md b/README.md index b4945b1..aa0e273 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ cd web && OIKOS_API_TOKEN=dev-token npm run dev # http://localhost:5173 | 5 — Secrets | ✅ | Infisical backend + SOPS fallback, rotation runbooks | | 6 — Deploy | ✅ | CI pipeline, cutover checklist, watchdog, rollback | -Full plan: [plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md). +Full plan: [plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/done/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md). ## Operations @@ -120,6 +120,7 @@ output). A native desktop wrapper is planned — see ``` cmd/oikos/ Go entry point — single binary cmd/nomos/ Nomos MCP client gateway +cmd/webhook/ Gitea deploy-webhook receiver (push-to-deploy on mac-mini) internal/ Go packages (httpapi, mcp, scheduler, actuator, learning, notifier, policy, secrets, db, config, ontology, domain, knowledge) @@ -129,11 +130,16 @@ migrations/ Forward-only SQL migrations (TimescaleDB) seeds/ Bootstrap YAML (ontology, inventory, policy, knowledge) compose/ Dockerfiles + Caddy config scripts/ Deploy, watchdog, verification, rollback +checks/ Host health-check scripts run over SSH by the scheduler +tools/ Client auto-setup scripts (checks) +ssh/ Deploy keys + authorized_keys management +vps/ Caddy/TURN config templates for the netbird VPS nomos/ Nomos config, persona, skills .agents/ Agent instruction files, shared conventions, skills archive/ Historical reference (legacy wiki, plans, SOPS backups) plans/ Design documents (active + done) docs/adr/ Architecture decision records +docs/operations/ Runbooks (rollback, etc.) ``` ## For agents diff --git a/bootstrap.sh b/bootstrap.sh index 14cd9d2..774b3a1 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -3,15 +3,14 @@ # # Thin client model (rev 2): no git clone, no sync timer. Fetches only the # agent orientation files (CLIENTS.md, AGENTS.md, OIKOS.md) and tooling -# (caveman, nomos-soul) from the raw Gitea URL. Enrolls via the Oikos API -# to receive an age keypair and Infisical machine identity. A lightweight -# context poller replaces the old 5-minute git pull. +# (checks) from the raw Gitea URL. Enrolls via the Oikos API to receive an +# age keypair and Infisical machine identity. A lightweight context poller +# replaces the old 5-minute git pull. # # Usage: # curl -fsSL https://git.hubris.network/dtoro/oikos/raw/main/bootstrap.sh \ # | sudo bash # curl ... | sudo bash -s -- --with-mcp # wire Claude's .mcp.json -# curl ... | sudo bash -s -- --with-nomos # install Goose + Nomos # curl ... | sudo bash -s -- --dry-run # show what would happen # # Prerequisites: @@ -28,16 +27,10 @@ REPO_RAW_URL="${HOMELAB_RAW_URL:-https://git.hubris.network/dtoro/oikos/raw/main OIKOS_API_URL="${HOMELAB_OIKOS_URL:-https://oikos.hubris.network/api/v1}" CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab}" MCP_URL="${HOMELAB_MCP_URL:-https://mcp.hubris.network/mcp}" -NOMOS_MCP_URI="${HOMELAB_NOMOS_MCP_URI:-https://mcp.hubris.network/mcp}" -NOMOS_MODEL="${HOMELAB_NOMOS_MODEL:-nousresearch/hermes-4-405b}" WITH_MCP=0 -WITH_NOMOS=0 DRY_RUN=0 -GITEA_TOKEN="${HOMELAB_GITEA_TOKEN:-}" -GITEA_USER="${HOMELAB_GITEA_USER:-dtoro}" - # ── helpers ────────────────────────────────────────────────────────── log() { echo "[oikos] $*"; } @@ -79,10 +72,7 @@ detect_mesh_ip() { while [ $# -gt 0 ]; do case "$1" in --with-mcp) WITH_MCP=1 ;; - --with-nomos) WITH_NOMOS=1 ;; --dry-run) DRY_RUN=1 ;; - --gitea-token) GITEA_TOKEN="$2"; shift ;; - --gitea-user) GITEA_USER="$2"; shift ;; *) die "unknown flag: $1" ;; esac shift @@ -148,7 +138,7 @@ done # ── fetch tools ────────────────────────────────────────────────────── log "fetching tools..." -for tool in setup-caveman.sh setup-nomos-soul.sh caveman.js caveman_wrapper.sh post-pull.sh; do +for tool in setup-checks.sh post-pull.sh; do url="$REPO_RAW_URL/tools/${tool}" dest="$CLONE_DIR/tools/${tool}" dry mkdir -p "$(dirname "$dest")" @@ -161,16 +151,6 @@ for tool in setup-caveman.sh setup-nomos-soul.sh caveman.js caveman_wrapper.sh p fi done -# ── fetch caveman templates ────────────────────────────────────────── -for tmpl in git_log.txt git_status.txt test_results.txt; do - url="$REPO_RAW_URL/tools/caveman/templates/${tmpl}" - dest="$CLONE_DIR/tools/caveman/templates/${tmpl}" - dry mkdir -p "$(dirname "$dest")" - if curl -fsSL --connect-timeout 10 "$url" -o "$dest.tmp" 2>/dev/null; then - mv "$dest.tmp" "$dest" - fi -done - # ── detect control-plane (use localhost if API is reachable directly) ─ if [ -z "${HOMELAB_OIKOS_URL:-}" ]; then if curl -s --connect-timeout 2 http://localhost:8090/api/v1/health >/dev/null 2>&1; then @@ -300,7 +280,7 @@ case "$OS" in esac # ── run auto-setup scripts ─────────────────────────────────────────── -for setup in "$CLONE_DIR"/tools/*.setup.sh; do +for setup in "$CLONE_DIR"/tools/setup-*.sh; do [ -f "$setup" ] || continue log "running setup: $(basename "$setup")" dry bash "$setup" @@ -319,17 +299,6 @@ if [ "$WITH_MCP" -eq 1 ]; then log " + MCP wired to $MCP_URL" fi -# ── --with-nomos: install Goose + Nomos wrapper ──────────────────── -if [ "$WITH_NOMOS" -eq 1 ]; then - log "installing Nomos agent..." - GOOSE_URL="https://github.com/block/goose/releases/latest/download/goose-${OS}-${ARCH:-amd64}" - if [ "$OS" = Darwin ]; then GOOSE_URL="https://github.com/block/goose/releases/latest/download/goose-darwin-${ARCH:-arm64}"; fi - dry curl -fsSL "$GOOSE_URL" -o /usr/local/bin/goose 2>/dev/null && chmod +x /usr/local/bin/goose || warn "goose not installed" - # Drop Nomos persona - cp "$CLONE_DIR/NOMOS.md" "$CLONE_DIR/.agents/NOMOS.md" 2>/dev/null || true - log " + Nomos agent installed" -fi - # ── netbird SSH JWT cache ──────────────────────────────────────────── if command -v netbird >/dev/null 2>&1; then dry netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400 2>/dev/null || true diff --git a/docs/adr/0013-signal-triggers.md b/docs/adr/0013-signal-triggers.md index b42eafc..b9a70f5 100644 --- a/docs/adr/0013-signal-triggers.md +++ b/docs/adr/0013-signal-triggers.md @@ -1,4 +1,7 @@ -# Signal Trigger Architecture +# ADR 0013 — Signal trigger architecture + +**Status:** Accepted +**Date:** 2026-07-08 ## Overview diff --git a/docs/adr/0014-entity-model.md b/docs/adr/0014-entity-model.md index 91866a4..4abab2f 100644 --- a/docs/adr/0014-entity-model.md +++ b/docs/adr/0014-entity-model.md @@ -1,6 +1,6 @@ -# Oikos Entity Model — Types, Relationships & Interactions +# ADR 0014 — Entity model: types, relationships & interactions -**Status:** Adopted +**Status:** Accepted **Date:** 2026-07-08 **Scope:** Full inventory of every entity type, relationship, state machine, and cognition pipeline — with clear markers for what is **code-real** vs **schema-only**. diff --git a/docs/adr/0015-api-bearer-auth-client-server-split.md b/docs/adr/0015-api-bearer-auth-client-server-split.md new file mode 100644 index 0000000..8369907 --- /dev/null +++ b/docs/adr/0015-api-bearer-auth-client-server-split.md @@ -0,0 +1,62 @@ +# ADR 0015 — Bearer-token auth for every route + client/server split + +Status: accepted (2026-07-12) · Plan: plans/2026-07-12-wails-desktop-app.md, Phase 0 + +## Context +The control-room SPA was embedded in the `oikos` binary (`go:embed`, +ADR 0001) and served at `/ui/*`. `combinedAuth` (`internal/httpapi/server.go`) +opened a dev-open bypass — no credential required at all — whenever +`OIKOS_ENV=dev` and no static token/OIDC issuer was configured. That was +true not just in local dev but in the actual mac-mini production deploy: +`docker-compose.yml`'s `api` service hardcoded `OIKOS_ENV: dev` with no +token set, so every route (`/api/v1/*`, `/mcp`, and an `/agent` reverse-proxy +mount to nomos that had never been wrapped in `combinedAuth` at all) was +reachable unauthenticated from anywhere on the mesh/LAN. A planned Wails +desktop client and any future non-browser client can't rely on same-origin +requests or a dev-open bypass; they need the SPA to be a standalone, +CORS-capable client that authenticates over HTTP like any other caller. + +## Decision +- Delete the SPA embed (`web/embed.go`, the `/ui/*` routes). `web/` is a + standalone static build, deployed separately (`make ui` / `make + deploy-ui`), served at `/` by Caddy with SPA fallback. +- Remove the dev-open bypass entirely. Every route requires a valid + static bearer token (`OIKOS_API_TOKEN` / `OIKOS_MCP_BEARER_TOKEN`) or an + OIDC JWT, with two narrow exceptions: `/healthz` (liveness) and + `POST /api/v1/clients/enroll` (IP-gated in the handler instead). + `GET /api/v1/events/stream` additionally accepts the token as a + `?token=` query param, since `EventSource` can't set custom headers. +- Add CORS (`github.com/go-chi/cors`, `OIKOS_CORS_ORIGIN`, default `*`) so a + cross-origin SPA (Vite dev server, a future Wails webview) can reach the + API. No `AllowCredentials` — auth is a header, not a cookie, so + credentialed CORS mode isn't needed and the two don't combine safely with + a wildcard origin. +- Wrap the previously-unauthenticated `/agent` proxy mount in the same + `combinedAuth` middleware as every other route. +- `cmd/nomos` becomes an authenticated client of `api`: it now sends + `Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN` on its own outbound calls + (MCP + the chat-assent approval-decision endpoint), which it never did + before — dev-open covered for it until now. +- The SPA gets a runtime config module (`web/src/lib/config.ts`) and a + first-launch `Config.svelte` screen: server URL + token, stored in + `localStorage`, injected into every `fetch()` via a shared + `fetchWithAuth` wrapper. Resolved fresh per request (not cached at + module-load time), so the same build works same-origin or cross-origin + without a rebuild. + +## Consequences +- Closing dev-open was a live security fix, not just future-proofing — + verified post-deploy that unauthenticated requests to production now 401. +- Nomos's *own* HTTP gateway (`cmd/nomos`, port 8092) still has no auth of + its own — out of scope here, tracked separately + (plans/2026-07-11-nomos-agent-code-review.md, finding C1). +- Production Caddy (`dtoro/caddy-conf`, not this repo) does not yet expose + `oikos.hubris.network` at all, so the interaction between Authentik + forward-auth and bearer-token clients (a non-browser client can't + complete a browser SSO redirect) is unresolved — needs an `@enroll`-style + bypass for `/api/v1/*`/`/mcp`/`/agent/*` before public exposure. This + repo's `compose/caddy/Caddyfile.oikos` (a reference copy, not deployed + from here) has the bypass; the real config does not yet. +- There is one shared bearer secret for all agents/clients, not per-client + tokens — acceptable for the current fleet size, revisit if per-client + revocation becomes necessary. diff --git a/docs/adr/README.md b/docs/adr/README.md index 3eb060f..69e9290 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -5,7 +5,7 @@ after acceptance — superseding decisions get a new ADR that links back. Statuses: proposed | accepted | superseded-by-NNNN. | ADR | Title | -|---|---|---| +|---|---| | [0001](0001-go-single-binary.md) | Go with single-binary role packaging | | [0002](0002-postgres-timescale-only-datastore.md) | PostgreSQL + TimescaleDB as the only datastore | | [0003](0003-db-native-ontology-yaml-seeds.md) | DB-native ontology with YAML seed manifests | @@ -16,7 +16,8 @@ Statuses: proposed | accepted | superseded-by-NNNN. | [0008](0008-forward-only-migrations.md) | Forward-only migrations | | [0009](0009-sse-over-websocket.md) | SSE over WebSocket for the event stream | | [0010](0010-infisical-with-sops-fallback.md) | Infisical secrets with SOPS DR fallback | -| [0011](0011-client-lifecycle-flows.md) | Client lifecycle flows — enrollment, bootstrap, sync | -| [0012](0012-hermes-oikos-interactions.md) | Hermes–Oikos interactions — agent/OS contract | -| [0013](0013-signal-triggers.md) | Signal triggers — host health checks via scheduler | +| [0011](0011-client-lifecycle-flows.md) | Client lifecycle sequence diagrams | +| [0012](0012-hermes-oikos-interactions.md) | Hermes/Oikos interaction architecture | +| [0013](0013-signal-triggers.md) | Signal trigger architecture | | [0014](0014-entity-model.md) | Entity model — types, relationships, state machines, OODA loop | +| [0015](0015-api-bearer-auth-client-server-split.md) | Bearer-token auth for every route + client/server split | diff --git a/internal/httpapi/impl.go b/internal/httpapi/impl.go index d96ed11..b07fcb1 100644 --- a/internal/httpapi/impl.go +++ b/internal/httpapi/impl.go @@ -1290,7 +1290,10 @@ func (s *Server) GetClientContext(ctx context.Context, req gen.GetClientContextR for rows.Next() { var p string if scanErr := rows.Scan(&p); scanErr == nil { - if strings.HasPrefix(p, "tools/") && strings.HasSuffix(p, ".setup.sh") { + // Matches tools/setup-*.sh (the auto-setup convention — + // see tools/post-pull.sh). Was tools/*.setup.sh until + // 2026-07-12, which never matched any real filename. + if strings.HasPrefix(p, "tools/setup-") && strings.HasSuffix(p, ".sh") { toolsChanged = append(toolsChanged, p) } else if p == ".sops.yaml" { sopsChanged = true diff --git a/oikos/gen-topology.py b/oikos/gen-topology.py deleted file mode 100644 index ca68d46..0000000 --- a/oikos/gen-topology.py +++ /dev/null @@ -1,237 +0,0 @@ -#!/usr/bin/env python3 -""" -Generate knowledge/wiki/infrastructure/topology.md (Mermaid views) and per-entity context -cards from inventory.yaml. - -Views: - 1. Compute & ingress — hypervisors → guests → services → public URLs - 2. Storage — mounts and pools per guest - -Context cards (oikos/cards/.md): one compact (~30-line) file per -host and service — identity, ontology edges, safe actions + risk class, -doc pointer, recent ledger history. This is the token-efficiency layer: -an agent orienting on an entity reads one card instead of several -search_docs/get_page round-trips. - -Run from the repo root: - python3 oikos/gen-topology.py # writes topology.md + cards/ - python3 oikos/gen-topology.py --check # exit 1 if output would change - -Wired into the same regeneration path as mcp/build_host_files.py so the -diagrams and cards never drift from inventory. Edges follow -oikos/ontology.yaml (hosts, provides, routes-to, mounts, stores-on). -""" - -from __future__ import annotations - -import argparse -import sys -from pathlib import Path - -try: - import yaml -except ImportError: # pragma: no cover - print("PyYAML is required: pip install pyyaml", file=sys.stderr) - sys.exit(2) - -REPO = Path(__file__).resolve().parent.parent -sys.path.insert(0, str(REPO)) -from oikos import gen_topology_lib as lib # noqa: E402 -from oikos import ledger as oikos_ledger # noqa: E402 -from oikos import policy as oikos_policy # noqa: E402 -from oikos import relations as oikos_relations # noqa: E402 - -INVENTORY = REPO / "inventory.yaml" -OUTPUT = REPO / "knowledge" / "wiki" / "infrastructure" / "topology.md" -CARDS_DIR = REPO / "oikos" / "cards" - -BANNER = ( - "\n" - "\n" -) - -# View/graph logic lives in oikos/gen_topology_lib.py (importable — this -# file's hyphenated name can't be). Re-exported here so existing call -# sites in this module don't need a rename. -node_id = lib.node_id -guest_label = lib.guest_label -compute_view = lib.compute_view -storage_view = lib.storage_view -archaeology_table = lib.archaeology_table - - -def _host_card(name: str, entry: dict, inv: dict) -> str: - lines = [f"# {name} (host:{name})\n"] - tag = "LXC" if entry.get("kind") == "lxc" else "VM" if entry.get("kind") == "vm" else entry.get("kind", "") - pve = entry.get("pve_id") - lines.append(f"- kind: {entry.get('kind', '?')}" + (f" ({tag} {pve})" if pve else "")) - lines.append(f"- state: {entry.get('state', 'active')}") - if entry.get("host"): - lines.append(f"- runs-on: host:{entry['host']}") - if entry.get("role"): - lines.append(f"- role: {entry['role']}") - addr = entry.get("lan_ip", "") - mesh = entry.get("mesh", {}) - mesh_bits = [] - for m, v in mesh.items(): - if isinstance(v, dict) and (v.get("ip") or v.get("fqdn")): - mesh_bits.append(f"{m}:{v.get('fqdn') or v.get('ip')}") - if addr or mesh_bits: - lines.append(f"- address: {addr}" + (f" (mesh: {', '.join(mesh_bits)})" if mesh_bits else "")) - if entry.get("mounts"): - lines.append(f"- mounts: {', '.join(entry['mounts'])}") - doc = None - if entry.get("kind") == "lxc" and pve: - cand = REPO / "knowledge" / "wiki" / "containers" / f"{pve}-{name}.md" - if cand.exists(): - doc = str(cand.relative_to(REPO)) - elif entry.get("kind") == "vm" and pve: - cand = REPO / "knowledge" / "wiki" / "vms" / f"{pve}-{name}.md" - if cand.exists(): - doc = str(cand.relative_to(REPO)) - elif entry.get("kind") == "proxmox-host": - cand = REPO / "knowledge" / "wiki" / "hosts" / f"{name}.md" - if cand.exists(): - doc = str(cand.relative_to(REPO)) - if doc: - lines.append(f"- doc: {doc}") - if entry.get("age_pubkey"): - lines.append("- secrets: enrolled (age key present)") - - rel = oikos_relations.relations(f"host:{name}", inv) - lines.append("\n## Blast radius") - lines.append(f"- impacts: {', '.join(rel['impacts']) or '(none)'}") - lines.append(f"- affected by: {', '.join(rel['affected_by']) or '(none)'}") - if rel["blast_radius"]: - lines.append(f"- full blast radius: {', '.join(rel['blast_radius'])}") - - lines.append("\n## Safe actions") - lines.append("- see the services this host runs for action-level risk classes") - - hist = oikos_ledger.history(f"host:{name}", limit=5) - lines.append("\n## Recent changes") - if hist: - for h in hist: - lines.append(f"- {h.get('ts', '?')} {h.get('action', '?')} ({h.get('risk', '?')}) — {h.get('result', '?')}") - else: - lines.append("- (none yet)") - return "\n".join(lines) + "\n" - - -def _service_card(name: str, entry: dict, inv: dict) -> str: - lines = [f"# {name} (service:{name})\n"] - if entry.get("backend"): - lines.append(f"- backend: host:{entry['backend']}") - url = entry.get("url") or entry.get("endpoint") - if url: - lines.append(f"- url: {url}") - if entry.get("doc_page"): - lines.append(f"- doc: {entry['doc_page']}") - if entry.get("config_repo"): - lines.append(f"- config repo: {entry['config_repo']}") - if entry.get("risk_notes"): - lines.append(f"- risk notes: {entry['risk_notes']}") - - rel = oikos_relations.relations(f"service:{name}", inv) - lines.append("\n## Blast radius") - lines.append(f"- impacts: {', '.join(rel['impacts']) or '(none)'}") - lines.append(f"- affected by: {', '.join(rel['affected_by']) or '(none)'}") - - lines.append("\n## Safe actions") - for a in oikos_policy.safe_actions_for_service(name, entry): - lines.append(f"- {a['action']} — {a['risk']} (approval: {a['approval']})") - - hist = oikos_ledger.history(f"service:{name}", limit=5) - lines.append("\n## Recent changes") - if hist: - for h in hist: - lines.append(f"- {h.get('ts', '?')} {h.get('action', '?')} ({h.get('risk', '?')}) — {h.get('result', '?')}") - else: - lines.append("- (none yet)") - return "\n".join(lines) + "\n" - - -def generate_cards(inv: dict) -> dict[Path, str]: - desired: dict[Path, str] = {} - for name, entry in inv.get("hosts", {}).items(): - desired[CARDS_DIR / f"host-{name}.md"] = _host_card(name, entry, inv) - for name, entry in inv.get("services", {}).items(): - if isinstance(entry, dict): - desired[CARDS_DIR / f"service-{name}.md"] = _service_card(name, entry, inv) - return desired - - -def write_cards(inv: dict, check: bool = False) -> int: - CARDS_DIR.mkdir(parents=True, exist_ok=True) - desired = generate_cards(inv) - diff_count = 0 - for path, content in desired.items(): - existing = path.read_text() if path.exists() else "" - if existing != content: - diff_count += 1 - if not check: - path.write_text(content) - for existing_path in CARDS_DIR.glob("*.md"): - if existing_path not in desired: - diff_count += 1 - if not check: - existing_path.unlink() - return diff_count - - -def render(inv: dict) -> str: - hosts = inv.get("hosts", {}) - services = inv.get("services", {}) - counts = ( - f"{sum(1 for e in hosts.values() if e.get('kind') == 'proxmox-host')} hypervisors, " - f"{sum(1 for e in hosts.values() if e.get('kind') == 'lxc')} LXCs, " - f"{sum(1 for e in hosts.values() if e.get('kind') == 'vm')} VMs, " - f"{sum(1 for e in hosts.values() if e.get('kind') == 'workstation')} workstations, " - f"{len(services)} services" - ) - parts = [ - BANNER, - "# Topology (generated)\n", - f"Source: [inventory.yaml](../../../inventory.yaml) — {counts}.", - "Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). " - "Operating model: [OIKOS.md](../../../.agents/OIKOS.md).\n", - "## Compute & ingress\n", - "\n".join(compute_view(inv)) + "\n", - "## Storage (mounts)\n", - "\n".join(storage_view(inv)) + "\n", - ] - arch = archaeology_table(inv) - if arch: - parts += ["## Archaeology (destroyed nodes)\n", "\n".join(arch) + "\n"] - return "\n".join(parts) - - -def main() -> int: - parser = argparse.ArgumentParser() - parser.add_argument("--check", action="store_true", - help="exit 1 if output would change (don't write)") - args = parser.parse_args() - - inv = yaml.safe_load(INVENTORY.read_text()) - content = render(inv) - existing = OUTPUT.read_text() if OUTPUT.exists() else "" - topology_changed = existing != content - card_diffs = write_cards(inv, check=args.check) - - if args.check: - if topology_changed: - print(f"{OUTPUT.relative_to(REPO)} would change", file=sys.stderr) - if card_diffs: - print(f"{card_diffs} card(s) in oikos/cards/ would change", file=sys.stderr) - return 1 if (topology_changed or card_diffs) else 0 - - if topology_changed: - OUTPUT.write_text(content) - print(f"wrote {OUTPUT.relative_to(REPO)}") - if card_diffs: - print(f"wrote/updated {card_diffs} card(s) in oikos/cards/") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/oikos/gen_topology_lib.py b/oikos/gen_topology_lib.py deleted file mode 100644 index 13c8548..0000000 --- a/oikos/gen_topology_lib.py +++ /dev/null @@ -1,112 +0,0 @@ -"""oikos/gen_topology_lib.py — shared Mermaid-view logic. - -Split out of oikos/gen-topology.py so it's importable (a hyphenated -filename can't be `import`ed as a module). oikos/gen-topology.py is the -CLI entrypoint that writes knowledge/wiki/infrastructure/topology.md + oikos/cards/; -oikos/console/app.py imports this module directly to render the live -/graph page without shelling out. -""" - -from __future__ import annotations - -from pathlib import Path - -import yaml - -REPO = Path(__file__).resolve().parent.parent -INVENTORY = REPO / "inventory.yaml" - - -def load_inventory() -> dict: - return yaml.safe_load(INVENTORY.read_text()) - - -def node_id(name: str) -> str: - """Mermaid-safe node id.""" - return name.replace("-", "_").replace(".", "_").replace("/", "_").strip("_") - - -def guest_label(name: str, entry: dict) -> str: - pve = entry.get("pve_id") - role = entry.get("role", "") - tag = f"LXC {pve}" if entry.get("kind") == "lxc" and pve else \ - f"VM {pve}" if entry.get("kind") == "vm" and pve else entry.get("kind", "") - ip = entry.get("lan_ip", "") - parts = [name, tag, role, ip] - return "
".join(str(p) for p in parts if p) - - -def compute_view(inv: dict) -> list[str]: - hosts = inv.get("hosts", {}) - services = inv.get("services", {}) - lines = ["```mermaid", "flowchart LR"] - - hypervisors = {n: e for n, e in hosts.items() if e.get("kind") == "proxmox-host"} - guests = {n: e for n, e in hosts.items() if e.get("kind") in ("lxc", "vm")} - others = {n: e for n, e in hosts.items() - if e.get("kind") in ("workstation", "external")} - - for hv in hypervisors: - lines.append(f' subgraph {node_id(hv)}_sub["{hv} (Proxmox)"]') - for g, e in guests.items(): - if e.get("host") == hv: - lines.append(f' {node_id(g)}["{guest_label(g, e)}"]') - lines.append(" end") - - # guests without a parent hypervisor recorded (e.g. rclone) - for g, e in guests.items(): - if e.get("host") not in hypervisors: - lines.append(f' {node_id(g)}["{guest_label(g, e)}"]') - - for n, e in others.items(): - shape = "([{}])" if e.get("kind") == "workstation" else "[[{}]]" - lines.append(f' {node_id(n)}{shape.format(guest_label(n, e))}') - - # ingress: public URL -> backend (routes-to) - for svc, e in sorted(services.items()): - if not isinstance(e, dict): - continue - backend = e.get("backend") - url = e.get("url") or ( - f'https://{e["public_host"]}' if e.get("public_host") else None) - if backend and url and backend in hosts: - host = url.removeprefix("https://").removeprefix("http://") - # hypervisors are rendered as subgraphs; point edges at the subgraph id - target = node_id(backend) + ("_sub" if backend in hypervisors else "") - lines.append( - f' {node_id("url_" + svc)}(["{host}"]) -->|routes-to| {target}') - - lines.append("```") - return lines - - -def storage_view(inv: dict) -> list[str]: - hosts = inv.get("hosts", {}) - lines = ["```mermaid", "flowchart LR"] - pools: set[str] = set() - edges: list[str] = [] - - for name, e in hosts.items(): - for mount in e.get("mounts", []): - pools.add(mount) - edges.append(f' {node_id(name)}["{name}"] -->|mounts| {node_id(mount)}') - - for pool in sorted(pools): - lines.append(f' {node_id(pool)}[("{pool}")]') - lines.extend(sorted(set(edges))) - lines.append("```") - return lines - - -def archaeology_table(inv: dict) -> list[str]: - arch = inv.get("archaeology", {}) - if not arch: - return [] - lines = ["| Node | ID | Destroyed | Reason |", "|---|---|---|---|"] - entries = sorted(arch.items(), key=lambda kv: str(kv[1].get("destroyed", "")), - reverse=True) - for name, e in entries: - lines.append( - f'| {name} | {e.get("pve_id", "")} | {e.get("destroyed", "")} ' - f'| {e.get("reason", "")} |') - return lines diff --git a/plans/2026-07-08-liveness-drift-and-ux-cohesion.md b/plans/2026-07-08-liveness-drift-and-ux-cohesion.md index 9f05ae5..7f511ec 100644 --- a/plans/2026-07-08-liveness-drift-and-ux-cohesion.md +++ b/plans/2026-07-08-liveness-drift-and-ux-cohesion.md @@ -1,8 +1,11 @@ # 2026-07-08 — Liveness, drift, and UX cohesion -**Status:** In Progress — Phases 1–4 code complete; not yet deployed. Phase 5 deferred. -(Audited 2026-07-11 — still accurate; prompt caching within Phase 4 also -confirmed not implemented.) +**Status:** In Progress — Phases 1–4 code complete and now deployed +(re-verified 2026-07-12: mac-mini was redeployed from `main` that day for +unrelated auth work — plans/2026-07-12-wails-desktop-app.md — which carried +every commit up to that point, including this plan's, so "not yet deployed" +below is stale). Phase 5 deferred. (Audited 2026-07-11 — still accurate; +prompt caching within Phase 4 also confirmed not implemented.) - **Phase 1 (drift/staleness):** done. Health/metrics/events misattribution fix, staleness sweep, `/entities` health+freshness, dashboard/fleet-health @@ -26,20 +29,19 @@ confirmed not implemented.) - **Phase 4 (agent efficiency):** core piece done — prior turns' tool calls/results are now replayed into the conversation (previously dropped entirely), and a compact live fleet-health snapshot is injected into the - system prompt each turn so the agent starts oriented. Prompt caching and - reconsidering the default model are **not done** (lower priority, no - measured regression without them). + system prompt each turn so the agent starts oriented. Prompt caching is + **not done** (lower priority, no measured regression without it). + Reconsidering the default model — done, but not by this plan: switched to + `deepseek/deepseek-v4-pro` on 2026-07-10 (`cmd/nomos/agent.go:70`) for + reliability, per that commit's own comment ("the flash tier over-narrates, + occasionally emits canned refusals, and is unreliable at multi-step tool + use"). - **Phase 5 (CRUD):** `PatchEntity` and a full `/checks` CRUD API (list/create/patch, including enable/disable) already existed server-side; the new Monitoring card's toggle uses `PatchCheck`. **Not done**: a "run check now" endpoint (no scheduler on-demand entrypoint exists yet), relationship editing, and an entity attribute editor UI. -**Not yet deployed** — the live `oikos-api`/`oikos-scheduler`/nomos -containers still run the pre-fix binaries; rebuilding and restarting them -needs an explicit go-ahead since it touches the running homelab control -plane. - Addresses five felt problems with the current system: (1) the agent reports stale machine state as if it were fresh, (2) sessions can't be opened and feel disconnected from chat, (3) the Nomos agent re-derives state every turn and diff --git a/plans/2026-07-08-oikos-gaps-and-improvements.md b/plans/2026-07-08-oikos-gaps-and-improvements.md index f5917b2..33d3706 100644 --- a/plans/2026-07-08-oikos-gaps-and-improvements.md +++ b/plans/2026-07-08-oikos-gaps-and-improvements.md @@ -1,14 +1,24 @@ # 2026-07-08 — Oikos gaps, broken things, and improvements -**Status:** In Progress — audited 2026-07-11. Done: A1 (approval FK bug), -A3 (Hermes→Nomos help text), D1 (`upsert_knowledge`), D4-partial (general -`run` tool). Still open: A2 (notifier flooding/dedup), A4 (`resolveHost` -dead code), A5 (`queryRows` stringly-typed columns), A6 (stale -`get_state_snapshot` description), B1-B5 (enrollment auth, fake Infisical -creds, `/query` mesh-only auth unenforced, insecure host key checking, -optional `caller_pubkey`), D2/D3 (no `get_approval_status`/ -`list_pending_approvals`/signal ack-resolve-mute tools), E (README tool -count, Caddyfile placeholders, NOMOS.md duplicate line). +**Status:** In Progress — audited 2026-07-11, re-audited 2026-07-12 for +drift from the `cmd/hermes`→`cmd/nomos` rename and later fixes. Done: A1 +(approval FK bug), A3 (Hermes→Nomos help text), **Section C** (toy NLU / +silent-wrong-answer fallback — nomos now calls real `listTools()` and +routes unmatched queries to `/chat` instead of guessing, per +`cmd/nomos/main.go:444-465`), **D.5** (SOUL.md/actuator architecture +mismatch — `nomos/SOUL.md:21-22,40` now accurately documents SSH via the +`run` tool), D1 (`upsert_knowledge`), D4-partial (general `run` tool). +Still open: A2 (notifier flooding/dedup), A4 (`resolveHost` dead code), A5 +(`queryRows` stringly-typed columns), A6 (stale `get_state_snapshot` +description), B1-B5 (enrollment auth, fake Infisical creds, `/query` +mesh-only auth unenforced, insecure host key checking, optional +`caller_pubkey`), D2/D3 (no `get_approval_status`/`list_pending_approvals`/ +signal ack-resolve-mute tools), E-partial (Caddyfile placeholders still +present; tool count now 33, documented in AGENTS.md as of 2026-07-12). +2026-07-12 re-audit also refreshed every `cmd/hermes`→`cmd/nomos` and +`internal/mcp/server.go` line-number citation below (the file grew from 28 +to 33 registered tools since 2026-07-11) — content/status of each finding +unchanged, only citations moved. ## Goal @@ -78,22 +88,24 @@ named `tools/list`, which doesn't exist. The correct `listTools()` helper ### A4. `resolveHost` never returns a per-entity SSH user -`internal/mcp/server.go:943` — the named return `sshUser` is always `""`; the -per-entity user branch is dead and everything relies on `sshExec`'s global -default fallback. **Fix:** read the SSH user from entity attributes or delete -the dead return to make the behavior honest. +`internal/mcp/server.go:1222` (was :943 — line moved) — the named return +`sshUser` is always `""`; the per-entity user branch is dead and everything +relies on `sshExec`'s global default fallback. **Fix:** read the SSH user +from entity attributes or delete the dead return to make the behavior +honest. ### A5. `queryRows` stringifies every column -`internal/mcp/server.go:861` renders all values via `fmt.Sprintf("%v", ...)`, -so numbers, bools, timestamps, and JSON all reach agents as strings. -**Fix:** type-preserving serialization (pass through pgx-native values into -`json.Marshal`) — improves every read tool at once. +`internal/mcp/server.go:1090` (was :861 — line moved) renders all values via +`fmt.Sprintf("%v", ...)`, so numbers, bools, timestamps, and JSON all reach +agents as strings. **Fix:** type-preserving serialization (pass through +pgx-native values into `json.Marshal`) — improves every read tool at once. ### A6. `get_state_snapshot` description is stale -`internal/mcp/server.go:689` still advertises "disk, drift count" — columns -removed in commit 3ea43ad. **Fix:** update the description. +`internal/mcp/server.go:863` (was :689 — line moved) still advertises "disk, +drift count" — columns removed in commit 3ea43ad. **Fix:** update the +description. --- @@ -101,12 +113,13 @@ removed in commit 3ea43ad. **Fix:** update the description. ### B1. Enrollment is unauthenticated, with a false comment -`internal/httpapi/server.go:97` says "unauthenticated (IP-gated in handler)" -but `EnrollClient` (`internal/httpapi/impl.go:1099`) performs no IP check at -all — the only gate is the target entity being in state -`planned`/`provisioning`. Caddy's `@enroll` matcher bypasses Authentik. -Anyone reaching `oikos.hubris.network` who knows (or guesses) a planned slug -receives that node's **age private key** in the HTTP response body. +`internal/httpapi/server.go:111` (was :97) says "unauthenticated (IP-gated +in handler)" but `EnrollClient` (`internal/httpapi/impl.go:1166`, was +:1099) performs no IP check at all — the only gate is the target entity +being in state `planned`/`provisioning`. Caddy's `@enroll` matcher bypasses +Authentik. Anyone reaching `oikos.hubris.network` who knows (or guesses) a +planned slug receives that node's **age private key** in the HTTP response +body. Still open — line numbers only, substance unchanged. **Fix:** enforce a real gate (mesh-CIDR check, one-time enrollment token minted when the entity is created, or both), and stop returning the age @@ -114,34 +127,49 @@ private key in the response — have the client fetch it from the secret store. ### B2. Fake Infisical credentials returned to enrollees -`internal/httpapi/impl.go:1191-1192` returns `"inf_client_"+uuid` / -`"inf_secret_"+uuid` — random strings wired to nothing. Enrolled clients hold -credentials that authenticate against nothing. +`internal/httpapi/impl.go:1260-1261` (was :1191-1192) returns +`"inf_client_"+uuid` / `"inf_secret_"+uuid` — random strings wired to +nothing. Enrolled clients hold credentials that authenticate against +nothing. Still open — line numbers only, substance unchanged. **Fix:** implement `CreateMachineIdentity` in `internal/secrets/infisical.go`, or return no credentials and document the manual step. -### B3. Hermes `/query` has no auth +### B3. Nomos's `/query` has no auth -`hermes/config.yaml:9` sets `mesh_only: true` but `cmd/hermes/main.go` never -reads or enforces it — it serves any caller on :8092, who can invoke -`request_execution`. **Fix:** enforce mesh-CIDR (or bearer token) in the -handler; fail closed. +`nomos/config.yaml:9` (was `hermes/config.yaml:9`) sets `mesh_only: true` +but `cmd/nomos/main.go` (was `cmd/hermes/main.go`) never reads or enforces +it — it serves any caller on :8092, who can invoke `request_execution`. +Still open, now also tracked as C1 in +[2026-07-11-nomos-agent-code-review.md](2026-07-11-nomos-agent-code-review.md), +deferred by the operator. **Fix:** enforce mesh-CIDR (or bearer token) in +the handler; fail closed. ### B4. SSH host keys not verified -`ssh.InsecureIgnoreHostKey()` at `internal/mcp/server.go:920`. +`ssh.InsecureIgnoreHostKey()` at `internal/mcp/server.go:1155` (was :920). +Still open — line number only, substance unchanged. **Fix:** known_hosts pinning (keys are already inventory-managed per node). ### B5. `list_my_secrets` enumerates all node pubkeys -Without `caller_pubkey`, `internal/mcp/server.go:709-720` returns every entity -that has an `age_pubkey`; nothing ties the caller to what it may list. +Without `caller_pubkey`, `internal/mcp/server.go:879-883` (was :709-720) +returns every entity that has an `age_pubkey`; nothing ties the caller to +what it may list. Still open — line numbers only, substance unchanged. **Fix:** require `caller_pubkey` and scope results to the caller's entitlements. --- -## C. User perspective (interacting via Hermes) +## C. User perspective (interacting via Hermes) — RESOLVED + +**Resolved as of the Hermes→Nomos rewrite (verified 2026-07-12).** This +entire section described `cmd/hermes`, which no longer exists — Hermes was +renamed and rebuilt as `cmd/nomos`, a real LLM-backed agent loop, which is +exactly the recommendation below. `cmd/nomos/main.go:444-465` now calls the +real `listTools()` for "help"/"what can you do", and routes unmatched +queries to "natural language queries belong to `/chat`..." instead of +silently falling back to `get_health_summary`. Kept below for history — +original text unchanged. - `routeQuery` NLU is hardcoded `strings.Contains`; `extractEntity` (`cmd/hermes/main.go:173`) recognizes only 5 services (`authentik, caddy, @@ -175,27 +203,41 @@ says 21 — both stale). Missing capabilities: `pending_approval`, an agent has no way to check or reference the approval. Add `get_approval_status` / `list_pending_approvals`. 4. Execution actions limited to `restart | systemctl | pct_exec | - apt_upgrade` — no deploy/rollback/config-edit path. -5. Architecture/doc mismatch: `hermes/SOUL.md` claims "no SSH access; all - mutations flow through the actuator", but the MCP server runs - `restart`/`pct_exec` synchronously over SSH from inside the api process - (`sshExec`, server.go:902). Align docs or move execution to the actuator. + apt_upgrade` — no deploy/rollback/config-edit path. Partially + superseded: the general `run` MCP tool (D4-partial, done) covers + arbitrary commands now; `request_execution`'s fixed enum is still there + for the specific actions it names (see + [2026-07-10-general-gated-execution.md](2026-07-10-general-gated-execution.md)). +5. **RESOLVED (verified 2026-07-12).** Architecture/doc mismatch: + `hermes/SOUL.md` claimed "no SSH access; all mutations flow through the + actuator", but the MCP server ran `restart`/`pct_exec` synchronously over + SSH from inside the api process. `nomos/SOUL.md:21-22,40` now accurately + documents SSH access via the policy-gated `run` tool — matches the + architecture the general-gated-execution plan built. No longer a + mismatch. --- ## E. Doc drift / housekeeping -- Tool counts: README 15 / AGENTS.md 21 / actual 28 — regenerate from - `internal/mcp/server.go` (consider a doc-gen make target). -- `compose/caddy/Caddyfile.oikos` retains literal `` - placeholders in all three vhosts. -- `.agents/HERMES.md` lists "`inventory.yaml`, `inventory.yaml`" (duplicate). -- `plans/index.md` drift: fix-MCP-tools row sat in Active with a broken link - after the file moved to `done/` (fixed alongside this plan); TRMNL listed - active though in `done/`; Grimmory header says `in-progress` though in - `done/`; `.hermes/plans/` (7 executed plans) missing from disk. -- `plans/2026-07-05-oikos-prometheus-lxc.md` (~0% done) references deleted - `oikos/scheduler.py` and `bin/homelab`; LXC 131 collision unresolved. +- **RESOLVED (verified 2026-07-12):** Tool counts. README 15 / AGENTS.md 21 + / actual 28 was already stale by 2026-07-11 (registered tools grew to + 33) — AGENTS.md now documents all 33 with the full catalog (2026-07-12). +- **Still open:** `compose/caddy/Caddyfile.oikos` retains literal + `` placeholders (this repo's copy is a reference only — + see [2026-07-12-wails-desktop-app.md](2026-07-12-wails-desktop-app.md)'s + "Plan review" — the real config lives in `dtoro/caddy-conf`). +- **RESOLVED:** `.agents/HERMES.md` renamed to `.agents/NOMOS.md`; the + duplicate-line bug itself is still present at `.agents/NOMOS.md:11` — + only the file citation was stale, the underlying nit is still open. +- **RESOLVED (verified 2026-07-12):** `plans/index.md` drift — the broken + link, TRMNL/Grimmory Active/Done mismatch, and missing `.hermes/plans/` + entries described here are no longer present in the current + `plans/index.md`; already fixed sometime after this plan was written. +- **RESOLVED (verified 2026-07-12):** `plans/2026-07-05-oikos-prometheus-lxc.md` + already self-corrected both the deleted-file references and the LXC 131 + collision in its own 2026-07-08 changelog — this bullet describes a + pre-fix state. --- diff --git a/plans/2026-07-11-nomos-agent-code-review.md b/plans/2026-07-11-nomos-agent-code-review.md index dbf619e..3d1c45b 100644 --- a/plans/2026-07-11-nomos-agent-code-review.md +++ b/plans/2026-07-11-nomos-agent-code-review.md @@ -182,16 +182,27 @@ board reflects reality instead of showing a task that looks perpetually ### C1. Nomos's own HTTP gateway has zero authentication -[docker-compose.yml:133](../docker-compose.yml) publishes port 8092 directly +[docker-compose.yml:144](../docker-compose.yml) publishes port 8092 directly (`"8092:8092"`, comment: *"mesh-published"*) and -[Caddyfile.oikos](../compose/caddy/Caddyfile.oikos:19,34) reverse-proxies to -it from two routes. `grep -n "Authorization\|Bearer\|auth" cmd/nomos/main.go` -returns **nothing** — `/chat`, `/sessions`, `/sessions/{id}` (including -`DELETE`), and `/query` have no credential check of any kind. Anyone who can -reach the LAN or mesh network can converse with Nomos directly: start tasks, -read/delete any session, answer pending questions, and — via chat-assent — -approve gated executions by typing "yes" or "I confirm" to whatever the agent -proposes, with no authentication at all. This is the same class of gap +[Caddyfile.oikos:52-54](../compose/caddy/Caddyfile.oikos) reverse-proxies to +it — as of the client/server split +([2026-07-12-wails-desktop-app.md](2026-07-12-wails-desktop-app.md)), only +from `nomos.hubris.network` now, not two routes: `/agent/*` on +`oikos.hubris.network` was repointed to go through `api`'s own authenticated +proxy mount instead of straight to nomos:8092, but that's `combinedAuth` +authenticating the *hop into api*, not anything nomos itself checks — this +finding is unaffected by that change, still fully open. `grep -n +"Authorization\|Bearer\|auth" cmd/nomos/main.go` still returns **nothing** +for nomos's inbound routes (nomos did gain outbound auth as *part of* the +client/server split — it now sends `Authorization: Bearer +$OIKOS_MCP_BEARER_TOKEN` on its own calls to `api` — but that's the opposite +direction from this finding) — `/chat`, `/sessions`, `/sessions/{id}` +(including `DELETE`), and `/query` have no credential check of any kind. +Anyone who can reach the LAN or mesh network can converse with Nomos +directly: start tasks, read/delete any session, answer pending questions, +and — via chat-assent — approve gated executions by typing "yes" or "I +confirm" to whatever the agent proposes, with no authentication at all. This +is the same class of gap [oikos-gaps-and-improvements](2026-07-08-oikos-gaps-and-improvements.md) flagged for the `api`/MCP surface (items B1-B5), but specifically for nomos's *own* port, which doesn't sit behind `combinedAuth` the way `api`'s routes do. diff --git a/plans/2026-07-12-wails-desktop-app.md b/plans/2026-07-12-wails-desktop-app.md index 4c79f5a..ab232fd 100644 --- a/plans/2026-07-12-wails-desktop-app.md +++ b/plans/2026-07-12-wails-desktop-app.md @@ -1,10 +1,33 @@ # 2026-07-12 — Wails desktop application -**Status:** In Progress — Phase 0 (0.1-0.4, 0.6) done and verified live -(browser: cross-origin static SPA + API on different ports, CORS, bearer -auth, SSE query-token auth, localStorage persistence across reload — see -"Plan review" for the gaps found and fixed along the way). Phase 1 (Wails -shell) not started. +**Status:** In Progress — Phase 0 (0.1-0.4, 0.6) done, verified live in a +local browser test, and **deployed to production** (mac-mini, commit +`0c0f35a`, 2026-07-12). Phase 1 (Wails shell) not started. + +**Production deploy (2026-07-12):** merged to `main`, picked up by the +2-minute deploy poller (`scripts/deploy.sh`: pg_dump backup → rebuild → +rolling restart → health check), `healthy after 1s`. Verified post-deploy: +unauthenticated `/api/v1/*` now 401s (the dev-open bypass was live in +production before this — `OIKOS_ENV=dev` with no token set — so this closed +a real, currently-exploitable hole, not just future prep); `/healthz` stayed +open; nomos reconnected its MCP session with the new +`OIKOS_MCP_BEARER_TOKEN` and a real tool call round-tripped end to end +(`get_health_summary` via `/query`). A real random token was generated and +added to mac-mini's `.env` (not committed — gitignored) before deploy, so +the `${OIKOS_MCP_BEARER_TOKEN:-dev-token}` fallback in `docker-compose.yml` +never activated with the weak literal default. + +**Deliberately not done as part of this deploy** (out of scope — a different +host/repo than "mac-mini", not touched): the Caddy LXC (121) and +`dtoro/caddy-conf`. Checked the real production Caddyfile directly — there is +**no `oikos.hubris.network` site block at all yet**, so the Authentik-bypass +risk (gap 1 below) doesn't apply yet; there's no public UI exposed to break. +`mcp.hubris.network` exists but still reverse-proxies to the old +pre-consolidation service on LXC 105 (`192.168.8.205:9810`), unrelated to +this stack — stale, but pre-existing and out of scope here. Exposing +`oikos.hubris.network` publicly (with the `@api` bypass this plan's +Caddyfile.oikos reference copy already has) is unstarted follow-up work, not +a regression from this deploy. ## Plan review — gaps found before starting Phase 0 diff --git a/plans/index.md b/plans/index.md index 09791fa..cd4ef05 100644 --- a/plans/index.md +++ b/plans/index.md @@ -10,11 +10,11 @@ went sideways, open an investigation. | ---- | ----- | ------ | | 2026-07-05 | [Oikos Prometheus LXC](2026-07-05-oikos-prometheus-lxc.md) | Planned — not started | | 2026-07-08 | [Oikos gaps, broken things, and improvements](2026-07-08-oikos-gaps-and-improvements.md) | In Progress — security items (B1-B5) and doc drift (E) still open | -| 2026-07-08 | [Control room web UI](2026-07-08-control-room-webui.md) | In Progress | +| 2026-07-08 | [Control room web UI](2026-07-08-control-room-webui.md) | In Progress — packaging/auth sections superseded by the Wails plan's Phase 0 (client/server split); M4 still open | | 2026-07-08 | [Liveness, drift, and UX cohesion](2026-07-08-liveness-drift-and-ux-cohesion.md) | In Progress — Phase 5 deferred | | 2026-07-10 | [General gated execution: unlimited actions, gated by risk](2026-07-10-general-gated-execution.md) | In Progress — enum retirement + auto-act revival still open | | 2026-07-11 | [Nomos agent code review: gaps and improvement plan](2026-07-11-nomos-agent-code-review.md) | In Progress — only C1 (unauthenticated nomos gateway) still open, deferred | -| 2026-07-12 | [Wails desktop application](2026-07-12-wails-desktop-app.md) | In Progress — Phase 0 done, Phase 1 not started | +| 2026-07-12 | [Wails desktop application](2026-07-12-wails-desktop-app.md) | In Progress — Phase 0 done and deployed to production (2026-07-12), Phase 1 not started | ## Done diff --git a/tools/caveman/caveman.js b/tools/caveman/caveman.js deleted file mode 100644 index 6f17d44..0000000 --- a/tools/caveman/caveman.js +++ /dev/null @@ -1,23 +0,0 @@ -#!/usr/bin/env node -// Caveman template renderer — reads template + data JSON files and renders output -// Installed automatically via homelab-context post-pull hook -const caveman = require("caveman"); -const fs = require("fs"); -const path = require("path"); - -const args = process.argv.slice(2); -if (args.length === 0) { - console.error("Usage: caveman