Compare commits
88 Commits
claude/goo
...
claude/cha
| Author | SHA1 | Date | |
|---|---|---|---|
| 614c38ea7c | |||
| 22412d2fa3 | |||
| 5686b9de40 | |||
| aa6017e0ca | |||
| cbfd09c5df | |||
| 851b5dce67 | |||
| 279549c8c9 | |||
| a39e67b6e9 | |||
| 551497e0b3 | |||
| 4a5e68bafe | |||
| ef00e5b8e4 | |||
| a512d40669 | |||
| b8bb29464b | |||
| 81acadec1d | |||
| 291b45565b | |||
| d45f2326b6 | |||
| 4bf811a383 | |||
| 35feada286 | |||
| cca2ae4621 | |||
| d13f6991b1 | |||
| 2de3602ebc | |||
| 2908b0a377 | |||
| cff05c0768 | |||
| 5d02126e16 | |||
| e8e230b4a5 | |||
| 2b3aa248b1 | |||
| 9d9cbb63c4 | |||
| ae3b150dcc | |||
| 750f0e088d | |||
| 84068cc40b | |||
| 5d15265f65 | |||
| 3ea43adcfd | |||
| 4724d6f297 | |||
| f07668c1c3 | |||
| 7e0566d2cd | |||
| 2e8ef75436 | |||
| ca16b75b90 | |||
| e148c7a981 | |||
| cefeba72b0 | |||
| e131215a25 | |||
| 7660e5681c | |||
| 28ab9b8088 | |||
| fcd9f23ee1 | |||
| efa66c7321 | |||
| 43aaf2a318 | |||
| a3ebd12e90 | |||
| 7c6cffb5f5 | |||
| 5b22f2367b | |||
| 84ecb6b895 | |||
| cfce35bee0 | |||
| a786107cc7 | |||
| 44e1e421e1 | |||
| 8653f3036d | |||
| 79dc87d584 | |||
| 638e313c66 | |||
| 85b541a1cc | |||
| 7f7d039e1b | |||
| 7ac2521a22 | |||
| 692800cf09 | |||
| bbd560bf19 | |||
| e93625b971 | |||
| bf250fc6a9 | |||
| b35825c50b | |||
| 806795c63d | |||
| 24b8772d72 | |||
| 129a6710cc | |||
| ae23311f17 | |||
| 2de23325c5 | |||
| 6b51c83b82 | |||
| 5e3b946ded | |||
| 5009a335bb | |||
| f04e0dc0d4 | |||
| 6b75f7302d | |||
| b2bfa26f64 | |||
| 4b6c02a88e | |||
| 11be45d307 | |||
| 0e3cbceeae | |||
| 128e11b823 | |||
| a0f059d19f | |||
| c9fb5fe553 | |||
| 8175d218c1 | |||
| c8b27b2c51 | |||
| dcd35b6315 | |||
| 890fe1a1c3 | |||
| f4a00a6cfd | |||
| 74a6b6bb18 | |||
| 3823a82417 | |||
| aa197190cd |
@@ -1,4 +1,4 @@
|
||||
# HERMES.md — Agent persona for homelab clients
|
||||
# NOMOS.md — Agent persona for homelab clients
|
||||
|
||||
This file is the canonical agent persona for **all** AI agents running on
|
||||
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
|
||||
@@ -8,7 +8,7 @@ conventions, and the source-of-truth hierarchy.
|
||||
|
||||
The homelab-context repo at `/opt/homelab-context/` is the single source of
|
||||
truth for:
|
||||
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
|
||||
- Fleet topology (`inventory.yaml`, `inventory.yaml`)
|
||||
- Service endpoints and credentials (via `homelab secret`)
|
||||
- Agent behaviour and conventions
|
||||
- Everything in this file
|
||||
@@ -22,7 +22,7 @@ deploy, client enrollment, incident investigation, and each node
|
||||
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
|
||||
acting. Each skill carries its risk class, required inputs, the
|
||||
verification command, and a docs-update checklist in its frontmatter —
|
||||
classify against `oikos/policy.yaml` using that risk class before any
|
||||
classify against `seeds/policy.yaml` using that risk class before any
|
||||
mutation. Don't re-derive topology or the mutation path by grepping the
|
||||
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
|
||||
operating model these runbooks execute inside (OODA loop, risk classes,
|
||||
@@ -32,8 +32,8 @@ approval flow, ontology).
|
||||
|
||||
| Agent | Loading mechanism |
|
||||
|-------|------------------|
|
||||
| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
|
||||
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints` → `/opt/homelab-context/HERMES.md` |
|
||||
| **Nomos** | `tools/setup-nomos-soul.sh` (auto-setup) → provisions `~/.nomos/SOUL.md` from this file |
|
||||
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints` → `/opt/homelab-context/NOMOS.md` |
|
||||
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
|
||||
|
||||
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
|
||||
@@ -87,10 +87,10 @@ Caveman templates live at `~/templates/`:
|
||||
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
|
||||
```
|
||||
|
||||
## Important note for Hermes agents
|
||||
## Important note for Nomos agents
|
||||
|
||||
If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
|
||||
by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
|
||||
If you are reading this as a Nomos agent, your SOUL.md was auto-provisioned
|
||||
by `tools/setup-nomos-soul.sh`. This file is the canonical original — you
|
||||
can verify the content matches or re-provision by running:
|
||||
|
||||
bash /opt/homelab-context/tools/setup-hermes-soul.sh
|
||||
bash /opt/homelab-context/tools/setup-nomos-soul.sh
|
||||
233
.agents/OIKOS.md
233
.agents/OIKOS.md
@@ -6,8 +6,8 @@ structure, the `homelab` CLI and MCP server are the syscall surface, and
|
||||
this page defines the rules everything above them follows.
|
||||
|
||||
Read this after [AGENTS.md](../AGENTS.md). Machine-readable companions:
|
||||
[oikos/ontology.yaml](../oikos/ontology.yaml) (systems model),
|
||||
[oikos/policy.yaml](../oikos/policy.yaml) (risk & approval).
|
||||
[seeds/ontology.yaml](../seeds/ontology.yaml) (systems model),
|
||||
[seeds/policy.yaml](../seeds/policy.yaml) (risk & approval).
|
||||
|
||||
## The kernel loop: OODA
|
||||
|
||||
@@ -38,19 +38,20 @@ one pass through **Observe → Orient → Decide → Act**:
|
||||
|
||||
| Primitive | What it is | Lives in |
|
||||
|---|---|---|
|
||||
| Host / Service | topology entities | `inventory.yaml` (+ generated `hosts/*.yaml`) |
|
||||
| Host / Service | topology entities | `inventory.yaml` |
|
||||
| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` |
|
||||
| Runbook | executable workflow with risk class + verification | `.agents/skills/<name>/SKILL.md` |
|
||||
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
|
||||
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
|
||||
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
|
||||
| Incident | investigation narrative | `knowledge/sources/investigations/` |
|
||||
| Signal | something needing attention, with lifecycle | DB `signals` table |
|
||||
| Change | one mutation: who, what, risk, approval, verification | DB `audit_log` + `executions` tables |
|
||||
| Approval | short-TTL signed grant for a gated action | DB `approvals` table |
|
||||
| Incident | investigation narrative | DB `knowledge_entities` (seeded from investigations) |
|
||||
| Knowledge | document, runbook, investigation | DB `knowledge_entities` (seeded from `seeds/knowledge.yaml`) |
|
||||
| Plan | design doc for non-trivial work | `plans/` |
|
||||
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
|
||||
|
||||
## Risk classes (enforced, not advisory)
|
||||
|
||||
From [oikos/policy.yaml](../oikos/policy.yaml):
|
||||
From [seeds/policy.yaml](../seeds/policy.yaml):
|
||||
|
||||
- **read_only** — status, logs, docs, inventory. Unattended.
|
||||
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger.
|
||||
@@ -69,7 +70,7 @@ Eight domains — physical, compute, network, storage, software,
|
||||
identity & access, operations, external — cover everything in the lab;
|
||||
entities are connected by typed edges (`hosts`, `provides`, `mounts`,
|
||||
`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …)
|
||||
defined in [oikos/ontology.yaml](../oikos/ontology.yaml). Rule of
|
||||
defined in [seeds/ontology.yaml](../seeds/ontology.yaml). Rule of
|
||||
completeness: **if it can break, be changed, or hold data, it has an
|
||||
entity and edges.** Blast-radius questions ("what breaks if strong goes
|
||||
down?") are graph walks, not doc archaeology.
|
||||
@@ -80,9 +81,9 @@ stored as `state:` in inventory (absent = active). Destroyed nodes live in
|
||||
the `archaeology:` section. Each transition is a runbook checklist;
|
||||
deprecation completes only when inbound edges reach zero.
|
||||
|
||||
Generated views: [infrastructure/topology.md](../knowledge/wiki/infrastructure/topology.md)
|
||||
(Mermaid, regenerated from inventory) and the live, clickable version at
|
||||
`oikos.hubris.network/graph` once the Console is deployed.
|
||||
Generated views: the live topology graph at `oikos.hubris.network/graph`
|
||||
via the API's `/api/v1/graph` endpoint, and the Mermaid export at
|
||||
`GET /api/v1/graph?format=mermaid`.
|
||||
|
||||
## Conventions carried forward
|
||||
|
||||
@@ -95,133 +96,95 @@ Generated views: [infrastructure/topology.md](../knowledge/wiki/infrastructure/t
|
||||
- Agents are terse ([caveman.md](shared/caveman.md)), verify claims, and fix
|
||||
collateral drift when found.
|
||||
|
||||
## Build status (30-day roadmap, started 2026-07-05)
|
||||
## Build status (Go rewrite — deployed 2026-07-07)
|
||||
|
||||
- **Week 1**: policy, ontology, service contract, archaeology, topology
|
||||
generator, this brief. Shipped.
|
||||
- **Week 2**: context cards, `homelab service <name> …`, change ledger,
|
||||
`node relations`, runbooks. Shipped.
|
||||
- **Week 3**: ops scheduler + state cache (`homelab service <name> health`
|
||||
is cache-first, `--live` forces a probe), drift detectors, signal engine
|
||||
(`homelab signal …`), decision classifier (`homelab decide …`), approval
|
||||
engine (`homelab approval …` — shared-HMAC grants; Matrix delivery is
|
||||
Hermes's existing `@dtoro:avispero` send path, not a new bot, see
|
||||
`oikos/approve.py`), daily brief + weekly report (`oikos/report.py`).
|
||||
Shipped, except: Prometheus is still `planned` (see
|
||||
[plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md)) —
|
||||
trend signals (disk-full prediction, temp creep) wait on that LXC; the
|
||||
scheduler's disk check today is point-in-time only, and CPU/NVMe
|
||||
temperature isn't probed at all yet (no confirmed sensor path on
|
||||
hubris/strong). DNS-vs-inventory and generic tracked-config-cleanliness
|
||||
drift checks are also deferred (see `oikos/drift.py` docstring).
|
||||
- **Week 4**: Oikos Console v0 shipped — signals landing page, service
|
||||
grid + detail, node/blast-radius view, live Mermaid graph, drift view,
|
||||
approvals queue (approve/deny, destructive confirmation-phrase
|
||||
enforced), daily/weekly reports. Server-rendered FastAPI + Jinja2, no
|
||||
SPA build chain, tested end-to-end against live production data (see
|
||||
`oikos/console/`). Deploys as a third webhook on `dtoro/Homelab-Docs`
|
||||
(`/opt/oikos-console`, port :9831) — see
|
||||
[oikos/console/deploy/README.md](../oikos/console/deploy/README.md) for
|
||||
the Caddy route and Gitea webhook registration this repo can't do for
|
||||
itself. Approval grants are now single-use (a second `check_grant` call
|
||||
for the same request fails even within the TTL) and already exact-bound
|
||||
to request id + entity + action.
|
||||
**Not shipped as originally planned:** per-agent *age-key-signed*
|
||||
request authentication — age has no signing primitive (it's an
|
||||
encryption-only keypair format), so "age-key-signed" wasn't
|
||||
buildable as stated. The real alternative (SSH-key signing via
|
||||
`ssh-keygen -Y sign`/`-Y verify`, using each host's already-provisioned
|
||||
SSH key) is real and buildable, but needs SSH public keys recorded in
|
||||
inventory first — not there today. Moved to the 60/90-day backlog.
|
||||
Authentik step-up re-auth on the approve/deny route is documented but
|
||||
needs a live Authentik instance to configure — also backlog.
|
||||
Docs pass done (this file, AGENTS.md, operations/commands.md); found
|
||||
and fixed two more stale references while at it (DNS section still
|
||||
pointed at destroyed LXC 124/dnsmasq instead of Technitium on 107, and
|
||||
a `claudio-monitor` reference that's been deprecated since 2026-06-04).
|
||||
The Oikos runtime was rewritten from Python to Go over 6 phases and is deployed
|
||||
in Docker on mac-mini. See
|
||||
[plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](../plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md)
|
||||
for the full plan. The Python codebase has been removed; all functionality runs
|
||||
in the Go binary.
|
||||
|
||||
### Real drift found while building Week 3 (unresolved, needs operator action)
|
||||
**Phase 1 — Ontology + DB (DONE):**
|
||||
- `migrations/` (001–011): TimescaleDB hypertables, entity_status, CAGGs,
|
||||
retention policies, knowledge entities with FTS. Forward-only, idempotent.
|
||||
- `seeds/{ontology,inventory,policy,knowledge}.yaml`: DB-native bootstrap +
|
||||
DR export. Knowledge seed contains 36 documents, 6 investigations, and 12
|
||||
runbooks.
|
||||
- `blast_radius()` SQL CTE, type hierarchy, abstract types, relationship
|
||||
validation.
|
||||
- Go packages: `internal/db/`, `internal/ontology/`, `internal/domain/`,
|
||||
`internal/knowledge/`.
|
||||
|
||||
The drift detectors surfaced genuine, currently-true findings on first
|
||||
run against production — recorded here rather than silently fixed, since
|
||||
each is a `config_mutation`/`destructive`-class decision:
|
||||
**Phase 2 — API (DONE):**
|
||||
- Single binary `cmd/oikos` with `oikos api` serving REST (:8090) + MCP
|
||||
on the same service layer. OpenAPI-first (`api/openapi.yaml`) with
|
||||
oapi-codegen + chi. RFC 9457 problem+json errors. Cursor pagination,
|
||||
If-Match/ETag optimistic concurrency, idempotency keys, SSE event stream,
|
||||
OIDC JWT + static bearer auth, audit middleware.
|
||||
- Go packages: `internal/httpapi/`, `internal/httpapi/gen/`.
|
||||
|
||||
- `republic-laptop` has no `age_pubkey:` in `inventory.yaml`, but its real
|
||||
age key is granted on nearly every shared secret in `.sops.yaml`
|
||||
(`age1vf8h7...`) — the enrollment write-back to inventory never
|
||||
happened. Fix: `homelab client add republic-laptop --finalize-pubkey
|
||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6`.
|
||||
- `grimmory` has an `age_pubkey` in inventory but is missing from
|
||||
`secrets/hello.yaml`'s recipient list — incomplete enrollment the
|
||||
other direction. Fix: re-run `homelab client add grimmory
|
||||
--finalize-pubkey <its key>`.
|
||||
- `pve_id 131` exists live on hubris (`pct list`) with no inventory entry
|
||||
— investigate before assuming it's a stale ID (see the Prometheus LXC
|
||||
plan doc above, which flags this explicitly).
|
||||
- Three `lifecycle-pve-id-reuse` info findings (100, 106, 107 each shared
|
||||
between an active host and an archaeology entry) — expected/benign ID
|
||||
reuse after destroy, no action needed.
|
||||
**Phase 3 — Control loop (DONE):**
|
||||
- Scheduler (`oikos scheduler`): check_defs runner, signal dedup/flap
|
||||
suppression, entity_status. HTTP, TCP, disk, cert-expiry probes.
|
||||
- Actuator: SSH skill procedure execution with context-aware timeouts,
|
||||
circuit breaker, retry budgets, error classification.
|
||||
- Learning engine: hourly pattern extraction, Wilson confidence bounds,
|
||||
anomaly detection, skills with validated patterns.
|
||||
- Notifier: Matrix badge delivery, approval token generation (HMAC,
|
||||
single-use, hashed), DB rendezvous pattern.
|
||||
- Policy classifier: risk class determination, autonomy routing,
|
||||
blast-radius computation, kill-switch support.
|
||||
- Go packages: `internal/scheduler/`, `internal/actuator/`,
|
||||
`internal/learning/`, `internal/notifier/`, `internal/policy/`.
|
||||
|
||||
## 60/90-day backlog
|
||||
**Phase 4 — Agent / Nomos (DONE):**
|
||||
- Standalone Nomos MCP client binary (`cmd/nomos`) with gateway mode
|
||||
(:8092). Structured queries + natural-language routing to 15 MCP tools.
|
||||
Agent activity logging on every tool call. No SSH keys.
|
||||
- `nomos/` directory with config, SOUL.md, homelab-ops skill.
|
||||
- Nomos Docker service in `docker-compose.yml` (profile: full).
|
||||
- Go packages: `cmd/nomos/`, `compose/nomos/`.
|
||||
|
||||
Derived from gaps observed while building the 30-day roadmap, not
|
||||
guesswork. Roughly ordered by what unblocks the most:
|
||||
**Phase 5 — Secrets / Infisical (DONE):**
|
||||
- `internal/secrets/`: backend abstraction (Manager) with primary
|
||||
(Infisical) and fallback (SOPS) backends. Machine identities via
|
||||
UniversalAuth. In-memory cache with TTL.
|
||||
- `oikos secret` CLI: list, migrate (SOPS → Infisical), export-sops
|
||||
(DR fallback). Infisical SDK v0.8.0.
|
||||
- Rotation runbook at `secrets/rotation.md`.
|
||||
- Docker compose: `infisical` + `redis` services (profile: infisical).
|
||||
|
||||
- **Fix the oikos-console deploy webhook's signature mismatch.** Console
|
||||
is live on apps (105) via a manual `deploy.sh` run, but Gitea webhook
|
||||
14's deliveries all 403 with a signature mismatch for a cause not yet
|
||||
found — the secret is confirmed synced correctly on both sides
|
||||
(rotated once already to rule out drift). Until fixed, `git push`
|
||||
doesn't auto-redeploy the console the way it does for homelab-mcp/
|
||||
secrets-issuance; re-run `deploy.sh` on apps manually after changes.
|
||||
See [oikos/console/deploy/README.md](../oikos/console/deploy/README.md).
|
||||
- **SSH-key-signed approval requests.** Replaces the design note in
|
||||
Week 4: age keys can't sign (encryption-only format), so per-agent
|
||||
request authentication needs `ssh-keygen -Y sign`/`-Y verify` against
|
||||
each host's existing SSH key. Blocked on a schema gap: inventory
|
||||
doesn't record SSH public keys today, only ports/users. First step is
|
||||
populating that field on enrollment, then wiring `oikos/approve.py` to
|
||||
require and verify a signature over the request payload.
|
||||
- **Authentik step-up re-auth** on the Console's `/approvals` POST route
|
||||
— needs a live Authentik `PromptStage`/reauth flow scoped to that path;
|
||||
not configurable without a running instance to test against.
|
||||
- **Prometheus provisioning** (see
|
||||
[plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md))
|
||||
— unblocks trend signals (disk-full prediction, temp creep) and real
|
||||
sparklines in the Console; investigate the undocumented `pve_id 131`
|
||||
on hubris first.
|
||||
- **CPU/NVMe temperature probing** in the scheduler — needs a confirmed
|
||||
sensor path on hubris and strong (lm-sensors vs vendor tool) before a
|
||||
real check can be written; guessing one risks a probe that silently
|
||||
never fires.
|
||||
- **DNS-vs-inventory drift check** — compare Technitium zone records
|
||||
against `services.*.url`/`public_host`; not implemented (`oikos/drift.py`
|
||||
has no Technitium API wiring yet).
|
||||
- **Generic tracked-config-cleanliness drift check** — today only caddy's
|
||||
`/etc/caddy` git-checkout path is hardcoded in `oikos/drift.py`; every
|
||||
other service with a `config_repo` needs its local checkout path
|
||||
recorded (a `mutation_path`-style field, same gap Week 1's service
|
||||
contract flagged but didn't backfill) before this generalizes.
|
||||
- **Per-service policy overrides** (`oikos/policy.yaml`
|
||||
`service_overrides`) — schema is ready (caddy/dns already use it);
|
||||
populate more as specific services turn out to need non-default risk
|
||||
classes.
|
||||
- **Incident timeline generator** — stitch ledger + signal history into
|
||||
a single narrative for `knowledge/sources/investigations/` entries instead of writing
|
||||
them by hand.
|
||||
- **Secret access audit** — who-can-decrypt-what report from
|
||||
`.sops.yaml` + inventory `age_pubkey`s, extending what
|
||||
`oikos/drift.py`'s SOPS check already partially does.
|
||||
- **Restore drills** — exercise `backs-up-to` (once populated) by
|
||||
actually restoring from a backup target on a schedule, not just
|
||||
checking freshness.
|
||||
- **Multi-agent delegation model** — more than one agent acting
|
||||
concurrently; needs the ledger's `agent` field to carry real identity
|
||||
(age pubkey, not just hostname) consistently, which it mostly does
|
||||
already but hasn't been stress-tested with concurrent writers.
|
||||
- **Grafana** — only if the Console's own Prometheus-backed sparklines
|
||||
turn out to be insufficient once Prometheus ships.
|
||||
- **"Generalize later" extraction** — the original decision was personal-
|
||||
first, generalize-later (see Week 1). Once patterns stabilize, extract
|
||||
a config-driven Oikos core with no `hubris.network`/`hubris`/`strong`
|
||||
hardcoding, so it's installable on a different homelab.
|
||||
**Phase 6 — Deploy + cutover (DONE, pending production cutover):**
|
||||
- CI pipeline: `.gitea/workflows/ci.yml` (Gitea Actions — build, vet,
|
||||
lint, test, docker build).
|
||||
- Deploy: `scripts/deploy.sh` (git pull → docker build → compose up →
|
||||
health check), SHA-tagged images, rolling restart.
|
||||
- Caddy config: `compose/caddy/Caddyfile.oikos` (oikos/mcp/nomos →
|
||||
mac-mini mesh :8090/:8092).
|
||||
- Watchdog: `scripts/watchdog.sh` (2min cron, Matrix alert on failure).
|
||||
- Verification: `scripts/verify-phase6.sh` (14/14 checks pass).
|
||||
- Rollback: `scripts/rollback.sh` (checkout SHA + pg_restore).
|
||||
- Cutover checklist: `scripts/cutover-checklist.md`.
|
||||
|
||||
**Current deployment:**
|
||||
- **Production**: Docker stack on mac-mini (`--profile full`: postgres, api,
|
||||
scheduler, notifier, nomos). Deployed 2026-07-07 with full knowledge seed.
|
||||
The Python MCP server and secrets-issuance on apps/105 have been stopped
|
||||
(see `scripts/cutover-checklist.md`).
|
||||
|
||||
## Python-era backlog (superseded)
|
||||
|
||||
The original 30-day roadmap (Python, shipped 2026-06/07) delivered:
|
||||
context cards, change ledger, node relations, runbooks, ops scheduler,
|
||||
drift detectors, signal engine, classifier, approval engine, and the
|
||||
FastAPI+Jinja2 Oikos Console. All of these have been re-implemented in
|
||||
the Go rewrite. The backlog items below that referenced Python paths
|
||||
(`oikos/approve.py`, `oikos/drift.py`, `oikos/console/`) are now addressed
|
||||
by the Go equivalents listed above.
|
||||
|
||||
Outstanding from the Python era (not yet in Go):
|
||||
- Prometheus provisioning (see [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md))
|
||||
- CPU/NVMe temperature probing (blocked on sensor path discovery)
|
||||
- SSH-key-signed approval requests (blocked on inventory schema)
|
||||
- Multi-agent delegation (blocked on ledger identity field)
|
||||
- Restore drills on a schedule
|
||||
|
||||
179
.agents/dev/CONTRIBUTING.md
Normal file
179
.agents/dev/CONTRIBUTING.md
Normal file
@@ -0,0 +1,179 @@
|
||||
# Agent developer guide
|
||||
|
||||
Instructions for AI agents working on the Oikos codebase. Read this after
|
||||
[AGENTS.md](../../AGENTS.md) and [OIKOS.md](../OIKOS.md). Human developers:
|
||||
see [CONTRIBUTING.md](../../CONTRIBUTING.md) for a human-friendly version.
|
||||
|
||||
## Codebase map
|
||||
|
||||
```
|
||||
cmd/oikos/main.go Entry point. Subcommands: api, scheduler, notifier, migrate,
|
||||
seed, export, secret, all
|
||||
cmd/nomos/main.go Nomos MCP client gateway (standalone binary, formerly Hermes)
|
||||
internal/httpapi/ REST + MCP server. Chi router. OpenAPI-generated types from
|
||||
internal/httpapi/gen/api.gen.go. Strict server in impl.go.
|
||||
internal/mcp/ MCP tool implementations (get_entity, search_knowledge, etc.)
|
||||
internal/db/ Connection pool (pool.go), seed ingestion (seed.go), DB→YAML
|
||||
export (export.go), type hierarchy (typetree.go)
|
||||
internal/db/queries/ SQL query files → sqlc generates internal/db/sqlcgen/
|
||||
internal/scheduler/ Observe loop: probes, signals, check_defs
|
||||
internal/actuator/ SSH execution with circuit breaker + retry
|
||||
internal/learning/ Pattern extraction, anomaly detection
|
||||
internal/notifier/ Matrix notification + approval token generation
|
||||
internal/policy/ Risk classifier (read policy.yaml → classify action)
|
||||
internal/secrets/ Backend abstraction: Infisical (primary) + SOPS (fallback)
|
||||
internal/domain/ Core types: entities, approvals, executions, signals, patterns
|
||||
internal/ontology/ Type hierarchy validation, relationship checks
|
||||
internal/knowledge/ Knowledge YAML seed ingestion
|
||||
internal/config/ Config loading from env vars
|
||||
api/openapi.yaml REST API contract. Source of truth for endpoints.
|
||||
api/codegen.yaml oapi-codegen config → generates internal/httpapi/gen/
|
||||
migrations/ Forward-only SQL. Format: NNN_name.up.sql. No down migrations.
|
||||
seeds/ Bootstrap YAML. ontology.yaml, inventory.yaml, policy.yaml,
|
||||
knowledge.yaml. Regenerated from DB via oikos export.
|
||||
compose/ Dockerfiles. oikos/ (multi-stage), nomos/ (distroless).
|
||||
Caddy config at compose/caddy/Caddyfile.oikos.
|
||||
scripts/ Deploy, rollback, watchdog, verification, cutover checklist.
|
||||
nomos/ Nomos config.yaml, SOUL.md, skills.
|
||||
.agents/ Agent instruction files, domains, shared conventions, skills.
|
||||
plans/ Design documents. active/ + done/.
|
||||
docs/adr/ Architecture decision records. Numbered, prefix-sorted.
|
||||
```
|
||||
|
||||
## Development loop
|
||||
|
||||
```bash
|
||||
# Start dependencies
|
||||
make dev
|
||||
|
||||
# Generate code after API/SQL changes
|
||||
make generate
|
||||
|
||||
# Build
|
||||
make build
|
||||
|
||||
# Run tests
|
||||
make test # all unit tests
|
||||
make test-db # integration tests (needs compose Postgres)
|
||||
|
||||
# Lint
|
||||
make lint
|
||||
|
||||
# CI drift guard (run before commit)
|
||||
make generate-check
|
||||
```
|
||||
|
||||
## Adding a feature or phase
|
||||
|
||||
Oikos features follow a phase model (read [OIKOS.md](../OIKOS.md) for the
|
||||
current phase status). To add a new capability:
|
||||
|
||||
1. **ADR first.** Write an architecture decision record in `docs/adr/` with
|
||||
the next sequence number. Document the decision, context, alternatives
|
||||
considered, and consequences.
|
||||
2. **Plan.** If the change is non-trivial, create a plan in `plans/` following
|
||||
the template in [page-templates.md](../shared/page-templates.md).
|
||||
3. **Schema.** If the feature needs new DB tables, write a forward-only
|
||||
migration in `migrations/`. Use `IF NOT EXISTS` for idempotency.
|
||||
4. **API.** If the feature exposes endpoints, define them in
|
||||
`api/openapi.yaml` first, then run `make generate`, then implement.
|
||||
5. **Domain.** Add types to `internal/domain/` before adding logic.
|
||||
6. **Tests.** Write tests alongside implementation. Integration tests go in
|
||||
`*_test.go` in the relevant package, using the compose Postgres.
|
||||
7. **Policy.** If the feature introduces new mutation types, update
|
||||
`seeds/policy.yaml` and the classifier in `internal/policy/`.
|
||||
8. **Run `make generate-check`** before commit to ensure generated code is
|
||||
current.
|
||||
|
||||
## SQL conventions
|
||||
|
||||
- Queries live in `internal/db/queries/*.sql` with `-- name: FuncName :exec`
|
||||
annotations for sqlc
|
||||
- Use `pgx/v5` driver. UUIDs use `pgtype.UUID`, timestamps use `time.Time`
|
||||
- CTEs for graph traversals (blast radius, dependency chains)
|
||||
- CAGGs and retention policies for TimescaleDB hypertables
|
||||
- FTS via `tsvector` + `tsquery` for knowledge search (migration 011)
|
||||
|
||||
## OpenAPI codegen
|
||||
|
||||
- Config: `api/codegen.yaml`. Uses `oapi-codegen/v2` with Chi server template
|
||||
- Generated output: `internal/httpapi/gen/api.gen.go` — never hand-edit
|
||||
- Strict server interface: `api.gen.go` generates the `StrictServerInterface`;
|
||||
implement it in `internal/httpapi/impl.go`
|
||||
- Problem+JSON errors via `internal/httpapi/problem.go` — RFC 9457 format
|
||||
- Cursor pagination, If-Match/ETag, idempotency keys, SSE streaming
|
||||
|
||||
## Testing philosophy
|
||||
|
||||
- **Race detector always on.** `make test` runs `go test -race -cover ./...`
|
||||
- **Integration tests** use the compose Postgres. Run with `make test-db`.
|
||||
Each test creates + tears down its own schema namespace.
|
||||
- **Coverage gates** in CI: policy + learning ≥ 80%, others ≥ 60%
|
||||
- Tests use `testing.T` directly, no assertion library
|
||||
- Table-driven tests for validation and classification logic
|
||||
|
||||
## Migration rules
|
||||
|
||||
- Forward-only. No down migrations (ADR 0008)
|
||||
- Idempotent: use `IF NOT EXISTS`, `DO $$ BEGIN ... END $$` blocks
|
||||
- Sequence numbers are sequential integers (001, 002, ...)
|
||||
- Each migration file is `NNN_name.up.sql`
|
||||
- Migrations are embedded in the binary via `migrations/embed.go`
|
||||
|
||||
## Seed files
|
||||
|
||||
- `seeds/ontology.yaml` — entity types, relationship types, lifecycles
|
||||
(validated against schema in `internal/ontology/`)
|
||||
- `seeds/inventory.yaml` — hosts, services, entities (the topology)
|
||||
- `seeds/policy.yaml` — risk classes, approval rules, autonomy settings
|
||||
- `seeds/knowledge.yaml` — documents, investigations, runbooks (DB is source
|
||||
of truth; this file is the DR export)
|
||||
- After DB changes via the API, run `make export` to regenerate seeds
|
||||
|
||||
## Secrets handling
|
||||
|
||||
- No secrets in code, config, or commits
|
||||
- Dev secrets in `.env` (gitignored)
|
||||
- Primary: Infisical (`internal/secrets/infisical.go`)
|
||||
- Fallback: SOPS + age (`internal/secrets/sops.go`)
|
||||
- Backend interface: `internal/secrets/backend.go`
|
||||
- Machine identities via Infisical UniversalAuth
|
||||
- In-memory cache with TTL for performance
|
||||
|
||||
## Staging and deployment
|
||||
|
||||
- CI pipeline: `.gitea/workflows/ci.yml` — lint, vet, vulncheck, test, docker build
|
||||
- Deploy: `scripts/deploy.sh` — git pull → docker build → compose up → health check
|
||||
- Watchdog: `scripts/watchdog.sh` — 2-minute cron, Matrix alert on failure
|
||||
- Rollback: `scripts/rollback.sh` — checkout SHA + pg_restore
|
||||
- Cutover checklist: `scripts/cutover-checklist.md`
|
||||
|
||||
## Writing conventions
|
||||
|
||||
Apply [writing-style.md](../shared/writing-style.md) for all committed prose.
|
||||
Terse, reference-style, no marketing vocabulary. Code comments explain intent
|
||||
and trade-offs, not mechanics.
|
||||
|
||||
Apply [caveman.md](../shared/caveman.md) for agent communication. The caveman
|
||||
standard applies to agent *chat responses*, not committed documentation.
|
||||
|
||||
## Skills
|
||||
|
||||
Agent skills live under `.agents/skills/<name>/SKILL.md`. Each skill has a
|
||||
frontmatter description that tools match against tasks. To add a skill:
|
||||
|
||||
1. Create `.agents/skills/<name>/SKILL.md`
|
||||
2. Include frontmatter with description field
|
||||
3. Document the procedure following the runbook template
|
||||
4. Reference relevant files, commands, and policy classes
|
||||
|
||||
Skills that require code (e.g. linting) may include companion scripts in the
|
||||
same directory.
|
||||
|
||||
## When in doubt
|
||||
|
||||
- Query MCP tools first (search_knowledge, get_entity)
|
||||
- Read the relevant ADR in `docs/adr/`
|
||||
- Grep the codebase: `rg <symbol> internal/`
|
||||
- Check `plans/` for in-progress work that may conflict
|
||||
- Classify any new mutation against `seeds/policy.yaml` before suggesting it
|
||||
@@ -17,18 +17,18 @@ fixed paths, so the wiki reorganization never moves it.
|
||||
| Layer | Location | Consumed by |
|
||||
|-------|----------|-------------|
|
||||
| Substrate — source of truth | `inventory.yaml` (root) | MCP server, `homelab` CLI, `oikos/` scheduler/drift/relations/gen-topology |
|
||||
| Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` |
|
||||
| Substrate — generated host records | `inventory.yaml` (root) | Go `internal/mcp/` server, `bin/homelab`; the single source of truth |
|
||||
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
|
||||
| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
|
||||
| Narrative — synthesized wiki | `archive/knowledge/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
|
||||
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
|
||||
|
||||
## Wiki pages
|
||||
|
||||
- **Node pages** (`knowledge/wiki/containers/<id>-<name>.md`, `.../vms/<id>-<name>.md`,
|
||||
- **Node pages** (`archive/knowledge/containers/<id>-<name>.md`, `.../vms/<id>-<name>.md`,
|
||||
`.../hosts/<name>.md`) follow the container/host template in
|
||||
[page-templates.md](../../shared/page-templates.md): opening definition, `## At a glance`,
|
||||
`## Role`, service/port map, storage, auto-deploy, `## Related`, `## Changelog`.
|
||||
- **Cross-cutting pages** (`knowledge/wiki/infrastructure/<topic>.md`) follow the cross-cutting
|
||||
- **Cross-cutting pages** (`archive/knowledge/infrastructure/<topic>.md`) follow the cross-cutting
|
||||
template: `## Why`, `## Components`, `## How to apply`, `## Gotchas`, `## Related`, `## Changelog`.
|
||||
- Each `inventory.yaml` host entry carries a `doc_page:` field pointing at its narrative page.
|
||||
Changing where a page lives means updating that field (read by `bin/homelab`).
|
||||
|
||||
@@ -40,7 +40,7 @@ transition: "<from> -> <to>" # only for lifecycle runbooks
|
||||
```
|
||||
|
||||
`risk_class` values and the lifecycle `transition` states must match
|
||||
[`oikos/policy.yaml`](../../../oikos/policy.yaml) and [`oikos/ontology.yaml`](../../../oikos/ontology.yaml).
|
||||
[`seeds/policy.yaml`](../../../seeds/policy.yaml) and [`seeds/ontology.yaml`](../../../seeds/ontology.yaml).
|
||||
|
||||
## Investigations
|
||||
|
||||
|
||||
@@ -6,8 +6,8 @@ this repo that auto-syncs every 5 min, a per-client age key for SOPS
|
||||
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
|
||||
|
||||
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
|
||||
> See [hermes-agent.md](hermes-agent.md). It uses the same `bootstrap.sh`
|
||||
> with an additional `--with-hermes` flag.
|
||||
> See [nomos-agent.md](nomos-agent.md). It uses the same `bootstrap.sh`
|
||||
> with an additional `--with-nomos` flag.
|
||||
|
||||
Architecture in [project_homelab_context_plan](https://… memory link); the
|
||||
operational reference is here.
|
||||
@@ -30,7 +30,7 @@ hostname doesn't match any inventory entry. Two fixes:
|
||||
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
|
||||
(Linux) or System Preferences → Sharing (macOS), then re-run.
|
||||
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
|
||||
regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
|
||||
update `inventory.yaml`, push. The next sync (≤5 min) propagates.
|
||||
|
||||
### Getting onto Netbird
|
||||
|
||||
@@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread
|
||||
|
||||
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
|
||||
|
||||
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
|
||||
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../archive/knowledge/containers/106-auth-outpost.md). Only needed once per account.
|
||||
|
||||
**Path A — setup-key (headless/scripted onboarding):**
|
||||
|
||||
@@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr
|
||||
### DNS prerequisite
|
||||
|
||||
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
|
||||
([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
|
||||
([dns.md](../../archive/knowledge/infrastructure/dns.md)) for LAN clients, **but only if the
|
||||
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
|
||||
workstations don't by default. Options:
|
||||
|
||||
@@ -81,7 +81,7 @@ If DNS isn't an option at all, override the URLs at bootstrap time:
|
||||
|
||||
```bash
|
||||
sudo HOMELAB_GITEA_TOKEN=... \
|
||||
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
|
||||
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git \
|
||||
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
|
||||
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
|
||||
bash /tmp/bootstrap.sh --with-mcp
|
||||
@@ -129,7 +129,7 @@ TOKEN=... # your Gitea PAT, scope read:repository
|
||||
|
||||
# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
|
||||
curl -fsSL -u "dtoro:$TOKEN" \
|
||||
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
|
||||
https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \
|
||||
-o /tmp/bootstrap.sh
|
||||
|
||||
# Run it.
|
||||
@@ -285,7 +285,7 @@ homelab client add my-new-machine
|
||||
# 4. On hubris: finalize the age public key.
|
||||
homelab client add my-new-machine --finalize-pubkey age1...
|
||||
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
|
||||
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
|
||||
# inventory.yaml, commits + pushes. The 5-min sync propagates.
|
||||
```
|
||||
|
||||
## Granting a secret to a new client
|
||||
@@ -333,7 +333,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
|
||||
| --- | --- | --- |
|
||||
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
|
||||
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
|
||||
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
|
||||
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git` |
|
||||
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
|
||||
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
|
||||
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
|
||||
@@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
|
||||
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
|
||||
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
|
||||
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
|
||||
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). |
|
||||
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../archive/knowledge/containers/106-auth-outpost.md). |
|
||||
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
|
||||
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
|
||||
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
|
||||
@@ -354,9 +354,9 @@ Added a new "Post-bootstrap: SSH reachability" section covering SSH key
|
||||
generation, pubkey publication, deployment to hosts, SSH config generation,
|
||||
and LAN IP registration. New workstations enrolled via this doc will
|
||||
automatically join the universal SSH mesh.
|
||||
### 2026-05-31 — cross-link to nomos-agent.md
|
||||
|
||||
### 2026-05-31 — cross-link to hermes-agent.md
|
||||
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
|
||||
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([nomos-agent.md](nomos-agent.md)) and noted it at the top of this page. The Nomos flow extends `bootstrap.sh` with `--with-nomos` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
|
||||
|
||||
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
|
||||
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Operations cheatsheet
|
||||
|
||||
Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
||||
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
||||
|
||||
## Proxmox CLI
|
||||
|
||||
@@ -8,13 +8,13 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
|
||||
| --- | --- |
|
||||
| `pct list` / `qm list` | List LXC containers / VMs |
|
||||
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
||||
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) |
|
||||
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
|
||||
| `pct enter <id>` | Shell into a container |
|
||||
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
||||
| `pvesm status` | Storage pools status |
|
||||
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
||||
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
||||
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
||||
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Nomos cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
||||
| `pveversion` | PVE version |
|
||||
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
||||
|
||||
@@ -22,22 +22,22 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
|
||||
|
||||
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
||||
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
||||
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
||||
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
||||
|
||||
## Reverse proxy
|
||||
|
||||
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md).
|
||||
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
||||
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
|
||||
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
||||
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
||||
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
||||
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
|
||||
|
||||
## DNS
|
||||
|
||||
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
||||
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
||||
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
|
||||
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
|
||||
- See [DNS](../../knowledge/wiki/infrastructure/dns.md).
|
||||
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
|
||||
|
||||
## Web access
|
||||
|
||||
@@ -49,7 +49,7 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
|
||||
- `ras-mc-ctl --errors` — full event log
|
||||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
||||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
||||
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md))
|
||||
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/archive/2026-04-21-hubris-crash-loop.md))
|
||||
|
||||
## Fleet apt operations
|
||||
|
||||
@@ -77,15 +77,15 @@ See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
|
||||
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
|
||||
| `homelab decide <action> <entity>` | Decision classifier: risk × blast radius × confidence → auto-act or escalate |
|
||||
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
|
||||
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
|
||||
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Nomos, or the Oikos Console's `/approvals` page) |
|
||||
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
|
||||
|
||||
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md).
|
||||
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../archive/oikos-cards/).
|
||||
|
||||
## Related
|
||||
- [Hubris host](../../knowledge/wiki/hosts/hubris.md)
|
||||
- [Containers index](../../knowledge/wiki/containers/index.md)
|
||||
- [DNS](../../knowledge/wiki/infrastructure/dns.md)
|
||||
- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md)
|
||||
- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)
|
||||
- [Hubris host](../../archive/knowledge/hosts/hubris.md)
|
||||
- [Containers index](../../archive/knowledge/containers/index.md)
|
||||
- [DNS](../../archive/knowledge/infrastructure/dns.md)
|
||||
- [Monitoring](../../archive/knowledge/infrastructure/monitoring.md)
|
||||
- [Auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)
|
||||
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Hermes agent — Nous-Hermes-powered Goose sessions on a homelab client
|
||||
# Nomos agent — LLM-powered terminal sessions on a homelab client
|
||||
|
||||
Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
|
||||
Llama variant) as a working terminal agent on a homelab client. Builds on top
|
||||
@@ -8,13 +8,13 @@ of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md))
|
||||
The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
|
||||
|
||||
- The chat loop, multi-turn history, and streaming
|
||||
- The OpenRouter provider that routes to Nous Hermes
|
||||
- The OpenRouter provider that routes to the configured LLM
|
||||
- The built-in `developer` extension (shell + file editor — same surface Claude
|
||||
Code has)
|
||||
- A remote MCP extension pointed at `mcp.hubris.network` for read-only
|
||||
homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
|
||||
|
||||
The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
|
||||
The persona is `/opt/homelab-context/NOMOS.md`, symlinked as Goose's global
|
||||
`.goosehints` so it's injected into the system prompt on every session.
|
||||
|
||||
## Prerequisites
|
||||
@@ -23,7 +23,7 @@ The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
|
||||
| --- | --- |
|
||||
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) |
|
||||
| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
|
||||
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` |
|
||||
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-nomos` |
|
||||
|
||||
## Onboarding flow
|
||||
|
||||
@@ -33,37 +33,37 @@ homelab client add new-machine
|
||||
|
||||
# 2. Join new-machine to Netbird (setup-key or OIDC).
|
||||
|
||||
# 3. On new-machine: bootstrap with --with-hermes.
|
||||
# 3. On new-machine: bootstrap with --with-nomos.
|
||||
TOKEN=... # gitea PAT, read:repository
|
||||
curl -fsSL -u "dtoro:$TOKEN" \
|
||||
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
|
||||
https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \
|
||||
-o /tmp/bootstrap.sh
|
||||
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-hermes
|
||||
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-nomos
|
||||
|
||||
# 4. Back on hubris: finalize the age pubkey AND grant the Hermes secret.
|
||||
# 4. Back on hubris: finalize the age pubkey AND grant the Nomos secret.
|
||||
homelab client add new-machine \
|
||||
--finalize-pubkey age1... \
|
||||
--with-hermes
|
||||
--with-nomos
|
||||
|
||||
# 5. Wait ≤5 min for sync, then on new-machine:
|
||||
hermes "what LXCs are running?"
|
||||
nomos "what LXCs are running?"
|
||||
```
|
||||
|
||||
The bootstrap `--with-hermes` flag does five things, all idempotent:
|
||||
The bootstrap `--with-nomos` flag does five things, all idempotent:
|
||||
|
||||
1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
|
||||
(upstream installer) and symlinks `/usr/local/bin/goose` to it.
|
||||
2. Symlinks `/opt/homelab-context/bin/hermes` → `/usr/local/bin/hermes`.
|
||||
3. Symlinks `/opt/homelab-context/HERMES.md` → `/root/HERMES.md` (Linux) or
|
||||
`/etc/HERMES.md` (macOS) for `cat`-as-operator convenience.
|
||||
2. Symlinks `/opt/homelab-context/bin/nomos` → `/usr/local/bin/nomos`.
|
||||
3. Symlinks `/opt/homelab-context/NOMOS.md` → `/root/NOMOS.md` (Linux) or
|
||||
`/etc/NOMOS.md` (macOS) for `cat`-as-operator convenience.
|
||||
4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
|
||||
extensions (preserves any keys the operator added by hand).
|
||||
5. Symlinks `~/.config/goose/.goosehints` → HERMES.md, so the persona is
|
||||
5. Symlinks `~/.config/goose/.goosehints` → NOMOS.md, so the persona is
|
||||
injected as the system prompt on every session.
|
||||
|
||||
## Seeding the OpenRouter key
|
||||
|
||||
The first time anyone enrolls with `--with-hermes`, the encrypted file
|
||||
The first time anyone enrolls with `--with-nomos`, the encrypted file
|
||||
`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
|
||||
existing recipient):
|
||||
|
||||
@@ -75,19 +75,19 @@ git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key'
|
||||
git -C /opt/homelab-context push
|
||||
```
|
||||
|
||||
Until this step happens, `hermes …` exits with `openrouter-api-key.yaml still
|
||||
Until this step happens, `nomos …` exits with `openrouter-api-key.yaml still
|
||||
contains the placeholder`. Subsequent enrollees get the real key automatically
|
||||
via `--with-hermes` (which adds them as a sops recipient on
|
||||
via `--with-nomos` (which adds them as a sops recipient on
|
||||
`secrets/openrouter-api-key.yaml`).
|
||||
|
||||
## Granting the OpenRouter key to an already-enrolled host
|
||||
|
||||
If a host was enrolled without `--with-hermes` and you want to add it later:
|
||||
If a host was enrolled without `--with-nomos` and you want to add it later:
|
||||
|
||||
```bash
|
||||
# On hubris:
|
||||
PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
|
||||
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-hermes
|
||||
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-nomos
|
||||
```
|
||||
|
||||
`--finalize-pubkey` is required by the existing flow even when the pubkey is
|
||||
@@ -101,12 +101,12 @@ re-run; only the secret recipient list changed.
|
||||
```bash
|
||||
homelab whoami # standard enrollment OK
|
||||
homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
|
||||
which goose && which hermes # binaries present
|
||||
which goose && which nomos # binaries present
|
||||
goose info -v # provider/model wiring sane
|
||||
hermes "what LXCs are running?" # interactive Goose session
|
||||
nomos "what LXCs are running?" # interactive Goose session
|
||||
|
||||
# Non-interactive smoke test:
|
||||
echo "List the homelab MCP tools you have available" | hermes
|
||||
echo "List the homelab MCP tools you have available" | nomos
|
||||
```
|
||||
|
||||
## Configuration
|
||||
@@ -135,9 +135,9 @@ extensions:
|
||||
Override via env on a single bootstrap run:
|
||||
|
||||
```bash
|
||||
HOMELAB_HERMES_MODEL=nousresearch/hermes-3-llama-3.1-405b \
|
||||
HOMELAB_HERMES_MCP_URI=https://mcp.hubris.network/mcp \
|
||||
sudo bash /tmp/bootstrap.sh --with-hermes
|
||||
HOMELAB_NOMOS_MODEL=nousresearch/hermes-3-llama-3.1-405b \
|
||||
HOMELAB_NOMOS_MCP_URI=https://mcp.hubris.network/mcp \
|
||||
sudo bash /tmp/bootstrap.sh --with-nomos
|
||||
```
|
||||
|
||||
Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
|
||||
@@ -156,33 +156,33 @@ every tool call, use `approve`. See
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
|
||||
| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
|
||||
| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
|
||||
| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py` — `mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
|
||||
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
|
||||
| `nomos: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-nomos` from hubris |
|
||||
| `nomos: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
|
||||
| Goose hangs on first `nomos` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
|
||||
| `homelab` extension fails to connect / no MCP tools listed | MCP server upgraded in Go rewrite (`internal/mcp/server.go`, Streamable HTTP via official MCP SDK). Old FastMCP SSE transport is deprecated. | Run `docker compose --profile full up` on mac-mini, or wait for the production cutover from apps/105. |
|
||||
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-nomos`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
|
||||
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
|
||||
|
||||
## Cross-references
|
||||
|
||||
- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the
|
||||
Hermes flow assumes is done.
|
||||
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
|
||||
Nomos flow assumes is done.
|
||||
- [`NOMOS.md`](../NOMOS.md) — the persona the Nomos agent reads on every
|
||||
session start (via `~/.config/goose/.goosehints`).
|
||||
- [`bin/hermes`](../../bin/hermes) — the wrapper that decrypts the OpenRouter key
|
||||
- [`bin/nomos`](../../bin/nomos) — the wrapper that decrypts the OpenRouter key
|
||||
and execs `goose session`.
|
||||
- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-hermes` flag's install block.
|
||||
- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-nomos` flag's install block.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
|
||||
(`"SSE transport is no longer supported - kept only for config file
|
||||
compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
|
||||
server at `mcp/server.py:336` still calls `mcp.run(transport="sse")`. Until
|
||||
server at `internal/mcp/server.go` uses Streamable HTTP (official MCP SDK). Until
|
||||
that's changed, the `homelab` MCP extension in Goose will fail to connect.
|
||||
The developer extension (shell + edit) covers most ops without it; this is
|
||||
a polish item, not a blocker.
|
||||
2. **Per-host OpenRouter keys** for billing attribution. Today all Hermes
|
||||
2. **Per-host OpenRouter keys** for billing attribution. Today all Nomos
|
||||
hosts share one key.
|
||||
3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
|
||||
directly — OpenRouter periodically rotates the underlying weights.
|
||||
@@ -204,7 +204,7 @@ templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token-
|
||||
efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
|
||||
Also created `tools/caveman/` with the wrapper script, JS renderer, and
|
||||
templates — the canonical source for all agent hosts.
|
||||
Captures the Hermes-on-Goose onboarding flow added in the same commit as
|
||||
`bootstrap.sh --with-hermes`, `bin/hermes`, the sops rule for
|
||||
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-hermes`
|
||||
Captures the Nomos-on-Goose onboarding flow added in the same commit as
|
||||
`bootstrap.sh --with-nomos`, `bin/nomos`, the sops rule for
|
||||
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-nomos`
|
||||
extension. MCP streamable_http migration is queued as follow-up #1.
|
||||
@@ -30,4 +30,4 @@ Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level p
|
||||
---
|
||||
|
||||
Source: https://github.com/JuliusBrussee/caveman
|
||||
Copy to `~/.hermes/skills/` for Hermes Agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.
|
||||
Copy to `~/.nomos/skills/` for Nomos agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# LLM Wiki — the documentation contract
|
||||
|
||||
How the narrative documentation in this repo is organized. The pattern is borrowed from the
|
||||
`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top
|
||||
`sources / wiki / index / log` model: a durable synthesized layer (`archive/knowledge/`) built on top
|
||||
of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an
|
||||
append-only operations log.
|
||||
|
||||
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
|
||||
generated `hosts/*.yaml`, `oikos/`, `mcp/`, `secrets/`, `bin/` — is not part of the wiki and never
|
||||
`secrets/`, `scripts/`, `bin/` — is not part of the wiki and never
|
||||
moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for the split.
|
||||
|
||||
## Layers
|
||||
@@ -14,7 +14,7 @@ moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for t
|
||||
- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference
|
||||
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
|
||||
Read them; do not rewrite them into other sources.
|
||||
- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per
|
||||
- **Wiki** (`archive/knowledge/`) is the synthesized, authoritative current-state layer: one page per
|
||||
node (`containers/`, `vms/`, host narratives) and per cross-cutting system (`infrastructure/`). A
|
||||
reader understands the topic from the wiki page without reading the sources.
|
||||
- **Index** (`index.md` / folder `README.md`) is a pure listing — every page in scope with a
|
||||
|
||||
@@ -160,7 +160,7 @@ session.** A change that touches a container page must also update:
|
||||
- The Caddy page site list (if the change affects `*.hubris.network` routing)
|
||||
- The DNS / ingress infrastructure pages (if the change affects routing)
|
||||
- The `hosts/{hubris,strong}.md` host page (if container count changes)
|
||||
- The `inventory.yaml` host entry (source of truth for the `hosts/*.yaml` generation)
|
||||
- The `inventory.yaml` host entry (single source of truth)
|
||||
- The `infrastructure/topology.md` (generated from inventory, but regen if needed)
|
||||
|
||||
The pattern of updating only one page and leaving stale references on others
|
||||
|
||||
@@ -36,7 +36,7 @@ Every doc-level page follows the same shape so a reader scans it in one pass.
|
||||
1. **One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`.
|
||||
2. **Opening definition.** First paragraph, 1–3 sentences, says what the thing is. No motivation, no marketing, no setup.
|
||||
3. **Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md).
|
||||
4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (`get_changelog` in `mcp/server.py`); keep the `### YYYY-MM-DD — title` shape.
|
||||
4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (Go MCP `get_changelog` in `internal/mcp/server.go`); keep the `### YYYY-MM-DD — title` shape.
|
||||
5. **Related links** only at the bottom, only when a reference cannot be woven inline.
|
||||
|
||||
## Section indexes (folder READMEs)
|
||||
@@ -58,7 +58,7 @@ duplicated prose, no narrative between the intro and the table.
|
||||
|
||||
## Diagrams
|
||||
|
||||
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it.
|
||||
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it. (Go DB-native topology generation planned.)
|
||||
- ASCII box diagrams are fine for small shape diagrams; keep them to one screen.
|
||||
|
||||
## Sourcing and cross-references
|
||||
|
||||
@@ -16,7 +16,7 @@ the full walkthrough; this runbook is the risk/lifecycle framing.
|
||||
|
||||
1. On any enrolled client: `homelab client add <hostname>` — appends a
|
||||
`hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned`
|
||||
→ `provisioning`, per [oikos/ontology.yaml](../../../oikos/ontology.yaml)),
|
||||
→ `provisioning`, per [seeds/ontology.yaml](../../../seeds/ontology.yaml)),
|
||||
commits + pushes.
|
||||
2. Netbird join is **optional, not a required step** — only needed for
|
||||
hosts that must be reachable off-LAN (workstations that roam, e.g.
|
||||
@@ -26,7 +26,7 @@ the full walkthrough; this runbook is the risk/lifecycle framing.
|
||||
routed `192.168.8.0/24` Netbird network resource. Skip this step for
|
||||
LAN-only nodes; do it (out-of-band, console or setup key) only for
|
||||
hosts that need independent off-LAN reachability.
|
||||
3. On the new host: run `bootstrap.sh` (add `--with-hermes` to also
|
||||
3. On the new host: run `bootstrap.sh` (add `--with-nomos` to also
|
||||
enroll the Hermes agent). This provisions `/etc/age/key.txt`, the
|
||||
sync timer, and prints an age pubkey.
|
||||
4. Back on an enrolled client: `homelab client add <hostname>
|
||||
@@ -40,4 +40,4 @@ Docs-update checklist: if the new host is an LXC/VM, add its narrative
|
||||
page under `containers/` or `vms/` and set `doc_page` in its inventory
|
||||
entry (host-level cards don't have a `doc_page` field yet — services do;
|
||||
narrative pages are still found via the generated `see_also` in
|
||||
`hosts/<name>.yaml`).
|
||||
`inventory.yaml`).
|
||||
|
||||
@@ -15,13 +15,13 @@ own repo) and get it live, safely.
|
||||
`config_repo`, its risk class, and the verification command to run
|
||||
after. If risk class requires approval (`config_mutation` or
|
||||
`destructive`), stop and get operator sign-off before editing — see
|
||||
`oikos/policy.yaml`.
|
||||
`seeds/policy.yaml`.
|
||||
2. Clone/pull the `config_repo` (never edit the backend's working tree
|
||||
directly — tracked configs change by commit + push, per
|
||||
[OIKOS.md](../../OIKOS.md) conventions).
|
||||
3. Make the change, commit, push to `main`.
|
||||
4. The Gitea webhook fires the deploy pipeline for that repo (see
|
||||
[infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for
|
||||
[infrastructure/auto-deploy.md](../../../archive/knowledge/infrastructure/auto-deploy.md) for
|
||||
the exact receiver/reload for this service).
|
||||
5. Run the preflight's verification command. If it fails, check
|
||||
`homelab service <name> log` for the reload/restart error.
|
||||
|
||||
@@ -15,11 +15,11 @@ links. Prose-voice rules are not machine-checkable — those stay a review respo
|
||||
Run from the repo root:
|
||||
|
||||
python3 .agents/skills/docs-lint/lint.py # default: knowledge/ .agents/ operations/ investigations/ plans/
|
||||
python3 .agents/skills/docs-lint/lint.py knowledge/wiki/containers/104-gitea.md
|
||||
python3 .agents/skills/docs-lint/lint.py archive/knowledge/containers/104-gitea.md
|
||||
|
||||
Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary
|
||||
list mirrors `writing-style.md`; update both together if the standard changes.
|
||||
|
||||
> **Known baseline.** `knowledge/wiki/containers/101-jellyfin.md` links into a sibling repo
|
||||
> **Known baseline.** `archive/knowledge/archive/knowledge/containers/101-jellyfin.md` links into a sibling repo
|
||||
> (`devops/homelab-authentik-admin`) that this checkout does not contain — expected, not a bug.
|
||||
> Any other broken link is a real regression; investigate before dismissing it as baseline noise.
|
||||
|
||||
@@ -9,7 +9,7 @@ transition: "provisioning -> active"
|
||||
|
||||
# Lifecycle: activate a node
|
||||
|
||||
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml). Requires: age key
|
||||
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml). Requires: age key
|
||||
enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
|
||||
ingress live if public, health check answering, doc page complete,
|
||||
ledger entry.
|
||||
@@ -32,5 +32,5 @@ ledger entry.
|
||||
Week 3's runbook automation lands).
|
||||
|
||||
Regenerate derived data: `python3 mcp/build_host_files.py && python3
|
||||
oikos/gen-topology.py` so `hosts/<name>.yaml`, the topology diagram, and
|
||||
inventory.yaml` so `inventory.yaml`, the topology diagram, and
|
||||
the context card all reflect the new state.
|
||||
|
||||
@@ -9,10 +9,10 @@ transition: "active -> deprecated"
|
||||
|
||||
# Lifecycle: deprecate a node
|
||||
|
||||
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml): a node keeps running
|
||||
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml): a node keeps running
|
||||
but takes no new dependents. **Completion condition: zero remaining
|
||||
inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a
|
||||
suggestion; `oikos/policy.yaml` `lifecycle_overrides.deprecated.refuse`
|
||||
suggestion; `seeds/policy.yaml` `lifecycle_overrides.deprecated.refuse`
|
||||
lists `new-inbound-edges` as refused going forward.
|
||||
|
||||
1. Set `state: deprecated` on the node.
|
||||
|
||||
@@ -10,7 +10,7 @@ transition: "deprecated -> destroyed"
|
||||
# Lifecycle: destroy a node
|
||||
|
||||
**Destructive.** Requires operator approval + typed confirmation phrase
|
||||
per `oikos/policy.yaml`. Requires (ontology): backups verified, secrets
|
||||
per `seeds/policy.yaml`. Requires (ontology): backups verified, secrets
|
||||
recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
|
||||
ledger entry.
|
||||
|
||||
@@ -33,7 +33,7 @@ ledger entry.
|
||||
data).
|
||||
7. `oikos/ledger.py append host:<name> destroy destructive --result ok`.
|
||||
8. Regenerate: `python3 mcp/build_host_files.py && python3
|
||||
oikos/gen-topology.py` — the node drops out of `hosts/*.yaml` and
|
||||
inventory.yaml` — the node drops out of `inventory.yaml` and
|
||||
appears in the topology doc's archaeology table.
|
||||
|
||||
If the destroy fails partway (e.g. secrets revoked but pct destroy
|
||||
|
||||
@@ -32,8 +32,8 @@ before returning to `active`.
|
||||
7. `homelab service <name> health` for every service the node hosts.
|
||||
8. Set `state: active`. Add a migration note to the node's doc page
|
||||
(old host/IP → new, date, phase reference) — this repo's convention
|
||||
for every past migration (see `containers/101-jellyfin.md`,
|
||||
for every past migration (see `archive/knowledge/containers/101-jellyfin.md`,
|
||||
`containers/129-house.md`).
|
||||
|
||||
Regenerate: `python3 mcp/build_host_files.py && python3
|
||||
oikos/gen-topology.py`.
|
||||
inventory.yaml`.
|
||||
|
||||
@@ -2,17 +2,17 @@
|
||||
name: lifecycle-provision-node
|
||||
risk_class: config_mutation
|
||||
inputs: [node_name, kind, storage_pool]
|
||||
verification: "grep 'state: provisioning' hosts/<name>.yaml"
|
||||
verification: "grep 'state: provisioning' inventory.yaml"
|
||||
docs_update_checklist: [doc_page_stub]
|
||||
transition: "planned -> provisioning"
|
||||
---
|
||||
|
||||
# Lifecycle: provision a node
|
||||
|
||||
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml) `lifecycle.transitions`.
|
||||
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml) `lifecycle.transitions`.
|
||||
Policy note: `provisioning` nodes get a lifecycle override —
|
||||
`config_mutation` actions downgrade to `reversible_low` because nothing
|
||||
depends on the node yet (see `oikos/policy.yaml` `lifecycle_overrides`).
|
||||
depends on the node yet (see `seeds/policy.yaml` `lifecycle_overrides`).
|
||||
|
||||
Requires (from ontology): inventory entry, IP reserved, storage pool
|
||||
chosen, doc page stub.
|
||||
|
||||
@@ -103,8 +103,8 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
|
||||
## Related
|
||||
|
||||
- [Operations cheatsheet](../../operations/commands.md)
|
||||
- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md)
|
||||
- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md)
|
||||
- [Auto-deploy pipelines](../../../archive/knowledge/infrastructure/auto-deploy.md)
|
||||
- [Hubris host page](../../../archive/knowledge/hosts/hubris.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ Goal: determine whether a service is actually healthy, without ad-hoc SSH.
|
||||
4. Cross-check blast radius: `homelab node <name> relations` — is this
|
||||
entity's own backend host healthy? A downstream failure (e.g. `strong`
|
||||
down) will show up here before the service's own logs explain anything.
|
||||
5. If the fix is a restart: classify first (`oikos/policy.yaml` —
|
||||
5. If the fix is a restart: classify first (`seeds/policy.yaml` —
|
||||
`service-restart` is `reversible_low` unless the service has a
|
||||
`service_overrides` entry, e.g. `caddy`/`dns` are `config_mutation`).
|
||||
Unattended agents may act on `reversible_low` without approval.
|
||||
|
||||
11
.claude/launch.json
Normal file
11
.claude/launch.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"version": "0.0.1",
|
||||
"configurations": [
|
||||
{
|
||||
"name": "web",
|
||||
"runtimeExecutable": "npm",
|
||||
"runtimeArgs": ["--prefix", "web", "run", "dev"],
|
||||
"port": 5173
|
||||
}
|
||||
]
|
||||
}
|
||||
24
.gitignore
vendored
24
.gitignore
vendored
@@ -2,9 +2,25 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Regenerated every scheduler run (every 10 min); no audit value in the
|
||||
# diff. Signals (signals/*.jsonl) ARE tracked — this is just the ephemeral
|
||||
# health-probe cache. See oikos/scheduler.py.
|
||||
# Regenerated every scheduler run; ephemeral health-probe cache.
|
||||
oikos/state.json
|
||||
|
||||
.worktrees/
|
||||
# Compiled binaries (Go rewrite — bin/oikos, bin/nomos)
|
||||
bin/oikos
|
||||
bin/nomos
|
||||
oikos/oikos
|
||||
|
||||
# Legacy Python oikos (superseded by cmd/oikos Go binary — Phase 1-6 rewrite).
|
||||
# oikos/ kernel files are still imported by bin/homelab for operational CLI
|
||||
# commands (ssh, pct, logs, restart, status, open, secret, client, sync, mcp).
|
||||
# Remove oikos/* when bin/homelab is ported to Go.
|
||||
backups/
|
||||
.env
|
||||
.infisical-credentials
|
||||
|
||||
# Web UI (Svelte 5) — build artifacts. Ignore built output but keep the
|
||||
# .gitkeep placeholder so `//go:embed all:dist` (web/embed.go) compiles on a
|
||||
# fresh checkout before the UI is built.
|
||||
web/dist/*
|
||||
!web/dist/.gitkeep
|
||||
web/node_modules/
|
||||
|
||||
42
.sops.yaml
42
.sops.yaml
@@ -1,14 +1,12 @@
|
||||
# SOPS recipient rules.
|
||||
#
|
||||
# Each rule pins one or more age public keys as recipients for files matching
|
||||
# the path_regex. The build_host_files.py generator doesn't read this file;
|
||||
# `sops` does — to encrypt a new secret, run `sops -e secrets/<name>.yaml`
|
||||
# from the repo root and SOPS will pick the matching rule below.
|
||||
# SOPS is DR-fallback only — Infisical is the active secrets backend.
|
||||
# Files live in archive/secrets-sops-backup/ for cold recovery.
|
||||
# To encrypt a new DR secret: sops -e archive/secrets-sops-backup/<name>.yaml
|
||||
#
|
||||
# To grant a secret to a new client: add their age public key (from
|
||||
# To grant a DR secret to a new client: add their age public key (from
|
||||
# inventory.yaml `hosts.<name>.age_pubkey`) to the relevant rule below, then
|
||||
# run `sops updatekeys secrets/<name>.yaml` to re-encrypt without rotating
|
||||
# the ciphertext payload.
|
||||
# run `sops updatekeys archive/secrets-sops-backup/<name>.yaml` to re-encrypt.
|
||||
#
|
||||
# To revoke: remove the recipient from the relevant rule and run
|
||||
# `sops updatekeys` (this is what `homelab client remove` calls). Past
|
||||
@@ -16,7 +14,7 @@
|
||||
# underlying credential if compromise is suspected.
|
||||
|
||||
creation_rules:
|
||||
- path_regex: ^secrets/hello\.yaml$
|
||||
- path_regex: ^archive/secrets-sops-backup/hello\.yaml$
|
||||
# The "hello" secret is encrypted to every enrolled client so the bootstrap
|
||||
# decrypt test works for everyone. Add each new client's age_pubkey when
|
||||
# they enrol; re-key with `sops updatekeys -y secrets/hello.yaml`.
|
||||
@@ -29,7 +27,7 @@ creation_rules:
|
||||
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
|
||||
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
|
||||
|
||||
- path_regex: ^secrets/gitea-pat\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupgitea-pat\.yaml$
|
||||
# Write-scoped Gitea PAT (dtoro user). Same recipient list as hello.yaml
|
||||
# since every enrolled client should be able to push (homelab client
|
||||
# add/remove, wiki edits, etc.).
|
||||
@@ -42,17 +40,17 @@ creation_rules:
|
||||
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
|
||||
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
|
||||
|
||||
- path_regex: ^secrets/gitea-tokens\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupgitea-tokens\.yaml$
|
||||
# Workstations only.
|
||||
age: >-
|
||||
# placeholder — fill with age_pubkey of: republic-laptop, mac-mini, strong, hubris
|
||||
|
||||
- path_regex: ^secrets/webhook-hmacs\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupwebhook-hmacs\.yaml$
|
||||
# LXCs that run a webhook receiver.
|
||||
age: >-
|
||||
# placeholder — fill with age_pubkey of: apps, caddy
|
||||
|
||||
- path_regex: ^secrets/turn-shared-secret\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupturn-shared-secret\.yaml$
|
||||
# coturn TURN long-term-credential password. Consumed by hubris (which
|
||||
# renders /etc/turnserver.conf + /opt/management.json on the VPS via
|
||||
# `homelab render-vps-configs`). Other recipients are convenience for
|
||||
@@ -65,7 +63,7 @@ creation_rules:
|
||||
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
|
||||
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
|
||||
|
||||
- path_regex: ^secrets/netbird-authentik-oidc\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupnetbird-authentik-oidc\.yaml$
|
||||
# Authentik OIDC client secret for the netbird-dashboard provider.
|
||||
# Consumed by hubris to render /opt/management.json on the VPS
|
||||
# (PKCEAuthorizationFlow.ProviderConfig.ClientSecret).
|
||||
@@ -77,7 +75,7 @@ creation_rules:
|
||||
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
|
||||
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
|
||||
|
||||
- path_regex: ^secrets/netbird-pat\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupnetbird-pat\.yaml$
|
||||
# NetBird API Personal Access Token. Consumed by the dns-sync job on the
|
||||
# `dns` LXC (107) to reconcile Technitium -> NetBird managed DNS zone.
|
||||
# (When 107 is enrolled, add its age_pubkey here and updatekeys.)
|
||||
@@ -87,19 +85,19 @@ creation_rules:
|
||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
|
||||
|
||||
- path_regex: ^secrets/openrouter-api-key\.yaml$
|
||||
# OpenRouter API key consumed by the `hermes` wrapper (bin/hermes) when
|
||||
- path_regex: archive/secrets-sops-backupopenrouter-api-key\.yaml$
|
||||
# OpenRouter API key consumed by the `nomos` wrapper (bin/nomos) when
|
||||
# spawning a Goose session. Recipients are any host that should run a
|
||||
# Nous-Hermes agent. Add a host's age_pubkey here, then
|
||||
# Nomos agent. Add a host's age_pubkey here, then
|
||||
# `sops updatekeys -y secrets/openrouter-api-key.yaml`.
|
||||
# See operations/hermes-agent.md.
|
||||
# See operations/nomos-agent.md.
|
||||
age: >-
|
||||
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
|
||||
|
||||
- path_regex: ^secrets/yuvomi-api-token\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupyuvomi-api-token\.yaml$
|
||||
# Named Bearer token for the Yuvomi REST API, consumed by yuvomi-mcp on
|
||||
# LXC 129 (house).
|
||||
age: >-
|
||||
@@ -108,7 +106,7 @@ creation_rules:
|
||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
|
||||
|
||||
- path_regex: ^secrets/hermes-house-users\.yaml$
|
||||
- path_regex: archive/secrets-sops-backuphermes-house-users\.yaml$
|
||||
# Signal number → Yuvomi user_id mapping (PII). Consumed by hermesd on LXC 129.
|
||||
age: >-
|
||||
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||
@@ -116,7 +114,7 @@ creation_rules:
|
||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
|
||||
|
||||
- path_regex: ^secrets/oikos-approval-hmac\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupoikos-approval-hmac\.yaml$
|
||||
# HMAC signing key for Oikos approval-grant tokens (oikos/approve.py).
|
||||
# Recipients: apps (105, runs the approval engine alongside homelab-mcp)
|
||||
# and hubris (admin/debug decrypt). See OIKOS.md "Approval engine".
|
||||
@@ -124,7 +122,7 @@ creation_rules:
|
||||
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
|
||||
|
||||
- path_regex: ^secrets/oikos-console-deploy-secret\.yaml$
|
||||
- path_regex: archive/secrets-sops-backupoikos-console-deploy-secret\.yaml$
|
||||
# Shared HMAC secret for the Gitea deploy webhook (id 14) ->
|
||||
# oikos-console-deploy.service on apps (105). Generated + registered
|
||||
# with Gitea before the apps-side install ran (see
|
||||
|
||||
File diff suppressed because one or more lines are too long
170
AGENTS.md
170
AGENTS.md
@@ -4,25 +4,33 @@ You are running on a machine that is part of the **hubris** homelab. The full
|
||||
context is in this checkout at `/opt/homelab-context/`. This file is the entry
|
||||
point. Read it once at start, then keep working.
|
||||
|
||||
- **New client?** Read [CLIENTS.md](CLIENTS.md) first.
|
||||
- **Developing on this repo?** Also read [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
|
||||
|
||||
The operating model — OODA loop, risk classes, approval rules, the ontology,
|
||||
and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation,
|
||||
classify the action against `oikos/policy.yaml`; when the class requires
|
||||
classify the action against `seeds/policy.yaml`; when the class requires
|
||||
approval, stop and ask the operator.
|
||||
|
||||
Agent-facing instruction is separated from human content under `.agents/`:
|
||||
Agent-facing instruction lives under `.agents/`:
|
||||
`.agents/shared/` holds the conventions every agent applies
|
||||
([writing-style](.agents/shared/writing-style.md), [caveman](.agents/shared/caveman.md),
|
||||
[page-templates](.agents/shared/page-templates.md), [llm-wiki](.agents/shared/llm-wiki.md)), and
|
||||
`.agents/domains/` holds the per-domain schemas
|
||||
([knowledge](.agents/domains/knowledge/schema.md), [operations](.agents/domains/operations/schema.md)).
|
||||
The narrative wiki lives under `knowledge/wiki/`; the machine-readable substrate
|
||||
(`inventory.yaml`, `hosts/*.yaml`, `oikos/`) stays at the repo root.
|
||||
|
||||
**Source of truth:** The Postgres database is the single source of truth for all
|
||||
structured data and knowledge. It is bootstrapped from `seeds/` at deploy time:
|
||||
`seeds/ontology.yaml` (entity types, relationships, lifecycles), `seeds/inventory.yaml`
|
||||
(hosts, services, entities), `seeds/policy.yaml` (risk classes, approval rules), and
|
||||
`seeds/knowledge.yaml` (documents, investigations, runbooks). The old narrative wiki
|
||||
is archived at `archive/knowledge/` for historical reference.
|
||||
|
||||
## 1. Who you are
|
||||
|
||||
Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read:
|
||||
|
||||
/opt/homelab-context/hosts/<your-hostname>.yaml
|
||||
/opt/homelab-context/inventory.yaml
|
||||
|
||||
That file tells you your role, your peers, what's mounted, and what services
|
||||
you host. If it does not exist, this client was not enrolled — stop and tell
|
||||
@@ -31,91 +39,102 @@ the operator to run `homelab client add <hostname>` from an existing client.
|
||||
## 2. The topology
|
||||
|
||||
- `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation
|
||||
with their mesh addresses, roles, and service mappings. Treat this file as
|
||||
authoritative; anything you read in narrative pages should agree with it.
|
||||
- `/opt/homelab-context/knowledge/wiki/infrastructure/mesh.md` — Tailscale → Netbird state.
|
||||
Both meshes are accepted today; Netbird is preferred for new traffic.
|
||||
- `/opt/homelab-context/knowledge/wiki/infrastructure/dns.md` — split-horizon DNS via
|
||||
Technitium on [dns (107)](knowledge/wiki/containers/107-dns.md). `*.hubris.network`
|
||||
resolves to 192.168.x.x on the LAN and to mesh addresses off-LAN.
|
||||
with their mesh addresses, roles, and service mappings. This is the seed file;
|
||||
at runtime the DB is authoritative (query via MCP `get_entity` or the REST API).
|
||||
- `/opt/homelab-context/seeds/knowledge.yaml` — full narrative knowledge: 36
|
||||
documents, 6 investigations, 12 runbooks. Ingested into the DB on deploy.
|
||||
- `/opt/homelab-context/.agents/operations/commands.md` — the operator's cheatsheet
|
||||
for pct, caddy, DNS, and the Oikos command surface. Use these verbs when
|
||||
you take actions.
|
||||
for pct, caddy, DNS, and the Oikos command surface.
|
||||
|
||||
## 3. The MCP server
|
||||
|
||||
The homelab exposes a Model Context Protocol server with structured tools.
|
||||
Endpoint is in `inventory.yaml` under `services.homelab_mcp.endpoint`.
|
||||
Endpoint: `https://mcp.hubris.network/mcp`.
|
||||
|
||||
Available tools:
|
||||
Available tools (21 total):
|
||||
|
||||
Context (pure read):
|
||||
get_host(name), list_services(), find_service(name_or_role),
|
||||
get_topology(), search_docs(query), get_page(path),
|
||||
get_changelog(page, since?), whoami(hostname),
|
||||
list_my_secrets(caller_pubkey?)
|
||||
Context — observe + orient:
|
||||
get_entity(slug), list_entities(type, limit, cursor),
|
||||
get_relations(entity), get_blast_radius(entity),
|
||||
search_knowledge(query) — ILIKE search over documents, investigations,
|
||||
runbooks in the knowledge_entities table
|
||||
get_patterns(status, entity_type, action) — learned action patterns
|
||||
get_skills(status) — available automation skills
|
||||
|
||||
Management (read-only):
|
||||
get_service_status(service), tail_log(service, lines=200),
|
||||
list_lxcs(), get_lxc_state(lxc), ping_service(service)
|
||||
Management — live state:
|
||||
get_service_status(service_slug) — systemctl is-active on target host
|
||||
tail_log(service_slug, lines=200) — journalctl
|
||||
list_lxcs() — all LXC containers with ID, host, IP, health
|
||||
get_lxc_state(lxc_slug) — pct status from Proxmox host
|
||||
ping_service(service_slug) — HTTP reachability from entity_status
|
||||
|
||||
Oikos (read-only; see OIKOS.md):
|
||||
explain(service) — compact context card, cheaper than search_docs+get_page
|
||||
preflight(service) — risk class, approval requirement, verification command
|
||||
get_relations(entity) — ontology blast-radius query (host: or service: id)
|
||||
get_change_history(entity, limit=20) — change-ledger entries
|
||||
get_state_snapshot() — last scheduler Observe-pass (health, disk, drift count)
|
||||
Oikos — decisions:
|
||||
explain(service_slug) — compact context card (type, state, health, relations)
|
||||
preflight(service_slug, action) — risk class + approval requirement
|
||||
whoami(hostname) — entity record, peers, health for a client
|
||||
get_change_history(entity_slug, limit=20) — last audit-log entries per entity
|
||||
get_state_snapshot() — fleet health, disk, drift count
|
||||
|
||||
Mutations are **not** exposed via MCP. Use the `homelab` CLI for those, with
|
||||
operator confirmation — see OIKOS.md's risk classes and approval flow.
|
||||
Operations — observe + act:
|
||||
get_health_summary() — fleet health counts (healthy/degraded/down/unknown)
|
||||
get_signal_history(entity_slug, state, limit) — open + recent signals
|
||||
get_audit_trail(entity_id) — audit log filter + browse
|
||||
get_agent_activity(limit) — agent self-inspection
|
||||
query_metrics(hours=24) — time-series metric bucketed averages
|
||||
get_trend(entity_id, days=7) — metric slope over time
|
||||
get_event_timeline(severity, entity_slug, limit) — recent events
|
||||
|
||||
**When to prefer MCP over grepping the clone:** any time you need to resolve a
|
||||
name to an address, look up service status, or search the wiki by content.
|
||||
Grep is fine for browsing or when MCP is unreachable.
|
||||
Execution — the single mutation path:
|
||||
request_execution(target, action, params) — policy-gated.
|
||||
reversible_low (restart, reload, pct_exec, apt audit) runs immediately;
|
||||
config_mutation (systemctl enable/disable, apt upgrade) queues for operator
|
||||
approval via Matrix, then executes on ✅.
|
||||
get_execution_status(execution_id) — poll progress
|
||||
|
||||
## 4. Wiki conventions
|
||||
**When to prefer MCP over grepping the clone:** always for knowledge queries.
|
||||
`search_knowledge("jellyfin hardware acceleration")` returns ranked results from
|
||||
the DB with entity links. `get_entity_knowledge("lxc:jellyfin")` returns documents,
|
||||
runbooks, and investigations in one call. Grep the clone only when MCP is
|
||||
unreachable.
|
||||
|
||||
See [page-templates.md](.agents/shared/page-templates.md) for file naming, page
|
||||
structure, and the tone standard. Quick reference:
|
||||
## 4. Knowledge conventions
|
||||
|
||||
- **File naming:** Foundational docs are ALL-CAPS (AGENTS.md, OIKOS.md, GLOSSARY.md);
|
||||
containers use `<id>-<name>.md`; infrastructure uses lowercase-with-dashes; plans
|
||||
and investigations use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`.
|
||||
- **Where pages live:** Narrative under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`;
|
||||
incident records under `knowledge/sources/investigations/`; runbook procedures under
|
||||
`.agents/skills/<name>/SKILL.md`; operator reference under `.agents/operations/`;
|
||||
design docs under `plans/`. Cross-link liberally; orphans are bugs.
|
||||
- **Changelog format:** Every page ends with a `## Changelog` section, entries in
|
||||
reverse-chronological order:
|
||||
All narrative knowledge (documents, investigations, runbooks) lives in the DB
|
||||
(`knowledge_entities` table) and is seeded from `seeds/knowledge.yaml`. Agents
|
||||
can register new knowledge via the API:
|
||||
|
||||
### YYYY-MM-DD — short title
|
||||
one or two lines describing what changed and why.
|
||||
```
|
||||
POST /api/v1/knowledge/{entity_slug}
|
||||
{"title": "...", "content": "...", "tags": ["..."]}
|
||||
```
|
||||
|
||||
The DB is the truth. The old wiki files are in `knowledge/wiki/` pending archive
|
||||
per the DB-as-source-of-truth plan.
|
||||
|
||||
- **Runbook procedures** live as `runbook` entities in the DB and as SKILL.md
|
||||
files under `.agents/skills/<name>/`. They carry `risk_class`, `procedure`
|
||||
(JSON-schema-validated), and are linked to entity types via `applies_to_type`.
|
||||
- **Investigations** are `investigation` entities linked to affected entities
|
||||
via `about` edges.
|
||||
- **Documents** are `document` entities linked to entities via `documents` edges.
|
||||
They carry `at_glance` (structured attributes) and `changelog` (parsed entries).
|
||||
- **Live state precedence.** If you observe a discrepancy between the docs and
|
||||
running state, update the docs *in the same session* (per the same-session update
|
||||
rule in [page-templates.md](.agents/shared/page-templates.md#same-session-update-rule)).
|
||||
running state, update the DB *in the same session* via the API. The `oikos export`
|
||||
command regenerates `seeds/knowledge.yaml` for version control.
|
||||
|
||||
## 5. Acting on the homelab
|
||||
|
||||
- **Read state**: prefer MCP tools, then files, then shell. Examples:
|
||||
`homelab whoami`, `homelab list`, `homelab status`, `homelab logs caddy`.
|
||||
- **Cross-host actions** (caddy reload, pct exec, etc.): use the `homelab`
|
||||
CLI — it resolves hostname → mesh address → ssh / pct path for you. Direct
|
||||
SSH still works; the CLI just removes the lookup burden.
|
||||
- **Secrets**: never hardcode. Call `homelab secret <name>` to decrypt on
|
||||
demand using the per-client age key at `/etc/age/key.txt`. Secrets ARE
|
||||
available in this system — `list_my_secrets()` (MCP) shows what you can
|
||||
decrypt.
|
||||
- **Mutations** (restart, edit configs, etc.): classify against
|
||||
`oikos/policy.yaml` first (`homelab decide <action> <entity>`).
|
||||
`reversible_low` actions just need the interactive confirmation prompt;
|
||||
`config_mutation`/`destructive` actions are mechanically refused without
|
||||
- **Read state**: use MCP tools. Nomos (the AI agent) is the primary
|
||||
operator interface — it has 21 MCP tools for observe/orient/decide/act.
|
||||
- **Actions** (restart, logs, apt, pct exec): Nomos calls `request_execution`
|
||||
via MCP. `reversible_low` actions execute immediately; `config_mutation`
|
||||
and `destructive` actions are queued for operator approval via Matrix.
|
||||
- **Secrets**: managed by Infisical (`oikos secret` subcommand for migration).
|
||||
Never hardcode secrets — use env vars from `.env`.
|
||||
- **Mutations** (restart, edit configs, etc.): classified against
|
||||
`seeds/policy.yaml`. `reversible_low` actions auto-execute;
|
||||
`config_mutation`/`destructive` actions require approval.
|
||||
a valid `--approval-id` from `homelab approval request` — see OIKOS.md.
|
||||
For ad-hoc work, SSH and edit directly — but commit changes that touch
|
||||
tracked configs (caddy, gitea custom, artifacto, mule-image, etc.; see
|
||||
`knowledge/wiki/infrastructure/auto-deploy.md`).
|
||||
- **Wiki updates**: same-session rule applies to any meaningful state change
|
||||
this client makes.
|
||||
|
||||
## 6. Communication mode
|
||||
|
||||
@@ -133,10 +152,10 @@ Currently auto-setup:
|
||||
- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm
|
||||
package, wrapper scripts, and compact output templates for token-efficient
|
||||
CLI output. Wrapper at `~/bin/caveman_wrapper.sh`.
|
||||
- **Hermes agent persona** (`tools/setup-hermes-soul.sh`): Provisions
|
||||
`~/.hermes/SOUL.md` from `HERMES.md` on Hermes agents. This ensures every
|
||||
Hermes agent follows the canonical homelab persona (token efficiency, source
|
||||
of truth hierarchy). No-op on non-Hermes agents.
|
||||
- **Nomos agent persona** (`tools/setup-nomos-soul.sh`): Provisions
|
||||
`~/.nomos/SOUL.md` from `NOMOS.md` on Nomos agents. This ensures every
|
||||
Nomos agent follows the canonical homelab persona (token efficiency, source
|
||||
of truth hierarchy). No-op on non-Nomos agents.
|
||||
|
||||
To add a new auto-setup, create `tools/<name>.setup.sh` in the repo,
|
||||
commit and push. All enrolled clients pick it up within 5 minutes.
|
||||
@@ -145,5 +164,6 @@ To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer.
|
||||
|
||||
## 8. When in doubt
|
||||
|
||||
Run `homelab mcp search_docs <query>` or `homelab mcp get_host <name>`.
|
||||
The clone is the fallback; MCP is the index.
|
||||
Use MCP tools: `search_knowledge <query>` for narrative context,
|
||||
`get_entity <slug>` for structured data, `get_entity_knowledge <slug>` for
|
||||
everything linked to an entity. The clone is the fallback; MCP is the index.
|
||||
139
CLIENTS.md
Normal file
139
CLIENTS.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Oikos — client guide
|
||||
|
||||
If you are a homelab machine, this is what Oikos is and what it gives you.
|
||||
If you are an AI agent running on this machine, also read [AGENTS.md](AGENTS.md).
|
||||
|
||||
## What is Oikos?
|
||||
|
||||
Oikos is the agentic operating system for the **hubris** Proxmox homelab. It
|
||||
observes state, classifies actions against policy, executes approved
|
||||
procedures, learns from outcomes, and notifies the operator. It runs as a
|
||||
Docker stack on mac-mini and exposes an MCP server + REST API.
|
||||
|
||||
## What Oikos provides
|
||||
|
||||
| Capability | How you access it |
|
||||
|------------|-------------------|
|
||||
| Entity query (topology, blast radius) | MCP `get_entity`, `get_blast_radius` |
|
||||
| Full-text knowledge search | MCP `search_knowledge` |
|
||||
| Service status + logs | MCP `get_service_status`, `tail_log` |
|
||||
| LXC inventory + state | MCP `list_lxcs`, `get_lxc_state` |
|
||||
| Context cards | MCP `explain` |
|
||||
| Pre-flight risk classification | MCP `preflight` |
|
||||
| Change history | MCP `get_change_history` |
|
||||
| State snapshot (health, disk, drift) | MCP `get_state_snapshot` |
|
||||
| Secrets (Infisical) | REST API + `oikos secret` CLI |
|
||||
| Approval tokens | Matrix via notifier |
|
||||
|
||||
All MCP tools are read-only. Mutations use the `homelab` CLI with operator
|
||||
approval.
|
||||
|
||||
## Enrollment
|
||||
|
||||
Thin client model — no git clone, no sync timer. `bootstrap.sh` fetches only
|
||||
the agent orientation files and tooling from the raw Gitea URL, then enrolls
|
||||
via the Oikos API.
|
||||
|
||||
To enroll:
|
||||
```bash
|
||||
# Run from any machine with mesh connectivity
|
||||
curl -fsSL https://git.hubris.network/dtoro/oikos/raw/main/bootstrap.sh | sudo bash
|
||||
|
||||
# Or with optional tooling:
|
||||
curl ... | sudo bash -s -- --with-mcp # wire Claude's MCP config
|
||||
curl ... | sudo bash -s -- --with-nomos # install Goose + Nomos
|
||||
```
|
||||
|
||||
This calls `POST /api/v1/clients/enroll` on the Oikos API, which:
|
||||
1. Validates the entity exists in DB (planned or provisioning state)
|
||||
2. Validates mesh IP against expected subnets
|
||||
3. Generates an age keypair and delivers it to the client
|
||||
4. Creates an Infisical machine identity
|
||||
5. Transitions the entity to provisioning state
|
||||
|
||||
## After enrollment
|
||||
|
||||
### What changes on your machine
|
||||
|
||||
- `/opt/homelab/` — agent orientation files (CLIENTS.md, AGENTS.md, OIKOS.md)
|
||||
- `/opt/homelab/tools/` — tooling scripts (caveman, nomos-soul)
|
||||
- `/etc/age/key.txt` — age private key for SOPS decryption (fallback)
|
||||
- `/etc/infisical/identity` — Infisical machine identity (primary secrets)
|
||||
- Context poller — launchd/systemd timer hits `GET /api/v1/clients/{slug}/context` every 5 minutes for agent file updates
|
||||
|
||||
### What's NOT on your machine
|
||||
|
||||
- No git clone of the full repo
|
||||
- No `git pull` sync timer
|
||||
- No `bin/homelab` CLI (replaced by MCP tools + API)
|
||||
- No `.sops.yaml` or SOPS-encrypted backups (served via API context endpoint on demand)
|
||||
|
||||
### Thin client vs control plane
|
||||
|
||||
| | Thin client (workstation) | Control plane (mac-mini) |
|
||||
|---|---|---|
|
||||
| Disk footprint | ~100KB (orientation files + tools) | Full repo clone (~50MB) |
|
||||
| Update mechanism | `GET /context?since=` poll | Git pull + post-pull.sh |
|
||||
| Source of truth | DB via MCP | DB + local seeds + archive |
|
||||
| Secrets access | Infisical (primary), age/SOPS served via API (fallback) | Infisical + local SOPS files |
|
||||
|
||||
### Your identity
|
||||
|
||||
Your identity in the homelab is defined in `inventory.yaml`. Run `hostname`
|
||||
(Linux) or `scutil --get LocalHostName` (macOS), then look up your entry.
|
||||
It tells you your role, what services you host, what's mounted, and your
|
||||
mesh address.
|
||||
|
||||
### Source of truth hierarchy
|
||||
|
||||
1. **Postgres database** (runtime) — authoritative for entities, knowledge,
|
||||
signals, ledger. Query via MCP or REST API.
|
||||
2. **Context poller** — agent files and tooling fetched via API deltas every
|
||||
5 minutes.
|
||||
3. **Never guess.** If data is missing, query MCP. If MCP is down, grep the
|
||||
local `/opt/homelab/` files.
|
||||
|
||||
## The context poller
|
||||
|
||||
Every 5 minutes, launchd (macOS) or systemd (Linux) hits:
|
||||
|
||||
```
|
||||
GET /api/v1/clients/ws:{hostname}/context?since={last_timestamp}
|
||||
```
|
||||
|
||||
The API returns which agent files, tools, and SOPS config changed since the
|
||||
last poll. Only changed files are downloaded. This replaces the old
|
||||
`git pull` with a lightweight HTTP delta.
|
||||
|
||||
To trigger manually: run `/opt/homelab/tools/context-poller.sh`.
|
||||
|
||||
## Making changes
|
||||
|
||||
- **Read state**: use MCP tools or the API
|
||||
- **Mutate state** (restart, edit config, deploy): classify the action against
|
||||
policy (query `preflight` MCP tool):
|
||||
- `read_only` / `reversible_low` — execute directly
|
||||
- `config_mutation` / `destructive` — request operator approval via
|
||||
`POST /api/v1/entities/{slug}/activate` (or equivalent lifecycle endpoint)
|
||||
- **Secrets**: use Infisical (primary) or SOPS (fallback). Never hardcode.
|
||||
- **Knowledge**: if you observe a discrepancy between docs and live state,
|
||||
update the DB via the API in the same session.
|
||||
|
||||
## MCP endpoint
|
||||
|
||||
```
|
||||
https://mcp.hubris.network/mcp
|
||||
```
|
||||
|
||||
Available tools are listed in [AGENTS.md](AGENTS.md#3-the-mcp-server).
|
||||
|
||||
## Communication mode
|
||||
|
||||
Apply [.agents/shared/caveman.md](.agents/shared/caveman.md) — terse,
|
||||
fragment-heavy communication. Drop filler, keep substance.
|
||||
|
||||
## Related
|
||||
|
||||
- [AGENTS.md](AGENTS.md) — full agent orientation (read this first)
|
||||
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, risk classes
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide for the Oikos codebase
|
||||
155
CONTRIBUTING.md
Normal file
155
CONTRIBUTING.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Contributing to Oikos
|
||||
|
||||
Developer guide for the Oikos codebase. If you are a homelab client consuming
|
||||
Oikos, see [CLIENTS.md](CLIENTS.md). If you are an AI agent working on the
|
||||
repo, see [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
|
||||
|
||||
## Dev setup
|
||||
|
||||
- **Go 1.26+** (see `go.mod` for pinned version)
|
||||
- **PostgreSQL with TimescaleDB** — the compose stack includes `timescale/timescaledb:2.17.2-pg16`
|
||||
- **Docker** for the full dev stack
|
||||
|
||||
```bash
|
||||
# Start dependencies (Postgres + Redis)
|
||||
docker compose --profile dev up -d
|
||||
|
||||
# Run all tests
|
||||
make test
|
||||
|
||||
# Run integration tests (needs compose Postgres)
|
||||
make test-db
|
||||
|
||||
# Build the binary
|
||||
make build
|
||||
```
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
cmd/oikos/ Single-binary entry point
|
||||
cmd/nomos/ Nomos MCP client gateway
|
||||
internal/ All Go packages
|
||||
httpapi/ REST + MCP server (OpenAPI-generated)
|
||||
mcp/ MCP tool implementations
|
||||
db/ Connection pool, migrations, seeds, sqlc queries
|
||||
scheduler/ Observe loop, probes, signals
|
||||
actuator/ SSH execution
|
||||
learning/ Pattern recognition, anomaly detection
|
||||
notifier/ Matrix notifications, approval tokens
|
||||
policy/ Risk classifier
|
||||
secrets/ Infisical + SOPS backend
|
||||
domain/ Core types: entities, approvals, signals, patterns
|
||||
ontology/ Type hierarchy, relationship validation
|
||||
knowledge/ Knowledge YAML seed ingestion
|
||||
api/openapi.yaml API contract — the source of truth for endpoints
|
||||
migrations/ Forward-only SQL migrations (TimescaleDB)
|
||||
seeds/ Bootstrap YAML: ontology, inventory, policy, knowledge
|
||||
compose/ Dockerfiles + Caddy config
|
||||
scripts/ Deploy, watchdog, rollback
|
||||
nomos/ Nomos config, persona, skills
|
||||
.agents/ Agent instruction files + skills
|
||||
plans/ Design documents
|
||||
docs/adr/ Architecture decision records
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `make build` | Build `oikos` binary |
|
||||
| `make test` | Run all tests with race detection |
|
||||
| `make test-db` | Run integration tests against compose Postgres |
|
||||
| `make lint` | `go vet` + `golangci-lint` |
|
||||
| `make generate` | Regenerate OpenAPI + sqlc code |
|
||||
| `make generate-check` | CI drift guard — fail if generated code is stale |
|
||||
| `make migrate` | Apply DB migrations |
|
||||
| `make seed` | Ingest seeds into DB |
|
||||
| `make export` | Export DB state to YAML seeds |
|
||||
| `make dev` | Start compose dev stack |
|
||||
| `make clean` | Remove binary + test cache |
|
||||
|
||||
## Conventions
|
||||
|
||||
### APIs are OpenAPI-first
|
||||
|
||||
The REST API is defined in `api/openapi.yaml`. Server code is generated with
|
||||
`oapi-codegen` into `internal/httpapi/gen/`. To add an endpoint:
|
||||
|
||||
1. Add the path + schema to `api/openapi.yaml`
|
||||
2. Run `make generate`
|
||||
3. Implement the handler in `internal/httpapi/impl.go`
|
||||
4. Add tests in `internal/httpapi/api_test.go`
|
||||
|
||||
Never hand-edit `internal/httpapi/gen/api.gen.go`.
|
||||
|
||||
### Database access is sqlc-first
|
||||
|
||||
SQL queries live in `internal/db/queries/*.sql`. Go code is generated with
|
||||
`sqlc` into `internal/db/sqlcgen/`. Config in `sqlc.yaml`.
|
||||
|
||||
- Queries target pgx/v5 with UUID + timestamptz overrides
|
||||
- Never hand-edit generated sqlc code
|
||||
|
||||
### Migrations are forward-only
|
||||
|
||||
SQL migrations live in `migrations/` as `NNN_name.up.sql`. There are no down
|
||||
migrations (see [ADR 0008](docs/adr/0008-forward-only-migrations.md)).
|
||||
Migrations are idempotent where possible (`IF NOT EXISTS`, `DO $$` blocks).
|
||||
|
||||
To add a migration:
|
||||
|
||||
1. Create `migrations/NNN_name.up.sql` with the next sequence number
|
||||
2. Write the DDL
|
||||
3. Run `make migrate` to apply
|
||||
|
||||
### Seeds are DB-generated
|
||||
|
||||
`seeds/*.yaml` are the bootstrap files used by `oikos seed`. After making
|
||||
changes via the API, run `make export` to regenerate the seed files. These
|
||||
files are version-controlled and serve as DR fallback.
|
||||
|
||||
### Writing style
|
||||
|
||||
Follow [.agents/shared/writing-style.md](.agents/shared/writing-style.md).
|
||||
Documentation is reference prose, not marketing. Banned vocabulary includes
|
||||
"robust", "seamless", "leverage", "utilize", "delve", "cutting-edge".
|
||||
|
||||
### Risk classification
|
||||
|
||||
Every mutation is classified against `seeds/policy.yaml` before execution.
|
||||
Four risk classes: `read_only`, `reversible_low`, `config_mutation`,
|
||||
`destructive`. The classifier can only lower autonomy relative to policy,
|
||||
never raise it. When in doubt, escalate.
|
||||
|
||||
## CI
|
||||
|
||||
Gitea Actions runs on push to `main` and pull requests (`ci.yml`):
|
||||
|
||||
1. `go vet` + `golangci-lint` + `govulncheck`
|
||||
2. Generated code drift check (`make generate-check`)
|
||||
3. Build (`go build ./...`)
|
||||
4. Test with race detector + coverage
|
||||
5. Docker build verification (no push)
|
||||
|
||||
Coverage gates: policy + learning packages ≥ 80%, others ≥ 60%.
|
||||
|
||||
## PR workflow
|
||||
|
||||
1. Create a branch from `main`
|
||||
2. Make changes, write tests
|
||||
3. Run `make lint test generate-check`
|
||||
4. Commit with a message following: problem → change → risk → verification
|
||||
5. Push to Gitea; CI gates PRs on green
|
||||
|
||||
## Secrets
|
||||
|
||||
Secrets are managed by Infisical (primary) with SOPS as DR fallback. Never
|
||||
hardcode secrets. Use environment variables from `.env` for local dev.
|
||||
The `.env` and `.infisical-credentials` files are gitignored.
|
||||
|
||||
## Related
|
||||
|
||||
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
|
||||
- [CLIENTS.md](CLIENTS.md) — for homelab clients consuming Oikos
|
||||
- [docs/adr/](docs/adr/) — architecture decision records
|
||||
268
README.md
268
README.md
@@ -1,190 +1,136 @@
|
||||
# Homelab OS
|
||||
# Oikos
|
||||
|
||||
Living documentation for the **hubris** Proxmox homelab + Oikos operating system.
|
||||
Agentic homelab operating system written in Go. Single binary (`cmd/oikos`),
|
||||
Docker-deployed on mac-mini, with a standalone Nomos MCP agent gateway
|
||||
(`cmd/nomos`). Manages the **hubris** Proxmox homelab autonomously — observes
|
||||
state, classifies actions against policy, executes approved procedures over SSH,
|
||||
learns from outcomes, and escalates when uncertain.
|
||||
|
||||
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md), then [OIKOS.md](.agents/OIKOS.md).
|
||||
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md).
|
||||
**For client machines:** see [CLIENTS.md](CLIENTS.md).
|
||||
**For developers:** see [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
---
|
||||
## Quick start
|
||||
|
||||
## For Agents — Navigation & Entry Points
|
||||
```bash
|
||||
# Dev stack (postgres + api + scheduler + notifier)
|
||||
docker compose --profile dev up -d
|
||||
|
||||
### You are running on a client enrolled in the hubris homelab
|
||||
# Full stack (adds Nomos agent gateway)
|
||||
docker compose --profile full up -d
|
||||
|
||||
1. **First:** Read [AGENTS.md](AGENTS.md) once. It explains who you are, the topology, available tools, conventions, and how to act.
|
||||
2. **Before any mutation:** Read [OIKOS.md](.agents/OIKOS.md). It defines the operating model, risk classes, approval flow, and the ontology you'll consult.
|
||||
3. **For specific workflows:** Load the matching skill from `.agents/skills/<name>/SKILL.md` (e.g., [service-health-check](.agents/skills/service-health-check/SKILL.md)).
|
||||
4. **When in doubt:** Use MCP tools (`search_docs`, `get_page`, `explain`, `get_changelog`) — they're cheaper and more reliable than grepping.
|
||||
# Build standalone binary
|
||||
go build -o bin/oikos -tags timetzdata ./cmd/oikos
|
||||
|
||||
### Key References for Agents
|
||||
|
||||
- **What am I?** → `/opt/homelab-context/hosts/<hostname>.yaml` (read on first run)
|
||||
- **Live topology** → `inventory.yaml` + `hosts/*.yaml` (canonical, always wins)
|
||||
- **Risk & approval** → [oikos/policy.yaml](oikos/policy.yaml) (enforced, not advisory)
|
||||
- **Runbooks & workflows** → [.agents/skills/](.agents/skills/) (risk class + verification checklist included)
|
||||
- **State of Oikos** → [OIKOS.md build status](.agents/OIKOS.md#build-status-30-day-roadmap) (scheduled probes, drift detectors, signals, approval engine)
|
||||
|
||||
### When to Use MCP vs Files vs Shell
|
||||
|
||||
| Task | Use | Tool |
|
||||
|------|-----|------|
|
||||
| Resolve hostname → address | MCP | `get_host(name)` or `list_services()` |
|
||||
| Search wiki by content | MCP | `search_docs(query)` |
|
||||
| Read a wiki page | MCP or file | `get_page(path)` or `cat knowledge/wiki/.../...md` |
|
||||
| Get changelog entries | MCP | `get_changelog(page, since?)` |
|
||||
| Understand a service | MCP | `explain(service)` — compact context card, cheaper than search+read |
|
||||
| Blast-radius query | MCP | `get_relations(entity)` (ontology walk) |
|
||||
| List available secrets | MCP | `list_my_secrets()` (scoped to your age key) |
|
||||
| Browse or grep | File | Raw `grep` when MCP unreachable, or exploratory browsing |
|
||||
|
||||
**When MCP is unreachable:** fall back to grepping the clone at `/opt/homelab-context/`. The local files are the same; MCP is just an index.
|
||||
|
||||
---
|
||||
|
||||
## Understanding the Operating Model
|
||||
|
||||
Before you act, **classify your action against [oikos/policy.yaml](oikos/policy.yaml)**.
|
||||
|
||||
### The Oikos OODA Loop + Decision Tree
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Observe["**Observe**<br/>probes, drift detectors, agent signals"]
|
||||
Orient["**Orient**<br/>ontology, context, state, entity relations"]
|
||||
Decide{"**Decide**<br/>classify against oikos/policy.yaml"}
|
||||
Auto["Auto-act<br/>(unattended)"]
|
||||
Escalate["Escalate<br/>homelab approval request"]
|
||||
Act["**Act**<br/>homelab CLI, runbooks, skills"]
|
||||
Verify["**Verify**<br/>checklist from SKILL.md"]
|
||||
Ledger["**Ledger**<br/>mutation record: who/what/risk"]
|
||||
Document["**Document**<br/>wiki update, same-session rule"]
|
||||
|
||||
Observe --> Orient --> Decide
|
||||
Decide -->|read_only, reversible_low| Auto
|
||||
Decide -->|config_mutation, destructive| Escalate
|
||||
Auto --> Act
|
||||
Escalate -->|approval granted| Act
|
||||
Act --> Verify --> Ledger --> Document
|
||||
Document -.loop.-> Observe
|
||||
# Run all roles in one process (dev mode)
|
||||
OIKOS_DATABASE_URL="postgres://oikos:oikos_dev@localhost:5432/oikos?sslmode=disable" \
|
||||
go run ./cmd/oikos all
|
||||
```
|
||||
|
||||
### Risk Classes (enforced, not advisory)
|
||||
## Architecture
|
||||
|
||||
From [oikos/policy.yaml](oikos/policy.yaml):
|
||||
```
|
||||
┌──────────────────────────────────┐
|
||||
│ mac-mini (Docker) │
|
||||
│ │
|
||||
Workstation ─── │ nomos (8092) ──MCP── api (8090) │
|
||||
(mesh) │ MCP gateway REST + MCP │
|
||||
│ │
|
||||
│ scheduler ── notifier ── postgres │
|
||||
│ (observe) (Matrix) (Timescale)│
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
- **read_only** — status, logs, docs, inventory queries. Unattended. MCP tools are all read_only.
|
||||
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger entry.
|
||||
- **config_mutation** — tracked-config edits (commit+push, never local), deploys, upgrades, DNS/ingress changes. **Operator approval required.**
|
||||
- **destructive** — destroy, format, wipe, rotate, revoke. **Approval + typed confirmation phrase.**
|
||||
| Component | Port | Role |
|
||||
|-----------|------|------|
|
||||
| `oikos api` | 8090 | REST API + MCP server (15 tools) |
|
||||
| `oikos scheduler` | — | Probe runner, signal lifecycle, metrics |
|
||||
| `oikos notifier` | — | Approval tokens, Matrix alerts |
|
||||
| `nomos serve` | 8092 | MCP client gateway, query routing |
|
||||
|
||||
### Decision Flow
|
||||
## Phases
|
||||
|
||||
1. **Decide:** Use `homelab decide <action> <entity>` to classify (risk class × blast radius × confidence).
|
||||
2. **Escalate if needed:** `homelab approval request` (Matrix-delivered to operator; see [operations/commands.md](.agents/operations/commands.md)).
|
||||
3. **Execute:** Use `homelab` CLI (not ad-hoc SSH) — it enforces policy, logs mutations, and verifies outcomes.
|
||||
4. **Document:** Update wiki in the same session (per [AGENTS.md §5](AGENTS.md#5-acting-on-the-homelab) and the [same-session rule](.agents/shared/page-templates.md#same-session-update-rule)).
|
||||
| Phase | Status | Description |
|
||||
|-------|--------|-------------|
|
||||
| 1 — Ontology + DB | ✅ | TimescaleDB, migrations, seeds, blast_radius |
|
||||
| 2 — API | ✅ | OpenAPI-first REST + MCP, auth, SSE, audit |
|
||||
| 3 — Control loop | ✅ | Scheduler, actuator, learning, classifier, notifier |
|
||||
| 4 — Nomos agent | ✅ | Standalone MCP client gateway, agent activity |
|
||||
| 5 — Secrets | ✅ | Infisical backend + SOPS fallback, rotation runbooks |
|
||||
| 6 — Deploy | ✅ | CI pipeline, cutover checklist, watchdog, rollback |
|
||||
|
||||
### The Ontology Graph
|
||||
Full plan: [plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md).
|
||||
|
||||
Everything that can break, be changed, or hold data has an entity in `inventory.yaml` + `oikos/ontology.yaml`. Blast-radius questions ("what breaks if strong goes down?") are graph walks via `homelab node <name> relations`, not doc archaeology.
|
||||
## Operations
|
||||
|
||||
**See:** [OIKOS.md](.agents/OIKOS.md) (full operating model, OODA loop, primitives, lifecycle gates, build status).
|
||||
### API endpoints
|
||||
|
||||
---
|
||||
```bash
|
||||
curl http://localhost:8090/api/v1/entities?type=service # fleet
|
||||
curl http://localhost:8090/api/v1/health # fleet health
|
||||
curl http://localhost:8090/api/v1/agent-activity # agent log
|
||||
```
|
||||
|
||||
## Finding & Understanding Information
|
||||
### Nomos queries
|
||||
|
||||
The narrative documentation is organized in **layers**:
|
||||
```bash
|
||||
# Structured tool call
|
||||
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
|
||||
-d '{"tool":"get_blast_radius","args":{"entity_id":"service:authentik"}}'
|
||||
|
||||
| Layer | What it is | Where | Immutable? | How agents use it |
|
||||
|-------|-----------|-------|-----------|-------------------|
|
||||
| **Sources** | Raw evidence: incidents, external refs, live state | `knowledge/sources/investigations/` | Yes | Read to understand root causes; do not rewrite |
|
||||
| **Wiki** | Synthesized current-state: one page per node & per system | `knowledge/wiki/{containers,hosts,vms,infrastructure}/` | No | This is the reference layer — if wiki disagrees with live state, update it *in the same session* |
|
||||
| **Index** | Pure listings — every page in scope with one-line summary | `index.md` / folder `README.md` | No | Navigation aid; keep it current when wiki restructures |
|
||||
| **Log** | Append-only doc-maintenance record (restructures, ingests, lints) | `knowledge/log.md` | Yes (append-only) | Read to understand past doc changes; never edit directly |
|
||||
# Natural language
|
||||
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
|
||||
-d '{"query":"what depends on authentik?"}'
|
||||
```
|
||||
|
||||
**Changelog ≠ Log:** Each wiki page ends with a `## Changelog` (infrastructure changes to that node, machine-parsed). That's not the Log; the Log records *doc operations* only.
|
||||
### CLI
|
||||
|
||||
**See:** [llm-wiki.md](.agents/shared/llm-wiki.md) (full rules, page structure, immutability contract).
|
||||
```bash
|
||||
oikos migrate # apply DB migrations
|
||||
oikos seed # ingest ontology/inventory/policy seeds
|
||||
oikos export # export DB state to YAML
|
||||
oikos api # serve REST + MCP
|
||||
oikos scheduler # run observe loop
|
||||
oikos notifier # run notification loop
|
||||
oikos all # all roles in one process
|
||||
oikos secret list # enumerate SOPS secrets
|
||||
oikos secret migrate # SOPS → Infisical
|
||||
```
|
||||
|
||||
---
|
||||
## Repo layout
|
||||
|
||||
## Map & Quick Navigation
|
||||
```
|
||||
cmd/oikos/ Go entry point — single binary
|
||||
cmd/nomos/ Nomos MCP client gateway
|
||||
internal/ Go packages (httpapi, mcp, scheduler, actuator, learning,
|
||||
notifier, policy, secrets, db, config, ontology, domain,
|
||||
knowledge)
|
||||
api/openapi.yaml API contract (OpenAPI 3.1)
|
||||
migrations/ Forward-only SQL migrations (TimescaleDB)
|
||||
seeds/ Bootstrap YAML (ontology, inventory, policy, knowledge)
|
||||
compose/ Dockerfiles + Caddy config
|
||||
scripts/ Deploy, watchdog, verification, rollback
|
||||
nomos/ Nomos config, persona, skills
|
||||
.agents/ Agent instruction files, shared conventions, skills
|
||||
archive/ Historical reference (legacy wiki, plans, SOPS backups)
|
||||
plans/ Design documents (active + done)
|
||||
docs/adr/ Architecture decision records
|
||||
```
|
||||
|
||||
### Agent Entry Points (Start Here)
|
||||
## For agents
|
||||
|
||||
- **You are an agent** → [AGENTS.md](AGENTS.md) (on deployed clients: `/opt/homelab-context/AGENTS.md`)
|
||||
- **Operating model & risk policy** → [OIKOS.md](.agents/OIKOS.md)
|
||||
- **Specific workflows** → [.agents/skills/](.agents/skills/) (load the matching SKILL.md before acting)
|
||||
- **Operations cheatsheet** → [.agents/operations/commands.md](.agents/operations/commands.md)
|
||||
- **Tools & MCP reference** → [AGENTS.md §3 — The MCP server](AGENTS.md#3-the-mcp-server)
|
||||
See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference:
|
||||
|
||||
### Topology & Infrastructure
|
||||
- **Source of truth:** DB (runtime) then seeds (bootstrap). Old wiki is
|
||||
archived at `archive/knowledge/` — use MCP `search_knowledge` instead.
|
||||
- **Mutations:** classify against policy, request approval for
|
||||
`destructive`/`config_mutation`
|
||||
- **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode
|
||||
|
||||
Node counts, IPs, and service lists change often — treat `inventory.yaml` and the index pages below as the source of truth, not this README.
|
||||
## Related
|
||||
|
||||
- **Proxmox hosts** → [knowledge/wiki/hosts/index.md](knowledge/wiki/hosts/index.md)
|
||||
- **VMs** → [knowledge/wiki/vms/index.md](knowledge/wiki/vms/index.md)
|
||||
- **LXC containers** → [knowledge/wiki/containers/index.md](knowledge/wiki/containers/index.md)
|
||||
- **Cross-cutting infrastructure** (DNS, ingress, mesh, backups, monitoring, auto-deploy, VPS) → [knowledge/wiki/infrastructure/index.md](knowledge/wiki/infrastructure/index.md)
|
||||
|
||||
### Knowledge & References
|
||||
|
||||
- **Glossary** — [GLOSSARY.md](knowledge/GLOSSARY.md)
|
||||
- **Incidents & investigations** — [knowledge/sources/investigations/index.md](knowledge/sources/investigations/index.md) (active + [archive](knowledge/sources/investigations/archive/))
|
||||
- **Plans & design docs** — [plans/index.md](plans/index.md)
|
||||
- **Hermes agent** (for Hermes-enrolled clients) — [HERMES.md](.agents/HERMES.md)
|
||||
|
||||
---
|
||||
|
||||
## Conventions
|
||||
|
||||
All pages follow:
|
||||
|
||||
- **File naming.** Foundational docs (entry-points, agent instruction, references) are ALL-CAPS (`AGENTS.md`, `OIKOS.md`, `GLOSSARY.md`); containers use `<id>-<name>.md`; infrastructure pages use lowercase-with-dashes; plans and incidents use `YYYY-MM-DD-slug.md`; skills are `<name>/SKILL.md`. See [page-templates.md](.agents/shared/page-templates.md#file-naming) for the full rules.
|
||||
- **Voice & vocabulary.** Concise, technical, sysadmin-to-sysadmin. No marketing prose, no puffers (seamless, robust, leverage, etc.). Full rules in [writing-style.md](.agents/shared/writing-style.md).
|
||||
- **Cross-linking is mandatory.** If a page references a node or system, link to it. Treat orphans as a bug.
|
||||
- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running state, fix the wiki *and* add a changelog entry *in the same session*.
|
||||
- **Tracked configs.** Pages for configs living in git repos (Caddy, Gitea, Artifacto, mule-image) must note the repo. Edits go through commit+push, never local changes. See [auto-deploy](knowledge/wiki/infrastructure/auto-deploy.md).
|
||||
- **No secrets.** This is a private repo, but still: reference secret *paths*, never secret *values*.
|
||||
|
||||
**For agents:** Read [caveman.md](.agents/shared/caveman.md) (terse communication standard). Use templates at [page-templates.md](.agents/shared/page-templates.md) when creating pages.
|
||||
|
||||
---
|
||||
|
||||
## Updating the Wiki
|
||||
|
||||
### When You Change Infrastructure
|
||||
|
||||
1. Update the relevant page (config snapshot, ports, mounts, IP address).
|
||||
2. Add a `### YYYY-MM-DD — title` entry to the page's `## Changelog` section (reverse chronological order).
|
||||
3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link from the changelog.
|
||||
4. If it's an incident, add a record to [`knowledge/sources/investigations/`](knowledge/sources/investigations/index.md).
|
||||
|
||||
### When You Restructure the Wiki
|
||||
|
||||
1. Update the relevant `index.md` / `README.md` in that section.
|
||||
2. Add a single-line entry to [`knowledge/log.md`](knowledge/log.md): `## [YYYY-MM-DD] <operation> | <summary>` (e.g., `## [2026-07-06] restructure | split infrastructure/dns into dns.md + dns-advanced.md`).
|
||||
|
||||
### The Same-Session Update Rule
|
||||
|
||||
**Any meaningful state change made in this session requires a wiki update before the session closes.** A change that touches a container page must also update:
|
||||
- The `containers/index.md` table (IPs, host, mounts, status)
|
||||
- The root `README.md` table (if affected)
|
||||
- The Caddy page site list (if affects `*.hubris.network` routing)
|
||||
- The DNS / ingress infrastructure pages (if affects routing)
|
||||
- The `hosts/hubris.md` or `hosts/strong.md` page (if container count changes)
|
||||
- The `inventory.yaml` host entry (source of truth for `hosts/*.yaml` generation)
|
||||
- The `knowledge/wiki/infrastructure/topology.md` (regenerate if needed)
|
||||
|
||||
Not updating all linked places is a bug. See [page-templates.md — same-session update rule](.agents/shared/page-templates.md#same-session-update-rule).
|
||||
|
||||
---
|
||||
|
||||
## More Information
|
||||
|
||||
- **For Hermes agents** → [HERMES.md](.agents/HERMES.md) (persona, source-of-truth hierarchy, token efficiency)
|
||||
- **For manual workflows** → [.agents/operations/](.agents/operations/) (commands cheatsheet, agent enrollment, Hermes guide)
|
||||
- **For skills/runbooks** → [.agents/skills/](.agents/skills/) (load the matching SKILL.md before acting; includes risk class + verification)
|
||||
- **MCP tools** → [AGENTS.md §3](AGENTS.md#3-the-mcp-server) (available tools, when to use MCP vs files)
|
||||
- **Page templates & voice** → [.agents/shared/](.agents/shared/) (page-templates.md, writing-style.md, caveman.md, llm-wiki.md)
|
||||
- **Machine-readable substrate** → `inventory.yaml`, `oikos/policy.yaml`, `oikos/ontology.yaml` (not part of the wiki; see [llm-wiki.md](.agents/shared/llm-wiki.md#rules))
|
||||
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
|
||||
- [CLIENTS.md](CLIENTS.md) — client onboarding guide
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide
|
||||
- [plans/](plans/) — design documents and cutover checklist
|
||||
- [docs/adr/](docs/adr/) — architecture decision records
|
||||
|
||||
412
api/openapi.yaml
412
api/openapi.yaml
@@ -41,6 +41,8 @@ servers:
|
||||
tags:
|
||||
- name: entities
|
||||
description: Inventory graph — entities and relationships
|
||||
- name: clients
|
||||
description: Client enrollment, context distribution, secrets
|
||||
- name: ontology
|
||||
description: Entity types, relationship types, lifecycles
|
||||
- name: signals
|
||||
@@ -310,6 +312,17 @@ paths:
|
||||
type: string
|
||||
style: form
|
||||
explode: true
|
||||
- name: include
|
||||
in: query
|
||||
schema:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
enum:
|
||||
- status
|
||||
style: form
|
||||
explode: true
|
||||
description: include=status joins entity_status and populates GraphView.health
|
||||
responses:
|
||||
'200':
|
||||
description: Graph view
|
||||
@@ -359,6 +372,129 @@ paths:
|
||||
type: integer
|
||||
default:
|
||||
$ref: '#/components/responses/Problem'
|
||||
/clients/enroll:
|
||||
post:
|
||||
tags:
|
||||
- clients
|
||||
operationId: enrollClient
|
||||
summary: Enroll a new client — issue age key, create Infisical identity
|
||||
description: >
|
||||
Validates mesh IP, generates an age keypair, creates an Infisical
|
||||
machine identity, and transitions the entity to provisioning.
|
||||
Caller must already have an entity in planned or provisioning state.
|
||||
x-required-scope: agent
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/EnrollRequest'
|
||||
responses:
|
||||
'200':
|
||||
description: Enrollment response with keys and identity
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/EnrollResponse'
|
||||
default:
|
||||
$ref: '#/components/responses/Problem'
|
||||
/clients/{slug}/context:
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/EntitySlug'
|
||||
get:
|
||||
tags:
|
||||
- clients
|
||||
operationId: getClientContext
|
||||
summary: Get agent context delta since a timestamp
|
||||
description: >
|
||||
Returns which agent files, tools, and SOPS config changed since the
|
||||
given timestamp. Thin clients poll this instead of git pull.
|
||||
x-required-scope: agent
|
||||
parameters:
|
||||
- name: since
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
format: date-time
|
||||
description: Return only changes since this timestamp (RFC 3339)
|
||||
responses:
|
||||
'200':
|
||||
description: Context delta
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ClientContext'
|
||||
default:
|
||||
$ref: '#/components/responses/Problem'
|
||||
/clients/{slug}/secrets:
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/EntitySlug'
|
||||
get:
|
||||
tags:
|
||||
- clients
|
||||
operationId: getClientSecrets
|
||||
summary: List secrets accessible to this client
|
||||
description: Infisical-secured secrets scoped to the client's machine identity
|
||||
x-required-scope: agent
|
||||
responses:
|
||||
'200':
|
||||
description: Secret keys accessible to this client
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ClientSecrets'
|
||||
default:
|
||||
$ref: '#/components/responses/Problem'
|
||||
/entities/provision:
|
||||
post:
|
||||
tags:
|
||||
- entities
|
||||
operationId: provisionEntity
|
||||
summary: Provision a compute entity (LXC, VM, container) on a host
|
||||
description: >
|
||||
Creates the entity in planned state, validates constraints (VMID, IP,
|
||||
capacity, template), classifies the action against policy, and
|
||||
transitions to provisioning on operator approval.
|
||||
x-required-scope: operator
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/IdempotencyKey'
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ProvisionRequest'
|
||||
responses:
|
||||
'201':
|
||||
description: Entity created, provisioning queued
|
||||
headers:
|
||||
ETag:
|
||||
$ref: '#/components/headers/ETag'
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ProvisionResponse'
|
||||
default:
|
||||
$ref: '#/components/responses/Problem'
|
||||
/entities/{slug}/provision/status:
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/EntitySlug'
|
||||
get:
|
||||
tags:
|
||||
- entities
|
||||
operationId: getProvisionStatus
|
||||
summary: Poll provisioning progress for a compute entity
|
||||
x-required-scope: viewer
|
||||
responses:
|
||||
'200':
|
||||
description: Provisioning steps with status
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ProvisionStatus'
|
||||
default:
|
||||
$ref: '#/components/responses/Problem'
|
||||
|
||||
/ontology:
|
||||
get:
|
||||
tags:
|
||||
@@ -888,6 +1024,9 @@ paths:
|
||||
- revoke
|
||||
note:
|
||||
type: string
|
||||
token:
|
||||
type: string
|
||||
description: HMAC approval token (single-use, verified server-side)
|
||||
responses:
|
||||
'200':
|
||||
description: Decision recorded
|
||||
@@ -1568,6 +1707,22 @@ paths:
|
||||
$ref: '#/components/schemas/HealthSummary'
|
||||
default:
|
||||
$ref: '#/components/responses/Problem'
|
||||
/dashboard/summary:
|
||||
get:
|
||||
tags:
|
||||
- observability
|
||||
operationId: getDashboardSummary
|
||||
summary: One-round-trip overview for the control room home page
|
||||
x-required-scope: viewer
|
||||
responses:
|
||||
'200':
|
||||
description: Dashboard summary
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/DashboardSummary'
|
||||
default:
|
||||
$ref: '#/components/responses/Problem'
|
||||
/export:
|
||||
get:
|
||||
tags:
|
||||
@@ -1658,6 +1813,13 @@ components:
|
||||
type: string
|
||||
maxLength: 128
|
||||
description: Client-generated key; replays within 24h return the original response
|
||||
EntitySlug:
|
||||
name: slug
|
||||
in: path
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
description: Entity slug (e.g. `ws:mac-mini`, `lxc:caddy`)
|
||||
headers:
|
||||
ETag:
|
||||
schema:
|
||||
@@ -1752,6 +1914,21 @@ components:
|
||||
updated_at:
|
||||
type: string
|
||||
format: date-time
|
||||
health:
|
||||
type: string
|
||||
description: last observed health, when the entity is monitored
|
||||
nullable: true
|
||||
enum:
|
||||
- healthy
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
- stale
|
||||
last_check_at:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
description: when health was last observed
|
||||
EntityCreate:
|
||||
type: object
|
||||
required:
|
||||
@@ -1848,6 +2025,17 @@ components:
|
||||
truncated:
|
||||
type: boolean
|
||||
description: True if node cap was hit
|
||||
health:
|
||||
type: object
|
||||
description: entity id -> health, present when include=status was requested
|
||||
additionalProperties:
|
||||
type: string
|
||||
enum:
|
||||
- healthy
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
- stale
|
||||
EntityType:
|
||||
type: object
|
||||
required:
|
||||
@@ -2072,6 +2260,7 @@ components:
|
||||
- disk
|
||||
- cert-expiry
|
||||
- drift
|
||||
- ping
|
||||
- ssh-script
|
||||
target:
|
||||
type: string
|
||||
@@ -2111,6 +2300,7 @@ components:
|
||||
- disk
|
||||
- cert-expiry
|
||||
- drift
|
||||
- ping
|
||||
- ssh-script
|
||||
target:
|
||||
type: string
|
||||
@@ -2857,6 +3047,9 @@ components:
|
||||
type: integer
|
||||
unknown:
|
||||
type: integer
|
||||
stale:
|
||||
type: integer
|
||||
description: last observation older than the check's expected cadence
|
||||
entities:
|
||||
type: array
|
||||
items:
|
||||
@@ -2877,6 +3070,7 @@ components:
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
- stale
|
||||
trend:
|
||||
type: string
|
||||
enum:
|
||||
@@ -2889,3 +3083,221 @@ components:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
DashboardSummary:
|
||||
type: object
|
||||
required:
|
||||
- entities_by_type
|
||||
- entities_by_state
|
||||
- health
|
||||
- signals_by_severity
|
||||
- approvals_pending
|
||||
- executions_by_state
|
||||
- event_rate
|
||||
properties:
|
||||
entities_by_type:
|
||||
type: object
|
||||
description: entity counts keyed by type
|
||||
additionalProperties:
|
||||
type: integer
|
||||
entities_by_state:
|
||||
type: object
|
||||
description: entity counts keyed by state
|
||||
additionalProperties:
|
||||
type: integer
|
||||
health:
|
||||
type: object
|
||||
required:
|
||||
- healthy
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
properties:
|
||||
healthy:
|
||||
type: integer
|
||||
degraded:
|
||||
type: integer
|
||||
down:
|
||||
type: integer
|
||||
unknown:
|
||||
type: integer
|
||||
stale:
|
||||
type: integer
|
||||
description: last observation older than the check's expected cadence
|
||||
signals_by_severity:
|
||||
type: object
|
||||
description: open (non-resolved) signal counts keyed by severity
|
||||
additionalProperties:
|
||||
type: integer
|
||||
approvals_pending:
|
||||
type: integer
|
||||
executions_by_state:
|
||||
type: object
|
||||
description: execution counts keyed by state, last 24h
|
||||
additionalProperties:
|
||||
type: integer
|
||||
event_rate:
|
||||
type: array
|
||||
description: event counts bucketed by 5-minute interval, most recent last
|
||||
items:
|
||||
type: object
|
||||
required:
|
||||
- bucket
|
||||
- count
|
||||
properties:
|
||||
bucket:
|
||||
type: string
|
||||
format: date-time
|
||||
count:
|
||||
type: integer
|
||||
EnrollRequest:
|
||||
type: object
|
||||
required:
|
||||
- slug
|
||||
properties:
|
||||
slug:
|
||||
type: string
|
||||
description: Entity slug (e.g. ws:new-laptop)
|
||||
hostname:
|
||||
type: string
|
||||
description: Actual hostname of the enrolling machine
|
||||
mesh_ip:
|
||||
type: string
|
||||
description: Source mesh IP for identity validation
|
||||
EnrollResponse:
|
||||
type: object
|
||||
required:
|
||||
- age_public_key
|
||||
- age_private_key
|
||||
- infisical_client_id
|
||||
- infisical_client_secret
|
||||
properties:
|
||||
age_public_key:
|
||||
type: string
|
||||
description: age1... public key for SOPS recipients
|
||||
age_private_key:
|
||||
type: string
|
||||
description: AGE-SECRET-KEY-... for local decryption
|
||||
infisical_client_id:
|
||||
type: string
|
||||
description: Infisical UniversalAuth client ID
|
||||
infisical_client_secret:
|
||||
type: string
|
||||
description: Infisical UniversalAuth client secret
|
||||
machine_identity_token:
|
||||
type: string
|
||||
description: Infisical machine identity access token
|
||||
ClientContext:
|
||||
type: object
|
||||
required:
|
||||
- version
|
||||
properties:
|
||||
version:
|
||||
type: integer
|
||||
description: Monotonic context version number
|
||||
agent_files_changed:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
description: Paths of agent instruction files that changed
|
||||
sops_config_changed:
|
||||
type: boolean
|
||||
description: True if .sops.yaml recipients changed
|
||||
tools_changed:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
description: Paths of tools/*.setup.sh that changed
|
||||
since:
|
||||
type: string
|
||||
format: date-time
|
||||
description: Timestamp for the next poll request
|
||||
ClientSecrets:
|
||||
type: object
|
||||
required:
|
||||
- keys
|
||||
properties:
|
||||
keys:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
description: Infisical secret keys accessible to this client
|
||||
ProvisionRequest:
|
||||
type: object
|
||||
required:
|
||||
- slug
|
||||
- type
|
||||
- name
|
||||
- host
|
||||
properties:
|
||||
slug:
|
||||
type: string
|
||||
description: e.g. lxc:jellyfin
|
||||
type:
|
||||
type: string
|
||||
description: Must be lxc, vm, or docker-container
|
||||
name:
|
||||
type: string
|
||||
description: Human-readable name
|
||||
host:
|
||||
type: string
|
||||
description: Slug of the Proxmox host (e.g. host:hubris)
|
||||
attributes:
|
||||
type: object
|
||||
description: VMID, cores, ram_mb, disk_gb, ip, template, mounts, services
|
||||
ProvisionResponse:
|
||||
type: object
|
||||
required:
|
||||
- entity
|
||||
- execution_id
|
||||
properties:
|
||||
entity:
|
||||
$ref: '#/components/schemas/Entity'
|
||||
execution_id:
|
||||
type: string
|
||||
format: uuid
|
||||
description: Execution ID to track provisioning progress
|
||||
ProvisionStatus:
|
||||
type: object
|
||||
required:
|
||||
- slug
|
||||
- state
|
||||
- steps
|
||||
properties:
|
||||
slug:
|
||||
type: string
|
||||
state:
|
||||
type: string
|
||||
description: Current entity state
|
||||
steps:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
required:
|
||||
- step
|
||||
- status
|
||||
properties:
|
||||
step:
|
||||
type: string
|
||||
status:
|
||||
type: string
|
||||
enum:
|
||||
- pending
|
||||
- running
|
||||
- ok
|
||||
- failed
|
||||
- skipped
|
||||
error_message:
|
||||
type: string
|
||||
nullable: true
|
||||
started_at:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
finished_at:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
error:
|
||||
type: string
|
||||
nullable: true
|
||||
description: Overall error if provisioning failed
|
||||
|
||||
@@ -45,7 +45,16 @@ Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webh
|
||||
### Portainer
|
||||
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
|
||||
|
||||
### homelab-mcp (`/opt/homelab-mcp/`)
|
||||
### homelab-mcp (`/opt/homelab-mcp/`) — DEPRECATED (Go rewrite, Phase 6)
|
||||
|
||||
> **Status:** This Python MCP server is being replaced by the Go `oikos api` binary
|
||||
> running in Docker on mac-mini. Cutover pending — see
|
||||
> [scripts/cutover-checklist.md](../../scripts/cutover-checklist.md) for the
|
||||
> execution plan. The Go MCP uses the official MCP Go SDK (Streamable HTTP, not
|
||||
> FastMCP) with 15 tools including `get_blast_radius`, `request_execution`, and
|
||||
> `get_agent_activity`. Source: `internal/mcp/server.go`.
|
||||
|
||||
**Current (Python) implementation — DO NOT MODIFY, awaiting cutover:**
|
||||
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
|
||||
`/opt/homelab-context/` (this LXC is itself an enrolled
|
||||
[homelab-context](../infrastructure/homelab-context.md) client). Source
|
||||
@@ -70,7 +79,10 @@ the server code). Listens on `0.0.0.0:9811`, secret in
|
||||
`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
|
||||
restarts on push.
|
||||
|
||||
### secrets-issuance (`/opt/secrets-issuance/`)
|
||||
### secrets-issuance (`/opt/secrets-issuance/`) — DEPRECATED (Go Phase 5)
|
||||
|
||||
> **Status:** Replaced by `internal/secrets/` in the Go rewrite. Machine identities
|
||||
> are now managed via Infisical (`docker compose --profile infisical up`).
|
||||
Tiny HTTP service that issues per-client age keypairs the first time
|
||||
each client calls `/issue`. Idempotent: subsequent calls return the
|
||||
same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in
|
||||
@@ -41,8 +41,8 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
|
||||
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
|
||||
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
|
||||
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
|
||||
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
|
||||
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
|
||||
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 (deprecated) | ~~reinstalls `homelab-mcp.service` + restart~~ → replaced by Go Docker stack on mac-mini |
|
||||
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 (deprecated) | ~~reinstalls `secrets-issuance.service` + restart~~ → replaced by `internal/secrets/` Go package |
|
||||
| `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` |
|
||||
| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) |
|
||||
|
||||
@@ -56,7 +56,7 @@ are documented in their own pages. Each system below links to its full doc.
|
||||
|
||||
- **[Topology diagram (generated)](topology.md)** — Mermaid graph of compute,
|
||||
ingress routing, and storage mounts. Auto-generated from `inventory.yaml`
|
||||
by `oikos/gen-topology.py`.
|
||||
by `oikos/gen-topology.py` (Python — Go DB-native replacement planned).
|
||||
|
||||
## Related
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user