cleanup: fix all stale references across .agents/ docs
- Rewrite AGENTS.md: DB as source of truth, MCP knowledge tools, archive refs - Fix OIKOS.md: seeds/ paths, remove Python-era notes, update deployment status - Fix commands.md, agent-enrollment.md: archive/knowledge/ links - Fix all SKILL.md files: remove hosts/*.yaml refs, point to inventory.yaml - Fix HERMES.md, schema.md, page-templates.md, llm-wiki.md: update paths - Fix bootstrap.sh: identity check reads inventory.yaml - Fix README.md, cutover-checklist.md: stale wiki references - Move convert-wiki.py to archive/ (one-shot done)
This commit is contained in:
@@ -8,7 +8,7 @@ conventions, and the source-of-truth hierarchy.
|
||||
|
||||
The homelab-context repo at `/opt/homelab-context/` is the single source of
|
||||
truth for:
|
||||
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
|
||||
- Fleet topology (`inventory.yaml`, `inventory.yaml`)
|
||||
- Service endpoints and credentials (via `homelab secret`)
|
||||
- Agent behaviour and conventions
|
||||
- Everything in this file
|
||||
@@ -22,7 +22,7 @@ deploy, client enrollment, incident investigation, and each node
|
||||
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
|
||||
acting. Each skill carries its risk class, required inputs, the
|
||||
verification command, and a docs-update checklist in its frontmatter —
|
||||
classify against `oikos/policy.yaml` using that risk class before any
|
||||
classify against `seeds/policy.yaml` using that risk class before any
|
||||
mutation. Don't re-derive topology or the mutation path by grepping the
|
||||
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
|
||||
operating model these runbooks execute inside (OODA loop, risk classes,
|
||||
|
||||
@@ -6,8 +6,8 @@ structure, the `homelab` CLI and MCP server are the syscall surface, and
|
||||
this page defines the rules everything above them follows.
|
||||
|
||||
Read this after [AGENTS.md](../AGENTS.md). Machine-readable companions:
|
||||
[oikos/ontology.yaml](../oikos/ontology.yaml) (systems model),
|
||||
[oikos/policy.yaml](../oikos/policy.yaml) (risk & approval).
|
||||
[seeds/ontology.yaml](../seeds/ontology.yaml) (systems model),
|
||||
[seeds/policy.yaml](../seeds/policy.yaml) (risk & approval).
|
||||
|
||||
## The kernel loop: OODA
|
||||
|
||||
@@ -41,16 +41,17 @@ one pass through **Observe → Orient → Decide → Act**:
|
||||
| Host / Service | topology entities | `inventory.yaml` |
|
||||
| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` |
|
||||
| Runbook | executable workflow with risk class + verification | `.agents/skills/<name>/SKILL.md` |
|
||||
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
|
||||
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
|
||||
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
|
||||
| Incident | investigation narrative | `knowledge/sources/investigations/` |
|
||||
| Signal | something needing attention, with lifecycle | DB `signals` table |
|
||||
| Change | one mutation: who, what, risk, approval, verification | DB `audit_log` + `executions` tables |
|
||||
| Approval | short-TTL signed grant for a gated action | DB `approvals` table |
|
||||
| Incident | investigation narrative | DB `knowledge_entities` (seeded from investigations) |
|
||||
| Knowledge | document, runbook, investigation | DB `knowledge_entities` (seeded from `seeds/knowledge.yaml`) |
|
||||
| Plan | design doc for non-trivial work | `plans/` |
|
||||
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
|
||||
|
||||
## Risk classes (enforced, not advisory)
|
||||
|
||||
From [oikos/policy.yaml](../oikos/policy.yaml):
|
||||
From [seeds/policy.yaml](../seeds/policy.yaml):
|
||||
|
||||
- **read_only** — status, logs, docs, inventory. Unattended.
|
||||
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger.
|
||||
@@ -69,7 +70,7 @@ Eight domains — physical, compute, network, storage, software,
|
||||
identity & access, operations, external — cover everything in the lab;
|
||||
entities are connected by typed edges (`hosts`, `provides`, `mounts`,
|
||||
`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …)
|
||||
defined in [oikos/ontology.yaml](../oikos/ontology.yaml). Rule of
|
||||
defined in [seeds/ontology.yaml](../seeds/ontology.yaml). Rule of
|
||||
completeness: **if it can break, be changed, or hold data, it has an
|
||||
entity and edges.** Blast-radius questions ("what breaks if strong goes
|
||||
down?") are graph walks, not doc archaeology.
|
||||
@@ -80,9 +81,9 @@ stored as `state:` in inventory (absent = active). Destroyed nodes live in
|
||||
the `archaeology:` section. Each transition is a runbook checklist;
|
||||
deprecation completes only when inbound edges reach zero.
|
||||
|
||||
Generated views: [infrastructure/topology.md](../knowledge/wiki/infrastructure/topology.md)
|
||||
(Mermaid, regenerated from inventory) and the live, clickable version at
|
||||
`oikos.hubris.network/graph` once the Console is deployed.
|
||||
Generated views: the live topology graph at `oikos.hubris.network/graph`
|
||||
via the API's `/api/v1/graph` endpoint, and the Mermaid export at
|
||||
`GET /api/v1/graph?format=mermaid`.
|
||||
|
||||
## Conventions carried forward
|
||||
|
||||
@@ -95,22 +96,24 @@ Generated views: [infrastructure/topology.md](../knowledge/wiki/infrastructure/t
|
||||
- Agents are terse ([caveman.md](shared/caveman.md)), verify claims, and fix
|
||||
collateral drift when found.
|
||||
|
||||
## Build status (Go rewrite — 2026-07-06 consolidation plan)
|
||||
## Build status (Go rewrite — deployed 2026-07-07)
|
||||
|
||||
The Oikos runtime was rewritten from Python to Go over 6 phases. See
|
||||
The Oikos runtime was rewritten from Python to Go over 6 phases and is deployed
|
||||
in Docker on mac-mini. See
|
||||
[plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](../plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md)
|
||||
for the full plan. The Python codebase (`oikos/*.py`, `mcp/server.py`,
|
||||
`oikos/console/`) is legacy — the Flask console and Python MCP server have
|
||||
been removed. The Python kernel files (`oikos/*.py`) remain only because
|
||||
`bin/homelab` still imports them for operational CLI commands.
|
||||
for the full plan. The Python codebase has been removed; all functionality runs
|
||||
in the Go binary.
|
||||
|
||||
**Phase 1 — Ontology + DB (DONE):**
|
||||
- `migrations/` (001–009): TimescaleDB hypertables, entity_status, CAGGs,
|
||||
retention policies. Forward-only, idempotent.
|
||||
- `seeds/{ontology,inventory,policy}.yaml`: DB-native bootstrap + DR export.
|
||||
- `migrations/` (001–011): TimescaleDB hypertables, entity_status, CAGGs,
|
||||
retention policies, knowledge entities with FTS. Forward-only, idempotent.
|
||||
- `seeds/{ontology,inventory,policy,knowledge}.yaml`: DB-native bootstrap +
|
||||
DR export. Knowledge seed contains 36 documents, 6 investigations, and 12
|
||||
runbooks.
|
||||
- `blast_radius()` SQL CTE, type hierarchy, abstract types, relationship
|
||||
validation.
|
||||
- Go packages: `internal/db/`, `internal/ontology/`, `internal/domain/`.
|
||||
- Go packages: `internal/db/`, `internal/ontology/`, `internal/domain/`,
|
||||
`internal/knowledge/`.
|
||||
|
||||
**Phase 2 — API (DONE):**
|
||||
- Single binary `cmd/oikos` with `oikos api` serving REST (:8090) + MCP
|
||||
@@ -164,11 +167,10 @@ been removed. The Python kernel files (`oikos/*.py`) remain only because
|
||||
- Cutover checklist: `scripts/cutover-checklist.md`.
|
||||
|
||||
**Current deployment:**
|
||||
- **Dev stack**: `docker compose --profile dev up` (postgres, api,
|
||||
scheduler, notifier) on mac-mini. Hermes runs with `--profile full`.
|
||||
- **Production**: apps/105 still hosts the Python MCP + secrets-issuance.
|
||||
Cutover checklist at `scripts/cutover-checklist.md` is the execution
|
||||
plan to move production to the Docker stack.
|
||||
- **Production**: Docker stack on mac-mini (`--profile full`: postgres, api,
|
||||
scheduler, notifier, hermes). Deployed 2026-07-07 with full knowledge seed.
|
||||
The Python MCP server and secrets-issuance on apps/105 have been stopped
|
||||
(see `scripts/cutover-checklist.md`).
|
||||
|
||||
## Python-era backlog (superseded)
|
||||
|
||||
|
||||
@@ -17,18 +17,18 @@ fixed paths, so the wiki reorganization never moves it.
|
||||
| Layer | Location | Consumed by |
|
||||
|-------|----------|-------------|
|
||||
| Substrate — source of truth | `inventory.yaml` (root) | MCP server, `homelab` CLI, `oikos/` scheduler/drift/relations/gen-topology |
|
||||
| Substrate — generated host records | `hosts/*.yaml` (root) | Go `internal/mcp/` server, `bin/homelab`; written by `mcp/build_host_files.py` |
|
||||
| Substrate — generated host records | `inventory.yaml` (root) | Go `internal/mcp/` server, `bin/homelab`; the single source of truth |
|
||||
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
|
||||
| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
|
||||
| Narrative — synthesized wiki | `archive/knowledge/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
|
||||
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
|
||||
|
||||
## Wiki pages
|
||||
|
||||
- **Node pages** (`knowledge/wiki/containers/<id>-<name>.md`, `.../vms/<id>-<name>.md`,
|
||||
- **Node pages** (`archive/knowledge/containers/<id>-<name>.md`, `.../vms/<id>-<name>.md`,
|
||||
`.../hosts/<name>.md`) follow the container/host template in
|
||||
[page-templates.md](../../shared/page-templates.md): opening definition, `## At a glance`,
|
||||
`## Role`, service/port map, storage, auto-deploy, `## Related`, `## Changelog`.
|
||||
- **Cross-cutting pages** (`knowledge/wiki/infrastructure/<topic>.md`) follow the cross-cutting
|
||||
- **Cross-cutting pages** (`archive/knowledge/infrastructure/<topic>.md`) follow the cross-cutting
|
||||
template: `## Why`, `## Components`, `## How to apply`, `## Gotchas`, `## Related`, `## Changelog`.
|
||||
- Each `inventory.yaml` host entry carries a `doc_page:` field pointing at its narrative page.
|
||||
Changing where a page lives means updating that field (read by `bin/homelab`).
|
||||
|
||||
@@ -40,7 +40,7 @@ transition: "<from> -> <to>" # only for lifecycle runbooks
|
||||
```
|
||||
|
||||
`risk_class` values and the lifecycle `transition` states must match
|
||||
[`oikos/policy.yaml`](../../../oikos/policy.yaml) and [`oikos/ontology.yaml`](../../../oikos/ontology.yaml).
|
||||
[`seeds/policy.yaml`](../../../seeds/policy.yaml) and [`seeds/ontology.yaml`](../../../seeds/ontology.yaml).
|
||||
|
||||
## Investigations
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ hostname doesn't match any inventory entry. Two fixes:
|
||||
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
|
||||
(Linux) or System Preferences → Sharing (macOS), then re-run.
|
||||
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
|
||||
regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
|
||||
update `inventory.yaml`, push. The next sync (≤5 min) propagates.
|
||||
|
||||
### Getting onto Netbird
|
||||
|
||||
@@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread
|
||||
|
||||
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
|
||||
|
||||
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
|
||||
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../archive/knowledge/containers/106-auth-outpost.md). Only needed once per account.
|
||||
|
||||
**Path A — setup-key (headless/scripted onboarding):**
|
||||
|
||||
@@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr
|
||||
### DNS prerequisite
|
||||
|
||||
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
|
||||
([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
|
||||
([dns.md](../../archive/knowledge/infrastructure/dns.md)) for LAN clients, **but only if the
|
||||
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
|
||||
workstations don't by default. Options:
|
||||
|
||||
@@ -285,7 +285,7 @@ homelab client add my-new-machine
|
||||
# 4. On hubris: finalize the age public key.
|
||||
homelab client add my-new-machine --finalize-pubkey age1...
|
||||
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
|
||||
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
|
||||
# inventory.yaml, commits + pushes. The 5-min sync propagates.
|
||||
```
|
||||
|
||||
## Granting a secret to a new client
|
||||
@@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
|
||||
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
|
||||
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
|
||||
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
|
||||
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). |
|
||||
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../archive/knowledge/containers/106-auth-outpost.md). |
|
||||
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
|
||||
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
|
||||
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Operations cheatsheet
|
||||
|
||||
Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
||||
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
||||
|
||||
## Proxmox CLI
|
||||
|
||||
@@ -8,13 +8,13 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
|
||||
| --- | --- |
|
||||
| `pct list` / `qm list` | List LXC containers / VMs |
|
||||
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
||||
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) |
|
||||
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
|
||||
| `pct enter <id>` | Shell into a container |
|
||||
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
||||
| `pvesm status` | Storage pools status |
|
||||
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
||||
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
||||
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
||||
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
||||
| `pveversion` | PVE version |
|
||||
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
||||
|
||||
@@ -22,22 +22,22 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
|
||||
|
||||
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
||||
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
||||
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
||||
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
||||
|
||||
## Reverse proxy
|
||||
|
||||
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md).
|
||||
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
||||
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
|
||||
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
||||
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
||||
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
||||
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
|
||||
|
||||
## DNS
|
||||
|
||||
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
||||
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
||||
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
|
||||
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
|
||||
- See [DNS](../../knowledge/wiki/infrastructure/dns.md).
|
||||
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
|
||||
|
||||
## Web access
|
||||
|
||||
@@ -49,7 +49,7 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
|
||||
- `ras-mc-ctl --errors` — full event log
|
||||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
||||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
||||
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md))
|
||||
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/archive/2026-04-21-hubris-crash-loop.md))
|
||||
|
||||
## Fleet apt operations
|
||||
|
||||
@@ -80,12 +80,12 @@ See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
|
||||
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
|
||||
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
|
||||
|
||||
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md).
|
||||
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../archive/oikos-cards/).
|
||||
|
||||
## Related
|
||||
- [Hubris host](../../knowledge/wiki/hosts/hubris.md)
|
||||
- [Containers index](../../knowledge/wiki/containers/index.md)
|
||||
- [DNS](../../knowledge/wiki/infrastructure/dns.md)
|
||||
- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md)
|
||||
- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)
|
||||
- [Hubris host](../../archive/knowledge/hosts/hubris.md)
|
||||
- [Containers index](../../archive/knowledge/containers/index.md)
|
||||
- [DNS](../../archive/knowledge/infrastructure/dns.md)
|
||||
- [Monitoring](../../archive/knowledge/infrastructure/monitoring.md)
|
||||
- [Auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)
|
||||
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# LLM Wiki — the documentation contract
|
||||
|
||||
How the narrative documentation in this repo is organized. The pattern is borrowed from the
|
||||
`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top
|
||||
`sources / wiki / index / log` model: a durable synthesized layer (`archive/knowledge/`) built on top
|
||||
of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an
|
||||
append-only operations log.
|
||||
|
||||
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
|
||||
generated `hosts/*.yaml`, `oikos/`, `mcp/`, `secrets/`, `bin/` — is not part of the wiki and never
|
||||
`secrets/`, `scripts/`, `bin/` — is not part of the wiki and never
|
||||
moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for the split.
|
||||
|
||||
## Layers
|
||||
@@ -14,7 +14,7 @@ moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for t
|
||||
- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference
|
||||
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
|
||||
Read them; do not rewrite them into other sources.
|
||||
- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per
|
||||
- **Wiki** (`archive/knowledge/`) is the synthesized, authoritative current-state layer: one page per
|
||||
node (`containers/`, `vms/`, host narratives) and per cross-cutting system (`infrastructure/`). A
|
||||
reader understands the topic from the wiki page without reading the sources.
|
||||
- **Index** (`index.md` / folder `README.md`) is a pure listing — every page in scope with a
|
||||
|
||||
@@ -160,7 +160,7 @@ session.** A change that touches a container page must also update:
|
||||
- The Caddy page site list (if the change affects `*.hubris.network` routing)
|
||||
- The DNS / ingress infrastructure pages (if the change affects routing)
|
||||
- The `hosts/{hubris,strong}.md` host page (if container count changes)
|
||||
- The `inventory.yaml` host entry (source of truth for the `hosts/*.yaml` generation)
|
||||
- The `inventory.yaml` host entry (single source of truth)
|
||||
- The `infrastructure/topology.md` (generated from inventory, but regen if needed)
|
||||
|
||||
The pattern of updating only one page and leaving stale references on others
|
||||
|
||||
@@ -16,7 +16,7 @@ the full walkthrough; this runbook is the risk/lifecycle framing.
|
||||
|
||||
1. On any enrolled client: `homelab client add <hostname>` — appends a
|
||||
`hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned`
|
||||
→ `provisioning`, per [oikos/ontology.yaml](../../../oikos/ontology.yaml)),
|
||||
→ `provisioning`, per [seeds/ontology.yaml](../../../seeds/ontology.yaml)),
|
||||
commits + pushes.
|
||||
2. Netbird join is **optional, not a required step** — only needed for
|
||||
hosts that must be reachable off-LAN (workstations that roam, e.g.
|
||||
@@ -40,4 +40,4 @@ Docs-update checklist: if the new host is an LXC/VM, add its narrative
|
||||
page under `containers/` or `vms/` and set `doc_page` in its inventory
|
||||
entry (host-level cards don't have a `doc_page` field yet — services do;
|
||||
narrative pages are still found via the generated `see_also` in
|
||||
`hosts/<name>.yaml`).
|
||||
`inventory.yaml`).
|
||||
|
||||
@@ -15,13 +15,13 @@ own repo) and get it live, safely.
|
||||
`config_repo`, its risk class, and the verification command to run
|
||||
after. If risk class requires approval (`config_mutation` or
|
||||
`destructive`), stop and get operator sign-off before editing — see
|
||||
`oikos/policy.yaml`.
|
||||
`seeds/policy.yaml`.
|
||||
2. Clone/pull the `config_repo` (never edit the backend's working tree
|
||||
directly — tracked configs change by commit + push, per
|
||||
[OIKOS.md](../../OIKOS.md) conventions).
|
||||
3. Make the change, commit, push to `main`.
|
||||
4. The Gitea webhook fires the deploy pipeline for that repo (see
|
||||
[infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for
|
||||
[infrastructure/auto-deploy.md](../../../archive/knowledge/infrastructure/auto-deploy.md) for
|
||||
the exact receiver/reload for this service).
|
||||
5. Run the preflight's verification command. If it fails, check
|
||||
`homelab service <name> log` for the reload/restart error.
|
||||
|
||||
@@ -15,11 +15,11 @@ links. Prose-voice rules are not machine-checkable — those stay a review respo
|
||||
Run from the repo root:
|
||||
|
||||
python3 .agents/skills/docs-lint/lint.py # default: knowledge/ .agents/ operations/ investigations/ plans/
|
||||
python3 .agents/skills/docs-lint/lint.py knowledge/wiki/containers/104-gitea.md
|
||||
python3 .agents/skills/docs-lint/lint.py archive/knowledge/containers/104-gitea.md
|
||||
|
||||
Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary
|
||||
list mirrors `writing-style.md`; update both together if the standard changes.
|
||||
|
||||
> **Known baseline.** `knowledge/wiki/containers/101-jellyfin.md` links into a sibling repo
|
||||
> **Known baseline.** `archive/knowledge/archive/knowledge/containers/101-jellyfin.md` links into a sibling repo
|
||||
> (`devops/homelab-authentik-admin`) that this checkout does not contain — expected, not a bug.
|
||||
> Any other broken link is a real regression; investigate before dismissing it as baseline noise.
|
||||
|
||||
@@ -9,7 +9,7 @@ transition: "provisioning -> active"
|
||||
|
||||
# Lifecycle: activate a node
|
||||
|
||||
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml). Requires: age key
|
||||
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml). Requires: age key
|
||||
enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
|
||||
ingress live if public, health check answering, doc page complete,
|
||||
ledger entry.
|
||||
@@ -32,5 +32,5 @@ ledger entry.
|
||||
Week 3's runbook automation lands).
|
||||
|
||||
Regenerate derived data: `python3 mcp/build_host_files.py && python3
|
||||
oikos/gen-topology.py` so `hosts/<name>.yaml`, the topology diagram, and
|
||||
inventory.yaml` so `inventory.yaml`, the topology diagram, and
|
||||
the context card all reflect the new state.
|
||||
|
||||
@@ -9,10 +9,10 @@ transition: "active -> deprecated"
|
||||
|
||||
# Lifecycle: deprecate a node
|
||||
|
||||
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml): a node keeps running
|
||||
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml): a node keeps running
|
||||
but takes no new dependents. **Completion condition: zero remaining
|
||||
inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a
|
||||
suggestion; `oikos/policy.yaml` `lifecycle_overrides.deprecated.refuse`
|
||||
suggestion; `seeds/policy.yaml` `lifecycle_overrides.deprecated.refuse`
|
||||
lists `new-inbound-edges` as refused going forward.
|
||||
|
||||
1. Set `state: deprecated` on the node.
|
||||
|
||||
@@ -10,7 +10,7 @@ transition: "deprecated -> destroyed"
|
||||
# Lifecycle: destroy a node
|
||||
|
||||
**Destructive.** Requires operator approval + typed confirmation phrase
|
||||
per `oikos/policy.yaml`. Requires (ontology): backups verified, secrets
|
||||
per `seeds/policy.yaml`. Requires (ontology): backups verified, secrets
|
||||
recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
|
||||
ledger entry.
|
||||
|
||||
@@ -33,7 +33,7 @@ ledger entry.
|
||||
data).
|
||||
7. `oikos/ledger.py append host:<name> destroy destructive --result ok`.
|
||||
8. Regenerate: `python3 mcp/build_host_files.py && python3
|
||||
oikos/gen-topology.py` — the node drops out of `hosts/*.yaml` and
|
||||
inventory.yaml` — the node drops out of `inventory.yaml` and
|
||||
appears in the topology doc's archaeology table.
|
||||
|
||||
If the destroy fails partway (e.g. secrets revoked but pct destroy
|
||||
|
||||
@@ -32,8 +32,8 @@ before returning to `active`.
|
||||
7. `homelab service <name> health` for every service the node hosts.
|
||||
8. Set `state: active`. Add a migration note to the node's doc page
|
||||
(old host/IP → new, date, phase reference) — this repo's convention
|
||||
for every past migration (see `containers/101-jellyfin.md`,
|
||||
for every past migration (see `archive/knowledge/containers/101-jellyfin.md`,
|
||||
`containers/129-house.md`).
|
||||
|
||||
Regenerate: `python3 mcp/build_host_files.py && python3
|
||||
oikos/gen-topology.py`.
|
||||
inventory.yaml`.
|
||||
|
||||
@@ -2,17 +2,17 @@
|
||||
name: lifecycle-provision-node
|
||||
risk_class: config_mutation
|
||||
inputs: [node_name, kind, storage_pool]
|
||||
verification: "grep 'state: provisioning' hosts/<name>.yaml"
|
||||
verification: "grep 'state: provisioning' inventory.yaml"
|
||||
docs_update_checklist: [doc_page_stub]
|
||||
transition: "planned -> provisioning"
|
||||
---
|
||||
|
||||
# Lifecycle: provision a node
|
||||
|
||||
Per [oikos/ontology.yaml](../../../oikos/ontology.yaml) `lifecycle.transitions`.
|
||||
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml) `lifecycle.transitions`.
|
||||
Policy note: `provisioning` nodes get a lifecycle override —
|
||||
`config_mutation` actions downgrade to `reversible_low` because nothing
|
||||
depends on the node yet (see `oikos/policy.yaml` `lifecycle_overrides`).
|
||||
depends on the node yet (see `seeds/policy.yaml` `lifecycle_overrides`).
|
||||
|
||||
Requires (from ontology): inventory entry, IP reserved, storage pool
|
||||
chosen, doc page stub.
|
||||
|
||||
@@ -103,8 +103,8 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
|
||||
## Related
|
||||
|
||||
- [Operations cheatsheet](../../operations/commands.md)
|
||||
- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md)
|
||||
- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md)
|
||||
- [Auto-deploy pipelines](../../../archive/knowledge/infrastructure/auto-deploy.md)
|
||||
- [Hubris host page](../../../archive/knowledge/hosts/hubris.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ Goal: determine whether a service is actually healthy, without ad-hoc SSH.
|
||||
4. Cross-check blast radius: `homelab node <name> relations` — is this
|
||||
entity's own backend host healthy? A downstream failure (e.g. `strong`
|
||||
down) will show up here before the service's own logs explain anything.
|
||||
5. If the fix is a restart: classify first (`oikos/policy.yaml` —
|
||||
5. If the fix is a restart: classify first (`seeds/policy.yaml` —
|
||||
`service-restart` is `reversible_low` unless the service has a
|
||||
`service_overrides` entry, e.g. `caddy`/`dns` are `config_mutation`).
|
||||
Unattended agents may act on `reversible_low` without approval.
|
||||
|
||||
Reference in New Issue
Block a user