cleanup: fix all stale references across .agents/ docs

- Rewrite AGENTS.md: DB as source of truth, MCP knowledge tools, archive refs
- Fix OIKOS.md: seeds/ paths, remove Python-era notes, update deployment status
- Fix commands.md, agent-enrollment.md: archive/knowledge/ links
- Fix all SKILL.md files: remove hosts/*.yaml refs, point to inventory.yaml
- Fix HERMES.md, schema.md, page-templates.md, llm-wiki.md: update paths
- Fix bootstrap.sh: identity check reads inventory.yaml
- Fix README.md, cutover-checklist.md: stale wiki references
- Move convert-wiki.py to archive/ (one-shot done)
This commit is contained in:
2026-07-07 21:00:39 +02:00
parent 5009a335bb
commit 5e3b946ded
24 changed files with 138 additions and 135 deletions

View File

@@ -30,7 +30,7 @@ hostname doesn't match any inventory entry. Two fixes:
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
(Linux) or System Preferences → Sharing (macOS), then re-run.
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
update `inventory.yaml`, push. The next sync (≤5 min) propagates.
### Getting onto Netbird
@@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../archive/knowledge/containers/106-auth-outpost.md). Only needed once per account.
**Path A — setup-key (headless/scripted onboarding):**
@@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr
### DNS prerequisite
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
([dns.md](../../archive/knowledge/infrastructure/dns.md)) for LAN clients, **but only if the
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
workstations don't by default. Options:
@@ -285,7 +285,7 @@ homelab client add my-new-machine
# 4. On hubris: finalize the age public key.
homelab client add my-new-machine --finalize-pubkey age1...
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
# inventory.yaml, commits + pushes. The 5-min sync propagates.
```
## Granting a secret to a new client
@@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). |
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../archive/knowledge/containers/106-auth-outpost.md). |
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |

View File

@@ -1,6 +1,6 @@
# Operations cheatsheet
Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
## Proxmox CLI
@@ -8,13 +8,13 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
| --- | --- |
| `pct list` / `qm list` | List LXC containers / VMs |
| `pct config <id>` / `qm config <id>` | Container / VM config |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) |
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
| `pct enter <id>` | Shell into a container |
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
| `pvesm status` | Storage pools status |
| `pvesh get /nodes --output-format json` | Node summary as JSON |
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
| `pveversion` | PVE version |
| `journalctl -u pve-cluster -n 100` | PVE service logs |
@@ -22,22 +22,22 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
## Reverse proxy
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
## DNS
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
- See [DNS](../../knowledge/wiki/infrastructure/dns.md).
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
## Web access
@@ -49,7 +49,7 @@ Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When w
- `ras-mc-ctl --errors` — full event log
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md))
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/archive/2026-04-21-hubris-crash-loop.md))
## Fleet apt operations
@@ -80,12 +80,12 @@ See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md).
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../archive/oikos-cards/).
## Related
- [Hubris host](../../knowledge/wiki/hosts/hubris.md)
- [Containers index](../../knowledge/wiki/containers/index.md)
- [DNS](../../knowledge/wiki/infrastructure/dns.md)
- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md)
- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)
- [Hubris host](../../archive/knowledge/hosts/hubris.md)
- [Containers index](../../archive/knowledge/containers/index.md)
- [DNS](../../archive/knowledge/infrastructure/dns.md)
- [Monitoring](../../archive/knowledge/infrastructure/monitoring.md)
- [Auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction