dtoro
d80a394b7f
Documentation and repo-hygiene pass following the client/server split:
Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.
Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).
Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).
Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.
Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.
Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
GetClientContext handler) since the mechanism's introduction on
2026-06-02 — never matched any real filename, so no client has ever
picked up an auto-setup script via git-pull or the context-poller sync.
Fixed all three; the Go server-side fix is the one that actually matters
since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
(parsed, never consumed) left over from an earlier clone-based model.
Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
94 lines
6.0 KiB
Markdown
94 lines
6.0 KiB
Markdown
# Operations cheatsheet
|
|
|
|
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
|
|
|
## Proxmox CLI
|
|
|
|
| Command | Use |
|
|
| --- | --- |
|
|
| `pct list` / `qm list` | List LXC containers / VMs |
|
|
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
|
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
|
|
| `pct enter <id>` | Shell into a container |
|
|
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
|
| `pvesm status` | Storage pools status |
|
|
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
|
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
|
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Nomos cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
|
| `pveversion` | PVE version |
|
|
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
|
|
|
## Storage
|
|
|
|
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
|
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
|
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
|
|
|
## Reverse proxy
|
|
|
|
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
|
|
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
|
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
|
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
|
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
|
|
|
|
## DNS
|
|
|
|
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
|
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
|
|
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
|
|
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
|
|
|
|
## Web access
|
|
|
|
- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
|
|
|
|
## Telemetry quick checks
|
|
|
|
- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
|
|
- `ras-mc-ctl --errors` — full event log
|
|
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
|
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
|
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/archive/2026-04-21-hubris-crash-loop.md))
|
|
|
|
## Fleet apt operations
|
|
|
|
**No current CLI equivalent.** `homelab apt-audit`/`apt-upgrade` (dpkg-state
|
|
audit, fanned-out apt upgrade with pre-upgrade snapshots) were part of the
|
|
retired Python `homelab` CLI and don't have a ported replacement — apt
|
|
patching today is ad hoc `run` MCP tool calls per host, without the
|
|
audit/snapshot/status wrapping this used to provide. If that wrapping is
|
|
still wanted, it needs to be rebuilt (e.g. as a runbook driving `run`, or a
|
|
new MCP tool) — see
|
|
[runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md) for
|
|
the dpkg-interrupted recovery procedure specifically.
|
|
|
|
## Oikos (agent OS layer)
|
|
|
|
See [OIKOS.md](../OIKOS.md) for the operating model. The `homelab` CLI this
|
|
section used to document is retired; the actual current interface is the
|
|
33 MCP tools cataloged in [AGENTS.md](../../AGENTS.md#3-the-mcp-server) plus
|
|
the REST API. Closest current equivalents for what used to live here:
|
|
|
|
| Old `homelab` command | Current equivalent |
|
|
| --- | --- |
|
|
| `homelab service <name> explain\|health\|docs\|log` | MCP `explain`, `get_service_status`, `tail_log`, `get_entity_knowledge` |
|
|
| `homelab node <name> relations` | MCP `get_blast_radius` |
|
|
| `homelab change preflight <service>` | MCP `preflight` |
|
|
| `homelab signal list\|ack\|resolve\|mute` | MCP `get_signal_history`, or REST `POST /api/v1/signals/{id}/ack\|resolve\|mute` (the control-room UI's Signals page wraps these) |
|
|
| `homelab approval request\|list\|reply\|check` | REST `GET/POST /api/v1/approvals*` (Matrix-delivered via the notifier, or the control-room UI's Operations page) |
|
|
| `homelab restart <service> --approval-id <id>` | MCP `run` (policy-gated — auto-executes if read-only/reversible_low, otherwise queues for the same Matrix/UI approval) |
|
|
| `homelab decide <action> <entity>` | No direct equivalent — classification now happens inline inside `run`/`request_execution`, not as a separate dry-run call |
|
|
|
|
There is no separately-deployed "Oikos Console" anymore — the control-room
|
|
SPA (`web/`) is the operator dashboard, served standalone (see
|
|
[plans/2026-07-12-wails-desktop-app.md](../../plans/2026-07-12-wails-desktop-app.md)).
|
|
|
|
## Related
|
|
- [Hubris host](../../archive/knowledge/hosts/hubris.md)
|
|
- [Containers index](../../archive/knowledge/containers/index.md)
|
|
- [DNS](../../archive/knowledge/infrastructure/dns.md)
|
|
- [Monitoring](../../archive/knowledge/infrastructure/monitoring.md)
|
|
- [Auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)
|
|
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction
|