dtoro
d80a394b7f
Documentation and repo-hygiene pass following the client/server split:
Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.
Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).
Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).
Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.
Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.
Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
GetClientContext handler) since the mechanism's introduction on
2026-06-02 — never matched any real filename, so no client has ever
picked up an auto-setup script via git-pull or the context-poller sync.
Fixed all three; the Go server-side fix is the one that actually matters
since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
(parsed, never consumed) left over from an earlier clone-based model.
Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
6.0 KiB
6.0 KiB
Operations cheatsheet
Run from the hubris host as root. When working from /root on Linux you're already on hubris — don't ssh hubris / ping hubris.
Proxmox CLI
| Command | Use |
|---|---|
pct list / qm list |
List LXC containers / VMs |
pct config <id> / qm config <id> |
Container / VM config |
pct exec <id> -- <cmd> |
Run command inside an LXC without entering it (no initgroups — see media permissions) |
pct enter <id> |
Shell into a container |
pct start <id> / pct stop <id> |
Boot / halt a container |
pvesm status |
Storage pools status |
pvesh get /nodes --output-format json |
Node summary as JSON |
pvesh get /nodes/hubris/lxc/<id>/status/current |
Live container status |
pvesh get /cluster/resources --type vm --output-format json |
Bulk per-LXC CPU/mem/disk (used by the homelab-health-watchdog Nomos cron — see monitoring; the old claudio-monitor this once fed is deprecated) |
pveversion |
PVE version |
journalctl -u pve-cluster -n 100 |
PVE service logs |
Storage
- Shared mount:
/mnt/library(ext4 on lvmthinlibrary). - Bind into a container:
pct set <id> -mp<N> /mnt/library/<sub>,mp=/data - For the standard whole-tree mount:
pct set <id> -mp0 /mnt/library,mp=/mnt/library. See media permissions for the GID-10000 onboarding recipe.
Reverse proxy
- Caddyfile:
/etc/caddy/Caddyfileon LXC 121. - CRITICAL: This file is tracked in
dtoro/caddy-conf(https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see auto-deploy). If you edit directly, the change will be lost on the next pull and agents won't know about it. - Hot reload:
pct exec 121 -- systemctl reload caddy. - Validate:
pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile. - Git workflow shortcut:
pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push".
DNS
- Split-horizon authority: Technitium DNS on dns (107) at
192.168.8.2:53. Web UI athttp://192.168.8.2. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.) - Add/edit records in the Technitium UI; the NetBird managed zone sync (
scripts/dns-sync.pycron on 107) picks changes up within ~10 minutes. - Verify:
dig @192.168.8.2 +short <host>.hubris.network. - See DNS.
Web access
https://proxmox.hubris.networkorhttps://192.168.8.77:8006— Proxmox UI
Telemetry quick checks
ras-mc-ctl --summary— summary of any RAS events (memory / PCIe AER / thermal) since bootras-mc-ctl --errors— full event logcat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference— should bebalance_powercat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor— should bepowersavels /sys/fs/pstore/ /var/lib/systemd/pstore/— panic traces from a previous crash (empty for pure hardware hangs — see investigation)
Fleet apt operations
No current CLI equivalent. homelab apt-audit/apt-upgrade (dpkg-state
audit, fanned-out apt upgrade with pre-upgrade snapshots) were part of the
retired Python homelab CLI and don't have a ported replacement — apt
patching today is ad hoc run MCP tool calls per host, without the
audit/snapshot/status wrapping this used to provide. If that wrapping is
still wanted, it needs to be rebuilt (e.g. as a runbook driving run, or a
new MCP tool) — see
runbook-dpkg-interrupted for
the dpkg-interrupted recovery procedure specifically.
Oikos (agent OS layer)
See OIKOS.md for the operating model. The homelab CLI this
section used to document is retired; the actual current interface is the
33 MCP tools cataloged in AGENTS.md plus
the REST API. Closest current equivalents for what used to live here:
Old homelab command |
Current equivalent |
|---|---|
homelab service <name> explain|health|docs|log |
MCP explain, get_service_status, tail_log, get_entity_knowledge |
homelab node <name> relations |
MCP get_blast_radius |
homelab change preflight <service> |
MCP preflight |
homelab signal list|ack|resolve|mute |
MCP get_signal_history, or REST POST /api/v1/signals/{id}/ack|resolve|mute (the control-room UI's Signals page wraps these) |
homelab approval request|list|reply|check |
REST GET/POST /api/v1/approvals* (Matrix-delivered via the notifier, or the control-room UI's Operations page) |
homelab restart <service> --approval-id <id> |
MCP run (policy-gated — auto-executes if read-only/reversible_low, otherwise queues for the same Matrix/UI approval) |
homelab decide <action> <entity> |
No direct equivalent — classification now happens inline inside run/request_execution, not as a separate dry-run call |
There is no separately-deployed "Oikos Console" anymore — the control-room
SPA (web/) is the operator dashboard, served standalone (see
plans/2026-07-12-wails-desktop-app.md).
Related
- Hubris host
- Containers index
- DNS
- Monitoring
- Auto-deploy
- Runbook: dpkg-interrupted recovery — what to do when apt got killed mid-transaction