dtoro
205d8a1a43
Oikos Console v0 (oikos/console/) — read-mostly, server-rendered FastAPI + Jinja2 web UI, no SPA build chain. Signals landing page, service grid + detail, node/blast-radius view, live Mermaid relationship graph, drift findings, approvals queue (approve/deny, destructive confirmation-phrase enforced), daily/weekly reports. Tested end-to-end via the preview tools against live production data, including a real click-through of the approve/deny flow. Found and fixed two bugs during that testing: - Severity-dot CSS classes didn't match the actual severity strings (dot-warn/dot-crit vs "warning"/"critical") — warning-severity signals rendered with no visible indicator at all. - The console's sys.path setup pointed at its own webhook checkout (/opt/oikos-console) rather than /opt/homelab-context, which would have made its oikos.* imports resolve to a SEPARATE copy of oikos/signal.py etc. than the scheduler and CLI use — silently forking signal/approval data into two locations in production. Fixed to match mcp/server.py's CONTEXT_DIR pattern. Also added _commit_push() so the console's writes (approval replies, signal ack/resolve) don't sit uncommitted against the 5-min-synced clone. Split oikos/gen_topology_lib.py out of oikos/gen-topology.py (hyphenated filenames aren't importable) so the console's /graph route can render live without shelling out. oikos/console/deploy/ — third webhook on dtoro/Homelab-Docs (port 9831), matching the homelab-mcp/secrets-issuance precedent. README documents the Caddy route and Gitea webhook registration this repo can't do for itself, and that Authentik step-up on /approvals needs a live instance to configure. Approval hardening: grants are now single-use (oikos/approve.py check_grant marks the request "executed" atomically, so a second call for the same id fails even within the TTL) — verified with a test. Per- agent age-key-signed requests, as originally planned, turned out not to be buildable as stated: age is encryption-only, no signing primitive. Documented the real alternative (SSH-key signing) and moved it to the 60/90-day backlog pending an inventory schema gap (no SSH pubkeys recorded today). Docs pass: added the Oikos command surface to operations/commands.md, new MCP tools to AGENTS.md. Found two more stale references while at it — commands.md and AGENTS.md both still pointed DNS at the destroyed LXC 124/dnsmasq instead of Technitium on dns (107), and a claudio-monitor reference deprecated since 2026-06-04 — fixed both. 60/90-day backlog written into OIKOS.md, derived from gaps actually observed this month, not guesswork. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
92 lines
6.3 KiB
Markdown
92 lines
6.3 KiB
Markdown
# Operations cheatsheet
|
||
|
||
Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
||
|
||
## Proxmox CLI
|
||
|
||
| Command | Use |
|
||
| --- | --- |
|
||
| `pct list` / `qm list` | List LXC containers / VMs |
|
||
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
||
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../infrastructure/media-permissions.md)) |
|
||
| `pct enter <id>` | Shell into a container |
|
||
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
||
| `pvesm status` | Storage pools status |
|
||
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
||
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
||
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
||
| `pveversion` | PVE version |
|
||
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
||
|
||
## Storage
|
||
|
||
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
||
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
||
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
||
|
||
## Reverse proxy
|
||
|
||
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md).
|
||
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
||
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
||
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
||
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
|
||
|
||
## DNS
|
||
|
||
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
||
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
|
||
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
|
||
- See [DNS](../infrastructure/dns.md).
|
||
|
||
## Web access
|
||
|
||
- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
|
||
|
||
## Telemetry quick checks
|
||
|
||
- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
|
||
- `ras-mc-ctl --errors` — full event log
|
||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
||
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md))
|
||
|
||
## Fleet apt operations
|
||
|
||
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
|
||
|
||
| Command | What it does |
|
||
| --- | --- |
|
||
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
|
||
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
|
||
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
|
||
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
|
||
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
|
||
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
|
||
|
||
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
|
||
|
||
## Oikos (agent OS layer)
|
||
|
||
See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
|
||
|
||
| Command | What it does |
|
||
| --- | --- |
|
||
| `homelab service <name> explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history |
|
||
| `homelab node <name> relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius |
|
||
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
|
||
| `homelab decide <action> <entity>` | Decision classifier: risk × blast radius × confidence → auto-act or escalate |
|
||
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
|
||
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
|
||
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
|
||
|
||
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../oikos/console/deploy/README.md).
|
||
|
||
## Related
|
||
- [Hubris host](../hosts/hubris.md)
|
||
- [Containers index](../containers/index.md)
|
||
- [DNS](../infrastructure/dns.md)
|
||
- [Monitoring](../infrastructure/monitoring.md)
|
||
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
||
- [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction
|