Files

dtoro 205d8a1a43 Oikos Week 4: Console v0, approval hardening, docs pass, backlog

Oikos Console v0 (oikos/console/) — read-mostly, server-rendered FastAPI
+ Jinja2 web UI, no SPA build chain. Signals landing page, service grid
+ detail, node/blast-radius view, live Mermaid relationship graph, drift
findings, approvals queue (approve/deny, destructive confirmation-phrase
enforced), daily/weekly reports. Tested end-to-end via the preview tools
against live production data, including a real click-through of the
approve/deny flow.

Found and fixed two bugs during that testing:
- Severity-dot CSS classes didn't match the actual severity strings
  (dot-warn/dot-crit vs "warning"/"critical") — warning-severity signals
  rendered with no visible indicator at all.
- The console's sys.path setup pointed at its own webhook checkout
  (/opt/oikos-console) rather than /opt/homelab-context, which would have
  made its oikos.* imports resolve to a SEPARATE copy of oikos/signal.py
  etc. than the scheduler and CLI use — silently forking signal/approval
  data into two locations in production. Fixed to match mcp/server.py's
  CONTEXT_DIR pattern. Also added _commit_push() so the console's writes
  (approval replies, signal ack/resolve) don't sit uncommitted against
  the 5-min-synced clone.

Split oikos/gen_topology_lib.py out of oikos/gen-topology.py (hyphenated
filenames aren't importable) so the console's /graph route can render
live without shelling out.

oikos/console/deploy/ — third webhook on dtoro/Homelab-Docs (port 9831),
matching the homelab-mcp/secrets-issuance precedent. README documents the
Caddy route and Gitea webhook registration this repo can't do for itself,
and that Authentik step-up on /approvals needs a live instance to
configure.

Approval hardening: grants are now single-use (oikos/approve.py
check_grant marks the request "executed" atomically, so a second call
for the same id fails even within the TTL) — verified with a test. Per-
agent age-key-signed requests, as originally planned, turned out not to
be buildable as stated: age is encryption-only, no signing primitive.
Documented the real alternative (SSH-key signing) and moved it to the
60/90-day backlog pending an inventory schema gap (no SSH pubkeys
recorded today).

Docs pass: added the Oikos command surface to operations/commands.md,
new MCP tools to AGENTS.md. Found two more stale references while at
it — commands.md and AGENTS.md both still pointed DNS at the destroyed
LXC 124/dnsmasq instead of Technitium on dns (107), and a claudio-monitor
reference deprecated since 2026-06-04 — fixed both.

60/90-day backlog written into OIKOS.md, derived from gaps actually
observed this month, not guesswork.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-06 00:03:51 +02:00

6.3 KiB

Raw Blame History

Operations cheatsheet

Run from the hubris host as root. When working from /root on Linux you're already on hubris — don't ssh hubris / ping hubris.

Proxmox CLI

Command	Use
`pct list` / `qm list`	List LXC containers / VMs
`pct config <id>` / `qm config <id>`	Container / VM config
`pct exec <id> -- <cmd>`	Run command inside an LXC without entering it (no initgroups — see media permissions)
`pct enter <id>`	Shell into a container
`pct start <id>` / `pct stop <id>`	Boot / halt a container
`pvesm status`	Storage pools status
`pvesh get /nodes --output-format json`	Node summary as JSON
`pvesh get /nodes/hubris/lxc/<id>/status/current`	Live container status
`pvesh get /cluster/resources --type vm --output-format json`	Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see monitoring; the old `claudio-monitor` this once fed is deprecated)
`pveversion`	PVE version
`journalctl -u pve-cluster -n 100`	PVE service logs

Storage

Shared mount: /mnt/library (ext4 on lvmthin library).
Bind into a container: pct set <id> -mp<N> /mnt/library/<sub>,mp=/data
For the standard whole-tree mount: pct set <id> -mp0 /mnt/library,mp=/mnt/library. See media permissions for the GID-10000 onboarding recipe.

Reverse proxy

Caddyfile: /etc/caddy/Caddyfile on LXC 121.
CRITICAL: This file is tracked in dtoro/caddy-conf (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see auto-deploy). If you edit directly, the change will be lost on the next pull and agents won't know about it.
Hot reload: pct exec 121 -- systemctl reload caddy.
Validate: pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile.
Git workflow shortcut: pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push".

DNS

Split-horizon authority: Technitium DNS on dns (107) at 192.168.8.2:53. Web UI at http://192.168.8.2. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
Add/edit records in the Technitium UI; the NetBird managed zone sync (scripts/dns-sync.py cron on 107) picks changes up within ~10 minutes.
Verify: dig @192.168.8.2 +short <host>.hubris.network.
See DNS.

Web access

https://proxmox.hubris.network or https://192.168.8.77:8006 — Proxmox UI

Telemetry quick checks

ras-mc-ctl --summary — summary of any RAS events (memory / PCIe AER / thermal) since boot
ras-mc-ctl --errors — full event log
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference — should be balance_power
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor — should be powersave
ls /sys/fs/pstore/ /var/lib/systemd/pstore/ — panic traces from a previous crash (empty for pure hardware hangs — see investigation)

Fleet apt operations

Two homelab subcommands wrap the common patterns; both fan out to hubris + every LXC.

Command	What it does
`homelab apt-audit [--target HOST]`	Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state.
`homelab apt-upgrade --target HOST`	Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`.
`homelab apt-upgrade --all`	Same, fanned out across the standard targets.
`homelab apt-upgrade ... --status`	Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target.
`homelab apt-upgrade ... --safe`	Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`.
`homelab apt-upgrade ... --force`	Skip both the dpkg-audit gate and snapshot-failure refusal.

PVE/kernel deferral on hubris: homelab apt-upgrade --target hubris will try every upgrade, including kernel + pve-*. To skip those, apt-mark hold the relevant packages on hubris first; homelab apt-audit shows held packages so you can confirm.

Oikos (agent OS layer)

See OIKOS.md for the operating model. Quick reference:

Command	What it does
`homelab service <name> explain\|health\|docs\|log\|actions\|history`	Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history
`homelab node <name> relations`	Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius
`homelab change preflight <service>`	Dry-run report before mutating: risk class, current health, config repo, verification command
`homelab decide <action> <entity>`	Decision classifier: risk × blast radius × confidence → auto-act or escalate
`homelab signal list\|raise\|ack\|resolve\|mute`	The attention layer — pending updates, thresholds, drift, anything needing attention
`homelab approval request\|list\|reply\|check`	Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page)
`homelab restart <service> [--approval-id <id>]`	`--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant

Oikos Console (read-mostly dashboard): oikos.hubris.network once deployed — see oikos/console/deploy/README.md.

Hubris host
Containers index
DNS
Monitoring
Auto-deploy
Runbook: dpkg-interrupted recovery — what to do when apt got killed mid-transaction

6.3 KiB Raw Blame History Unescape Escape