oikos/plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md

Plan: Consolidate the Oikos control plane onto mac-mini

Status: Planned (2026-07-06) — approved, not yet executed.

Context

The goal is an agentic OS that runs the homelab and keeps the operator in control. Today the Oikos control plane is scattered and, worse, mis-placed: homelab-mcp, secrets-issuance, and oikos-console all run on LXC 105 (apps) — a container inside hubris, oikos-scheduler (the Observe stage) is undeployed, and the Hermes agent runtime is spread across several clients. Two problems follow:

• The observer lives inside the observed. A scheduler on apps/105 can't report "hubris is down" — it dies with hubris. The Observe/Act loop needs an always-on node outside both hypervisors.
• Deploy is fragmented. Three Gitea webhooks (ids 10/11/14); webhook 14 (console) has been 403-broken for a cause never found, so console never auto-redeploys.

Decision: make mac-mini the single Oikos control node — it's always-on, external to both hubris and strong, on the mesh, can SSH/curl both, and already runs Hermes. Move everything (MCP, secrets-issuance, console, scheduler, the new actuator, Hermes) there; apps/105 stops running Oikos. Because mac-mini already has a launchd git-sync (5-min pull), we replace all three Gitea webhooks with pull-based deploy — which eliminates the webhook-403 problem entirely.

This is a macOS re-platforming (systemd → launchd) plus an ingress re-point. The public URLs (mcp./secrets./oikos.hubris.network) and DNS are unchanged — only the Caddy backend and the runtime host move.

Target architecture

Caddy (LXC 121, ingress)  ──reverse_proxy over netbird──▶  mac-mini (always-on, macOS)
  mcp.hubris.network      ─────────────────────────────▶    homelab-mcp        :9810
  secrets.hubris.network  ─────────────────────────────▶    secrets-issuance   :9820
  oikos.hubris.network    ─────────────────────────────▶    oikos-console      :8091

mac-mini also runs (no ingress, mesh/SSH outward):
  oikos-scheduler + actuator (launchd StartInterval 600s)  ──ssh/curl──▶ hubris, strong
  Hermes agent runtime (already here)

Single synced checkout: /opt/homelab-context (5-min launchd git pull = deploy trigger)
apps/105: Oikos services stopped + removed; keeps its non-Oikos roles (artifacto, plantuml)

Workstreams

1. macOS runtime — launchd plists (deploy/macos/, new)

One LaunchDaemon plist per service (run at boot, no GUI login), mirroring each systemd unit's ExecStart/Environment, replacing [Timer] with launchd keys:

Service	plist	launchd equivalent of
homelab-mcp	network.hubris.homelab-mcp.plist	mcp/deploy/homelab-mcp.service (KeepAlive)
secrets-issuance	network.hubris.secrets-issuance.plist	its systemd unit (KeepAlive)
oikos-console	network.hubris.oikos-console.plist	oikos/console/deploy/oikos-console.service (KeepAlive)
oikos-scheduler	network.hubris.oikos-scheduler.plist	oikos/systemd/oikos-scheduler.{service,timer} (StartInterval 600)

• Single checkout, shared venv. All services run in-place from /opt/homelab-context with one shared .venv (gitignored), instead of the Linux per-service checkouts (/opt/homelab-mcp, /opt/oikos-console). This is the concrete consolidation win: one clone, one sync, N daemons.
• Confinement gap (tradeoff, document it): the systemd units use ProtectSystem=strict, ReadOnlyPaths, NoNewPrivileges, PrivateTmp — launchd has no equivalent. Mitigate by running the daemons as a dedicated non-root _oikos user (or dtoro) with least-privilege file perms, and note the reduced isolation in the deploy README + OIKOS.md risk section.
• install-macos.sh — create _oikos user + venv (pip install the deps), copy plists to /Library/LaunchDaemons, launchctl bootstrap system …. Idempotent, mirroring mcp/deploy/deploy.sh's shape.

2. macOS deploy = the existing pull-sync (retire the 3 webhooks)

• Extend mac-mini's existing launchd git-sync post-pull hook (the enrollment tools/post-pull.sh path) to launchctl kickstart -k any service whose files changed after a pull. Pull-based deploy replaces webhook-push deploy.
• Retire Gitea webhooks 10, 11, 14 (operator step in Gitea UI). This removes the never-solved console-webhook 403 by deleting the webhook, not fixing it.
• Update knowledge/wiki/infrastructure/auto-deploy.md: mac-mini services deploy by pull-sync, not webhook; the webhook shape now applies only to remaining in-lab config repos.

3. Build the actuator + autonomy kill-switch (the Act stage)

Still required — it's the control loop, now living on mac-mini.

• oikos/act.py (new): read open Signals (oikos_signal.list_signals( state="raised")); for each with a recommended_action, call oikos_decide.classify(). auto-act (reversible_low, contained, confident) → start_acting() (the existing unused transition), execute via homelab restart <svc> -y (reuses cmd_restart + its _record_change ledger write), run the Signal's verification, then resolve() or leave open with a result=failed ledger entry so the next pass escalates. Everything else → oikos_approve.request(...) + acknowledge(note="escalated: <id>"). A ledger-history loop-guard caps auto-retries per (entity, kind).
• Autonomy kill-switch in oikos/policy.yaml (autonomy.auto_act: off|reversible_low, never_auto_act: entity list) + loaders in oikos/policy.py — the "keep control" guarantee.
• Wire python3 oikos/act.py run into the scheduler's launchd pass (after the observe pass) and commit signals/ + ledger/.

4. Ingress re-point (Caddy backends: apps → mac-mini mesh)

• In dtoro/caddy-conf (separate repo, tracked-config commit+push): change the three reverse_proxy backends for mcp./secrets./oikos.hubris.network from 192.168.8.205:<port> to mac-mini's netbird address (mac-mini-234-17.netbird.selfhosted:<port>), since Caddy (8.x) and mac-mini (178.x) are cross-subnet — mesh is the robust path. DNS and public URLs unchanged.
• Verify Caddy → mac-mini reachability first (Caddy container must resolve + reach the netbird name; confirm netbird is on the caddy LXC or route via hubris). If mesh isn't reachable from Caddy, fall back to the 178.x LAN IP and confirm hubris routes 8.x↔178.x (it's dual-homed: hubris 178.181, apps 8.205).

5. Secrets & SSH keys on mac-mini

• SOPS recipients: grant mac-mini's age_pubkey (age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs) on every secret the moved services decrypt at runtime (secrets-issuance's issuing key, approval HMAC secrets/oikos-approval-hmac.yaml, any service creds), then sops updatekeys. Edit .sops.yaml recipient lists + re-key; commit via homelab client add mac-mini --finalize-pubkey … path (comment-preserving).
• SSH keys: provision on mac-mini the mcp-reader.key restricted key the MCP server SSH-proxies to hubris with (HOMELAB_MCP_SSH_KEY), plus root SSH access to hubris + strong for the scheduler's disk probe (_ssh in oikos/scheduler.py:102). Store under /etc/homelab-mcp/ and _oikos's ~/.ssh respectively.
• Security note (document): the age-key issuer now runs on a workstation. Record the widened surface in OIKOS.md; least-privilege _oikos user + keychain/file perms are the mitigation.

6. Inventory / topology truth (same-session rule)

• inventory.yaml: set services.{homelab_mcp,secrets_issuance}.backend: mac-mini; add oikos_console + oikos_scheduler service entries backed by mac-mini; give mac-mini a runs: list (homelab-mcp, secrets-issuance, oikos-console, oikos-scheduler); remove homelab-mcp/secrets-issuance from apps.runs. Keep systemd_unit→ note launchd label.
• Regenerate derived views: mcp/build_host_files.py (hosts/*.yaml) and oikos/gen-topology.py (cards + topology.md). Update knowledge/wiki/containers/105-apps.md and add a mac-mini control-node page under knowledge/wiki/hosts/ (or note on the workstation).

7. Decommission on apps/105 (operator steps, documented not executed)

systemctl disable --now homelab-mcp secrets-issuance oikos-console + their -deploy webhook units on apps; remove /opt/homelab-mcp, /opt/oikos-console checkouts. Captured in the deploy README as a cutover checklist.

8. Docs

.agents/OIKOS.md (control node = mac-mini; loop closed; risks), auto-deploy.md (pull-sync), deploy/macos/README.md (cutover runbook), .agents/skills/autopilot-pass/SKILL.md (supervised actuator run), host/service cards (regenerated).

Critical files

File	Change
deploy/macos/*.plist + install-macos.sh + README.md	New — launchd runtime + cutover
oikos/act.py	New — actuator (Act stage)
oikos/policy.yaml, oikos/policy.py	Autonomy kill-switch + loaders
oikos/systemd/run-scheduler.sh → macOS variant	Observe+Act pass, commit signals/+ledger/
inventory.yaml	Move service backends apps→mac-mini; mac-mini runs:
hosts/*.yaml, oikos/cards/*, topology.md	Regenerated
dtoro/caddy-conf (external repo)	3 reverse_proxy backends → mac-mini mesh
.sops.yaml	Grant mac-mini recipient on moved-service secrets; re-key
bin/homelab	Optional autopilot subcommand
Docs: OIKOS.md, auto-deploy.md, 105-apps.md	Reflect consolidation

Reuse (do not reimplement)

• Deploy scaffolding shape: mcp/deploy/deploy.sh, oikos/console/deploy/.
• Service definitions to port: mcp/deploy/homelab-mcp.service, oikos/console/deploy/oikos-console.service, oikos/systemd/*.
• Actuator primitives: oikos_decide.classify (decide.py:64), signal transitions incl. unused start_acting (signal.py:125), cmd_restart + _record_change (bin/homelab:603,190), oikos_approve.request (bin/homelab:1370), oikos_ledger.history (ledger.py:59).
• Comment-preserving inventory + .sops.yaml edits + sops updatekeys: homelab client add --finalize-pubkey path in bin/homelab.

Risks / assumptions

• Caddy → mac-mini reachability over netbird is unverified — workstream 4 verifies before cutover; LAN 178.x is the fallback (hubris is dual-homed).
• Lost sandboxing on macOS (no ProtectSystem/ReadOnlyPaths) — mitigated by a least-privilege _oikos daemon user, not eliminated.
• secrets-issuance on a workstation widens the security surface — accepted per scope decision; documented.
• Single-checkout race: a git pull mid-request could serve half-updated code — post-pull kickstart -k restarts services after the pull completes.
• macOS /opt/homelab-context already exists (mac-mini is enrolled); daemons need it readable by _oikos.

Verification (end to end)

1. Runtime: install-macos.sh on mac-mini; launchctl print system/… shows all four daemons running; curl 127.0.0.1:{9810,9820,8091} responds locally.
2. Ingress: after the caddy-conf push + reload, curl -I https://mcp.hubris.network/mcp, .../ (console → 302 Authentik), secrets.hubris.network/health → 200/302 through Caddy to mac-mini.
3. Observe+Act loop: trigger a service-down Signal; confirm the launchd scheduler pass runs it raised→acting→resolved, writes a restart/result: ok ledger entry, and pushes signals/+ledger/. Kill-switch: set autonomy.auto_act: off → next pass escalates instead.
4. Deploy-by-pull: git push a trivial change; within 5 min mac-mini pulls and kickstarts the changed daemon (no webhook involved).
5. Cutover: after decommission, systemctl is-active on apps shows the three services inactive; production traffic served only by mac-mini; MCP tools + console + a test enrollment all still work.

Out of scope (per prior direction)

SSH-signed approval requests, Authentik step-up on the console POST, Prometheus / temperature trend signals, and the generalize-to-any-homelab refactor.