chore: add plan
This commit is contained in:
@@ -0,0 +1,220 @@
|
|||||||
|
# Plan: Consolidate the Oikos control plane onto mac-mini
|
||||||
|
|
||||||
|
**Status:** Planned (2026-07-06) — approved, not yet executed.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The goal is an agentic OS that runs the homelab and keeps the operator in
|
||||||
|
control. Today the Oikos control plane is scattered and, worse, mis-placed:
|
||||||
|
`homelab-mcp`, `secrets-issuance`, and `oikos-console` all run on **LXC 105
|
||||||
|
(apps) — a container inside hubris**, `oikos-scheduler` (the Observe stage) is
|
||||||
|
undeployed, and the Hermes agent runtime is spread across several clients. Two
|
||||||
|
problems follow:
|
||||||
|
|
||||||
|
- **The observer lives inside the observed.** A scheduler on apps/105 can't
|
||||||
|
report "hubris is down" — it dies with hubris. The Observe/Act loop needs an
|
||||||
|
always-on node *outside* both hypervisors.
|
||||||
|
- **Deploy is fragmented.** Three Gitea webhooks (ids 10/11/14); webhook 14
|
||||||
|
(console) has been 403-broken for a cause never found, so console never
|
||||||
|
auto-redeploys.
|
||||||
|
|
||||||
|
**Decision:** make **mac-mini** the single Oikos control node — it's always-on,
|
||||||
|
external to both hubris and strong, on the mesh, can SSH/curl both, and already
|
||||||
|
runs Hermes. Move *everything* (MCP, secrets-issuance, console, scheduler, the
|
||||||
|
new actuator, Hermes) there; apps/105 stops running Oikos. Because mac-mini
|
||||||
|
already has a launchd git-sync (5-min pull), we replace all three Gitea webhooks
|
||||||
|
with pull-based deploy — which eliminates the webhook-403 problem entirely.
|
||||||
|
|
||||||
|
This is a **macOS re-platforming** (systemd → launchd) plus an **ingress
|
||||||
|
re-point**. The public URLs (`mcp.`/`secrets.`/`oikos.hubris.network`) and DNS
|
||||||
|
are unchanged — only the Caddy backend and the runtime host move.
|
||||||
|
|
||||||
|
## Target architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Caddy (LXC 121, ingress) ──reverse_proxy over netbird──▶ mac-mini (always-on, macOS)
|
||||||
|
mcp.hubris.network ─────────────────────────────▶ homelab-mcp :9810
|
||||||
|
secrets.hubris.network ─────────────────────────────▶ secrets-issuance :9820
|
||||||
|
oikos.hubris.network ─────────────────────────────▶ oikos-console :8091
|
||||||
|
|
||||||
|
mac-mini also runs (no ingress, mesh/SSH outward):
|
||||||
|
oikos-scheduler + actuator (launchd StartInterval 600s) ──ssh/curl──▶ hubris, strong
|
||||||
|
Hermes agent runtime (already here)
|
||||||
|
|
||||||
|
Single synced checkout: /opt/homelab-context (5-min launchd git pull = deploy trigger)
|
||||||
|
apps/105: Oikos services stopped + removed; keeps its non-Oikos roles (artifacto, plantuml)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Workstreams
|
||||||
|
|
||||||
|
### 1. macOS runtime — launchd plists (`deploy/macos/`, new)
|
||||||
|
|
||||||
|
One `LaunchDaemon` plist per service (run at boot, no GUI login), mirroring each
|
||||||
|
systemd unit's `ExecStart`/`Environment`, replacing `[Timer]` with launchd keys:
|
||||||
|
|
||||||
|
| Service | plist | launchd equivalent of |
|
||||||
|
|---|---|---|
|
||||||
|
| `homelab-mcp` | `network.hubris.homelab-mcp.plist` | `mcp/deploy/homelab-mcp.service` (KeepAlive) |
|
||||||
|
| `secrets-issuance` | `network.hubris.secrets-issuance.plist` | its systemd unit (KeepAlive) |
|
||||||
|
| `oikos-console` | `network.hubris.oikos-console.plist` | `oikos/console/deploy/oikos-console.service` (KeepAlive) |
|
||||||
|
| `oikos-scheduler` | `network.hubris.oikos-scheduler.plist` | `oikos/systemd/oikos-scheduler.{service,timer}` (`StartInterval` 600) |
|
||||||
|
|
||||||
|
- **Single checkout, shared venv.** All services run in-place from
|
||||||
|
`/opt/homelab-context` with one shared `.venv` (gitignored), instead of the
|
||||||
|
Linux per-service checkouts (`/opt/homelab-mcp`, `/opt/oikos-console`). This is
|
||||||
|
the concrete consolidation win: one clone, one sync, N daemons.
|
||||||
|
- **Confinement gap (tradeoff, document it):** the systemd units use
|
||||||
|
`ProtectSystem=strict`, `ReadOnlyPaths`, `NoNewPrivileges`, `PrivateTmp` —
|
||||||
|
launchd has no equivalent. Mitigate by running the daemons as a dedicated
|
||||||
|
non-root `_oikos` user (or `dtoro`) with least-privilege file perms, and note
|
||||||
|
the reduced isolation in the deploy README + OIKOS.md risk section.
|
||||||
|
- **`install-macos.sh`** — create `_oikos` user + venv (`pip install` the deps),
|
||||||
|
copy plists to `/Library/LaunchDaemons`, `launchctl bootstrap system …`.
|
||||||
|
Idempotent, mirroring `mcp/deploy/deploy.sh`'s shape.
|
||||||
|
|
||||||
|
### 2. macOS deploy = the existing pull-sync (retire the 3 webhooks)
|
||||||
|
|
||||||
|
- Extend mac-mini's existing launchd git-sync post-pull hook (the enrollment
|
||||||
|
`tools/post-pull.sh` path) to `launchctl kickstart -k` any service whose files
|
||||||
|
changed after a pull. Pull-based deploy replaces webhook-push deploy.
|
||||||
|
- **Retire Gitea webhooks 10, 11, 14** (operator step in Gitea UI). This removes
|
||||||
|
the never-solved console-webhook 403 by deleting the webhook, not fixing it.
|
||||||
|
- Update `knowledge/wiki/infrastructure/auto-deploy.md`: mac-mini services deploy
|
||||||
|
by pull-sync, not webhook; the webhook shape now applies only to remaining
|
||||||
|
in-lab config repos.
|
||||||
|
|
||||||
|
### 3. Build the actuator + autonomy kill-switch (the Act stage)
|
||||||
|
|
||||||
|
Still required — it's the control loop, now living on mac-mini.
|
||||||
|
|
||||||
|
- **`oikos/act.py`** (new): read open Signals (`oikos_signal.list_signals(
|
||||||
|
state="raised")`); for each with a `recommended_action`, call
|
||||||
|
`oikos_decide.classify()`. `auto-act` (reversible_low, contained, confident)
|
||||||
|
→ `start_acting()` (the existing unused transition), execute via
|
||||||
|
`homelab restart <svc> -y` (reuses `cmd_restart` + its `_record_change` ledger
|
||||||
|
write), run the Signal's `verification`, then `resolve()` or leave open with a
|
||||||
|
`result=failed` ledger entry so the next pass escalates. Everything else →
|
||||||
|
`oikos_approve.request(...)` + `acknowledge(note="escalated: <id>")`. A
|
||||||
|
ledger-history loop-guard caps auto-retries per (entity, kind).
|
||||||
|
- **Autonomy kill-switch** in `oikos/policy.yaml` (`autonomy.auto_act:
|
||||||
|
off|reversible_low`, `never_auto_act:` entity list) + loaders in
|
||||||
|
`oikos/policy.py` — the "keep control" guarantee.
|
||||||
|
- Wire `python3 oikos/act.py run` into the scheduler's launchd pass (after the
|
||||||
|
observe pass) and commit `signals/` + `ledger/`.
|
||||||
|
|
||||||
|
### 4. Ingress re-point (Caddy backends: apps → mac-mini mesh)
|
||||||
|
|
||||||
|
- In `dtoro/caddy-conf` (separate repo, tracked-config commit+push): change the
|
||||||
|
three `reverse_proxy` backends for `mcp.`/`secrets.`/`oikos.hubris.network`
|
||||||
|
from `192.168.8.205:<port>` to **mac-mini's netbird address**
|
||||||
|
(`mac-mini-234-17.netbird.selfhosted:<port>`), since Caddy (8.x) and mac-mini
|
||||||
|
(178.x) are cross-subnet — mesh is the robust path. **DNS and public URLs
|
||||||
|
unchanged.**
|
||||||
|
- **Verify Caddy → mac-mini reachability first** (Caddy container must resolve +
|
||||||
|
reach the netbird name; confirm netbird is on the caddy LXC or route via
|
||||||
|
hubris). If mesh isn't reachable from Caddy, fall back to the 178.x LAN IP and
|
||||||
|
confirm hubris routes 8.x↔178.x (it's dual-homed: hubris 178.181, apps 8.205).
|
||||||
|
|
||||||
|
### 5. Secrets & SSH keys on mac-mini
|
||||||
|
|
||||||
|
- **SOPS recipients:** grant mac-mini's `age_pubkey`
|
||||||
|
(`age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs`) on every
|
||||||
|
secret the moved services decrypt at runtime (secrets-issuance's issuing key,
|
||||||
|
approval HMAC `secrets/oikos-approval-hmac.yaml`, any service creds), then
|
||||||
|
`sops updatekeys`. Edit `.sops.yaml` recipient lists + re-key; commit via
|
||||||
|
`homelab client add mac-mini --finalize-pubkey …` path (comment-preserving).
|
||||||
|
- **SSH keys:** provision on mac-mini the `mcp-reader.key` restricted key the MCP
|
||||||
|
server SSH-proxies to hubris with (`HOMELAB_MCP_SSH_KEY`), plus root SSH access
|
||||||
|
to hubris + strong for the scheduler's disk probe (`_ssh` in
|
||||||
|
`oikos/scheduler.py:102`). Store under `/etc/homelab-mcp/` and `_oikos`'s
|
||||||
|
`~/.ssh` respectively.
|
||||||
|
- **Security note (document):** the age-key *issuer* now runs on a workstation.
|
||||||
|
Record the widened surface in OIKOS.md; least-privilege `_oikos` user +
|
||||||
|
keychain/file perms are the mitigation.
|
||||||
|
|
||||||
|
### 6. Inventory / topology truth (same-session rule)
|
||||||
|
|
||||||
|
- `inventory.yaml`: set `services.{homelab_mcp,secrets_issuance}.backend:
|
||||||
|
mac-mini`; add `oikos_console` + `oikos_scheduler` service entries backed by
|
||||||
|
mac-mini; give `mac-mini` a `runs:` list (homelab-mcp, secrets-issuance,
|
||||||
|
oikos-console, oikos-scheduler); remove `homelab-mcp`/`secrets-issuance` from
|
||||||
|
`apps.runs`. Keep `systemd_unit`→ note launchd label.
|
||||||
|
- Regenerate derived views: `mcp/build_host_files.py` (hosts/*.yaml) and
|
||||||
|
`oikos/gen-topology.py` (cards + `topology.md`). Update
|
||||||
|
`knowledge/wiki/containers/105-apps.md` and add a mac-mini control-node page
|
||||||
|
under `knowledge/wiki/hosts/` (or note on the workstation).
|
||||||
|
|
||||||
|
### 7. Decommission on apps/105 (operator steps, documented not executed)
|
||||||
|
|
||||||
|
`systemctl disable --now homelab-mcp secrets-issuance oikos-console` + their
|
||||||
|
`-deploy` webhook units on apps; remove `/opt/homelab-mcp`, `/opt/oikos-console`
|
||||||
|
checkouts. Captured in the deploy README as a cutover checklist.
|
||||||
|
|
||||||
|
### 8. Docs
|
||||||
|
|
||||||
|
`.agents/OIKOS.md` (control node = mac-mini; loop closed; risks),
|
||||||
|
`auto-deploy.md` (pull-sync), `deploy/macos/README.md` (cutover runbook),
|
||||||
|
`.agents/skills/autopilot-pass/SKILL.md` (supervised actuator run), host/service
|
||||||
|
cards (regenerated).
|
||||||
|
|
||||||
|
## Critical files
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|---|---|
|
||||||
|
| `deploy/macos/*.plist` + `install-macos.sh` + `README.md` | **New** — launchd runtime + cutover |
|
||||||
|
| `oikos/act.py` | **New** — actuator (Act stage) |
|
||||||
|
| `oikos/policy.yaml`, `oikos/policy.py` | Autonomy kill-switch + loaders |
|
||||||
|
| `oikos/systemd/run-scheduler.sh` → macOS variant | Observe+Act pass, commit signals/+ledger/ |
|
||||||
|
| `inventory.yaml` | Move service backends apps→mac-mini; mac-mini `runs:` |
|
||||||
|
| `hosts/*.yaml`, `oikos/cards/*`, `topology.md` | Regenerated |
|
||||||
|
| `dtoro/caddy-conf` (external repo) | 3 reverse_proxy backends → mac-mini mesh |
|
||||||
|
| `.sops.yaml` | Grant mac-mini recipient on moved-service secrets; re-key |
|
||||||
|
| `bin/homelab` | Optional `autopilot` subcommand |
|
||||||
|
| Docs: `OIKOS.md`, `auto-deploy.md`, `105-apps.md` | Reflect consolidation |
|
||||||
|
|
||||||
|
## Reuse (do not reimplement)
|
||||||
|
|
||||||
|
- Deploy scaffolding shape: `mcp/deploy/deploy.sh`, `oikos/console/deploy/`.
|
||||||
|
- Service definitions to port: `mcp/deploy/homelab-mcp.service`,
|
||||||
|
`oikos/console/deploy/oikos-console.service`, `oikos/systemd/*`.
|
||||||
|
- Actuator primitives: `oikos_decide.classify` (`decide.py:64`), signal
|
||||||
|
transitions incl. unused `start_acting` (`signal.py:125`), `cmd_restart` +
|
||||||
|
`_record_change` (`bin/homelab:603,190`), `oikos_approve.request`
|
||||||
|
(`bin/homelab:1370`), `oikos_ledger.history` (`ledger.py:59`).
|
||||||
|
- Comment-preserving inventory + `.sops.yaml` edits + `sops updatekeys`:
|
||||||
|
`homelab client add --finalize-pubkey` path in `bin/homelab`.
|
||||||
|
|
||||||
|
## Risks / assumptions
|
||||||
|
|
||||||
|
- **Caddy → mac-mini reachability** over netbird is unverified — workstream 4
|
||||||
|
verifies before cutover; LAN 178.x is the fallback (hubris is dual-homed).
|
||||||
|
- **Lost sandboxing** on macOS (no `ProtectSystem`/`ReadOnlyPaths`) — mitigated
|
||||||
|
by a least-privilege `_oikos` daemon user, not eliminated.
|
||||||
|
- **secrets-issuance on a workstation** widens the security surface — accepted
|
||||||
|
per scope decision; documented.
|
||||||
|
- **Single-checkout race:** a git pull mid-request could serve half-updated code
|
||||||
|
— post-pull `kickstart -k` restarts services after the pull completes.
|
||||||
|
- macOS `/opt/homelab-context` already exists (mac-mini is enrolled); daemons
|
||||||
|
need it readable by `_oikos`.
|
||||||
|
|
||||||
|
## Verification (end to end)
|
||||||
|
|
||||||
|
1. **Runtime:** `install-macos.sh` on mac-mini; `launchctl print system/…` shows
|
||||||
|
all four daemons running; `curl 127.0.0.1:{9810,9820,8091}` responds locally.
|
||||||
|
2. **Ingress:** after the caddy-conf push + reload, `curl -I
|
||||||
|
https://mcp.hubris.network/mcp`, `.../` (console → 302 Authentik),
|
||||||
|
`secrets.hubris.network/health` → 200/302 through Caddy to mac-mini.
|
||||||
|
3. **Observe+Act loop:** trigger a `service-down` Signal; confirm the launchd
|
||||||
|
scheduler pass runs it `raised→acting→resolved`, writes a `restart`/`result:
|
||||||
|
ok` ledger entry, and pushes `signals/`+`ledger/`. Kill-switch: set
|
||||||
|
`autonomy.auto_act: off` → next pass escalates instead.
|
||||||
|
4. **Deploy-by-pull:** `git push` a trivial change; within 5 min mac-mini pulls
|
||||||
|
and `kickstart`s the changed daemon (no webhook involved).
|
||||||
|
5. **Cutover:** after decommission, `systemctl is-active` on apps shows the three
|
||||||
|
services `inactive`; production traffic served only by mac-mini; MCP tools +
|
||||||
|
console + a test enrollment all still work.
|
||||||
|
|
||||||
|
## Out of scope (per prior direction)
|
||||||
|
|
||||||
|
SSH-signed approval requests, Authentik step-up on the console POST, Prometheus /
|
||||||
|
temperature trend signals, and the generalize-to-any-homelab refactor.
|
||||||
Reference in New Issue
Block a user