Files
oikos/infrastructure/homelab-context.md
dtoro e32b21072d claudio-bot: decommission LXC 123, archive repos, migrate monitoring to Hermes
- Destroy LXC 123 (claudio-bot) — freed 8 GiB rootfs, 512 MiB RAM, 1 core
- Archive dtoro/claudio-bot and dtoro/claudio-monitor on Gitea (read-only)
- Stop claudio-monitor.timer on hubris, remove /opt/claudio-monitor
- Extend homelab-hardware-health skill with LXC resources, service health, apt/docker drift
- Create homelab-health-watchdog cron (15 min, Matrix alerts, actionable options)
- Wire Matrix (matrix:dtoro) as health alert delivery platform
- Update 13 files: inventory, containers/*, infrastructure/*, hosts/*, README, .sops.yaml
- Add deprecation plan at plans/2026-06-04_130000-deprecate-claudio-bot.md
2026-06-04 10:14:56 +02:00

144 lines
7.9 KiB
Markdown

# Homelab context distribution
The cross-client context-and-secrets system that makes every agent (Claude
Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
self-locating and able to read the same source of truth.
Operational walkthrough for enrolling a new client lives in
[operations/agent-enrollment.md](../operations/agent-enrollment.md); this
page is the architecture reference.
## What's where
| Piece | Host | Path | Role |
| --- | --- | --- | --- |
| Source of truth | [gitea (104)](../containers/104-gitea.md) | `dtoro/Homelab-Docs.git` | Inventory + wiki + service code |
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, Hermes Agent |
| `homelab` CLI | every enrolled client | `/usr/local/bin/homelab``/opt/homelab-context/bin/homelab` (symlink) | Operator surface for enroll/secret/ssh/pct |
| Per-client age key | every enrolled client | `/etc/age/key.txt` (0600 root) | Decrypts SOPS-encrypted secrets the client is a recipient on |
| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/mcp) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets |
| Secrets-issuance | [apps (105)](../containers/105-apps.md) | `secrets-issuance.service` on port 9820 (https://secrets.hubris.network/issue) | Generates per-client age keypair on first bootstrap; idempotent; admin-token-gated `/revoke` |
| Sync timer | every enrolled client | `homelab-context-sync.timer` (Linux) / `network.hubris.homelab-context-sync.plist` (macOS) | `git pull --ff-only` every 5 min |
| Encrypted secrets | `dtoro/Homelab-Docs` | `secrets/*.yaml` (SOPS+age) | Recipients declared in `.sops.yaml` |
| Read-only context PAT | `dtoro` Gitea user, scope `read:repository` | given to operators out-of-band | Bootstrap-only — for the initial clone before SOPS works |
| Write-scoped PAT | `secrets/gitea-pat.yaml` (SOPS) | `homelab refresh-creds` swaps it into `/etc/homelab-context/git-credentials` | All post-bootstrap pushes (client lifecycle, wiki edits) |
## Data flow
```
dtoro/Homelab-Docs (gitea)
┌────────── push ────────┤ ◀── git push (write PAT or SSH)
│ │
│ ┌────── push ──────┘
│ │ │
│ │ ▼ webhook (push event)
│ │ ┌─── homelab-mcp-deploy ──── (LXC 105:9811)
│ │ └─── secrets-issuance-deploy (LXC 105:9821)
│ │ │
│ │ ▼
│ │ git pull → deploy.sh → restart service
│ │
│ └── on every client:
│ timer (5 min) → git pull --ff-only into /opt/homelab-context
homelab CLI / MCP server reads /opt/homelab-context for everything
```
## Why two clones on LXC 105
The MCP server and secrets-issuance each have their own clone
(`/opt/homelab-mcp`, `/opt/secrets-issuance`) **in addition to**
`/opt/homelab-context`. Reasons:
- The deploy webhook for each service updates its own clone, runs
`deploy.sh` from there, and re-installs the systemd unit. Mixing this
with the client-context clone would create a circular dependency
(deploy reinstalls the unit that pulled it).
- The MCP server reads its data from `/opt/homelab-context` (the same path
every client uses) so changes to inventory propagate identically. Code
changes live in `/opt/homelab-mcp` and trigger a service restart.
## Mesh / network gates
- Both services bind `0.0.0.0:<port>`. The trust boundary is
`MESH_SUBNETS` in the service's environment + nftables (planned). Today
`MESH_SUBNETS=100.122.0.0/16,100.64.0.0/10,192.168.8.0/24` — Netbird +
Tailscale + the homelab LAN. Adjust if the LAN ever has untrusted
devices.
- Caddy fronts both with Let's Encrypt certs via the IONOS DNS challenge:
`mcp.hubris.network``192.168.8.205:9810`,
`secrets.hubris.network``192.168.8.205:9820`. Off-LAN clients on
Netbird reach them via the `192.168.8.0/24` network resource routed
through the PVE peer ([mesh.md](mesh.md)).
- Clients with default-public DNS (workstations not on Netbird, LXCs
using router DNS) need a `/etc/hosts` override pointing
`mcp.hubris.network` and `secrets.hubris.network` at the caddy LXC
(`192.168.8.175`) — same caveat as every other `*.hubris.network`
service, see [dns.md](dns.md).
## Secrets model
- Each enrolled client gets one **age private key** issued by
secrets-issuance on first bootstrap. The key file stays root-only on
the client; the public key is committed to `inventory.yaml` (and
becomes a recipient on SOPS-encrypted files via `.sops.yaml`).
- `secrets/*.yaml` are SOPS+age. Recipients per file are pinned in
`.sops.yaml` `creation_rules` by `path_regex`. Re-encrypting a file is
`sops updatekeys -y secrets/<name>.yaml`.
- The MCP server's `list_my_secrets(caller_pubkey)` tool returns only
secret *names* a given pubkey can decrypt — the server never sees
plaintext. Decryption is local-on-client (`homelab secret <name>`
shells out to `sops -d` with the client's key).
- The "all-clients" secrets (`hello.yaml` for the bootstrap decrypt
test, `gitea-pat.yaml` for the write-scoped PAT) are auto-granted to
every newly enrolled client by `homelab client add --finalize-pubkey`
(which appends the pubkey to the matching `.sops.yaml` rule and runs
`sops updatekeys`).
- **Removal does not erase past disclosure.** Revoking a client via
`homelab client remove` shreds the issuance-side key, denylists the
hostname, removes them from the recipient list, and re-keys all
shared secrets — but anything they already decrypted to disk is out of
your control. Rotate the underlying credential if compromise is
suspected.
## Why this design
- **One source of truth** keeps inventory, code, secrets, and docs
versioned together. A `git log` of `inventory.yaml` is the history of
the homelab.
- **Per-client age keys** scale better than a shared admin secret —
removing a client is a real revocation (for new ciphertext), not just
removing them from a wiki page.
- **MCP layer over the same clone** gives MCP-capable agents structured
query (`find_service`, `search_docs`) without forcing non-MCP tools to
go without — anything can still `cat` the markdown.
- **Sync timer rather than push fan-out** keeps the failure mode
contained: one client's webhook outage doesn't block a push from
landing on the others. Sub-5-min staleness is fine for docs and rare
enough for secrets that we don't need lower latency.
## Related
- [Operations: agent enrollment](../operations/agent-enrollment.md) — the
step-by-step for adding a new client
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
pipelines (and the rest of the lab's webhook pipelines)
- [Mesh](mesh.md) — Netbird / Tailscale paths and the `192.168.8.0/24`
network resource
- [Apps (105)](../containers/105-apps.md) — where both services run
- [Gitea (104)](../containers/104-gitea.md) — the source of truth
## Changelog
### 2026-05-20 — system live across hubris, apps, republic-laptop
Phase 1 of the [cross-client context plan](../README.md) merged. Three
clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS
decrypt verified on each. Webhook auto-deploy for both LXC 105 services
wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
`client add --finalize-pubkey` grant flow live so new clients are one
ceremony instead of four manual steps. Outstanding: bootstrap mac-mini
(macOS, exercises launchd) + ludo-mini + the remaining LXCs;
Hermes Agent integration so the agent uses inventory at chat-time.