From 3c25f936d3a1aa7ca399913d9724d82bfd496101 Mon Sep 17 00:00:00 2001 From: root Date: Wed, 20 May 2026 15:47:48 +0200 Subject: [PATCH] Phase 1: cross-client homelab context + MCP scaffolding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the foundation for distributing homelab context to every client (LXCs, VMs, workstations including republic-laptop, mac-mini, ludo-mini) with a single source of truth, structured query layer (MCP), and per-client age-key issuance for secrets: - inventory.yaml — canonical topology (hosts, services, mesh addresses) - hosts/*.yaml — per-host identity files generated from inventory by mcp/build_host_files.py; do not edit by hand - AGENTS.md — orientation doc symlinked to /root/AGENTS.md on every client - bootstrap.sh — one-shot enroll (Linux + macOS), clones repo, fetches age key from issuance, installs sync timer/launchd job, drops the homelab CLI - bin/homelab — single-binary Python CLI: whoami, list, ssh, pct, logs, restart, open, status, secret, sync, mcp, client add/remove, nuke - mcp/server.py — FastMCP server: context tools + read-only management tools (no mutations exposed); shell-outs use mcp-reader restricted ssh key - mcp/deploy/ — claudio-monitor-style gitea webhook deploy scaffold for the MCP service on LXC 105 (ports 9810 mcp, 9811 webhook) - secrets-issuance/ — per-client age key auto-provisioning over the mesh; source-IP gated against inventory, with denylist for revoked clients (ports 9820 issue, 9821 webhook) - secrets/, .sops.yaml — SOPS recipient scaffolding; the operator fills in age public keys after Phase 3a generates them - scripts/sync/ — systemd timer (Linux) + launchd plist (macOS) pulling /opt/homelab-context every 5 min Mesh: both Netbird (preferred, 100.122.0.0/16) and Tailscale accepted during the in-flight migration; no client is gated on completing the move. Plan reference: /root/.claude/plans/lets-make-a-plan-fluttering-trinket.md Co-Authored-By: Claude Opus 4.7 (1M context) --- .sops.yaml | 36 ++ AGENTS.md | 89 ++++ bin/homelab | 494 ++++++++++++++++++ bootstrap.sh | 266 ++++++++++ hosts/apps.yaml | 50 ++ hosts/arriman.yaml | 34 ++ hosts/authentik.yaml | 35 ++ hosts/caddy.yaml | 35 ++ hosts/claudio-bot.yaml | 24 + hosts/elementsynapse.yaml | 29 + hosts/gitea.yaml | 34 ++ hosts/haos.yaml | 26 + hosts/hubris.yaml | 33 ++ hosts/jellyfin.yaml | 31 ++ hosts/ludo-mini.yaml | 17 + hosts/mac-mini.yaml | 19 + hosts/mule-images.yaml | 31 ++ hosts/mule-photos-new.yaml | 29 + hosts/nextcloud.yaml | 31 ++ hosts/nfs-export.yaml | 22 + hosts/paperless.yaml | 31 ++ hosts/plato.yaml | 31 ++ hosts/republic-laptop.yaml | 17 + hosts/sophia.yaml | 24 + hosts/zimaos.yaml | 28 + inventory.yaml | 365 +++++++++++++ mcp/build_host_files.py | 158 ++++++ mcp/deploy/deploy.sh | 46 ++ mcp/deploy/homelab-mcp.service | 24 + mcp/deploy/webhook/homelab-mcp-deploy.service | 13 + mcp/deploy/webhook/install.sh | 31 ++ mcp/deploy/webhook/webhook.py | 95 ++++ mcp/server.py | 310 +++++++++++ scripts/sync/install.sh | 41 ++ .../sync/linux/homelab-context-sync.service | 15 + scripts/sync/linux/homelab-context-sync.timer | 11 + .../network.hubris.homelab-context-sync.plist | 32 ++ secrets-issuance/deploy/deploy.sh | 57 ++ secrets-issuance/deploy/webhook/install.sh | 30 ++ .../webhook/secrets-issuance-deploy.service | 13 + secrets-issuance/deploy/webhook/webhook.py | 91 ++++ secrets-issuance/server.py | 266 ++++++++++ secrets-issuance/server.service | 23 + secrets/README.md | 63 +++ 44 files changed, 3180 insertions(+) create mode 100644 .sops.yaml create mode 100644 AGENTS.md create mode 100755 bin/homelab create mode 100755 bootstrap.sh create mode 100644 hosts/apps.yaml create mode 100644 hosts/arriman.yaml create mode 100644 hosts/authentik.yaml create mode 100644 hosts/caddy.yaml create mode 100644 hosts/claudio-bot.yaml create mode 100644 hosts/elementsynapse.yaml create mode 100644 hosts/gitea.yaml create mode 100644 hosts/haos.yaml create mode 100644 hosts/hubris.yaml create mode 100644 hosts/jellyfin.yaml create mode 100644 hosts/ludo-mini.yaml create mode 100644 hosts/mac-mini.yaml create mode 100644 hosts/mule-images.yaml create mode 100644 hosts/mule-photos-new.yaml create mode 100644 hosts/nextcloud.yaml create mode 100644 hosts/nfs-export.yaml create mode 100644 hosts/paperless.yaml create mode 100644 hosts/plato.yaml create mode 100644 hosts/republic-laptop.yaml create mode 100644 hosts/sophia.yaml create mode 100644 hosts/zimaos.yaml create mode 100644 inventory.yaml create mode 100755 mcp/build_host_files.py create mode 100755 mcp/deploy/deploy.sh create mode 100644 mcp/deploy/homelab-mcp.service create mode 100644 mcp/deploy/webhook/homelab-mcp-deploy.service create mode 100755 mcp/deploy/webhook/install.sh create mode 100755 mcp/deploy/webhook/webhook.py create mode 100755 mcp/server.py create mode 100755 scripts/sync/install.sh create mode 100644 scripts/sync/linux/homelab-context-sync.service create mode 100644 scripts/sync/linux/homelab-context-sync.timer create mode 100644 scripts/sync/macos/network.hubris.homelab-context-sync.plist create mode 100755 secrets-issuance/deploy/deploy.sh create mode 100755 secrets-issuance/deploy/webhook/install.sh create mode 100644 secrets-issuance/deploy/webhook/secrets-issuance-deploy.service create mode 100755 secrets-issuance/deploy/webhook/webhook.py create mode 100755 secrets-issuance/server.py create mode 100644 secrets-issuance/server.service create mode 100644 secrets/README.md diff --git a/.sops.yaml b/.sops.yaml new file mode 100644 index 0000000..c84f3e0 --- /dev/null +++ b/.sops.yaml @@ -0,0 +1,36 @@ +# SOPS recipient rules. +# +# Each rule pins one or more age public keys as recipients for files matching +# the path_regex. The build_host_files.py generator doesn't read this file; +# `sops` does — to encrypt a new secret, run `sops -e secrets/.yaml` +# from the repo root and SOPS will pick the matching rule below. +# +# To grant a secret to a new client: add their age public key (from +# inventory.yaml `hosts..age_pubkey`) to the relevant rule below, then +# run `sops updatekeys secrets/.yaml` to re-encrypt without rotating +# the ciphertext payload. +# +# To revoke: remove the recipient from the relevant rule and run +# `sops updatekeys` (this is what `homelab client remove` calls). Past +# ciphertext the client already decrypted is not affected — rotate the +# underlying credential if compromise is suspected. + +creation_rules: + - path_regex: ^secrets/hello\.yaml$ + # The "hello" secret is encrypted to every enrolled client so the bootstrap + # decrypt test works for everyone. Populated by the operator with the + # age_pubkey from each hosts/.yaml after Phase 3a generates keys. + age: >- + # placeholder — operator fills in after first issuance run. + # comma-separated age public keys, e.g.: + # age1xxx,age1yyy,age1zzz + + - path_regex: ^secrets/gitea-tokens\.yaml$ + # Workstations only. + age: >- + # placeholder — fill with age_pubkey of: republic-laptop, mac-mini, ludo-mini, hubris + + - path_regex: ^secrets/webhook-hmacs\.yaml$ + # LXCs that run a webhook receiver. + age: >- + # placeholder — fill with age_pubkey of: apps, caddy, claudio-bot, claudio-monitor host diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..31888bf --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,89 @@ +# AGENTS.md — orientation for any agent on a homelab client + +You are running on a machine that is part of the **hubris** homelab. The full +context is in this checkout at `/opt/homelab-context/`. This file is the entry +point. Read it once at start, then keep working. + +## 1. Who you are + +Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read: + + /opt/homelab-context/hosts/.yaml + +That file tells you your role, your peers, what's mounted, and what services +you host. If it does not exist, this client was not enrolled — stop and tell +the operator to run `homelab client add ` from an existing client. + +## 2. The topology + +- `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation + with their mesh addresses, roles, and service mappings. Treat this file as + authoritative; anything you read in narrative pages should agree with it. +- `/opt/homelab-context/infrastructure/mesh.md` — Tailscale → Netbird state. + Both meshes are accepted today; Netbird is preferred for new traffic. +- `/opt/homelab-context/infrastructure/dns.md` — split-horizon DNS via + dnsmasq on LXC 124. `*.hubris.network` resolves to 192.168.x.x on the LAN + and to mesh addresses off-LAN. +- `/opt/homelab-context/operations/commands.md` — the operator's cheatsheet + for pct, caddy, dnsmasq. Use these verbs when you take actions. + +## 3. The MCP server + +The homelab exposes a Model Context Protocol server with structured tools. +Endpoint is in `inventory.yaml` under `services.homelab_mcp.endpoint`. + +Available tools: + + Context (pure read): + get_host(name), list_services(), find_service(name_or_role), + get_topology(), search_docs(query), get_page(path), + get_changelog(page, since?), whoami(hostname), + list_my_secrets(caller_pubkey?) + + Management (read-only): + get_service_status(service), tail_log(service, lines=200), + list_lxcs(), get_lxc_state(lxc), ping_service(service) + +Mutations are **not** exposed via MCP. Use the `homelab` CLI for those, with +operator confirmation. + +**When to prefer MCP over grepping the clone:** any time you need to resolve a +name to an address, look up service status, or search the wiki by content. +Grep is fine for browsing or when MCP is unreachable. + +## 4. Wiki conventions + +- Pages live under `containers/`, `hosts/`, `vms/`, `infrastructure/`, + `investigations/`, `operations/`. Cross-link liberally; orphans are bugs. +- Every page ends with a `## Changelog` section, entries in reverse-chrono + order: + + ### YYYY-MM-DD — short title + one or two lines describing what changed and why. + +- Investigation files are dated and slugged: `YYYY-MM-DD-slug.md`. +- Live state takes precedence over docs. If you observe a discrepancy, update + the docs *in the same session* (per the same-session update rule). + +## 5. Acting on the homelab + +- **Read state**: prefer MCP tools, then files, then shell. Examples: + `homelab whoami`, `homelab list`, `homelab status`, `homelab logs caddy`. +- **Cross-host actions** (caddy reload, pct exec, etc.): use the `homelab` + CLI — it resolves hostname → mesh address → ssh / pct path for you. Direct + SSH still works; the CLI just removes the lookup burden. +- **Secrets**: never hardcode. Call `homelab secret ` to decrypt on + demand using the per-client age key at `/etc/age/key.txt`. Secrets ARE + available in this system — `list_my_secrets()` (MCP) shows what you can + decrypt. +- **Mutations** (restart, edit configs, etc.): the `homelab` CLI's mutating + subcommands ask for confirmation. For ad-hoc work, SSH and edit directly — + but commit changes that touch tracked configs (caddy, gitea custom, + artifacto, mule-image, etc.; see `infrastructure/auto-deploy.md`). +- **Wiki updates**: same-session rule applies to any meaningful state change + this client makes. + +## 6. When in doubt + +Run `homelab mcp search_docs ` or `homelab mcp get_host `. +The clone is the fallback; MCP is the index. diff --git a/bin/homelab b/bin/homelab new file mode 100755 index 0000000..a5d4835 --- /dev/null +++ b/bin/homelab @@ -0,0 +1,494 @@ +#!/usr/bin/env python3 +""" +homelab — single-binary CLI for managing the hubris homelab from any client. + +Reads /opt/homelab-context/inventory.yaml as the source of truth, resolves +hostnames to mesh addresses, and wraps the common ops (ssh, pct, logs, restart, +status, open, secret, client add/remove, sync, mcp). + +Mutating subcommands prompt unless -y / --yes is passed. + +Run `homelab --help` or `homelab --help` for usage. +""" + +from __future__ import annotations + +import argparse +import json +import os +import shutil +import subprocess +import sys +import urllib.request +from pathlib import Path + +try: + import yaml +except ImportError: + print("PyYAML required (apt install python3-yaml or pip install pyyaml)", file=sys.stderr) + sys.exit(2) + +CONTEXT = Path(os.environ.get("HOMELAB_CONTEXT_DIR", "/opt/homelab-context")) +INVENTORY = CONTEXT / "inventory.yaml" +HOSTS_DIR = CONTEXT / "hosts" +AGE_KEY = Path(os.environ.get("SOPS_AGE_KEY_FILE", "/etc/age/key.txt")) + + +# ---------- helpers ---------- + +def die(msg: str, code: int = 1) -> None: + print(f"homelab: {msg}", file=sys.stderr) + sys.exit(code) + + +def inventory() -> dict: + if not INVENTORY.exists(): + die(f"no inventory at {INVENTORY} — has bootstrap run?") + return yaml.safe_load(INVENTORY.read_text()) + + +def host(name: str) -> dict: + inv = inventory() + if name not in inv.get("hosts", {}): + die(f"unknown host: {name}") + return inv["hosts"][name] + + +def host_address(name: str, prefer_lan: bool = False) -> str: + h = host(name) + mesh = h.get("mesh", {}) + candidates = [] + if prefer_lan and h.get("lan_ip"): + candidates.append(h["lan_ip"]) + candidates.extend([ + mesh.get("netbird", {}).get("fqdn") if isinstance(mesh.get("netbird"), dict) else None, + mesh.get("netbird", {}).get("ip") if isinstance(mesh.get("netbird"), dict) else None, + h.get("lan_ip"), + mesh.get("tailscale", {}).get("fqdn") if isinstance(mesh.get("tailscale"), dict) else None, + mesh.get("tailscale", {}).get("ip") if isinstance(mesh.get("tailscale"), dict) else None, + ]) + for c in candidates: + if c: + return c + die(f"no reachable address for host {name}") + + +def hubris_ssh() -> list[str]: + """SSH base command for hubris, honoring its non-default Netbird SSH port.""" + h = host("hubris") + port = h.get("ssh", {}).get("netbird_port", 22) + return ["ssh", "-p", str(port), f"root@{host_address('hubris')}"] + + +def confirm(question: str, default_no: bool = True) -> bool: + suffix = "[y/N]" if default_no else "[Y/n]" + try: + answer = input(f"{question} {suffix} ").strip().lower() + except (EOFError, KeyboardInterrupt): + return False + if not answer: + return not default_no + return answer in ("y", "yes") + + +def service(name: str) -> dict: + inv = inventory() + svc = inv.get("services", {}).get(name) + if not svc: + die(f"unknown service: {name}") + return svc + + +def service_backend_host(name: str) -> str: + return service(name)["backend"] + + +def push_inventory(message: str) -> None: + """Stage + commit + push inventory + regenerated hosts/.""" + subprocess.run(["python3", str(CONTEXT / "mcp" / "build_host_files.py")], + check=True, cwd=CONTEXT) + subprocess.run(["git", "add", "inventory.yaml", "hosts/"], check=True, cwd=CONTEXT) + if subprocess.run(["git", "diff", "--cached", "--quiet"], cwd=CONTEXT).returncode == 0: + print("(no changes to commit)") + return + subprocess.run(["git", "commit", "-m", message], check=True, cwd=CONTEXT) + subprocess.run(["git", "push"], check=True, cwd=CONTEXT) + + +# ---------- subcommands ---------- + +def cmd_whoami(args: argparse.Namespace) -> int: + name = args.hostname or os.uname().nodename.split(".")[0] + path = HOSTS_DIR / f"{name}.yaml" + if not path.exists(): + die(f"no hosts/{name}.yaml — has this client been enrolled?") + print(path.read_text()) + return 0 + + +def cmd_list(args: argparse.Namespace) -> int: + inv = inventory() + print(f"{'NAME':<22} {'KIND':<14} {'OS':<6} {'ROLE':<22} ADDRESS") + for name, entry in inv.get("hosts", {}).items(): + addr = "" + try: + addr = host_address(name) + except SystemExit: + addr = "?" + print(f"{name:<22} {entry.get('kind','?'):<14} {entry.get('os','?'):<6} " + f"{entry.get('role','?'):<22} {addr}") + print() + print("Services:") + for svc, entry in inv.get("services", {}).items(): + url = entry.get("url") or entry.get("endpoint") or "" + backend = entry.get("backend", "?") + print(f" {svc:<22} backend={backend:<15} {url}") + return 0 + + +def cmd_ssh(args: argparse.Namespace) -> int: + name = args.host + h = host(name) + addr = host_address(name) + cmd = ["ssh"] + # hubris uses a non-default Netbird SSH port for the mesh path + if name == "hubris": + port = h.get("ssh", {}).get("netbird_port", 22) + cmd.extend(["-p", str(port)]) + user = args.user or "root" + cmd.append(f"{user}@{addr}") + if args.command: + cmd.append(" ".join(args.command)) + os.execvp(cmd[0], cmd) + + +def cmd_pct(args: argparse.Namespace) -> int: + lxc = args.lxc + inv = inventory() + entry = inv.get("hosts", {}).get(lxc) + if not entry or entry.get("kind") != "lxc": + die(f"{lxc} is not an LXC in inventory") + pve_id = str(entry["pve_id"]) + action = args.action + if action == "exec": + if not args.rest: + die("pct exec needs a command") + remote_cmd = ["pct", "exec", pve_id, "--"] + args.rest + elif action in ("status", "config", "start", "stop", "reboot", "shutdown"): + remote_cmd = ["pct", action, pve_id] + elif action == "enter": + remote_cmd = ["pct", "enter", pve_id] + else: + die(f"unknown pct action: {action}") + if action in ("stop", "reboot", "shutdown") and not args.yes: + if not confirm(f"run 'pct {action} {pve_id}' (={lxc}) on hubris?"): + return 1 + cmd = hubris_ssh() + ["--"] + remote_cmd + return subprocess.call(cmd) + + +def cmd_logs(args: argparse.Namespace) -> int: + svc = args.service + host_name = service_backend_host(svc) + unit = service(svc).get("systemd_unit", svc) + if host_name == "hubris": + base = hubris_ssh() + else: + base = ["ssh", f"root@{host_address(host_name)}"] + remote = ["journalctl", "-u", unit, "-n", str(args.lines), "--no-pager"] + if args.follow: + remote.append("-f") + return subprocess.call(base + ["--"] + remote) + + +def cmd_restart(args: argparse.Namespace) -> int: + svc = args.service + host_name = service_backend_host(svc) + unit = service(svc).get("systemd_unit", svc) + if not args.yes: + if not confirm(f"restart systemd unit '{unit}' on {host_name}?"): + return 1 + if host_name == "hubris": + base = hubris_ssh() + else: + base = ["ssh", f"root@{host_address(host_name)}"] + return subprocess.call(base + ["--", "systemctl", "restart", unit]) + + +def cmd_open(args: argparse.Namespace) -> int: + url = service(args.service).get("url") + if not url: + die(f"service {args.service} has no url in inventory") + opener = "open" if sys.platform == "darwin" else "xdg-open" + if shutil.which(opener) is None: + print(url) + return 0 + return subprocess.call([opener, url]) + + +def cmd_status(args: argparse.Namespace) -> int: + inv = inventory() + print(f"{'NAME':<22} REACH") + for name, entry in inv.get("hosts", {}).items(): + try: + addr = host_address(name) + except SystemExit: + print(f"{name:<22} no-address") + continue + # Quick ping-ish check via mesh. + proc = subprocess.run( + ["ping", "-c", "1", "-W", "2", addr], + capture_output=True, text=True, + ) + reach = "ok" if proc.returncode == 0 else "down" + print(f"{name:<22} {reach:<5} {addr}") + print() + print(f"{'SERVICE':<22} URL HEALTH") + for svc, entry in inv.get("services", {}).items(): + url = entry.get("url") or entry.get("endpoint") + if not url: + continue + proc = subprocess.run( + ["curl", "-sS", "-o", "/dev/null", "-w", "%{http_code}", + "--max-time", "3", url], + capture_output=True, text=True, + ) + code = proc.stdout.strip() or "---" + print(f"{svc:<22} {url:<50} {code}") + return 0 + + +def cmd_secret(args: argparse.Namespace) -> int: + name = args.name + path = CONTEXT / "secrets" / f"{name}.yaml" + if not path.exists(): + die(f"no secret '{name}' (looked for {path})") + if not AGE_KEY.exists(): + die(f"no age key at {AGE_KEY} — has bootstrap run?") + env = {**os.environ, "SOPS_AGE_KEY_FILE": str(AGE_KEY)} + return subprocess.call(["sops", "-d", str(path)], env=env) + + +def cmd_sync(args: argparse.Namespace) -> int: + if sys.platform == "darwin": + return subprocess.call( + ["sudo", "launchctl", "kickstart", "-k", + "system/network.hubris.homelab-context-sync"] + ) + return subprocess.call( + ["sudo", "systemctl", "start", "homelab-context-sync.service"] + ) + + +def cmd_mcp(args: argparse.Namespace) -> int: + svc = service("homelab_mcp") + endpoint = svc.get("endpoint") + if not endpoint: + die("no MCP endpoint in inventory") + # Minimal SSE invocation via mcp-cli if available; otherwise instruct. + if shutil.which("mcp"): + return subprocess.call(["mcp", "call", endpoint, args.tool, *(args.args or [])]) + die("'mcp' CLI not installed. Install with: pip install 'mcp[cli]'") + + +def cmd_client_add(args: argparse.Namespace) -> int: + name = args.name + inv = inventory() + if not args.finalize_pubkey: + if name in inv["hosts"]: + die(f"{name} already in inventory (use --finalize-pubkey to update age_pubkey)") + print(f"Adding new client '{name}' to inventory.yaml.") + kind = input(" kind [workstation/lxc/vm] (default: workstation): ").strip() or "workstation" + os_name = input(f" os [linux/macos] (default: linux): ").strip() or "linux" + netbird_fqdn = input(f" netbird FQDN (default: {name}.netbird.selfhosted): ").strip() \ + or f"{name}.netbird.selfhosted" + role = input(" role (e.g. primary-dev, dev): ").strip() or "dev" + entry = { + "kind": kind, + "os": os_name, + "role": role, + "mesh": {"netbird": {"fqdn": netbird_fqdn}}, + "age_pubkey": "", + } + inv["hosts"][name] = entry + INVENTORY.write_text(yaml.safe_dump(inv, sort_keys=False)) + push_inventory(f"client-add: {name}") + print() + print("Next steps:") + print(f" 1. Join {name} to Netbird (out-of-band, Netbird console / setup key).") + print(f" 2. On {name}: curl -fsSL /dtoro/Homelab-Docs/raw/main/bootstrap.sh | sudo bash") + print(f" 3. bootstrap prints an age pubkey — bring it back here and run:") + print(f" homelab client add {name} --finalize-pubkey ") + return 0 + + # finalize_pubkey path + if name not in inv["hosts"]: + die(f"{name} not in inventory — run 'homelab client add {name}' first (no --finalize-pubkey)") + inv["hosts"][name]["age_pubkey"] = args.finalize_pubkey + INVENTORY.write_text(yaml.safe_dump(inv, sort_keys=False)) + push_inventory(f"client-add: {name} (finalize age_pubkey)") + print(f"finalized age_pubkey for {name}.") + return 0 + + +def cmd_client_remove(args: argparse.Namespace) -> int: + name = args.name + inv = inventory() + if name not in inv["hosts"]: + die(f"{name} not in inventory") + if not args.yes: + print(f"This will:") + print(f" 1. Remove {name} from inventory.yaml and hosts/") + print(f" 2. Re-encrypt every secret without {name} as recipient") + print(f" 3. Revoke {name}'s age key on the issuance server (shred + denylist)") + print(f" 4. Commit + push the change") + print(f"After: rotate any credentials inside secrets {name} previously had access to,") + print(f" and revoke {name}'s Netbird peer in the console.") + if not confirm(f"proceed removing {name}?"): + return 1 + + # 1. Inventory + del inv["hosts"][name] + INVENTORY.write_text(yaml.safe_dump(inv, sort_keys=False)) + + # 2. SOPS — remove recipient. Requires `sops updatekeys` after we edit .sops.yaml. + # We don't try to programmatically edit .sops.yaml because the recipient list + # there is keyed by path-glob rules; the operator must remove the pubkey line + # if it's listed by-name. We'll trigger updatekeys after the operator confirms. + secrets_dir = CONTEXT / "secrets" + if secrets_dir.exists(): + print() + print("[remove] re-encrypting secrets without removed recipient (sops updatekeys)") + sops_yaml = CONTEXT / ".sops.yaml" + if sops_yaml.exists(): + print(f" Note: review {sops_yaml} for hard-coded recipients of '{name}' " + "and remove them before sops updatekeys.") + for f in sorted(secrets_dir.glob("*.yaml")): + subprocess.run(["sops", "updatekeys", "-y", str(f)], check=False) + + # 3. Revoke on issuance server + admin_token_url = inv["services"].get("secrets_issuance", {}).get("endpoint", "").replace("/issue", "") + if admin_token_url: + revoke_url = admin_token_url.rstrip("/") + "/revoke" + token = os.environ.get("HOMELAB_ISSUANCE_ADMIN_TOKEN", "") + if not token: + print(f" WARNING: set HOMELAB_ISSUANCE_ADMIN_TOKEN to call {revoke_url} — skipping revocation") + else: + req = urllib.request.Request( + revoke_url, data=json.dumps({"hostname": name}).encode(), + headers={"X-Admin-Token": token, "Content-Type": "application/json"}, + method="POST", + ) + try: + with urllib.request.urlopen(req, timeout=10) as resp: + print(f" issuance revoke: {resp.status} {resp.read().decode().strip()}") + except Exception as e: + print(f" issuance revoke failed: {e}") + + # 4. Commit + push + push_inventory(f"client-remove: {name}") + + print() + print("Follow-up checklist (the CLI cannot do these automatically):") + print(f" [ ] Revoke {name}'s Netbird peer in the Netbird console.") + print(f" [ ] Rotate any credentials whose ciphertext {name} already has on disk") + print(f" — the only real revocation for past-disclosed secrets is rotation.") + print(f" [ ] If the machine is reachable and decommissioned, run:") + print(f" homelab nuke {name} (shreds /etc/age/key.txt, removes /opt/homelab-context)") + return 0 + + +def cmd_nuke(args: argparse.Namespace) -> int: + name = args.name + if not args.yes: + if not confirm(f"destroy /etc/age/key.txt + /opt/homelab-context on {name}?"): + return 1 + addr = host_address(name) + remote = ("set -euo pipefail; " + "shred -u /etc/age/key.txt 2>/dev/null || true; " + "rm -rf /opt/homelab-context; " + "systemctl disable --now homelab-context-sync.timer 2>/dev/null || true; " + "launchctl bootout system/network.hubris.homelab-context-sync 2>/dev/null || true; " + "echo nuked") + return subprocess.call(["ssh", f"root@{addr}", remote]) + + +# ---------- argparse ---------- + +def main() -> int: + p = argparse.ArgumentParser(prog="homelab", description=__doc__) + sub = p.add_subparsers(dest="cmd", required=True) + + sp = sub.add_parser("whoami", help="print this host's hosts/.yaml") + sp.add_argument("hostname", nargs="?") + sp.set_defaults(func=cmd_whoami) + + sp = sub.add_parser("list", help="list hosts and services from inventory") + sp.set_defaults(func=cmd_list) + + sp = sub.add_parser("ssh", help="ssh to a host via mesh") + sp.add_argument("host") + sp.add_argument("--user", "-u", default=None) + sp.add_argument("command", nargs=argparse.REMAINDER) + sp.set_defaults(func=cmd_ssh) + + sp = sub.add_parser("pct", help="proxy pct commands via ssh to hubris") + sp.add_argument("lxc") + sp.add_argument("action") + sp.add_argument("rest", nargs=argparse.REMAINDER) + sp.add_argument("--yes", "-y", action="store_true") + sp.set_defaults(func=cmd_pct) + + sp = sub.add_parser("logs", help="journalctl for a service") + sp.add_argument("service") + sp.add_argument("--lines", "-n", type=int, default=200) + sp.add_argument("--follow", "-f", action="store_true") + sp.set_defaults(func=cmd_logs) + + sp = sub.add_parser("restart", help="restart a service") + sp.add_argument("service") + sp.add_argument("--yes", "-y", action="store_true") + sp.set_defaults(func=cmd_restart) + + sp = sub.add_parser("open", help="open a service's URL in browser") + sp.add_argument("service") + sp.set_defaults(func=cmd_open) + + sp = sub.add_parser("status", help="ping every host + HTTP-check every service") + sp.set_defaults(func=cmd_status) + + sp = sub.add_parser("secret", help="decrypt a secret (sops -d wrapper)") + sp.add_argument("name") + sp.set_defaults(func=cmd_secret) + + sp = sub.add_parser("sync", help="manually trigger homelab-context-sync") + sp.set_defaults(func=cmd_sync) + + sp = sub.add_parser("mcp", help="call an MCP tool (requires 'mcp' CLI installed)") + sp.add_argument("tool") + sp.add_argument("args", nargs=argparse.REMAINDER) + sp.set_defaults(func=cmd_mcp) + + sp = sub.add_parser("nuke", help="shred /etc/age/key.txt + /opt/homelab-context on a host") + sp.add_argument("name") + sp.add_argument("--yes", "-y", action="store_true") + sp.set_defaults(func=cmd_nuke) + + client = sub.add_parser("client", help="client lifecycle (add/remove)") + csub = client.add_subparsers(dest="action", required=True) + csub_add = csub.add_parser("add") + csub_add.add_argument("name") + csub_add.add_argument("--finalize-pubkey", default=None, + help="set/update age_pubkey for an already-added client") + csub_add.set_defaults(func=cmd_client_add) + csub_rm = csub.add_parser("remove") + csub_rm.add_argument("name") + csub_rm.add_argument("--yes", "-y", action="store_true") + csub_rm.set_defaults(func=cmd_client_remove) + + args = p.parse_args() + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bootstrap.sh b/bootstrap.sh new file mode 100755 index 0000000..888cbbf --- /dev/null +++ b/bootstrap.sh @@ -0,0 +1,266 @@ +#!/usr/bin/env bash +# bootstrap.sh — enroll a new client into the homelab context system. +# +# Usage: +# curl -fsSL https://git.hubris.network/dtoro/Homelab-Docs/raw/main/bootstrap.sh \ +# | sudo bash +# curl ... | sudo bash -s -- --with-mcp # also wire Claude's .mcp.json +# curl ... | sudo bash -s -- --dry-run # show what would happen +# curl ... | sudo bash -s -- --no-secrets # skip age-key issuance +# +# Prerequisites the script verifies: +# - running as root +# - OS is Linux or macOS +# - git, age, sops are installed +# - at least one mesh (netbird OR tailscale) is connected +# - this host has an `hosts/.yaml` entry in the repo (or refuses) + +set -euo pipefail + +# -------- defaults -------- +REPO_HTTPS="${HOMELAB_REPO_URL:-https://git.hubris.network/dtoro/Homelab-Docs.git}" +CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab-context}" +ISSUANCE_URL_NETBIRD="${HOMELAB_ISSUANCE_NETBIRD:-http://apps.netbird.selfhosted:9820/issue}" +ISSUANCE_URL_TAILSCALE="${HOMELAB_ISSUANCE_TAILSCALE:-http://apps.ts:9820/issue}" +MCP_URL="${HOMELAB_MCP_URL:-http://apps.netbird.selfhosted:9810/sse}" + +WITH_MCP=0 +DRY_RUN=0 +NO_SECRETS=0 + +# -------- flag parsing -------- +while [ $# -gt 0 ]; do + case "$1" in + --with-mcp) WITH_MCP=1; shift ;; + --dry-run) DRY_RUN=1; shift ;; + --no-secrets) NO_SECRETS=1; shift ;; + --help|-h) + sed -n '2,11p' "$0" | sed 's/^# *//' + exit 0 + ;; + *) + echo "unknown flag: $1" >&2; exit 2 ;; + esac +done + +run() { + if [ "$DRY_RUN" -eq 1 ]; then + printf '+ %s\n' "$*" + else + eval "$*" + fi +} + +# -------- preflight -------- +if [ "$(id -u)" -ne 0 ]; then + echo "bootstrap.sh must run as root (use sudo)." >&2 + exit 1 +fi + +OS="$(uname -s)" +case "$OS" in + Linux|Darwin) ;; + *) echo "unsupported OS: $OS" >&2; exit 1 ;; +esac + +# Resolve hostname; on macOS prefer LocalHostName if set. +if [ "$OS" = "Darwin" ]; then + HNAME="$(scutil --get LocalHostName 2>/dev/null || hostname -s)" + HNAME_ALT="$(hostname -s)" + if [ "$HNAME" != "$HNAME_ALT" ]; then + echo "note: scutil LocalHostName=$HNAME differs from hostname=$HNAME_ALT" + echo " using LocalHostName for inventory lookup." + fi +else + HNAME="$(hostname -s)" +fi +echo "[bootstrap] hostname: $HNAME" + +# Check dependencies. +missing=() +for cmd in git; do command -v "$cmd" >/dev/null || missing+=("$cmd"); done +if [ "$NO_SECRETS" -eq 0 ]; then + for cmd in age sops; do command -v "$cmd" >/dev/null || missing+=("$cmd"); done +fi +if [ "${#missing[@]}" -gt 0 ]; then + echo "missing required tools: ${missing[*]}" >&2 + if [ "$OS" = "Darwin" ]; then + echo " brew install ${missing[*]}" + else + echo " apt install -y ${missing[*]} (or platform equivalent)" + fi + exit 1 +fi + +# Mesh check — accept either Netbird OR Tailscale. +MESH_CONNECTED="" +if command -v netbird >/dev/null && netbird status 2>/dev/null | grep -q "Management: Connected"; then + MESH_CONNECTED="netbird" +elif command -v tailscale >/dev/null && tailscale status >/dev/null 2>&1; then + MESH_CONNECTED="tailscale" +fi +if [ -z "$MESH_CONNECTED" ] && [ "$NO_SECRETS" -eq 0 ]; then + echo "neither netbird nor tailscale is connected; cannot bootstrap." >&2 + echo "either bring up the mesh first, or pass --no-secrets to skip issuance." >&2 + exit 1 +fi +echo "[bootstrap] mesh: ${MESH_CONNECTED:-none (skipped, --no-secrets)}" + +# -------- clone -------- +if [ -d "$CLONE_DIR/.git" ]; then + existing_remote="$(git -C "$CLONE_DIR" remote get-url origin 2>/dev/null || true)" + if [ -n "$existing_remote" ] && [ "$existing_remote" != "$REPO_HTTPS" ]; then + echo "$CLONE_DIR already exists with a different remote ($existing_remote);" >&2 + echo "refusing to overwrite. Move it aside or set HOMELAB_REPO_URL." >&2 + exit 1 + fi + echo "[bootstrap] clone exists; pulling" + run "git -C '$CLONE_DIR' pull --ff-only --quiet" +else + echo "[bootstrap] cloning to $CLONE_DIR" + run "git clone --quiet '$REPO_HTTPS' '$CLONE_DIR'" +fi + +# -------- identity check -------- +HOST_YAML="$CLONE_DIR/hosts/$HNAME.yaml" +if [ ! -f "$HOST_YAML" ]; then + cat >&2 <' on Linux, or System Preferences → +Sharing on macOS.) +EOF + exit 1 +fi + +# -------- secrets issuance -------- +if [ "$NO_SECRETS" -eq 0 ]; then + if [ "$MESH_CONNECTED" = "netbird" ]; then + URL="$ISSUANCE_URL_NETBIRD" + else + URL="$ISSUANCE_URL_TAILSCALE" + fi + KEY_FILE=/etc/age/key.txt + run "mkdir -p /etc/age && chmod 0700 /etc/age" + + if [ -f "$KEY_FILE" ]; then + echo "[bootstrap] age key already exists at $KEY_FILE — verifying with issuance" + fi + echo "[bootstrap] requesting age key from $URL" + if [ "$DRY_RUN" -eq 0 ]; then + # The issuance endpoint identifies us by source mesh IP. No body needed. + HTTP_CODE=$(curl -sS -o /tmp/homelab-age-key -w '%{http_code}' \ + -X POST -H "Content-Type: application/json" \ + -d "{\"hostname\":\"$HNAME\"}" \ + "$URL") || HTTP_CODE=000 + case "$HTTP_CODE" in + 200) + mv /tmp/homelab-age-key "$KEY_FILE" + chmod 0600 "$KEY_FILE" + echo "[bootstrap] age key installed at $KEY_FILE" + # Capture the PR snippet (if returned in a sidecar header) so the + # operator knows the pubkey to add to inventory.yaml. The server + # includes it in the JSON response when generating a new key. + if grep -q '"pubkey"' "$KEY_FILE" 2>/dev/null; then + # Shouldn't happen — server should return raw key, not JSON. + echo "[bootstrap] unexpected: key file contains JSON, please inspect" >&2 + fi + ;; + 403) + echo "[bootstrap] issuance returned 403 — caller not recognized" >&2 + echo "is this peer in the Netbird/Tailscale console? is the hostname" >&2 + echo "above ('$HNAME') matching the inventory entry?" >&2 + exit 1 + ;; + *) + echo "[bootstrap] issuance failed (HTTP $HTTP_CODE)" >&2 + cat /tmp/homelab-age-key >&2 || true + exit 1 + ;; + esac + fi +fi + +# -------- install sync timer / launchd plist -------- +echo "[bootstrap] installing sync mechanism for $OS" +run "bash '$CLONE_DIR/scripts/sync/install.sh'" + +# -------- install homelab CLI -------- +echo "[bootstrap] installing homelab CLI to /usr/local/bin/homelab" +run "install -m 0755 '$CLONE_DIR/bin/homelab' /usr/local/bin/homelab" + +# -------- AGENTS.md symlink -------- +case "$OS" in + Linux) + AGENTS_LINK=/root/AGENTS.md + ;; + Darwin) + AGENTS_LINK=/etc/AGENTS.md + ;; +esac +run "ln -sfn '$CLONE_DIR/AGENTS.md' '$AGENTS_LINK'" +echo "[bootstrap] linked AGENTS.md → $AGENTS_LINK" + +# -------- MCP wiring -------- +if [ "$WITH_MCP" -eq 1 ]; then + # Pick the right user's home — when invoked via sudo, SUDO_USER is set. + if [ -n "${SUDO_USER:-}" ] && [ "$SUDO_USER" != "root" ]; then + USER_HOME=$(eval echo "~$SUDO_USER") + else + USER_HOME="$HOME" + fi + MCP_CONFIG="$USER_HOME/.claude/.mcp.json" + run "mkdir -p '$USER_HOME/.claude'" + + # Merge endpoint into existing config (or create new). Use python for the merge + # because shell JSON juggling is error-prone. + PY_MERGE=$(cat <.yaml (key at $KEY_FILE)" +fi +if [ "$WITH_MCP" -eq 1 ]; then + echo "MCP: merged into $MCP_CONFIG" +fi diff --git a/hosts/apps.yaml b/hosts/apps.yaml new file mode 100644 index 0000000..daef02b --- /dev/null +++ b/hosts/apps.yaml @@ -0,0 +1,50 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: apps +kind: lxc +os: linux +role: docker-apps +host: hubris +pve_id: 105 +lan_ip: 192.168.8.205 +mesh: + netbird: + fqdn: apps.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +public_hosts: +- artifacto.hubris.network +runs: +- artifacto +- plantuml +- homelab-mcp +- secrets-issuance +- artifacto +- homelab_mcp +- secrets_issuance +services_hosted: +- name: artifacto + backend: apps + url: https://artifacto.hubris.network +- name: homelab_mcp + backend: apps + port: 9810 + netbird_host: apps.netbird.selfhosted + endpoint: http://apps.netbird.selfhosted:9810/sse + note: MCP server. Read-only context + management. Mesh-only. +- name: secrets_issuance + backend: apps + port: 9820 + netbird_host: apps.netbird.selfhosted + endpoint: http://apps.netbird.selfhosted:9820/issue + note: Issues per-client age private keys. Mesh-only. +see_also: +- containers/105-apps.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/arriman.yaml b/hosts/arriman.yaml new file mode 100644 index 0000000..2c529fc --- /dev/null +++ b/hosts/arriman.yaml @@ -0,0 +1,34 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: arriman +kind: lxc +os: linux +role: arr-stack +host: hubris +pve_id: 122 +lan_ip: 192.168.8.132 +mesh: + netbird: + fqdn: arriman.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +public_hosts: +- jellyseerr.hubris.network +- qbit.hubris.network +- sab.hubris.network +runs: +- arr_stack +services_hosted: +- name: arr_stack + backend: arriman + note: jellyseerr / qbit / sab on docker compose +see_also: +- containers/122-arriman.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/authentik.yaml b/hosts/authentik.yaml new file mode 100644 index 0000000..ed2620a --- /dev/null +++ b/hosts/authentik.yaml @@ -0,0 +1,35 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: authentik +kind: lxc +os: linux +role: idp +host: hubris +pve_id: 124 +lan_ip: 192.168.8.180 +mesh: + netbird: + fqdn: authentik.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +public_host: auth.hubris.network +runs: +- authentik +- dnsmasq +services_hosted: +- name: authentik + url: https://auth.hubris.network + backend: authentik +- name: dnsmasq + backend: authentik + note: split-horizon DNS, /etc/dnsmasq.d/hubris-split.conf +notes: +- 'Also hosts split-horizon dnsmasq: /etc/dnsmasq.d/hubris-split.conf' +see_also: +- containers/124-authentik.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/caddy.yaml b/hosts/caddy.yaml new file mode 100644 index 0000000..c0231c6 --- /dev/null +++ b/hosts/caddy.yaml @@ -0,0 +1,35 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: caddy +kind: lxc +os: linux +role: reverse-proxy +host: hubris +pve_id: 121 +lan_ip: 192.168.8.175 +mesh: + netbird: + fqdn: caddy.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +peers: +- authentik +- gitea +runs: +- caddy +services_hosted: +- name: caddy + backend: caddy + role: reverse-proxy + note: terminates all *.hubris.network +notes: +- Terminates all *.hubris.network +- /etc/caddy is a git checkout of dtoro/caddy-conf +see_also: +- containers/121-caddy.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/claudio-bot.yaml b/hosts/claudio-bot.yaml new file mode 100644 index 0000000..843565a --- /dev/null +++ b/hosts/claudio-bot.yaml @@ -0,0 +1,24 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: claudio-bot +kind: lxc +os: linux +role: matrix-agent +host: hubris +pve_id: 123 +lan_ip: 192.168.8.230 +mesh: + netbird: + fqdn: claudio-bot.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +notes: +- Reads /opt/homelab-context/ on startup +see_also: +- containers/123-claudio-bot.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/elementsynapse.yaml b/hosts/elementsynapse.yaml new file mode 100644 index 0000000..2de15fa --- /dev/null +++ b/hosts/elementsynapse.yaml @@ -0,0 +1,29 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: elementsynapse +kind: lxc +os: linux +role: matrix-server +host: hubris +pve_id: 118 +lan_ip: 192.168.8.239 +mesh: + netbird: + fqdn: elementsynapse.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +public_host: matrix.hubris.network +runs: +- matrix +services_hosted: +- name: matrix + url: https://matrix.hubris.network + backend: elementsynapse +see_also: +- containers/118-elementsynapse.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/gitea.yaml b/hosts/gitea.yaml new file mode 100644 index 0000000..89fc268 --- /dev/null +++ b/hosts/gitea.yaml @@ -0,0 +1,34 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: gitea +kind: lxc +os: linux +role: git-server +host: hubris +pve_id: 104 +lan_ip: 192.168.8.121 +mesh: + netbird: + fqdn: gitea.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +public_host: git.hubris.network +runs: +- gitea +services_hosted: +- name: gitea + url: https://git.hubris.network + backend: gitea + backend_url: http://192.168.8.121:3000 +notes: +- Bare repos live at /mnt/library/repos/dtoro/*.git +see_also: +- containers/104-gitea.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/haos.yaml b/hosts/haos.yaml new file mode 100644 index 0000000..f83a4e8 --- /dev/null +++ b/hosts/haos.yaml @@ -0,0 +1,26 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: haos +kind: vm +os: linux +role: home-automation +host: hubris +pve_id: 108 +mesh: + netbird: + fqdn: haos.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +runs: +- haos +services_hosted: +- name: haos + backend: haos +see_also: +- vms/108-haos.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/hubris.yaml b/hosts/hubris.yaml new file mode 100644 index 0000000..bc507db --- /dev/null +++ b/hosts/hubris.yaml @@ -0,0 +1,33 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: hubris +kind: proxmox-host +os: linux +role: hypervisor +lan_ip: 192.168.8.77 +mesh: + netbird: + ip: 100.122.38.109 + fqdn: proxmox-server.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +ssh: + port: 22 + netbird_port: 22022 +runs: +- proxmox_ui +services_hosted: +- name: proxmox_ui + url: https://proxmox.hubris.network + backend: hubris + port: 8006 +see_also: +- hosts/hubris.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/jellyfin.yaml b/hosts/jellyfin.yaml new file mode 100644 index 0000000..795f0aa --- /dev/null +++ b/hosts/jellyfin.yaml @@ -0,0 +1,31 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: jellyfin +kind: lxc +os: linux +role: media-server +host: hubris +pve_id: 101 +lan_ip: 192.168.8.206 +mesh: + netbird: + fqdn: jellyfin.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +public_host: media.hubris.network +runs: +- jellyfin +services_hosted: +- name: jellyfin + url: https://media.hubris.network + backend: jellyfin +see_also: +- containers/101-jellyfin.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/ludo-mini.yaml b/hosts/ludo-mini.yaml new file mode 100644 index 0000000..16a2c23 --- /dev/null +++ b/hosts/ludo-mini.yaml @@ -0,0 +1,17 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: ludo-mini +kind: workstation +os: linux +role: dev +mesh: + netbird: + fqdn: ludo-mini.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/mac-mini.yaml b/hosts/mac-mini.yaml new file mode 100644 index 0000000..024b2cc --- /dev/null +++ b/hosts/mac-mini.yaml @@ -0,0 +1,19 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: mac-mini +kind: workstation +os: macos +role: dev +mesh: + netbird: + fqdn: mac-mini.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +notes: +- Only macOS in the fleet. Bootstrap uses launchd. +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/mule-images.yaml b/hosts/mule-images.yaml new file mode 100644 index 0000000..745389e --- /dev/null +++ b/hosts/mule-images.yaml @@ -0,0 +1,31 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: mule-images +kind: lxc +os: linux +role: photo-management +host: hubris +pve_id: 120 +lan_ip: 192.168.8.136 +mesh: + netbird: + fqdn: mule-images.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +public_host: photos.hubris.network +runs: +- photos +services_hosted: +- name: photos + url: https://photos.hubris.network + backend: mule-images +see_also: +- containers/120-mule-images.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/mule-photos-new.yaml b/hosts/mule-photos-new.yaml new file mode 100644 index 0000000..cdcb5e5 --- /dev/null +++ b/hosts/mule-photos-new.yaml @@ -0,0 +1,29 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: mule-photos-new +kind: lxc +os: linux +role: photo-management +host: hubris +pve_id: 127 +lan_ip: 192.168.8.181 +mesh: + netbird: + fqdn: mule-photos-new.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +public_host: photos-new.hubris.network +runs: +- photos_new +services_hosted: +- name: photos_new + url: https://photos-new.hubris.network + backend: mule-photos-new +see_also: +- containers/127-mule-photos-new.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/nextcloud.yaml b/hosts/nextcloud.yaml new file mode 100644 index 0000000..4dbdab5 --- /dev/null +++ b/hosts/nextcloud.yaml @@ -0,0 +1,31 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: nextcloud +kind: lxc +os: linux +role: file-sync +host: hubris +pve_id: 114 +lan_ip: 192.168.8.224 +mesh: + netbird: + fqdn: nextcloud.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +public_host: cloud.hubris.network +runs: +- nextcloud +services_hosted: +- name: nextcloud + url: https://cloud.hubris.network + backend: nextcloud +see_also: +- containers/114-nextcloud.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/nfs-export.yaml b/hosts/nfs-export.yaml new file mode 100644 index 0000000..d858fbe --- /dev/null +++ b/hosts/nfs-export.yaml @@ -0,0 +1,22 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: nfs-export +kind: lxc +os: linux +role: storage-export +host: hubris +pve_id: 102 +lan_ip: 192.168.8.200 +mesh: + netbird: + fqdn: nfs-export.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +see_also: +- containers/102-nfs-export.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/paperless.yaml b/hosts/paperless.yaml new file mode 100644 index 0000000..c9e201b --- /dev/null +++ b/hosts/paperless.yaml @@ -0,0 +1,31 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: paperless +kind: lxc +os: linux +role: document-archive +host: hubris +pve_id: 103 +lan_ip: 192.168.8.130 +mesh: + netbird: + fqdn: paperless.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +public_host: paperless.hubris.network +runs: +- paperless +services_hosted: +- name: paperless + url: https://paperless.hubris.network + backend: paperless +see_also: +- containers/103-paperless.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/plato.yaml b/hosts/plato.yaml new file mode 100644 index 0000000..d0c9af6 --- /dev/null +++ b/hosts/plato.yaml @@ -0,0 +1,31 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: plato +kind: lxc +os: linux +role: app +host: hubris +pve_id: 126 +lan_ip: 192.168.8.190 +mesh: + netbird: + fqdn: plato.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library/documents/plato +public_host: plato.hubris.network +runs: +- plato +services_hosted: +- name: plato + url: https://plato.hubris.network + backend: plato +see_also: +- containers/126-plato.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/republic-laptop.yaml b/hosts/republic-laptop.yaml new file mode 100644 index 0000000..3a393c6 --- /dev/null +++ b/hosts/republic-laptop.yaml @@ -0,0 +1,17 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: republic-laptop +kind: workstation +os: linux +role: primary-dev +mesh: + netbird: + fqdn: republic-laptop.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/sophia.yaml b/hosts/sophia.yaml new file mode 100644 index 0000000..f9df16b --- /dev/null +++ b/hosts/sophia.yaml @@ -0,0 +1,24 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: sophia +kind: lxc +os: linux +role: workshop +host: hubris +pve_id: 119 +lan_ip: 192.168.8.157 +mesh: + netbird: + fqdn: sophia.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +see_also: +- containers/119-sophia.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/hosts/zimaos.yaml b/hosts/zimaos.yaml new file mode 100644 index 0000000..2fde196 --- /dev/null +++ b/hosts/zimaos.yaml @@ -0,0 +1,28 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: zimaos +kind: vm +os: linux +role: nas-frontend-eval +host: hubris +pve_id: 100 +mesh: + netbird: + fqdn: zimaos.netbird.selfhosted +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +public_host: zimaos.hubris.network +runs: +- zimaos +services_hosted: +- name: zimaos + url: https://zimaos.hubris.network + backend: zimaos +see_also: +- vms/100-zimaos.md +mcp_endpoint: http://apps.netbird.selfhosted:9810/sse +secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue diff --git a/inventory.yaml b/inventory.yaml new file mode 100644 index 0000000..6a1fa6e --- /dev/null +++ b/inventory.yaml @@ -0,0 +1,365 @@ +# Homelab inventory — canonical structured topology +# +# This file is the single source of truth for hosts, services, and mesh +# addresses. Everything in hosts/*.yaml is generated from this file by +# mcp/build_host_files.py — do not edit hosts/*.yaml by hand. +# +# Conventions: +# - hostname keys here MUST match the actual `hostname` of the machine +# (or, on macOS, `scutil --get LocalHostName` if set). +# - `os:` one of: linux, macos +# - `kind:` one of: proxmox-host, lxc, vm, workstation +# - `mesh:` lists addresses the host is reachable at. Both `netbird` and +# `tailscale` are accepted during the in-flight migration +# (see infrastructure/mesh.md). Prefer netbird FQDNs over raw IPs. +# - `age_pubkey:` is provisioned by the secrets-issuance service on first +# bootstrap and committed back here. New entries leave it empty. +# - When a service moves hosts, update only the `services:` section here; +# never duplicate the address elsewhere. + +# --------------------------------------------------------------------------- +# Static config (mesh + service endpoints) +# --------------------------------------------------------------------------- + +mesh: + primary: netbird # preferred for new clients + accepted: [netbird, tailscale] + netbird_subnet: 100.122.0.0/16 + netbird_domain: netbird.selfhosted + +# Top-level service registry. Each entry resolves to a host above plus a URL. +services: + proxmox_ui: + url: https://proxmox.hubris.network + backend: hubris + port: 8006 + + gitea: + url: https://git.hubris.network + backend: gitea + backend_url: http://192.168.8.121:3000 + + caddy: + backend: caddy + role: reverse-proxy + note: terminates all *.hubris.network + + authentik: + url: https://auth.hubris.network + backend: authentik + + dnsmasq: + backend: authentik + note: split-horizon DNS, /etc/dnsmasq.d/hubris-split.conf + + jellyfin: + url: https://media.hubris.network + backend: jellyfin + + nextcloud: + url: https://cloud.hubris.network + backend: nextcloud + + paperless: + url: https://paperless.hubris.network + backend: paperless + + matrix: + url: https://matrix.hubris.network + backend: elementsynapse + + photos: + url: https://photos.hubris.network + backend: mule-images + + photos_new: + url: https://photos-new.hubris.network + backend: mule-photos-new + + plato: + url: https://plato.hubris.network + backend: plato + + arr_stack: + backend: arriman + note: jellyseerr / qbit / sab on docker compose + + artifacto: + backend: apps + url: https://artifacto.hubris.network + + zimaos: + url: https://zimaos.hubris.network + backend: zimaos + + haos: + backend: haos + + # New services introduced by this plan. + homelab_mcp: + backend: apps + port: 9810 + netbird_host: apps.netbird.selfhosted + endpoint: http://apps.netbird.selfhosted:9810/sse + note: MCP server. Read-only context + management. Mesh-only. + + secrets_issuance: + backend: apps + port: 9820 + netbird_host: apps.netbird.selfhosted + endpoint: http://apps.netbird.selfhosted:9820/issue + note: Issues per-client age private keys. Mesh-only. + +# --------------------------------------------------------------------------- +# Hosts (Proxmox host + LXCs + VMs + workstations) +# --------------------------------------------------------------------------- + +hosts: + + # ----- The hypervisor ----- + hubris: + kind: proxmox-host + os: linux + role: hypervisor + lan_ip: 192.168.8.77 + mesh: + netbird: + ip: 100.122.38.109 + fqdn: proxmox-server.netbird.selfhosted + ssh: + port: 22 # OpenSSH + netbird_port: 22022 # see infrastructure/ssh-access.md + mounts: + - /mnt/library # shared lvmthin + age_pubkey: "" # filled by issuance on first bootstrap + + # ----- LXC containers (all on hubris) ----- + jellyfin: + kind: lxc + pve_id: 101 + host: hubris + os: linux + role: media-server + lan_ip: 192.168.8.206 + public_host: media.hubris.network + mesh: {netbird: {fqdn: jellyfin.netbird.selfhosted}} + mounts: [/mnt/library] + age_pubkey: "" + + nfs-export: + kind: lxc + pve_id: 102 + host: hubris + os: linux + role: storage-export + lan_ip: 192.168.8.200 + mesh: {netbird: {fqdn: nfs-export.netbird.selfhosted}} + age_pubkey: "" + + paperless: + kind: lxc + pve_id: 103 + host: hubris + os: linux + role: document-archive + lan_ip: 192.168.8.130 + public_host: paperless.hubris.network + mesh: {netbird: {fqdn: paperless.netbird.selfhosted}} + mounts: [/mnt/library] + age_pubkey: "" + + gitea: + kind: lxc + pve_id: 104 + host: hubris + os: linux + role: git-server + lan_ip: 192.168.8.121 + public_host: git.hubris.network + backend_port: 3000 + mesh: {netbird: {fqdn: gitea.netbird.selfhosted}} + mounts: [/mnt/library] + notes: + - "Bare repos live at /mnt/library/repos/dtoro/*.git" + age_pubkey: "" + + apps: + kind: lxc + pve_id: 105 + host: hubris + os: linux + role: docker-apps + lan_ip: 192.168.8.205 + public_hosts: [artifacto.hubris.network] + mesh: {netbird: {fqdn: apps.netbird.selfhosted}} + mounts: [/mnt/library] + runs: + - artifacto + - plantuml + - homelab-mcp # introduced by this plan + - secrets-issuance # introduced by this plan + age_pubkey: "" + + nextcloud: + kind: lxc + pve_id: 114 + host: hubris + os: linux + role: file-sync + lan_ip: 192.168.8.224 + public_host: cloud.hubris.network + mesh: {netbird: {fqdn: nextcloud.netbird.selfhosted}} + mounts: [/mnt/library] + age_pubkey: "" + + elementsynapse: + kind: lxc + pve_id: 118 + host: hubris + os: linux + role: matrix-server + lan_ip: 192.168.8.239 + public_host: matrix.hubris.network + mesh: {netbird: {fqdn: elementsynapse.netbird.selfhosted}} + age_pubkey: "" + + sophia: + kind: lxc + pve_id: 119 + host: hubris + os: linux + role: workshop + lan_ip: 192.168.8.157 + mesh: {netbird: {fqdn: sophia.netbird.selfhosted}} + mounts: [/mnt/library] + age_pubkey: "" + + mule-images: + kind: lxc + pve_id: 120 + host: hubris + os: linux + role: photo-management + lan_ip: 192.168.8.136 + public_host: photos.hubris.network + mesh: {netbird: {fqdn: mule-images.netbird.selfhosted}} + mounts: [/mnt/library] + age_pubkey: "" + + caddy: + kind: lxc + pve_id: 121 + host: hubris + os: linux + role: reverse-proxy + lan_ip: 192.168.8.175 + notes: + - "Terminates all *.hubris.network" + - "/etc/caddy is a git checkout of dtoro/caddy-conf" + peers: [authentik, gitea] + mesh: {netbird: {fqdn: caddy.netbird.selfhosted}} + age_pubkey: "" + + arriman: + kind: lxc + pve_id: 122 + host: hubris + os: linux + role: arr-stack + lan_ip: 192.168.8.132 + public_hosts: [jellyseerr.hubris.network, qbit.hubris.network, sab.hubris.network] + mesh: {netbird: {fqdn: arriman.netbird.selfhosted}} + mounts: [/mnt/library] + age_pubkey: "" + + claudio-bot: + kind: lxc + pve_id: 123 + host: hubris + os: linux + role: matrix-agent + lan_ip: 192.168.8.230 + mesh: {netbird: {fqdn: claudio-bot.netbird.selfhosted}} + notes: + - "Reads /opt/homelab-context/ on startup" + age_pubkey: "" + + authentik: + kind: lxc + pve_id: 124 + host: hubris + os: linux + role: idp + lan_ip: 192.168.8.180 + public_host: auth.hubris.network + notes: + - "Also hosts split-horizon dnsmasq: /etc/dnsmasq.d/hubris-split.conf" + mesh: {netbird: {fqdn: authentik.netbird.selfhosted}} + age_pubkey: "" + + plato: + kind: lxc + pve_id: 126 + host: hubris + os: linux + role: app + lan_ip: 192.168.8.190 + public_host: plato.hubris.network + mesh: {netbird: {fqdn: plato.netbird.selfhosted}} + mounts: [/mnt/library/documents/plato] + age_pubkey: "" + + mule-photos-new: + kind: lxc + pve_id: 127 + host: hubris + os: linux + role: photo-management + lan_ip: 192.168.8.181 + public_host: photos-new.hubris.network + mesh: {netbird: {fqdn: mule-photos-new.netbird.selfhosted}} + age_pubkey: "" + + # ----- VMs (on hubris) ----- + zimaos: + kind: vm + pve_id: 100 + host: hubris + os: linux + role: nas-frontend-eval + public_host: zimaos.hubris.network + mesh: {netbird: {fqdn: zimaos.netbird.selfhosted}} + age_pubkey: "" + + haos: + kind: vm + pve_id: 108 + host: hubris + os: linux + role: home-automation + mesh: {netbird: {fqdn: haos.netbird.selfhosted}} + age_pubkey: "" + + # ----- Workstations (roaming, primary agent clients) ----- + republic-laptop: + kind: workstation + os: linux + role: primary-dev + mesh: {netbird: {fqdn: republic-laptop.netbird.selfhosted}} + age_pubkey: "" + + mac-mini: + kind: workstation + os: macos + role: dev + mesh: {netbird: {fqdn: mac-mini.netbird.selfhosted}} + notes: + - "Only macOS in the fleet. Bootstrap uses launchd." + age_pubkey: "" + + ludo-mini: + kind: workstation + os: linux + role: dev + mesh: {netbird: {fqdn: ludo-mini.netbird.selfhosted}} + age_pubkey: "" diff --git a/mcp/build_host_files.py b/mcp/build_host_files.py new file mode 100755 index 0000000..10099ff --- /dev/null +++ b/mcp/build_host_files.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 +""" +Generate hosts/.yaml from inventory.yaml. + +Run from the repo root: + python3 mcp/build_host_files.py # writes files, exits non-zero on diff + python3 mcp/build_host_files.py --check # exits non-zero if any output differs + +Designed to be wired into a pre-commit hook or Gitea Action so generated +hosts/*.yaml never drift from inventory.yaml. +""" + +from __future__ import annotations + +import argparse +import difflib +import os +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("PyYAML is required: pip install pyyaml", file=sys.stderr) + sys.exit(2) + + +REPO = Path(__file__).resolve().parent.parent +INVENTORY = REPO / "inventory.yaml" +HOSTS_DIR = REPO / "hosts" + +GENERATED_BANNER = ( + "# Generated by mcp/build_host_files.py from inventory.yaml.\n" + "# Do NOT edit by hand — your changes will be overwritten.\n" + "# Source of truth: ../inventory.yaml\n" +) + + +def narrative_page(name: str, kind: str, pve_id: int | None) -> str | None: + """Best-guess path to the human-authored narrative page for this host.""" + if kind == "proxmox-host": + candidate = REPO / "hosts" / f"{name}.md" + elif kind == "lxc": + candidate = REPO / "containers" / f"{pve_id}-{name}.md" + elif kind == "vm": + candidate = REPO / "vms" / f"{pve_id}-{name}.md" + else: + return None + if candidate.exists(): + return str(candidate.relative_to(REPO)) + return None + + +def build_one(name: str, entry: dict, inventory: dict) -> dict: + """Project the entry for a single host into a per-host yaml record.""" + services = inventory.get("services", {}) + mesh = inventory.get("mesh", {}) + pve_id = entry.get("pve_id") + + # Services this host runs: scan inventory.services for matching backend. + runs_services = sorted( + svc for svc, sentry in services.items() + if isinstance(sentry, dict) and sentry.get("backend") == name + ) + + record = { + "name": name, + "kind": entry.get("kind"), + "os": entry.get("os"), + "role": entry.get("role"), + "host": entry.get("host"), + "pve_id": pve_id, + "lan_ip": entry.get("lan_ip"), + "mesh": entry.get("mesh", {}), + "mesh_globals": { + "primary": mesh.get("primary"), + "accepted": mesh.get("accepted"), + }, + "peers": entry.get("peers", []), + "mounts": entry.get("mounts", []), + "public_host": entry.get("public_host"), + "public_hosts": entry.get("public_hosts", []), + "ssh": entry.get("ssh", {}), + "runs": entry.get("runs", []) + runs_services, + "services_hosted": [ + {"name": svc, **services[svc]} for svc in runs_services + ], + "notes": entry.get("notes", []), + "age_pubkey": entry.get("age_pubkey", ""), + "see_also": [ + page for page in [narrative_page(name, entry.get("kind", ""), pve_id)] + if page + ], + "mcp_endpoint": services.get("homelab_mcp", {}).get("endpoint"), + "secrets_issuance_endpoint": ( + services.get("secrets_issuance", {}).get("endpoint") + ), + } + # Strip None and empty containers so the file stays readable. + return {k: v for k, v in record.items() if v not in (None, {}, [], "")} + + +def serialize(record: dict) -> str: + return GENERATED_BANNER + yaml.safe_dump( + record, sort_keys=False, default_flow_style=False, width=100 + ) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--check", action="store_true", + help="exit 1 if any output would change (don't write)") + args = parser.parse_args() + + inventory = yaml.safe_load(INVENTORY.read_text()) + hosts = inventory.get("hosts", {}) + + HOSTS_DIR.mkdir(exist_ok=True) + desired: dict[Path, str] = {} + for name, entry in hosts.items(): + desired[HOSTS_DIR / f"{name}.yaml"] = serialize(build_one(name, entry, inventory)) + + diff_count = 0 + for path, content in desired.items(): + existing = path.read_text() if path.exists() else "" + if existing != content: + diff_count += 1 + if args.check: + diff = difflib.unified_diff( + existing.splitlines(keepends=True), + content.splitlines(keepends=True), + fromfile=str(path), + tofile=str(path) + " (generated)", + ) + sys.stdout.writelines(diff) + else: + path.write_text(content) + print(f"wrote {path.relative_to(REPO)}") + + # Clean up orphans (file exists but host removed from inventory). + for existing_path in HOSTS_DIR.glob("*.yaml"): + if existing_path not in desired: + diff_count += 1 + if args.check: + print(f"orphan: {existing_path.relative_to(REPO)} (would delete)") + else: + existing_path.unlink() + print(f"deleted orphan {existing_path.relative_to(REPO)}") + + if args.check and diff_count > 0: + print(f"\n{diff_count} file(s) would change. Run without --check to write.", + file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/mcp/deploy/deploy.sh b/mcp/deploy/deploy.sh new file mode 100755 index 0000000..1cea2a8 --- /dev/null +++ b/mcp/deploy/deploy.sh @@ -0,0 +1,46 @@ +#!/bin/bash +# Install/update homelab-mcp on LXC 105 (apps). Idempotent. +# Triggered by the gitea webhook or run by hand. +set -euo pipefail + +REPO_DIR=${REPO_DIR:-/opt/homelab-mcp} + +cd "$REPO_DIR" +echo "[deploy] git pull" +git pull --ff-only + +echo "[deploy] ensure python venv + deps" +if [ ! -d /opt/homelab-mcp/.venv ]; then + python3 -m venv /opt/homelab-mcp/.venv +fi +/opt/homelab-mcp/.venv/bin/pip install --quiet --upgrade pip +/opt/homelab-mcp/.venv/bin/pip install --quiet "mcp[cli]" pyyaml + +echo "[deploy] install systemd units" +install -m 644 mcp/deploy/homelab-mcp.service \ + /etc/systemd/system/homelab-mcp.service +install -m 644 mcp/deploy/webhook/homelab-mcp-deploy.service \ + /etc/systemd/system/homelab-mcp-deploy.service + +echo "[deploy] context clone for the MCP server" +# The MCP server reads from a local clone of Homelab-Docs at /opt/homelab-context +# (the same path every client uses). Bootstrap should already have created this; +# if not, fail loudly. +if [ ! -d /opt/homelab-context/.git ]; then + echo " /opt/homelab-context is not a git clone — run bootstrap.sh first." >&2 + exit 1 +fi + +systemctl daemon-reload +if systemctl is-active --quiet homelab-mcp.service; then + systemctl restart homelab-mcp.service +fi +if systemctl is-active --quiet homelab-mcp-deploy.service; then + systemctl restart homelab-mcp-deploy.service +fi + +echo "[deploy] done" +echo "First-time enable:" +echo " systemctl enable --now homelab-mcp.service homelab-mcp-deploy.service" +echo "Webhook first-time setup (generates secret):" +echo " $REPO_DIR/mcp/deploy/webhook/install.sh" diff --git a/mcp/deploy/homelab-mcp.service b/mcp/deploy/homelab-mcp.service new file mode 100644 index 0000000..ad30b6c --- /dev/null +++ b/mcp/deploy/homelab-mcp.service @@ -0,0 +1,24 @@ +[Unit] +Description=Homelab MCP server (read-only context + management) +After=network-online.target homelab-context-sync.service +Wants=network-online.target + +[Service] +Type=simple +WorkingDirectory=/opt/homelab-context +Environment=HOMELAB_CONTEXT_DIR=/opt/homelab-context +Environment=HOMELAB_MCP_SSH_KEY=/etc/homelab-mcp/mcp-reader.key +Environment=HOMELAB_MCP_SSH_USER=mcp-reader +ExecStart=/opt/homelab-mcp/.venv/bin/python /opt/homelab-mcp/mcp/server.py +Restart=on-failure +RestartSec=5 +# Stay confined. +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +NoNewPrivileges=true +ReadOnlyPaths=/opt/homelab-context /opt/homelab-mcp +ReadWritePaths=/var/log/homelab-mcp + +[Install] +WantedBy=multi-user.target diff --git a/mcp/deploy/webhook/homelab-mcp-deploy.service b/mcp/deploy/webhook/homelab-mcp-deploy.service new file mode 100644 index 0000000..a995a0a --- /dev/null +++ b/mcp/deploy/webhook/homelab-mcp-deploy.service @@ -0,0 +1,13 @@ +[Unit] +Description=Gitea deploy webhook for dtoro/Homelab-Docs → homelab-mcp +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/bin/python3 /opt/homelab-mcp/mcp/deploy/webhook/webhook.py +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target diff --git a/mcp/deploy/webhook/install.sh b/mcp/deploy/webhook/install.sh new file mode 100755 index 0000000..8d61858 --- /dev/null +++ b/mcp/deploy/webhook/install.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# First-time setup for the homelab-mcp deploy webhook. Generates a secret, +# installs the systemd unit, and starts it. Re-run is safe (won't regenerate +# the secret if one exists). +set -euo pipefail + +SECRET_DIR=/etc/homelab-mcp-deploy +SECRET=$SECRET_DIR/secret +UNIT=homelab-mcp-deploy.service + +install -d -m 700 "$SECRET_DIR" +if [ ! -s "$SECRET" ]; then + head -c 32 /dev/urandom | base64 > "$SECRET" + chmod 600 "$SECRET" + echo "[install] generated webhook secret at $SECRET" +fi + +systemctl daemon-reload +systemctl enable --now "$UNIT" +systemctl status "$UNIT" --no-pager | head -10 + +cat <:9811/deploy + HTTP Method: POST + Content-Type: application/json + Secret: $(cat $SECRET) + Trigger: Push events +EOF diff --git a/mcp/deploy/webhook/webhook.py b/mcp/deploy/webhook/webhook.py new file mode 100755 index 0000000..a22f128 --- /dev/null +++ b/mcp/deploy/webhook/webhook.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +"""Deploy webhook for dtoro/Homelab-Docs on LXC 105 (apps). + +Listens on 0.0.0.0:9811/deploy. Validates Gitea HMAC, runs deploy.sh. +Port 9811: claudio-bot-deploy=9797, backup-library-deploy=9798, + claudio-monitor=9799, homelab-mcp=9811, secrets-issuance=9821. +""" +from __future__ import annotations + +import hashlib +import hmac +import json +import logging +import os +import subprocess +import sys +import threading +from http.server import BaseHTTPRequestHandler, HTTPServer + +BIND_HOST = "0.0.0.0" +BIND_PORT = 9811 +SECRET_PATH = "/etc/homelab-mcp-deploy/secret" +DEPLOY_CMD = ["/opt/homelab-mcp/mcp/deploy/deploy.sh"] +TARGET_REF = "refs/heads/main" + +logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") +log = logging.getLogger("homelab-mcp-deploy") + +_deploy_lock = threading.Lock() + + +def load_secret() -> bytes: + with open(SECRET_PATH, "rb") as f: + return f.read().strip() + + +class Handler(BaseHTTPRequestHandler): + def log_message(self, fmt, *args): + log.info("%s - %s", self.address_string(), fmt % args) + + def _reply(self, status: int, msg: str = "") -> None: + self.send_response(status) + self.send_header("Content-Type", "text/plain") + self.end_headers() + if msg: + self.wfile.write(msg.encode()) + + def do_POST(self) -> None: + if self.path != "/deploy": + self._reply(404, "not found") + return + length = int(self.headers.get("Content-Length", "0")) + body = self.rfile.read(length) if length else b"" + sig_header = self.headers.get("X-Gitea-Signature", "") + secret = load_secret() + expected = hmac.new(secret, body, hashlib.sha256).hexdigest() + if not hmac.compare_digest(expected, sig_header): + log.warning("signature mismatch") + self._reply(403, "bad signature") + return + try: + payload = json.loads(body) + except json.JSONDecodeError: + self._reply(400, "bad json") + return + if payload.get("ref", "") != TARGET_REF: + self._reply(204) + return + if not _deploy_lock.acquire(blocking=False): + self._reply(202, "already running") + return + try: + log.info("running deploy: %s", DEPLOY_CMD) + result = subprocess.run(DEPLOY_CMD, capture_output=True, text=True, timeout=180) + if result.returncode != 0: + log.error("deploy failed: %s\n%s", result.stdout, result.stderr) + self._reply(500, "deploy failed") + return + self._reply(204) + finally: + _deploy_lock.release() + + +def main() -> int: + if not os.path.exists(SECRET_PATH): + log.error("secret file missing: %s", SECRET_PATH) + return 1 + server = HTTPServer((BIND_HOST, BIND_PORT), Handler) + log.info("listening on %s:%d", BIND_HOST, BIND_PORT) + server.serve_forever() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/mcp/server.py b/mcp/server.py new file mode 100755 index 0000000..a00fbc5 --- /dev/null +++ b/mcp/server.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +""" +Homelab MCP server. + +Reads the canonical state from /opt/homelab-context/ (a git clone of +dtoro/Homelab-Docs) and exposes structured tools to any MCP-capable agent. + +Two tool groups: + - Context (pure read of the clone, no shell-outs) + - Management (read-only ssh/pct/systemctl/curl, NO mutations) + +Mutations live in the `homelab` CLI on each client, behind operator +confirmation. The MCP server never restarts, edits, or executes arbitrary +commands. +""" + +from __future__ import annotations + +import os +import re +import shlex +import subprocess +from functools import lru_cache +from pathlib import Path +from typing import Any + +import yaml +from mcp.server.fastmcp import FastMCP + +CONTEXT_DIR = Path(os.environ.get("HOMELAB_CONTEXT_DIR", "/opt/homelab-context")) +INVENTORY = CONTEXT_DIR / "inventory.yaml" +HOSTS_DIR = CONTEXT_DIR / "hosts" +SSH_IDENTITY = os.environ.get("HOMELAB_MCP_SSH_KEY", "/etc/homelab-mcp/mcp-reader.key") +SSH_USER = os.environ.get("HOMELAB_MCP_SSH_USER", "mcp-reader") +SSH_TIMEOUT = int(os.environ.get("HOMELAB_MCP_SSH_TIMEOUT", "10")) + +mcp = FastMCP("homelab") + + +def _load_inventory() -> dict: + if not INVENTORY.exists(): + raise RuntimeError(f"inventory not found: {INVENTORY}") + return yaml.safe_load(INVENTORY.read_text()) + + +@lru_cache(maxsize=1) +def _inventory_cache_key() -> int: + """Cache buster keyed on mtime so we re-read after each sync pull.""" + return INVENTORY.stat().st_mtime_ns + + +def inventory() -> dict: + _inventory_cache_key.cache_clear() + _ = _inventory_cache_key() # warm + return _load_inventory() + + +def _host_ssh_target(name: str) -> str: + """Return the ssh target (user@host) the MCP reader uses for `name`.""" + inv = inventory() + h = inv.get("hosts", {}).get(name) + if h is None: + raise ValueError(f"unknown host: {name}") + mesh = h.get("mesh", {}) + # Netbird FQDN if available, then lan_ip, then tailscale. + target = ( + mesh.get("netbird", {}).get("fqdn") + or mesh.get("netbird", {}).get("ip") + or h.get("lan_ip") + or mesh.get("tailscale", {}).get("fqdn") + ) + if not target: + raise ValueError(f"no reachable address for host {name}") + return f"{SSH_USER}@{target}" + + +def _ssh(host: str, *cmd: str, timeout: int | None = None) -> subprocess.CompletedProcess: + """Run a command on a remote host as the restricted mcp-reader user.""" + target = _host_ssh_target(host) + full = [ + "ssh", "-i", SSH_IDENTITY, "-o", "BatchMode=yes", + "-o", "StrictHostKeyChecking=accept-new", + "-o", f"ConnectTimeout={SSH_TIMEOUT}", + target, *cmd, + ] + return subprocess.run(full, capture_output=True, text=True, + timeout=timeout or SSH_TIMEOUT * 3) + + +def _service_to_host(service: str) -> str: + """Resolve a service name to its backend host name.""" + svc = inventory().get("services", {}).get(service) + if not svc: + raise ValueError(f"unknown service: {service}") + return svc["backend"] + + +# --------------------------------------------------------------------------- +# Context tools +# --------------------------------------------------------------------------- + +@mcp.tool() +def get_host(name: str) -> dict: + """Return the structured record for a host (LXC, VM, workstation, or hubris).""" + inv = inventory() + entry = inv.get("hosts", {}).get(name) + if entry is None: + raise ValueError(f"unknown host: {name}") + out = {"name": name, **entry} + yaml_path = HOSTS_DIR / f"{name}.yaml" + if yaml_path.exists(): + out["host_yaml_path"] = str(yaml_path) + return out + + +@mcp.tool() +def list_services() -> dict: + """List every service registered in inventory.yaml.""" + return inventory().get("services", {}) + + +@mcp.tool() +def find_service(query: str) -> list[dict]: + """Find services by name substring, role, or backend host name.""" + q = query.lower() + inv = inventory() + out = [] + for name, entry in inv.get("services", {}).items(): + haystack = " ".join([ + name, + str(entry.get("backend", "")), + str(entry.get("role", "")), + str(entry.get("note", "")), + ]).lower() + if q in haystack: + out.append({"service": name, **entry}) + return out + + +@mcp.tool() +def get_topology() -> dict: + """Return the full inventory (hosts + services + mesh).""" + return inventory() + + +@mcp.tool() +def search_docs(query: str, max_results: int = 10) -> list[dict]: + """Ripgrep the markdown wiki for query, return file:line hits.""" + rg = subprocess.run( + ["rg", "--no-heading", "-n", "-i", "--type", "md", "-m", "5", query, str(CONTEXT_DIR)], + capture_output=True, text=True, timeout=10, + ) + results = [] + for line in rg.stdout.splitlines()[: max_results * 5]: + m = re.match(r"^(.+?):(\d+):(.*)$", line) + if m: + results.append({ + "path": str(Path(m.group(1)).relative_to(CONTEXT_DIR)), + "line": int(m.group(2)), + "text": m.group(3).strip(), + }) + if len(results) >= max_results: + break + return results + + +@mcp.tool() +def get_page(path: str) -> str: + """Return a markdown page verbatim. Path is relative to the repo root.""" + full = (CONTEXT_DIR / path).resolve() + if not str(full).startswith(str(CONTEXT_DIR.resolve())): + raise ValueError("path escapes repo") + if not full.exists(): + raise ValueError(f"no such page: {path}") + return full.read_text() + + +@mcp.tool() +def get_changelog(path: str, since: str | None = None) -> str: + """Extract the trailing ## Changelog section from a page; optionally filter.""" + text = get_page(path) + m = re.search(r"^##\s+Changelog\s*\n(.*)", text, re.MULTILINE | re.DOTALL) + if not m: + return "" + body = m.group(1).strip() + if since: + kept = [] + for entry in re.split(r"^###\s+", body, flags=re.MULTILINE): + if not entry.strip(): + continue + head = entry.split("\n", 1)[0] + date_m = re.match(r"(\d{4}-\d{2}-\d{2})", head) + if not date_m or date_m.group(1) >= since: + kept.append("### " + entry.rstrip()) + return "\n\n".join(kept).strip() + return body + + +@mcp.tool() +def whoami(hostname: str) -> dict: + """Given a hostname, return that host's full yaml record from hosts/.yaml.""" + candidate = HOSTS_DIR / f"{hostname}.yaml" + if not candidate.exists(): + raise ValueError(f"no host record for {hostname}") + return yaml.safe_load(candidate.read_text()) + + +@mcp.tool() +def list_my_secrets(caller_pubkey: str) -> list[str]: + """Return the names of secrets the caller (identified by age pubkey) can decrypt. + + Metadata only — the server never returns plaintext. The client decrypts + locally with its own /etc/age/key.txt. + """ + if not caller_pubkey: + return [] + secrets_dir = CONTEXT_DIR / "secrets" + if not secrets_dir.exists(): + return [] + out = [] + for path in secrets_dir.glob("*.yaml"): + try: + data = yaml.safe_load(path.read_text()) or {} + except yaml.YAMLError: + continue + recipients = ( + data.get("sops", {}) + .get("age", []) + ) + for r in recipients: + if r.get("recipient") == caller_pubkey: + out.append(path.stem) + break + return sorted(out) + + +# --------------------------------------------------------------------------- +# Management tools (read-only) +# --------------------------------------------------------------------------- + +@mcp.tool() +def get_service_status(service: str) -> dict: + """systemctl is-active + is-enabled for the named service on its backend host.""" + host = _service_to_host(service) + inv_svc = inventory()["services"][service] + unit = inv_svc.get("systemd_unit", service) + active = _ssh(host, "systemctl", "is-active", unit) + enabled = _ssh(host, "systemctl", "is-enabled", unit) + return { + "service": service, + "host": host, + "unit": unit, + "active": active.stdout.strip(), + "enabled": enabled.stdout.strip(), + } + + +@mcp.tool() +def tail_log(service: str, lines: int = 200) -> str: + """Last N journalctl lines for the named service on its backend host.""" + host = _service_to_host(service) + inv_svc = inventory()["services"][service] + unit = inv_svc.get("systemd_unit", service) + proc = _ssh(host, "journalctl", "-u", unit, "-n", str(lines), "--no-pager") + return proc.stdout + + +@mcp.tool() +def list_lxcs() -> str: + """Run `pct list` on hubris.""" + return _ssh("hubris", "pct", "list").stdout + + +@mcp.tool() +def get_lxc_state(lxc: str) -> dict: + """`pct status` and a quick `pct config` snapshot for the named LXC.""" + inv = inventory().get("hosts", {}) + if lxc not in inv: + raise ValueError(f"unknown lxc: {lxc}") + pve_id = inv[lxc].get("pve_id") + if pve_id is None: + raise ValueError(f"{lxc} has no pve_id (is it actually an LXC?)") + status = _ssh("hubris", "pct", "status", str(pve_id)) + cfg = _ssh("hubris", "pct", "config", str(pve_id)) + return {"lxc": lxc, "pve_id": pve_id, + "status": status.stdout.strip(), + "config": cfg.stdout} + + +@mcp.tool() +def ping_service(service: str) -> dict: + """HTTP check against the service's URL (from inventory).""" + inv_svc = inventory().get("services", {}).get(service) + if not inv_svc: + raise ValueError(f"unknown service: {service}") + url = inv_svc.get("url") or inv_svc.get("endpoint") or inv_svc.get("backend_url") + if not url: + return {"service": service, "ok": False, "reason": "no URL in inventory"} + proc = subprocess.run( + ["curl", "-sS", "-o", "/dev/null", "-w", "%{http_code} %{time_total}", + "--max-time", "5", url], + capture_output=True, text=True, + ) + code, t = (proc.stdout.strip().split() + ["", ""])[:2] + return {"service": service, "url": url, "http_code": code, "time_s": t, + "ok": code.startswith("2") or code.startswith("3")} + + +if __name__ == "__main__": + mcp.run(transport="sse") diff --git a/scripts/sync/install.sh b/scripts/sync/install.sh new file mode 100755 index 0000000..de610cf --- /dev/null +++ b/scripts/sync/install.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Install the homelab-context sync mechanism for the current OS. +# Called by bootstrap.sh; safe to re-run (idempotent). + +set -euo pipefail + +REPO_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab-context}" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +case "$(uname -s)" in + Linux) + install -m 0644 "$SCRIPT_DIR/linux/homelab-context-sync.service" \ + /etc/systemd/system/homelab-context-sync.service + install -m 0644 "$SCRIPT_DIR/linux/homelab-context-sync.timer" \ + /etc/systemd/system/homelab-context-sync.timer + systemctl daemon-reload + systemctl enable --now homelab-context-sync.timer + # Kick once so the first pull happens immediately. + systemctl start homelab-context-sync.service || true + echo "[install.sh] linux: timer installed and started" + ;; + Darwin) + PLIST=/Library/LaunchDaemons/network.hubris.homelab-context-sync.plist + install -m 0644 -o root -g wheel \ + "$SCRIPT_DIR/macos/network.hubris.homelab-context-sync.plist" "$PLIST" + # Touch the log path so the daemon can write without a TOCTOU. + touch /var/log/homelab-context-sync.log + chmod 0644 /var/log/homelab-context-sync.log + # bootstrap may have loaded a previous version; reload. + launchctl bootout system "$PLIST" 2>/dev/null || true + launchctl bootstrap system "$PLIST" + launchctl enable system/network.hubris.homelab-context-sync + # Kick once. + launchctl kickstart -k system/network.hubris.homelab-context-sync || true + echo "[install.sh] macos: launchd job loaded and kicked" + ;; + *) + echo "[install.sh] unsupported OS: $(uname -s)" >&2 + exit 1 + ;; +esac diff --git a/scripts/sync/linux/homelab-context-sync.service b/scripts/sync/linux/homelab-context-sync.service new file mode 100644 index 0000000..adc650b --- /dev/null +++ b/scripts/sync/linux/homelab-context-sync.service @@ -0,0 +1,15 @@ +[Unit] +Description=Pull /opt/homelab-context from Gitea +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/bin/git -C /opt/homelab-context pull --ff-only --quiet +TimeoutStartSec=60 +# Don't fail aggressively — a missed pull just retries next tick. +SuccessExitStatus=0 1 +Nice=10 + +[Install] +WantedBy=multi-user.target diff --git a/scripts/sync/linux/homelab-context-sync.timer b/scripts/sync/linux/homelab-context-sync.timer new file mode 100644 index 0000000..42c90f1 --- /dev/null +++ b/scripts/sync/linux/homelab-context-sync.timer @@ -0,0 +1,11 @@ +[Unit] +Description=Periodic pull of /opt/homelab-context + +[Timer] +OnBootSec=2min +OnUnitActiveSec=5min +AccuracySec=30s +Unit=homelab-context-sync.service + +[Install] +WantedBy=timers.target diff --git a/scripts/sync/macos/network.hubris.homelab-context-sync.plist b/scripts/sync/macos/network.hubris.homelab-context-sync.plist new file mode 100644 index 0000000..814dc6e --- /dev/null +++ b/scripts/sync/macos/network.hubris.homelab-context-sync.plist @@ -0,0 +1,32 @@ + + + + + Label + network.hubris.homelab-context-sync + + ProgramArguments + + /usr/bin/git + -C + /opt/homelab-context + pull + --ff-only + --quiet + + + StartInterval + 300 + + RunAtLoad + + + StandardOutPath + /var/log/homelab-context-sync.log + StandardErrorPath + /var/log/homelab-context-sync.log + + Nice + 10 + + diff --git a/secrets-issuance/deploy/deploy.sh b/secrets-issuance/deploy/deploy.sh new file mode 100755 index 0000000..65a40c7 --- /dev/null +++ b/secrets-issuance/deploy/deploy.sh @@ -0,0 +1,57 @@ +#!/bin/bash +# Install/update secrets-issuance on LXC 105 (apps). Idempotent. +set -euo pipefail + +REPO_DIR=${REPO_DIR:-/opt/secrets-issuance} + +cd "$REPO_DIR" +echo "[deploy] git pull" +git pull --ff-only + +echo "[deploy] ensure python venv + deps" +if [ ! -d /opt/secrets-issuance/.venv ]; then + python3 -m venv /opt/secrets-issuance/.venv +fi +/opt/secrets-issuance/.venv/bin/pip install --quiet --upgrade pip +/opt/secrets-issuance/.venv/bin/pip install --quiet pyyaml + +echo "[deploy] verify dependencies (age, shred)" +command -v age-keygen >/dev/null || { echo "age-keygen not installed: apt install -y age" >&2; exit 1; } +command -v shred >/dev/null || { echo "shred not installed: apt install -y coreutils" >&2; exit 1; } + +echo "[deploy] state dir" +install -d -m 700 /var/lib/secrets-issuance +install -d -m 700 /var/lib/secrets-issuance/keys + +echo "[deploy] admin token dir" +install -d -m 700 /etc/secrets-issuance +if [ ! -s /etc/secrets-issuance/admin-token ]; then + head -c 32 /dev/urandom | base64 > /etc/secrets-issuance/admin-token + chmod 600 /etc/secrets-issuance/admin-token + echo "[deploy] generated admin token at /etc/secrets-issuance/admin-token" +fi + +echo "[deploy] install systemd units" +install -m 644 secrets-issuance/server.service \ + /etc/systemd/system/secrets-issuance.service +install -m 644 secrets-issuance/deploy/webhook/secrets-issuance-deploy.service \ + /etc/systemd/system/secrets-issuance-deploy.service + +if [ ! -d /opt/homelab-context/.git ]; then + echo " /opt/homelab-context is not a git clone — run bootstrap.sh first." >&2 + exit 1 +fi + +systemctl daemon-reload +if systemctl is-active --quiet secrets-issuance.service; then + systemctl restart secrets-issuance.service +fi +if systemctl is-active --quiet secrets-issuance-deploy.service; then + systemctl restart secrets-issuance-deploy.service +fi + +echo "[deploy] done" +echo "First-time enable:" +echo " systemctl enable --now secrets-issuance.service secrets-issuance-deploy.service" +echo "Webhook first-time setup:" +echo " $REPO_DIR/secrets-issuance/deploy/webhook/install.sh" diff --git a/secrets-issuance/deploy/webhook/install.sh b/secrets-issuance/deploy/webhook/install.sh new file mode 100755 index 0000000..beb99c5 --- /dev/null +++ b/secrets-issuance/deploy/webhook/install.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# First-time setup for the secrets-issuance deploy webhook. Generates a +# secret, installs the systemd unit, and starts it. +set -euo pipefail + +SECRET_DIR=/etc/secrets-issuance-deploy +SECRET=$SECRET_DIR/secret +UNIT=secrets-issuance-deploy.service + +install -d -m 700 "$SECRET_DIR" +if [ ! -s "$SECRET" ]; then + head -c 32 /dev/urandom | base64 > "$SECRET" + chmod 600 "$SECRET" + echo "[install] generated webhook secret at $SECRET" +fi + +systemctl daemon-reload +systemctl enable --now "$UNIT" +systemctl status "$UNIT" --no-pager | head -10 + +cat <:9821/deploy + HTTP Method: POST + Content-Type: application/json + Secret: $(cat $SECRET) + Trigger: Push events +EOF diff --git a/secrets-issuance/deploy/webhook/secrets-issuance-deploy.service b/secrets-issuance/deploy/webhook/secrets-issuance-deploy.service new file mode 100644 index 0000000..9097042 --- /dev/null +++ b/secrets-issuance/deploy/webhook/secrets-issuance-deploy.service @@ -0,0 +1,13 @@ +[Unit] +Description=Gitea deploy webhook for dtoro/Homelab-Docs → secrets-issuance +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/bin/python3 /opt/secrets-issuance/secrets-issuance/deploy/webhook/webhook.py +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target diff --git a/secrets-issuance/deploy/webhook/webhook.py b/secrets-issuance/deploy/webhook/webhook.py new file mode 100755 index 0000000..1decd7b --- /dev/null +++ b/secrets-issuance/deploy/webhook/webhook.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +"""Deploy webhook for dtoro/Homelab-Docs → secrets-issuance on LXC 105. + +Listens on 0.0.0.0:9821/deploy. Validates Gitea HMAC, runs deploy.sh. +""" +from __future__ import annotations + +import hashlib +import hmac +import json +import logging +import os +import subprocess +import sys +import threading +from http.server import BaseHTTPRequestHandler, HTTPServer + +BIND_HOST = "0.0.0.0" +BIND_PORT = 9821 +SECRET_PATH = "/etc/secrets-issuance-deploy/secret" +DEPLOY_CMD = ["/opt/secrets-issuance/secrets-issuance/deploy/deploy.sh"] +TARGET_REF = "refs/heads/main" + +logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") +log = logging.getLogger("secrets-issuance-deploy") +_deploy_lock = threading.Lock() + + +def load_secret() -> bytes: + with open(SECRET_PATH, "rb") as f: + return f.read().strip() + + +class Handler(BaseHTTPRequestHandler): + def log_message(self, fmt, *args): + log.info("%s - %s", self.address_string(), fmt % args) + + def _reply(self, status: int, msg: str = "") -> None: + self.send_response(status) + self.send_header("Content-Type", "text/plain") + self.end_headers() + if msg: + self.wfile.write(msg.encode()) + + def do_POST(self) -> None: + if self.path != "/deploy": + self._reply(404, "not found") + return + length = int(self.headers.get("Content-Length", "0")) + body = self.rfile.read(length) if length else b"" + sig = self.headers.get("X-Gitea-Signature", "") + secret = load_secret() + expected = hmac.new(secret, body, hashlib.sha256).hexdigest() + if not hmac.compare_digest(expected, sig): + log.warning("signature mismatch") + self._reply(403, "bad signature") + return + try: + payload = json.loads(body) + except json.JSONDecodeError: + self._reply(400, "bad json") + return + if payload.get("ref", "") != TARGET_REF: + self._reply(204) + return + if not _deploy_lock.acquire(blocking=False): + self._reply(202, "already running") + return + try: + result = subprocess.run(DEPLOY_CMD, capture_output=True, text=True, timeout=180) + if result.returncode != 0: + log.error("deploy failed: %s\n%s", result.stdout, result.stderr) + self._reply(500, "deploy failed") + return + self._reply(204) + finally: + _deploy_lock.release() + + +def main() -> int: + if not os.path.exists(SECRET_PATH): + log.error("secret file missing: %s", SECRET_PATH) + return 1 + server = HTTPServer((BIND_HOST, BIND_PORT), Handler) + log.info("listening on %s:%d", BIND_HOST, BIND_PORT) + server.serve_forever() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/secrets-issuance/server.py b/secrets-issuance/server.py new file mode 100755 index 0000000..2ccdd31 --- /dev/null +++ b/secrets-issuance/server.py @@ -0,0 +1,266 @@ +#!/usr/bin/env python3 +""" +Secrets-issuance HTTP service. + +Lives on LXC 105 alongside the MCP server. Mesh-bound (nftables restricts to +Netbird + Tailscale subnets). Identifies callers by source mesh IP and returns +the per-client age private key. + +State: + /var/lib/secrets-issuance/keys/.key (private, 0600 root) + /var/lib/secrets-issuance/keys/.pub (public) + /var/lib/secrets-issuance/denylist.txt (one hostname per line) + +Inventory lookup: + /opt/homelab-context/inventory.yaml — the issuance server is itself a + homelab-context client, so it sees inventory updates within 5 min. + +Endpoints: + POST /issue body: {"hostname": "..."} -> raw age private key + POST /revoke body: {"hostname": "..."} -> shred key + add to denylist + (called by 'homelab client remove') + GET /health +""" + +from __future__ import annotations + +import ipaddress +import json +import logging +import os +import secrets as sysrandom +import subprocess +import sys +from http.server import BaseHTTPRequestHandler, HTTPServer +from pathlib import Path + +import yaml + +BIND_HOST = "0.0.0.0" +BIND_PORT = 9820 +CONTEXT_DIR = Path(os.environ.get("HOMELAB_CONTEXT_DIR", "/opt/homelab-context")) +STATE_DIR = Path(os.environ.get("SECRETS_ISSUANCE_STATE", "/var/lib/secrets-issuance")) +KEYS_DIR = STATE_DIR / "keys" +DENYLIST = STATE_DIR / "denylist.txt" +ADMIN_TOKEN_PATH = Path(os.environ.get("SECRETS_ISSUANCE_ADMIN_TOKEN", + "/etc/secrets-issuance/admin-token")) +MESH_SUBNETS = [ + ipaddress.ip_network(s.strip()) + for s in os.environ.get("MESH_SUBNETS", "100.64.0.0/10").split(",") + if s.strip() +] + +logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") +log = logging.getLogger("secrets-issuance") + + +def load_inventory() -> dict: + return yaml.safe_load((CONTEXT_DIR / "inventory.yaml").read_text()) + + +def known_hosts() -> dict[str, dict]: + return load_inventory().get("hosts", {}) + + +def host_by_mesh_ip(ip: str) -> str | None: + """Resolve a mesh source IP to a hostname per inventory.""" + for name, entry in known_hosts().items(): + mesh = entry.get("mesh", {}) + for proto in ("netbird", "tailscale"): + p = mesh.get(proto, {}) + if not isinstance(p, dict): + continue + if p.get("ip") == ip: + return name + return None + + +def in_mesh(ip: str) -> bool: + addr = ipaddress.ip_address(ip) + return any(addr in subnet for subnet in MESH_SUBNETS) + + +def is_denied(hostname: str) -> bool: + if not DENYLIST.exists(): + return False + return hostname in {line.strip() for line in DENYLIST.read_text().splitlines() if line.strip()} + + +def deny(hostname: str) -> None: + DENYLIST.parent.mkdir(parents=True, exist_ok=True) + existing = set() + if DENYLIST.exists(): + existing = {line.strip() for line in DENYLIST.read_text().splitlines() if line.strip()} + existing.add(hostname) + DENYLIST.write_text("\n".join(sorted(existing)) + "\n") + + +def generate_key(hostname: str) -> tuple[Path, str]: + """Generate an age keypair for hostname; return (priv_path, pubkey).""" + KEYS_DIR.mkdir(parents=True, exist_ok=True) + priv = KEYS_DIR / f"{hostname}.key" + pub_path = KEYS_DIR / f"{hostname}.pub" + proc = subprocess.run(["age-keygen"], capture_output=True, text=True, check=True) + priv.write_text(proc.stdout) + priv.chmod(0o600) + # Extract the public key. age-keygen writes "# public key: ageXXX" line. + pubkey = "" + for line in proc.stdout.splitlines(): + if line.startswith("# public key:"): + pubkey = line.split(":", 1)[1].strip() + if not pubkey: + priv.unlink(missing_ok=True) + raise RuntimeError("age-keygen did not emit a public key") + pub_path.write_text(pubkey + "\n") + log.info("generated new age key for %s (pubkey: %s)", hostname, pubkey) + return priv, pubkey + + +def existing_pubkey(hostname: str) -> str | None: + pub_path = KEYS_DIR / f"{hostname}.pub" + if pub_path.exists(): + return pub_path.read_text().strip() + return None + + +def read_admin_token() -> str | None: + if ADMIN_TOKEN_PATH.exists(): + return ADMIN_TOKEN_PATH.read_text().strip() + return None + + +class Handler(BaseHTTPRequestHandler): + def log_message(self, fmt, *args): + log.info("%s - %s", self.address_string(), fmt % args) + + def _reply(self, status: int, body: bytes | str = b"", + content_type: str = "text/plain") -> None: + self.send_response(status) + self.send_header("Content-Type", content_type) + if isinstance(body, str): + body = body.encode() + self.send_header("Content-Length", str(len(body))) + self.end_headers() + if body: + self.wfile.write(body) + + def _read_json(self) -> dict | None: + length = int(self.headers.get("Content-Length", "0")) + if not length: + return {} + body = self.rfile.read(length) + try: + return json.loads(body) + except json.JSONDecodeError: + return None + + def do_GET(self) -> None: + if self.path == "/health": + self._reply(200, "ok") + return + self._reply(404, "not found") + + def do_POST(self) -> None: + client_ip = self.client_address[0] + if not in_mesh(client_ip): + log.warning("rejecting non-mesh source %s", client_ip) + self._reply(403, "non-mesh source") + return + + if self.path == "/issue": + self._handle_issue(client_ip) + elif self.path == "/revoke": + self._handle_revoke() + else: + self._reply(404, "not found") + + def _handle_issue(self, client_ip: str) -> None: + payload = self._read_json() + if payload is None: + self._reply(400, "bad json") + return + hostname = (payload.get("hostname") or "").strip() + if not hostname: + self._reply(400, "hostname required") + return + + # Cross-check: does inventory list this hostname, and does its mesh IP + # match the caller (if known)? + hosts = known_hosts() + if hostname not in hosts: + log.warning("unknown hostname in /issue: %s (from %s)", hostname, client_ip) + self._reply(403, f"unknown hostname: {hostname} (run 'homelab client add {hostname}' first)") + return + + if is_denied(hostname): + log.warning("denied hostname %s tried to issue (from %s)", hostname, client_ip) + self._reply(403, "hostname on denylist (was removed); operator must clear before re-enrolling") + return + + inv_ip = host_by_mesh_ip(client_ip) + if inv_ip is not None and inv_ip != hostname: + log.warning("source %s maps to inventory host %s but body claims %s", + client_ip, inv_ip, hostname) + self._reply(403, "source IP / hostname mismatch") + return + if inv_ip is None: + # Inventory has no IP for this hostname yet (first-bootstrap state). + # Accept the call but log it loudly so the operator backfills. + log.info("issuing for %s (source %s) — inventory has no mesh IP yet", + hostname, client_ip) + + priv = KEYS_DIR / f"{hostname}.key" + existing_pub = existing_pubkey(hostname) + if priv.exists() and existing_pub: + # Re-issue: client lost its key but is still authorized. Return + # the existing key. + log.info("returning existing key for %s", hostname) + self._reply(200, priv.read_text()) + return + + # Fresh provisioning. + priv, pubkey = generate_key(hostname) + # Tell the operator (via response body comment) which pubkey to commit. + body = priv.read_text() + body += f"\n# operator: add this pubkey to inventory.yaml under hosts.{hostname}.age_pubkey:\n" + body += f"# {pubkey}\n" + self._reply(200, body) + + def _handle_revoke(self) -> None: + token = self.headers.get("X-Admin-Token", "") + expected = read_admin_token() + if not expected or token != expected: + self._reply(403, "admin token required") + return + payload = self._read_json() + if payload is None: + self._reply(400, "bad json") + return + hostname = (payload.get("hostname") or "").strip() + if not hostname: + self._reply(400, "hostname required") + return + priv = KEYS_DIR / f"{hostname}.key" + pub = KEYS_DIR / f"{hostname}.pub" + for p in (priv, pub): + if p.exists(): + # shred-then-unlink + subprocess.run(["shred", "-u", str(p)], check=False) + if p.exists(): + p.unlink(missing_ok=True) + deny(hostname) + log.warning("revoked %s (shredded keys, added to denylist)", hostname) + self._reply(200, f"revoked {hostname}\n") + + +def main() -> int: + KEYS_DIR.mkdir(parents=True, exist_ok=True) + server = HTTPServer((BIND_HOST, BIND_PORT), Handler) + log.info("listening on %s:%d (context=%s, state=%s)", + BIND_HOST, BIND_PORT, CONTEXT_DIR, STATE_DIR) + server.serve_forever() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/secrets-issuance/server.service b/secrets-issuance/server.service new file mode 100644 index 0000000..24b9134 --- /dev/null +++ b/secrets-issuance/server.service @@ -0,0 +1,23 @@ +[Unit] +Description=Homelab secrets-issuance (per-client age key provisioning) +After=network-online.target homelab-context-sync.service +Wants=network-online.target + +[Service] +Type=simple +WorkingDirectory=/opt/secrets-issuance +Environment=HOMELAB_CONTEXT_DIR=/opt/homelab-context +Environment=SECRETS_ISSUANCE_STATE=/var/lib/secrets-issuance +Environment=MESH_SUBNETS=100.122.0.0/16,100.64.0.0/10 +ExecStart=/opt/secrets-issuance/.venv/bin/python /opt/secrets-issuance/secrets-issuance/server.py +Restart=on-failure +RestartSec=5 +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +NoNewPrivileges=true +ReadOnlyPaths=/opt/homelab-context /opt/secrets-issuance +ReadWritePaths=/var/lib/secrets-issuance + +[Install] +WantedBy=multi-user.target diff --git a/secrets/README.md b/secrets/README.md new file mode 100644 index 0000000..af534a6 --- /dev/null +++ b/secrets/README.md @@ -0,0 +1,63 @@ +# secrets/ + +SOPS-encrypted YAML files. The plaintext lives only in transit and in the +operator's head — committed files are always ciphertext. + +## Conventions + +- One file per logical grouping (e.g. `gitea-tokens.yaml`, `webhook-hmacs.yaml`, + `api-keys.yaml`). +- Recipients are declared in `../.sops.yaml` by path-regex, not per-file. +- The plaintext schema inside each file is free-form YAML; the consumer code + decides what it expects (e.g. `gitea-tokens.yaml` contains + `{"": "ghp_xxx"}`). + +## How to add a secret + +```bash +# 1. Decide which clients should be able to decrypt it; edit ../.sops.yaml to +# list their age public keys for the new path_regex. +# 2. Create the plaintext, encrypt in place: +sops -e --in-place secrets/my-thing.yaml +# 3. Commit + push. The 5-min sync propagates to every recipient. +``` + +## How to consume a secret + +```bash +# On any client that's a recipient: +homelab secret my-thing # prints plaintext +# Or programmatically: +sops -d /opt/homelab-context/secrets/my-thing.yaml +``` + +The `mcp` tool `list_my_secrets(caller_pubkey)` returns the names of secrets +the caller can decrypt. The MCP server never reads plaintext — decryption +stays client-side. + +## Granting / revoking access + +To grant a new recipient: edit `../.sops.yaml` to add their age pubkey, then +re-key every affected file: + +```bash +sops updatekeys -y secrets/my-thing.yaml +``` + +To revoke: remove the recipient from `../.sops.yaml` and `sops updatekeys` — +but remember this only protects future ciphertext. Past plaintext the client +already decrypted is gone from your control. Rotate the underlying credential +if compromise is suspected. + +`homelab client remove ` does the recipient removal + `updatekeys` for +you, and prints the rotation checklist as a follow-up. + +## hello.yaml — bootstrap decrypt test + +`secrets/hello.yaml` is encrypted to every enrolled client. Used by Phase 3a +verification to confirm the end-to-end decrypt path works on a freshly- +bootstrapped machine. Content is intentionally trivial: + +```yaml +greeting: hello from the homelab +```