From c299409431df315c573237f749020616ebda12d9 Mon Sep 17 00:00:00 2001 From: dtoro Date: Tue, 2 Jun 2026 00:20:28 +0200 Subject: [PATCH] docs: update ssh-access.md + agent-enrollment.md with universal SSH setup --- infrastructure/ssh-access.md | 199 +++++++++++++++++++++++++++++---- operations/agent-enrollment.md | 77 +++++++++++++ 2 files changed, 254 insertions(+), 22 deletions(-) diff --git a/infrastructure/ssh-access.md b/infrastructure/ssh-access.md index 0cb907b..5a89f04 100644 --- a/infrastructure/ssh-access.md +++ b/infrastructure/ssh-access.md @@ -1,45 +1,200 @@ # SSH access -How to reach hubris and the VPS over SSH, and the dual-server gotcha. +How to reach every host in the fleet from any workstation, with LAN as +the primary path and Netbird as the automatic backup. -## Hubris +## Architecture -Two SSH endpoints — easy to hit the wrong one. +SSH access relies on three layers: -| Server | Listen | Auth | Notes | -| -------------- | ---------------------------- | --------------------------------- | ----- | -| OpenSSH | `0.0.0.0:22` | `authorized_keys` at `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced; symlinked from `/root/.ssh/authorized_keys`) | Standard. | -| Netbird SSH | `100.122.38.109:22022` | OIDC / browser auth — bypasses `authorized_keys` | If a client lands here it'll open a browser tab to authenticate, then sometimes hang. Force port 22 or use the LAN IP. | +1. **Homelab inventory (`inventory.yaml`)** — the single source of truth + for every host's LAN IP, Netbird addresses, SSH user, and port. +2. **Key distribution (`ssh/deploy-keys.sh`)** — deploys workstation SSH + public keys to hubris and every running LXC, so any key-authorized + workstation can log in anywhere. +3. **Config generation (`homelab ssh-config --install`)** — generates + `~/.ssh/config.d/homelab` with short hostname aliases for every host, + using LAN IPs (routed via Netbird's `192.168.8.0/24` subnet route when + off-LAN) with Netbird FQDN fallbacks (`-mesh`) for roaming + workstations. -### Authorized root keys -- `root@hubris` (self, RSA) — original. -- `d.toro.v@pm.me` (ed25519) — user's iMac (`mac-mini.netbird.selfhosted`, LAN `192.168.8.174`), added 2026-04-22. +### How it works -### Notes -- Password auth is enabled on hubris but the root password is **not** the one the user expects. Prefer key flows; don't try `ssh-copy-id` blind. -- Off-LAN access from the iMac uses the LAN path. As of 2026-04-22 the iMac's Netbird tunnel to hubris was P2P healthy but no packets were captured on `wt0`; needs revisit if remote access becomes critical. +- **From on-LAN:** `ssh gitea` resolves to `192.168.8.121` directly. +- **From off-LAN (Netbird):** The same `192.168.8.121` works because + hubris routes the `192.168.8.0/24` subnet through Netbird. +- **Roaming workstations:** `ssh mac-mini-mesh` or `ssh republic-laptop-mesh` + uses the Netbird FQDN as a fallback when the workstation is off its + home subnet. -## VPS (`82.165.190.79` / `100.122.165.149`) +The `homelab ssh ` CLI command also has built-in LAN probing: +it tries a 1.5s TCP connect to the LAN IP, and if that fails, falls +back to the Netbird FQDN. -- **Mesh-only.** Public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`. -- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.`. -- `ListenAddress` is still 0.0.0.0; gating is firewall-layer. -- Authorized root keys: PVE (`root@hubris`), iMac (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer before disabling its access paths. +## Key distribution -See [VPS hardening](vps-hardening.md) for the firewall + fail2ban rules and recovery paths. +Each workstation's SSH public key lives in the repo at: +`ssh/authorized_keys/.pub` + +To deploy or re-deploy all workstation keys to hubris + every running LXC: + +```bash +# From hubris (or via homelab pct): +sudo bash /opt/homelab-context/ssh/deploy-keys.sh + +# Or from any workstation: +ssh root@192.168.8.77 "bash /opt/homelab-context/ssh/deploy-keys.sh" +``` + +This script: +- Reads all `.pub` files from `ssh/authorized_keys/` +- Adds any missing keys to `/etc/pve/priv/authorized_keys` on hubris +- For each running LXC, appends keys to `/root/.ssh/authorized_keys` +- Is idempotent — skips keys already present + +## Config generation + +To generate the SSH config on any workstation: + +```bash +homelab ssh-config --install +``` + +This writes to `~/.ssh/config.d/homelab` and ensures +`Include ~/.ssh/config.d/homelab` is present in `~/.ssh/config`. + +The config is regenerated automatically on every `homelab sync` (which +kicks the 5-minute context sync timer). + +## Adding a new workstation + +When onboarding a new machine: + +1. Hostname must match an entry in `inventory.yaml`. +2. If the workstation will be on the LAN, add its `lan_ip` to + `inventory.yaml` and push. This gives it a primary LAN entry in the + generated SSH config. +3. Enable SSH Remote Login: + - **macOS:** `sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist` + - **Linux:** `sudo systemctl enable --now sshd` +4. Generate an SSH keypair if one doesn't exist: + ```bash + ssh-keygen -t ed25519 -a 100 + ``` +5. Publish the public key to the repo: + ```bash + cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/.pub + cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add pubkey' && git push + ``` +6. Deploy the key to all hosts: + ```bash + ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh" + ``` +7. Generate the local SSH config: + ```bash + homelab ssh-config --install + ``` + +## Hosts + +### Hubris (PVE host) + +| Detail | Value | +|--------|-------| +| LAN IP | `192.168.8.77` | +| Netbird | `100.122.38.109` (FQDN: `proxmox-server.netbird.selfhosted`) | +| Netbird SSH port | `22022` (mesh-only, OIDC auth) | +| SSH user | `root` | +| Authorized keys | `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced) | + +Authorized root keys currently deployed: +- `root@hubris` (self, RSA) +- `d.toro.v@pm.me` (ed25519) — mac-mini + +### LXCs + +Every LXC at `192.168.8.x` accepts root SSH via authorized_keys. Keys +are managed by `ssh/deploy-keys.sh`. SSH user is `root`. + +| LXC | Name | LAN IP | Role | +|-----|------|--------|------| +| 101 | jellyfin | `192.168.8.206` | media-server | +| 102 | nfs-export | `192.168.8.200` | storage-export | +| 103 | paperless | `192.168.8.130` | document-archive | +| 104 | gitea | `192.168.8.121` | git-server | +| 105 | apps | `192.168.8.205` | docker-apps | +| 106 | auth-outpost | `192.168.8.184` | authentik-outpost | +| 107 | dns | `192.168.8.185` | dns-helper | +| 114 | nextcloud | `192.168.8.224` | file-sync | +| 118 | elementsynapse | `192.168.8.239` | matrix-server | +| 119 | sophia | `192.168.8.157` | workshop | +| 120 | mule-images | `192.168.8.136` | photo-management | +| 121 | caddy | `192.168.8.175` | reverse-proxy | +| 122 | arriman | `192.168.8.132` | arr-stack | +| 123 | claudio-bot | `192.168.8.230` | matrix-agent | +| 126 | plato | `192.168.8.190` | app | + +### Workstations + +| Name | OS | LAN IP | Netbird FQDN | SSH user | +|------|----|--------|--------------|----------| +| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` | +| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` | +| ludo-mini | Linux | `192.168.8.133` | `ludo-mini.netbird.selfhosted` | TBD | + +### VPS (external) + +| Detail | Value | +|--------|-------| +| Public IP | `82.165.190.79` | +| Netbird | `100.122.165.149` (FQDN: `netbird-ionos.netbird.selfhosted`) | +| SSH user | `root` | +| Access | Mesh-only — public port 22 is blocked by nftables. Key-only auth. | + +## VPS + +Access is mesh-only. From a mesh-connected peer: + +```bash +ssh root@100.122.165.149 +ssh root@netbird-ionos.netbird.selfhosted +# or via homelab: +homelab ssh netbird-vps +``` + +## Verification + +```bash +# From any workstation after running homelab ssh-config --install: +for name in hubris gitea apps sophia paperless caddy jellyfin nextcloud; do + ssh -o BatchMode=yes "$name" "hostname" && echo "$name OK" +done +``` ## Related -- [Hubris host](../hosts/hubris.md) + - [Mesh migration](mesh.md) - [VPS hardening](vps-hardening.md) +- [Agent enrollment](../operations/agent-enrollment.md) +- [Homelab CLI](../bin/homelab) ## Changelog +### 2026-06-02 — universal SSH reachability + +Replaced ad-hoc per-workstation SSH configs with inventory-generated +configs (`ssh/gen-config.py`, `homelab ssh-config`). Added centralized +key distribution (`ssh/deploy-keys.sh`, `ssh/authorized_keys/`). All +LXCs now accept root SSH from any workstation whose pubkey is in the +repo. mac-mini Remote Login enabled. Netbird subnet route +(192.168.8.0/24 via hubris) provides off-LAN reachability for all LAN +IPs. + ### 2026-04-28 — wiki entry created Initial documentation. ### 2026-04-23 — VPS SSH hardened to mesh-only -Public `:22` blocked at nftables. Key-only sshd. See [VPS hardening](vps-hardening.md). +Public `:22` blocked at nftables. Key-only sshd. ### 2026-04-22 — iMac key authorized on hubris -`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`. +`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`. \ No newline at end of file diff --git a/operations/agent-enrollment.md b/operations/agent-enrollment.md index 6b0263c..d48a2a3 100644 --- a/operations/agent-enrollment.md +++ b/operations/agent-enrollment.md @@ -163,6 +163,77 @@ For Claude Code: start a new session — the `homelab` MCP server appears in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management, 1 secrets-metadata). +## Post-bootstrap: SSH reachability + +A new workstation must be reachable from other workstations and must be +able to reach every host by short hostname. Run these steps after the +bootstrap verify passes: + +### 1. Enable SSH server + +```bash +# macOS: +sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist + +# Linux: +sudo systemctl enable --now sshd +``` + +### 2. Generate SSH key (if missing) + +```bash +ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100 +``` + +### 3. Publish pubkey to the repo + +```bash +cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub +cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push +``` + +### 4. Deploy keys to all hosts + +From any existing enrolled machine (hubris or another workstation): + +```bash +ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh" +``` + +This adds the new workstation's pubkey to hubris and every running LXC. + +### 5. Generate SSH config + +```bash +homelab ssh-config --install +``` + +Verify: + +```bash +ssh hubris hostname # should return "hubris" without password +ssh gitea hostname # should return "gitea" without password +ssh mac-mini hostname # should return "mac-mini" without password (workstation-to-workstation) +``` + +### 6. Add LAN IP to inventory (if on LAN) + +If the workstation has a static or reserved LAN IP, add it to +`inventory.yaml`: + +```yaml +hosts: + your-hostname: + lan_ip: 192.168.8.xxx +``` + +This gives it a primary LAN entry in the generated SSH config (faster +than the Netbird fallback). Commit + push, then: + +```bash +cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install +``` + ## Claude Code permissions for fleet ops By default Claude Code's auto-mode classifier asks for confirmation on every @@ -278,6 +349,12 @@ The CLI prints a follow-up checklist that the operator must do manually: ## Changelog +### 2026-06-02 — SSH reachability post-bootstrap steps +Added a new "Post-bootstrap: SSH reachability" section covering SSH key +generation, pubkey publication, deployment to hosts, SSH config generation, +and LAN IP registration. New workstations enrolled via this doc will +automatically join the universal SSH mesh. + ### 2026-05-31 — cross-link to hermes-agent.md Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](./hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.