docs: move narrative wiki under knowledge/wiki/ (phase 3)
Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.
Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
outbound + between-moved-files), including .hermes/, runbooks, operations,
investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
embed doc_page); update oikos/gen-topology.py output path, candidate doc
paths, and footer links; update code-comment doc paths.
Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.
Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
topology regen fixed one, introduced none). Remaining are pre-existing refs
to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
149
knowledge/wiki/infrastructure/auto-deploy.md
Normal file
149
knowledge/wiki/infrastructure/auto-deploy.md
Normal file
@@ -0,0 +1,149 @@
|
||||
# Auto-deploy — gitea-webhook pipelines
|
||||
|
||||
Several configs and apps in the lab live in `dtoro/*` repos on [gitea (104)](../containers/104-gitea.md) and auto-redeploy on push. All pipelines follow one of two shapes.
|
||||
|
||||
## Two shapes
|
||||
|
||||
### Shape A — checkout IS the working tree (config repos)
|
||||
|
||||
`/etc/<thing>` or `/var/lib/<thing>/...` is itself a `git clone`. Push triggers `git pull` + a reload command. Used for pure-config repos where re-cloning is cheap.
|
||||
|
||||
### Shape B — receiver outside the app repo (compose stacks)
|
||||
|
||||
The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`webhook.py`, `deploy.sh`, systemd unit) lives in a sibling `/opt/<thing>-deploy/` so the app repo stays portable. Push triggers `git pull` + `docker compose up -d --build`. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout.
|
||||
|
||||
## Common
|
||||
|
||||
- All receivers validate `X-Gitea-Signature` HMAC-SHA256 against a per-pipeline secret in `/etc/<thing>-deploy/secret`.
|
||||
- All filter to `refs/heads/main` (or `master` for older repos). Gitea's "test delivery" button sends `ref=main` (without `refs/heads/`) — those will log "ignoring ref main" and 204. Real pushes work. **Don't "fix" the ref filter to accept both** — it'd also accept PR merges from side branches that got fast-forwarded.
|
||||
- Gitea's `app.ini` `[webhook] ALLOWED_HOST_LIST` must include every receiver IP. Currently:
|
||||
- `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
|
||||
- `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
|
||||
- `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
|
||||
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
|
||||
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
|
||||
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
|
||||
- ~~`192.168.8.190` ([plato (126)](../containers/126-plato.md))~~ (destroyed 2026-06-28)
|
||||
- `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito)
|
||||
|
||||
**Don't strip these when editing app.ini.**
|
||||
|
||||
- Git creds for root-run deploy services live in `/etc/<thing>-deploy/git-credentials` (mode 600) and are wired via `credential.helper = store --file=/etc/<thing>-deploy/git-credentials` in the repo's `.git/config`. Necessary because the unit typically runs with `ProtectHome=true`, which blocks `/root`.
|
||||
|
||||
## Pipelines
|
||||
|
||||
| Repo | Target | Shape | Receiver | Webhook id | Reload action |
|
||||
| ------------------------------- | -------------------------------------------- | ----- | ------------------------------------- | ---------- | ------------- |
|
||||
| `dtoro/caddy-conf` | [caddy (121)](../containers/121-caddy.md) `/etc/caddy/` | A | `http://192.168.8.175:9797/deploy` | 2 | `caddy validate` + `systemctl reload caddy` |
|
||||
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
|
||||
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
|
||||
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
|
||||
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/126-plato.md) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
|
||||
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
|
||||
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
|
||||
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
|
||||
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
|
||||
| `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` |
|
||||
| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) |
|
||||
|
||||
> Note: `dtoro/Homelab-Docs` has **three webhooks** firing on the same push.
|
||||
> Each owns its own clone on LXC 105. They don't conflict because each
|
||||
> deploy.sh only touches its own service unit + venv.
|
||||
|
||||
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
|
||||
|
||||
## When you change a tracked config
|
||||
|
||||
Always commit + push. Local-only edits drift. Common ones:
|
||||
|
||||
- `/etc/caddy/Caddyfile` ↔ `dtoro/caddy-conf` (auto-deploys)
|
||||
- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys)
|
||||
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
|
||||
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
|
||||
- ~~`/opt/plato/app/` ↔ `dtoro/Plato`~~ (destroyed 2026-06-28)
|
||||
- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
|
||||
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
|
||||
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
|
||||
|
||||
## Per-pipeline notes / gotchas
|
||||
|
||||
### caddy-conf
|
||||
- Repo includes `scripts/webhook/install.sh`. Editing the systemd unit *inside the repo* does **not** auto-reinstall — re-run `install.sh` manually after unit edits.
|
||||
- The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`).
|
||||
|
||||
### gitea-customizations
|
||||
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
|
||||
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
|
||||
|
||||
### mule-image / Artifacto
|
||||
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
|
||||
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
|
||||
- Manual deploy: `pct exec <id> -- /opt/<thing>-deploy/deploy.sh`.
|
||||
- Health: `pct exec <id> -- curl -s http://127.0.0.1:<port>/health` → `ok`.
|
||||
- Setup tokens used to register the webhook (e.g., `artifacto-deploy-setup`, `artifacto-deploy-setup-2`, `artifacto-cleanup` on user `dtoro`) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's `/users/{u}/tokens` endpoints require basic auth (not bearer), so cleanup couldn't be automated.
|
||||
|
||||
### backup-library
|
||||
- Currently the only deploy that targets the host directly (`192.168.8.77:9798`).
|
||||
- `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run.
|
||||
|
||||
### homelab-mcp / secrets-issuance
|
||||
- Both ride a single push to `dtoro/Homelab-Docs`. Two clones on LXC 105
|
||||
(`/opt/homelab-mcp`, `/opt/secrets-issuance`) — each is an independent
|
||||
Shape-B target with its own webhook receiver.
|
||||
- The deploy script restarts the service it just updated. Because the
|
||||
webhook receiver itself is a separate systemd unit (`*-deploy.service`),
|
||||
it does NOT restart itself — but `deploy.sh` running `systemctl
|
||||
restart homelab-mcp-deploy.service` (or the secrets-issuance one)
|
||||
would create a kill-self loop. The current `deploy.sh` is careful
|
||||
to only restart the main service.
|
||||
- Both services consume `/opt/homelab-context` for their runtime data
|
||||
(inventory, secret recipient lookup). That clone is **the same clone
|
||||
every other client has** — kept fresh by `homelab-context-sync.timer`,
|
||||
not by these webhooks.
|
||||
|
||||
## Custom-built binaries that overlap apt-managed paths
|
||||
|
||||
If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly `/usr/bin/<name>` — then the next `apt upgrade` of the corresponding package will silently clobber the custom build. That's exactly how [LXC 121 caddy](../containers/121-caddy.md) went down for ~10 min on 2026-05-21: an xcaddy build with `caddy-dns/ionos` lived at `/usr/bin/caddy` and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile.
|
||||
|
||||
Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary:
|
||||
|
||||
1. **Ship the build as a `.deb` with an epoch-bumped version.** Use `dpkg-deb --build` (or `nfpm`) to package the binary as `Package: <name>`, `Version: 1:<upstream>-hubris<n>`. The epoch (`1:`) means it beats any non-epoch upstream version regardless of point bumps, so `apt upgrade` is a no-op for that package. Used by caddy: `caddy 1:2.11.3-hubris1` (see commit `2026-05-21` in [121-caddy.md](../containers/121-caddy.md)).
|
||||
|
||||
2. **Hold the apt package.** `apt-mark hold <pkg>` on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than `.deb` packaging but: (a) the hold flag isn't preserved by `dpkg -i` of a new version, (b) it's invisible unless you check `apt-mark showhold`, (c) you have to remember to `apt-mark unhold` when you intentionally want a new version. The new `homelab apt-audit` subcommand surfaces holds across the fleet so they don't get forgotten.
|
||||
|
||||
If you're not sure what's already lurking, run `homelab apt-audit --fleet` and look at the `NONAPT` column — that's a count of binaries in `/usr/bin/{caddy,docker,jellyfin}` + `/usr/local/bin/*` that no apt package owns.
|
||||
|
||||
## Related
|
||||
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||
- [Backups (disabled)](backups.md)
|
||||
- [Operations cheatsheet](../../../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-28 — Plato pipeline decommissioned
|
||||
LXC 126 destroyed, webhook id 8 on `dtoro/Plato` removed. `192.168.8.190` removed from gitea `app.ini` `ALLOWED_HOST_LIST`.
|
||||
|
||||
### 2026-06-24 — terminalito pipeline added
|
||||
Webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` on [trmnl (128)](../containers/128-trmnl.md). Shape B (`server/deploy/webhook.py` receiver, in-repo `server/deploy/deploy.sh`; secret `/etc/terminalito-deploy/secret`). `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. Verified end-to-end with a push. Repo-local `credential.helper` in `/opt/terminalito/.git/config` (the unit can't read root's global git config).
|
||||
|
||||
### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added
|
||||
Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
|
||||
|
||||
### 2026-05-13 — Plato pipeline added
|
||||
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation. Six active pipelines.
|
||||
|
||||
### 2026-04-22 — Artifacto pipeline added
|
||||
Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
|
||||
|
||||
### 2026-06-04 — claudio-bot pipeline decommissioned
|
||||
LXC 123 destroyed, `dtoro/claudio-bot` archived. Webhook port 9797 dead.
|
||||
|
||||
### 2026-04-21 — mule-image + claudio-bot pipelines added
|
||||
Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
|
||||
|
||||
### 2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped
|
||||
Initial three. Set the conventions everything else follows.
|
||||
142
knowledge/wiki/infrastructure/backups.md
Normal file
142
knowledge/wiki/infrastructure/backups.md
Normal file
@@ -0,0 +1,142 @@
|
||||
# Backups — restic on external drive (DEPRECATED — superseded)
|
||||
|
||||
> **DEPRECATED 2026-07-01.** Superseded by the **rclone → Proton Drive** off-host mirror on
|
||||
> [LXC 132 `rclone`](../containers/132-rclone.md). That job finally closes the off-host / 3-2-1 gap
|
||||
> this page flagged for months. The restic-on-USB job below is kept for archaeology; it has been
|
||||
> **DISABLED since 2026-04-22** and is not coming back in its old form.
|
||||
|
||||
## Current backup — rclone → Proton Drive (LXC 132)
|
||||
|
||||
- **Where:** [LXC 132 `rclone`](../containers/132-rclone.md) (`192.168.8.214`), `/mnt/library`
|
||||
mounted **read-only**.
|
||||
- **What:** plain `rclone sync` (Proton mirrors local; browsable files, no versioning) of the
|
||||
folders listed in `/etc/rclone-backup/folders.list`, to `proton:library-backup/…`.
|
||||
- **When:** monthly — `rclone-backup.timer` (`OnCalendar=*-*-01 03:00`).
|
||||
- **UI:** rclone Web GUI on `192.168.8.214:5572` (LAN-only, no auth).
|
||||
- **Encryption:** Proton's built-in E2E (no rclone `crypt` overlay).
|
||||
- **Runs / logs:** `/var/log/rclone-backup/` + `runs.jsonl`.
|
||||
- **Still a single off-host target** (Proton only). Not yet a full 3-2-1 (no second independent
|
||||
copy), but strictly better than the previous "no off-host copy at all."
|
||||
|
||||
See [132-rclone](../containers/132-rclone.md) for the full design.
|
||||
|
||||
---
|
||||
|
||||
## Legacy — restic on external drive (DISABLED 2026-04-22)
|
||||
|
||||
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../../investigations/2026-04-21-hubris-crash-loop.md).
|
||||
|
||||
## Status
|
||||
|
||||
**DISABLED 2026-04-22.** All four timers `systemctl disable --now`'d:
|
||||
- `backup-library@homecloud.timer`
|
||||
- `backup-library@images.timer`
|
||||
- `backup-library@small.timer`
|
||||
- `backup-library-check.timer`
|
||||
|
||||
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
|
||||
|
||||
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md).
|
||||
|
||||
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
|
||||
|
||||
## Design
|
||||
|
||||
Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs `/mnt/library`) into thermal danger.
|
||||
|
||||
Retention per tag: `--keep-last 3 --keep-monthly 12 --keep-yearly 3` with `--group-by host,tags,paths`.
|
||||
|
||||
## Components
|
||||
|
||||
- **Repo:** `dtoro/backup-library`
|
||||
- **Checkout:** `/opt/backup-library` on the [hubris host](../hosts/hubris.md)
|
||||
- **Auto-deploys** via gitea webhook → `http://192.168.8.77:9798/deploy`. See [auto-deploy](auto-deploy.md). [Gitea (104)](../containers/104-gitea.md) `app.ini` `ALLOWED_HOST_LIST` includes `192.168.8.77` for this.
|
||||
- **Restic repo:** `/mnt/backup/restic-library`. Passphrase `/etc/restic/passphrase` (mode 600). **Escrow in password manager — loss = permanent data loss.**
|
||||
- **External drive:** `/dev/sda1` ext4 label `backup-library` UUID `ff46e775-1ba1-4892-82c9-e5cac5be933a`. Fstab uses `noauto` + `nofail,x-systemd.device-timeout=10s,errors=remount-ro`.
|
||||
|
||||
## Mount-on-demand
|
||||
|
||||
`/usr/local/sbin/backup-usb.sh attach|detach|status`. All three backup units (`backup-library.service`, `backup-library@.service`, `backup-library-check.service`) have `ExecStartPre=backup-usb.sh attach` and `ExecStopPost=backup-usb.sh detach`. The helper toggles `/sys/bus/usb/devices/*/authorized` by matching vendor:product `090c:2320`, then mounts/unmounts `/mnt/backup`. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.
|
||||
|
||||
## UAS blacklist
|
||||
|
||||
`/etc/modprobe.d/usb-storage-quirks.conf`:
|
||||
```
|
||||
options usb-storage quirks=090c:2320:u
|
||||
```
|
||||
Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with `dmesg | grep "UAS is ignored"`.
|
||||
|
||||
## Schedule
|
||||
|
||||
Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:
|
||||
|
||||
| Timer | When | Include list | Approx size |
|
||||
| ---------------------------------- | -------------- | ------------------------------------ | ----------- |
|
||||
| `backup-library@homecloud.timer` | day 1 / month | `/etc/restic/include-homecloud.list` | ~315 G |
|
||||
| `backup-library@images.timer` | day 10 / month | `/etc/restic/include-images.list` | ~103 G |
|
||||
| `backup-library@small.timer` | day 20 / month | `/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper) | ~11 G |
|
||||
|
||||
Snapshots tagged `chunk-<group>` so forget/prune treats each series independently.
|
||||
|
||||
Ad-hoc full run (kept for manual use): `systemctl start backup-library.service` (no arg → uses `/etc/restic/include.list`, tag `monthly`).
|
||||
|
||||
Yearly integrity: `backup-library-check.timer` (`OnCalendar=yearly`) runs full `restic check --read-data`.
|
||||
|
||||
## Thermal caps
|
||||
|
||||
Baked into the systemd units:
|
||||
- `IOReadBandwidthMax=/mnt/library 50M`
|
||||
- `IOWriteBandwidthMax=/mnt/backup 30M`
|
||||
- `--read-concurrency=1` on restic.
|
||||
|
||||
## Wrapper
|
||||
|
||||
`/usr/local/sbin/backup-library.sh` — preflight → unlock → backup → forget/prune (`--group-by host,tags,paths`) → `check --read-data-subset=5%` → notify. Takes optional `<group>` arg or `GROUP=` env.
|
||||
|
||||
## Notifications
|
||||
|
||||
~~POST to claudio-bot (123) `http://192.168.8.230:9090/notify`~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes `send_message` via Matrix instead.
|
||||
|
||||
`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
|
||||
|
||||
## Recovery
|
||||
|
||||
Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/RECOVERY.md`). Covers `restic snapshots/ls/find/restore/mount`, uid/gid gotcha, cross-host recovery.
|
||||
|
||||
## Known SPOF
|
||||
|
||||
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) were not implemented before this job was retired — the **off-host copy is now provided by [rclone → Proton Drive (LXC 132)](../containers/132-rclone.md)** instead. A second independent copy is still outstanding.
|
||||
|
||||
## Drive history
|
||||
|
||||
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
|
||||
|
||||
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../../investigations/2026-04-21-hubris-crash-loop.md).
|
||||
|
||||
## Thermal monitoring
|
||||
|
||||
Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213`). See [monitoring](monitoring.md).
|
||||
|
||||
## Related
|
||||
- [Hubris host](../hosts/hubris.md)
|
||||
- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
||||
- [Monitoring](monitoring.md)
|
||||
- [Auto-deploy](auto-deploy.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../../../investigations/2026-04-21-hubris-crash-loop.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-07-01 — DEPRECATED; superseded by rclone → Proton Drive (LXC 132)
|
||||
Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclone`](../containers/132-rclone.md) (`/mnt/library` → Proton Drive, monthly, LAN Web GUI). This finally provides the off-host copy the "Known SPOF" note wanted. The restic-on-USB units on hubris remain `disabled` (drive already removed 2026-04-22); page restructured to lead with the current job and demote restic to "Legacy".
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation. Status remains DISABLED.
|
||||
|
||||
### 2026-04-22 — DISABLED
|
||||
Drive removed as the A/B test in the [crash investigation](../../../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
||||
|
||||
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
|
||||
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
|
||||
|
||||
### 2026-04-20 — deployed; redesigned for thermal-gentleness
|
||||
Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.
|
||||
137
knowledge/wiki/infrastructure/dns.md
Normal file
137
knowledge/wiki/infrastructure/dns.md
Normal file
@@ -0,0 +1,137 @@
|
||||
# DNS — split-horizon `*.hubris.network`
|
||||
|
||||
LAN clients resolve `*.hubris.network` to the [Caddy reverse proxy](../containers/121-caddy.md) (`192.168.8.175`). Public clients resolve to the IONOS VPS (`82.165.190.79`) via an IONOS wildcard, where they hit the [VPS traefik public ingress](ingress.md).
|
||||
|
||||
There is **no wildcard on the LAN side**. Every subdomain needs an explicit entry.
|
||||
|
||||
## Components
|
||||
|
||||
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
|
||||
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
|
||||
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
|
||||
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
|
||||
|
||||
## Live entries (as of 2026-06-04)
|
||||
|
||||
```
|
||||
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
|
||||
address=/sso.hubris.network/192.168.8.175 # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
|
||||
address=/git.hubris.network/192.168.8.175
|
||||
address=/media.hubris.network/192.168.8.175
|
||||
address=/paperless.hubris.network/192.168.8.175
|
||||
address=/books.hubris.network/192.168.8.175
|
||||
address=/home.hubris.network/192.168.8.175
|
||||
address=/cloud.hubris.network/192.168.8.175
|
||||
address=/matrix.hubris.network/192.168.8.175
|
||||
address=/proxmox.hubris.network/192.168.8.175
|
||||
address=/docker.hubris.network/192.168.8.175
|
||||
address=/jellyseerr.hubris.network/192.168.8.175
|
||||
address=/qbit.hubris.network/192.168.8.175
|
||||
address=/sab.hubris.network/192.168.8.175
|
||||
address=/blog.hubris.network/192.168.8.175
|
||||
address=/photos.hubris.network/192.168.8.175
|
||||
address=/photos-new.hubris.network/192.168.8.175
|
||||
address=/artifacto.hubris.network/192.168.8.175
|
||||
address=/zimaos.hubris.network/192.168.8.175
|
||||
address=/nfs-export.hubris.network/192.168.8.200
|
||||
```
|
||||
|
||||
Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do.
|
||||
|
||||
## Why split-horizon
|
||||
|
||||
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.
|
||||
|
||||
## The gotcha that cost a debug session (2026-04-22)
|
||||
|
||||
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
|
||||
|
||||
## Recipe — adding a new subdomain
|
||||
|
||||
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
|
||||
2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)` — the NetBird managed DNS zone sync picks it up within ~10 minutes via cron. Or add directly to the NetBird managed zone via API if you need it faster.
|
||||
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
|
||||
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
|
||||
|
||||
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../../../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
|
||||
|
||||
## Public path — what does and doesn't follow the LAN map
|
||||
|
||||
- Hostnames published in [VPS traefik dynamic config](ingress.md) (currently `artifacto.hubris.network`, `blog.hubris.network`) reach a real backend over the netbird mesh.
|
||||
- Anything else with a `*.hubris.network` URL hits the VPS but isn't routed anywhere — returns 404.
|
||||
- `netbird.hubris.network` is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.
|
||||
|
||||
## Long-term plan
|
||||
|
||||
Either:
|
||||
- Move split-horizon DNS to the LAN router so `*.hubris.network → 192.168.8.175` is answered for every LAN client. Eliminates per-LXC overrides.
|
||||
- Or, once the [Tailscale → Netbird migration](mesh.md) completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.
|
||||
|
||||
## Related
|
||||
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
|
||||
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
|
||||
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
|
||||
- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-28 — `plato.hubris.network` removed
|
||||
Plato (LXC 126) decommissioned. Technitium entry deleted; dns-sync cron reaped the NetBird managed zone record.
|
||||
|
||||
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
|
||||
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
|
||||
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
|
||||
- Caddy: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxied to VPS
|
||||
- `header_up Host auth.hubris.network` added to strip `:443` from upstream Host header
|
||||
- All 9 LXCs' /etc/hosts updated: `auth.hubris.network → 192.168.8.175` (Caddy proxy)
|
||||
- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
|
||||
- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
|
||||
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
|
||||
|
||||
- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
|
||||
- A **sync job on [dns (107)](../containers/107-dns.md)** (`/opt/dns-sync/sync.py`, cron */10) reconciles Technitium's named A-records → the **NetBird managed DNS zone** via the NetBird API (`/api/dns/zones/{id}/records`, PAT in sops `secrets/netbird-pat.yaml`). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (`.79`) — correct.
|
||||
- This **killed the manual drift** that caused the whole `auth`/`sso`/`nfs-export` saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.
|
||||
|
||||
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
|
||||
|
||||
> Reference: [scripts/dns-sync.py](../../../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
|
||||
|
||||
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
|
||||
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
|
||||
|
||||
**Fixed:** added `/etc/cron.d/dns-sync` (`*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1`).
|
||||
|
||||
Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backend-health`) that runs `scripts/check-caddy-backends.sh` every 10 minutes.
|
||||
|
||||
### 2026-06-02 — 8 LXCs moved from DHCP to static IP
|
||||
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
|
||||
|
||||
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
|
||||
Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
|
||||
|
||||
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
|
||||
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
||||
|
||||
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
|
||||
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
|
||||
|
||||
### 2026-05-14 — `zimaos.hubris.network` added (Caddy-fronted, standard pattern)
|
||||
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
|
||||
|
||||
### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
|
||||
New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
|
||||
|
||||
### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
|
||||
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation. 16 active entries.
|
||||
|
||||
### 2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79
|
||||
Public path now lands on the VPS traefik, not the old yunohost. Necessary for the [public ingress](ingress.md) pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.
|
||||
|
||||
### 2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)
|
||||
Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.
|
||||
|
||||
### 2026-04-21 — dnsmasq stood up on LXC 124
|
||||
Co-located with Authentik. Initial entries cover everything routed through Caddy.
|
||||
143
knowledge/wiki/infrastructure/homelab-context.md
Normal file
143
knowledge/wiki/infrastructure/homelab-context.md
Normal file
@@ -0,0 +1,143 @@
|
||||
# Homelab context distribution
|
||||
|
||||
The cross-client context-and-secrets system that makes every agent (Claude
|
||||
Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
|
||||
self-locating and able to read the same source of truth.
|
||||
|
||||
Operational walkthrough for enrolling a new client lives in
|
||||
[operations/agent-enrollment.md](../../../operations/agent-enrollment.md); this
|
||||
page is the architecture reference.
|
||||
|
||||
## What's where
|
||||
|
||||
| Piece | Host | Path | Role |
|
||||
| --- | --- | --- | --- |
|
||||
| Source of truth | [gitea (104)](../containers/104-gitea.md) | `dtoro/Homelab-Docs.git` | Inventory + wiki + service code |
|
||||
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, Hermes Agent |
|
||||
| `homelab` CLI | every enrolled client | `/usr/local/bin/homelab` → `/opt/homelab-context/bin/homelab` (symlink) | Operator surface for enroll/secret/ssh/pct |
|
||||
| Per-client age key | every enrolled client | `/etc/age/key.txt` (0600 root) | Decrypts SOPS-encrypted secrets the client is a recipient on |
|
||||
| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/mcp) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets |
|
||||
| Secrets-issuance | [apps (105)](../containers/105-apps.md) | `secrets-issuance.service` on port 9820 (https://secrets.hubris.network/issue) | Generates per-client age keypair on first bootstrap; idempotent; admin-token-gated `/revoke` |
|
||||
| Sync timer | every enrolled client | `homelab-context-sync.timer` (Linux) / `network.hubris.homelab-context-sync.plist` (macOS) | `git pull --ff-only` every 5 min |
|
||||
| Encrypted secrets | `dtoro/Homelab-Docs` | `secrets/*.yaml` (SOPS+age) | Recipients declared in `.sops.yaml` |
|
||||
| Read-only context PAT | `dtoro` Gitea user, scope `read:repository` | given to operators out-of-band | Bootstrap-only — for the initial clone before SOPS works |
|
||||
| Write-scoped PAT | `secrets/gitea-pat.yaml` (SOPS) | `homelab refresh-creds` swaps it into `/etc/homelab-context/git-credentials` | All post-bootstrap pushes (client lifecycle, wiki edits) |
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
dtoro/Homelab-Docs (gitea)
|
||||
│
|
||||
┌────────── push ────────┤ ◀── git push (write PAT or SSH)
|
||||
│ │
|
||||
│ ┌────── push ──────┘
|
||||
│ │ │
|
||||
│ │ ▼ webhook (push event)
|
||||
│ │ ┌─── homelab-mcp-deploy ──── (LXC 105:9811)
|
||||
│ │ └─── secrets-issuance-deploy (LXC 105:9821)
|
||||
│ │ │
|
||||
│ │ ▼
|
||||
│ │ git pull → deploy.sh → restart service
|
||||
│ │
|
||||
│ └── on every client:
|
||||
│ timer (5 min) → git pull --ff-only into /opt/homelab-context
|
||||
│
|
||||
▼
|
||||
homelab CLI / MCP server reads /opt/homelab-context for everything
|
||||
```
|
||||
|
||||
## Why two clones on LXC 105
|
||||
|
||||
The MCP server and secrets-issuance each have their own clone
|
||||
(`/opt/homelab-mcp`, `/opt/secrets-issuance`) **in addition to**
|
||||
`/opt/homelab-context`. Reasons:
|
||||
|
||||
- The deploy webhook for each service updates its own clone, runs
|
||||
`deploy.sh` from there, and re-installs the systemd unit. Mixing this
|
||||
with the client-context clone would create a circular dependency
|
||||
(deploy reinstalls the unit that pulled it).
|
||||
- The MCP server reads its data from `/opt/homelab-context` (the same path
|
||||
every client uses) so changes to inventory propagate identically. Code
|
||||
changes live in `/opt/homelab-mcp` and trigger a service restart.
|
||||
|
||||
## Mesh / network gates
|
||||
|
||||
- Both services bind `0.0.0.0:<port>`. The trust boundary is
|
||||
`MESH_SUBNETS` in the service's environment + nftables (planned). Today
|
||||
`MESH_SUBNETS=100.122.0.0/16,100.64.0.0/10,192.168.8.0/24` — Netbird +
|
||||
Tailscale + the homelab LAN. Adjust if the LAN ever has untrusted
|
||||
devices.
|
||||
- Caddy fronts both with Let's Encrypt certs via the IONOS DNS challenge:
|
||||
`mcp.hubris.network` → `192.168.8.205:9810`,
|
||||
`secrets.hubris.network` → `192.168.8.205:9820`. Off-LAN clients on
|
||||
Netbird reach them via the `192.168.8.0/24` network resource routed
|
||||
through the PVE peer ([mesh.md](mesh.md)).
|
||||
- Clients with default-public DNS (workstations not on Netbird, LXCs
|
||||
using router DNS) need a `/etc/hosts` override pointing
|
||||
`mcp.hubris.network` and `secrets.hubris.network` at the caddy LXC
|
||||
(`192.168.8.175`) — same caveat as every other `*.hubris.network`
|
||||
service, see [dns.md](dns.md).
|
||||
|
||||
## Secrets model
|
||||
|
||||
- Each enrolled client gets one **age private key** issued by
|
||||
secrets-issuance on first bootstrap. The key file stays root-only on
|
||||
the client; the public key is committed to `inventory.yaml` (and
|
||||
becomes a recipient on SOPS-encrypted files via `.sops.yaml`).
|
||||
- `secrets/*.yaml` are SOPS+age. Recipients per file are pinned in
|
||||
`.sops.yaml` `creation_rules` by `path_regex`. Re-encrypting a file is
|
||||
`sops updatekeys -y secrets/<name>.yaml`.
|
||||
- The MCP server's `list_my_secrets(caller_pubkey)` tool returns only
|
||||
secret *names* a given pubkey can decrypt — the server never sees
|
||||
plaintext. Decryption is local-on-client (`homelab secret <name>`
|
||||
shells out to `sops -d` with the client's key).
|
||||
- The "all-clients" secrets (`hello.yaml` for the bootstrap decrypt
|
||||
test, `gitea-pat.yaml` for the write-scoped PAT) are auto-granted to
|
||||
every newly enrolled client by `homelab client add --finalize-pubkey`
|
||||
(which appends the pubkey to the matching `.sops.yaml` rule and runs
|
||||
`sops updatekeys`).
|
||||
- **Removal does not erase past disclosure.** Revoking a client via
|
||||
`homelab client remove` shreds the issuance-side key, denylists the
|
||||
hostname, removes them from the recipient list, and re-keys all
|
||||
shared secrets — but anything they already decrypted to disk is out of
|
||||
your control. Rotate the underlying credential if compromise is
|
||||
suspected.
|
||||
|
||||
## Why this design
|
||||
|
||||
- **One source of truth** keeps inventory, code, secrets, and docs
|
||||
versioned together. A `git log` of `inventory.yaml` is the history of
|
||||
the homelab.
|
||||
- **Per-client age keys** scale better than a shared admin secret —
|
||||
removing a client is a real revocation (for new ciphertext), not just
|
||||
removing them from a wiki page.
|
||||
- **MCP layer over the same clone** gives MCP-capable agents structured
|
||||
query (`find_service`, `search_docs`) without forcing non-MCP tools to
|
||||
go without — anything can still `cat` the markdown.
|
||||
- **Sync timer rather than push fan-out** keeps the failure mode
|
||||
contained: one client's webhook outage doesn't block a push from
|
||||
landing on the others. Sub-5-min staleness is fine for docs and rare
|
||||
enough for secrets that we don't need lower latency.
|
||||
|
||||
## Related
|
||||
|
||||
- [Operations: agent enrollment](../../../operations/agent-enrollment.md) — the
|
||||
step-by-step for adding a new client
|
||||
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
|
||||
pipelines (and the rest of the lab's webhook pipelines)
|
||||
- [Mesh](mesh.md) — Netbird / Tailscale paths and the `192.168.8.0/24`
|
||||
network resource
|
||||
- [Apps (105)](../containers/105-apps.md) — where both services run
|
||||
- [Gitea (104)](../containers/104-gitea.md) — the source of truth
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-05-20 — system live across hubris, apps, republic-laptop
|
||||
Phase 1 of the [cross-client context plan](../../../README.md) merged. Three
|
||||
clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS
|
||||
decrypt verified on each. Webhook auto-deploy for both LXC 105 services
|
||||
wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
|
||||
`client add --finalize-pubkey` grant flow live so new clients are one
|
||||
ceremony instead of four manual steps. Outstanding: bootstrap mac-mini
|
||||
(macOS, exercises launchd) + ludo-mini + the remaining LXCs;
|
||||
Hermes Agent integration so the agent uses inventory at chat-time.
|
||||
66
knowledge/wiki/infrastructure/index.md
Normal file
66
knowledge/wiki/infrastructure/index.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Infrastructure — cross-cutting systems
|
||||
|
||||
The homelab's shared infrastructure: systems that span multiple nodes and
|
||||
are documented in their own pages. Each system below links to its full doc.
|
||||
|
||||
## Network
|
||||
|
||||
- **[Network](network.md)** — physical topology, subnets, routing, DHCP.
|
||||
Homelab `192.168.8.0/24` isolated from household `192.168.178.0/24`.
|
||||
Proxmox hubris acts as subnet router.
|
||||
- **[DNS — split-horizon](dns.md)** — Technitium DNS on LXC 107,
|
||||
`192.168.8.2:53`. `*.hubris.network` resolves to LAN IPs on the homelab
|
||||
network and to mesh addresses off-LAN.
|
||||
|
||||
## Connectivity / mesh
|
||||
|
||||
- **[Mesh — Tailscale → Netbird migration](mesh.md)** — overlay networking.
|
||||
Netbird is the preferred path; Tailscale is legacy.
|
||||
- **[SSH access](ssh-access.md)** — dual-server SSH (OpenSSH + Netbird SSH)
|
||||
on hubris, key distribution.
|
||||
|
||||
## Public ingress
|
||||
|
||||
- **[Public ingress — VPS traefik + cert mirror](ingress.md)** — how home
|
||||
services reach the open internet. Two-stage: VPS traefik (IONOS) terminates
|
||||
TLS, proxies over Netbird to home Caddy.
|
||||
- **[Caddy reverse proxy](../containers/121-caddy.md)** — LAN endpoint.
|
||||
Terminates TLS for every `*.hubris.network` hostname, forwards to backends.
|
||||
|
||||
## Storage
|
||||
|
||||
- **[Media permissions — GID 10000 standard](media-permissions.md)** — shared
|
||||
group permission model across all LXCs that read/write the media library.
|
||||
- **[Backups — rclone → Proton Drive](backups.md)** — off-host backup strategy.
|
||||
LXC 132 handles rclone to Proton Drive; restic-on-USB deprecated.
|
||||
|
||||
## Identity & access
|
||||
|
||||
- **[Authentik SSO](../containers/124-authentik.md)** — identity provider.
|
||||
Core server runs on the VPS; LAN forward-auth outpost at LXC 106.
|
||||
OIDC providers configured for Jellyfin, Jellyseerr, Sabnzbd, qBittorrent,
|
||||
Yuvomi, and more.
|
||||
|
||||
## Management & automation
|
||||
|
||||
- **[Homelab context distribution](homelab-context.md)** — `/opt/homelab-context`
|
||||
clone, MCP server, secrets issuance, cross-client sync.
|
||||
- **[Auto-deploy — gitea-webhook pipelines](auto-deploy.md)** — push-to-deploy
|
||||
for Caddy config, mule-image, and other tracked repos.
|
||||
- **[Monitoring](monitoring.md)** — health checks, watchdogs, alerting
|
||||
(migrated from claudio-bot to Hermes cron).
|
||||
- **[VPS hardening](vps-hardening.md)** — IONOS netbird VPS: fail2ban,
|
||||
nftables, OIDC SSH, security posture.
|
||||
|
||||
## Topology
|
||||
|
||||
- **[Topology diagram (generated)](topology.md)** — Mermaid graph of compute,
|
||||
ingress routing, and storage mounts. Auto-generated from `inventory.yaml`
|
||||
by `oikos/gen-topology.py`.
|
||||
|
||||
## Related
|
||||
|
||||
- [README](../../../README.md) — entry point
|
||||
- [Containers index](../containers/index.md)
|
||||
- [Operations cheatsheet](../../../operations/commands.md)
|
||||
- [OIKOS operating model](../../../OIKOS.md)
|
||||
103
knowledge/wiki/infrastructure/ingress.md
Normal file
103
knowledge/wiki/infrastructure/ingress.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Public ingress — VPS traefik + cert mirror
|
||||
|
||||
How home services reach the open internet without exposing the home network. Two-stage pattern: traefik on the IONOS VPS terminates TLS at the public edge, then reverse-proxies over the netbird mesh to home Caddy / direct backends.
|
||||
|
||||
## The shape
|
||||
|
||||
```
|
||||
Public client
|
||||
│ *.hubris.network → 82.165.190.79 (IONOS wildcard)
|
||||
▼
|
||||
[VPS traefik] priority 1: HostSNI(*) → netbird-proxy:8443 ← netbird control plane
|
||||
priority 10: per-host HTTP routers ← home services
|
||||
│ HTTP over netbird mesh
|
||||
▼
|
||||
[Home backend on 192.168.8.x]
|
||||
```
|
||||
|
||||
LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
|
||||
|
||||
## Why this shape
|
||||
|
||||
- VPS traefik already has a HostSNI(*) TCP passthrough at priority 1 so netbird's own ingress (`netbird.hubris.network`, future `*.proxy.hubris.network`) is unaffected.
|
||||
- Per-hostname HTTP file-provider routers at priority 10 win over the passthrough for the listed hosts and let traefik terminate TLS itself for those.
|
||||
- Traefik's own ACME (`letsencrypt` resolver) fails on this box: HostSNI(*) grabs TLS-ALPN-01 challenges before traefik's `allowACMEByPass=true` can respond. Solution: home Caddy obtains certs via IONOS DNS-01 (no conflict) and the VPS *mirrors* the result over.
|
||||
|
||||
## Components
|
||||
|
||||
### On the VPS (`82.165.190.79`)
|
||||
|
||||
- `/opt/traefik-dynamic.yaml` — file-watched dynamic config. One `http.routers.<name>-public` + one `http.services.<name>-public` per exposed service, plus one entry in the top-level `tls.certificates` list per hostname.
|
||||
- Let's-Encrypt-effective directory: `/letsencrypt/` inside the traefik container, backed by the host-side docker volume `opt_netbird_traefik_letsencrypt`.
|
||||
- Backups before edits: `cp /opt/traefik-dynamic.yaml /opt/traefik-dynamic.yaml.bak.$(date +%s)` — several bak files live alongside.
|
||||
|
||||
### On the PVE host (`192.168.8.77`)
|
||||
|
||||
- `/usr/local/bin/hubris-public-cert-sync.sh` — runs daily via `hubris-public-cert-sync.timer`. Maps source hostname → VPS cert filenames in a bash assoc array. For each mapping: `pct pull` cert+key from [Caddy (121)](../containers/121-caddy.md)'s store, diff against the VPS copy, scp only on change.
|
||||
- Filenames are stable per host so the dynamic.yaml never needs editing on renewal — traefik file-watches and hot-reloads the cert.
|
||||
|
||||
## Services currently exposed
|
||||
|
||||
| Hostname | Path scope | Backend | Middlewares | Cert files on VPS |
|
||||
| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ |
|
||||
| `artifacto.hubris.network` | `/p/*`, `/static/*`, `/healthz` | `192.168.8.205:3100` | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key` |
|
||||
| `blog.hubris.network` | whole host | `192.168.8.205:8080` | `blog-ratelimit` (100 rps / 200 burst) | `blog.fullchain.crt` / `blog.privkey.key` |
|
||||
| `trmnl.hubris.network` | whole host | `192.168.8.211:9851` ([trmnl 128](../containers/128-trmnl.md)) | `trmnl-ratelimit` (20 rps / 40 burst) | `trmnl.fullchain.crt` / `trmnl.privkey.key` |
|
||||
| `house.hubris.network` | whole host | `192.168.8.212:3000` ([house 129](../containers/129-house.md)) | `house-ratelimit` (30 rps / 60 burst) | `house.fullchain.crt` / `house.privkey.key` |
|
||||
|
||||
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
|
||||
|
||||
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
|
||||
|
||||
Since 2026-05-31 [Authentik runs on the VPS itself](../../../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
|
||||
|
||||
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
|
||||
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
|
||||
- Traefik reaches it over the `auth` Docker network (`172.30.1.0/24`); Postgres/Redis on that net are isolated from the netbird containers.
|
||||
- Admin UI is IP-gated: an `admin-allowlist` ipAllowList middleware on `PathPrefix(/if/admin/)` (currently `5.61.168.0/24`). Login/flow endpoints stay public.
|
||||
|
||||
No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `auth`.
|
||||
|
||||
## Recipe — exposing another service
|
||||
|
||||
1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`).
|
||||
2. Add an entry to `HOSTS` in `/usr/local/bin/hubris-public-cert-sync.sh` mapping the hostname → VPS filenames. Run once: `systemctl start hubris-public-cert-sync.service`. Confirm the cert landed.
|
||||
3. Edit `/opt/traefik-dynamic.yaml` on the VPS:
|
||||
- Add to `tls.certificates`: paths `/letsencrypt/<crt>` and `/letsencrypt/<key>`.
|
||||
- Add `http.routers.<host>-public`: `rule: 'Host(\`<host>\`)'` (or with path matchers if scope-gating), `entryPoints: [websecure]`, `priority: 10`, `tls: {}`, `service: <host>-public`, `middlewares: [...]`.
|
||||
- Add a ratelimit middleware under `http.middlewares` if wanted.
|
||||
- Add `http.services.<host>-public.loadBalancer.servers[0].url: 'http://<lan-ip>:<port>'`.
|
||||
4. Verify:
|
||||
```
|
||||
ssh root@100.122.165.149 'curl -skI --resolve <host>:443:127.0.0.1 https://<host>/' # 2xx/3xx
|
||||
curl -skI --resolve <host>:443:<vps-public-ip> https://<host>/ # same
|
||||
```
|
||||
5. **No DNS edit needed** — the IONOS wildcard already points at the VPS.
|
||||
|
||||
## What does NOT follow this pattern
|
||||
|
||||
- `netbird.hubris.network` (and any future `*.proxy.hubris.network`) uses the netbird-proxy / HostSNI passthrough path. Netbird handles its own cert via ACME cleanly because it *is* the passthrough target.
|
||||
|
||||
## Related
|
||||
- [DNS split-horizon](dns.md)
|
||||
- [Caddy (121)](../containers/121-caddy.md) — cert source, internal counterpart
|
||||
- [Mesh migration](mesh.md) — netbird is the transport between VPS and home
|
||||
- [VPS hardening](vps-hardening.md) — fail2ban / nftables that the access logs feed
|
||||
- [Artifacto on apps (105)](../containers/105-apps.md) — first publicly-exposed service
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-24 — `trmnl.hubris.network` exposed
|
||||
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
|
||||
|
||||
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
|
||||
Authentik migrated onto the VPS ([investigation](../../../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
### 2026-04-23 — `blog.hubris.network` exposed
|
||||
WriteFreely on [apps (105)](../containers/105-apps.md). Whole host is public.
|
||||
|
||||
### 2026-04-22 — pattern established with Artifacto
|
||||
First service through the file-provider router. IONOS wildcard moved to the VPS this day. Cert mirror script + timer deployed on the PVE host.
|
||||
101
knowledge/wiki/infrastructure/media-permissions.md
Normal file
101
knowledge/wiki/infrastructure/media-permissions.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# Media permissions — `media` GID 10000
|
||||
|
||||
Standard for any LXC reading/writing `/mnt/library` on [hubris](../hosts/hubris.md). Applied 2026-04-20.
|
||||
|
||||
## Standard
|
||||
|
||||
Every LXC that mounts `/mnt/library` participates in a shared `media` group with **GID 10000**. Shared subtrees are owned by that group with the setgid bit (`drwxrwsr-x`, mode `2775`), so new files auto-inherit the right group regardless of which container wrote them.
|
||||
|
||||
## Why
|
||||
|
||||
`/mnt/library` is a cross-container storage pool. \*arr writes, jellyfin reads, mulita scans, paperless ingests. Without a shared group, each container sees files as `nobody:nogroup` (unprivileged) or `www-data` (privileged 1:1) and the permission web collapses into one-off chmods. GID 10000 bridges privileged and unprivileged containers.
|
||||
|
||||
## Onboarding a new LXC
|
||||
|
||||
1. `pct set <id> -mp0 /mnt/library,mp=/mnt/library` (if not already mounted).
|
||||
2. Inside the container:
|
||||
```
|
||||
groupadd -g 10000 media
|
||||
usermod -aG media <service-user> # for every user that needs library access
|
||||
```
|
||||
3. If the container is **unprivileged** (check `pct config <id> | grep unprivileged`), append this idmap block to `/etc/pve/lxc/<id>.conf` (back up first):
|
||||
```
|
||||
lxc.idmap: u 0 100000 65536
|
||||
lxc.idmap: g 0 100000 10000
|
||||
lxc.idmap: g 10000 10000 1
|
||||
lxc.idmap: g 10001 110001 55535
|
||||
```
|
||||
Then `pct stop <id> && pct start <id>`.
|
||||
4. For systemd services running with `User=root` (not typical), add a drop-in with `SupplementaryGroups=media`. Systemd skips `initgroups()` for `User=root`.
|
||||
5. `pct exec` sessions don't get supplementary groups (no initgroups). Use `sudo -i` or `su - <user>` inside the container to verify membership interactively. Real services use `initgroups` and work correctly.
|
||||
|
||||
## State snapshot
|
||||
|
||||
### Host
|
||||
|
||||
- Group `media` GID 10000 exists.
|
||||
- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000).
|
||||
- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid):
|
||||
- `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries
|
||||
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / grimmory
|
||||
- `downloads` — \*arr stack output
|
||||
- `images` — photoprism / immich / mulita
|
||||
- `roms` — emu frontends
|
||||
- `syncthing` — empty subtree, retained for archaeology (LXC 109 destroyed 2026-05-14)
|
||||
- Container-specific subtrees intentionally **not** migrated (keep their own owner:group):
|
||||
- `documents` (paperless, `www-data:www-data 750`)
|
||||
- `homecloud` (nextcloud — its own permission model, easy to break)
|
||||
- `marimo` (marimo venv) — *LXC since destroyed; review whether subtree still serves a purpose*
|
||||
- `notes`, `sophia` (single-container use); `heaper` — orphaned data subtree (LXC since destroyed 2026-05-14, 224 MiB retained)
|
||||
- `repos` (owner UID 102 GID 105 from inside [gitea](../containers/104-gitea.md) — don't touch)
|
||||
|
||||
### LXCs with media-group membership
|
||||
|
||||
| ID | Name | Priv | Media-group members |
|
||||
| --- | --------------------------------------------- | ---- | --------------------------------------------- |
|
||||
| 101 | [jellyfin](../containers/101-jellyfin.md) | **unpriv + idmap** | jellyfin |
|
||||
| 103 | [paperless](../containers/103-paperless.md) | priv | www-data |
|
||||
| 104 | [gitea](../containers/104-gitea.md) | priv | www-data, gitea |
|
||||
| 105 | [apps](../containers/105-apps.md) | priv | www-data |
|
||||
| 114 | [nextcloud](../containers/114-nextcloud.md) | priv | www-data |
|
||||
| 119 | [sophia](../containers/119-sophia.md) | priv | www-data |
|
||||
| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data |
|
||||
| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
|
||||
| 130 | [grimmory](../containers/130-grimmory.md) | priv | Docker container uses `GROUP_ID=10000` env var (linuxserver pattern) — no in-LXC group needed |
|
||||
| 132 | [rclone](../containers/132-rclone.md) | priv | **read-only** mount; runs as root → reads all subtrees. No media group needed |
|
||||
|
||||
> Some entries from earlier snapshots — 100 (arr-yunohost), 107 (marimo), 109 (syncthing), 110 (photoprism), 112 (immich), 116 (heaper) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).
|
||||
|
||||
Config backups: `/root/101.conf.bak.*`, `/root/109.conf.bak.*` (109 destroyed 2026-05-14).
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **[apps (105)](../containers/105-apps.md) and [grimmory (130)](../containers/130-grimmory.md) are Docker hosts.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000`, `user: "<uid>:10000"`, or `GROUP_ID=10000` (linuxserver images) in compose. Grimmory, audiobookshelf-in-docker, etc. need this per-container.
|
||||
- **`pct exec` does NOT run initgroups.** So `pct exec <id> -- id` shows only the primary group. For interactive verification, use `pct exec <id> -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine.
|
||||
- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed.
|
||||
- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page.
|
||||
- **`/etc/subgid`** must retain both `root:100000:65536` AND `root:10000:1`. Dropping the second breaks startup of any unprivileged LXC with the idmap block.
|
||||
- **\*arr "Set Permissions" options** can override the setgid inheritance by explicitly chown'ing files. Leave those off, or set the group to `media`. Relevant to Sonarr/Radarr/qBittorrent on [arriman (122)](../containers/122-arriman.md).
|
||||
- **Nextcloud** files under `/mnt/library/homecloud` are deliberately NOT in the media group. NC manages its own permission model. See [nextcloud (114)](../containers/114-nextcloud.md).
|
||||
- **\*arr stack on arriman** required `MEDIACENTER_GID=10000` (not 13000) in `.env` because s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. See [arriman (122)](../containers/122-arriman.md#changelog).
|
||||
|
||||
## Related
|
||||
- [Hubris host](../hosts/hubris.md)
|
||||
- All container pages list whether they're in the standard
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-05-14 — LXC 109 (syncthing) destroyed
|
||||
Removed the syncthing row from the membership table and the syncthing-as-`User=root` example from the onboarding section. `/mnt/library/syncthing` subtree was already empty and retained as an empty dir.
|
||||
|
||||
### 2026-05-14 — LXC 116 (heaper) destroyed
|
||||
Removed the heaper row from the LXC membership table and noted the orphaned `/mnt/library/heaper` subtree (224 MiB retained). See [host changelog](../hosts/hubris.md#changelog).
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
### 2026-04-26 — `MEDIACENTER_GID` fix on [arriman (122)](../containers/122-arriman.md)
|
||||
qBit was erroring every torrent with "Permission denied" because `MEDIACENTER_GID=13000` was set as a supplementary GID via `group_add:`. Changed to 10000 (primary GID); fix described above is now standard.
|
||||
|
||||
### 2026-04-20 — standard rolled out
|
||||
GID 10000 hostgroup, idmap blocks for unprivileged LXCs, setgid 2775 on shared subtrees, `media` membership for service users in every participating LXC.
|
||||
194
knowledge/wiki/infrastructure/mesh.md
Normal file
194
knowledge/wiki/infrastructure/mesh.md
Normal file
@@ -0,0 +1,194 @@
|
||||
# Mesh — Tailscale → Netbird migration
|
||||
|
||||
The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target end-state. In-progress as of 2026-04-21.
|
||||
|
||||
## Current state
|
||||
|
||||
- **PVE host** uses Netbird (`wt0`, `100.122.38.109/16`). Its resolver is the local netbird daemon, which forwards to LAN/upstream — so the PVE host gets `*.hubris.network → 192.168.8.175` via the system resolver chain.
|
||||
- **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration.
|
||||
- **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service.
|
||||
|
||||
## ICE / STUN / TURN
|
||||
|
||||
**Today** (post-2026-05-21 migration):
|
||||
|
||||
- External STUN servers (Google + Cloudflare) declared under `server.Stuns` in `/opt/management.json`. Embedded STUN is no longer in use.
|
||||
- coturn runs on the VPS host (apt-installed, systemd-managed). TURN URL `turn:netbird.hubris.network:3478?transport=tcp` is advertised to peers via mgmt's `TURNConfig.Turns` block. Long-term credentials at `user=netbird:<pwd from /root/turn-pass.txt>`.
|
||||
- For non-symmetric peers, ICE picks direct `srflx/srflx` (P2P). For symmetric-NAT peers, ICE falls through to TURN-relay before falling back to the WSS relay (`rels://netbird.hubris.network:443`).
|
||||
|
||||
**IONOS port-3478 caveat** (load-bearing — undocumented before 2026-05-21):
|
||||
|
||||
IONOS upstream filters STUN-class traffic on port 3478 **for both UDP and TCP** by default. The UDP block was already known (verified 2026-05-10 with `tcpdump -ni any udp port 3478`). The TCP block was discovered 2026-05-21 when TURN-allocate retries timed out from a symmetric-NAT laptop: TCP handshake completed (carried by kernel-only SYN exchange) but data packets never reached the VPS's `ens6` interface.
|
||||
|
||||
**Resolution**: operator must add an inbound firewall exception in the IONOS dashboard (Cloud Panel → Networks → Firewall policy on the VPS) for inbound TCP 3478 (and UDP 3478 if STUN-via-VPS-listener is ever needed; we use external STUN so this is optional). Without that exception, coturn is unreachable from outside even though host nftables + listener look fine.
|
||||
|
||||
**Verifying TURN works** end-to-end from an outside peer:
|
||||
|
||||
```python
|
||||
# python3
|
||||
import socket, struct, secrets
|
||||
s = socket.create_connection(("netbird.hubris.network", 3478), timeout=10)
|
||||
tid = secrets.token_bytes(12)
|
||||
attrs = struct.pack("!HHI", 0x0019, 4, 17 << 24) # REQUESTED-TRANSPORT = UDP
|
||||
msg = struct.pack("!HHI", 0x0003, len(attrs), 0x2112A442) + tid + attrs # Allocate
|
||||
s.sendall(msg)
|
||||
print(s.recv(4096).hex()) # expect 120 bytes starting 0x0113 (Allocate Error = auth challenge)
|
||||
```
|
||||
|
||||
A 120-byte `0x0113`-typed response means coturn is reachable and responding. An indefinite TCP timeout (after `connected from ...` prints) means the IONOS rule was reverted or narrowed.
|
||||
|
||||
**If a peer is still on the WSS relay after this** — verify in `netbird status -d`: `Connection type: P2P` with srflx/srflx is the cone-NAT happy path; `Connection type: P2P` with relay/srflx (or relay/prflx) means the peer's network is symmetric-NAT and it's using TURN. Falling back to `Relayed (rels://...)` only happens if TURN allocate also fails — investigate the IONOS exception first.
|
||||
|
||||
**Old combined-server note (history, kept for context):**
|
||||
|
||||
Pre-migration, the bundled `netbirdio/netbird-server` combined image silently ignored both `server.turns:` and top-level `TURNConfig:` YAML, so TURN was non-functional. That's why the architectural migration to the canonical multi-container stack (`mgmt + signal + relay + coturn`) happened. See the 2026-05-21 changelog entry below and the [Netbird-combined-no-TURN finding](https://github.com/netbirdio/netbird/issues) in homelab memory for the full discovery.
|
||||
|
||||
## Consequence — every LXC wired to Authentik needs an internal override
|
||||
|
||||
Until each LXC is migrated to Netbird, anything that needs to reach `auth.hubris.network` (Authentik), `cloud.hubris.network` (Nextcloud), etc., must override the public answer with `192.168.8.175`.
|
||||
|
||||
Two techniques. Pick by HTTP-client behavior.
|
||||
|
||||
### A) `/etc/hosts` override
|
||||
|
||||
Works for libc `getaddrinfo` clients: curl, wget, most Go/Python/Ruby apps, gitea.
|
||||
|
||||
- Place the line **outside** the `# --- BEGIN PVE ---` / `# --- END PVE ---` markers. Proxmox rewrites everything inside that block on every container start.
|
||||
- Belt-and-suspenders: `/etc/systemd/system/hubris-hosts-override.service` (oneshot, enabled, idempotent).
|
||||
|
||||
### B) Local dnsmasq
|
||||
|
||||
Required for clients that bypass `/etc/hosts`. **Nextcloud (PHP Guzzle + `OC\Http\Client\DnsPinMiddleware`) is one** — uses `dns_get_record()`, not `getaddrinfo`. Likely candidates for the same treatment: Python/Ruby/Java apps that do explicit DNS resolution before connecting.
|
||||
|
||||
Recipe:
|
||||
```
|
||||
apt install dnsmasq
|
||||
|
||||
cat > /etc/dnsmasq.d/hubris-internal.conf <<EOF
|
||||
address=/auth.hubris.network/192.168.8.175
|
||||
server=192.168.8.1
|
||||
server=1.1.1.1
|
||||
interface=lo
|
||||
bind-interfaces
|
||||
no-hosts
|
||||
no-resolv
|
||||
EOF
|
||||
|
||||
# Set LXC default nameservers and live resolv.conf
|
||||
pct set <id> --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"
|
||||
# Then update /etc/resolv.conf inside the LXC too.
|
||||
```
|
||||
|
||||
### Known overrides applied
|
||||
|
||||
| LXC | Technique | Notes |
|
||||
| ------------------------------------------ | ---------------------------------------- | ----- |
|
||||
| [104 (gitea)](../containers/104-gitea.md) | `/etc/hosts` + `hubris-hosts-override.service` | Standard |
|
||||
| [114 (nextcloud)](../containers/114-nextcloud.md) | local dnsmasq @ 127.0.0.1 + resolv.conf reorder | Guzzle bypasses /etc/hosts |
|
||||
| [105 (apps)](../containers/105-apps.md), inner containers | `extra_hosts:` in compose | Booklore, mulita, WriteFreely each ship with this |
|
||||
|
||||
## Adding new LXCs
|
||||
|
||||
- Don't add new LXCs to Tailscale; add them to Netbird. Tailscale is being decommissioned on hubris.
|
||||
- When wiring a new app into Authentik: `cat /etc/resolv.conf` on the target LXC. If nameserver is `192.168.8.1` or `100.100.100.100`, add the hosts override. If it's the netbird daemon IP, skip.
|
||||
|
||||
## Long-term fix
|
||||
|
||||
Either:
|
||||
- Split-horizon DNS at LAN/router level so `*.hubris.network → 192.168.8.175` for every LAN client. Eliminates all per-LXC overrides.
|
||||
- Once Netbird migration completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain on the PVE host.
|
||||
|
||||
## CRITICAL — never `docker compose up` Portainer-managed stacks
|
||||
|
||||
[apps (105)](../containers/105-apps.md) runs multiple stacks deployed via the Portainer UI (`/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`). Running `docker compose up -d <svc>` from the host shell on these triggers a recreate of OTHER services in the stack — labels drift, compose treats the project as reconciling, containers get destroyed and remade. **This wiped Booklore's mariadb data on 2026-04-22** (bind mount `./mariadb/config` re-initialized fresh).
|
||||
|
||||
Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-managed stacks:
|
||||
1. Edit the compose in Portainer UI → **Stacks → <stack> → Editor → Update stack**. Portainer handles the recreate cleanly with its own state tracking.
|
||||
2. Do NOT edit `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/docker-compose.yml` directly.
|
||||
3. For data-preserving DNS overrides on stateful services, prefer Caddy-side handling (forward-auth) or the app's built-in OIDC over container-level `extra_hosts` when possible.
|
||||
|
||||
## Related
|
||||
- [DNS split-horizon](dns.md)
|
||||
- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides
|
||||
- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B
|
||||
- [Gitea (104)](../containers/104-gitea.md) — example of Technique A
|
||||
- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
|
||||
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../../investigations/2026-05-31-authentik-vps-migration.md).
|
||||
|
||||
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
|
||||
|
||||
**Symptom:** `netbird-mgmt` crash-looped for ~9 days (since 2026-05-21 migration). All peers showed `Connecting`, management returned `404 (Not Found)` for gRPC → `EOF` on startup.
|
||||
|
||||
**Root cause:** The 2026-05-21 migration configured `management.json` with `OIDCConfigEndpoint: https://auth.hubris.network/...`. The public DNS for `auth.hubris.network` already pointed to `82.165.190.79` (VPS), but the VPS Traefik had no route for that host → management got EOF on every boot, crash-looped.
|
||||
|
||||
**Fix:**
|
||||
1. Added `auth-hubris` router to `/opt/traefik-dynamic.yaml` — `Host(auth.hubris.network)` with `certResolver: letsencrypt` → service `auth-authentik`.
|
||||
2. Service backend: `https://192.168.8.175` (Caddy on hubris LAN) via the Netbird mesh.
|
||||
3. ServersTransport `skip-verify` with `serverName: auth.hubris.network` + `insecureSkipVerify: true` so Traefik sends the correct SNI to Caddy.
|
||||
4. Upgraded VPS Netbird client `0.69.0 → 0.71.3` via apt (`apt install netbird=0.71.3`).
|
||||
5. Mesh fully recovered; management connected to peers within ~1 min.
|
||||
|
||||
**Bootstrap note (if mesh is dark and management must restart):** If the WireGuard tunnels are fully dead AND management needs to restart (after e.g. a VPS reboot with no peer handshakes), the Traefik backend `192.168.8.175` will be unreachable and management will crash again. Recovery: temporarily open VPS port 22 via IONOS console (`nft insert rule inet hubris-fw input iifname "ens6" tcp dport 22 accept`), SSH in, run `socat TCP-LISTEN:8443,bind=172.30.0.1,reuseaddr,fork TCP:127.0.0.1:8443 &`, then from hubris `ssh -f -N -R 127.0.0.1:8443:192.168.8.175:443 root@82.165.190.79` — this bootstraps one management start, after which the mesh self-heals.
|
||||
|
||||
**Architecture after this change:** `auth.hubris.network` is publicly accessible (HTTPS via VPS Traefik → Caddy on hubris → Authentik LXC 124). External devices authenticating to Netbird now hit this public path. Phase 6 (Authentik as Netbird IdP) is complete and live.
|
||||
|
||||
### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
|
||||
The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
|
||||
|
||||
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
|
||||
|
||||
Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
|
||||
|
||||
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/124-authentik.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
|
||||
|
||||
**Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21):
|
||||
|
||||
Existing peers — registered against the old combined image's embedded Dex IdP at `https://netbird.hubris.network/oauth2` — cache the OLD expected SSH-JWT issuer in the netbird daemon's in-memory state. After the migration, incoming `netbird ssh` connections were rejected with:
|
||||
|
||||
```
|
||||
JWT authentication failed: validate token (
|
||||
expected issuer=https://netbird.hubris.network/oauth2,
|
||||
audiences=[netbird-dashboard netbird-cli],
|
||||
actual issuer=https://auth.hubris.network/application/o/netbird/,
|
||||
audience=netbird-dashboard
|
||||
)
|
||||
```
|
||||
|
||||
Neither `systemctl restart netbird` nor `netbird down && netbird up` clears the cache. Root cause: in `client/internal/engine_ssh.go`, `updateSSH()` bails out with `if e.sshServer != nil { return nil }` whenever the SSH server is already running, so mgmt-pushed JWT config updates are silently ignored. Only a full daemon-process tear-down lets the SSH server re-initialize with the new validator config:
|
||||
|
||||
```
|
||||
sudo systemctl stop netbird
|
||||
sleep 3
|
||||
sudo systemctl start netbird
|
||||
```
|
||||
|
||||
After that, `grep -iE "issuer|audience" /var/log/netbird/client.log | tail` shows the new Authentik issuer. Run this on every existing peer (PVE host + every LXC + every workstation) once after a future IdP swap.
|
||||
|
||||
**Username gotcha (related):** `netbird ssh` defaults the remote username to the LOCAL one (e.g. `dtoro` from the operator's laptop). Hubris + the LXCs only have `root`, so the JWT is accepted but the session immediately fails with `user dtoro not found`. Always use the explicit `root@` prefix when invoking netbird-ssh manually:
|
||||
|
||||
```
|
||||
netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted
|
||||
```
|
||||
|
||||
The `homelab` CLI handles this automatically via the per-host `ssh.user` field in `inventory.yaml` (defaults to `root`; set explicitly only for workstations whose login user isn't `root`).
|
||||
|
||||
Open follow-up: TURN-over-TLS on TCP 5349 (cert via certbot or extract Traefik's acme.json) for hostile-middlebox networks; plain TCP 3478 is sufficient for current usage.
|
||||
|
||||
### 2026-05-10 — ICE direct p2p restored (external STUN swap)
|
||||
All peers were `Connection type: Relayed` because the embedded STUN on the VPS was unreachable from outside (IONOS drops UDP 3478 inbound). Swapped to Google + Cloudflare STUN in `/opt/config.yaml`. After `netbird down/up`, peers now report `P2P` with srflx/host candidates. Backup of pre-change config at `/opt/config.yaml.bak-20260510-185050`. Fixes a real-world Nextcloud upload bottleneck (~366 kB/s through the relay → LAN-direct on same-network peers).
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
### 2026-04-22 — Booklore mariadb data wiped (lesson recorded)
|
||||
The "never docker compose up Portainer-managed stacks" rule comes from this. See [apps (105)](../containers/105-apps.md#changelog).
|
||||
|
||||
### 2026-04-22 — netbird mgmt host joined its own mesh
|
||||
`82.165.190.79` is now a peer (`100.122.165.149`). LAN access + split-horizon DNS via PVE peer. See [VPS hardening](vps-hardening.md).
|
||||
|
||||
### 2026-04-21 — overrides applied to gitea (104) and nextcloud (114)
|
||||
Two techniques documented; Nextcloud forced the dnsmasq route because of Guzzle.
|
||||
60
knowledge/wiki/infrastructure/monitoring.md
Normal file
60
knowledge/wiki/infrastructure/monitoring.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# Monitoring — Hermes health watchdog
|
||||
|
||||
Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy
|
||||
`claudio-monitor` + `claudio-bot` IPC pipeline on 2026-06-04.
|
||||
|
||||
## Current approach
|
||||
|
||||
Two layers:
|
||||
|
||||
1. **On-demand:** ask Hermes "how's the homelab?" or run `homelab health` — loads
|
||||
the `homelab-hardware-health` skill, checks hardware temps, LXC resources,
|
||||
service reachability, and apt/docker drift across all hosts.
|
||||
|
||||
2. **Cron watchdog:** `homelab-health-watchdog` runs every 15 minutes via Hermes
|
||||
cron. Silent when healthy. When thresholds breach, sends an actionable alert
|
||||
to Matrix (`@dtoro:avispero`) with options the user can reply to directly
|
||||
(e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on
|
||||
the selected option via SSH.
|
||||
|
||||
Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C,
|
||||
apt >10/>50 upgradable, services down.
|
||||
|
||||
Home Assistant pulls PVE metrics independently via its Proxmox VE integration
|
||||
(unaffected by this change).
|
||||
|
||||
## Legacy: claudio-monitor (deprecated 2026-06-04)
|
||||
|
||||
The old system was a bash watchdog on hubris (`claudio-monitor.timer`, every 5
|
||||
min) that POSTed alerts to a Matrix bot (`@claudio:avispero`) via an IPC server
|
||||
on LXC 123:9090. All components decommissioned:
|
||||
|
||||
| Component | Fate |
|
||||
|-----------|------|
|
||||
| LXC 123 (claudio-bot) | Destroyed 2026-06-04 |
|
||||
| `dtoro/claudio-bot` | Archived (read-only) on Gitea |
|
||||
| `dtoro/claudio-monitor` | Archived (read-only) on Gitea |
|
||||
| `claudio-monitor.timer` | Disabled on hubris |
|
||||
| `/opt/claudio-monitor/` | Still on hubris (cleanup pending) |
|
||||
| `/etc/claudio-monitor/` | Still on hubris (cleanup pending) |
|
||||
|
||||
For the full deprecation plan, see `plans/2026-06-04_130000-deprecate-claudio-bot.md`.
|
||||
|
||||
## Related pages
|
||||
- [Hubris host](../hosts/hubris.md)
|
||||
- [HAOS VM (108)](../vms/108-haos.md)
|
||||
- [Backups (disabled)](backups.md)
|
||||
- [Homelab context distribution](homelab-context.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-04 — migrated to Hermes health watchdog
|
||||
claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring.
|
||||
On-demand `homelab health` via extended skill; 15-min cron watchdog with actionable
|
||||
Matrix alerts. LXC 123 destroyed, repos archived.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
|
||||
General health monitor with per-LXC checks. MQTT/REST push paths ripped out.
|
||||
88
knowledge/wiki/infrastructure/network.md
Normal file
88
knowledge/wiki/infrastructure/network.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# Network
|
||||
|
||||
Physical and logical network topology for the homelab.
|
||||
|
||||
## Why
|
||||
|
||||
The homelab runs on a dedicated internal subnet (`192.168.8.0/24`) isolated from the main household LAN (`192.168.178.0/24`). Isolation is enforced at Proxmox: LXC/VM traffic is bridged only on the internal `vmbr0` bridge; Proxmox routes packets out to Fritz!Box via `vmbr1`. The main LAN cannot reach homelab services directly without a Fritz!Box static route (which is configured to allow inbound).
|
||||
|
||||
Fritz!OS 8.x does not support second IP networks on LAN ports, so Proxmox (`hubris`) acts as the subnet router rather than the Fritz!Box.
|
||||
|
||||
## Hardware
|
||||
|
||||
| Device | Role |
|
||||
|---|---|
|
||||
| Fritz!Box 7590 | Main router / ISP gateway (`192.168.178.1`) |
|
||||
| SODOLA 5-Port 2.5Gbit Managed | Homelab switch — flat L2, all ports native |
|
||||
| hubris (Proxmox) | Subnet router — routes between `192.168.8.0/24` and `192.168.178.0/24` |
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
ISP
|
||||
└── Fritz!Box 7590 (192.168.178.1)
|
||||
│ static route: 192.168.8.0/24 → 192.168.178.10
|
||||
│
|
||||
└── SODOLA 5-Port 2.5Gbit
|
||||
├── Port 1 uplink → Fritz!Box LAN
|
||||
├── Port 2 hubris eno1 → vmbr1 (192.168.178.10)
|
||||
├── Port 3 [device]
|
||||
├── Port 4 [device]
|
||||
└── Port 5 spare
|
||||
|
||||
hubris internal bridges:
|
||||
vmbr1 192.168.178.10/24 eno1 (uplink, DHCP-reserved) gateway 192.168.178.1
|
||||
vmbr0 192.168.8.77/24 no physical port (internal)
|
||||
192.168.8.1/24 alias — LXC default gateway
|
||||
├── all 16 LXCs
|
||||
└── HAOS VM
|
||||
```
|
||||
|
||||
## Subnets
|
||||
|
||||
| Subnet | Gateway | Purpose |
|
||||
|---|---|---|
|
||||
| `192.168.178.0/24` | `192.168.178.1` | Household LAN — laptops, phones, Fritz!Box DHCP |
|
||||
| `192.168.8.0/24` | `192.168.8.1` (Proxmox `vmbr0` alias) | Homelab — all LXCs and VMs |
|
||||
|
||||
## DHCP
|
||||
|
||||
- **Household (`192.168.178.x`)**: Fritz!Box built-in DHCP. Proxmox `vmbr1` has a reservation: MAC `84:47:09:6b:e7:58` → `192.168.178.10`.
|
||||
- **Homelab (`192.168.8.x`)**: Technitium on [CT 107](../containers/107-dns.md) at `192.168.8.2`. Range `192.168.8.241–192.168.8.254`, gateway `192.168.8.1`, DNS `192.168.8.2`.
|
||||
|
||||
Static IPs span `.101–.239` (all LXCs, VMs, and workstations). DHCP pool narrowed to `.241–.254` (2026-06-03) to avoid overlap and IP conflicts.
|
||||
|
||||
## DNS
|
||||
|
||||
Split-horizon DNS for `*.hubris.network` served by Technitium on [CT 107](../containers/107-dns.md) at `192.168.8.2:53`. See [dns.md](dns.md) for full detail.
|
||||
|
||||
## Routing
|
||||
|
||||
Proxmox has `net.ipv4.ip_forward=1` (already enabled by PVE). Packets from LXCs on `vmbr0` destined for the internet exit via `vmbr1` → Fritz!Box. Fritz!Box masquerades all outbound WAN traffic. Fritz!Box has a static route (`192.168.8.0/24 → 192.168.178.10`) so return traffic reaches the LXCs.
|
||||
|
||||
No NAT on Proxmox — traffic flows without double-NAT.
|
||||
|
||||
## Remote access
|
||||
|
||||
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../../../vps/).
|
||||
- **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md).
|
||||
|
||||
## Related
|
||||
|
||||
- [DNS](dns.md) — split-horizon config and entry list
|
||||
- [Ingress](ingress.md) — public entry points via VPS traefik
|
||||
- [Mesh](mesh.md) — NetBird / Tailscale VPN overlay
|
||||
- [hosts/hubris.md](../hosts/hubris.md) — Proxmox host (vmbr0/vmbr1 config)
|
||||
- [CT 107 — dns](../containers/107-dns.md) — Technitium DNS + DHCP server
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2)
|
||||
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed.
|
||||
Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
|
||||
|
||||
### 2026-06-02 — Executed migration; Proxmox as subnet router
|
||||
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/2026-06-01-slate-ax-to-sodola-migration.md).
|
||||
|
||||
### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added
|
||||
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/2026-06-01-slate-ax-to-sodola-migration.md).
|
||||
206
knowledge/wiki/infrastructure/ssh-access.md
Normal file
206
knowledge/wiki/infrastructure/ssh-access.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# SSH access
|
||||
|
||||
How to reach every host in the fleet from any workstation, with LAN as
|
||||
the primary path and Netbird as the automatic backup.
|
||||
|
||||
## Architecture
|
||||
|
||||
SSH access relies on three layers:
|
||||
|
||||
1. **Homelab inventory (`inventory.yaml`)** — the single source of truth
|
||||
for every host's LAN IP, Netbird addresses, SSH user, and port.
|
||||
2. **Key distribution (`ssh/deploy-keys.sh`)** — deploys workstation SSH
|
||||
public keys to hubris and every running LXC, so any key-authorized
|
||||
workstation can log in anywhere.
|
||||
3. **Config generation (`homelab ssh-config --install`)** — generates
|
||||
`~/.ssh/config.d/homelab` with short hostname aliases for every host,
|
||||
using LAN IPs (routed via Netbird's `192.168.8.0/24` subnet route when
|
||||
off-LAN) with Netbird FQDN fallbacks (`<name>-mesh`) for roaming
|
||||
workstations.
|
||||
|
||||
### How it works
|
||||
|
||||
- **From on-LAN:** `ssh gitea` resolves to `192.168.8.121` directly.
|
||||
- **From off-LAN (Netbird):** The same `192.168.8.121` works because
|
||||
hubris routes the `192.168.8.0/24` subnet through Netbird.
|
||||
- **Roaming workstations:** `ssh mac-mini-mesh` or `ssh republic-laptop-mesh`
|
||||
uses the Netbird FQDN as a fallback when the workstation is off its
|
||||
home subnet.
|
||||
|
||||
The `homelab ssh <host>` CLI command also has built-in LAN probing:
|
||||
it tries a 1.5s TCP connect to the LAN IP, and if that fails, falls
|
||||
back to the Netbird FQDN.
|
||||
|
||||
## Key distribution
|
||||
|
||||
Each workstation's SSH public key lives in the repo at:
|
||||
`ssh/authorized_keys/<hostname>.pub`
|
||||
|
||||
To deploy or re-deploy all workstation keys to hubris + every running LXC:
|
||||
|
||||
```bash
|
||||
# From hubris (or via homelab pct):
|
||||
sudo bash /opt/homelab-context/ssh/deploy-keys.sh
|
||||
|
||||
# Or from any workstation:
|
||||
ssh root@192.168.8.77 "bash /opt/homelab-context/ssh/deploy-keys.sh"
|
||||
```
|
||||
|
||||
This script:
|
||||
- Reads all `.pub` files from `ssh/authorized_keys/`
|
||||
- Adds any missing keys to `/etc/pve/priv/authorized_keys` on hubris
|
||||
- For each running LXC, appends keys to `/root/.ssh/authorized_keys`
|
||||
- Is idempotent — skips keys already present
|
||||
|
||||
## Config generation
|
||||
|
||||
To generate the SSH config on any workstation:
|
||||
|
||||
```bash
|
||||
homelab ssh-config --install
|
||||
```
|
||||
|
||||
This writes to `~/.ssh/config.d/homelab` and ensures
|
||||
`Include ~/.ssh/config.d/homelab` is present in `~/.ssh/config`.
|
||||
|
||||
The config is regenerated automatically on every `homelab sync` (which
|
||||
kicks the 5-minute context sync timer).
|
||||
|
||||
## Adding a new workstation
|
||||
|
||||
When onboarding a new machine:
|
||||
|
||||
1. Hostname must match an entry in `inventory.yaml`.
|
||||
2. If the workstation will be on the LAN, add its `lan_ip` to
|
||||
`inventory.yaml` and push. This gives it a primary LAN entry in the
|
||||
generated SSH config.
|
||||
3. Enable SSH Remote Login:
|
||||
- **macOS:** `sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist`
|
||||
- **Linux:** `sudo systemctl enable --now sshd`
|
||||
4. Generate an SSH keypair if one doesn't exist:
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -a 100
|
||||
```
|
||||
5. Publish the public key to the repo:
|
||||
```bash
|
||||
cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/<hostname>.pub
|
||||
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add <hostname> pubkey' && git push
|
||||
```
|
||||
6. Deploy the key to all hosts:
|
||||
```bash
|
||||
ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
|
||||
```
|
||||
7. Generate the local SSH config:
|
||||
```bash
|
||||
homelab ssh-config --install
|
||||
```
|
||||
|
||||
## Hosts
|
||||
|
||||
### Hubris + strong (PVE cluster: `Homelab`)
|
||||
|
||||
Both nodes share `/etc/pve/priv/authorized_keys` — it's Proxmox
|
||||
cluster-synced, so a key added on either node is authorized on both.
|
||||
|
||||
| Detail | hubris | strong |
|
||||
|--------|--------|-----------|
|
||||
| LAN IP | `192.168.8.77` | `192.168.178.181` |
|
||||
| Cluster node name | `hubris` | `strong` (OS hostname kept as-is from install) |
|
||||
| Netbird | `100.122.38.109` (`proxmox-server.netbird.selfhosted`) | not enrolled yet |
|
||||
| Netbird SSH port | `22022` (mesh-only, OIDC auth) | n/a |
|
||||
| SSH user | `root` | `root` |
|
||||
|
||||
Authorized root keys currently deployed (cluster-wide):
|
||||
- `root@hubris` (self, RSA)
|
||||
- `d.toro.v@pm.me` (ed25519) — mac-mini
|
||||
- `root@strong` (RSA) — strong's own key, added 2026-07-01 for the cluster join
|
||||
|
||||
### LXCs
|
||||
|
||||
Every LXC at `192.168.8.x` accepts root SSH via authorized_keys. Keys
|
||||
are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
|
||||
|
||||
| LXC | Name | LAN IP | Role |
|
||||
|-----|------|--------|------|
|
||||
| 101 | jellyfin | `192.168.8.206` | media-server |
|
||||
| 102 | nfs-export | `192.168.8.200` | storage-export |
|
||||
| 103 | paperless | `192.168.8.130` | document-archive |
|
||||
| 104 | gitea | `192.168.8.121` | git-server |
|
||||
| 105 | apps | `192.168.8.205` | docker-apps |
|
||||
| 106 | auth-outpost | `192.168.8.184` | authentik-outpost |
|
||||
| 107 | dns | `192.168.8.185` | dns-helper |
|
||||
| 114 | nextcloud | `192.168.8.224` | file-sync |
|
||||
| 118 | elementsynapse | `192.168.8.239` | matrix-server |
|
||||
| 119 | sophia | `192.168.8.157` | workshop |
|
||||
| 120 | mule-images | `192.168.8.136` | photo-management |
|
||||
| 121 | caddy | `192.168.8.175` | reverse-proxy |
|
||||
| 122 | arriman | `192.168.8.132` | arr-stack |
|
||||
|
||||
### Workstations
|
||||
|
||||
| Name | OS | LAN IP | Netbird FQDN | SSH user |
|
||||
|------|----|--------|--------------|----------|
|
||||
| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
|
||||
| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
|
||||
|
||||
strong moved out of this table 2026-07-01 — it's a Proxmox host now, see the cluster table above.
|
||||
|
||||
### VPS (external)
|
||||
|
||||
| Detail | Value |
|
||||
|--------|-------|
|
||||
| Public IP | `82.165.190.79` |
|
||||
| Netbird | `100.122.165.149` (FQDN: `netbird-ionos.netbird.selfhosted`) |
|
||||
| SSH user | `root` |
|
||||
| Access | Mesh-only — public port 22 is blocked by nftables. Key-only auth. |
|
||||
|
||||
## VPS
|
||||
|
||||
Access is mesh-only. From a mesh-connected peer:
|
||||
|
||||
```bash
|
||||
ssh root@100.122.165.149
|
||||
ssh root@netbird-ionos.netbird.selfhosted
|
||||
# or via homelab:
|
||||
homelab ssh netbird-vps
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# From any workstation after running homelab ssh-config --install:
|
||||
for name in hubris gitea apps sophia paperless caddy jellyfin nextcloud; do
|
||||
ssh -o BatchMode=yes "$name" "hostname" && echo "$name OK"
|
||||
done
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [Mesh migration](mesh.md)
|
||||
- [VPS hardening](vps-hardening.md)
|
||||
- [Agent enrollment](../../../operations/agent-enrollment.md)
|
||||
- [Homelab CLI](../../../bin/homelab)
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-07-01 — strong reformatted to Proxmox, joined cluster; table corrected
|
||||
strong moved from the Workstations table to the PVE-cluster table (was showing a stale `192.168.8.133`, never actually reachable — the real LAN IP has always been `192.168.178.181`, matching hosts/strong.yaml). Root key access bootstrapped via one-time console password, then key-only going forward. See [hosts/hubris.md#cluster](../hosts/hubris.md#cluster) and [hosts/strong.md](../hosts/strong.md).
|
||||
|
||||
### 2026-06-02 — universal SSH reachability
|
||||
|
||||
Replaced ad-hoc per-workstation SSH configs with inventory-generated
|
||||
configs (`ssh/gen-config.py`, `homelab ssh-config`). Added centralized
|
||||
key distribution (`ssh/deploy-keys.sh`, `ssh/authorized_keys/`). All
|
||||
LXCs now accept root SSH from any workstation whose pubkey is in the
|
||||
repo. mac-mini Remote Login enabled. Netbird subnet route
|
||||
(192.168.8.0/24 via hubris) provides off-LAN reachability for all LAN
|
||||
IPs.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
### 2026-04-23 — VPS SSH hardened to mesh-only
|
||||
Public `:22` blocked at nftables. Key-only sshd.
|
||||
|
||||
### 2026-04-22 — iMac key authorized on hubris
|
||||
`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`.
|
||||
96
knowledge/wiki/infrastructure/topology.md
Normal file
96
knowledge/wiki/infrastructure/topology.md
Normal file
@@ -0,0 +1,96 @@
|
||||
<!-- Generated by oikos/gen-topology.py from inventory.yaml. -->
|
||||
<!-- Do NOT edit by hand - your changes will be overwritten. -->
|
||||
|
||||
# Topology (generated)
|
||||
|
||||
Source: [inventory.yaml](../../../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services.
|
||||
Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). Operating model: [OIKOS.md](../../../.agents/OIKOS.md).
|
||||
|
||||
## Compute & ingress
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph hubris_sub["hubris (Proxmox)"]
|
||||
trmnl["trmnl<br/>LXC 128<br/>trmnl-middleware<br/>192.168.8.211"]
|
||||
nfs_export["nfs-export<br/>LXC 102<br/>storage-export<br/>192.168.8.200"]
|
||||
paperless["paperless<br/>LXC 103<br/>document-archive<br/>192.168.8.130"]
|
||||
gitea["gitea<br/>LXC 104<br/>git-server<br/>192.168.8.121"]
|
||||
apps["apps<br/>LXC 105<br/>docker-apps<br/>192.168.8.205"]
|
||||
auth_outpost["auth-outpost<br/>LXC 106<br/>authentik-gateway<br/>192.168.8.6"]
|
||||
dns["dns<br/>LXC 107<br/>dns-server<br/>192.168.8.2"]
|
||||
nextcloud["nextcloud<br/>LXC 114<br/>file-sync<br/>192.168.8.224"]
|
||||
sophia["sophia<br/>LXC 119<br/>workshop<br/>192.168.8.109"]
|
||||
mule_images["mule-images<br/>LXC 120<br/>photo-management<br/>192.168.8.136"]
|
||||
caddy["caddy<br/>LXC 121<br/>reverse-proxy<br/>192.168.8.175"]
|
||||
teddycloud["teddycloud<br/>LXC 131<br/>teddycloud<br/>192.168.8.150"]
|
||||
zimaos["zimaos<br/>VM 100<br/>nas-frontend-eval<br/>192.168.8.195"]
|
||||
haos["haos<br/>VM 108<br/>home-automation<br/>192.168.8.101"]
|
||||
end
|
||||
subgraph strong_sub["strong (Proxmox)"]
|
||||
house["house<br/>LXC 129<br/>family-planner<br/>192.168.8.244"]
|
||||
jellyfin["jellyfin<br/>LXC 101<br/>media-server<br/>192.168.8.246"]
|
||||
elementsynapse["elementsynapse<br/>LXC 118<br/>matrix-server<br/>192.168.8.242"]
|
||||
arriman["arriman<br/>LXC 122<br/>arr-stack<br/>192.168.8.245"]
|
||||
grimmory["grimmory<br/>LXC 130<br/>book-library<br/>192.168.8.247"]
|
||||
seanime["seanime<br/>LXC 133<br/>anime-media-server<br/>192.168.8.248"]
|
||||
romm["romm<br/>LXC 134<br/>rom-manager<br/>192.168.8.249"]
|
||||
end
|
||||
rclone["rclone<br/>lxc<br/>backup"]
|
||||
republic_laptop([republic-laptop<br/>workstation<br/>primary-dev])
|
||||
mac_mini([mac-mini<br/>workstation<br/>dev<br/>192.168.178.182])
|
||||
netbird_vps[[netbird-vps<br/>external<br/>netbird-mgmt]]
|
||||
url_artifacto(["artifacto.hubris.network"]) -->|routes-to| apps
|
||||
url_authentik(["auth.hubris.network"]) -->|routes-to| netbird_vps
|
||||
url_gitea(["git.hubris.network"]) -->|routes-to| gitea
|
||||
url_homelab_mcp(["mcp.hubris.network"]) -->|routes-to| apps
|
||||
url_jellyfin(["media.hubris.network"]) -->|routes-to| jellyfin
|
||||
url_matrix(["matrix.hubris.network"]) -->|routes-to| elementsynapse
|
||||
url_nextcloud(["cloud.hubris.network"]) -->|routes-to| nextcloud
|
||||
url_paperless(["paperless.hubris.network"]) -->|routes-to| paperless
|
||||
url_photos(["photos.hubris.network"]) -->|routes-to| mule_images
|
||||
url_proxmox_ui(["proxmox.hubris.network"]) -->|routes-to| hubris_sub
|
||||
url_secrets_issuance(["secrets.hubris.network"]) -->|routes-to| apps
|
||||
url_teddycloud(["teddy.hubris.network"]) -->|routes-to| teddycloud
|
||||
url_trmnl(["trmnl.hubris.network"]) -->|routes-to| trmnl
|
||||
url_zimaos(["zimaos.hubris.network"]) -->|routes-to| zimaos
|
||||
```
|
||||
|
||||
## Storage (mounts)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
mnt_library[("/mnt/library")]
|
||||
mnt_media_local[("/mnt/media_local")]
|
||||
mnt_media_local_anime[("/mnt/media_local/anime")]
|
||||
apps["apps"] -->|mounts| mnt_library
|
||||
arriman["arriman"] -->|mounts| mnt_media_local
|
||||
gitea["gitea"] -->|mounts| mnt_library
|
||||
grimmory["grimmory"] -->|mounts| mnt_media_local
|
||||
hubris["hubris"] -->|mounts| mnt_library
|
||||
jellyfin["jellyfin"] -->|mounts| mnt_media_local
|
||||
mule_images["mule-images"] -->|mounts| mnt_library
|
||||
nextcloud["nextcloud"] -->|mounts| mnt_library
|
||||
paperless["paperless"] -->|mounts| mnt_library
|
||||
romm["romm"] -->|mounts| mnt_media_local
|
||||
seanime["seanime"] -->|mounts| mnt_media_local_anime
|
||||
sophia["sophia"] -->|mounts| mnt_library
|
||||
teddycloud["teddycloud"] -->|mounts| mnt_library
|
||||
```
|
||||
|
||||
## Archaeology (destroyed nodes)
|
||||
|
||||
| Node | ID | Destroyed | Reason |
|
||||
|---|---|---|---|
|
||||
| plato | 126 | 2026-06-28 | notes workspace decommissioned; data retained at /mnt/library/documents/plato |
|
||||
| claudio-bot | 123 | 2026-06-04 | replaced by Hermes Agent on mac-mini; monitoring moved to homelab-health-watchdog cron |
|
||||
| mule-photos-new | 127 | 2026-05-22 | PhotoPrism test stack promoted to LXC 120 (Mulimage 2.0 merge) |
|
||||
| heaper | 116 | 2026-05-14 | decommissioned; data retained at /mnt/library/heaper |
|
||||
| syncthing | 109 | 2026-05-14 | decommissioned; library subtree was empty |
|
||||
| seafile | 125 | 2026-05-13 | Seafile Pro evaluation rejected; files.hubris.network removed from caddy + dns |
|
||||
| arr-yunohost | 100 | 2026-04-28 | migrated to docker stack on arriman (LXC 122) |
|
||||
| flaresolverr | 106 | 2026-04-28 | folded into the arriman docker compose |
|
||||
| marimo | 107 | 2026-04-28 | decommissioned |
|
||||
| photoprism | 110 | 2026-04-28 | replaced by mule-images (LXC 120) |
|
||||
| karakeep | 111 | 2026-04-28 | decommissioned |
|
||||
| immich | 112 | 2026-04-28 | replaced by mule-images (LXC 120) |
|
||||
| reticulum | 115 | 2026-04-28 | decommissioned |
|
||||
91
knowledge/wiki/infrastructure/vps-hardening.md
Normal file
91
knowledge/wiki/infrastructure/vps-hardening.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# VPS hardening — `82.165.190.79` / `100.122.165.149`
|
||||
|
||||
IONOS VPS that runs the Netbird control plane and the [public ingress traefik](ingress.md). Hardened 2026-04-23 from its stock-Plesk state.
|
||||
|
||||
## At a glance
|
||||
- **Hostname:** `inspiring-ramanujan.82-165-190-79.plesk.page`
|
||||
- **OS:** Debian 13
|
||||
- **Mesh:** netbird `100.122.165.149` (peer of the lab mesh; routes `192.168.8.0/24` via [hubris](../hosts/hubris.md)).
|
||||
- **Public:** `82.165.190.79` (`ens6`).
|
||||
- **Public DNS:** IONOS wildcard `*.hubris.network → 82.165.190.79`.
|
||||
- **Docker stack** at `/opt/docker-compose.yml`: `traefik` (TLS/ACME) + `dashboard` + `mgmt` + `signal` + `relay` + `proxy` — netbird-mgmt 0.71.3 vanilla deploy since 2026-05-21 (see [mesh.md changelog](mesh.md#changelog)).
|
||||
- **Host services (outside docker):** `coturn` (TURN-TCP on :3478, long-term creds rendered into `/etc/turnserver.conf` by `homelab render-vps-configs` from sops-encrypted `secrets/turn-shared-secret.yaml`).
|
||||
- **Config rendering:** `/etc/turnserver.conf` + `/opt/management.json` are generated from templates in `vps/*.tmpl` on this repo by `homelab render-vps-configs`. Secret placeholders (`{{TURN_PASSWORD}}`, `{{AUTHENTIK_CLIENT_SECRET}}`) are substituted from sops-encrypted secrets decrypted on hubris and pushed over ssh. **Do not hand-edit those two files on the VPS** — the next render will overwrite them.
|
||||
|
||||
## SSH
|
||||
|
||||
- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`.
|
||||
- **Mesh-only**: public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`. `ListenAddress` itself is still `0.0.0.0` — gating is firewall-layer.
|
||||
- Authorized root keys: PVE (`root@hubris`), Mac Mini (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer **before** disabling its access paths.
|
||||
|
||||
## Firewall — nftables (`inet hubris-fw`)
|
||||
|
||||
Config at `/etc/nftables.conf`, service enabled.
|
||||
|
||||
- Public iface `ens6`. Wireguard iface `wt0`.
|
||||
- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6, **TCP 3478** (coturn TURN-TCP, added 2026-05-21). Everything else drops.
|
||||
- `wt0` fully accepted in INPUT. `lo` accepted.
|
||||
- **IONOS upstream firewall** also gates inbound traffic before it reaches `ens6`. Open ports today: TCP 80/443 (traefik), UDP 51820 (netbird-proxy), TCP 3478 (coturn, added 2026-05-21). UDP 3478 is dropped by IONOS upstream regardless of local nftables. See [mesh.md ICE/STUN](mesh.md) for the STUN/TURN port matrix.
|
||||
- **FORWARD chain at priority `filter-10`** (runs before Docker's FORWARD) hosts the fail2ban ban enforcement — see below.
|
||||
- Set `banned4` (typed `ipv4_addr`, flag `timeout`) holds fail2ban's drops.
|
||||
- Coexists with Docker's `ip nat` / `ip filter` tables (iptables-nft compat). **Do NOT `flush ruleset`** in this config — it'll wipe Docker's state too.
|
||||
|
||||
## fail2ban
|
||||
|
||||
- **Jail `traefik-4xx`** tails `/var/log/traefik/access.log` (bind-mounted from container). Filter at `/etc/fail2ban/filter.d/traefik-4xx.conf` matches 401/403/404/429 from `blog-public@file` or `artifacto-public@file` routers only — netbird-grpc traffic isn't considered.
|
||||
- Tunables: `findtime=600, maxretry=30, bantime=3600`.
|
||||
- **Action** at `/etc/fail2ban/action.d/nft-hubris.conf` adds/removes elements from `inet hubris-fw banned4` with per-element timeout.
|
||||
|
||||
### CRITICAL invariant — wireguard / fail2ban
|
||||
|
||||
**Bans must never affect `wt0` or wireguard UDP.** The FORWARD chain explicitly `accept`s the following *before* the ban check:
|
||||
- `udp 51820` (wireguard)
|
||||
- `udp 3478` (STUN)
|
||||
- `ct state established,related`
|
||||
|
||||
The INPUT ban rule is scoped to `iifname "ens6"`.
|
||||
|
||||
Violating this takes the mesh down for every home device (they share one public IP) and the only recovery is IONOS console → `nft flush set inet hubris-fw banned4`.
|
||||
|
||||
## Traefik access log
|
||||
|
||||
- Written to `/var/log/traefik/access.log` on the host via a bind mount added to `/opt/docker-compose.yml` (traefik volumes include `/var/log/traefik:/logs`) plus `--accesslog.filepath=/logs/access.log`.
|
||||
- CLF format. Real client IP arrives correctly because docker userland-proxy is off — see [public ingress](ingress.md).
|
||||
|
||||
## Plesk / mail / FTP / Dr.Web
|
||||
|
||||
Stopped and disabled (not uninstalled). All of:
|
||||
`dovecot`, `dovecot.socket`, `postfix`, `postfix@-`, `pc-remote`, `xinetd`, `plesk-task-manager`, `plesk-web-socket`, `sw-cp-server`, `sw-engine`, `plesk-repaird`, `plesk-repaird.socket`, `drwebd`.
|
||||
|
||||
`psa.service` is masked (was a one-shot boot bootstrap). `/etc/cron.d/plesk-backup-manager-task` renamed to `.disabled`.
|
||||
|
||||
Reverse: `systemctl unmask psa; systemctl enable --now <svc>`.
|
||||
|
||||
## Auto-patching
|
||||
|
||||
- `unattended-upgrades` enabled (stock).
|
||||
- Drop-in at `/etc/apt/apt.conf.d/52hubris-reboot.conf` sets auto-reboot at **04:00 UTC** when `/var/run/reboot-required` is set.
|
||||
- Runs inside the stock `apt-daily-upgrade.timer`.
|
||||
|
||||
## Recovery paths
|
||||
|
||||
Ordered by preference:
|
||||
1. **SSH via mesh** — primary. Any mesh peer with an authorized key.
|
||||
2. **IONOS web console** (my.ionos.com → VPS → Console) — uses the system password, not SSH keys. Bypasses any firewall misconfig.
|
||||
3. **IONOS rescue mode** — boot rescue, mount rootfs, edit `/etc/nftables.conf` or `/etc/ssh/sshd_config.d/10-hubris-hardening.conf` to a known-good state, reboot.
|
||||
|
||||
## Related
|
||||
- [Public ingress (VPS traefik)](ingress.md)
|
||||
- [Mesh migration](mesh.md) — VPS as a mesh peer
|
||||
- [SSH access](ssh-access.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-05-21 — netbird stack migrated combined → vanilla; coturn added
|
||||
Replaced the `netbirdio/netbird-server` combined image with the canonical `mgmt + signal + relay + dashboard` containers (0.71.3). Added host-side `coturn` for external TURN, with nftables rule `iifname "ens6" tcp dport 3478 accept` and an IONOS upstream firewall exception. Authentik on LXC 124 now provides OIDC for the netbird dashboard. Full context in [mesh.md changelog](mesh.md#changelog).
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
### 2026-04-23 — hardened
|
||||
nftables firewall, mesh-only SSH, fail2ban traefik jail, Plesk disabled, auto-reboot 04:00 UTC, wireguard/fail2ban invariant established.
|
||||
Reference in New Issue
Block a user