dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Bootstrap Homelab-Docs wiki

Browse Source 
Initial documentation of the hubris Proxmox homelab as a cross-linked
markdown wiki. Per-node pages, cross-cutting infrastructure pages, an
investigation log, and an operations cheatsheet. Each node and topic
ends with a Changelog section so changes can be tracked in-place going
forward.

Refreshed against live state on 2026-04-28 — 14 active LXCs (109
syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test
state of the 2026-04-21 hubris crash-loop investigation.
This commit is contained in:
dtoro
2026-04-28 22:53:09 +02:00
commit f5cd320433
31 changed files with 2157 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
infrastructure/auto-deploy.md Normal file
View File

@@ -0,0 +1,93 @@
# Auto-deploy — gitea-webhook pipelines
Several configs and apps in the lab live in `dtoro/*` repos on [gitea (104)](../containers/104-gitea.md) and auto-redeploy on push. All pipelines follow one of two shapes.
## Two shapes
### Shape A — checkout IS the working tree (config repos)
`/etc/<thing>` or `/var/lib/<thing>/...` is itself a `git clone`. Push triggers `git pull` + a reload command. Used for pure-config repos where re-cloning is cheap.
### Shape B — receiver outside the app repo (compose stacks)
The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`webhook.py`, `deploy.sh`, systemd unit) lives in a sibling `/opt/<thing>-deploy/` so the app repo stays portable. Push triggers `git pull` + `docker compose up -d --build`. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout.
## Common
- All receivers validate `X-Gitea-Signature` HMAC-SHA256 against a per-pipeline secret in `/etc/<thing>-deploy/secret`.
- All filter to `refs/heads/main` (or `master` for older repos). Gitea's "test delivery" button sends `ref=main` (without `refs/heads/`) — those will log "ignoring ref main" and 204. Real pushes work. **Don't "fix" the ref filter to accept both** — it'd also accept PR merges from side branches that got fast-forwarded.
- Gitea's `app.ini` `[webhook] ALLOWED_HOST_LIST` must include every receiver IP. Currently:
- `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
- `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
- `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
- `192.168.8.230` ([claudio-bot (123)](../containers/123-claudio-bot.md))
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
**Don't strip these when editing app.ini.**
- Git creds for root-run deploy services live in `/etc/<thing>-deploy/git-credentials` (mode 600) and are wired via `credential.helper = store --file=/etc/<thing>-deploy/git-credentials` in the repo's `.git/config`. Necessary because the unit typically runs with `ProtectHome=true`, which blocks `/root`.
## Pipelines
| Repo | Target | Shape | Receiver | Webhook id | Reload action |
| ------------------------------- | -------------------------------------------- | ----- | ------------------------------------- | ---------- | ------------- |
| `dtoro/caddy-conf` | [caddy (121)](../containers/121-caddy.md) `/etc/caddy/` | A | `http://192.168.8.175:9797/deploy` | 2 | `caddy validate` + `systemctl reload caddy` |
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
| `dtoro/claudio-bot` | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` |
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern.
## When you change a tracked config
Always commit + push. Local-only edits drift. Common ones:
- `/etc/caddy/Caddyfile` ↔ `dtoro/caddy-conf` (auto-deploys)
- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys)
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
- `/opt/claudio-bot/` ↔ `dtoro/claudio-bot` (auto-deploys)
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
## Per-pipeline notes / gotchas
### caddy-conf
- Repo includes `scripts/webhook/install.sh`. Editing the systemd unit *inside the repo* does **not** auto-reinstall — re-run `install.sh` manually after unit edits.
- The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`).
### gitea-customizations
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
### mule-image / Artifacto
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
- Manual deploy: `pct exec <id> -- /opt/<thing>-deploy/deploy.sh`.
- Health: `pct exec <id> -- curl -s http://127.0.0.1:<port>/health` → `ok`.
- Setup tokens used to register the webhook (e.g., `artifacto-deploy-setup`, `artifacto-deploy-setup-2`, `artifacto-cleanup` on user `dtoro`) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's `/users/{u}/tokens` endpoints require basic auth (not bearer), so cleanup couldn't be automated.
### backup-library
- Currently the only deploy that targets the host directly (`192.168.8.77:9798`).
- `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run.
## Related
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
- [Backups (disabled)](backups.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation. Six active pipelines.
### 2026-04-22 — Artifacto pipeline added
Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
### 2026-04-21 — mule-image + claudio-bot pipelines added
Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
### 2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped
Initial three. Set the conventions everything else follows.

115
infrastructure/backups.md Normal file
View File

@@ -0,0 +1,115 @@
# Backups — restic on external drive (DISABLED)
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md).
## Status
**DISABLED 2026-04-22.** All four timers `systemctl disable --now`'d:
- `backup-library@homecloud.timer`
- `backup-library@images.timer`
- `backup-library@small.timer`
- `backup-library-check.timer`
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
## Design
Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs `/mnt/library`) into thermal danger.
Retention per tag: `--keep-last 3 --keep-monthly 12 --keep-yearly 3` with `--group-by host,tags,paths`.
## Components
- **Repo:** `dtoro/backup-library`
- **Checkout:** `/opt/backup-library` on the [hubris host](../hosts/hubris.md)
- **Auto-deploys** via gitea webhook → `http://192.168.8.77:9798/deploy`. See [auto-deploy](auto-deploy.md). [Gitea (104)](../containers/104-gitea.md) `app.ini` `ALLOWED_HOST_LIST` includes `192.168.8.77` for this.
- **Restic repo:** `/mnt/backup/restic-library`. Passphrase `/etc/restic/passphrase` (mode 600). **Escrow in password manager — loss = permanent data loss.**
- **External drive:** `/dev/sda1` ext4 label `backup-library` UUID `ff46e775-1ba1-4892-82c9-e5cac5be933a`. Fstab uses `noauto` + `nofail,x-systemd.device-timeout=10s,errors=remount-ro`.
## Mount-on-demand
`/usr/local/sbin/backup-usb.sh attach|detach|status`. All three backup units (`backup-library.service`, `backup-library@.service`, `backup-library-check.service`) have `ExecStartPre=backup-usb.sh attach` and `ExecStopPost=backup-usb.sh detach`. The helper toggles `/sys/bus/usb/devices/*/authorized` by matching vendor:product `090c:2320`, then mounts/unmounts `/mnt/backup`. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.
## UAS blacklist
`/etc/modprobe.d/usb-storage-quirks.conf`:
```
options usb-storage quirks=090c:2320:u
```
Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with `dmesg | grep "UAS is ignored"`.
## Schedule
Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:
| Timer | When | Include list | Approx size |
| ---------------------------------- | -------------- | ------------------------------------ | ----------- |
| `backup-library@homecloud.timer` | day 1 / month | `/etc/restic/include-homecloud.list` | ~315 G |
| `backup-library@images.timer` | day 10 / month | `/etc/restic/include-images.list` | ~103 G |
| `backup-library@small.timer` | day 20 / month | `/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper) | ~11 G |
Snapshots tagged `chunk-<group>` so forget/prune treats each series independently.
Ad-hoc full run (kept for manual use): `systemctl start backup-library.service` (no arg → uses `/etc/restic/include.list`, tag `monthly`).
Yearly integrity: `backup-library-check.timer` (`OnCalendar=yearly`) runs full `restic check --read-data`.
## Thermal caps
Baked into the systemd units:
- `IOReadBandwidthMax=/mnt/library 50M`
- `IOWriteBandwidthMax=/mnt/backup 30M`
- `--read-concurrency=1` on restic.
## Wrapper
`/usr/local/sbin/backup-library.sh` — preflight → unlock → backup → forget/prune (`--group-by host,tags,paths`) → `check --read-data-subset=5%` → notify. Takes optional `<group>` arg or `GROUP=` env.
## Notifications
POST to [claudio-bot (123)](../containers/123-claudio-bot.md) `http://192.168.8.230:9090/notify` with header `X-Bot-Token: $(cat /etc/restic/bot.token)`. Token in `/etc/restic/bot.token` is the same value as the bot's `ipc.token` — rotate together.
`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
## Recovery
Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/RECOVERY.md`). Covers `restic snapshots/ls/find/restore/mount`, uid/gid gotcha, cross-host recovery.
## Known SPOF
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented.
## Drive history
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
## Thermal monitoring
Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213`). See [monitoring](monitoring.md).
## Related
- [Hubris host](../hosts/hubris.md)
- [claudio-bot (123)](../containers/123-claudio-bot.md)
- [Monitoring](monitoring.md)
- [Auto-deploy](auto-deploy.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation. Status remains DISABLED.
### 2026-04-22 — DISABLED
Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
### 2026-04-20 — deployed; redesigned for thermal-gentleness
Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.

86
infrastructure/dns.md Normal file
View File

@@ -0,0 +1,86 @@
# DNS — split-horizon `*.hubris.network`
LAN clients resolve `*.hubris.network` to the [Caddy reverse proxy](../containers/121-caddy.md) (`192.168.8.175`). Public clients resolve to the IONOS VPS (`82.165.190.79`) via an IONOS wildcard, where they hit the [VPS traefik public ingress](ingress.md).
There is **no wildcard on the LAN side**. Every subdomain needs an explicit entry.
## Components
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
- **LAN authoritative for `hubris.network` records:** dnsmasq on [authentik (124)](../containers/124-authentik.md), `192.168.8.180:53` and `127.0.0.1:53`. Config at `/etc/dnsmasq.d/hubris-split.conf`. Forwards everything else to `1.1.1.1` and `8.8.8.8` (`no-resolv`, `server=...`).
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
## Live entries (as of 2026-04-28)
```
address=/auth.hubris.network/192.168.8.175
address=/git.hubris.network/192.168.8.175
address=/media.hubris.network/192.168.8.175
address=/paperless.hubris.network/192.168.8.175
address=/books.hubris.network/192.168.8.175
address=/home.hubris.network/192.168.8.175
address=/cloud.hubris.network/192.168.8.175
address=/matrix.hubris.network/192.168.8.175
address=/proxmox.hubris.network/192.168.8.175
address=/docker.hubris.network/192.168.8.175
address=/jellyseerr.hubris.network/192.168.8.175
address=/qbit.hubris.network/192.168.8.175
address=/sab.hubris.network/192.168.8.175
address=/blog.hubris.network/192.168.8.175
address=/photos.hubris.network/192.168.8.175
address=/artifacto.hubris.network/192.168.8.175
```
## Why split-horizon
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy.
## The gotcha that cost a debug session (2026-04-22)
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the LXC-124 dnsmasq entry, LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
## Recipe — adding a new subdomain
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
2. Append to `/etc/dnsmasq.d/hubris-split.conf` on [authentik (124)](../containers/124-authentik.md):
```
address=/<new>.hubris.network/192.168.8.175
```
3. `pct exec 124 -- systemctl restart dnsmasq` — reload/SIGHUP may not pick up the new `address=` line; hard restart is the reliable path.
4. Verify: `dig @192.168.8.180 +short <new>.hubris.network` → `192.168.8.175`.
5. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
> The dnsmasq config is **not** tracked in git. If you stand up a `dtoro/dnsmasq-conf` repo, wire it like `caddy-conf` (webhook on LXC 124).
## Public path — what does and doesn't follow the LAN map
- Hostnames published in [VPS traefik dynamic config](ingress.md) (currently `artifacto.hubris.network`, `blog.hubris.network`) reach a real backend over the netbird mesh.
- Anything else with a `*.hubris.network` URL hits the VPS but isn't routed anywhere — returns 404.
- `netbird.hubris.network` is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.
## Long-term plan
Either:
- Move split-horizon DNS to the LAN router so `*.hubris.network → 192.168.8.175` is answered for every LAN client. Eliminates per-LXC overrides.
- Or, once the [Tailscale → Netbird migration](mesh.md) completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.
## Related
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
- [Authentik (124)](../containers/124-authentik.md) — host of the split-horizon dnsmasq
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation. 16 active entries.
### 2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79
Public path now lands on the VPS traefik, not the old yunohost. Necessary for the [public ingress](ingress.md) pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.
### 2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)
Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.
### 2026-04-21 — dnsmasq stood up on LXC 124
Co-located with Authentik. Initial entries cover everything routed through Caddy.

84
infrastructure/ingress.md Normal file
View File

@@ -0,0 +1,84 @@
# Public ingress — VPS traefik + cert mirror
How home services reach the open internet without exposing the home network. Two-stage pattern: traefik on the IONOS VPS terminates TLS at the public edge, then reverse-proxies over the netbird mesh to home Caddy / direct backends.
## The shape
```
Public client
│ *.hubris.network → 82.165.190.79 (IONOS wildcard)
▼
[VPS traefik] priority 1: HostSNI(*) → netbird-proxy:8443 ← netbird control plane
priority 10: per-host HTTP routers ← home services
│ HTTP over netbird mesh
▼
[Home backend on 192.168.8.x]
```
Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
## Why this shape
- VPS traefik already has a HostSNI(*) TCP passthrough at priority 1 so netbird's own ingress (`netbird.hubris.network`, future `*.proxy.hubris.network`) is unaffected.
- Per-hostname HTTP file-provider routers at priority 10 win over the passthrough for the listed hosts and let traefik terminate TLS itself for those.
- Traefik's own ACME (`letsencrypt` resolver) fails on this box: HostSNI(*) grabs TLS-ALPN-01 challenges before traefik's `allowACMEByPass=true` can respond. Solution: home Caddy obtains certs via IONOS DNS-01 (no conflict) and the VPS *mirrors* the result over.
## Components
### On the VPS (`82.165.190.79`)
- `/opt/traefik-dynamic.yaml` — file-watched dynamic config. One `http.routers.<name>-public` + one `http.services.<name>-public` per exposed service, plus one entry in the top-level `tls.certificates` list per hostname.
- Let's-Encrypt-effective directory: `/letsencrypt/` inside the traefik container, backed by the host-side docker volume `opt_netbird_traefik_letsencrypt`.
- Backups before edits: `cp /opt/traefik-dynamic.yaml /opt/traefik-dynamic.yaml.bak.$(date +%s)` — several bak files live alongside.
### On the PVE host (`192.168.8.77`)
- `/usr/local/bin/hubris-public-cert-sync.sh` — runs daily via `hubris-public-cert-sync.timer`. Maps source hostname → VPS cert filenames in a bash assoc array. For each mapping: `pct pull` cert+key from [Caddy (121)](../containers/121-caddy.md)'s store, diff against the VPS copy, scp only on change.
- Filenames are stable per host so the dynamic.yaml never needs editing on renewal — traefik file-watches and hot-reloads the cert.
## Services currently exposed
| Hostname | Path scope | Backend | Middlewares | Cert files on VPS |
| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ |
| `artifacto.hubris.network` | `/p/*`, `/static/*`, `/healthz` | `192.168.8.205:3100` | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key` |
| `blog.hubris.network` | whole host | `192.168.8.205:8080` | `blog-ratelimit` (100 rps / 200 burst) | `blog.fullchain.crt` / `blog.privkey.key` |
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
## Recipe — exposing another service
1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`).
2. Add an entry to `HOSTS` in `/usr/local/bin/hubris-public-cert-sync.sh` mapping the hostname → VPS filenames. Run once: `systemctl start hubris-public-cert-sync.service`. Confirm the cert landed.
3. Edit `/opt/traefik-dynamic.yaml` on the VPS:
- Add to `tls.certificates`: paths `/letsencrypt/<crt>` and `/letsencrypt/<key>`.
- Add `http.routers.<host>-public`: `rule: 'Host(\`<host>\`)'` (or with path matchers if scope-gating), `entryPoints: [websecure]`, `priority: 10`, `tls: {}`, `service: <host>-public`, `middlewares: [...]`.
- Add a ratelimit middleware under `http.middlewares` if wanted.
- Add `http.services.<host>-public.loadBalancer.servers[0].url: 'http://<lan-ip>:<port>'`.
4. Verify:
```
ssh root@100.122.165.149 'curl -skI --resolve <host>:443:127.0.0.1 https://<host>/' # 2xx/3xx
curl -skI --resolve <host>:443:<vps-public-ip> https://<host>/ # same
```
5. **No DNS edit needed** — the IONOS wildcard already points at the VPS.
## What does NOT follow this pattern
- `netbird.hubris.network` (and any future `*.proxy.hubris.network`) uses the netbird-proxy / HostSNI passthrough path. Netbird handles its own cert via ACME cleanly because it *is* the passthrough target.
## Related
- [DNS split-horizon](dns.md)
- [Caddy (121)](../containers/121-caddy.md) — cert source, internal counterpart
- [Mesh migration](mesh.md) — netbird is the transport between VPS and home
- [VPS hardening](vps-hardening.md) — fail2ban / nftables that the access logs feed
- [Artifacto on apps (105)](../containers/105-apps.md) — first publicly-exposed service
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-23 — `blog.hubris.network` exposed
WriteFreely on [apps (105)](../containers/105-apps.md). Whole host is public.
### 2026-04-22 — pattern established with Artifacto
First service through the file-provider router. IONOS wildcard moved to the VPS this day. Cert mirror script + timer deployed on the PVE host.

95
infrastructure/media-permissions.md Normal file
View File

@@ -0,0 +1,95 @@
# Media permissions — `media` GID 10000
Standard for any LXC reading/writing `/mnt/library` on [hubris](../hosts/hubris.md). Applied 2026-04-20.
## Standard
Every LXC that mounts `/mnt/library` participates in a shared `media` group with **GID 10000**. Shared subtrees are owned by that group with the setgid bit (`drwxrwsr-x`, mode `2775`), so new files auto-inherit the right group regardless of which container wrote them.
## Why
`/mnt/library` is a cross-container storage pool. \*arr writes, jellyfin reads, syncthing syncs, photoprism / immich / mulita scan, paperless ingests. Without a shared group, each container sees files as `nobody:nogroup` (unprivileged) or `www-data` (privileged 1:1) and the permission web collapses into one-off chmods. GID 10000 bridges privileged and unprivileged containers.
## Onboarding a new LXC
1. `pct set <id> -mp0 /mnt/library,mp=/mnt/library` (if not already mounted).
2. Inside the container:
```
groupadd -g 10000 media
usermod -aG media <service-user> # for every user that needs library access
```
3. If the container is **unprivileged** (check `pct config <id> | grep unprivileged`), append this idmap block to `/etc/pve/lxc/<id>.conf` (back up first):
```
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 10000
lxc.idmap: g 10000 10000 1
lxc.idmap: g 10001 110001 55535
```
Then `pct stop <id> && pct start <id>`.
4. For systemd services running with `User=root` (not typical — but [syncthing (109)](../containers/109-syncthing.md) is one), add a drop-in with `SupplementaryGroups=media`. Systemd skips `initgroups()` for `User=root`.
5. `pct exec` sessions don't get supplementary groups (no initgroups). Use `sudo -i` or `su - <user>` inside the container to verify membership interactively. Real services use `initgroups` and work correctly.
## State snapshot
### Host
- Group `media` GID 10000 exists.
- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000).
- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid):
- `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / booklore
- `downloads` — \*arr stack output
- `images` — photoprism / immich / mulita
- `roms` — emu frontends
- `syncthing` — cross-device sync
- Container-specific subtrees intentionally **not** migrated (keep their own owner:group):
- `documents` (paperless, `www-data:www-data 750`)
- `homecloud` (nextcloud — its own permission model, easy to break)
- `marimo` (marimo venv) — *LXC since destroyed; review whether subtree still serves a purpose*
- `notes`, `heaper`, `sophia` (single-container use)
- `repos` (owner UID 102 GID 105 from inside [gitea](../containers/104-gitea.md) — don't touch)
### LXCs with media-group membership
| ID | Name | Priv | Media-group members |
| --- | --------------------------------------------- | ---- | --------------------------------------------- |
| 101 | [jellyfin](../containers/101-jellyfin.md) | **unpriv + idmap** | jellyfin |
| 103 | [paperless](../containers/103-paperless.md) | priv | www-data |
| 104 | [gitea](../containers/104-gitea.md) | priv | www-data, gitea |
| 105 | [apps](../containers/105-apps.md) | priv | www-data |
| 109 | [syncthing](../containers/109-syncthing.md) | **unpriv + idmap** | root (via `SupplementaryGroups=media` drop-in on `syncthing@.service`) |
| 114 | [nextcloud](../containers/114-nextcloud.md) | priv | www-data |
| 116 | [heaper](../containers/116-heaper.md) | priv | www-data |
| 119 | [sophia](../containers/119-sophia.md) | priv | www-data |
| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data |
| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
> Some entries from the older snapshot — 100 (arr-yunohost), 107 (marimo), 110 (photoprism), 112 (immich) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).
Config backups: `/root/101.conf.bak.*`, `/root/109.conf.bak.*`.
## Gotchas
- **[apps (105)](../containers/105-apps.md) is a Docker host.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000` or `user: "<uid>:10000"` in compose. Booklore, audiobookshelf-in-docker, etc. need this per-container.
- **`pct exec` does NOT run initgroups.** So `pct exec 109 -- id` shows only the primary group. For interactive verification, use `pct exec 109 -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine.
- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed.
- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page.
- **`/etc/subgid`** must retain both `root:100000:65536` AND `root:10000:1`. Dropping the second breaks startup of any unprivileged LXC with the idmap block.
- **\*arr "Set Permissions" options** can override the setgid inheritance by explicitly chown'ing files. Leave those off, or set the group to `media`. Relevant to Sonarr/Radarr/qBittorrent on [arriman (122)](../containers/122-arriman.md).
- **Nextcloud** files under `/mnt/library/homecloud` are deliberately NOT in the media group. NC manages its own permission model. See [nextcloud (114)](../containers/114-nextcloud.md).
- **\*arr stack on arriman** required `MEDIACENTER_GID=10000` (not 13000) in `.env` because s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. See [arriman (122)](../containers/122-arriman.md#changelog).
## Related
- [Hubris host](../hosts/hubris.md)
- All container pages list whether they're in the standard
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-26 — `MEDIACENTER_GID` fix on [arriman (122)](../containers/122-arriman.md)
qBit was erroring every torrent with "Permission denied" because `MEDIACENTER_GID=13000` was set as a supplementary GID via `group_add:`. Changed to 10000 (primary GID); fix described above is now standard.
### 2026-04-20 — standard rolled out
GID 10000 hostgroup, idmap blocks for unprivileged LXCs, setgid 2775 on shared subtrees, `media` membership for service users in every participating LXC.

94
infrastructure/mesh.md Normal file
View File

@@ -0,0 +1,94 @@
# Mesh — Tailscale → Netbird migration
The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target end-state. In-progress as of 2026-04-21.
## Current state
- **PVE host** uses Netbird (`wt0`, `100.122.38.109/16`). Its resolver is the local netbird daemon, which forwards to LAN/upstream — so the PVE host gets `*.hubris.network → 192.168.8.175` via the system resolver chain.
- **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration.
- **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service.
## Consequence — every LXC wired to Authentik needs an internal override
Until each LXC is migrated to Netbird, anything that needs to reach `auth.hubris.network` (Authentik), `cloud.hubris.network` (Nextcloud), etc., must override the public answer with `192.168.8.175`.
Two techniques. Pick by HTTP-client behavior.
### A) `/etc/hosts` override
Works for libc `getaddrinfo` clients: curl, wget, most Go/Python/Ruby apps, gitea.
- Place the line **outside** the `# --- BEGIN PVE ---` / `# --- END PVE ---` markers. Proxmox rewrites everything inside that block on every container start.
- Belt-and-suspenders: `/etc/systemd/system/hubris-hosts-override.service` (oneshot, enabled, idempotent).
### B) Local dnsmasq
Required for clients that bypass `/etc/hosts`. **Nextcloud (PHP Guzzle + `OC\Http\Client\DnsPinMiddleware`) is one** — uses `dns_get_record()`, not `getaddrinfo`. Likely candidates for the same treatment: Python/Ruby/Java apps that do explicit DNS resolution before connecting.
Recipe:
```
apt install dnsmasq
cat > /etc/dnsmasq.d/hubris-internal.conf <<EOF
address=/auth.hubris.network/192.168.8.175
server=192.168.8.1
server=1.1.1.1
interface=lo
bind-interfaces
no-hosts
no-resolv
EOF
# Set LXC default nameservers and live resolv.conf
pct set <id> --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"
# Then update /etc/resolv.conf inside the LXC too.
```
### Known overrides applied
| LXC | Technique | Notes |
| ------------------------------------------ | ---------------------------------------- | ----- |
| [104 (gitea)](../containers/104-gitea.md) | `/etc/hosts` + `hubris-hosts-override.service` | Standard |
| [114 (nextcloud)](../containers/114-nextcloud.md) | local dnsmasq @ 127.0.0.1 + resolv.conf reorder | Guzzle bypasses /etc/hosts |
| [105 (apps)](../containers/105-apps.md), inner containers | `extra_hosts:` in compose | Booklore, mulita, WriteFreely each ship with this |
## Adding new LXCs
- Don't add new LXCs to Tailscale; add them to Netbird. Tailscale is being decommissioned on hubris.
- When wiring a new app into Authentik: `cat /etc/resolv.conf` on the target LXC. If nameserver is `192.168.8.1` or `100.100.100.100`, add the hosts override. If it's the netbird daemon IP, skip.
## Long-term fix
Either:
- Split-horizon DNS at LAN/router level so `*.hubris.network → 192.168.8.175` for every LAN client. Eliminates all per-LXC overrides.
- Once Netbird migration completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain on the PVE host.
## CRITICAL — never `docker compose up` Portainer-managed stacks
[apps (105)](../containers/105-apps.md) runs multiple stacks deployed via the Portainer UI (`/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`). Running `docker compose up -d <svc>` from the host shell on these triggers a recreate of OTHER services in the stack — labels drift, compose treats the project as reconciling, containers get destroyed and remade. **This wiped Booklore's mariadb data on 2026-04-22** (bind mount `./mariadb/config` re-initialized fresh).
Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-managed stacks:
1. Edit the compose in Portainer UI → **Stacks → <stack> → Editor → Update stack**. Portainer handles the recreate cleanly with its own state tracking.
2. Do NOT edit `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/docker-compose.yml` directly.
3. For data-preserving DNS overrides on stateful services, prefer Caddy-side handling (forward-auth) or the app's built-in OIDC over container-level `extra_hosts` when possible.
## Related
- [DNS split-horizon](dns.md)
- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides
- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B
- [Gitea (104)](../containers/104-gitea.md) — example of Technique A
- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-22 — Booklore mariadb data wiped (lesson recorded)
The "never docker compose up Portainer-managed stacks" rule comes from this. See [apps (105)](../containers/105-apps.md#changelog).
### 2026-04-22 — netbird mgmt host joined its own mesh
`82.165.190.79` is now a peer (`100.122.165.149`). LAN access + split-horizon DNS via PVE peer. See [VPS hardening](vps-hardening.md).
### 2026-04-21 — overrides applied to gitea (104) and nextcloud (114)
Two techniques documented; Nextcloud forced the dnsmasq route because of Guzzle.

75
infrastructure/monitoring.md Normal file
View File

@@ -0,0 +1,75 @@
# Monitoring — `claudio-monitor`
Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.
## Why
After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.
## Components
- **Repo:** `dtoro/claudio-monitor`
- **Checkout:** `/opt/claudio-monitor` on the PVE host
- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).
## Per-LXC data
A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.
## Matrix-side plugin
`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
- `/monitor status`
- `/monitor history [N]`
- `/monitor clear <key>`
Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.
Events posted to claudio-bot `/notify`:
```json
{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
```
`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.
## Current thresholds & active alerts
Active warnings at the time monitor was deployed (2026-04-21):
- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%
## Testing
- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.
## Not yet deployed (tracked in plan file)
- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.
## Related repos
- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.
## Related pages
- [Hubris host](../hosts/hubris.md)
- [claudio-bot (123)](../containers/123-claudio-bot.md)
- [HAOS VM (108)](../vms/108-haos.md)
- [Backups (disabled)](backups.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.

45
infrastructure/ssh-access.md Normal file
View File

@@ -0,0 +1,45 @@
# SSH access
How to reach hubris and the VPS over SSH, and the dual-server gotcha.
## Hubris
Two SSH endpoints — easy to hit the wrong one.
| Server | Listen | Auth | Notes |
| -------------- | ---------------------------- | --------------------------------- | ----- |
| OpenSSH | `0.0.0.0:22` | `authorized_keys` at `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced; symlinked from `/root/.ssh/authorized_keys`) | Standard. |
| Netbird SSH | `100.122.38.109:22022` | OIDC / browser auth — bypasses `authorized_keys` | If a client lands here it'll open a browser tab to authenticate, then sometimes hang. Force port 22 or use the LAN IP. |
### Authorized root keys
- `root@hubris` (self, RSA) — original.
- `d.toro.v@pm.me` (ed25519) — user's iMac (`mac-mini.netbird.selfhosted`, LAN `192.168.8.174`), added 2026-04-22.
### Notes
- Password auth is enabled on hubris but the root password is **not** the one the user expects. Prefer key flows; don't try `ssh-copy-id` blind.
- Off-LAN access from the iMac uses the LAN path. As of 2026-04-22 the iMac's Netbird tunnel to hubris was P2P healthy but no packets were captured on `wt0`; needs revisit if remote access becomes critical.
## VPS (`82.165.190.79` / `100.122.165.149`)
- **Mesh-only.** Public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`.
- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`.
- `ListenAddress` is still 0.0.0.0; gating is firewall-layer.
- Authorized root keys: PVE (`root@hubris`), iMac (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer before disabling its access paths.
See [VPS hardening](vps-hardening.md) for the firewall + fail2ban rules and recovery paths.
## Related
- [Hubris host](../hosts/hubris.md)
- [Mesh migration](mesh.md)
- [VPS hardening](vps-hardening.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-23 — VPS SSH hardened to mesh-only
Public `:22` blocked at nftables. Key-only sshd. See [VPS hardening](vps-hardening.md).
### 2026-04-22 — iMac key authorized on hubris
`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`.

84
infrastructure/vps-hardening.md Normal file
View File

@@ -0,0 +1,84 @@
# VPS hardening — `82.165.190.79` / `100.122.165.149`
IONOS VPS that runs the Netbird control plane and the [public ingress traefik](ingress.md). Hardened 2026-04-23 from its stock-Plesk state.
## At a glance
- **Hostname:** `inspiring-ramanujan.82-165-190-79.plesk.page`
- **OS:** Debian 13
- **Mesh:** netbird `100.122.165.149` (peer of the lab mesh; routes `192.168.8.0/24` via [hubris](../hosts/hubris.md)).
- **Public:** `82.165.190.79` (`ens6`).
- **Public DNS:** IONOS wildcard `*.hubris.network → 82.165.190.79`.
## SSH
- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`.
- **Mesh-only**: public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`. `ListenAddress` itself is still `0.0.0.0` — gating is firewall-layer.
- Authorized root keys: PVE (`root@hubris`), Mac Mini (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer **before** disabling its access paths.
## Firewall — nftables (`inet hubris-fw`)
Config at `/etc/nftables.conf`, service enabled.
- Public iface `ens6`. Wireguard iface `wt0`.
- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6. Everything else drops.
- `wt0` fully accepted in INPUT. `lo` accepted.
- **FORWARD chain at priority `filter-10`** (runs before Docker's FORWARD) hosts the fail2ban ban enforcement — see below.
- Set `banned4` (typed `ipv4_addr`, flag `timeout`) holds fail2ban's drops.
- Coexists with Docker's `ip nat` / `ip filter` tables (iptables-nft compat). **Do NOT `flush ruleset`** in this config — it'll wipe Docker's state too.
## fail2ban
- **Jail `traefik-4xx`** tails `/var/log/traefik/access.log` (bind-mounted from container). Filter at `/etc/fail2ban/filter.d/traefik-4xx.conf` matches 401/403/404/429 from `blog-public@file` or `artifacto-public@file` routers only — netbird-grpc traffic isn't considered.
- Tunables: `findtime=600, maxretry=30, bantime=3600`.
- **Action** at `/etc/fail2ban/action.d/nft-hubris.conf` adds/removes elements from `inet hubris-fw banned4` with per-element timeout.
### CRITICAL invariant — wireguard / fail2ban
**Bans must never affect `wt0` or wireguard UDP.** The FORWARD chain explicitly `accept`s the following *before* the ban check:
- `udp 51820` (wireguard)
- `udp 3478` (STUN)
- `ct state established,related`
The INPUT ban rule is scoped to `iifname "ens6"`.
Violating this takes the mesh down for every home device (they share one public IP) and the only recovery is IONOS console → `nft flush set inet hubris-fw banned4`.
## Traefik access log
- Written to `/var/log/traefik/access.log` on the host via a bind mount added to `/opt/docker-compose.yml` (traefik volumes include `/var/log/traefik:/logs`) plus `--accesslog.filepath=/logs/access.log`.
- CLF format. Real client IP arrives correctly because docker userland-proxy is off — see [public ingress](ingress.md).
## Plesk / mail / FTP / Dr.Web
Stopped and disabled (not uninstalled). All of:
`dovecot`, `dovecot.socket`, `postfix`, `postfix@-`, `pc-remote`, `xinetd`, `plesk-task-manager`, `plesk-web-socket`, `sw-cp-server`, `sw-engine`, `plesk-repaird`, `plesk-repaird.socket`, `drwebd`.
`psa.service` is masked (was a one-shot boot bootstrap). `/etc/cron.d/plesk-backup-manager-task` renamed to `.disabled`.
Reverse: `systemctl unmask psa; systemctl enable --now <svc>`.
## Auto-patching
- `unattended-upgrades` enabled (stock).
- Drop-in at `/etc/apt/apt.conf.d/52hubris-reboot.conf` sets auto-reboot at **04:00 UTC** when `/var/run/reboot-required` is set.
- Runs inside the stock `apt-daily-upgrade.timer`.
## Recovery paths
Ordered by preference:
1. **SSH via mesh** — primary. Any mesh peer with an authorized key.
2. **IONOS web console** (my.ionos.com → VPS → Console) — uses the system password, not SSH keys. Bypasses any firewall misconfig.
3. **IONOS rescue mode** — boot rescue, mount rootfs, edit `/etc/nftables.conf` or `/etc/ssh/sshd_config.d/10-hubris-hardening.conf` to a known-good state, reboot.
## Related
- [Public ingress (VPS traefik)](ingress.md)
- [Mesh migration](mesh.md) — VPS as a mesh peer
- [SSH access](ssh-access.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-23 — hardened
nftables firewall, mesh-only SSH, fail2ban traefik jail, Plesk disabled, auto-reboot 04:00 UTC, wireguard/fail2ban invariant established.