dtoro
f9f2e8ab5e
Root cause of "NetBird won't forward to Technitium" was NOT a nameserver bug — it was a missing route. The home-lab-dns nameserver group (-> 192.168.8.2, domain hubris.network) was applied to all peers, but the 192.168.8.0/24 route (home-lab-network resource) was distributed only to the Services group. Roaming peers (Core: iphone + laptops) had no route to 192.168.8.2, so forwarding silently failed (Networks: -). Fix: added Core to the home-lab-network resource distribution via the NetBird API. Roaming peers now get the subnet route + the already-applied nameserver forwarding -> *.hubris.network resolves off-LAN. Also grants roaming devices full homelab service access. No CT 107 mesh-join needed (original Phase 2 hypothesis obsolete). Docs: corrected dns.md + 107-dns.md root-cause claims. Managed zone kept as fallback pending Phase 4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
158 lines
16 KiB
Markdown
158 lines
16 KiB
Markdown
# DNS — split-horizon `*.hubris.network`
|
||
|
||
LAN clients resolve `*.hubris.network` to the [Caddy reverse proxy](../containers/121-caddy.md) (`192.168.8.175`). Public clients resolve to the IONOS VPS (`82.165.190.79`) via an IONOS wildcard, where they hit the [VPS traefik public ingress](ingress.md).
|
||
|
||
There is **no wildcard on the LAN side**. Every subdomain needs an explicit entry.
|
||
|
||
## Components
|
||
|
||
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
|
||
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
|
||
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
|
||
- **All LXCs** now point at Technitium (`192.168.8.2`) directly (since 2026-06-21 — see changelog). The earlier mix of router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`) — which returned the public IONOS A record and forced `/etc/hosts` overrides — has been removed.
|
||
|
||
## Live entries (as of 2026-06-04)
|
||
|
||
```
|
||
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
|
||
address=/sso.hubris.network/192.168.8.175 # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
|
||
address=/git.hubris.network/192.168.8.175
|
||
address=/media.hubris.network/192.168.8.175
|
||
address=/paperless.hubris.network/192.168.8.175
|
||
address=/books.hubris.network/192.168.8.175
|
||
address=/home.hubris.network/192.168.8.175
|
||
address=/cloud.hubris.network/192.168.8.175
|
||
address=/matrix.hubris.network/192.168.8.175
|
||
address=/proxmox.hubris.network/192.168.8.175
|
||
address=/docker.hubris.network/192.168.8.175
|
||
address=/jellyseerr.hubris.network/192.168.8.175
|
||
address=/qbit.hubris.network/192.168.8.175
|
||
address=/sab.hubris.network/192.168.8.175
|
||
address=/blog.hubris.network/192.168.8.175
|
||
address=/photos.hubris.network/192.168.8.175
|
||
address=/photos-new.hubris.network/192.168.8.175
|
||
address=/artifacto.hubris.network/192.168.8.175
|
||
address=/plato.hubris.network/192.168.8.175
|
||
address=/zimaos.hubris.network/192.168.8.175
|
||
address=/nfs-export.hubris.network/192.168.8.200
|
||
```
|
||
|
||
Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do.
|
||
|
||
## Why split-horizon
|
||
|
||
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.
|
||
|
||
## The gotcha that cost a debug session (2026-04-22)
|
||
|
||
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
|
||
|
||
## Recipe — adding a new subdomain
|
||
|
||
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
|
||
2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)` — the NetBird managed DNS zone sync picks it up within ~10 minutes via cron. Or add directly to the NetBird managed zone via API if you need it faster.
|
||
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
|
||
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
|
||
|
||
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
|
||
|
||
## Public path — what does and doesn't follow the LAN map
|
||
|
||
- Hostnames published in [VPS traefik dynamic config](ingress.md) (currently `artifacto.hubris.network`, `blog.hubris.network`) reach a real backend over the netbird mesh.
|
||
- Anything else with a `*.hubris.network` URL hits the VPS but isn't routed anywhere — returns 404.
|
||
- `netbird.hubris.network` is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.
|
||
|
||
## Long-term plan
|
||
|
||
Either:
|
||
- Move split-horizon DNS to the LAN router so `*.hubris.network → 192.168.8.175` is answered for every LAN client. Eliminates per-LXC overrides.
|
||
- Or, once the [Tailscale → Netbird migration](mesh.md) completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.
|
||
|
||
## Related
|
||
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
|
||
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
|
||
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
|
||
- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority
|
||
|
||
## Changelog
|
||
|
||
### 2026-06-21 — DNS single-source, Phase 2: roaming-peer forwarding fixed (root cause was a missing route, not a NetBird bug)
|
||
The long-standing belief that "NetBird won't forward to Technitium for mesh peers" (which is *why* the managed-zone sync was built) turned out to be **wrong**. The NetBird API showed the real cause:
|
||
|
||
- The `home-lab-dns` nameserver group (→ `192.168.8.2`, match-domain `hubris.network`) was already applied to **all** peers (via the `All` group). So every peer *had* the forwarding rule.
|
||
- But the `192.168.8.0/24` route (network resource `home-lab-network`) was distributed to the **`Services`** group only = `{netbird-ionos, proxmox-server}`. Roaming peers (`dtoro-iphone`, `mac-mini`, `republic-laptop`, `muli-laptop`, `ludo-mini`) are in **`Core`**, which had **no route to the homelab subnet** → they couldn't reach `192.168.8.2` → forwarding silently failed (`Networks: -`).
|
||
|
||
**Fix:** added the `Core` group to the `home-lab-network` resource distribution (now `[Services, Core]`) via `PUT /api/networks/.../resources/...`. Roaming peers immediately picked up `Networks: 192.168.8.0/24` and `[192.168.8.2:53] for [hubris.network] is Available`. No CT 107 mesh-join / TUN / restart was needed (the original Phase 2 hypothesis is obsolete). This also gives roaming devices full homelab **service** access, not just DNS.
|
||
|
||
The managed zone is still in place as a fallback pending the Phase 3 roaming test (iPhone on cellular); Phase 4 then deletes the managed zone + `dns-sync` cron. Rollback: PUT the resource back to `[Services]` only.
|
||
|
||
### 2026-06-21 — DNS single-source, Phase 1: on-prem LXCs decoupled from NetBird
|
||
Goal: collapse the three overlapping DNS sources (Technitium + NetBird managed zone + per-LXC band-aids) toward **one zone**, keeping NetBird. Phase 1 (the safe, mesh-independent half) is done:
|
||
|
||
- **Every homelab LXC now resolves via Technitium (`192.168.8.2`).** Fixed 8 boxes that were on a dead resolver, the router, or Tailscale MagicDNS:
|
||
- `192.168.8.180` (dead ex-Authentik): 102, 106, 126
|
||
- `192.168.8.1` (router → public answer): 101
|
||
- `100.100.100.100` (Tailscale MagicDNS): 104, 105, 114, 119 — also ran `tailscale set --accept-dns=false` so `tailscaled` stops rewriting `/etc/resolv.conf`.
|
||
- Already correct (`.2`): 103, 118, 120, 121, 122. CT 107 stays on `1.1.1.1` by design (no self-dependency).
|
||
- **Removed the redundant `/etc/hosts` band-aids** (`auth`/`mcp`/`secrets` → `192.168.8.175`) on 101, 103, 104, 105, 114, 118, 120, 121, 122, 126; disabled `hubris-hosts-override.service` where enabled. Backups at `/etc/hosts.bak-dnsplan`. PVE-managed lines and self-hostname maps preserved. Technitium already returns identical-or-better answers (verified: `auth → 82.165.190.79`, `mcp`/`secrets`/`cloud`/`git` → `192.168.8.175`).
|
||
- **Nextcloud (114):** its documented Guzzle-workaround dnsmasq is not running; resolves correctly straight from Technitium, so no special-casing remains.
|
||
- Net effect: **NetBird's DNS now only matters for off-LAN roaming peers** (Tier 2). On-prem (LXCs + household via Fritz!Box→Technitium) is fully NetBird-independent — so dropping the managed zone later can no longer break on-LAN resolution. Phases 2–4 (mesh-IP forwarding, roaming test, managed-zone + sync deletion) still pending.
|
||
|
||
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
|
||
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
|
||
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
|
||
- Caddy: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxied to VPS
|
||
- `header_up Host auth.hubris.network` added to strip `:443` from upstream Host header
|
||
- All 9 LXCs' /etc/hosts updated: `auth.hubris.network → 192.168.8.175` (Caddy proxy)
|
||
- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
|
||
- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
|
||
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
|
||
|
||
- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
|
||
- A **sync job on [dns (107)](../containers/107-dns.md)** (`/opt/dns-sync/sync.py`, cron */10) reconciles Technitium's named A-records → the **NetBird managed DNS zone** via the NetBird API (`/api/dns/zones/{id}/records`, PAT in sops `secrets/netbird-pat.yaml`). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (`.79`) — correct.
|
||
- This **killed the manual drift** that caused the whole `auth`/`sso`/`nfs-export` saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.
|
||
|
||
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
|
||
|
||
> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
|
||
|
||
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
|
||
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
|
||
|
||
**Fixed:** added `/etc/cron.d/dns-sync` (`*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1`).
|
||
|
||
Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backend-health`) that runs `scripts/check-caddy-backends.sh` every 10 minutes.
|
||
|
||
### 2026-06-02 — 8 LXCs moved from DHCP to static IP
|
||
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
|
||
|
||
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
|
||
Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
|
||
|
||
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
|
||
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
||
|
||
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
|
||
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
|
||
|
||
### 2026-05-14 — `zimaos.hubris.network` added (Caddy-fronted, standard pattern)
|
||
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
|
||
|
||
### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
|
||
New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
|
||
|
||
### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
|
||
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.
|
||
|
||
### 2026-04-28 — wiki entry created
|
||
Initial documentation. 16 active entries.
|
||
|
||
### 2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79
|
||
Public path now lands on the VPS traefik, not the old yunohost. Necessary for the [public ingress](ingress.md) pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.
|
||
|
||
### 2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)
|
||
Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.
|
||
|
||
### 2026-04-21 — dnsmasq stood up on LXC 124
|
||
Co-located with Authentik. Initial entries cover everything routed through Caddy.
|