dtoro
85457095e4
Phase 1 of the DNS-redundancy cleanup (keep NetBird, collapse toward one zone) — the safe, mesh-independent half: - Every homelab LXC now resolves via Technitium (192.168.8.2). Fixed 8 boxes on a dead resolver (.180), the router (.1), or Tailscale MagicDNS (100.100.100.100): 101,102,104,105,106,114,119,126. - Removed the redundant /etc/hosts auth/mcp/secrets overrides (Technitium returns identical-or-better answers); disabled hubris-hosts-override. - Net effect: on-prem DNS (LXCs + household via Fritz!Box->Technitium) is now NetBird-independent, so dropping the managed zone later can't break on-LAN resolution. Phases 2-4 still pending. Tailscale decommissioned (was legacy/being-phased-out): - Removed from the 6 LXCs still running it (101,103,104,105,114,119): logout, disable tailscaled, apt purge, state cleared. - inventory.yaml: dropped tailscale from accepted + all mesh blocks; regenerated hosts/*.yaml (also pruned orphan authentik/claudio-bot). - Tightened secrets-issuance MESH_SUBNETS: removed the now-vestigial Tailscale CGNAT range 100.64.0.0/10. - Updated narrative docs (mesh, dns, network, README, AGENTS, agent-enrollment, homelab-context, 105-apps, 107-dns). Live infra changed on the fleet + Mac; this commit records the docs/inventory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
148 lines
14 KiB
Markdown
148 lines
14 KiB
Markdown
# DNS — split-horizon `*.hubris.network`
|
||
|
||
LAN clients resolve `*.hubris.network` to the [Caddy reverse proxy](../containers/121-caddy.md) (`192.168.8.175`). Public clients resolve to the IONOS VPS (`82.165.190.79`) via an IONOS wildcard, where they hit the [VPS traefik public ingress](ingress.md).
|
||
|
||
There is **no wildcard on the LAN side**. Every subdomain needs an explicit entry.
|
||
|
||
## Components
|
||
|
||
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
|
||
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
|
||
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
|
||
- **All LXCs** now point at Technitium (`192.168.8.2`) directly (since 2026-06-21 — see changelog). The earlier mix of router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`) — which returned the public IONOS A record and forced `/etc/hosts` overrides — has been removed.
|
||
|
||
## Live entries (as of 2026-06-04)
|
||
|
||
```
|
||
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
|
||
address=/sso.hubris.network/192.168.8.175 # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
|
||
address=/git.hubris.network/192.168.8.175
|
||
address=/media.hubris.network/192.168.8.175
|
||
address=/paperless.hubris.network/192.168.8.175
|
||
address=/books.hubris.network/192.168.8.175
|
||
address=/home.hubris.network/192.168.8.175
|
||
address=/cloud.hubris.network/192.168.8.175
|
||
address=/matrix.hubris.network/192.168.8.175
|
||
address=/proxmox.hubris.network/192.168.8.175
|
||
address=/docker.hubris.network/192.168.8.175
|
||
address=/jellyseerr.hubris.network/192.168.8.175
|
||
address=/qbit.hubris.network/192.168.8.175
|
||
address=/sab.hubris.network/192.168.8.175
|
||
address=/blog.hubris.network/192.168.8.175
|
||
address=/photos.hubris.network/192.168.8.175
|
||
address=/photos-new.hubris.network/192.168.8.175
|
||
address=/artifacto.hubris.network/192.168.8.175
|
||
address=/plato.hubris.network/192.168.8.175
|
||
address=/zimaos.hubris.network/192.168.8.175
|
||
address=/nfs-export.hubris.network/192.168.8.200
|
||
```
|
||
|
||
Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do.
|
||
|
||
## Why split-horizon
|
||
|
||
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.
|
||
|
||
## The gotcha that cost a debug session (2026-04-22)
|
||
|
||
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
|
||
|
||
## Recipe — adding a new subdomain
|
||
|
||
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
|
||
2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)` — the NetBird managed DNS zone sync picks it up within ~10 minutes via cron. Or add directly to the NetBird managed zone via API if you need it faster.
|
||
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
|
||
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
|
||
|
||
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
|
||
|
||
## Public path — what does and doesn't follow the LAN map
|
||
|
||
- Hostnames published in [VPS traefik dynamic config](ingress.md) (currently `artifacto.hubris.network`, `blog.hubris.network`) reach a real backend over the netbird mesh.
|
||
- Anything else with a `*.hubris.network` URL hits the VPS but isn't routed anywhere — returns 404.
|
||
- `netbird.hubris.network` is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.
|
||
|
||
## Long-term plan
|
||
|
||
Either:
|
||
- Move split-horizon DNS to the LAN router so `*.hubris.network → 192.168.8.175` is answered for every LAN client. Eliminates per-LXC overrides.
|
||
- Or, once the [Tailscale → Netbird migration](mesh.md) completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.
|
||
|
||
## Related
|
||
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
|
||
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
|
||
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
|
||
- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority
|
||
|
||
## Changelog
|
||
|
||
### 2026-06-21 — DNS single-source, Phase 1: on-prem LXCs decoupled from NetBird
|
||
Goal: collapse the three overlapping DNS sources (Technitium + NetBird managed zone + per-LXC band-aids) toward **one zone**, keeping NetBird. Phase 1 (the safe, mesh-independent half) is done:
|
||
|
||
- **Every homelab LXC now resolves via Technitium (`192.168.8.2`).** Fixed 8 boxes that were on a dead resolver, the router, or Tailscale MagicDNS:
|
||
- `192.168.8.180` (dead ex-Authentik): 102, 106, 126
|
||
- `192.168.8.1` (router → public answer): 101
|
||
- `100.100.100.100` (Tailscale MagicDNS): 104, 105, 114, 119 — also ran `tailscale set --accept-dns=false` so `tailscaled` stops rewriting `/etc/resolv.conf`.
|
||
- Already correct (`.2`): 103, 118, 120, 121, 122. CT 107 stays on `1.1.1.1` by design (no self-dependency).
|
||
- **Removed the redundant `/etc/hosts` band-aids** (`auth`/`mcp`/`secrets` → `192.168.8.175`) on 101, 103, 104, 105, 114, 118, 120, 121, 122, 126; disabled `hubris-hosts-override.service` where enabled. Backups at `/etc/hosts.bak-dnsplan`. PVE-managed lines and self-hostname maps preserved. Technitium already returns identical-or-better answers (verified: `auth → 82.165.190.79`, `mcp`/`secrets`/`cloud`/`git` → `192.168.8.175`).
|
||
- **Nextcloud (114):** its documented Guzzle-workaround dnsmasq is not running; resolves correctly straight from Technitium, so no special-casing remains.
|
||
- Net effect: **NetBird's DNS now only matters for off-LAN roaming peers** (Tier 2). On-prem (LXCs + household via Fritz!Box→Technitium) is fully NetBird-independent — so dropping the managed zone later can no longer break on-LAN resolution. Phases 2–4 (mesh-IP forwarding, roaming test, managed-zone + sync deletion) still pending.
|
||
|
||
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
|
||
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
|
||
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
|
||
- Caddy: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxied to VPS
|
||
- `header_up Host auth.hubris.network` added to strip `:443` from upstream Host header
|
||
- All 9 LXCs' /etc/hosts updated: `auth.hubris.network → 192.168.8.175` (Caddy proxy)
|
||
- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
|
||
- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
|
||
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
|
||
|
||
- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
|
||
- A **sync job on [dns (107)](../containers/107-dns.md)** (`/opt/dns-sync/sync.py`, cron */10) reconciles Technitium's named A-records → the **NetBird managed DNS zone** via the NetBird API (`/api/dns/zones/{id}/records`, PAT in sops `secrets/netbird-pat.yaml`). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (`.79`) — correct.
|
||
- This **killed the manual drift** that caused the whole `auth`/`sso`/`nfs-export` saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.
|
||
|
||
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
|
||
|
||
> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
|
||
|
||
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
|
||
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
|
||
|
||
**Fixed:** added `/etc/cron.d/dns-sync` (`*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1`).
|
||
|
||
Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backend-health`) that runs `scripts/check-caddy-backends.sh` every 10 minutes.
|
||
|
||
### 2026-06-02 — 8 LXCs moved from DHCP to static IP
|
||
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
|
||
|
||
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
|
||
Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
|
||
|
||
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
|
||
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
||
|
||
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
|
||
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
|
||
|
||
### 2026-05-14 — `zimaos.hubris.network` added (Caddy-fronted, standard pattern)
|
||
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
|
||
|
||
### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
|
||
New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
|
||
|
||
### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
|
||
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.
|
||
|
||
### 2026-04-28 — wiki entry created
|
||
Initial documentation. 16 active entries.
|
||
|
||
### 2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79
|
||
Public path now lands on the VPS traefik, not the old yunohost. Necessary for the [public ingress](ingress.md) pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.
|
||
|
||
### 2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)
|
||
Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.
|
||
|
||
### 2026-04-21 — dnsmasq stood up on LXC 124
|
||
Co-located with Authentik. Initial entries cover everything routed through Caddy.
|