oikos/infrastructure/dns.md

# DNS — split-horizon `*.hubris.network`

LAN clients resolve `*.hubris.network` to the [Caddy reverse proxy](../containers/121-caddy.md) (`192.168.8.175`). Public clients resolve to the IONOS VPS (`82.165.190.79`) via an IONOS wildcard, where they hit the [VPS traefik public ingress](ingress.md).

There is **no wildcard on the LAN side**. Every subdomain needs an explicit entry.

## Components

- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53` — the **single DNS source** for LAN, household, and mesh. (Through 2026-06-21 it also synced A-records into a NetBird managed zone; that replica was retired in Phase 4, see changelog.) Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **All LXCs** now point at Technitium (`192.168.8.2`) directly (since 2026-06-21 — see changelog). The earlier mix of router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`) — which returned the public IONOS A record and forced `/etc/hosts` overrides — has been removed.

## Live entries (as of 2026-06-04)

```
address=/auth.hubris.network/82.165.190.79   # → VPS, not Caddy (Authentik migrated 2026-05-31)
address=/sso.hubris.network/192.168.8.175    # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
address=/git.hubris.network/192.168.8.175
address=/media.hubris.network/192.168.8.175
address=/paperless.hubris.network/192.168.8.175
address=/books.hubris.network/192.168.8.175
address=/home.hubris.network/192.168.8.175
address=/cloud.hubris.network/192.168.8.175
address=/matrix.hubris.network/192.168.8.175
address=/proxmox.hubris.network/192.168.8.175
address=/docker.hubris.network/192.168.8.175
address=/jellyseerr.hubris.network/192.168.8.175
address=/qbit.hubris.network/192.168.8.175
address=/sab.hubris.network/192.168.8.175
address=/blog.hubris.network/192.168.8.175
address=/photos.hubris.network/192.168.8.175
address=/photos-new.hubris.network/192.168.8.175
address=/artifacto.hubris.network/192.168.8.175
address=/plato.hubris.network/192.168.8.175
address=/zimaos.hubris.network/192.168.8.175
address=/nfs-export.hubris.network/192.168.8.200
```

Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do.

## Why split-horizon

The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.

## The gotcha that cost a debug session (2026-04-22)

Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.

## Recipe — adding a new subdomain

1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)`. That's it — Technitium is the single DNS source; mesh peers forward to it live and LAN/household query it directly. (No managed-zone sync to wait for — retired 2026-06-21, Phase 4.)
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`. On a NetBird peer that caches a stale answer, `netbird service restart` clears its resolver cache.

> Technitium on LXC 107 is the single source of truth. (Through 2026-06-21 a `dns-sync.py` cron mirrored it into a NetBird managed zone; that replica + cron were removed in Phase 4 once mesh peers were forwarding directly — see [changelog](#2026-06-21--dns-single-source-phase-4-complete--managed-zone-removed-technitium-is-the-single-source).)

## Public path — what does and doesn't follow the LAN map

- Hostnames published in [VPS traefik dynamic config](ingress.md) (currently `artifacto.hubris.network`, `blog.hubris.network`) reach a real backend over the netbird mesh.
- Anything else with a `*.hubris.network` URL hits the VPS but isn't routed anywhere — returns 404.
- `netbird.hubris.network` is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.

## Long-term plan

Either:
- Move split-horizon DNS to the LAN router so `*.hubris.network → 192.168.8.175` is answered for every LAN client. Eliminates per-LXC overrides.
- Or, once the [Tailscale → Netbird migration](mesh.md) completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.

## Related
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority

## Changelog

### 2026-06-21 — DNS single-source, Phase 4: COMPLETE — managed zone removed, Technitium is the single source
After upgrading the Mac client `0.68.3 → 0.71.3` (matching mgmt) and clearing its NetBird resolver cache (`netbird service restart`), the managed-zone deletion was retried and **works**: with 0 managed-zone records, the Mac resolves `git`/`cloud`/`nfs-export`/`auth`.hubris.network entirely by forwarding to Technitium (`192.168.8.2`). The iPhone confirmed the same on **cellular** (no LAN/Fritz path — proof it's the mesh-forward path).

So the first deletion attempt (the REVERTED entry below) was a **misdiagnosis**: forwarding wasn't broken — the old 0.68.3 client's resolver cache (`100.122.255.254`) held stale/empty answers and wouldn't clear on `down/up` (only a full daemon restart clears it), and the Mac's dual LAN+mesh resolver paths muddied the test. A direct query (`dig @100.122.255.254 <unsynced-name>`) had in fact shown forwarding working all along.

**Done:**
- Deleted all 23 NetBird managed-zone A-records via API.
- Removed the `*/10` sync cron (`/etc/cron.d/dns-sync`). Kept `/opt/dns-sync/sync.py` + token + a pre-deletion backup as an emergency-restore tool only (run it once to rebuild the managed zone if ever reverting).

**End state — one zone:** Technitium (`192.168.8.2`) is the single authoring + serving source. Mesh peers forward to it (route via `Core` → `192.168.8.0/24`); LAN/household query it directly (Fritz!Box DNSv4 → Technitium). No managed-zone replica, no sync.

**Prereqs for it to keep working:** mesh clients on NetBird **0.71.x+**, and roaming peers in the `Core` group (route) — both satisfied. Other peers (laptops, proxmox-server) pick up forwarding as their resolver caches expire, or after a `netbird service restart`.

**Optional, not done:** flip the Technitium wildcard `*.hubris.network` from `→ 82.165.190.79` (VPS, mirrors public) to `→ 192.168.8.175` (Caddy) so new Caddy services need zero DNS entries. Deferred — decide separately.

### 2026-06-21 — DNS single-source, Phase 4: ATTEMPTED + REVERTED — managed zone is load-bearing
Tried to collapse to a single zone by deleting the NetBird managed-zone replica (and pausing `dns-sync`). Result: **mesh-peer DNS broke.** With the managed zone gone, the Mac (NetBird 0.68.3) failed to resolve `git`/`cloud`/`nfs-export`.hubris.network via the NetBird resolver (`100.122.255.254`) — the `home-lab-dns` nameserver group (`→ 192.168.8.2`) reports `Available` but does **not** actually serve forwarded queries. Restored the managed zone via `dns-sync.py` (23 records) and re-enabled the cron; resolution recovered immediately.

**Correction to the Phase 2 entry below:** the missing route was *a* real problem (it blocked roaming-peer *connectivity* to services — the actual user-facing win), but it was **not** the whole story. DNS forwarding to the routed-LAN IP `192.168.8.2` still does not work for mesh peers, so the original "NetBird won't forward to Technitium" finding stands and the managed zone stays.

**To actually finish single-source later**, forwarding must first be made to work. Two candidates, untried:
- Upgrade NetBird clients to 0.71.x (the Mac is 0.68.3 — version skew with mgmt 0.71.3 is a known source of resolver bugs), then retest.
- Point `home-lab-dns` at a **mesh-native** DNS IP — join CT 107 to the mesh and use its `100.122.x` address instead of the routed-LAN `192.168.8.2` (the original Phase 2 hypothesis; needs a brief 107 restart for `/dev/net/tun`).

Net state after today: Phase 1 (on-prem single resolver) and Phase 2 (roaming route → iPhone reaches the homelab) stand and deliver the practical goals. The managed-zone replica + sync are **retained** as load-bearing.

### 2026-06-21 — DNS single-source, Phase 2: roaming-peer route fixed (see Phase 4 correction above — forwarding still doesn't serve queries; this fixed *connectivity*, not DNS)
The long-standing belief that "NetBird won't forward to Technitium for mesh peers" (which is *why* the managed-zone sync was built) turned out to be **wrong**. The NetBird API showed the real cause:

- The `home-lab-dns` nameserver group (→ `192.168.8.2`, match-domain `hubris.network`) was already applied to **all** peers (via the `All` group). So every peer *had* the forwarding rule.
- But the `192.168.8.0/24` route (network resource `home-lab-network`) was distributed to the **`Services`** group only = `{netbird-ionos, proxmox-server}`. Roaming peers (`dtoro-iphone`, `mac-mini`, `republic-laptop`, `muli-laptop`, `ludo-mini`) are in **`Core`**, which had **no route to the homelab subnet** → they couldn't reach `192.168.8.2` → forwarding silently failed (`Networks: -`).

**Fix:** added the `Core` group to the `home-lab-network` resource distribution (now `[Services, Core]`) via `PUT /api/networks/.../resources/...`. Roaming peers immediately picked up `Networks: 192.168.8.0/24` and `[192.168.8.2:53] for [hubris.network] is Available`. No CT 107 mesh-join / TUN / restart was needed (the original Phase 2 hypothesis is obsolete). This also gives roaming devices full homelab **service** access, not just DNS.

The managed zone is still in place as a fallback pending the Phase 3 roaming test (iPhone on cellular); Phase 4 then deletes the managed zone + `dns-sync` cron. Rollback: PUT the resource back to `[Services]` only.

### 2026-06-21 — DNS single-source, Phase 1: on-prem LXCs decoupled from NetBird
Goal: collapse the three overlapping DNS sources (Technitium + NetBird managed zone + per-LXC band-aids) toward **one zone**, keeping NetBird. Phase 1 (the safe, mesh-independent half) is done:

- **Every homelab LXC now resolves via Technitium (`192.168.8.2`).** Fixed 8 boxes that were on a dead resolver, the router, or Tailscale MagicDNS:
  - `192.168.8.180` (dead ex-Authentik): 102, 106, 126
  - `192.168.8.1` (router → public answer): 101
  - `100.100.100.100` (Tailscale MagicDNS): 104, 105, 114, 119 — also ran `tailscale set --accept-dns=false` so `tailscaled` stops rewriting `/etc/resolv.conf`.
  - Already correct (`.2`): 103, 118, 120, 121, 122. CT 107 stays on `1.1.1.1` by design (no self-dependency).
- **Removed the redundant `/etc/hosts` band-aids** (`auth`/`mcp`/`secrets` → `192.168.8.175`) on 101, 103, 104, 105, 114, 118, 120, 121, 122, 126; disabled `hubris-hosts-override.service` where enabled. Backups at `/etc/hosts.bak-dnsplan`. PVE-managed lines and self-hostname maps preserved. Technitium already returns identical-or-better answers (verified: `auth → 82.165.190.79`, `mcp`/`secrets`/`cloud`/`git` → `192.168.8.175`).
- **Nextcloud (114):** its documented Guzzle-workaround dnsmasq is not running; resolves correctly straight from Technitium, so no special-casing remains.
- Net effect: **NetBird's DNS now only matters for off-LAN roaming peers** (Tier 2). On-prem (LXCs + household via Fritz!Box→Technitium) is fully NetBird-independent — so dropping the managed zone later can no longer break on-LAN resolution. Phases 2–4 (mesh-IP forwarding, roaming test, managed-zone + sync deletion) still pending.

### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
- Caddy: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxied to VPS
- `header_up Host auth.hubris.network` added to strip `:443` from upstream Host header
- All 9 LXCs' /etc/hosts updated: `auth.hubris.network → 192.168.8.175` (Caddy proxy)
- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:

- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
- A **sync job on [dns (107)](../containers/107-dns.md)** (`/opt/dns-sync/sync.py`, cron */10) reconciles Technitium's named A-records → the **NetBird managed DNS zone** via the NetBird API (`/api/dns/zones/{id}/records`, PAT in sops `secrets/netbird-pat.yaml`). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (`.79`) — correct.
- This **killed the manual drift** that caused the whole `auth`/`sso`/`nfs-export` saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.

**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.

> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).

### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.

**Fixed:** added `/etc/cron.d/dns-sync` (`*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1`).

Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backend-health`) that runs `scripts/check-caddy-backends.sh` every 10 minutes.

### 2026-06-02 — 8 LXCs moved from DHCP to static IP
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).

### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.

### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).

### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.

### 2026-05-14 — `zimaos.hubris.network` added (Caddy-fronted, standard pattern)
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).

### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).

### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.

### 2026-04-28 — wiki entry created
Initial documentation. 16 active entries.

### 2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79
Public path now lands on the VPS traefik, not the old yunohost. Necessary for the [public ingress](ingress.md) pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.

### 2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)
Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.

### 2026-04-21 — dnsmasq stood up on LXC 124
Co-located with Authentik. Initial entries cover everything routed through Caddy.