oikos/infrastructure/dns.md

DNS — split-horizon *.hubris.network

LAN clients resolve *.hubris.network to the Caddy reverse proxy (192.168.8.175). Public clients resolve to the IONOS VPS (82.165.190.79) via an IONOS wildcard, where they hit the VPS traefik public ingress.

There is no wildcard on the LAN side. Every subdomain needs an explicit entry.

Components

• Authoritative public DNS: IONOS. *.hubris.network → 82.165.190.79 (was 74.118.126.4 until 2026-04-22).
• LAN authoritative for hubris.network records: Technitium DNS on dns (107) at 192.168.8.2:53 — the single DNS source for LAN, household, and mesh. (Through 2026-06-21 it also synced A-records into a NetBird managed zone; that replica was retired in Phase 4, see changelog.) Formerly dnsmasq on authentik (124) (decommissioned 2026-06-04).
• PVE host (192.168.8.77): resolver is the local Netbird daemon at 100.122.38.109:53, which forwards to the LAN/upstream and learns hubris.network answers via that path. netbird status says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
• All LXCs now point at Technitium (192.168.8.2) directly (since 2026-06-21 — see changelog). The earlier mix of router DNS (192.168.8.1) / Tailscale MagicDNS (100.100.100.100) — which returned the public IONOS A record and forced /etc/hosts overrides — has been removed.

Live entries (as of 2026-06-04)

address=/auth.hubris.network/82.165.190.79   # → VPS, not Caddy (Authentik migrated 2026-05-31)
address=/sso.hubris.network/192.168.8.175    # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
address=/git.hubris.network/192.168.8.175
address=/media.hubris.network/192.168.8.175
address=/paperless.hubris.network/192.168.8.175
address=/books.hubris.network/192.168.8.175
address=/home.hubris.network/192.168.8.175
address=/cloud.hubris.network/192.168.8.175
address=/matrix.hubris.network/192.168.8.175
address=/proxmox.hubris.network/192.168.8.175
address=/docker.hubris.network/192.168.8.175
address=/jellyseerr.hubris.network/192.168.8.175
address=/qbit.hubris.network/192.168.8.175
address=/sab.hubris.network/192.168.8.175
address=/blog.hubris.network/192.168.8.175
address=/photos.hubris.network/192.168.8.175
address=/photos-new.hubris.network/192.168.8.175
address=/artifacto.hubris.network/192.168.8.175
address=/plato.hubris.network/192.168.8.175
address=/zimaos.hubris.network/192.168.8.175
address=/nfs-export.hubris.network/192.168.8.200

Note: nfs-export.hubris.network is the only .hubris.network entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses caddy (121) because NFS is L4, not HTTP — Caddy has nothing to do.

Why split-horizon

The IONOS wildcard points at the VPS for public ingress (per-host routers in VPS traefik). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The Technitium DNS override on 192.168.8.2 keeps LAN traffic on the home Caddy.

The gotcha that cost a debug session (2026-04-22)

Creating a new Caddyfile site block is necessary but not sufficient. Without the Technitium entry on dns (107), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.

Recipe — adding a new subdomain

1. Edit /etc/caddy/Caddyfile on caddy (121), commit + push to dtoro/caddy-conf. Webhook reloads caddy. See auto-deploy.
2. Add the A record in the Technitium UI at dns (107). That's it — Technitium is the single DNS source; mesh peers forward to it live and LAN/household query it directly. (No managed-zone sync to wait for — retired 2026-06-21, Phase 4.)
3. Verify: dig @192.168.8.2 +short <new>.hubris.network → 192.168.8.175.
4. On macOS clients, flush: sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder. On a NetBird peer that caches a stale answer, netbird service restart clears its resolver cache.

Technitium on LXC 107 is the single source of truth. (Through 2026-06-21 a dns-sync.py cron mirrored it into a NetBird managed zone; that replica + cron were removed in Phase 4 once mesh peers were forwarding directly — see changelog.)

Public path — what does and doesn't follow the LAN map

• Hostnames published in VPS traefik dynamic config (currently artifacto.hubris.network, blog.hubris.network) reach a real backend over the netbird mesh.
• Anything else with a *.hubris.network URL hits the VPS but isn't routed anywhere — returns 404.
• netbird.hubris.network is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.

Long-term plan

Either:

• Move split-horizon DNS to the LAN router so *.hubris.network → 192.168.8.175 is answered for every LAN client. Eliminates per-LXC overrides.
• Or, once the Tailscale → Netbird migration completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.

Related

• Caddy (121) — every LAN entry points here
• Ingress (VPS traefik) — public-side counterpart
• Mesh migration — per-LXC DNS workarounds during the transition
• DNS server (107) — Technitium, current DNS authority

Changelog

2026-06-21 — DNS single-source, Phase 4: COMPLETE — managed zone removed, Technitium is the single source

After upgrading the Mac client 0.68.3 → 0.71.3 (matching mgmt) and clearing its NetBird resolver cache (netbird service restart), the managed-zone deletion was retried and works: with 0 managed-zone records, the Mac resolves git/cloud/nfs-export/auth.hubris.network entirely by forwarding to Technitium (192.168.8.2). The iPhone confirmed the same on cellular (no LAN/Fritz path — proof it's the mesh-forward path).

So the first deletion attempt (the REVERTED entry below) was a misdiagnosis: forwarding wasn't broken — the old 0.68.3 client's resolver cache (100.122.255.254) held stale/empty answers and wouldn't clear on down/up (only a full daemon restart clears it), and the Mac's dual LAN+mesh resolver paths muddied the test. A direct query (dig @100.122.255.254 <unsynced-name>) had in fact shown forwarding working all along.

Done:

• Deleted all 23 NetBird managed-zone A-records via API.
• Removed the */10 sync cron (/etc/cron.d/dns-sync). Kept /opt/dns-sync/sync.py + token + a pre-deletion backup as an emergency-restore tool only (run it once to rebuild the managed zone if ever reverting).

End state — one zone: Technitium (192.168.8.2) is the single authoring + serving source. Mesh peers forward to it (route via Core → 192.168.8.0/24); LAN/household query it directly (Fritz!Box DNSv4 → Technitium). No managed-zone replica, no sync.

Prereqs for it to keep working: mesh clients on NetBird 0.71.x+, and roaming peers in the Core group (route) — both satisfied. Other peers (laptops, proxmox-server) pick up forwarding as their resolver caches expire, or after a netbird service restart.

Optional, not done: flip the Technitium wildcard *.hubris.network from → 82.165.190.79 (VPS, mirrors public) to → 192.168.8.175 (Caddy) so new Caddy services need zero DNS entries. Deferred — decide separately.

2026-06-21 — DNS single-source, Phase 4: ATTEMPTED + REVERTED — managed zone is load-bearing

Tried to collapse to a single zone by deleting the NetBird managed-zone replica (and pausing dns-sync). Result: mesh-peer DNS broke. With the managed zone gone, the Mac (NetBird 0.68.3) failed to resolve git/cloud/nfs-export.hubris.network via the NetBird resolver (100.122.255.254) — the home-lab-dns nameserver group (→ 192.168.8.2) reports Available but does not actually serve forwarded queries. Restored the managed zone via dns-sync.py (23 records) and re-enabled the cron; resolution recovered immediately.

Correction to the Phase 2 entry below: the missing route was a real problem (it blocked roaming-peer connectivity to services — the actual user-facing win), but it was not the whole story. DNS forwarding to the routed-LAN IP 192.168.8.2 still does not work for mesh peers, so the original "NetBird won't forward to Technitium" finding stands and the managed zone stays.

To actually finish single-source later, forwarding must first be made to work. Two candidates, untried:

• Upgrade NetBird clients to 0.71.x (the Mac is 0.68.3 — version skew with mgmt 0.71.3 is a known source of resolver bugs), then retest.
• Point home-lab-dns at a mesh-native DNS IP — join CT 107 to the mesh and use its 100.122.x address instead of the routed-LAN 192.168.8.2 (the original Phase 2 hypothesis; needs a brief 107 restart for /dev/net/tun).

Net state after today: Phase 1 (on-prem single resolver) and Phase 2 (roaming route → iPhone reaches the homelab) stand and deliver the practical goals. The managed-zone replica + sync are retained as load-bearing.

2026-06-21 — DNS single-source, Phase 2: roaming-peer route fixed (see Phase 4 correction above — forwarding still doesn't serve queries; this fixed connectivity, not DNS)

The long-standing belief that "NetBird won't forward to Technitium for mesh peers" (which is why the managed-zone sync was built) turned out to be wrong. The NetBird API showed the real cause:

• The home-lab-dns nameserver group (→ 192.168.8.2, match-domain hubris.network) was already applied to all peers (via the All group). So every peer had the forwarding rule.
• But the 192.168.8.0/24 route (network resource home-lab-network) was distributed to the Services group only = {netbird-ionos, proxmox-server}. Roaming peers (dtoro-iphone, mac-mini, republic-laptop, muli-laptop, ludo-mini) are in Core, which had no route to the homelab subnet → they couldn't reach 192.168.8.2 → forwarding silently failed (Networks: -).

Fix: added the Core group to the home-lab-network resource distribution (now [Services, Core]) via PUT /api/networks/.../resources/.... Roaming peers immediately picked up Networks: 192.168.8.0/24 and [192.168.8.2:53] for [hubris.network] is Available. No CT 107 mesh-join / TUN / restart was needed (the original Phase 2 hypothesis is obsolete). This also gives roaming devices full homelab service access, not just DNS.

The managed zone is still in place as a fallback pending the Phase 3 roaming test (iPhone on cellular); Phase 4 then deletes the managed zone + dns-sync cron. Rollback: PUT the resource back to [Services] only.

2026-06-21 — DNS single-source, Phase 1: on-prem LXCs decoupled from NetBird

Goal: collapse the three overlapping DNS sources (Technitium + NetBird managed zone + per-LXC band-aids) toward one zone, keeping NetBird. Phase 1 (the safe, mesh-independent half) is done:

• Every homelab LXC now resolves via Technitium (192.168.8.2). Fixed 8 boxes that were on a dead resolver, the router, or Tailscale MagicDNS:
  • 192.168.8.180 (dead ex-Authentik): 102, 106, 126
  • 192.168.8.1 (router → public answer): 101
  • 100.100.100.100 (Tailscale MagicDNS): 104, 105, 114, 119 — also ran tailscale set --accept-dns=false so tailscaled stops rewriting /etc/resolv.conf.
  • Already correct (.2): 103, 118, 120, 121, 122. CT 107 stays on 1.1.1.1 by design (no self-dependency).
• Removed the redundant /etc/hosts band-aids (auth/mcp/secrets → 192.168.8.175) on 101, 103, 104, 105, 114, 118, 120, 121, 122, 126; disabled hubris-hosts-override.service where enabled. Backups at /etc/hosts.bak-dnsplan. PVE-managed lines and self-hostname maps preserved. Technitium already returns identical-or-better answers (verified: auth → 82.165.190.79, mcp/secrets/cloud/git → 192.168.8.175).
• Nextcloud (114): its documented Guzzle-workaround dnsmasq is not running; resolves correctly straight from Technitium, so no special-casing remains.
• Net effect: NetBird's DNS now only matters for off-LAN roaming peers (Tier 2). On-prem (LXCs + household via Fritz!Box→Technitium) is fully NetBird-independent — so dropping the managed zone later can no longer break on-LAN resolution. Phases 2–4 (mesh-IP forwarding, roaming test, managed-zone + sync deletion) still pending.

2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved

Household LAN clients (192.168.178.x) now resolve *.hubris.network to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = 192.168.8.2 (User-defined). Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).

• Caddy: auth.hubris.network, authentik snippet, and sso.hubris.network all proxied to VPS

• header_up Host auth.hubris.network added to strip :443 from upstream Host header

• All 9 LXCs' /etc/hosts updated: auth.hubris.network → 192.168.8.175 (Caddy proxy)

• Inventory: removed hosts.authentik, renamed dnsmasq service → dns

• Docs: 124-authentik.md deleted; dns.md references updated to Technitium (107) The "delete NetBird managed zone → forward everything to Technitium" plan was abandoned — NetBird's DNS defeats it: it won't apply a nameserver group that contains the peer's own mesh IP (the Mac's 100.122.234.17 → Nameservers: 0/0 Available), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the managed zone was doing all the real work; disabling it broke all mesh resolution). So the model is now:

• Technitium (192.168.8.2) is the single place you author DNS (UI/API, MX/SPF/CAA, full zone).

• A sync job on dns (107) (/opt/dns-sync/sync.py, cron */10) reconciles Technitium's named A-records → the NetBird managed DNS zone via the NetBird API (/api/dns/zones/{id}/records, PAT in sops secrets/netbird-pat.yaml). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (.79) — correct.

• This killed the manual drift that caused the whole auth/sso/nfs-export saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.

Cleanup done same day: removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's zoneTransfer=Allow; fixed home-lab-dns group → [192.168.8.2] (dropped the self-referencing Mac IP → now 1/1 Available); deleted the vestigial Proxmox Names group.

Reference: scripts/dns-sync.py. The sync's source of truth is Technitium; it deletes NetBird records absent from Technitium (so obsolete names like files, photos-new get reaped).

2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)

The dns-sync.py script on LXC 107 had been placed at /opt/dns-sync/sync.py on 2026-06-04 but no crontab was configured — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.

Fixed: added /etc/cron.d/dns-sync (*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1).

Also added a Caddy backend health check cron on hubris (/etc/cron.d/caddy-backend-health) that runs scripts/check-caddy-backends.sh every 10 minutes.

2026-06-02 — 8 LXCs moved from DHCP to static IP

All LXCs that Caddy reverse-proxies to by IP were on ip=dhcp and could float on reboot (arriman got a different lease mid-session and broke). Fixed via pct set + in-LXC /etc/network/interfaces. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See arriman changelog.

2026-06-01 — dnsmasq replaced by Technitium on dns (107); LXC 124 retired

Split-horizon DNS moved off 124 to a dedicated Technitium LXC at 192.168.8.2 (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird home-lab-dns nameserver group cut over to 192.168.8.2 (with .180 as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, LXC 124 shut down. Caveat: the NetBird managed DNS zone still answers most app names directly (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). Action needed: update router DHCP DNS from the dead .180 → 192.168.8.2 for any plain-LAN (non-mesh) clients.

2026-05-31 — auth.hubris.network re-pointed to the VPS (82.165.190.79)

Authentik migrated off LXC 124 onto the VPS (see investigation). The dnsmasq entry changed from 192.168.8.175 (home Caddy) to 82.165.190.79 (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — auth is now a genuinely public service served directly from the VPS. Gotcha logged: the NetBird per-client resolver (100.122.255.254) caches dnsmasq answers and does not clear on netbird down/up; clients needed /etc/hosts overrides or resolvectl flush-caches to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).

2026-05-14 — nfs-export.hubris.network added (direct, non-HTTP)

NFSv4 export server nfs-export (102) at 192.168.8.200. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.

2026-05-14 — zimaos.hubris.network added (Caddy-fronted, standard pattern)

New LAN entry for 100-zimaos → caddy (121) → 192.168.8.195. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (reverse_proxy 192.168.8.195 + IONOS DNS-01 TLS).

2026-05-13 — plato.hubris.network added; files.hubris.network removed

New LAN-only entry for plato (126). Same day, the files.hubris.network entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).

2026-05-12 — files.hubris.network added (since removed 2026-05-13)

Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.

2026-04-28 — wiki entry created

Initial documentation. 16 active entries.

2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79

Public path now lands on the VPS traefik, not the old yunohost. Necessary for the public ingress pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.

2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)

Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.

2026-04-21 — dnsmasq stood up on LXC 124

Co-located with Authentik. Initial entries cover everything routed through Caddy.