oikos/infrastructure/dns.md

DNS — split-horizon *.hubris.network

LAN clients resolve *.hubris.network to the Caddy reverse proxy (192.168.8.175). Public clients resolve to the IONOS VPS (82.165.190.79) via an IONOS wildcard, where they hit the VPS traefik public ingress.

There is no wildcard on the LAN side. Every subdomain needs an explicit entry.

Components

• Authoritative public DNS: IONOS. *.hubris.network → 82.165.190.79 (was 74.118.126.4 until 2026-04-22).
• LAN authoritative for hubris.network records: Technitium DNS on dns (107) at 192.168.8.2:53. Syncs A records to the NetBird managed DNS zone via cron (see dns-sync.py). Formerly dnsmasq on authentik (124) (decommissioned 2026-06-04).
• PVE host (192.168.8.77): resolver is the local Netbird daemon at 100.122.38.109:53, which forwards to the LAN/upstream and learns hubris.network answers via that path. netbird status says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
• Some LXCs keep router DNS (192.168.8.1) or Tailscale MagicDNS (100.100.100.100), both of which return the public IONOS A record. Those LXCs need either a /etc/hosts override or local dnsmasq — see mesh migration for which technique applies where.

Live entries (as of 2026-06-04)

address=/auth.hubris.network/82.165.190.79   # → VPS, not Caddy (Authentik migrated 2026-05-31)
address=/sso.hubris.network/192.168.8.175    # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
address=/git.hubris.network/192.168.8.175
address=/media.hubris.network/192.168.8.175
address=/paperless.hubris.network/192.168.8.175
address=/books.hubris.network/192.168.8.175
address=/home.hubris.network/192.168.8.175
address=/cloud.hubris.network/192.168.8.175
address=/matrix.hubris.network/192.168.8.175
address=/proxmox.hubris.network/192.168.8.175
address=/docker.hubris.network/192.168.8.175
address=/jellyseerr.hubris.network/192.168.8.175
address=/qbit.hubris.network/192.168.8.175
address=/sab.hubris.network/192.168.8.175
address=/blog.hubris.network/192.168.8.175
address=/photos.hubris.network/192.168.8.175
address=/photos-new.hubris.network/192.168.8.175
address=/artifacto.hubris.network/192.168.8.175
address=/zimaos.hubris.network/192.168.8.175
address=/teddy.hubris.network/192.168.8.175
address=/nfs-export.hubris.network/192.168.8.200

Non-hubris.network override (Toniebox device traffic):

address=/prod.de.bb-online.com/192.168.8.214   # → TeddyCloud (131) direct on :443

This intercepts Toniebox DNS locally without touching public DNS. The dns-sync.py cron on LXC 107 skips non-hubris.network records — this entry is Technitium-only.

Note: nfs-export.hubris.network is the only .hubris.network entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses caddy (121) because NFS is L4, not HTTP — Caddy has nothing to do.

Why split-horizon

The IONOS wildcard points at the VPS for public ingress (per-host routers in VPS traefik). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The Technitium DNS override on 192.168.8.2 keeps LAN traffic on the home Caddy.

The gotcha that cost a debug session (2026-04-22)

Creating a new Caddyfile site block is necessary but not sufficient. Without the Technitium entry on dns (107), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.

Recipe — adding a new subdomain

1. Edit /etc/caddy/Caddyfile on caddy (121), commit + push to dtoro/caddy-conf. Webhook reloads caddy. See auto-deploy.
2. Add the A record in the Technitium UI at dns (107) — the NetBird managed DNS zone sync picks it up within ~10 minutes via cron. Or add directly to the NetBird managed zone via API if you need it faster.
3. Verify: dig @192.168.8.2 +short <new>.hubris.network → 192.168.8.175.
4. On macOS clients, flush: sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder.

The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the scripts/dns-sync.py cron on 107 reconciles them and reaps stale records. See dns.md changelog 2026-06-03.

Public path — what does and doesn't follow the LAN map

• Hostnames published in VPS traefik dynamic config (currently artifacto.hubris.network, blog.hubris.network) reach a real backend over the netbird mesh.
• Anything else with a *.hubris.network URL hits the VPS but isn't routed anywhere — returns 404.
• netbird.hubris.network is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.

Long-term plan

Either:

• Move split-horizon DNS to the LAN router so *.hubris.network → 192.168.8.175 is answered for every LAN client. Eliminates per-LXC overrides.
• Or, once the Tailscale → Netbird migration completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.

Related

• Caddy (121) — every LAN entry points here
• Ingress (VPS traefik) — public-side counterpart
• Mesh migration — per-LXC DNS workarounds during the transition
• DNS server (107) — Technitium, current DNS authority

Changelog

2026-06-29 — teddy.hubris.network added; prod.de.bb-online.com override added

TeddyCloud (LXC 131) provisioned. teddy.hubris.network → 192.168.8.175 (Caddy → TeddyCloud web UI at :8080). Non-hubris override prod.de.bb-online.com → 192.168.8.214 routes Toniebox device HTTPS traffic directly to TeddyCloud port 443 — this bypasses Caddy and is Technitium-only (dns-sync cron does not replicate non-hubris.network records to the NetBird managed zone).

2026-06-28 — plato.hubris.network removed

Plato (LXC 126) decommissioned. Technitium entry deleted; dns-sync cron reaped the NetBird managed zone record.

2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved

Household LAN clients (192.168.178.x) now resolve *.hubris.network to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = 192.168.8.2 (User-defined). Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).

• Caddy: auth.hubris.network, authentik snippet, and sso.hubris.network all proxied to VPS

• header_up Host auth.hubris.network added to strip :443 from upstream Host header

• All 9 LXCs' /etc/hosts updated: auth.hubris.network → 192.168.8.175 (Caddy proxy)

• Inventory: removed hosts.authentik, renamed dnsmasq service → dns

• Docs: 124-authentik.md deleted; dns.md references updated to Technitium (107) The "delete NetBird managed zone → forward everything to Technitium" plan was abandoned — NetBird's DNS defeats it: it won't apply a nameserver group that contains the peer's own mesh IP (the Mac's 100.122.234.17 → Nameservers: 0/0 Available), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the managed zone was doing all the real work; disabling it broke all mesh resolution). So the model is now:

• Technitium (192.168.8.2) is the single place you author DNS (UI/API, MX/SPF/CAA, full zone).

• A sync job on dns (107) (/opt/dns-sync/sync.py, cron */10) reconciles Technitium's named A-records → the NetBird managed DNS zone via the NetBird API (/api/dns/zones/{id}/records, PAT in sops secrets/netbird-pat.yaml). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (.79) — correct.

• This killed the manual drift that caused the whole auth/sso/nfs-export saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.

Cleanup done same day: removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's zoneTransfer=Allow; fixed home-lab-dns group → [192.168.8.2] (dropped the self-referencing Mac IP → now 1/1 Available); deleted the vestigial Proxmox Names group.

Reference: scripts/dns-sync.py. The sync's source of truth is Technitium; it deletes NetBird records absent from Technitium (so obsolete names like files, photos-new get reaped).

2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)

The dns-sync.py script on LXC 107 had been placed at /opt/dns-sync/sync.py on 2026-06-04 but no crontab was configured — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.

Fixed: added /etc/cron.d/dns-sync (*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1).

Also added a Caddy backend health check cron on hubris (/etc/cron.d/caddy-backend-health) that runs scripts/check-caddy-backends.sh every 10 minutes.

2026-06-02 — 8 LXCs moved from DHCP to static IP

All LXCs that Caddy reverse-proxies to by IP were on ip=dhcp and could float on reboot (arriman got a different lease mid-session and broke). Fixed via pct set + in-LXC /etc/network/interfaces. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See arriman changelog.

2026-06-01 — dnsmasq replaced by Technitium on dns (107); LXC 124 retired

Split-horizon DNS moved off 124 to a dedicated Technitium LXC at 192.168.8.2 (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird home-lab-dns nameserver group cut over to 192.168.8.2 (with .180 as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, LXC 124 shut down. Caveat: the NetBird managed DNS zone still answers most app names directly (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). Action needed: update router DHCP DNS from the dead .180 → 192.168.8.2 for any plain-LAN (non-mesh) clients.

2026-05-31 — auth.hubris.network re-pointed to the VPS (82.165.190.79)

Authentik migrated off LXC 124 onto the VPS (see investigation). The dnsmasq entry changed from 192.168.8.175 (home Caddy) to 82.165.190.79 (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — auth is now a genuinely public service served directly from the VPS. Gotcha logged: the NetBird per-client resolver (100.122.255.254) caches dnsmasq answers and does not clear on netbird down/up; clients needed /etc/hosts overrides or resolvectl flush-caches to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).

2026-05-14 — nfs-export.hubris.network added (direct, non-HTTP)

NFSv4 export server nfs-export (102) at 192.168.8.200. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.

2026-05-14 — zimaos.hubris.network added (Caddy-fronted, standard pattern)

New LAN entry for 100-zimaos → caddy (121) → 192.168.8.195. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (reverse_proxy 192.168.8.195 + IONOS DNS-01 TLS).

2026-05-13 — plato.hubris.network added; files.hubris.network removed

New LAN-only entry for plato (126). Same day, the files.hubris.network entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).

2026-05-12 — files.hubris.network added (since removed 2026-05-13)

Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.

2026-04-28 — wiki entry created

Initial documentation. 16 active entries.

2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79

Public path now lands on the VPS traefik, not the old yunohost. Necessary for the public ingress pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.

2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)

Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.

2026-04-21 — dnsmasq stood up on LXC 124

Co-located with Authentik. Initial entries cover everything routed through Caddy.