oikos/.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md

Plan: Prevent DHCP IP drift from breaking Caddy backends

Date: 2026-06-05
Slug: prevent-dhcp-ip-drift

---

Goal

Eliminate the root cause of services becoming unreachable when DHCP lease renewals change backend IPs that Caddy's reverse_proxy directives hardcode.

Triggering incident: Paperless (LXC 103) and HAOS (VM 108) had DHCP-assigned IPs change from .130→.243 and .101→.241 respectively. Caddyfile still pointed at the old IPs → services unreachable from iPhone on Netbird.

Current context

DHCP vs static IP inventory

Machine	Type	PVE ID	Current IP	Allocation method	Status
jellyfin	LXC	101	192.168.8.206	Static (ip=.../24)	✅
paperless	LXC	103	192.168.8.243	DHCP (ip=dhcp)	❌ broken, hotfixed to .243
gitea	LXC	104	192.168.8.121	Static	✅
apps	LXC	105	192.168.8.205	Static	✅
nextcloud	LXC	114	192.168.8.224	Static	✅
elementsynapse	LXC	118	192.168.8.239	Static	✅
mule-images	LXC	120	192.168.8.136	Static	✅
caddy	LXC	121	192.168.8.175	Static	✅
arriman	LXC	122	192.168.8.132	Static	✅
sophia	LXC	119	192.168.8.157	Static?	? (not in 2026-06-02 list)
nfs-export	LXC	102	192.168.8.200	Static?	?
plato	LXC	126	192.168.8.190	Static?	?
HAOS	VM	108	192.168.8.241	DHCP (VM — OS-managed)	❌ broken, hotfixed to .241
zimaos	VM	100	192.168.8.195	DHCP (known stale lease, see note)	⚠️ open issue
authentik	—	—	—	migrated to VPS (external)	N/A

Infrastructure facts

• DHCP server: Technitium on CT 107 (192.168.8.2), pool .241–.254
• Static IP range: .101–.239
• DNS: Split-horizon on Technitium — *.hubris.network → 192.168.8.175 (Caddy itself)
• Caddyfile: Has 29 reverse_proxy directives, all using hardcoded IP:port
• Caddy reload: Auto-deployed via webhook on git push to dtoro/caddy-conf
• Documentation: inventory.yaml updated, hosts/*.yaml stale-before-regenerate

Why it happened

1. Paperless LXC 103 was missed during the 2026-06-02 static-IP migration (Proxmox config still shows ip=dhcp)
2. HAOS VM 108 uses DHCP internally (HAOS manages its own network — can't just pct set)
3. Both got new IPs from the Technitium .241–.254 pool after a lease renewal
4. Caddyfile still pointed at the old static-range IPs → connection refused

---

Proposed approach

Three-layer solution:

Layer 1: Fix the immediate offenders (static assignment)

Paperless LXC 103:

• pct set 103 --net0 name=eth0,bridge=vmbr0,gw=192.168.8.1,hwaddr=BC:24:11:0A:8D:C2,ip=192.168.8.130/24,ip6=auto,type=veth
• Inside the LXC, update /etc/network/interfaces to match
• Reboot/restart networking

HAOS VM 108:

• Set a DHCP reservation in Technitium for the VM's MAC address, pinning it to 192.168.8.101
• This avoids needing to reconfigure HAOS internally (which is tedious)
• Alternatively: use ha network update from the HAOS console to set a static IP

Layer 2: Audit and fix all remaining DHCP hosts

Check every LXC/VM hosted on hubris:

for ct in $(pct list | awk 'NR>1{print $1}'); do
  echo "=== CT $ct ==="
  pct config $ct | grep "^net0"
done

Any with ip=dhcp that Caddy reverse-proxies to → convert to static.

Known candidates to check:

• CT 102 (nfs-export) — .200 but not in Caddy. May not need static.
• CT 119 (sophia) — .157 — is this static or DHCP? Not sure.
• CT 126 (plato) — .190 — same question.
• VM 100 (zimaos) — .195 but known to have a stale lease (see 2026-06-03 changelog)

Layer 3: Add validation

Create a script that runs periodically (cron or homelab cronjob):

/opt/homelab-context/scripts/check-caddy-backends.sh:

1. Parse /etc/caddy/Caddyfile on CT 121 to extract all reverse_proxy IP:port targets
2. For each IP:port, attempt a TCP connect (timeout 3s)
3. Report any that fail

Could also run as a homelab cron job that notifies dtoro on Matrix if a backend is unreachable.

This catches any future drift proactively (before a user reports it).

Files likely to change

File	Change
inventory.yaml	May update paperless/HAOS IPs if we choose different static IPs
LXC 103 Proxmux config (via pct set)	Set static IP
Technitium DHCP reservations	Add HAOS reservation
/etc/caddy/Caddyfile on CT 121	Already fixed — only changes again if we re-assign paperless IP to .130
scripts/check-caddy-backends.sh	New validation script (new file in homelab-context)

Risks / Tradeoffs

• Pinning paperless to .130 — if the LXC was reinstalled since then, .130 may already be in use. Verify first with arp-scan or nmap.
• HAOS static IP via Technitium reservation vs inside HAOS: Technitium reservation is simpler (no HA config changes), but if HAOS's DHCP lease expires and the Technitium server is down, the reservation won't help. A static IP inside HAOS is more robust but requires poking the HA console.
• Validation script false positives — a service might be legitimately down for maintenance. The script should be a warning, not an alert.
• Caddy reload — each Caddyfile edit triggers an auto-reload via webhook. If the backend is down during reload, Caddy itself stays up (it's just a reverse_proxy target).

Verification

1. After setting paperless static: ssh root@192.168.8.175 "curl -s -o /dev/null -w '%{http_code}' http://192.168.8.130:8000" → 302
2. After Technitium HAOS reservation: curl -s -o /dev/null -w '%{http_code}' http://192.168.8.101:8123 → 200
3. Run validation script → all targets reachable
4. Confirm from iPhone: both paperless.hubris.network and home.hubris.network load

Open questions

1. Should paperless go back to .130 (its original), or stay at .243 (current)? Going back to .130 means updating the Caddyfile again, but keeps the static range allocation consistent.
2. HAOS: Technitium reservation or HAOS-internal static config? Reservation is easier; HAOS-internal is more robust.
3. Should the Caddyfile validation script run as a homelab cron job, or as a cron on the caddy LXC itself?
4. ZimaOS (VM 100) — should we also pin its IP while we're at it?