fix: gitea DHCP drift + add caddy backend validation script
This commit is contained in:
129
.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md
Normal file
129
.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Plan: Prevent DHCP IP drift from breaking Caddy backends
|
||||
|
||||
**Date:** 2026-06-05
|
||||
**Slug:** prevent-dhcp-ip-drift
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Eliminate the root cause of services becoming unreachable when DHCP lease renewals change backend IPs that Caddy's `reverse_proxy` directives hardcode.
|
||||
|
||||
**Triggering incident:** Paperless (LXC 103) and HAOS (VM 108) had DHCP-assigned IPs change from `.130→.243` and `.101→.241` respectively. Caddyfile still pointed at the old IPs → services unreachable from iPhone on Netbird.
|
||||
|
||||
## Current context
|
||||
|
||||
### DHCP vs static IP inventory
|
||||
|
||||
| Machine | Type | PVE ID | Current IP | Allocation method | Status |
|
||||
|---------|------|--------|-----------|-------------------|--------|
|
||||
| **jellyfin** | LXC | 101 | 192.168.8.206 | Static (`ip=.../24`) | ✅ |
|
||||
| **paperless** | LXC | 103 | 192.168.8.243 | **DHCP** (`ip=dhcp`) | ❌ broken, hotfixed to .243 |
|
||||
| **gitea** | LXC | 104 | 192.168.8.121 | Static | ✅ |
|
||||
| **apps** | LXC | 105 | 192.168.8.205 | Static | ✅ |
|
||||
| **nextcloud** | LXC | 114 | 192.168.8.224 | Static | ✅ |
|
||||
| **elementsynapse** | LXC | 118 | 192.168.8.239 | Static | ✅ |
|
||||
| **mule-images** | LXC | 120 | 192.168.8.136 | Static | ✅ |
|
||||
| **caddy** | LXC | 121 | 192.168.8.175 | Static | ✅ |
|
||||
| **arriman** | LXC | 122 | 192.168.8.132 | Static | ✅ |
|
||||
| **sophia** | LXC | 119 | 192.168.8.157 | Static? | ? (not in 2026-06-02 list) |
|
||||
| **nfs-export** | LXC | 102 | 192.168.8.200 | Static? | ? |
|
||||
| **plato** | LXC | 126 | 192.168.8.190 | Static? | ? |
|
||||
| **HAOS** | VM | 108 | 192.168.8.241 | **DHCP** (VM — OS-managed) | ❌ broken, hotfixed to .241 |
|
||||
| **zimaos** | VM | 100 | 192.168.8.195 | DHCP (known stale lease, see note) | ⚠️ open issue |
|
||||
| **authentik** | — | — | — | migrated to VPS (external) | N/A |
|
||||
|
||||
### Infrastructure facts
|
||||
|
||||
- **DHCP server:** Technitium on CT 107 (192.168.8.2), pool `.241–.254`
|
||||
- **Static IP range:** `.101–.239`
|
||||
- **DNS:** Split-horizon on Technitium — `*.hubris.network → 192.168.8.175` (Caddy itself)
|
||||
- **Caddyfile:** Has 29 `reverse_proxy` directives, all using **hardcoded IP:port**
|
||||
- **Caddy reload:** Auto-deployed via webhook on git push to `dtoro/caddy-conf`
|
||||
- **Documentation:** `inventory.yaml` updated, `hosts/*.yaml` stale-before-regenerate
|
||||
|
||||
### Why it happened
|
||||
|
||||
1. Paperless LXC 103 was **missed** during the 2026-06-02 static-IP migration (Proxmox config still shows `ip=dhcp`)
|
||||
2. HAOS VM 108 uses DHCP internally (HAOS manages its own network — can't just `pct set`)
|
||||
3. Both got new IPs from the Technitium `.241–.254` pool after a lease renewal
|
||||
4. Caddyfile still pointed at the old static-range IPs → connection refused
|
||||
|
||||
---
|
||||
|
||||
## Proposed approach
|
||||
|
||||
Three-layer solution:
|
||||
|
||||
### Layer 1: Fix the immediate offenders (static assignment)
|
||||
|
||||
**Paperless LXC 103:**
|
||||
- `pct set 103 --net0 name=eth0,bridge=vmbr0,gw=192.168.8.1,hwaddr=BC:24:11:0A:8D:C2,ip=192.168.8.130/24,ip6=auto,type=veth`
|
||||
- Inside the LXC, update `/etc/network/interfaces` to match
|
||||
- Reboot/restart networking
|
||||
|
||||
**HAOS VM 108:**
|
||||
- Set a **DHCP reservation** in Technitium for the VM's MAC address, pinning it to `192.168.8.101`
|
||||
- This avoids needing to reconfigure HAOS internally (which is tedious)
|
||||
- Alternatively: use `ha network update` from the HAOS console to set a static IP
|
||||
|
||||
### Layer 2: Audit and fix all remaining DHCP hosts
|
||||
|
||||
Check every LXC/VM hosted on hubris:
|
||||
```bash
|
||||
for ct in $(pct list | awk 'NR>1{print $1}'); do
|
||||
echo "=== CT $ct ==="
|
||||
pct config $ct | grep "^net0"
|
||||
done
|
||||
```
|
||||
Any with `ip=dhcp` that Caddy reverse-proxies to → convert to static.
|
||||
|
||||
**Known candidates to check:**
|
||||
- CT 102 (nfs-export) — `.200` but not in Caddy. May not need static.
|
||||
- CT 119 (sophia) — `.157` — is this static or DHCP? Not sure.
|
||||
- CT 126 (plato) — `.190` — same question.
|
||||
- VM 100 (zimaos) — `.195` but known to have a stale lease (see 2026-06-03 changelog)
|
||||
|
||||
### Layer 3: Add validation
|
||||
|
||||
Create a script that runs periodically (cron or homelab cronjob):
|
||||
|
||||
**`/opt/homelab-context/scripts/check-caddy-backends.sh`:**
|
||||
1. Parse `/etc/caddy/Caddyfile` on CT 121 to extract all `reverse_proxy IP:port` targets
|
||||
2. For each `IP:port`, attempt a TCP connect (timeout 3s)
|
||||
3. Report any that fail
|
||||
|
||||
Could also run as a homelab cron job that notifies dtoro on Matrix if a backend is unreachable.
|
||||
|
||||
This catches any future drift proactively (before a user reports it).
|
||||
|
||||
### Files likely to change
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `inventory.yaml` | May update paperless/HAOS IPs if we choose different static IPs |
|
||||
| LXC 103 Proxmux config (via `pct set`) | Set static IP |
|
||||
| Technitium DHCP reservations | Add HAOS reservation |
|
||||
| `/etc/caddy/Caddyfile` on CT 121 | Already fixed — only changes again if we re-assign paperless IP to `.130` |
|
||||
| `scripts/check-caddy-backends.sh` | New validation script (new file in homelab-context) |
|
||||
|
||||
### Risks / Tradeoffs
|
||||
|
||||
- **Pinning paperless to `.130`** — if the LXC was reinstalled since then, `.130` may already be in use. Verify first with `arp-scan` or `nmap`.
|
||||
- **HAOS static IP via Technitium reservation** vs **inside HAOS**: Technitium reservation is simpler (no HA config changes), but if HAOS's DHCP lease expires and the Technitium server is down, the reservation won't help. A static IP inside HAOS is more robust but requires poking the HA console.
|
||||
- **Validation script false positives** — a service might be legitimately down for maintenance. The script should be a warning, not an alert.
|
||||
- **Caddy reload** — each Caddyfile edit triggers an auto-reload via webhook. If the backend is down during reload, Caddy itself stays up (it's just a reverse_proxy target).
|
||||
|
||||
### Verification
|
||||
|
||||
1. After setting paperless static: `ssh root@192.168.8.175 "curl -s -o /dev/null -w '%{http_code}' http://192.168.8.130:8000"` → 302
|
||||
2. After Technitium HAOS reservation: `curl -s -o /dev/null -w '%{http_code}' http://192.168.8.101:8123` → 200
|
||||
3. Run validation script → all targets reachable
|
||||
4. Confirm from iPhone: both `paperless.hubris.network` and `home.hubris.network` load
|
||||
|
||||
### Open questions
|
||||
|
||||
1. Should paperless go back to `.130` (its original), or stay at `.243` (current)? Going back to `.130` means updating the Caddyfile again, but keeps the static range allocation consistent.
|
||||
2. HAOS: Technitium reservation or HAOS-internal static config? Reservation is easier; HAOS-internal is more robust.
|
||||
3. Should the Caddyfile validation script run as a homelab cron job, or as a cron on the caddy LXC itself?
|
||||
4. ZimaOS (VM 100) — should we also pin its IP while we're at it?
|
||||
Reference in New Issue
Block a user