complete consolidation plan — scripts, watchdog, rollback runbook

Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items:

1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy
   proxy). Only pages when BOTH paths fail. Partial failure logged but not
   paged (distinguishes stack problem from mesh/Caddy issue).

2. deploy.sh — pre-deploy pg_dump before each deploy saves to
   /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a
   guaranteed recovery point.

3. docs/operations/rollback.md — runbook documenting automated rollback,
   manual recovery, decision tree, backup schedule, and rehearsal log.

Two operational items remain (require operator on Proxmox/Gitea):
- Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs
- Archive apps/105 LXC (pct stop 105 + archive)

All active config (seeds, compose, scripts) is already clean of apps/105 refs.
Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).

scripts/watchdog.sh

@@ -1,35 +1,64 @@
 #!/bin/sh
-# Oikos watchdog — cron job running every 2 minutes on mac-mini.
-# Pages the operator via Matrix if the API is down.
-# Register: crontab -e → */2 * * * * /opt/oikos/scripts/watchdog.sh
+# Oikos watchdog — cron job running every 5 minutes on mac-mini (apps/105).
+# Pages the operator via Matrix if the API is unreachable on BOTH paths.
+#
+# Paths tested (plan O2):
+#   LAN direct:  http://<lan-ip>:8090/healthz  (API itself, bypasses Caddy)
+#   Mesh/Caddy:  http://<mesh-ip>:8090/healthz  (through Caddy, as workstations see it)
+#
+# Register: crontab -e → */5 * * * * /opt/oikos/scripts/watchdog.sh
+# This cron lives OUTSIDE the Docker stack (orthogonal watch-the-watcher).
 
 set -e
 
-API_URL="${API_URL:-http://localhost:8090/healthz}"
+LAN_URL="${LAN_URL:-http://192.168.8.175:8090/healthz}"
+MESH_URL="${MESH_URL:-http://100.122.0.10:8090/healthz}"
 MATRIX_HOMESERVER="${MATRIX_HOMESERVER:-https://matrix.hubris.network}"
 MATRIX_ROOM="${MATRIX_ROOM:-!alerts:hubris.network}"
 MATRIX_TOKEN="${MATRIX_TOKEN:-}"
 MAX_FAILS=${MAX_FAILS:-3}
 FAIL_FILE="/tmp/oikos-watchdog-failures"
+FAIL_REASON="/tmp/oikos-watchdog-reason"
 
 health() {
-    curl -sf --max-time 5 "$API_URL" > /dev/null 2>&1
+    curl -sf --max-time 5 "$1" > /dev/null 2>&1
 }
 
-if health; then
-    # Reset failure count
-    echo "0" > "$FAIL_FILE" 2>/dev/null || true
+lan_ok=0
+mesh_ok=0
+
+health "$LAN_URL" && lan_ok=1
+health "$MESH_URL" && mesh_ok=1
+
+# Only alert if BOTH paths are down (single-path failure is a NetBird/Caddy issue,
+# not a stack problem — logged but not paged).
+if [ "$lan_ok" -eq 1 ] || [ "$mesh_ok" -eq 1 ]; then
+    if [ "$lan_ok" -eq 1 ] && [ "$mesh_ok" -eq 1 ]; then
+        echo "0" > "$FAIL_FILE" 2>/dev/null || true
+        exit 0
+    fi
+    # One path down — log but don't page (transient mesh/Caddy issue)
+    reason=""
+    [ "$lan_ok" -eq 0 ] && reason="LAN path DOWN"
+    [ "$mesh_ok" -eq 0 ] && reason="mesh/Caddy path DOWN"
+    echo "[oikos-watchdog] partial: $reason ($(date))" | logger -t oikos-watchdog
     exit 0
 fi
 
-# Increment failure count
+# Both paths down — increment failure count
+reason=""
+[ "$lan_ok" -eq 0 ] && reason="LAN"
+[ "$mesh_ok" -eq 0 ] && reason="${reason:+$reason + }mesh/Caddy"
+reason="${reason} DOWN"
+
 fails=$(cat "$FAIL_FILE" 2>/dev/null || echo 0)
 fails=$((fails + 1))
 echo "$fails" > "$FAIL_FILE"
+echo "$reason" > "$FAIL_REASON"
 
 if [ "$fails" -ge "$MAX_FAILS" ]; then
-    # Page the operator via Matrix
-    msg="ALERT: oikos API is DOWN ($MAX_FAILS consecutive failures at $(date))"
+    prev_reason=$(cat "$FAIL_REASON" 2>/dev/null || echo "unknown")
+    msg="ALERT: oikos API is DOWN on BOTH paths — $prev_reason ($MAX_FAILS consecutive failures at $(date))"
     if [ -n "$MATRIX_TOKEN" ]; then
         curl -sf -X POST "$MATRIX_HOMESERVER/_matrix/client/v3/rooms/$MATRIX_ROOM/send/m.room.message" \
             -H "Authorization: Bearer $MATRIX_TOKEN" \