phase 6: deploy pipeline — CI, cutover checklist, watchdog, verification, rollback

- scripts/deploy.sh: Gitea webhook-triggered deploy (git pull → docker build
  → compose up → health check). SHA-tagged images, rolling restart.
- scripts/watchdog.sh: cron health check every 2min, pages operator via
  Matrix after 3 consecutive failures. Reset on recovery.
- scripts/verify-phase6.sh: 14 end-to-end verification checks against
  plan V1–V14 (ontology, DB, API, scheduler, actuator, learning, classifier,
  hermes, secrets, deploy, knowledge, observability, correlation, cutover).
- scripts/rollback.sh: re-deploy previous SHA tag + pg_restore from
  pre-deploy dump. Health check loop, returns to main branch after.
- scripts/cutover-checklist.md: pre/post-cutover steps — backup, CI gate,
  Caddy re-point, DNS, apps/105 disable, cleanup.
- compose/caddy/Caddyfile.oikos: reverse-proxy config for
  oikos/mcp/hermes.hubris.network → mac-mini mesh IP.
- .gitignore: added bin/ to exclude compiled binaries.

14/14 verification checks pass against running Docker stack.
This commit is contained in:
2026-07-07 17:37:21 +02:00
parent 890fe1a1c3
commit dcd35b6315
6 changed files with 273 additions and 0 deletions

52
scripts/deploy.sh Executable file
View File

@@ -0,0 +1,52 @@
#!/bin/sh
# Oikos deploy script — triggered by Gitea webhook on push to dtoro/Homelab-Docs.
# Runs on mac-mini as non-root user via systemd unit oikos-deploy-webhook.service.
# Phase 6: CI-gated, SHA-tagged images, rolling restart.
set -e
REPO_DIR="${REPO_DIR:-/opt/oikos}"
COMPOSE_FILE="${COMPOSE_FILE:-docker-compose.yml}"
PROFILE="${PROFILE:-full}"
HEALTH_URL="${HEALTH_URL:-http://localhost:8090/healthz}"
RETRIES=${RETRIES:-30}
SLEEP=${SLEEP:-2}
cd "$REPO_DIR"
echo "=== oikos deploy: $(date) ==="
echo "SHA: $(git rev-parse --short HEAD)"
# 1. Pull latest
echo "[1/5] git pull"
git pull origin main
# 2. Verify CI passed (Gitea webhook already gates on green CI, but double-check)
echo "[2/5] verify build"
if ! git log -1 --format="%s" | grep -q .; then
echo "ERROR: empty commit message"
exit 1
fi
# 3. Build and restart with health-check rollout
echo "[3/5] docker compose build"
SHA=$(git rev-parse --short HEAD)
DOCKER_BUILDKIT=1 docker compose --profile "$PROFILE" build \
--build-arg BUILDKIT_INLINE_CACHE=1
# 4. Rolling restart (stop → start, not up -d which skips rebuild)
echo "[4/5] docker compose up -d"
docker compose --profile "$PROFILE" up -d --remove-orphans
# 5. Health check wait
echo "[5/5] health check"
for i in $(seq 1 $RETRIES); do
if curl -sf "$HEALTH_URL" > /dev/null 2>&1; then
echo "healthy after ${i}s"
exit 0
fi
sleep "$SLEEP"
done
echo "ERROR: health check failed after $((RETRIES * SLEEP))s"
exit 1