Files

dtoro 7660e5681c complete consolidation plan — scripts, watchdog, rollback runbook

Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items:

1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy
   proxy). Only pages when BOTH paths fail. Partial failure logged but not
   paged (distinguishes stack problem from mesh/Caddy issue).

2. deploy.sh — pre-deploy pg_dump before each deploy saves to
   /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a
   guaranteed recovery point.

3. docs/operations/rollback.md — runbook documenting automated rollback,
   manual recovery, decision tree, backup schedule, and rehearsal log.

Two operational items remain (require operator on Proxmox/Gitea):
- Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs
- Archive apps/105 LXC (pct stop 105 + archive)

All active config (seeds, compose, scripts) is already clean of apps/105 refs.
Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).

2026-07-08 11:25:43 +02:00

2.5 KiB

Raw Blame History

Oikos rollback runbook

Risk class: reversible_low (rollback restores previous version) RTO: < 10 minutes (scripted) RPO: last pre-deploy pg_dump (created by deploy.sh before each deploy)

When to roll back

Health check fails after deploy (API returns 503 or times out)
API returns errors that didn't exist before deploy
Configuration regression (wrong routes, missing tools)
Migration failure (deploy.sh finishes but seeds fail)

Procedure

Automated rollback

cd /opt/oikos
./scripts/rollback.sh <previous-sha>

The script performs:

Stop all Docker services
Restore DB from /opt/oikos/backups/pre-deploy-<sha>.sql
Check out the previous SHA
Rebuild and restart
Health check loop (30 attempts, 2s each)

Find the previous SHA:

git log --oneline -5

Manual rollback (if script fails)

# 1. Stop everything
cd /opt/oikos
docker compose --profile full down

# 2. Restore DB manually
DUMP_FILE=/opt/oikos/backups/pre-deploy-<sha>.sql
docker compose --profile full up -d postgres
sleep 5
docker compose exec -T postgres psql -U oikos oikos < "$DUMP_FILE"

# 3. Re-deploy previous version
git checkout <previous-sha>
DOCKER_BUILDKIT=1 docker compose --profile full build
docker compose --profile full up -d

# 4. Health check
curl http://localhost:8090/healthz

After rollback (re-deploy latest)

git checkout main
./scripts/deploy.sh

Recovery verification

# API responds
curl http://localhost:8090/healthz

# Entity count matches
curl -s http://localhost:8090/api/v1/entities?limit=1 | jq '.items | length'

# MCP tools working (via Hermes)
curl -s http://localhost:8092/query -d '{"tool":"get_health_summary"}'

Rollback decision tree

Deploy fails health check
├── Migration problem → rollback.sh + DB restore
├── Code regression → rollback.sh (DB restore optional)
├── Config change → rollback.sh (DB restore optional)
└── Docker/infra issue → rollback.sh + full DB restore

Backups

Automated: deploy.sh runs pg_dump before every deploy → /opt/oikos/backups/pre-deploy-<sha>.sql
Scheduled: daily pg_dump via scheduler housekeeping + rclone push to Proton Drive (plan A3)
Retention: keep last 30 pre-deploy dumps locally; 30 daily + 12 monthly on Proton Drive

Rehearsal log

Date	Trigger	Previous SHA	Result	Duration
2026-07-07	Cutover test	`7ac2521`	Healthy, 20 tools	~2 min

2.5 KiB Raw Blame History