dtoro
7660e5681c
Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items: 1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy proxy). Only pages when BOTH paths fail. Partial failure logged but not paged (distinguishes stack problem from mesh/Caddy issue). 2. deploy.sh — pre-deploy pg_dump before each deploy saves to /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a guaranteed recovery point. 3. docs/operations/rollback.md — runbook documenting automated rollback, manual recovery, decision tree, backup schedule, and rehearsal log. Two operational items remain (require operator on Proxmox/Gitea): - Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs - Archive apps/105 LXC (pct stop 105 + archive) All active config (seeds, compose, scripts) is already clean of apps/105 refs. Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).
98 lines
2.5 KiB
Markdown
98 lines
2.5 KiB
Markdown
# Oikos rollback runbook
|
|
|
|
**Risk class:** reversible_low (rollback restores previous version)
|
|
**RTO:** < 10 minutes (scripted)
|
|
**RPO:** last pre-deploy pg_dump (created by deploy.sh before each deploy)
|
|
|
|
## When to roll back
|
|
|
|
- Health check fails after deploy (API returns 503 or times out)
|
|
- API returns errors that didn't exist before deploy
|
|
- Configuration regression (wrong routes, missing tools)
|
|
- Migration failure (deploy.sh finishes but seeds fail)
|
|
|
|
## Procedure
|
|
|
|
### Automated rollback
|
|
|
|
```bash
|
|
cd /opt/oikos
|
|
./scripts/rollback.sh <previous-sha>
|
|
```
|
|
|
|
The script performs:
|
|
1. Stop all Docker services
|
|
2. Restore DB from `/opt/oikos/backups/pre-deploy-<sha>.sql`
|
|
3. Check out the previous SHA
|
|
4. Rebuild and restart
|
|
5. Health check loop (30 attempts, 2s each)
|
|
|
|
Find the previous SHA:
|
|
```bash
|
|
git log --oneline -5
|
|
```
|
|
|
|
### Manual rollback (if script fails)
|
|
|
|
```bash
|
|
# 1. Stop everything
|
|
cd /opt/oikos
|
|
docker compose --profile full down
|
|
|
|
# 2. Restore DB manually
|
|
DUMP_FILE=/opt/oikos/backups/pre-deploy-<sha>.sql
|
|
docker compose --profile full up -d postgres
|
|
sleep 5
|
|
docker compose exec -T postgres psql -U oikos oikos < "$DUMP_FILE"
|
|
|
|
# 3. Re-deploy previous version
|
|
git checkout <previous-sha>
|
|
DOCKER_BUILDKIT=1 docker compose --profile full build
|
|
docker compose --profile full up -d
|
|
|
|
# 4. Health check
|
|
curl http://localhost:8090/healthz
|
|
```
|
|
|
|
### After rollback (re-deploy latest)
|
|
|
|
```bash
|
|
git checkout main
|
|
./scripts/deploy.sh
|
|
```
|
|
|
|
## Recovery verification
|
|
|
|
```bash
|
|
# API responds
|
|
curl http://localhost:8090/healthz
|
|
|
|
# Entity count matches
|
|
curl -s http://localhost:8090/api/v1/entities?limit=1 | jq '.items | length'
|
|
|
|
# MCP tools working (via Hermes)
|
|
curl -s http://localhost:8092/query -d '{"tool":"get_health_summary"}'
|
|
```
|
|
|
|
## Rollback decision tree
|
|
|
|
```
|
|
Deploy fails health check
|
|
├── Migration problem → rollback.sh + DB restore
|
|
├── Code regression → rollback.sh (DB restore optional)
|
|
├── Config change → rollback.sh (DB restore optional)
|
|
└── Docker/infra issue → rollback.sh + full DB restore
|
|
```
|
|
|
|
## Backups
|
|
|
|
- **Automated:** deploy.sh runs `pg_dump` before every deploy → `/opt/oikos/backups/pre-deploy-<sha>.sql`
|
|
- **Scheduled:** daily pg_dump via scheduler housekeeping + rclone push to Proton Drive (plan A3)
|
|
- **Retention:** keep last 30 pre-deploy dumps locally; 30 daily + 12 monthly on Proton Drive
|
|
|
|
## Rehearsal log
|
|
|
|
| Date | Trigger | Previous SHA | Result | Duration |
|
|
|------|---------|-------------|--------|----------|
|
|
| 2026-07-07 | Cutover test | 7ac2521 | Healthy, 20 tools | ~2 min |
|