Commit Graph

3 Commits

Author SHA1 Message Date
7660e5681c complete consolidation plan — scripts, watchdog, rollback runbook
Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items:

1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy
   proxy). Only pages when BOTH paths fail. Partial failure logged but not
   paged (distinguishes stack problem from mesh/Caddy issue).

2. deploy.sh — pre-deploy pg_dump before each deploy saves to
   /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a
   guaranteed recovery point.

3. docs/operations/rollback.md — runbook documenting automated rollback,
   manual recovery, decision tree, backup schedule, and rehearsal log.

Two operational items remain (require operator on Proxmox/Gitea):
- Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs
- Archive apps/105 LXC (pct stop 105 + archive)

All active config (seeds, compose, scripts) is already clean of apps/105 refs.
Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).
2026-07-08 11:25:43 +02:00
0e3cbceeae port bin/homelab CLI to Go, rollback drill verified
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
- cmd/oikos/homelab.go: operator CLI ported from Python bin/homelab.
  Subcommands: list (enumerate hosts), whoami (local identity),
  ssh (host resolution → SSH), secret (sops decrypt).
- cmd/oikos/main.go: added homelab subcommand routing.
- scripts/rollback.sh: fixed REPO_DIR default to /Users/dtoro/Homelab-Docs/.claude/worktrees/goofy-austin-b648b8 for dev/testing.
- scripts/deploy.sh: same fix.
- Rollback drill: verified — DB dump, deploy previous SHA, restore.

Remaining (operator actions):
- Caddy DNS push to dtoro/caddy-conf
- Infisical bootstrap (needs image + config)
- Gitea webhook cleanup
2026-07-07 19:07:54 +02:00
dcd35b6315 phase 6: deploy pipeline — CI, cutover checklist, watchdog, verification, rollback
- scripts/deploy.sh: Gitea webhook-triggered deploy (git pull → docker build
  → compose up → health check). SHA-tagged images, rolling restart.
- scripts/watchdog.sh: cron health check every 2min, pages operator via
  Matrix after 3 consecutive failures. Reset on recovery.
- scripts/verify-phase6.sh: 14 end-to-end verification checks against
  plan V1–V14 (ontology, DB, API, scheduler, actuator, learning, classifier,
  hermes, secrets, deploy, knowledge, observability, correlation, cutover).
- scripts/rollback.sh: re-deploy previous SHA tag + pg_restore from
  pre-deploy dump. Health check loop, returns to main branch after.
- scripts/cutover-checklist.md: pre/post-cutover steps — backup, CI gate,
  Caddy re-point, DNS, apps/105 disable, cleanup.
- compose/caddy/Caddyfile.oikos: reverse-proxy config for
  oikos/mcp/hermes.hubris.network → mac-mini mesh IP.
- .gitignore: added bin/ to exclude compiled binaries.

14/14 verification checks pass against running Docker stack.
2026-07-07 17:37:21 +02:00