101 lines
2.6 KiB
Markdown
101 lines
2.6 KiB
Markdown
# Secret rotation runbook (Phase 5)
|
|
|
|
Rotation cadences per secret type. All rotation is automated via Infisical;
|
|
this runbook covers the manual verification and DR procedures.
|
|
|
|
## Rotation schedule
|
|
|
|
| Secret | Cadence | Method |
|
|
|--------|---------|--------|
|
|
| OpenRouter API key | 90 days | Infisical rotation policy → update `OPENROUTER_API_KEY` env |
|
|
| MCP bearer token | 30 days | Infisical random password generation |
|
|
| Approval HMAC secret | 90 days | Infisical random password generation |
|
|
| Matrix access token | 90 days | Manual (Matrix does not support automated rotation) |
|
|
| Age DR key (SOPS fallback) | Never | Static — stored offline for DR only |
|
|
|
|
## How to rotate a secret
|
|
|
|
### Automated (Infisical)
|
|
```bash
|
|
# Secrets managed by Infisical rotate automatically per the policy above.
|
|
# To force an immediate rotation:
|
|
infisical secrets rotate --project-id $INFISICAL_PROJECT_ID \
|
|
--secret-name <secret-key> --env dev
|
|
|
|
# Verify the new value is available:
|
|
oikos secret list
|
|
```
|
|
|
|
### Manual (SOPS fallback)
|
|
```bash
|
|
# If Infisical is unavailable, use the SOPS DR fallback:
|
|
sops -d secrets/<name>.yaml
|
|
|
|
# To rotate a SOPS secret:
|
|
sops -e --in-place secrets/<name>.yaml # edit in place
|
|
```
|
|
|
|
## Rotation verification
|
|
|
|
After any rotation, verify the consuming services still work:
|
|
|
|
```bash
|
|
# 1. OpenRouter key: test Hermes query
|
|
curl -s -X POST http://localhost:8092/query \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"query":"fleet health"}'
|
|
|
|
# 2. MCP bearer token: test MCP connection
|
|
curl -s -X POST http://localhost:8092/query \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"tool":"list_entities","args":{"limit":1}}'
|
|
|
|
# 3. Approval HMAC: create a test execution
|
|
curl -s -X POST http://localhost:8092/query \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"query":"restart caddy"}'
|
|
```
|
|
|
|
## Disaster recovery
|
|
|
|
If Infisical is completely unavailable:
|
|
|
|
```bash
|
|
# 1. Export SOPS DR fallback
|
|
oikos secret export-sops > /tmp/sops-dr-backup.txt
|
|
|
|
# 2. Configure services to use SOPS fallback
|
|
# Set OIKOS_SECRETS_DIR=/opt/homelab-context/secrets
|
|
# This switches the secrets manager to SOPS-only mode.
|
|
|
|
# 3. Restart services
|
|
docker compose restart api hermes notifier scheduler
|
|
```
|
|
|
|
## Restore drill
|
|
|
|
Run monthly:
|
|
|
|
```bash
|
|
# 1. Export all secrets from Infisical
|
|
oikos secret list
|
|
|
|
# 2. Simulate Infisical outage: stop the container
|
|
docker compose stop infisical
|
|
|
|
# 3. Verify SOPS fallback works
|
|
OIKOS_SECRETS_DIR=./secrets oikos secret list
|
|
|
|
# 4. Restore Infisical
|
|
docker compose start infisical
|
|
sleep 5
|
|
|
|
# 5. Verify Infisical primary works again
|
|
oikos secret list
|
|
```
|
|
|
|
## Changelog
|
|
|
|
### 2026-07-07 — initial rotation runbook
|
|
Phase 5 rotation cadences, verification steps, and DR restore drill.
|