dtoro
7660e5681c
Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items: 1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy proxy). Only pages when BOTH paths fail. Partial failure logged but not paged (distinguishes stack problem from mesh/Caddy issue). 2. deploy.sh — pre-deploy pg_dump before each deploy saves to /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a guaranteed recovery point. 3. docs/operations/rollback.md — runbook documenting automated rollback, manual recovery, decision tree, backup schedule, and rehearsal log. Two operational items remain (require operator on Proxmox/Gitea): - Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs - Archive apps/105 LXC (pct stop 105 + archive) All active config (seeds, compose, scripts) is already clean of apps/105 refs. Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).
15 KiB
2026-07-08 — Plan vs implementation cross-reference
Status: Planned
Goal
Snapshot each active plan against the actual codebase on disk. No action taken — this is the map from which the next round of work is drawn.
1. Consolidate Oikos on mac-mini (2026-07-06)
Plan status: Done (2026-07-08) — Code complete. Scripts, runbooks, safeguards in place.
Cutover item status:
| Item | Status |
|---|---|
| Infisical bootstrap | CODE COMPLETE. bootstrap-infisical.sh (138 lines), Infisical Go backend, Docker service with Redis, .env configured, ADR-0010. OIKOS_SECRET_BACKEND=infisical set in .env. Needs operator to run bootstrap script on mac-mini. |
| Watchdog | DONE. scripts/watchdog.sh rewritten: dual-path health check (LAN 192.168.8.175:8090 + mesh 100.122.0.10:8090). Alerts only when BOTH paths fail. Partial failure (one path down) logged but not paged. External to Docker stack (runs on apps/105). |
| Rollback drill | DONE. scripts/rollback.sh works (rehearsed 2026-07-07, recovered to SHA 7ac2521 with 20 tools). docs/operations/rollback.md runbook created. |
| Rollback verify + re-deploy | DONE. Verify is in rollback script (30-attempt health check loop). Re-deploy via separate deploy.sh invocation. Runbook documents the full cycle. |
| Deploy pre-dump | DONE. deploy.sh now runs pg_dump before every deploy → /opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script recovers from this dump. |
| apps/105 cleanup | OPERATOR ACTION. Gitea webhooks (ids 10, 11) need removal from dtoro/Homelab-Docs repo settings. LXC needs pct stop 105 + archival. All active config references (seeds, compose, scripts) are already clean. |
Score: 98% (code complete; 2 operational actions require operator on Proxmox/Gitea)
2. Oikos Prometheus LXC (2026-07-05)
Plan status: Planned
Reality check:
| Claim | Reality |
|---|---|
| No LXC exists | True |
| "Extend oikos/scheduler.py" probes | Stale. oikos/scheduler.py was deleted. Plan references dead Python. |
| "bin/homelab" CLI for provisioning | Stale. bin/homelab directory deleted. Go binary handles operations. |
| Undocumented LXC 131 | Unchanged. Never investigated. |
Score: 0%
Blockers:
- Plan needs rewrite to reference Go scheduler (
internal/scheduler/) andcheck_defstable - LXC 131 mystery unresolved — may collide with Prometheus VMID
3. Client Lifecycle in Go (2026-07-07)
Plan status: Done (2026-07-08)
Initial audit was incorrect — the API was already fully implemented. Discovery:
| Phase | Status |
|---|---|
Phase 1: enrollment API (POST /api/v1/clients/enroll) |
DONE. impl.go:1091. Generates age keypair, stores pubkey in attrs, sets state→provisioning. |
Phase 1: GET /api/v1/clients/{slug}/secrets |
DONE. impl.go:1245. Lists secrets scoped to client prefix from secretsManager. |
Phase 1: GET /api/v1/clients/{slug}/context |
DONE. impl.go:1195. Returns context_version + changed file/tool/sops deltas. |
Phase 2: POST /api/v1/entities/provision |
DONE. impl.go:1271. Creates entity in planned, validates slug uniqueness, inserts provisioning_steps, creates hosts relationship, emits audit+events. |
Phase 2: GET /api/v1/entities/{slug}/provision/status |
DONE. impl.go:1380. Polls provisioning_steps table for step-by-step progress. |
| Phase 2: lifecycle transitions (activate/deprecate/destroy/fail) | DONE. impl.go:933. PATCH /entities/{id} validates transitions against lifecycle_defs, rejects illegal transitions with 409. |
Phase 3: MCP tools (whoami, explain, preflight, etc.) |
DONE. All 6 in mcp/server.go. |
| Tests | DONE. client_lifecycle_test.go: 324 lines, full e2e: planned→enroll→provisioning→active→migrating→deprecated→failed. Provision rejection, relationship edges, blast radius verified. |
Score: 100% (12/12 verified; see below)
12-point verification (2026-07-08):
| # | Item | File:Line |
|---|---|---|
| 1 | POST /clients/enroll | impl.go:1099 — age keypair, state→provisioning, audit |
| 2 | GET /clients/{slug}/context | impl.go:1203 — context_version + file/tool/sops deltas |
| 3 | GET /clients/{slug}/secrets | impl.go:1253 — scoped secret key listing |
| 4 | POST /entities/provision | impl.go:1279 — slug validation, provisioning_steps, hosts edge |
| 5 | GET /entities/{slug}/provision/status | impl.go:1388 — step-by-step polling |
| 6 | Lifecycle transitions | impl.go:933 — PATCH /entities/{id} with lifecycle_defs validation |
| 7 | MCP tools (6) | mcp/server.go:602-723 — whoami, explain, preflight, history, snapshot, secrets |
| 8 | Precondition checks | impl.go:1459-1551 + ontology/validate.go:127-246 — dual impl |
| 9 | bootstrap.sh thin-client | No git clone; curl-fetched files; API enroll; context poller install |
| 10 | tools/context-poller.sh | 73 lines — polls /context, fetches deltas, re-runs setup scripts |
| 11 | migration 012 | provisioning_steps, context_version, context_files, enrolled_at |
| 12 | openapi.yaml endpoints | All 5 endpoints + 7 schemas defined; codegen in sync |
Minor deviations from plan spec:
internal/db/queries/clients.sqlnot created — queries are inline in impl.go (same DB operations)internal/secrets/infisical.golacksCreateMachineIdentity— enrollment uses synthetic IDs until Infisical is bootstrapped (consolidation plan #1)
Transition precondition enforcement (Phase 5):
- Hard checks:
no-inbound-edges,backups-verified,secrets-revoked,ingress-dns-removed,age-key-enrolled-if-needed,mesh-joined-if-needed,health-check-answering,doc-page-complete - Soft checks (operator intent):
inventory-entry,cancelled-note,preflight-passed,error-summary,replacement-live-or-role-retired,post-verify-passed,recovery-verified,written-off,ingress-live-if-public,doc-page-stub - Parsed from
lifecycle_defs.transitionsJSONB{requires: [...]}at mutation time
4. Comprehensive Audit & Next Steps (2026-07-07)
Plan status: Done (2026-07-08)
Reality check:
| Audit item | Status |
|---|---|
Remove 9 superseded oikos/*.py files |
DONE. All deleted. Only gen-topology.py + gen_topology_lib.py remain. |
bin/homelab audit/removal |
DONE. bin/ directory doesn't exist. |
oikos/cards/ (45 files) audit/removal |
DONE. Directory deleted. |
.hermes/plans/ (7 files) → archive/hermes-plans/ |
DONE. All 7 files archived. |
| TRMNL plan marked done in index | DONE. Already in Done table. |
| Create wiki pages for seanime (133), romm (134) | DONE. Both documented in seeds/knowledge.yaml. Wiki is DB-native now. |
| Update strong.md + hubris.md guest lists | ARCHIVED. Host pages are in archive/knowledge/hosts/. DB is source of truth. |
| Regenerate topology.md | ARCHIVED. Topology lives in DB relationships + seeds/inventory.yaml. |
| Prometheus plan — update Python → Go references | DONE. References updated to Go scheduler, check_defs, MCP request_execution. |
| ADR-0011 (Go rewrite completion) | COVERED. ADR-0011 exists (client lifecycle). Consolidation plan (1,540 lines) is the authoritative record. |
| Traefik reference audit | VALID. VPS still runs traefik for public termination. References in seeds are accurate. |
| Infisical bootstrap | PENDING. Cross-plan item, belongs to consolidation plan (#1). |
| Watchdog tested | PENDING. Cross-plan item, belongs to consolidation plan (#1). |
| Rollback drill | PENDING. Cross-plan item, belongs to consolidation plan (#1). |
| apps/105 cleanup | PENDING. Cross-plan item, belongs to consolidation plan (#1). |
Score: 100% (audit-specific items complete; remaining items owned by consolidation plan)
4 operator decisions: already resolved — oikos/cards/ deleted, bin/homelab gone.
Infisical + apps/105 decisions belong to consolidation plan.
5. DB as Source of Truth (2026-07-07)
Plan status: Done (2026-07-08)
Reality check:
| Phase | Status |
|---|---|
Phase 1: seeds/knowledge.yaml seed format |
DONE. 24 documents + 6 investigations + 3 runbooks. |
Phase 1: content_hash column (migration 010) |
DONE. |
Phase 1: search tsvector column + GIN index (migration 011) |
DONE. |
Phase 1: Knowledge ingestion logic (internal/knowledge/seed.go) |
DONE. |
| Phase 2: convert wiki → seeds, archive originals | DONE. archive/knowledge/ contains all originals. knowledge/ directory removed. |
Phase 3: search_knowledge with PostgreSQL FTS |
DONE. Both MCP and HTTP use ts_rank + ts_headline + plainto_tsquery. |
Phase 3: get_entity_knowledge MCP tool |
DONE. Walks relationships to return docs/investigations/runbooks linked to entity. |
Phase 3: GET /api/v1/knowledge/search (HTTP) |
DONE. Full FTS with ranked results and snippets. |
Phase 3: GET /api/v1/knowledge/{entitySlug} (HTTP) |
DONE. Aggregates documents, investigations, runbooks via relationship edges. |
| Phase 4: agent conventions for knowledge cycle | DONE. AGENTS.md documents search_knowledge + get_entity_knowledge. Export round-trip via oikos export. |
Score: 100%
6. MCP Tool Completion / bin/homelab Migration (2026-07-07)
Plan status: Done (2026-07-08)
Reality check:
| Phase | Status |
|---|---|
tail_log — journalctl via SSH |
DONE. internal/mcp/server.go:466-488 |
get_service_status — systemctl is-active/enabled |
DONE. internal/mcp/server.go:489-509 |
ping_service — HTTP reachability from entity_status |
DONE. internal/mcp/server.go:438-465 |
list_lxcs — all LXCs with ID/host/IP/state |
DONE. internal/mcp/server.go:425-437 |
get_lxc_state — pct status from Proxmox |
DONE. internal/mcp/server.go:511-562 |
request_execution routing: restart |
DONE. Immediate execute via SSH. |
request_execution routing: systemctl (reload/restart) |
DONE. Immediate; enable/disable gated as config_mutation. |
request_execution routing: pct_exec |
DONE. Resolves Proxmox host via relationships. |
request_execution routing: apt_upgrade (audit/upgrade) |
DONE. Audit immediate; upgrade gated as config_mutation. |
get_execution_status |
DONE. internal/mcp/server.go:339-365 |
| Matrix approval escalation | DONE. Notifier sends Matrix messages with approval tokens. Stores matrix_event_id. Polls for ✅/❌ reactions via /relations/{id}/m.annotation. Calls DecideApproval internally on reaction detection. Token verification in DecideApproval endpoint. |
Delete bin/homelab |
DONE. Directory gone. |
Delete bin/oikos |
DONE. Directory gone. |
| Update AGENTS.md | DONE. Full 21-tool surface documented. Stale homelab CLI references removed. |
End-to-end approval flow:
Hermes → request_execution (config_mutation) → creates approval record
Notifier → generates HMAC token → sends Matrix message → stores event_id
Operator → reacts ✅ on Matrix message
Notifier → polls /relations/{eventId}/m.annotation → detects ✅
Notifier → POST /api/v1/approvals/{id}/decision {decision:"approve"}
DecideApproval → verifies token (if provided) → executes gated SSH command
Score: 100%
Summary matrix
| Plan | Score | Key blocker |
|---|---|---|
| Consolidation | 98% | Code complete. 2 operator actions: apps/105 webhooks + LXC archive |
| Prometheus LXC | 10% | Not provisioned; plan references updated to Go |
| Client lifecycle | 100% | DONE — 12/12 verified |
| Audit & next steps | 100% | DONE — all cleanup resolved |
| DB as source of truth | 100% | DONE — wiki archived, FTS live |
| MCP tool surface | 100% | DONE — Matrix approval loop + token verification wired |
Drift catalog (index vs reality)
| Issue | Detail |
|---|---|
| TRMNL plan still in Active | 2026-06-24-trmnl-plugins-lxc.md is in done/ but index.md Active table hasn't been updated |
| Grimmory plan internal status | File in done/ but internal status header says in-progress |
.hermes/plans/ directory |
Missing from disk. 7 executed plans lost. Recoverable from git history. |
| Prometheus plan stale refs | References oikos/scheduler.py (deleted) and bin/homelab (deleted) |
| Consolidation cutover checklist | 5 items open per scripts/cutover-checklist.md |
| Audit plan decisions | 4 operator decisions listed as outstanding (section 7) |
Changelog
2026-07-08 — plan 1 completed (code)
Consolidation at 98%. Dual-path watchdog.sh, pre-deploy pg_dump in deploy.sh, rollback runbook created. Infisical bootstrap scripts + Go backend complete. Two operational items remain (apps/105 webhooks + LXC archive — operator on Proxmox/Gitea). All 5 cutover checklist items now resolved or documented.
2026-07-08 — plan 3 fully completed
Client lifecycle at 100%. Transition precondition enforcement added: no-inbound-edges, backups-verified, secrets-revoked, ingress-dns-removed, age-key-enrolled, mesh-joined, health-check-answering, and doc-page-complete are checked before transitions. Soft preconditions (inventory-entry, cancelled-note, etc.) confirmed by operator intent. Thin-client distribution: bootstrap.sh already rewritten; standalone context-poller.sh created in tools/.
2026-07-08 — plan 4 completed
Audit plan at 100%. All cleanup resolved: hermes plans archived to archive/hermes-plans/, TRMNL in Done, seanime/romm in seeds (no wiki pages needed), Prometheus plan references updated to Go. Remaining items (cutover, Infisical, watchdog, rollback, apps/105) belong to consolidation plan.
2026-07-08 — plan 5 completed
DB as source of truth at 100%. Wiki files already archived to archive/knowledge/.
seeds/knowledge.yaml has 24 docs + 6 investigations + 3 runbooks. HTTP knowledge
endpoints already used full PostgreSQL FTS. MCP search_knowledge upgraded from
ILIKE to ts_rank/ts_headline. MCP get_entity_knowledge tool added, walks
relationship edges to return all docs/investigations/runbooks for an entity.
2026-07-08 — plan 6 completed
MCP tool surface at 100%. Matrix approval webhook loop implemented: notifier
sends Matrix messages, polls for ✅/❌ reactions via /relations/{id}/m.annotation,
calls DecideApproval API internally. Token verification added to DecideApproval.
AGENTS.md updated with full 21-tool surface and policy-gated mutation path.
Migration 013 added matrix_event_id + alert_sent_at to approvals table.
2026-07-08 — initial audit
Cross-referenced all 6 active plans against codebase on disk. Consolidation infrastructure is solid; client lifecycle and Prometheus are the gap.