dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

plans: comprehensive audit + DB-as-truth architecture

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-07 20:09:32 +02:00
parent 4b6c02a88e
commit b2bfa26f64
3 changed files with 991 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

364
plans/2026-07-07-comprehensive-audit-and-next-steps.md Normal file
View File

@@ -0,0 +1,364 @@
# 2026-07-07 — Comprehensive audit: stale files, state gaps, and next steps
**Status:** Planned
## Executive summary
Oikos is in a stable intermediate state: Go rewrite (Phases 1–6) is implemented and
running in Docker on mac-mini, the narrative wiki has been restructured into
`knowledge/wiki/`, and the strong migration (Tier 1 + 2) is complete. But the
transition left behind stale artifacts, half-updated plans, and several documentation
gaps that create confusion for agents and operators.
This plan catalogs every stale file and gap found, assigns ownership, and proposes a
prioritized work queue for the next 2–4 weeks. **No action is taken by this plan
itself** — it is the map from which concrete work items are drawn.
---
## 1. Stale files — catalog
### 1.1 Python kernel (`oikos/*.py`) — 11 files
| File | Risk | Verdict |
|------|------|---------|
| `oikos/__init__.py` | Low | Package init. Harmless but unused. |
| `oikos/approve.py` | Low | Superseded by `internal/domain/approval.go`. Remove. |
| `oikos/decide.py` | Low | Superseded by `internal/policy/classify.go`. Remove. |
| `oikos/drift.py` | **Medium** | Superseded by `internal/scheduler/scheduler.go`. References old paths (`containers/121-caddy.md` at L218). Remove. |
| `oikos/ledger.py` | **Medium** | Superseded by DB view + `internal/notifier/notifier.go`. Remove. |
| `oikos/policy.py` | Low | Superseded by `internal/policy/classify.go`. Remove. |
| `oikos/relations.py` | Low | Superseded by `blast_radius()` SQL function + `internal/ontology/`. Remove. |
| `oikos/scheduler.py` | **Medium** | Superseded by `internal/scheduler/`. Referenced by Prometheus plan. Remove. |
| `oikos/signal.py` | Low | Superseded by signals table (migration 003) + `internal/domain/signal.go`. Remove. |
| `oikos/gen-topology.py` | **High** | Still active — writes `knowledge/wiki/infrastructure/topology.md`, consumed by `homelab` CLI. **Keep until ported to Go.** |
| `oikos/gen_topology_lib.py` | **High** | Imported by `gen-topology.py`. **Keep until ported.** |
**Action:** Remove the 9 superseded `.py` files. Keep `gen-topology.py` +
`gen_topology_lib.py` until a Go equivalent exists. Update `.gitignore` and any
cross-references (Prometheus plan, bin/homelab docs).
### 1.2 `.hermes/plans/` — 7 files
These plans live in `.hermes/plans/` but the wiki-hq convention (`plans/index.md` L125,
`.agents/domains/operations/schema.md`) mandates all plans go in `plans/`. The strong
migration assessment (`.hermes/plans/2026-07-05_strong-migration-assessment.md`) is
particularly valuable and has already been executed.
| File | Status | Action |
|------|--------|--------|
| `2026-06-03_110000-library-ssd-migration-to-ludo-mini.md` | Superseded by strong migration | Move to `plans/done/` |
| `2026-06-03_150000-homelab-structure-revision.md` | Executed (is this repo structure) | Move to `plans/done/` |
| `2026-06-03_223218-dhcp-pool-exclude-static-ips.md` | Executed | Move to `plans/done/` |
| `2026-06-05_170000-prevent-dhcp-ip-drift.md` | Executed | Move to `plans/done/` |
| `2026-06-06_232200-authentik-frequent-login-fix.md` | Executed | Move to `plans/done/` |
| `2026-06-06_234500-caddyfile-truncation-permanent-fix.md` | Executed | Move to `plans/done/` |
| `2026-07-05_strong-migration-assessment.md` | Executed (Phases 1–2d done) | Move to `plans/done/`, add Phase 3 status note |
### 1.3 `oikos/cards/` — 45 context cards
These `.md` files are consumed by the Python MCP server's `explain` tool
(`mcp/server.py:38` referenced `CARD_DIR`). The Go MCP server (`internal/mcp/`) now
serves entity context from the database via `get_entity` / `list_entities`.
| Risk | Verdict |
|------|---------|
| **Medium** | Cards are redundant with DB entities but may still be read by the legacy `bin/homelab` Python CLI for `homelab explain`. If `bin/homelab` is ported to Go (via generated OpenAPI client), these become dead weight. |
**Action:** Verify whether `bin/homelab explain` reads `oikos/cards/` or queries the
API. If it reads cards: mark them as "keep until homelab CLI ported." If it queries the
API: remove cards and mention in the next Go release.
### 1.4 `secrets-issuance/` — identity issuance service
Still running on apps/105 (LXC) as `secrets-issuance.service`. The Docker stack
hasn't taken over this function yet. The cutover checklist (`scripts/cutover-checklist.md:5`)
notes "Disable apps/105 services" as ticked, but secrets-issuance was specifically stopped,
not replaced.
| Risk | Verdict |
|------|---------|
| **Medium** | If apps/105 is destroyed (cutover cleanup), secrets issuance must be replaced or explicitly decommissioned. |
**Action:** Decide: (a) port secrets issuance to Docker stack, or (b) decommission it
(age keys can be generated ad-hoc on each client). Document the decision.
### 1.5 Traefik references
`knowledge/sources/references/cert-sync-and-traefik-config.md` and
`inventory.yaml:245` (`vps/management.json.tmpl` references traefik) refer to a reverse
proxy that was replaced by Caddy for internal routing. Traefik **still runs on the VPS**
for public termination, so the references may be valid — but the doc is unclear.
**Action:** Audit traefik references. If VPS traefik is still active, document its
scope (public termination only, not internal). If fully replaced, remove the references.
---
## 2. Plan status drift
### 2.1 TRMNL (128) — marked "Planned" but done
`plans/2026-06-24-trmnl-plugins-lxc.md` shows status "Planned" but:
- LXC 128 exists in `inventory.yaml` with IP `192.168.8.211`
- Wiki page `knowledge/wiki/containers/128-trmnl.md` exists
- `hosts/trmnl.yaml` exists
- Caddy route `trmnl.hubris.network` exists
- Service `services.trmnl` exists in `inventory.yaml`
**Action:** Mark TRMNL plan as "Done", move to `plans/done/`, add changelog entries
on affected pages (as specified in the plan's "Post-migration" section).
### 2.2 Consolidation plan — cutover incomplete
`plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md` status says
"Phase 1-6 implemented, pending cutover." The cutover checklist has 5 pending items:
1. **Infisical bootstrap** — `profiles: [infisical]` in compose but `OIKOS_SECRET_BACKEND`
env not set. SOPS fallback is active.
2. **Watchdog tested** — pending API stop + Matrix alert verification.
3. **Rollback drill** — `scripts/rollback.sh` never rehearsed.
4. **Rollback verify health + re-deploy** — depends on #3.
5. **Cleanup** — apps/105 webhooks not removed, LXC not archived.
**Action:** These 5 items are the highest-priority operational work. Schedule a 1-hour
window to complete them.
### 2.3 Prometheus plan — references Python
`plans/2026-07-05-oikos-prometheus-lxc.md:60` says "Extend `oikos/scheduler.py`'s
disk/temp probes" but the scheduler is now Go (`internal/scheduler/`). The plan also
references `bin/homelab` Python CLI commands for provisioning.
**Action:** Update the plan to reference the Go check_defs system (migration 003,
`check_defs` table) and the new `homelab` CLI (or `oikos` binary commands).
---
## 3. Documentation gaps
### 3.1 Missing container wiki pages
Two containers exist in `inventory.yaml` and `hosts/*.yaml` but have no wiki page:
| Container | PVE ID | Host | Wiki page |
|-----------|--------|------|-----------|
| seanime | 133 | strong | **Missing** |
| romm | 134 | strong | **Missing** |
**Action:** Create `knowledge/wiki/containers/133-seanime.md` and
`knowledge/wiki/containers/134-romm.md` from the data in `hosts/seanime.yaml` and
`hosts/romm.yaml`.
### 3.2 Host pages for `strong.md` and `hubris.md` — stale resource math
`knowledge/wiki/hosts/hubris.md` and `knowledge/wiki/hosts/strong.md` may still
reflect pre-migration resource allocation. The strong migration moved elementsynapse,
house, arriman, jellyfin, and grimmory — both pages need updated guest lists and
resource math.
**Action:** Audit both host pages against live `pct list` on each Proxmox node.
Update guest tables and RAM/CPU summaries.
### 3.3 `knowledge/wiki/infrastructure/topology.md` — Mermaid graph
Generated by `oikos/gen-topology.py`. Should be verified to include:
- strong and its guests (vmbr1 range 192.168.8.241–249)
- romm (134) and seanime (133)
- teddycloud (131)
- grimmory (130) on strong (not hubris)
**Action:** Run `oikos/gen-topology.py` and verify topology reflects current state.
Commit the regenerated file.
### 3.4 No ADR for Go rewrite completion
ADR-0001 ("Go single binary") was written as a forward-looking decision. No ADR
documents the completed rewrite as an accepted/implemented decision with retroactive
context on why the Python→Go transition happened when it did.
**Action:** Add an ADR-0011 documenting the completed Go rewrite with dates, phases,
and the rationale for the timing.
### 3.5 Un-enrolled hosts tracked in inventory
Several hosts have `age_pubkey: ''` in `inventory.yaml`:
- jellyfin, paperless, gitea, nextcloud, sophia, mule-images, caddy, arriman
These are documented as "not enrolled" in their host files. This is intentional
(not all LXCs need homelab-context) but the pattern should be documented in the
wiki as a convention.
**Action:** Add a section to `knowledge/wiki/infrastructure/homelab-context.md`
(or `.agents/operations/agent-enrollment.md`) explaining when enrollment is expected
vs. skipped.
---
## 4. Technical debt — Go codebase
### 4.1 `bin/homelab` — still a Python shim?
The `.gitignore` says "oikos/ kernel files are still imported by bin/homelab for
operational CLI commands." If `bin/homelab` is a compiled Go binary (as the README
and ADR-0001 suggest), then the Python kernel files should not be needed.
**Action:** Verify what `bin/homelab` is (run `file bin/homelab`). If it's a Go
binary and doesn't import Python, the 9 superseded `.py` files in `oikos/` are
safe to delete.
### 4.2 `oikos/build_hosts.go` — Python equivalent still exists
`cmd/oikos/build_hosts.go` exists alongside `mcp/build_host_files.py`. Both
generate `hosts/*.yaml` from `inventory.yaml`. Are both active? If the Go version
works, the Python version is stale.
**Action:** Compare output of both generators. If identical, remove the Python
version and update the `homelab` CLI to use the Go generator.
### 4.3 `scripts/dns-sync.py` — standalone Python
This script syncs DNS records to Technitium. It's operational and has no Go
equivalent yet. Not stale, but should be tracked as "port to Go."
**Action:** Add to Go rewrite backlog as a follow-up item.
### 4.4 `scripts/validate-seeds.py` — standalone Python
Validates `seeds/*.yaml`. Could be folded into `oikos seed` as a `--validate` flag.
**Action:** Add to Go rewrite backlog. Low priority.
---
## 5. Operational backlog (prioritized)
### P0 — Immediate (this week)
1. **[Cutover] Complete cutover checklist** — the 5 pending items from `scripts/cutover-checklist.md`:
- Infisical bootstrap (or decide: defer and stay on SOPS)
- Watchdog end-to-end test
- Rollback drill (`scripts/rollback.sh` rehearsal)
- Rollback verify + re-deploy
- apps/105 cleanup (remove Gitea webhooks, archive LXC)
2. **[Docs] Create wiki pages for seanime (133) and romm (134)**
3. **[Plans] Mark TRMNL plan as done**, move to `plans/done/`, write changelogs
### P1 — Near-term (next 1-2 weeks)
4. **[Cleanup] Migrate `.hermes/plans/` → `plans/done/`** for all 7 files
5. **[Cleanup] Remove 9 superseded `oikos/*.py` files** (all except gen-topology*)
— after verifying `bin/homelab` doesn't import them
6. **[Docs] Audit and update strong.md + hubris.md** guest lists and resource math
7. **[Docs] Regenerate `knowledge/wiki/infrastructure/topology.md`** to reflect
current state (strong guests, new LXCs)
8. **[Plans] Update Prometheus plan** — port Python references to Go equivalents
### P2 — Medium-term (2-4 weeks)
9. **[Docs] ADR-0011** — document the completed Go rewrite
10. **[Docs] Document enrollment convention** — when LXCs get homelab-context vs.
when they skip it
11. **[Cleanup] Audit and resolve oikos/cards/** — verify whether still consumed
12. **[Cleanup] Traefik reference audit** — document or remove old traefik mentions
13. **[Plans] Complete wiki-hq adoption Phases 3–5** from
`plans/2026-07-06-adopt-wiki-hq-doc-architecture.md`:
- Phase 4 — runbooks → skills (reshape remaining runbooks into `.agents/skills/`)
- Phase 5 — writing-style + README pass + docs-lint
14. **[Code] Compare `mcp/build_host_files.py` vs `cmd/oikos/build_hosts.go`**
— remove duplicate if outputs match
### P3 — Backlog (when capacity allows)
15. **[Code] Port `oikos/gen-topology.py` to Go** — the last Python holdout
16. **[Code] Port `scripts/dns-sync.py` to Go** — add to `oikos` binary
17. **[Code] Fold `scripts/validate-seeds.py` into `oikos seed --validate`**
18. **[Infra] Prometheus LXC provisioning** — per the updated plan
19. **[Infra] Strong Phase 3** — migrate mule-images (120) per strong migration
assessment
20. **[Docs] Port `knowledge/sources/references/cert-sync-and-traefik-config.md`**
to reflect current Caddy-based cert management
---
## 6. File inventory — what to keep, what to remove
### Keep (active)
```
cmd/oikos/** cmd/hermes/**, internal/**, api/openapi.yaml, compose/**, migrations/**,
seeds/**, docs/adr/**, docker-compose.yml, Makefile, go.mod, go.sum, sqlc.yaml,
inventory.yaml, hosts/*.yaml, .sops.yaml, .gitignore, .gitea/**,
knowledge/**, .agents/**, plans/**, README.md, AGENTS.md,
hermes/**, oikos/gen-topology.py, oikos/gen_topology_lib.py,
scripts/{deploy,rollback,watchdog,verify-phase6}.sh, scripts/sync/**,
scripts/check-caddy-backends.sh, scripts/cutover-checklist.md,
secrets/**, secrets-issuance/**, bin/homelab, mcp/build_host_files.py,
ssh/**, tools/**, vps/**, ledger/**, bootstrap.sh,
scripts/dns-sync.py, scripts/validate-seeds.py (keep until ported)
```
### Remove
```
oikos/__init__.py, oikos/approve.py, oikos/decide.py, oikos/drift.py,
oikos/ledger.py, oikos/policy.py, oikos/relations.py, oikos/scheduler.py,
oikos/signal.py
```
### Migrate
```
.hermes/plans/*.md → plans/done/
oikos/cards/*.md → TBD (remove if unused, keep if bin/homelab reads them)
```
### Decide
```
secrets-issuance/ → port to Docker or decommission?
oikos/cards/ → still consumed by bin/homelab explain?
mcp/build_host_files.py → redundant with cmd/oikos/build_hosts.go?
```
---
## 7. Decisions requested
These require operator input before proceeding:
1. **Infisical now or later?** Bootstrap Infisical (Phase 5 production) now or defer
and continue with SOPS-only? Risk: Infisical adds Redis + Infisical container to
the Docker stack on mac-mini with non-trivial setup.
2. **Secrets issuance: port or kill?** If we decommission `secrets-issuance/`, new
clients generate their own age key and the operator adds the pubkey manually.
Simpler but loses automated enrollment.
3. **apps/105: archive or destroy?** After Docker cutover cleanup, does apps/105
stay as a warm spare or get archived per lifecycle rules?
4. **oikos/cards/: keep or drop?** Verify `bin/homelab explain` behavior. If it
queries the API, cards are dead weight. If it reads files, they stay until the
CLI is ported.
---
## Changelog
### 2026-07-07 — comprehensive audit created
Catalog of 20+ stale files, plan status drift, documentation gaps, and
prioritized operational backlog. Covers Python kernel cleanup, .hermes/plans/
migration, oikos/cards/ assessment, cutover completion, and missing wiki pages.

625
plans/2026-07-07-db-as-source-of-truth.md Normal file
View File

@@ -0,0 +1,625 @@
# 2026-07-07 — DB as single source of truth for agent knowledge
**Status:** Proposed
## Goal
Convert `knowledge/wiki/` into DB seeds, archive all original markdown files, and
archive every other file in the repo that the Oikos Go binary does not read. After
this, the DB is the **sole source of truth** for agents — one query surface, no grep
fallback. All old wiki files go to root `archive/`. New knowledge is registered via
the API, and `oikos export` writes it back to seed files for version control.
## Decisions from operator
| Question | Decision |
|----------|----------|
| Plans in DB? | No — agent construction docs only |
| Ingestion trigger | Seed-time (`oikos seed` reads `seeds/knowledge.yaml`) |
| Archive location | Root `archive/` |
| Content granularity | Structured sections: At-a-glance → entity attributes, procedures → runbooks, changelog → parsed entries, rest → `knowledge_entities.content` |
| Wiki files after conversion | **Archive them all** — DB is the only truth |
| Stale files not used by Oikos | **Archive them all** |
## What goes to `archive/`
Everything the Go binary (`cmd/oikos`, `internal/`) does **not** read at runtime.
### Full archive list
| Path | Destination | Why |
|------|-------------|-----|
| `knowledge/wiki/containers/*.md` (20 pages) | `archive/knowledge/containers/` | Converted to seeds |
| `knowledge/wiki/hosts/*.md` (3 pages) | `archive/knowledge/hosts/` | Converted to seeds |
| `knowledge/wiki/infrastructure/*.md` (12 pages) | `archive/knowledge/infrastructure/` | Converted to seeds |
| `knowledge/wiki/vms/*.md` (2 pages) | `archive/knowledge/vms/` | Converted to seeds |
| `knowledge/sources/investigations/*.md` (5 pages) | `archive/knowledge/investigations/` | Converted to seeds |
| `knowledge/sources/references/*.md` (1 page) | `archive/knowledge/references/` | Converted to seeds |
| `knowledge/sources/index.md` | `archive/knowledge/` | Index page, no entity mapping |
| `knowledge/index.md` | `archive/knowledge/` | Index page |
| `knowledge/log.md` | `archive/knowledge/` | Doc-maintenance log, superseded by audit_log |
| `knowledge/GLOSSARY.md` | `archive/knowledge/` | Glossary (can be re-ingested as `document:glossary` later) |
| `oikos/cards/` (45 files) | `archive/oikos-cards/` | Python MCP explain tool; superseded by DB entities |
| `.hermes/plans/` (7 files) | `archive/hermes-plans/` | Agent construction docs, never in DB |
| `ledger/2026-07.jsonl` | `archive/ledger/` | Python-era artifact; Go uses audit_log table |
| `mcp/build_host_files.py` | `archive/mcp/` | Redundant with `cmd/oikos/build_hosts.go` |
### Full delete list (no archival value)
| Path | Why |
|------|-----|
| `oikos/approve.py` | Superseded by `internal/domain/approval.go` |
| `oikos/decide.py` | Superseded by `internal/policy/classify.go` |
| `oikos/drift.py` | Superseded by `internal/scheduler/scheduler.go` |
| `oikos/ledger.py` | Superseded by `internal/db/` + `audit_log` table |
| `oikos/policy.py` | Superseded by `internal/policy/classify.go` |
| `oikos/relations.py` | Superseded by `blast_radius()` SQL function |
| `oikos/scheduler.py` | Superseded by `internal/scheduler/scheduler.go` |
| `oikos/signal.py` | Superseded by `signals` table + `internal/domain/signal.go` |
| `oikos/__init__.py` | Package init, never imported by Go binary |
### What stays (read by Oikos Go binary)
| Path | Read by | Why keep |
|------|---------|----------|
| `seeds/ontology.yaml` | `oikos seed` | DB bootstrap |
| `seeds/inventory.yaml` | `oikos seed` | DB bootstrap |
| `seeds/policy.yaml` | `oikos seed` | DB bootstrap |
| `seeds/knowledge.yaml` | `oikos seed` | **NEW** — knowledge bootstrap |
| `inventory.yaml` | `oikos build-hosts`, `oikos homelab list` | Topology source of truth |
| `secrets/*.yaml` | `oikos secret list`, `oikos homelab secret` | SOPS-encrypted secrets |
| `migrations/*.sql` | `oikos migrate` | Embedded SQL migrations |
| `api/openapi.yaml` | `make generate` (oapi-codegen) | API contract |
| `compose/`, `Dockerfile`, `docker-compose.yml` | Docker build/deploy | Runtime packaging |
| `cmd/`, `internal/` | `go build` | Source code |
| `bin/homelab` | `oikos homelab` subcommand | CLI binary |
| `oikos/gen-topology.py` | `bin/homelab` (Python shim) | Topology generation (until ported) |
| `oikos/gen_topology_lib.py` | imported by gen-topology.py | Same |
| `scripts/deploy.sh` | Gitea webhook → deploy | Production deploy |
| `scripts/rollback.sh` | Operator manual | Rollback |
| `scripts/watchdog.sh` | cron on mac-mini | Health monitoring |
| `scripts/verify-phase6.sh` | Operator manual | Acceptance verification |
| `scripts/sync/` | systemd/launchd timers | Repo sync to clients |
| `scripts/check-caddy-backends.sh` | Operator diagnostic | Health check utility |
| `scripts/cutover-checklist.md` | Operator reference | Cutover tracking |
| `scripts/dns-sync.py` | Operator manual (not yet ported) | DNS sync to Technitium |
| `scripts/validate-seeds.py` | CI / operator manual | Seed validation |
| `tools/` | `post-pull.sh` auto-setup | Client tooling install |
| `ssh/` | `homelab ssh` | SSH key deployment |
| `vps/` | `homelab render-vps-configs` | VPS templates |
| `secrets-issuance/` | Running on apps/105 | Identity issuance (until decommissioned) |
| `AGENTS.md` | Agent onboarding | Entry point |
| `README.md` | Human onboarding | Entry point |
| `.agents/` | Agent conventions, skills, operating model | Agent runtime guidance |
| `plans/` | Agent construction docs | Intent documents |
| `hermes/` | Hermes config + persona | Agent gateway config |
| `Makefile`, `go.mod`, `go.sum`, `sqlc.yaml`, `.gitignore`, `.sops.yaml`, `.gitea/` | Build/config | Build tooling |
### Ambiguous — needs confirmation
| Path | Question |
|------|----------|
| `.agents/` | Contains OIKOS.md (operating model), SKILL.md runbooks, conventions. Go binary doesn't read these at runtime — they're agent orientation docs. Should they also be converted to seeds and archived? Or do they stay as the "agent manual" alongside AGENTS.md? |
| `scripts/dns-sync.py`, `scripts/validate-seeds.py` | Actively used Python scripts with no Go equivalent. Keep until ported? |
| `secrets-issuance/` | Running on apps/105. Archive after decommission? |
## Architecture change
```
BEFORE (split):
knowledge/wiki/ → agents grep clone for narrative
Postgres DB → agents query MCP for structured data
Two surfaces, no bridge
AFTER (DB-only):
seeds/knowledge.yaml → oikos seed → knowledge_entities table
↓
MCP search_knowledge("jellyfin transcode")
MCP get_entity_knowledge("lxc:jellyfin")
HTTP GET /api/v1/knowledge/search
↓
One surface, everything indexed
```
```
┌──────────────────────────────────┐
│ seeds/knowledge.yaml │ Version-controlled source material
│ (generated by oikos export) │ Round-trip: seed → DB → export → seed
│ documents: │
│ - slug: containers/101-jellyfin│
│ entity: lxc:jellyfin │
│ title: "101 — jellyfin" │
│ content: "..." │
│ at_glance: {host: strong, ..}│
│ changelog: [{date, title, ..}]│
│ tags: [container, media, ..] │
│ investigations: │
│ - slug: 2026-06-06-caddy-... │
│ about: [service:caddy, ...] │
│ ... │
│ runbooks: │
│ - slug: service-health-check │
│ procedure_for: [service] │
│ ... │
└──────────────┬───────────────────┘
│ oikos seed
▼
┌──────────────────────────────────┐
│ knowledge_entities table │
│ title | content | content_hash │
│ source | tags │
│ ─ FTS: ts_vector(title+content) │
│ │
│ + entities table: │
│ document:containers/101-jellyfin│
│ investigation:2026-06-06-... │
│ runbook:service-health-check │
│ │
│ + relationships: │
│ document ──documents──► lxc │
│ investigation ──about──► service│
│ runbook ──procedure-for──► type│
└──────────────────────────────────┘
│
▼
┌──────────────────────────────────┐
│ MCP + HTTP API │
│ search_knowledge("jellyfin") │
│ get_entity_knowledge("lxc:...") │
│ POST /api/v1/knowledge/{id} │
└──────────────────────────────────┘
```
## Execution — Fresh install on mac-mini
The current Docker stack on mac-mini was deployed during cutover testing with dev data.
Replace it with a fresh production instance using the real seeds (ontology, inventory,
policy, knowledge).
### Steps
```bash
# On mac-mini (192.168.8.175):
# 1. Stop current stack
docker compose --profile full down
# 2. Wipe old Postgres volume (fresh start)
docker volume rm oikos_pg-data
# 3. Pull latest repo (already committed to local main)
git pull origin main
# 4. Rebuild image with current code
docker compose build
# 5. Fresh migrate + seed (now includes knowledge)
docker compose run --rm migrate
docker compose run --rm seed
# 6. Verify seed ingested
docker compose run --rm -e OIKOS_DATABASE_URL=... oikos \
psql -c "SELECT count(*) FROM knowledge_entities;"
# Expect: 54+ rows
# 7. Start production stack
docker compose --profile full up -d
# 8. Health check
curl http://localhost:8090/healthz
./scripts/verify-phase6.sh
# 9. Verify knowledge endpoints
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin"
```
After fresh install succeeds, the old `backups/pre-cutover-20260707.sql` dump becomes
stale (it's from the dev instance). The new instance is the canonical production DB.
---
## Phase 1 — Seed format + ingestion
### 1.1 Add `content_hash` to `knowledge_entities`
**Migration 010:**
```sql
ALTER TABLE knowledge_entities ADD COLUMN content_hash TEXT;
```
### 1.2 Seed format: `seeds/knowledge.yaml`
One YAML file with three sections. Ingested by `oikos seed` after ontology, inventory,
and policy.
```yaml
# Oikos knowledge seed — documents, investigations, and runbooks.
# Generated by oikos export. Ingested on deploy.
# After ingest the DB is authoritative.
version: 1
documents:
- slug: "containers/101-jellyfin"
title: "101 — jellyfin"
content: |
# 101 — jellyfin
Jellyfin media server with hardware-accelerated transcoding...
## Service / port
| Service | Port | Notes |
|---------|------|-------|
| jellyfin | 8096 | Web UI + API |
...
## SSO
Authentik OIDC via SSO-Auth plugin v4.0.0.4. No Caddy forward-auth gate.
...
entity_slug: "lxc:jellyfin" # links via `documents` edge
tags: ["container", "media", "jellyfin"]
at_glance: # structured: pushed to entity attributes if empty
host: strong
ip: "192.168.8.246"
cores: 4
ram: "8 GiB"
mounts: ["/mnt/media_local"]
public_host: media.hubris.network
changelog:
- date: "2026-07-05"
title: "migrated from hubris to strong"
body: "Phase 2 of strong migration..."
- date: "2026-06-10"
title: "VAAPI configured on Radeon 680M"
body: "GPU passthrough via dev0 + dev1..."
- slug: "hosts/strong"
title: "strong — Proxmox host"
content: |
# strong — Proxmox host
Reformatted from Linux workstation to Proxmox VE 9.2.3...
entity_slug: "proxmox-host:strong"
tags: ["host", "proxmox", "hypervisor"]
at_glance:
role: hypervisor
hardware: "Minisforum UM773 Lite (Ryzen 7 PRO 6850U, 28 GiB RAM)"
os: linux
changelog:
- date: "2026-07-05"
title: "strong migration Phase 2 complete"
body: "arriman + jellyfin + grimmory migrated. ludo-lvm..."
# ... 35+ more document entries
investigations:
- slug: "2026-06-06-caddyfile-truncation"
title: "Caddyfile truncation — all LAN services down"
date: "2026-06-06"
status: resolved
duration: "~45 min"
content: |
## Symptom
All *.hubris.network services returned 502...
## Root cause
...
about_slugs: ["service:caddy", "service:dns"] # links via `about` edges
tags: ["incident", "caddy", "dns", "downtime"]
- slug: "2026-06-06-authentik-session-lifetime"
title: "Frequent Authentik login prompts"
date: "2026-06-06"
status: resolved
duration: "~30 min"
content: |
## Summary
...
about_slugs: ["service:authentik"]
tags: ["incident", "authentik", "oidc"]
# ... 3 more investigation entries
runbooks:
- slug: "service-health-check"
name: "Service health check"
risk_class: read_only
entity_type: "service" # applies to all service entities
procedure:
params_schema:
type: object
properties:
unit: {type: string}
required: ["unit"]
steps:
- name: "check systemd unit"
runner: ssh
command: "systemctl is-active {{ .unit }}"
timeout_s: 10
verify:
- runner: ssh
command: "systemctl is-active {{ .unit }}"
expect: {exit_code: 0, stdout_contains: "active"}
- slug: "lifecycle-provision-node"
name: "Provision new node"
risk_class: config_mutation
entity_type: "lxc"
procedure:
params_schema:
type: object
properties:
name: {type: string}
template: {type: string}
cores: {type: integer}
memory_mb: {type: integer}
required: ["name", "template"]
steps:
- name: "create container"
runner: ssh
target: "{{ .host }}"
command: "pct create $(pvesh get /nextid) {{ .template }} ..."
timeout_s: 120
verify: [...]
# ... 10 more runbook entries
```
### 1.3 Ingest logic
`oikos seed` already runs: migrate → ontology → inventory → policy.
Add knowledge as step 5:
```
oikos seed
1. migrate (DDL)
2. ontology seed (entity_types, relationship_types, lifecycles)
3. inventory seed (entities, relationships)
4. policy seed (risk_classes, approval_rules, autonomy)
5. knowledge seed (documents, investigations, runbooks) ← NEW
```
Knowledge ingest for each section:
**Documents:**
- For each entry, create `document:<slug>` entity in `entities` table (type: `document`)
- Insert into `knowledge_entities` (title, content, source=slug, tags, content_hash)
- Create `documents` edge from `document:<slug>` to `entity_slug`
- Push `at_glance` fields into the linked entity's `attributes` if those keys are empty (backfill)
- Store `changelog` as JSONB in document entity `attributes.changelog`
- Skip if `content_hash` matches existing row (idempotent)
**Investigations:**
- Create `investigation:<slug>` entity (type: `investigation`, lifecycle: `infrastructure`)
- Insert into `knowledge_entities`
- Create `about` edges to each entity in `about_slugs`
- Store `date`, `status`, `duration` in entity `attributes`
**Runbooks:**
- Create `runbook:<slug>` entity (type: `runbook`, lifecycle: `skill`)
- Insert into `knowledge_entities`
- Create `procedure-for` edges to `entity_type` (abstract type — applies to all instances)
- If a `skills` row with matching slug already exists, only update `knowledge_entities.content` and `content_hash` (don't overwrite the structured `procedure` JSONB in the skills table — that's managed separately)
### 1.4 Export: DB → seed
`oikos export` already writes `seeds/ontology.yaml`, `seeds/inventory.yaml`,
`seeds/policy.yaml`. Add `seeds/knowledge.yaml`:
```go
// internal/db/export.go
func ExportKnowledge(ctx context.Context, pool *pgxpool.Pool) ([]byte, error) {
// SELECT all document + investigation + runbook entities
// JOIN knowledge_entities for content
// JOIN relationships for entity links
// Serialize to seeds/knowledge.yaml format
}
```
Round-trip guarantee: `seed → DB → export → seed` is byte-stable. Tested in CI.
## Phase 2 — Convert wiki to seeds, archive originals
### 2.1 Conversion script (one-time)
```
oikos knowledge convert [--dry-run]
```
- Reads every page in `knowledge/wiki/`, `knowledge/sources/`, `knowledge/GLOSSARY.md`
- Parses structured sections (at-glance, changelog, procedures)
- Maps each page to entity slugs using the path convention + `inventory.yaml` doc_page
- Generates `seeds/knowledge.yaml`
- Reports: "would write N documents, M investigations, K runbooks"
- Without `--dry-run`: writes `seeds/knowledge.yaml`
Implementation: initially a Go version in `cmd/oikos/`, can fall back to a one-shot
Python script if that's faster to build.
### 2.2 Archive
After conversion verified (seeds ingest cleanly):
```bash
# Create archive directory
mkdir -p archive/knowledge/{containers,hosts,infrastructure,vms,investigations,references}
# Move all knowledge files
mv knowledge/wiki/containers/*.md archive/knowledge/containers/
mv knowledge/wiki/hosts/*.md archive/knowledge/hosts/
mv knowledge/wiki/infrastructure/*.md archive/knowledge/infrastructure/
mv knowledge/wiki/vms/*.md archive/knowledge/vms/
mv knowledge/sources/investigations/*.md archive/knowledge/investigations/
mv knowledge/sources/references/*.md archive/knowledge/references/
mv knowledge/index.md knowledge/log.md knowledge/GLOSSARY.md archive/knowledge/
# Move other stale artifacts
mv oikos/cards/ archive/oikos-cards/
mv .hermes/plans/*.md archive/hermes-plans/
mv ledger/2026-07.jsonl archive/ledger/
mv mcp/build_host_files.py archive/mcp/
# Delete superseded Python
rm oikos/__init__.py oikos/approve.py oikos/decide.py oikos/drift.py \
oikos/ledger.py oikos/policy.py oikos/relations.py oikos/scheduler.py \
oikos/signal.py
# Clean empty directories
rmdir knowledge/sources/investigations/archive/ # already moved
rmdir knowledge/sources/investigations/
rmdir knowledge/sources/references/
rmdir knowledge/sources/
rmdir knowledge/wiki/containers/archive/ # already moved
rmdir knowledge/wiki/containers/
rmdir knowledge/wiki/hosts/
rmdir knowledge/wiki/infrastructure/
rmdir knowledge/wiki/vms/
rmdir knowledge/wiki/
```
### 2.3 What the repo looks like after
```
/
├── seeds/
│ ├── ontology.yaml
│ ├── inventory.yaml
│ ├── policy.yaml
│ └── knowledge.yaml ← NEW: documents + investigations + runbooks
├── archive/
│ ├── knowledge/ ← all old wiki markdown
│ ├── oikos-cards/ ← 45 Python explain cards
│ ├── hermes-plans/ ← 7 agent construction docs
│ ├── ledger/ ← Python-era JSONL
│ └── mcp/ ← old Python build_host_files.py
├── .agents/ ← agent conventions + skills (stays)
├── plans/ ← construction plans (stays)
├── cmd/, internal/ ← Go source
├── migrations/, api/, compose/
├── inventory.yaml, hosts/*.yaml
├── AGENTS.md, README.md
└── ...
```
The repo shrinks by ~65 narrative files and ~45 card files. All knowledge is in
`seeds/knowledge.yaml` — one file, git-tracked, round-trippable.
## Phase 3 — Un-stub MCP + HTTP
Same as previous revision:
- `search_knowledge(query)` → PostgreSQL FTS with `ts_rank`, `ts_headline`
- `get_entity_knowledge(slug)` → aggregates documents, investigations, runbooks linked to entity
- `GET /api/v1/knowledge/search` → FTS endpoint
- `GET /api/v1/knowledge/{slug}` → entity-linked knowledge
- `POST /api/v1/knowledge/{slug}` → agent registers new knowledge
- `PATCH /api/v1/knowledge/{slug}` → agent updates knowledge
## Phase 4 — Agent conventions
### Knowledge registration
Agents call `POST /api/v1/knowledge/{entity_slug}` when they:
- Discover an undocumented detail (e.g., config quirk, gotcha)
- Complete an investigation
- Create a new entity and need to attach docs
- Update entity attributes discovered during operation
### Knowledge query (single call pattern)
```python
# One call replaces grep + get_entity + get_relations
ctx = mcp.get_entity_knowledge("lxc:jellyfin")
# Returns:
# - entity attributes (structured)
# - documents (full-text indexed narrative)
# - investigations (any incidents involving this entity)
# - runbooks (procedures that apply to this entity type)
# - relationships (blast radius, dependencies)
```
### Export cycle
```
agent discovers → POST /api/v1/knowledge → DB updated
operator reviews → oikos export → seeds/knowledge.yaml updated
git commit + push → version-controlled, diffable
next deploy → oikos seed → DB back in sync
```
## Files to create/modify
| File | Action |
|------|--------|
| `migrations/010_knowledge_hash.up.sql` | New — `content_hash TEXT` |
| `internal/knowledge/seed.go` | New — knowledge seed ingest (read seeds/knowledge.yaml → DB) |
| `internal/knowledge/seed_test.go` | New — round-trip tests |
| `internal/knowledge/convert.go` | New — one-shot wiki→seed converter |
| `internal/knowledge/parser.go` | New — section parser (at-glance, changelog) |
| `internal/db/seed.go` | Modify — add knowledge step after policy |
| `internal/db/export.go` | Modify — add `ExportKnowledge` |
| `internal/httpapi/impl.go` | Modify — un-stub knowledge endpoints |
| `internal/httpapi/knowledge.go` | New — knowledge handlers |
| `internal/mcp/server.go` | Modify — un-stub `search_knowledge`, `get_entity_knowledge` |
| `cmd/oikos/main.go` | Modify — add `knowledge convert` subcommand |
| `api/openapi.yaml` | Modify — knowledge endpoint schemas |
| `seeds/knowledge.yaml` | New — generated by converter, ingested on seed |
| `archive/` | New — all old wiki + stale files |
| `plans/2026-07-07-comprehensive-audit-and-next-steps.md` | Modify — mark knowledge items as addressed |
## Verification
```bash
# 1. Convert wiki to seed
oikos knowledge convert --dry-run
# → "would write 37 documents, 5 investigations, 12 runbooks"
oikos knowledge convert
# 2. Fresh DB, full seed
oikos migrate && oikos seed
# 3. Verify knowledge populated
psql -c "SELECT title, source, cardinality(tags) FROM knowledge_entities;"
# → 54 rows (37 docs + 5 investigations + 12 runbooks)
# 4. Verify FTS
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin+transcode"
# → returns document:containers/101-jellyfin with ts_headline snippet
# 5. Verify entity-linked knowledge
curl "http://localhost:8090/api/v1/knowledge/lxc:jellyfin"
# → {entity, documents: [...], investigations: [], runbooks: [...], relationships: {...}}
# 6. Verify MCP
curl -X POST localhost:8092/query \
-d '{"tool":"search_knowledge","args":{"query":"caddy truncation"}}'
# → returns investigation:2026-06-06-caddyfile-truncation
# 7. Verify agent registration
curl -X POST localhost:8090/api/v1/knowledge/lxc:jellyfin \
-H "Content-Type: application/json" \
-d '{"title":"VAAPI note","content":"Requires /dev/dri/renderD128 passthrough","tags":["gpu"]}'
# → 201 Created
# 8. Verify export round-trip
oikos export # writes seeds/*.yaml
oikos seed # re-ingests (skips all — hashes match)
# → "knowledge: ingested 0, skipped 54, errors 0"
# 9. Archive originals (manual, after verification)
# → move all wiki files to archive/ as listed in Phase 2.2
# 10. Verify repo still builds and deploys
make generate && make build && make test
docker compose --profile dev up -d
./scripts/verify-phase6.sh
```
---
## Changelog
### 2026-07-07 — v2: full archive, wiki→seeds only
Convert wiki to seeds, archive ALL originals, archive all stale files. No human-editing
frontend retained — DB is the only source of truth. Knowledge cycle: agent API →
oikos export → seeds/knowledge.yaml → git → oikos seed → DB.

2
plans/index.md
View File

@@ -12,6 +12,8 @@ went sideways, open an investigation.
| 2026-07-05 | [Oikos Prometheus LXC](2026-07-05-oikos-prometheus-lxc.md) | Planned |
| 2026-07-06 | [Adopt wiki-hq doc architecture](2026-07-06-adopt-wiki-hq-doc-architecture.md) | In Progress |
| 2026-07-06 | [Consolidate Oikos control plane onto mac-mini](2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md) | In Progress (Phase 1-6 implemented, pending cutover) |
| 2026-07-07 | [Comprehensive audit: stale files, state gaps, and next steps](2026-07-07-comprehensive-audit-and-next-steps.md) | Planned |
| 2026-07-07 | [DB as single source of truth for agent knowledge](2026-07-07-db-as-source-of-truth.md) | Proposed |
## Done