oikos/plans/2026-07-07-db-as-source-of-truth.md

2026-07-07 — DB as single source of truth for agent knowledge

Status: Done (2026-07-08) — wiki archived, FTS live, MCP + HTTP knowledge surface complete.

Goal

Convert knowledge/wiki/ into DB seeds, archive all original markdown files, and archive every other file in the repo that the Oikos Go binary does not read. After this, the DB is the sole source of truth for agents — one query surface, no grep fallback. All old wiki files go to root archive/. New knowledge is registered via the API, and oikos export writes it back to seed files for version control.

Decisions from operator

Question	Decision
Plans in DB?	No — agent construction docs only
Ingestion trigger	Seed-time (oikos seed reads seeds/knowledge.yaml)
Archive location	Root archive/
Content granularity	Structured sections: At-a-glance → entity attributes, procedures → runbooks, changelog → parsed entries, rest → knowledge_entities.content
Wiki files after conversion	Archive them all — DB is the only truth
Stale files not used by Oikos	Archive them all

What goes to archive/

Everything the Go binary (cmd/oikos, internal/) does not read at runtime.

Full archive list

Path	Destination	Why
knowledge/wiki/containers/*.md (20 pages)	archive/knowledge/containers/	Converted to seeds
knowledge/wiki/hosts/*.md (3 pages)	archive/knowledge/hosts/	Converted to seeds
knowledge/wiki/infrastructure/*.md (12 pages)	archive/knowledge/infrastructure/	Converted to seeds
knowledge/wiki/vms/*.md (2 pages)	archive/knowledge/vms/	Converted to seeds
knowledge/sources/investigations/*.md (5 pages)	archive/knowledge/investigations/	Converted to seeds
knowledge/sources/references/*.md (1 page)	archive/knowledge/references/	Converted to seeds
knowledge/sources/index.md	archive/knowledge/	Index page, no entity mapping
knowledge/index.md	archive/knowledge/	Index page
knowledge/log.md	archive/knowledge/	Doc-maintenance log, superseded by audit_log
knowledge/GLOSSARY.md	archive/knowledge/	Glossary (can be re-ingested as document:glossary later)
oikos/cards/ (45 files)	archive/oikos-cards/	Python MCP explain tool; superseded by DB entities
.hermes/plans/ (7 files)	archive/hermes-plans/	Agent construction docs, never in DB
ledger/2026-07.jsonl	archive/ledger/	Python-era artifact; Go uses audit_log table
mcp/build_host_files.py	archive/mcp/	Redundant with cmd/oikos/build_hosts.go

Full delete list (no archival value)

Path	Why
oikos/approve.py	Superseded by internal/domain/approval.go
oikos/decide.py	Superseded by internal/policy/classify.go
oikos/drift.py	Superseded by internal/scheduler/scheduler.go
oikos/ledger.py	Superseded by internal/db/ + audit_log table
oikos/policy.py	Superseded by internal/policy/classify.go
oikos/relations.py	Superseded by blast_radius() SQL function
oikos/scheduler.py	Superseded by internal/scheduler/scheduler.go
oikos/signal.py	Superseded by signals table + internal/domain/signal.go
oikos/__init__.py	Package init, never imported by Go binary

What stays (read by Oikos Go binary)

Path	Read by	Why keep
seeds/ontology.yaml	oikos seed	DB bootstrap
seeds/inventory.yaml	oikos seed	DB bootstrap
seeds/policy.yaml	oikos seed	DB bootstrap
seeds/knowledge.yaml	oikos seed	NEW — knowledge bootstrap
inventory.yaml	oikos build-hosts, oikos homelab list	Topology source of truth
secrets/*.yaml	oikos secret list, oikos homelab secret	SOPS-encrypted secrets
migrations/*.sql	oikos migrate	Embedded SQL migrations
api/openapi.yaml	make generate (oapi-codegen)	API contract
compose/, Dockerfile, docker-compose.yml	Docker build/deploy	Runtime packaging
cmd/, internal/	go build	Source code
bin/homelab	oikos homelab subcommand	CLI binary
oikos/gen-topology.py	bin/homelab (Python shim)	Topology generation (until ported)
oikos/gen_topology_lib.py	imported by gen-topology.py	Same
scripts/deploy.sh	Gitea webhook → deploy	Production deploy
scripts/rollback.sh	Operator manual	Rollback
scripts/watchdog.sh	cron on mac-mini	Health monitoring
scripts/verify-phase6.sh	Operator manual	Acceptance verification
scripts/sync/	systemd/launchd timers	Repo sync to clients
scripts/check-caddy-backends.sh	Operator diagnostic	Health check utility
scripts/cutover-checklist.md	Operator reference	Cutover tracking
scripts/dns-sync.py	Operator manual (not yet ported)	DNS sync to Technitium
scripts/validate-seeds.py	CI / operator manual	Seed validation
tools/	post-pull.sh auto-setup	Client tooling install
ssh/	homelab ssh	SSH key deployment
vps/	homelab render-vps-configs	VPS templates
secrets-issuance/	Running on apps/105	Identity issuance (until decommissioned)
AGENTS.md	Agent onboarding	Entry point
README.md	Human onboarding	Entry point
.agents/	Agent conventions, skills, operating model	Agent runtime guidance
plans/	Agent construction docs	Intent documents
hermes/	Hermes config + persona	Agent gateway config
Makefile, go.mod, go.sum, sqlc.yaml, .gitignore, .sops.yaml, .gitea/	Build/config	Build tooling

Ambiguous — needs confirmation

Path	Question
.agents/	Contains OIKOS.md (operating model), SKILL.md runbooks, conventions. Go binary doesn't read these at runtime — they're agent orientation docs. Should they also be converted to seeds and archived? Or do they stay as the "agent manual" alongside AGENTS.md?
scripts/dns-sync.py, scripts/validate-seeds.py	Actively used Python scripts with no Go equivalent. Keep until ported?
secrets-issuance/	Running on apps/105. Archive after decommission?

Architecture change

BEFORE (split):
  knowledge/wiki/  →  agents grep clone for narrative
  Postgres DB      →  agents query MCP for structured data
  Two surfaces, no bridge

AFTER (DB-only):
  seeds/knowledge.yaml  →  oikos seed  →  knowledge_entities table
                                            ↓
                                     MCP search_knowledge("jellyfin transcode")
                                     MCP get_entity_knowledge("lxc:jellyfin")
                                     HTTP GET /api/v1/knowledge/search
                                            ↓
                                     One surface, everything indexed

┌──────────────────────────────────┐
│  seeds/knowledge.yaml             │  Version-controlled source material
│  (generated by oikos export)      │  Round-trip: seed → DB → export → seed
│  documents:                       │
│    - slug: containers/101-jellyfin│
│      entity: lxc:jellyfin         │
│      title: "101 — jellyfin"      │
│      content: "..."               │
│      at_glance: {host: strong, ..}│
│      changelog: [{date, title, ..}]│
│      tags: [container, media, ..] │
│  investigations:                  │
│    - slug: 2026-06-06-caddy-...   │
│      about: [service:caddy, ...]  │
│      ...                          │
│  runbooks:                        │
│    - slug: service-health-check   │
│      procedure_for: [service]     │
│      ...                          │
└──────────────┬───────────────────┘
               │ oikos seed
               ▼
┌──────────────────────────────────┐
│  knowledge_entities table         │
│  title | content | content_hash   │
│  source | tags                    │
│  ─ FTS: ts_vector(title+content)  │
│                                   │
│  + entities table:                │
│    document:containers/101-jellyfin│
│    investigation:2026-06-06-...   │
│    runbook:service-health-check   │
│                                   │
│  + relationships:                 │
│    document ──documents──► lxc    │
│    investigation ──about──► service│
│    runbook ──procedure-for──► type│
└──────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────┐
│  MCP + HTTP API                   │
│  search_knowledge("jellyfin")     │
│  get_entity_knowledge("lxc:...")  │
│  POST /api/v1/knowledge/{id}      │
└──────────────────────────────────┘

Execution — Fresh install on mac-mini

The current Docker stack on mac-mini was deployed during cutover testing with dev data. Replace it with a fresh production instance using the real seeds (ontology, inventory, policy, knowledge).

Steps

# On mac-mini (192.168.8.175):

# 1. Stop current stack
docker compose --profile full down

# 2. Wipe old Postgres volume (fresh start)
docker volume rm oikos_pg-data

# 3. Pull latest repo (already committed to local main)
git pull origin main

# 4. Rebuild image with current code
docker compose build

# 5. Fresh migrate + seed (now includes knowledge)
docker compose run --rm migrate
docker compose run --rm seed

# 6. Verify seed ingested
docker compose run --rm -e OIKOS_DATABASE_URL=... oikos \
  psql -c "SELECT count(*) FROM knowledge_entities;"
# Expect: 54+ rows

# 7. Start production stack
docker compose --profile full up -d

# 8. Health check
curl http://localhost:8090/healthz
./scripts/verify-phase6.sh

# 9. Verify knowledge endpoints
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin"

After fresh install succeeds, the old backups/pre-cutover-20260707.sql dump becomes stale (it's from the dev instance). The new instance is the canonical production DB.

---

Phase 1 — Seed format + ingestion

1.1 Add content_hash to knowledge_entities

Migration 010:

ALTER TABLE knowledge_entities ADD COLUMN content_hash TEXT;

1.2 Seed format: seeds/knowledge.yaml

One YAML file with three sections. Ingested by oikos seed after ontology, inventory, and policy.

# Oikos knowledge seed — documents, investigations, and runbooks.
# Generated by oikos export. Ingested on deploy.
# After ingest the DB is authoritative.

version: 1

documents:
  - slug: "containers/101-jellyfin"
    title: "101 — jellyfin"
    content: |
      # 101 — jellyfin

      Jellyfin media server with hardware-accelerated transcoding...

      ## Service / port
      | Service | Port | Notes |
      |---------|------|-------|
      | jellyfin | 8096 | Web UI + API |
      ...

      ## SSO
      Authentik OIDC via SSO-Auth plugin v4.0.0.4. No Caddy forward-auth gate.
      ...
    entity_slug: "lxc:jellyfin"         # links via `documents` edge
    tags: ["container", "media", "jellyfin"]
    at_glance:                          # structured: pushed to entity attributes if empty
      host: strong
      ip: "192.168.8.246"
      cores: 4
      ram: "8 GiB"
      mounts: ["/mnt/media_local"]
      public_host: media.hubris.network
    changelog:
      - date: "2026-07-05"
        title: "migrated from hubris to strong"
        body: "Phase 2 of strong migration..."
      - date: "2026-06-10"
        title: "VAAPI configured on Radeon 680M"
        body: "GPU passthrough via dev0 + dev1..."

  - slug: "hosts/strong"
    title: "strong — Proxmox host"
    content: |
      # strong — Proxmox host
      Reformatted from Linux workstation to Proxmox VE 9.2.3...
    entity_slug: "proxmox-host:strong"
    tags: ["host", "proxmox", "hypervisor"]
    at_glance:
      role: hypervisor
      hardware: "Minisforum UM773 Lite (Ryzen 7 PRO 6850U, 28 GiB RAM)"
      os: linux
    changelog:
      - date: "2026-07-05"
        title: "strong migration Phase 2 complete"
        body: "arriman + jellyfin + grimmory migrated. ludo-lvm..."

  # ... 35+ more document entries

investigations:
  - slug: "2026-06-06-caddyfile-truncation"
    title: "Caddyfile truncation — all LAN services down"
    date: "2026-06-06"
    status: resolved
    duration: "~45 min"
    content: |
      ## Symptom
      All *.hubris.network services returned 502...
      ## Root cause
      ...
    about_slugs: ["service:caddy", "service:dns"]  # links via `about` edges
    tags: ["incident", "caddy", "dns", "downtime"]

  - slug: "2026-06-06-authentik-session-lifetime"
    title: "Frequent Authentik login prompts"
    date: "2026-06-06"
    status: resolved
    duration: "~30 min"
    content: |
      ## Summary
      ...
    about_slugs: ["service:authentik"]
    tags: ["incident", "authentik", "oidc"]

  # ... 3 more investigation entries

runbooks:
  - slug: "service-health-check"
    name: "Service health check"
    risk_class: read_only
    entity_type: "service"              # applies to all service entities
    procedure:
      params_schema:
        type: object
        properties:
          unit: {type: string}
        required: ["unit"]
      steps:
        - name: "check systemd unit"
          runner: ssh
          command: "systemctl is-active {{ .unit }}"
          timeout_s: 10
      verify:
        - runner: ssh
          command: "systemctl is-active {{ .unit }}"
          expect: {exit_code: 0, stdout_contains: "active"}

  - slug: "lifecycle-provision-node"
    name: "Provision new node"
    risk_class: config_mutation
    entity_type: "lxc"
    procedure:
      params_schema:
        type: object
        properties:
          name: {type: string}
          template: {type: string}
          cores: {type: integer}
          memory_mb: {type: integer}
        required: ["name", "template"]
      steps:
        - name: "create container"
          runner: ssh
          target: "{{ .host }}"
          command: "pct create $(pvesh get /nextid) {{ .template }} ..."
          timeout_s: 120
      verify: [...]

  # ... 10 more runbook entries

1.3 Ingest logic

oikos seed already runs: migrate → ontology → inventory → policy.

Add knowledge as step 5:

oikos seed
  1. migrate (DDL)
  2. ontology seed (entity_types, relationship_types, lifecycles)
  3. inventory seed (entities, relationships)
  4. policy seed (risk_classes, approval_rules, autonomy)
  5. knowledge seed (documents, investigations, runbooks)  ← NEW

Knowledge ingest for each section:

Documents:

• For each entry, create document:<slug> entity in entities table (type: document)
• Insert into knowledge_entities (title, content, source=slug, tags, content_hash)
• Create documents edge from document:<slug> to entity_slug
• Push at_glance fields into the linked entity's attributes if those keys are empty (backfill)
• Store changelog as JSONB in document entity attributes.changelog
• Skip if content_hash matches existing row (idempotent)

Investigations:

• Create investigation:<slug> entity (type: investigation, lifecycle: infrastructure)
• Insert into knowledge_entities
• Create about edges to each entity in about_slugs
• Store date, status, duration in entity attributes

Runbooks:

• Create runbook:<slug> entity (type: runbook, lifecycle: skill)
• Insert into knowledge_entities
• Create procedure-for edges to entity_type (abstract type — applies to all instances)
• If a skills row with matching slug already exists, only update knowledge_entities.content and content_hash (don't overwrite the structured procedure JSONB in the skills table — that's managed separately)

1.4 Export: DB → seed

oikos export already writes seeds/ontology.yaml, seeds/inventory.yaml, seeds/policy.yaml. Add seeds/knowledge.yaml:

// internal/db/export.go
func ExportKnowledge(ctx context.Context, pool *pgxpool.Pool) ([]byte, error) {
    // SELECT all document + investigation + runbook entities
    // JOIN knowledge_entities for content
    // JOIN relationships for entity links
    // Serialize to seeds/knowledge.yaml format
}

Round-trip guarantee: seed → DB → export → seed is byte-stable. Tested in CI.

Phase 2 — Convert wiki to seeds, archive originals

2.1 Conversion script (one-time)

oikos knowledge convert [--dry-run]

• Reads every page in knowledge/wiki/, knowledge/sources/, knowledge/GLOSSARY.md
• Parses structured sections (at-glance, changelog, procedures)
• Maps each page to entity slugs using the path convention + inventory.yaml doc_page
• Generates seeds/knowledge.yaml
• Reports: "would write N documents, M investigations, K runbooks"
• Without --dry-run: writes seeds/knowledge.yaml

Implementation: initially a Go version in cmd/oikos/, can fall back to a one-shot Python script if that's faster to build.

2.2 Archive

After conversion verified (seeds ingest cleanly):

# Create archive directory
mkdir -p archive/knowledge/{containers,hosts,infrastructure,vms,investigations,references}

# Move all knowledge files
mv knowledge/wiki/containers/*.md archive/knowledge/containers/
mv knowledge/wiki/hosts/*.md archive/knowledge/hosts/
mv knowledge/wiki/infrastructure/*.md archive/knowledge/infrastructure/
mv knowledge/wiki/vms/*.md archive/knowledge/vms/
mv knowledge/sources/investigations/*.md archive/knowledge/investigations/
mv knowledge/sources/references/*.md archive/knowledge/references/
mv knowledge/index.md knowledge/log.md knowledge/GLOSSARY.md archive/knowledge/

# Move other stale artifacts
mv oikos/cards/ archive/oikos-cards/
mv .hermes/plans/*.md archive/hermes-plans/
mv ledger/2026-07.jsonl archive/ledger/
mv mcp/build_host_files.py archive/mcp/

# Delete superseded Python
rm oikos/__init__.py oikos/approve.py oikos/decide.py oikos/drift.py \
   oikos/ledger.py oikos/policy.py oikos/relations.py oikos/scheduler.py \
   oikos/signal.py

# Clean empty directories
rmdir knowledge/sources/investigations/archive/  # already moved
rmdir knowledge/sources/investigations/
rmdir knowledge/sources/references/
rmdir knowledge/sources/
rmdir knowledge/wiki/containers/archive/          # already moved
rmdir knowledge/wiki/containers/
rmdir knowledge/wiki/hosts/
rmdir knowledge/wiki/infrastructure/
rmdir knowledge/wiki/vms/
rmdir knowledge/wiki/

2.3 What the repo looks like after

/
├── seeds/
│   ├── ontology.yaml
│   ├── inventory.yaml
│   ├── policy.yaml
│   └── knowledge.yaml        ← NEW: documents + investigations + runbooks
├── archive/
│   ├── knowledge/            ← all old wiki markdown
│   ├── oikos-cards/          ← 45 Python explain cards
│   ├── hermes-plans/         ← 7 agent construction docs
│   ├── ledger/               ← Python-era JSONL
│   └── mcp/                  ← old Python build_host_files.py
├── .agents/                  ← agent conventions + skills (stays)
├── plans/                    ← construction plans (stays)
├── cmd/, internal/           ← Go source
├── migrations/, api/, compose/
├── inventory.yaml, hosts/*.yaml
├── AGENTS.md, README.md
└── ...

The repo shrinks by ~65 narrative files and ~45 card files. All knowledge is in seeds/knowledge.yaml — one file, git-tracked, round-trippable.

Phase 3 — Un-stub MCP + HTTP

Same as previous revision:

• search_knowledge(query) → PostgreSQL FTS with ts_rank, ts_headline
• get_entity_knowledge(slug) → aggregates documents, investigations, runbooks linked to entity
• GET /api/v1/knowledge/search → FTS endpoint
• GET /api/v1/knowledge/{slug} → entity-linked knowledge
• POST /api/v1/knowledge/{slug} → agent registers new knowledge
• PATCH /api/v1/knowledge/{slug} → agent updates knowledge

Phase 4 — Agent conventions

Knowledge registration

Agents call POST /api/v1/knowledge/{entity_slug} when they:

• Discover an undocumented detail (e.g., config quirk, gotcha)
• Complete an investigation
• Create a new entity and need to attach docs
• Update entity attributes discovered during operation

Knowledge query (single call pattern)

# One call replaces grep + get_entity + get_relations
ctx = mcp.get_entity_knowledge("lxc:jellyfin")
# Returns:
#   - entity attributes (structured)
#   - documents (full-text indexed narrative)
#   - investigations (any incidents involving this entity)
#   - runbooks (procedures that apply to this entity type)
#   - relationships (blast radius, dependencies)

Export cycle

agent discovers → POST /api/v1/knowledge → DB updated
operator reviews → oikos export → seeds/knowledge.yaml updated
git commit + push → version-controlled, diffable
next deploy → oikos seed → DB back in sync

Files to create/modify

File	Action
migrations/010_knowledge_hash.up.sql	New — content_hash TEXT
internal/knowledge/seed.go	New — knowledge seed ingest (read seeds/knowledge.yaml → DB)
internal/knowledge/seed_test.go	New — round-trip tests
internal/knowledge/convert.go	New — one-shot wiki→seed converter
internal/knowledge/parser.go	New — section parser (at-glance, changelog)
internal/db/seed.go	Modify — add knowledge step after policy
internal/db/export.go	Modify — add ExportKnowledge
internal/httpapi/impl.go	Modify — un-stub knowledge endpoints
internal/httpapi/knowledge.go	New — knowledge handlers
internal/mcp/server.go	Modify — un-stub search_knowledge, get_entity_knowledge
cmd/oikos/main.go	Modify — add knowledge convert subcommand
api/openapi.yaml	Modify — knowledge endpoint schemas
seeds/knowledge.yaml	New — generated by converter, ingested on seed
archive/	New — all old wiki + stale files
plans/2026-07-07-comprehensive-audit-and-next-steps.md	Modify — mark knowledge items as addressed

Verification

# 1. Convert wiki to seed
oikos knowledge convert --dry-run
# → "would write 37 documents, 5 investigations, 12 runbooks"
oikos knowledge convert

# 2. Fresh DB, full seed
oikos migrate && oikos seed

# 3. Verify knowledge populated
psql -c "SELECT title, source, cardinality(tags) FROM knowledge_entities;"
# → 54 rows (37 docs + 5 investigations + 12 runbooks)

# 4. Verify FTS
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin+transcode"
# → returns document:containers/101-jellyfin with ts_headline snippet

# 5. Verify entity-linked knowledge
curl "http://localhost:8090/api/v1/knowledge/lxc:jellyfin"
# → {entity, documents: [...], investigations: [], runbooks: [...], relationships: {...}}

# 6. Verify MCP
curl -X POST localhost:8092/query \
  -d '{"tool":"search_knowledge","args":{"query":"caddy truncation"}}'
# → returns investigation:2026-06-06-caddyfile-truncation

# 7. Verify agent registration
curl -X POST localhost:8090/api/v1/knowledge/lxc:jellyfin \
  -H "Content-Type: application/json" \
  -d '{"title":"VAAPI note","content":"Requires /dev/dri/renderD128 passthrough","tags":["gpu"]}'
# → 201 Created

# 8. Verify export round-trip
oikos export   # writes seeds/*.yaml
oikos seed     # re-ingests (skips all — hashes match)
# → "knowledge: ingested 0, skipped 54, errors 0"

# 9. Archive originals (manual, after verification)
# → move all wiki files to archive/ as listed in Phase 2.2

# 10. Verify repo still builds and deploys
make generate && make build && make test
docker compose --profile dev up -d
./scripts/verify-phase6.sh

---

Changelog

2026-07-07 — v2: full archive, wiki→seeds only

Convert wiki to seeds, archive ALL originals, archive all stale files. No human-editing frontend retained — DB is the only source of truth. Knowledge cycle: agent API → oikos export → seeds/knowledge.yaml → git → oikos seed → DB.