oikos/plans/done/2026-07-07-db-as-source-of-truth.md

# 2026-07-07 — DB as single source of truth for agent knowledge

**Status:** Done (2026-07-08) — wiki archived, FTS live, MCP + HTTP knowledge surface complete.

## Goal

Convert `knowledge/wiki/` into DB seeds, archive all original markdown files, and
archive every other file in the repo that the Oikos Go binary does not read. After
this, the DB is the **sole source of truth** for agents — one query surface, no grep
fallback. All old wiki files go to root `archive/`. New knowledge is registered via
the API, and `oikos export` writes it back to seed files for version control.

## Decisions from operator

| Question | Decision |
|----------|----------|
| Plans in DB? | No — agent construction docs only |
| Ingestion trigger | Seed-time (`oikos seed` reads `seeds/knowledge.yaml`) |
| Archive location | Root `archive/` |
| Content granularity | Structured sections: At-a-glance → entity attributes, procedures → runbooks, changelog → parsed entries, rest → `knowledge_entities.content` |
| Wiki files after conversion | **Archive them all** — DB is the only truth |
| Stale files not used by Oikos | **Archive them all** |

## What goes to `archive/`

Everything the Go binary (`cmd/oikos`, `internal/`) does **not** read at runtime.

### Full archive list

| Path | Destination | Why |
|------|-------------|-----|
| `knowledge/wiki/containers/*.md` (20 pages) | `archive/knowledge/containers/` | Converted to seeds |
| `knowledge/wiki/hosts/*.md` (3 pages) | `archive/knowledge/hosts/` | Converted to seeds |
| `knowledge/wiki/infrastructure/*.md` (12 pages) | `archive/knowledge/infrastructure/` | Converted to seeds |
| `knowledge/wiki/vms/*.md` (2 pages) | `archive/knowledge/vms/` | Converted to seeds |
| `knowledge/sources/investigations/*.md` (5 pages) | `archive/knowledge/investigations/` | Converted to seeds |
| `knowledge/sources/references/*.md` (1 page) | `archive/knowledge/references/` | Converted to seeds |
| `knowledge/sources/index.md` | `archive/knowledge/` | Index page, no entity mapping |
| `knowledge/index.md` | `archive/knowledge/` | Index page |
| `knowledge/log.md` | `archive/knowledge/` | Doc-maintenance log, superseded by audit_log |
| `knowledge/GLOSSARY.md` | `archive/knowledge/` | Glossary (can be re-ingested as `document:glossary` later) |
| `oikos/cards/` (45 files) | `archive/oikos-cards/` | Python MCP explain tool; superseded by DB entities |
| `.hermes/plans/` (7 files) | `archive/hermes-plans/` | Agent construction docs, never in DB |
| `ledger/2026-07.jsonl` | `archive/ledger/` | Python-era artifact; Go uses audit_log table |
| `mcp/build_host_files.py` | `archive/mcp/` | Redundant with `cmd/oikos/build_hosts.go` |

### Full delete list (no archival value)

| Path | Why |
|------|-----|
| `oikos/approve.py` | Superseded by `internal/domain/approval.go` |
| `oikos/decide.py` | Superseded by `internal/policy/classify.go` |
| `oikos/drift.py` | Superseded by `internal/scheduler/scheduler.go` |
| `oikos/ledger.py` | Superseded by `internal/db/` + `audit_log` table |
| `oikos/policy.py` | Superseded by `internal/policy/classify.go` |
| `oikos/relations.py` | Superseded by `blast_radius()` SQL function |
| `oikos/scheduler.py` | Superseded by `internal/scheduler/scheduler.go` |
| `oikos/signal.py` | Superseded by `signals` table + `internal/domain/signal.go` |
| `oikos/__init__.py` | Package init, never imported by Go binary |

### What stays (read by Oikos Go binary)

| Path | Read by | Why keep |
|------|---------|----------|
| `seeds/ontology.yaml` | `oikos seed` | DB bootstrap |
| `seeds/inventory.yaml` | `oikos seed` | DB bootstrap |
| `seeds/policy.yaml` | `oikos seed` | DB bootstrap |
| `seeds/knowledge.yaml` | `oikos seed` | **NEW** — knowledge bootstrap |
| `inventory.yaml` | `oikos build-hosts`, `oikos homelab list` | Topology source of truth |
| `secrets/*.yaml` | `oikos secret list`, `oikos homelab secret` | SOPS-encrypted secrets |
| `migrations/*.sql` | `oikos migrate` | Embedded SQL migrations |
| `api/openapi.yaml` | `make generate` (oapi-codegen) | API contract |
| `compose/`, `Dockerfile`, `docker-compose.yml` | Docker build/deploy | Runtime packaging |
| `cmd/`, `internal/` | `go build` | Source code |
| `bin/homelab` | `oikos homelab` subcommand | CLI binary |
| `oikos/gen-topology.py` | `bin/homelab` (Python shim) | Topology generation (until ported) |
| `oikos/gen_topology_lib.py` | imported by gen-topology.py | Same |
| `scripts/deploy.sh` | Gitea webhook → deploy | Production deploy |
| `scripts/rollback.sh` | Operator manual | Rollback |
| `scripts/watchdog.sh` | cron on mac-mini | Health monitoring |
| `scripts/verify-phase6.sh` | Operator manual | Acceptance verification |
| `scripts/sync/` | systemd/launchd timers | Repo sync to clients |
| `scripts/check-caddy-backends.sh` | Operator diagnostic | Health check utility |
| `scripts/cutover-checklist.md` | Operator reference | Cutover tracking |
| `scripts/dns-sync.py` | Operator manual (not yet ported) | DNS sync to Technitium |
| `scripts/validate-seeds.py` | CI / operator manual | Seed validation |
| `tools/` | `post-pull.sh` auto-setup | Client tooling install |
| `ssh/` | `homelab ssh` | SSH key deployment |
| `vps/` | `homelab render-vps-configs` | VPS templates |
| `secrets-issuance/` | Running on apps/105 | Identity issuance (until decommissioned) |
| `AGENTS.md` | Agent onboarding | Entry point |
| `README.md` | Human onboarding | Entry point |
| `.agents/` | Agent conventions, skills, operating model | Agent runtime guidance |
| `plans/` | Agent construction docs | Intent documents |
| `hermes/` | Hermes config + persona | Agent gateway config |
| `Makefile`, `go.mod`, `go.sum`, `sqlc.yaml`, `.gitignore`, `.sops.yaml`, `.gitea/` | Build/config | Build tooling |

### Ambiguous — needs confirmation

| Path | Question |
|------|----------|
| `.agents/` | Contains OIKOS.md (operating model), SKILL.md runbooks, conventions. Go binary doesn't read these at runtime — they're agent orientation docs. Should they also be converted to seeds and archived? Or do they stay as the "agent manual" alongside AGENTS.md? |
| `scripts/dns-sync.py`, `scripts/validate-seeds.py` | Actively used Python scripts with no Go equivalent. Keep until ported? |
| `secrets-issuance/` | Running on apps/105. Archive after decommission? |

## Architecture change

```
BEFORE (split):
  knowledge/wiki/  →  agents grep clone for narrative
  Postgres DB      →  agents query MCP for structured data
  Two surfaces, no bridge

AFTER (DB-only):
  seeds/knowledge.yaml  →  oikos seed  →  knowledge_entities table
                                            ↓
                                     MCP search_knowledge("jellyfin transcode")
                                     MCP get_entity_knowledge("lxc:jellyfin")
                                     HTTP GET /api/v1/knowledge/search
                                            ↓
                                     One surface, everything indexed
```

```
┌──────────────────────────────────┐
│  seeds/knowledge.yaml             │  Version-controlled source material
│  (generated by oikos export)      │  Round-trip: seed → DB → export → seed
│  documents:                       │
│    - slug: containers/101-jellyfin│
│      entity: lxc:jellyfin         │
│      title: "101 — jellyfin"      │
│      content: "..."               │
│      at_glance: {host: strong, ..}│
│      changelog: [{date, title, ..}]│
│      tags: [container, media, ..] │
│  investigations:                  │
│    - slug: 2026-06-06-caddy-...   │
│      about: [service:caddy, ...]  │
│      ...                          │
│  runbooks:                        │
│    - slug: service-health-check   │
│      procedure_for: [service]     │
│      ...                          │
└──────────────┬───────────────────┘
               │ oikos seed
               ▼
┌──────────────────────────────────┐
│  knowledge_entities table         │
│  title | content | content_hash   │
│  source | tags                    │
│  ─ FTS: ts_vector(title+content)  │
│                                   │
│  + entities table:                │
│    document:containers/101-jellyfin│
│    investigation:2026-06-06-...   │
│    runbook:service-health-check   │
│                                   │
│  + relationships:                 │
│    document ──documents──► lxc    │
│    investigation ──about──► service│
│    runbook ──procedure-for──► type│
└──────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────┐
│  MCP + HTTP API                   │
│  search_knowledge("jellyfin")     │
│  get_entity_knowledge("lxc:...")  │
│  POST /api/v1/knowledge/{id}      │
└──────────────────────────────────┘
```

## Execution — Fresh install on mac-mini

The current Docker stack on mac-mini was deployed during cutover testing with dev data.
Replace it with a fresh production instance using the real seeds (ontology, inventory,
policy, knowledge).

### Steps

```bash
# On mac-mini (192.168.8.175):

# 1. Stop current stack
docker compose --profile full down

# 2. Wipe old Postgres volume (fresh start)
docker volume rm oikos_pg-data

# 3. Pull latest repo (already committed to local main)
git pull origin main

# 4. Rebuild image with current code
docker compose build

# 5. Fresh migrate + seed (now includes knowledge)
docker compose run --rm migrate
docker compose run --rm seed

# 6. Verify seed ingested
docker compose run --rm -e OIKOS_DATABASE_URL=... oikos \
  psql -c "SELECT count(*) FROM knowledge_entities;"
# Expect: 54+ rows

# 7. Start production stack
docker compose --profile full up -d

# 8. Health check
curl http://localhost:8090/healthz
./scripts/verify-phase6.sh

# 9. Verify knowledge endpoints
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin"
```

After fresh install succeeds, the old `backups/pre-cutover-20260707.sql` dump becomes
stale (it's from the dev instance). The new instance is the canonical production DB.

---

## Phase 1 — Seed format + ingestion

### 1.1 Add `content_hash` to `knowledge_entities`

**Migration 010:**

```sql
ALTER TABLE knowledge_entities ADD COLUMN content_hash TEXT;
```

### 1.2 Seed format: `seeds/knowledge.yaml`

One YAML file with three sections. Ingested by `oikos seed` after ontology, inventory,
and policy.

```yaml
# Oikos knowledge seed — documents, investigations, and runbooks.
# Generated by oikos export. Ingested on deploy.
# After ingest the DB is authoritative.

version: 1

documents:
  - slug: "containers/101-jellyfin"
    title: "101 — jellyfin"
    content: |
      # 101 — jellyfin

      Jellyfin media server with hardware-accelerated transcoding...

      ## Service / port
      | Service | Port | Notes |
      |---------|------|-------|
      | jellyfin | 8096 | Web UI + API |
      ...

      ## SSO
      Authentik OIDC via SSO-Auth plugin v4.0.0.4. No Caddy forward-auth gate.
      ...
    entity_slug: "lxc:jellyfin"         # links via `documents` edge
    tags: ["container", "media", "jellyfin"]
    at_glance:                          # structured: pushed to entity attributes if empty
      host: strong
      ip: "192.168.8.246"
      cores: 4
      ram: "8 GiB"
      mounts: ["/mnt/media_local"]
      public_host: media.hubris.network
    changelog:
      - date: "2026-07-05"
        title: "migrated from hubris to strong"
        body: "Phase 2 of strong migration..."
      - date: "2026-06-10"
        title: "VAAPI configured on Radeon 680M"
        body: "GPU passthrough via dev0 + dev1..."

  - slug: "hosts/strong"
    title: "strong — Proxmox host"
    content: |
      # strong — Proxmox host
      Reformatted from Linux workstation to Proxmox VE 9.2.3...
    entity_slug: "proxmox-host:strong"
    tags: ["host", "proxmox", "hypervisor"]
    at_glance:
      role: hypervisor
      hardware: "Minisforum UM773 Lite (Ryzen 7 PRO 6850U, 28 GiB RAM)"
      os: linux
    changelog:
      - date: "2026-07-05"
        title: "strong migration Phase 2 complete"
        body: "arriman + jellyfin + grimmory migrated. ludo-lvm..."

  # ... 35+ more document entries

investigations:
  - slug: "2026-06-06-caddyfile-truncation"
    title: "Caddyfile truncation — all LAN services down"
    date: "2026-06-06"
    status: resolved
    duration: "~45 min"
    content: |
      ## Symptom
      All *.hubris.network services returned 502...
      ## Root cause
      ...
    about_slugs: ["service:caddy", "service:dns"]  # links via `about` edges
    tags: ["incident", "caddy", "dns", "downtime"]

  - slug: "2026-06-06-authentik-session-lifetime"
    title: "Frequent Authentik login prompts"
    date: "2026-06-06"
    status: resolved
    duration: "~30 min"
    content: |
      ## Summary
      ...
    about_slugs: ["service:authentik"]
    tags: ["incident", "authentik", "oidc"]

  # ... 3 more investigation entries

runbooks:
  - slug: "service-health-check"
    name: "Service health check"
    risk_class: read_only
    entity_type: "service"              # applies to all service entities
    procedure:
      params_schema:
        type: object
        properties:
          unit: {type: string}
        required: ["unit"]
      steps:
        - name: "check systemd unit"
          runner: ssh
          command: "systemctl is-active {{ .unit }}"
          timeout_s: 10
      verify:
        - runner: ssh
          command: "systemctl is-active {{ .unit }}"
          expect: {exit_code: 0, stdout_contains: "active"}

  - slug: "lifecycle-provision-node"
    name: "Provision new node"
    risk_class: config_mutation
    entity_type: "lxc"
    procedure:
      params_schema:
        type: object
        properties:
          name: {type: string}
          template: {type: string}
          cores: {type: integer}
          memory_mb: {type: integer}
        required: ["name", "template"]
      steps:
        - name: "create container"
          runner: ssh
          target: "{{ .host }}"
          command: "pct create $(pvesh get /nextid) {{ .template }} ..."
          timeout_s: 120
      verify: [...]

  # ... 10 more runbook entries
```

### 1.3 Ingest logic

`oikos seed` already runs: migrate → ontology → inventory → policy.

Add knowledge as step 5:

```
oikos seed
  1. migrate (DDL)
  2. ontology seed (entity_types, relationship_types, lifecycles)
  3. inventory seed (entities, relationships)
  4. policy seed (risk_classes, approval_rules, autonomy)
  5. knowledge seed (documents, investigations, runbooks)  ← NEW
```

Knowledge ingest for each section:

**Documents:**
- For each entry, create `document:<slug>` entity in `entities` table (type: `document`)
- Insert into `knowledge_entities` (title, content, source=slug, tags, content_hash)
- Create `documents` edge from `document:<slug>` to `entity_slug`
- Push `at_glance` fields into the linked entity's `attributes` if those keys are empty (backfill)
- Store `changelog` as JSONB in document entity `attributes.changelog`
- Skip if `content_hash` matches existing row (idempotent)

**Investigations:**
- Create `investigation:<slug>` entity (type: `investigation`, lifecycle: `infrastructure`)
- Insert into `knowledge_entities`
- Create `about` edges to each entity in `about_slugs`
- Store `date`, `status`, `duration` in entity `attributes`

**Runbooks:**
- Create `runbook:<slug>` entity (type: `runbook`, lifecycle: `skill`)
- Insert into `knowledge_entities`
- Create `procedure-for` edges to `entity_type` (abstract type — applies to all instances)
- If a `skills` row with matching slug already exists, only update `knowledge_entities.content` and `content_hash` (don't overwrite the structured `procedure` JSONB in the skills table — that's managed separately)

### 1.4 Export: DB → seed

`oikos export` already writes `seeds/ontology.yaml`, `seeds/inventory.yaml`,
`seeds/policy.yaml`. Add `seeds/knowledge.yaml`:

```go
// internal/db/export.go
func ExportKnowledge(ctx context.Context, pool *pgxpool.Pool) ([]byte, error) {
    // SELECT all document + investigation + runbook entities
    // JOIN knowledge_entities for content
    // JOIN relationships for entity links
    // Serialize to seeds/knowledge.yaml format
}
```

Round-trip guarantee: `seed → DB → export → seed` is byte-stable. Tested in CI.

## Phase 2 — Convert wiki to seeds, archive originals

### 2.1 Conversion script (one-time)

```
oikos knowledge convert [--dry-run]
```

- Reads every page in `knowledge/wiki/`, `knowledge/sources/`, `knowledge/GLOSSARY.md`
- Parses structured sections (at-glance, changelog, procedures)
- Maps each page to entity slugs using the path convention + `inventory.yaml` doc_page
- Generates `seeds/knowledge.yaml`
- Reports: "would write N documents, M investigations, K runbooks"
- Without `--dry-run`: writes `seeds/knowledge.yaml`

Implementation: initially a Go version in `cmd/oikos/`, can fall back to a one-shot
Python script if that's faster to build.

### 2.2 Archive

After conversion verified (seeds ingest cleanly):

```bash
# Create archive directory
mkdir -p archive/knowledge/{containers,hosts,infrastructure,vms,investigations,references}

# Move all knowledge files
mv knowledge/wiki/containers/*.md archive/knowledge/containers/
mv knowledge/wiki/hosts/*.md archive/knowledge/hosts/
mv knowledge/wiki/infrastructure/*.md archive/knowledge/infrastructure/
mv knowledge/wiki/vms/*.md archive/knowledge/vms/
mv knowledge/sources/investigations/*.md archive/knowledge/investigations/
mv knowledge/sources/references/*.md archive/knowledge/references/
mv knowledge/index.md knowledge/log.md knowledge/GLOSSARY.md archive/knowledge/

# Move other stale artifacts
mv oikos/cards/ archive/oikos-cards/
mv .hermes/plans/*.md archive/hermes-plans/
mv ledger/2026-07.jsonl archive/ledger/
mv mcp/build_host_files.py archive/mcp/

# Delete superseded Python
rm oikos/__init__.py oikos/approve.py oikos/decide.py oikos/drift.py \
   oikos/ledger.py oikos/policy.py oikos/relations.py oikos/scheduler.py \
   oikos/signal.py

# Clean empty directories
rmdir knowledge/sources/investigations/archive/  # already moved
rmdir knowledge/sources/investigations/
rmdir knowledge/sources/references/
rmdir knowledge/sources/
rmdir knowledge/wiki/containers/archive/          # already moved
rmdir knowledge/wiki/containers/
rmdir knowledge/wiki/hosts/
rmdir knowledge/wiki/infrastructure/
rmdir knowledge/wiki/vms/
rmdir knowledge/wiki/
```

### 2.3 What the repo looks like after

```
/
├── seeds/
│   ├── ontology.yaml
│   ├── inventory.yaml
│   ├── policy.yaml
│   └── knowledge.yaml        ← NEW: documents + investigations + runbooks
├── archive/
│   ├── knowledge/            ← all old wiki markdown
│   ├── oikos-cards/          ← 45 Python explain cards
│   ├── hermes-plans/         ← 7 agent construction docs
│   ├── ledger/               ← Python-era JSONL
│   └── mcp/                  ← old Python build_host_files.py
├── .agents/                  ← agent conventions + skills (stays)
├── plans/                    ← construction plans (stays)
├── cmd/, internal/           ← Go source
├── migrations/, api/, compose/
├── inventory.yaml, hosts/*.yaml
├── AGENTS.md, README.md
└── ...
```

The repo shrinks by ~65 narrative files and ~45 card files. All knowledge is in
`seeds/knowledge.yaml` — one file, git-tracked, round-trippable.

## Phase 3 — Un-stub MCP + HTTP

Same as previous revision:

- `search_knowledge(query)` → PostgreSQL FTS with `ts_rank`, `ts_headline`
- `get_entity_knowledge(slug)` → aggregates documents, investigations, runbooks linked to entity
- `GET /api/v1/knowledge/search` → FTS endpoint
- `GET /api/v1/knowledge/{slug}` → entity-linked knowledge
- `POST /api/v1/knowledge/{slug}` → agent registers new knowledge
- `PATCH /api/v1/knowledge/{slug}` → agent updates knowledge

## Phase 4 — Agent conventions

### Knowledge registration

Agents call `POST /api/v1/knowledge/{entity_slug}` when they:
- Discover an undocumented detail (e.g., config quirk, gotcha)
- Complete an investigation
- Create a new entity and need to attach docs
- Update entity attributes discovered during operation

### Knowledge query (single call pattern)

```python
# One call replaces grep + get_entity + get_relations
ctx = mcp.get_entity_knowledge("lxc:jellyfin")
# Returns:
#   - entity attributes (structured)
#   - documents (full-text indexed narrative)
#   - investigations (any incidents involving this entity)
#   - runbooks (procedures that apply to this entity type)
#   - relationships (blast radius, dependencies)
```

### Export cycle

```
agent discovers → POST /api/v1/knowledge → DB updated
operator reviews → oikos export → seeds/knowledge.yaml updated
git commit + push → version-controlled, diffable
next deploy → oikos seed → DB back in sync
```

## Files to create/modify

| File | Action |
|------|--------|
| `migrations/010_knowledge_hash.up.sql` | New — `content_hash TEXT` |
| `internal/knowledge/seed.go` | New — knowledge seed ingest (read seeds/knowledge.yaml → DB) |
| `internal/knowledge/seed_test.go` | New — round-trip tests |
| `internal/knowledge/convert.go` | New — one-shot wiki→seed converter |
| `internal/knowledge/parser.go` | New — section parser (at-glance, changelog) |
| `internal/db/seed.go` | Modify — add knowledge step after policy |
| `internal/db/export.go` | Modify — add `ExportKnowledge` |
| `internal/httpapi/impl.go` | Modify — un-stub knowledge endpoints |
| `internal/httpapi/knowledge.go` | New — knowledge handlers |
| `internal/mcp/server.go` | Modify — un-stub `search_knowledge`, `get_entity_knowledge` |
| `cmd/oikos/main.go` | Modify — add `knowledge convert` subcommand |
| `api/openapi.yaml` | Modify — knowledge endpoint schemas |
| `seeds/knowledge.yaml` | New — generated by converter, ingested on seed |
| `archive/` | New — all old wiki + stale files |
| `plans/2026-07-07-comprehensive-audit-and-next-steps.md` | Modify — mark knowledge items as addressed |

## Verification

```bash
# 1. Convert wiki to seed
oikos knowledge convert --dry-run
# → "would write 37 documents, 5 investigations, 12 runbooks"
oikos knowledge convert

# 2. Fresh DB, full seed
oikos migrate && oikos seed

# 3. Verify knowledge populated
psql -c "SELECT title, source, cardinality(tags) FROM knowledge_entities;"
# → 54 rows (37 docs + 5 investigations + 12 runbooks)

# 4. Verify FTS
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin+transcode"
# → returns document:containers/101-jellyfin with ts_headline snippet

# 5. Verify entity-linked knowledge
curl "http://localhost:8090/api/v1/knowledge/lxc:jellyfin"
# → {entity, documents: [...], investigations: [], runbooks: [...], relationships: {...}}

# 6. Verify MCP
curl -X POST localhost:8092/query \
  -d '{"tool":"search_knowledge","args":{"query":"caddy truncation"}}'
# → returns investigation:2026-06-06-caddyfile-truncation

# 7. Verify agent registration
curl -X POST localhost:8090/api/v1/knowledge/lxc:jellyfin \
  -H "Content-Type: application/json" \
  -d '{"title":"VAAPI note","content":"Requires /dev/dri/renderD128 passthrough","tags":["gpu"]}'
# → 201 Created

# 8. Verify export round-trip
oikos export   # writes seeds/*.yaml
oikos seed     # re-ingests (skips all — hashes match)
# → "knowledge: ingested 0, skipped 54, errors 0"

# 9. Archive originals (manual, after verification)
# → move all wiki files to archive/ as listed in Phase 2.2

# 10. Verify repo still builds and deploys
make generate && make build && make test
docker compose --profile dev up -d
./scripts/verify-phase6.sh
```

---

## Changelog

### 2026-07-07 — v2: full archive, wiki→seeds only
Convert wiki to seeds, archive ALL originals, archive all stale files. No human-editing
frontend retained — DB is the only source of truth. Knowledge cycle: agent API →
oikos export → seeds/knowledge.yaml → git → oikos seed → DB.