dtoro/oikos

Fork 0

Files

dtoro b2bfa26f64 plans: comprehensive audit + DB-as-truth architecture

2026-07-07 20:09:32 +02:00

24 KiB

Raw Blame History

2026-07-07 — DB as single source of truth for agent knowledge

Status: Proposed

Goal

Convert knowledge/wiki/ into DB seeds, archive all original markdown files, and archive every other file in the repo that the Oikos Go binary does not read. After this, the DB is the sole source of truth for agents — one query surface, no grep fallback. All old wiki files go to root archive/. New knowledge is registered via the API, and oikos export writes it back to seed files for version control.

Decisions from operator

Question	Decision
Plans in DB?	No — agent construction docs only
Ingestion trigger	Seed-time (`oikos seed` reads `seeds/knowledge.yaml`)
Archive location	Root `archive/`
Content granularity	Structured sections: At-a-glance → entity attributes, procedures → runbooks, changelog → parsed entries, rest → `knowledge_entities.content`
Wiki files after conversion	Archive them all — DB is the only truth
Stale files not used by Oikos	Archive them all

What goes to `archive/`

Everything the Go binary (cmd/oikos, internal/) does not read at runtime.

Full archive list

Path	Destination	Why
`knowledge/wiki/containers/*.md` (20 pages)	`archive/knowledge/containers/`	Converted to seeds
`knowledge/wiki/hosts/*.md` (3 pages)	`archive/knowledge/hosts/`	Converted to seeds
`knowledge/wiki/infrastructure/*.md` (12 pages)	`archive/knowledge/infrastructure/`	Converted to seeds
`knowledge/wiki/vms/*.md` (2 pages)	`archive/knowledge/vms/`	Converted to seeds
`knowledge/sources/investigations/*.md` (5 pages)	`archive/knowledge/investigations/`	Converted to seeds
`knowledge/sources/references/*.md` (1 page)	`archive/knowledge/references/`	Converted to seeds
`knowledge/sources/index.md`	`archive/knowledge/`	Index page, no entity mapping
`knowledge/index.md`	`archive/knowledge/`	Index page
`knowledge/log.md`	`archive/knowledge/`	Doc-maintenance log, superseded by audit_log
`knowledge/GLOSSARY.md`	`archive/knowledge/`	Glossary (can be re-ingested as `document:glossary` later)
`oikos/cards/` (45 files)	`archive/oikos-cards/`	Python MCP explain tool; superseded by DB entities
`.hermes/plans/` (7 files)	`archive/hermes-plans/`	Agent construction docs, never in DB
`ledger/2026-07.jsonl`	`archive/ledger/`	Python-era artifact; Go uses audit_log table
`mcp/build_host_files.py`	`archive/mcp/`	Redundant with `cmd/oikos/build_hosts.go`

Full delete list (no archival value)

Path	Why
`oikos/approve.py`	Superseded by `internal/domain/approval.go`
`oikos/decide.py`	Superseded by `internal/policy/classify.go`
`oikos/drift.py`	Superseded by `internal/scheduler/scheduler.go`
`oikos/ledger.py`	Superseded by `internal/db/` + `audit_log` table
`oikos/policy.py`	Superseded by `internal/policy/classify.go`
`oikos/relations.py`	Superseded by `blast_radius()` SQL function
`oikos/scheduler.py`	Superseded by `internal/scheduler/scheduler.go`
`oikos/signal.py`	Superseded by `signals` table + `internal/domain/signal.go`
`oikos/__init__.py`	Package init, never imported by Go binary

What stays (read by Oikos Go binary)

Path	Read by	Why keep
`seeds/ontology.yaml`	`oikos seed`	DB bootstrap
`seeds/inventory.yaml`	`oikos seed`	DB bootstrap
`seeds/policy.yaml`	`oikos seed`	DB bootstrap
`seeds/knowledge.yaml`	`oikos seed`	NEW — knowledge bootstrap
`inventory.yaml`	`oikos build-hosts`, `oikos homelab list`	Topology source of truth
`secrets/*.yaml`	`oikos secret list`, `oikos homelab secret`	SOPS-encrypted secrets
`migrations/*.sql`	`oikos migrate`	Embedded SQL migrations
`api/openapi.yaml`	`make generate` (oapi-codegen)	API contract
`compose/`, `Dockerfile`, `docker-compose.yml`	Docker build/deploy	Runtime packaging
`cmd/`, `internal/`	`go build`	Source code
`bin/homelab`	`oikos homelab` subcommand	CLI binary
`oikos/gen-topology.py`	`bin/homelab` (Python shim)	Topology generation (until ported)
`oikos/gen_topology_lib.py`	imported by gen-topology.py	Same
`scripts/deploy.sh`	Gitea webhook → deploy	Production deploy
`scripts/rollback.sh`	Operator manual	Rollback
`scripts/watchdog.sh`	cron on mac-mini	Health monitoring
`scripts/verify-phase6.sh`	Operator manual	Acceptance verification
`scripts/sync/`	systemd/launchd timers	Repo sync to clients
`scripts/check-caddy-backends.sh`	Operator diagnostic	Health check utility
`scripts/cutover-checklist.md`	Operator reference	Cutover tracking
`scripts/dns-sync.py`	Operator manual (not yet ported)	DNS sync to Technitium
`scripts/validate-seeds.py`	CI / operator manual	Seed validation
`tools/`	`post-pull.sh` auto-setup	Client tooling install
`ssh/`	`homelab ssh`	SSH key deployment
`vps/`	`homelab render-vps-configs`	VPS templates
`secrets-issuance/`	Running on apps/105	Identity issuance (until decommissioned)
`AGENTS.md`	Agent onboarding	Entry point
`README.md`	Human onboarding	Entry point
`.agents/`	Agent conventions, skills, operating model	Agent runtime guidance
`plans/`	Agent construction docs	Intent documents
`hermes/`	Hermes config + persona	Agent gateway config
`Makefile`, `go.mod`, `go.sum`, `sqlc.yaml`, `.gitignore`, `.sops.yaml`, `.gitea/`	Build/config	Build tooling

Ambiguous — needs confirmation

Path	Question
`.agents/`	Contains OIKOS.md (operating model), SKILL.md runbooks, conventions. Go binary doesn't read these at runtime — they're agent orientation docs. Should they also be converted to seeds and archived? Or do they stay as the "agent manual" alongside AGENTS.md?
`scripts/dns-sync.py`, `scripts/validate-seeds.py`	Actively used Python scripts with no Go equivalent. Keep until ported?
`secrets-issuance/`	Running on apps/105. Archive after decommission?

Architecture change

BEFORE (split):
  knowledge/wiki/  →  agents grep clone for narrative
  Postgres DB      →  agents query MCP for structured data
  Two surfaces, no bridge

AFTER (DB-only):
  seeds/knowledge.yaml  →  oikos seed  →  knowledge_entities table
                                            ↓
                                     MCP search_knowledge("jellyfin transcode")
                                     MCP get_entity_knowledge("lxc:jellyfin")
                                     HTTP GET /api/v1/knowledge/search
                                            ↓
                                     One surface, everything indexed

┌──────────────────────────────────┐
│  seeds/knowledge.yaml             │  Version-controlled source material
│  (generated by oikos export)      │  Round-trip: seed → DB → export → seed
│  documents:                       │
│    - slug: containers/101-jellyfin│
│      entity: lxc:jellyfin         │
│      title: "101 — jellyfin"      │
│      content: "..."               │
│      at_glance: {host: strong, ..}│
│      changelog: [{date, title, ..}]│
│      tags: [container, media, ..] │
│  investigations:                  │
│    - slug: 2026-06-06-caddy-...   │
│      about: [service:caddy, ...]  │
│      ...                          │
│  runbooks:                        │
│    - slug: service-health-check   │
│      procedure_for: [service]     │
│      ...                          │
└──────────────┬───────────────────┘
               │ oikos seed
               ▼
┌──────────────────────────────────┐
│  knowledge_entities table         │
│  title | content | content_hash   │
│  source | tags                    │
│  ─ FTS: ts_vector(title+content)  │
│                                   │
│  + entities table:                │
│    document:containers/101-jellyfin│
│    investigation:2026-06-06-...   │
│    runbook:service-health-check   │
│                                   │
│  + relationships:                 │
│    document ──documents──► lxc    │
│    investigation ──about──► service│
│    runbook ──procedure-for──► type│
└──────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────┐
│  MCP + HTTP API                   │
│  search_knowledge("jellyfin")     │
│  get_entity_knowledge("lxc:...")  │
│  POST /api/v1/knowledge/{id}      │
└──────────────────────────────────┘

Execution — Fresh install on mac-mini

The current Docker stack on mac-mini was deployed during cutover testing with dev data. Replace it with a fresh production instance using the real seeds (ontology, inventory, policy, knowledge).

Steps

# On mac-mini (192.168.8.175):

# 1. Stop current stack
docker compose --profile full down

# 2. Wipe old Postgres volume (fresh start)
docker volume rm oikos_pg-data

# 3. Pull latest repo (already committed to local main)
git pull origin main

# 4. Rebuild image with current code
docker compose build

# 5. Fresh migrate + seed (now includes knowledge)
docker compose run --rm migrate
docker compose run --rm seed

# 6. Verify seed ingested
docker compose run --rm -e OIKOS_DATABASE_URL=... oikos \
  psql -c "SELECT count(*) FROM knowledge_entities;"
# Expect: 54+ rows

# 7. Start production stack
docker compose --profile full up -d

# 8. Health check
curl http://localhost:8090/healthz
./scripts/verify-phase6.sh

# 9. Verify knowledge endpoints
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin"

After fresh install succeeds, the old backups/pre-cutover-20260707.sql dump becomes stale (it's from the dev instance). The new instance is the canonical production DB.

Phase 1 — Seed format + ingestion

1.1 Add `content_hash` to `knowledge_entities`

Migration 010:

ALTER TABLE knowledge_entities ADD COLUMN content_hash TEXT;

1.2 Seed format: `seeds/knowledge.yaml`

One YAML file with three sections. Ingested by oikos seed after ontology, inventory, and policy.

# Oikos knowledge seed — documents, investigations, and runbooks.
# Generated by oikos export. Ingested on deploy.
# After ingest the DB is authoritative.

version: 1

documents:
  - slug: "containers/101-jellyfin"
    title: "101 — jellyfin"
    content: |
      # 101 — jellyfin

      Jellyfin media server with hardware-accelerated transcoding...

      ## Service / port
      | Service | Port | Notes |
      |---------|------|-------|
      | jellyfin | 8096 | Web UI + API |
      ...

      ## SSO
      Authentik OIDC via SSO-Auth plugin v4.0.0.4. No Caddy forward-auth gate.
      ...
    entity_slug: "lxc:jellyfin"         # links via `documents` edge
    tags: ["container", "media", "jellyfin"]
    at_glance:                          # structured: pushed to entity attributes if empty
      host: strong
      ip: "192.168.8.246"
      cores: 4
      ram: "8 GiB"
      mounts: ["/mnt/media_local"]
      public_host: media.hubris.network
    changelog:
      - date: "2026-07-05"
        title: "migrated from hubris to strong"
        body: "Phase 2 of strong migration..."
      - date: "2026-06-10"
        title: "VAAPI configured on Radeon 680M"
        body: "GPU passthrough via dev0 + dev1..."

  - slug: "hosts/strong"
    title: "strong — Proxmox host"
    content: |
      # strong — Proxmox host
      Reformatted from Linux workstation to Proxmox VE 9.2.3...
    entity_slug: "proxmox-host:strong"
    tags: ["host", "proxmox", "hypervisor"]
    at_glance:
      role: hypervisor
      hardware: "Minisforum UM773 Lite (Ryzen 7 PRO 6850U, 28 GiB RAM)"
      os: linux
    changelog:
      - date: "2026-07-05"
        title: "strong migration Phase 2 complete"
        body: "arriman + jellyfin + grimmory migrated. ludo-lvm..."

  # ... 35+ more document entries

investigations:
  - slug: "2026-06-06-caddyfile-truncation"
    title: "Caddyfile truncation — all LAN services down"
    date: "2026-06-06"
    status: resolved
    duration: "~45 min"
    content: |
      ## Symptom
      All *.hubris.network services returned 502...
      ## Root cause
      ...
    about_slugs: ["service:caddy", "service:dns"]  # links via `about` edges
    tags: ["incident", "caddy", "dns", "downtime"]

  - slug: "2026-06-06-authentik-session-lifetime"
    title: "Frequent Authentik login prompts"
    date: "2026-06-06"
    status: resolved
    duration: "~30 min"
    content: |
      ## Summary
      ...
    about_slugs: ["service:authentik"]
    tags: ["incident", "authentik", "oidc"]

  # ... 3 more investigation entries

runbooks:
  - slug: "service-health-check"
    name: "Service health check"
    risk_class: read_only
    entity_type: "service"              # applies to all service entities
    procedure:
      params_schema:
        type: object
        properties:
          unit: {type: string}
        required: ["unit"]
      steps:
        - name: "check systemd unit"
          runner: ssh
          command: "systemctl is-active {{ .unit }}"
          timeout_s: 10
      verify:
        - runner: ssh
          command: "systemctl is-active {{ .unit }}"
          expect: {exit_code: 0, stdout_contains: "active"}

  - slug: "lifecycle-provision-node"
    name: "Provision new node"
    risk_class: config_mutation
    entity_type: "lxc"
    procedure:
      params_schema:
        type: object
        properties:
          name: {type: string}
          template: {type: string}
          cores: {type: integer}
          memory_mb: {type: integer}
        required: ["name", "template"]
      steps:
        - name: "create container"
          runner: ssh
          target: "{{ .host }}"
          command: "pct create $(pvesh get /nextid) {{ .template }} ..."
          timeout_s: 120
      verify: [...]

  # ... 10 more runbook entries

1.3 Ingest logic

oikos seed already runs: migrate → ontology → inventory → policy.

Add knowledge as step 5:

oikos seed
  1. migrate (DDL)
  2. ontology seed (entity_types, relationship_types, lifecycles)
  3. inventory seed (entities, relationships)
  4. policy seed (risk_classes, approval_rules, autonomy)
  5. knowledge seed (documents, investigations, runbooks)  ← NEW

Knowledge ingest for each section:

Documents:

For each entry, create document:<slug> entity in entities table (type: document)
Insert into knowledge_entities (title, content, source=slug, tags, content_hash)
Create documents edge from document:<slug> to entity_slug
Push at_glance fields into the linked entity's attributes if those keys are empty (backfill)
Store changelog as JSONB in document entity attributes.changelog
Skip if content_hash matches existing row (idempotent)

Investigations:

Create investigation:<slug> entity (type: investigation, lifecycle: infrastructure)
Insert into knowledge_entities
Create about edges to each entity in about_slugs
Store date, status, duration in entity attributes

Runbooks:

Create runbook:<slug> entity (type: runbook, lifecycle: skill)
Insert into knowledge_entities
Create procedure-for edges to entity_type (abstract type — applies to all instances)
If a skills row with matching slug already exists, only update knowledge_entities.content and content_hash (don't overwrite the structured procedure JSONB in the skills table — that's managed separately)

1.4 Export: DB → seed

oikos export already writes seeds/ontology.yaml, seeds/inventory.yaml, seeds/policy.yaml. Add seeds/knowledge.yaml:

// internal/db/export.go
func ExportKnowledge(ctx context.Context, pool *pgxpool.Pool) ([]byte, error) {
    // SELECT all document + investigation + runbook entities
    // JOIN knowledge_entities for content
    // JOIN relationships for entity links
    // Serialize to seeds/knowledge.yaml format
}

Round-trip guarantee: seed → DB → export → seed is byte-stable. Tested in CI.

Phase 2 — Convert wiki to seeds, archive originals

2.1 Conversion script (one-time)

oikos knowledge convert [--dry-run]

Reads every page in knowledge/wiki/, knowledge/sources/, knowledge/GLOSSARY.md
Parses structured sections (at-glance, changelog, procedures)
Maps each page to entity slugs using the path convention + inventory.yaml doc_page
Generates seeds/knowledge.yaml
Reports: "would write N documents, M investigations, K runbooks"
Without --dry-run: writes seeds/knowledge.yaml

Implementation: initially a Go version in cmd/oikos/, can fall back to a one-shot Python script if that's faster to build.

2.2 Archive

After conversion verified (seeds ingest cleanly):

# Create archive directory
mkdir -p archive/knowledge/{containers,hosts,infrastructure,vms,investigations,references}

# Move all knowledge files
mv knowledge/wiki/containers/*.md archive/knowledge/containers/
mv knowledge/wiki/hosts/*.md archive/knowledge/hosts/
mv knowledge/wiki/infrastructure/*.md archive/knowledge/infrastructure/
mv knowledge/wiki/vms/*.md archive/knowledge/vms/
mv knowledge/sources/investigations/*.md archive/knowledge/investigations/
mv knowledge/sources/references/*.md archive/knowledge/references/
mv knowledge/index.md knowledge/log.md knowledge/GLOSSARY.md archive/knowledge/

# Move other stale artifacts
mv oikos/cards/ archive/oikos-cards/
mv .hermes/plans/*.md archive/hermes-plans/
mv ledger/2026-07.jsonl archive/ledger/
mv mcp/build_host_files.py archive/mcp/

# Delete superseded Python
rm oikos/__init__.py oikos/approve.py oikos/decide.py oikos/drift.py \
   oikos/ledger.py oikos/policy.py oikos/relations.py oikos/scheduler.py \
   oikos/signal.py

# Clean empty directories
rmdir knowledge/sources/investigations/archive/  # already moved
rmdir knowledge/sources/investigations/
rmdir knowledge/sources/references/
rmdir knowledge/sources/
rmdir knowledge/wiki/containers/archive/          # already moved
rmdir knowledge/wiki/containers/
rmdir knowledge/wiki/hosts/
rmdir knowledge/wiki/infrastructure/
rmdir knowledge/wiki/vms/
rmdir knowledge/wiki/

2.3 What the repo looks like after

/
├── seeds/
│   ├── ontology.yaml
│   ├── inventory.yaml
│   ├── policy.yaml
│   └── knowledge.yaml        ← NEW: documents + investigations + runbooks
├── archive/
│   ├── knowledge/            ← all old wiki markdown
│   ├── oikos-cards/          ← 45 Python explain cards
│   ├── hermes-plans/         ← 7 agent construction docs
│   ├── ledger/               ← Python-era JSONL
│   └── mcp/                  ← old Python build_host_files.py
├── .agents/                  ← agent conventions + skills (stays)
├── plans/                    ← construction plans (stays)
├── cmd/, internal/           ← Go source
├── migrations/, api/, compose/
├── inventory.yaml, hosts/*.yaml
├── AGENTS.md, README.md
└── ...

The repo shrinks by ~65 narrative files and ~45 card files. All knowledge is in seeds/knowledge.yaml — one file, git-tracked, round-trippable.

Phase 3 — Un-stub MCP + HTTP

Same as previous revision:

search_knowledge(query) → PostgreSQL FTS with ts_rank, ts_headline
get_entity_knowledge(slug) → aggregates documents, investigations, runbooks linked to entity
GET /api/v1/knowledge/search → FTS endpoint
GET /api/v1/knowledge/{slug} → entity-linked knowledge
POST /api/v1/knowledge/{slug} → agent registers new knowledge
PATCH /api/v1/knowledge/{slug} → agent updates knowledge

Phase 4 — Agent conventions

Knowledge registration

Agents call POST /api/v1/knowledge/{entity_slug} when they:

Discover an undocumented detail (e.g., config quirk, gotcha)
Complete an investigation
Create a new entity and need to attach docs
Update entity attributes discovered during operation

Knowledge query (single call pattern)

# One call replaces grep + get_entity + get_relations
ctx = mcp.get_entity_knowledge("lxc:jellyfin")
# Returns:
#   - entity attributes (structured)
#   - documents (full-text indexed narrative)
#   - investigations (any incidents involving this entity)
#   - runbooks (procedures that apply to this entity type)
#   - relationships (blast radius, dependencies)

Export cycle

agent discovers → POST /api/v1/knowledge → DB updated
operator reviews → oikos export → seeds/knowledge.yaml updated
git commit + push → version-controlled, diffable
next deploy → oikos seed → DB back in sync

Files to create/modify

File	Action
`migrations/010_knowledge_hash.up.sql`	New — `content_hash TEXT`
`internal/knowledge/seed.go`	New — knowledge seed ingest (read seeds/knowledge.yaml → DB)
`internal/knowledge/seed_test.go`	New — round-trip tests
`internal/knowledge/convert.go`	New — one-shot wiki→seed converter
`internal/knowledge/parser.go`	New — section parser (at-glance, changelog)
`internal/db/seed.go`	Modify — add knowledge step after policy
`internal/db/export.go`	Modify — add `ExportKnowledge`
`internal/httpapi/impl.go`	Modify — un-stub knowledge endpoints
`internal/httpapi/knowledge.go`	New — knowledge handlers
`internal/mcp/server.go`	Modify — un-stub `search_knowledge`, `get_entity_knowledge`
`cmd/oikos/main.go`	Modify — add `knowledge convert` subcommand
`api/openapi.yaml`	Modify — knowledge endpoint schemas
`seeds/knowledge.yaml`	New — generated by converter, ingested on seed
`archive/`	New — all old wiki + stale files
`plans/2026-07-07-comprehensive-audit-and-next-steps.md`	Modify — mark knowledge items as addressed

Verification

# 1. Convert wiki to seed
oikos knowledge convert --dry-run
# → "would write 37 documents, 5 investigations, 12 runbooks"
oikos knowledge convert

# 2. Fresh DB, full seed
oikos migrate && oikos seed

# 3. Verify knowledge populated
psql -c "SELECT title, source, cardinality(tags) FROM knowledge_entities;"
# → 54 rows (37 docs + 5 investigations + 12 runbooks)

# 4. Verify FTS
curl "http://localhost:8090/api/v1/knowledge/search?q=jellyfin+transcode"
# → returns document:containers/101-jellyfin with ts_headline snippet

# 5. Verify entity-linked knowledge
curl "http://localhost:8090/api/v1/knowledge/lxc:jellyfin"
# → {entity, documents: [...], investigations: [], runbooks: [...], relationships: {...}}

# 6. Verify MCP
curl -X POST localhost:8092/query \
  -d '{"tool":"search_knowledge","args":{"query":"caddy truncation"}}'
# → returns investigation:2026-06-06-caddyfile-truncation

# 7. Verify agent registration
curl -X POST localhost:8090/api/v1/knowledge/lxc:jellyfin \
  -H "Content-Type: application/json" \
  -d '{"title":"VAAPI note","content":"Requires /dev/dri/renderD128 passthrough","tags":["gpu"]}'
# → 201 Created

# 8. Verify export round-trip
oikos export   # writes seeds/*.yaml
oikos seed     # re-ingests (skips all — hashes match)
# → "knowledge: ingested 0, skipped 54, errors 0"

# 9. Archive originals (manual, after verification)
# → move all wiki files to archive/ as listed in Phase 2.2

# 10. Verify repo still builds and deploys
make generate && make build && make test
docker compose --profile dev up -d
./scripts/verify-phase6.sh

Changelog

2026-07-07 — v2: full archive, wiki→seeds only

Convert wiki to seeds, archive ALL originals, archive all stale files. No human-editing frontend retained — DB is the only source of truth. Knowledge cycle: agent API → oikos export → seeds/knowledge.yaml → git → oikos seed → DB.

24 KiB Raw Blame History