Sophia/README.md

Sophia

A knowledge engine where each idea is an autonomous agent — an Engram — that moves, resonates with similar ideas, forms synapses with them, and answers questions by gathering its own perspective into a single integrated reply.

The whole system is a 3D simulation you can watch: ingested text becomes glowing dots that fly out from the centre of a torus, drift toward similar peers, and light up when a query's broadcast wave touches them.

ingest text → embed → spawn engram at center
                        ↓
                  spiral into the tube
                        ↓
        gravity pulls toward similar engrams
                        ↓
         co-resonating pairs form synapses
                        ↓
   query broadcast → cosine + 1-hop synaptic →
        N responders contribute snippets →
              streamed integrated answer

Stack

• Rust workspace — domain types, event-driven sim, axum HTTP/WS server, sled persistence, LM Studio client
• Three.js + WebGL — additive-blended particle cloud, post-processing bloom, custom GLSL for per-engram pulse and hue variance
• LM Studio — local LLM (gemma-4-e4b) for introspection + query responders + integrator; embeddings via nomic-embed-text-v1.5

The full design lives in docs/system-analysis.md.

Quickstart

You need:

• Rust 1.80+ (via rustup)
• Node 20+ + npm
• LM Studio running locally with both google/gemma-4-e4b and text-embedding-nomic-embed-text-v1.5 loaded; copy your developer API token from LM Studio into config.local.toml:

  [lm_studio]
  api_token = "lms-..."
  

Build the frontend once:

(cd web && npm install && npm run build)

Then run the auto-ingesting demo:

cargo run --release -- demo
# → http://127.0.0.1:7777

The bin boots the server, hydrates any saved galaxies (data/), then spawns a background task that ingests assets/demo_corpus.txt (~50 paragraphs across cooking, ML, and history). Open the URL and you should see three jewel-tone clusters form within ~30 s.

Without demo, the server boots empty and you ingest manually via the bottom- right text area, or:

curl -X POST -H 'content-type: application/json' \
  -d '{"texts":["Pasta carbonara uses ...","..."]}' \
  http://127.0.0.1:7777/api/galaxy/<gid>/ingest

Asking a question

Use the bottom-left ask box. The query materialises as a pinned engram at the donut center; relevant engrams light up briefly and contribute a snippet; the integrated answer streams back over SSE. Try:

• What did we ingest about pasta?
• How does gradient descent work?
• When did the Berlin Wall fall?

API

route	what
GET /healthz	process status + LM Studio reachability
GET /api/stats	live HUD numbers (ticks_per_sec, llm_queue_depth, …)
GET /api/galaxy	list galaxies
POST /api/galaxy	{name} → new galaxy
DELETE /api/galaxy/:id	remove from sim + disk
POST /api/galaxy/:id/seed?n=200	spawn N synthetic engrams
POST /api/galaxy/:id/ingest	{texts: [...]}
POST /api/galaxy/:id/resize	{major_radius, minor_radius}
POST /api/galaxy/:id/query	{text} → {query_id}
GET /api/query/:id	latest snapshot
GET /api/query/:id/stream	SSE updates of running answer
GET /api/galaxy/:gid/engrams/:eid	inspector payload (manifest, memories…)
WS /ws/galaxy/:id/events	event stream (positions, synapses, query lifecycle)

Layout

crates/
  sophia-core/    # domain types, no I/O
  sophia-sim/     # event scheduler, physics, kiddo spatial index, broadcast
  sophia-llm/     # LM Studio client + caveman prompts (introspection, query)
  sophia-store/   # sled+JSON snapshot persistence
  sophia-server/  # axum HTTP + WS, query orchestrator, SSE answer stream
  sophia-bin/     # main; loads config, hydrates store, runs demo subcommand
web/              # Vite + TypeScript + Three.js
assets/           # demo corpus
data/             # sled DB (gitignored)
docs/             # design + plan

Configuration

config.toml is versioned and contains defaults; copy what you want to override into config.local.toml (gitignored — for the LM Studio token, or for tweaking the snapshot interval, etc.). Keys:

• [server] — host, port, static dir
• [storage] — data_dir, snapshot_interval_secs
• [lm_studio] — base URL, model names, API token, parallel-op cap
• [galaxy_defaults] — major_radius, minor_radius
• [token_caps] — per-call input/output caps for each LLM use site (note: gemma-4-e4b is a thinking model; output caps must budget for hidden reasoning plus visible content)

Persistence

Each galaxy snapshots to one row in a sled DB at data/, JSON-encoded, every 60 s. Kill the server with Ctrl-C or kill -9; on restart it loads the latest snapshot and resumes — same engram positions, slates, manifests, memories, and synapses.

Stage map

The implementation grew incrementally; each stage produced a runnable demo:

0. Scaffolding — workspace + black canvas
1. Engrams in space — physics, scheduler, position WS stream
2. Slate + LM Studio — embed, ingest, manifest
3. Gravity + clustering + synapses
4. Birth introspection (taxonomy/goals) + memories + click-to-inspect
5. Queries as conversations — broadcast, responders, integrator, SSE
6. Persistence + replay-on-boot
7. Polish — HUD (FPS, ticks/sec, LLM queue), demo subcommand

Each is verifiable end-to-end and the full plan is in docs/system-analysis.md.