Implement Sophia MVP stages 4-7 (introspection, queries, persistence, polish)

Brings the system from "engrams cluster + form synapses" to a complete end-to-end demo: ingest text, watch it cluster, ask questions, restart with state intact. - Stage 4: birth introspection (taxonomy/goals/open_questions via LLM, bounded by the global parallel-op semaphore), per-engram memory log, click-to-inspect side panel. - Stage 5: queries as conversations. POST /api/galaxy/:id/query embeds the question, materializes a pinned Query-Engram at the donut center, runs broadcast retrieval (global cosine scan + 1-hop synaptic expansion with attenuation) and fans out responder LLM calls. The integrator runs every 2s on accumulated snippets and streams the refining answer back over SSE; responders briefly transition to Conversing on the WS bus so the right dots light up. - Stage 6: snapshot persistence. sled-backed store keyed by galaxy id, JSON-encoded values (bincode chokes on internally-tagged enums like Manifest/MemoryKind), 60s periodic snapshot task, hydrate-on-boot, DELETE /api/galaxy/:id wired through. State survives kill -9. - Stage 7: HUD additions (sim ticks/sec, LLM queue depth, FPS) via a new GET /api/stats polled at 1Hz. `sophia demo` subcommand boots the server then auto-ingests a 50-paragraph corpus baked into the binary with include_str!. README quickstart added. Token caps for query_responder/integrator bumped (gemma-4-e4b is a thinking model — output budget must cover hidden reasoning + visible answer, otherwise content comes back empty). Pinned engrams skip physics; their tick scheduling is also skipped at materialization so they stay perfectly still at the donut center. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 11:53:11 +02:00
parent 8688f632bf
commit bae084cd76
36 changed files with 3137 additions and 58 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,156 @@
+# Sophia
+
+A knowledge engine where each idea is an autonomous agent — an *Engram* — that
+moves, resonates with similar ideas, forms synapses with them, and answers
+questions by gathering its own perspective into a single integrated reply.
+
+The whole system is a 3D simulation you can watch: ingested text becomes glowing
+dots that fly out from the centre of a torus, drift toward similar peers, and
+light up when a query's broadcast wave touches them.
+
+```
+ingest text → embed → spawn engram at center
+                        ↓
+                  spiral into the tube
+                        ↓
+        gravity pulls toward similar engrams
+                        ↓
+         co-resonating pairs form synapses
+                        ↓
+   query broadcast → cosine + 1-hop synaptic →
+        N responders contribute snippets →
+              streamed integrated answer
+```
+
+## Stack
+
+- **Rust** workspace — domain types, event-driven sim, axum HTTP/WS server,
+  sled persistence, LM Studio client
+- **Three.js + WebGL** — additive-blended particle cloud, post-processing
+  bloom, custom GLSL for per-engram pulse and hue variance
+- **LM Studio** — local LLM (`gemma-4-e4b`) for introspection + query
+  responders + integrator; embeddings via `nomic-embed-text-v1.5`
+
+The full design lives in [`docs/system-analysis.md`](docs/system-analysis.md).
+
+## Quickstart
+
+You need:
+
+- Rust 1.80+ (via `rustup`)
+- Node 20+ + `npm`
+- LM Studio running locally with both `google/gemma-4-e4b` and
+  `text-embedding-nomic-embed-text-v1.5` loaded; copy your developer API
+  token from LM Studio into `config.local.toml`:
+  ```toml
+  [lm_studio]
+  api_token = "lms-..."
+  ```
+
+Build the frontend once:
+
+```bash
+(cd web && npm install && npm run build)
+```
+
+Then run the auto-ingesting demo:
+
+```bash
+cargo run --release -- demo
+# → http://127.0.0.1:7777
+```
+
+The bin boots the server, hydrates any saved galaxies (`data/`), then spawns a
+background task that ingests `assets/demo_corpus.txt` (~50 paragraphs across
+cooking, ML, and history). Open the URL and you should see three jewel-tone
+clusters form within ~30 s.
+
+Without `demo`, the server boots empty and you ingest manually via the bottom-
+right text area, or:
+
+```bash
+curl -X POST -H 'content-type: application/json' \
+  -d '{"texts":["Pasta carbonara uses ...","..."]}' \
+  http://127.0.0.1:7777/api/galaxy/<gid>/ingest
+```
+
+## Asking a question
+
+Use the bottom-left **ask** box. The query materialises as a pinned engram at
+the donut center; relevant engrams light up briefly and contribute a snippet;
+the integrated answer streams back over SSE. Try:
+
+- `What did we ingest about pasta?`
+- `How does gradient descent work?`
+- `When did the Berlin Wall fall?`
+
+## API
+
+| route | what |
+| --- | --- |
+| `GET /healthz` | process status + LM Studio reachability |
+| `GET /api/stats` | live HUD numbers (`ticks_per_sec`, `llm_queue_depth`, …) |
+| `GET /api/galaxy` | list galaxies |
+| `POST /api/galaxy` | `{name}` → new galaxy |
+| `DELETE /api/galaxy/:id` | remove from sim + disk |
+| `POST /api/galaxy/:id/seed?n=200` | spawn N synthetic engrams |
+| `POST /api/galaxy/:id/ingest` | `{texts: [...]}` |
+| `POST /api/galaxy/:id/resize` | `{major_radius, minor_radius}` |
+| `POST /api/galaxy/:id/query` | `{text}` → `{query_id}` |
+| `GET /api/query/:id` | latest snapshot |
+| `GET /api/query/:id/stream` | SSE updates of running answer |
+| `GET /api/galaxy/:gid/engrams/:eid` | inspector payload (manifest, memories…) |
+| `WS /ws/galaxy/:id/events` | event stream (positions, synapses, query lifecycle) |
+
+## Layout
+
+```
+crates/
+  sophia-core/    # domain types, no I/O
+  sophia-sim/     # event scheduler, physics, kiddo spatial index, broadcast
+  sophia-llm/     # LM Studio client + caveman prompts (introspection, query)
+  sophia-store/   # sled+JSON snapshot persistence
+  sophia-server/  # axum HTTP + WS, query orchestrator, SSE answer stream
+  sophia-bin/     # main; loads config, hydrates store, runs demo subcommand
+web/              # Vite + TypeScript + Three.js
+assets/           # demo corpus
+data/             # sled DB (gitignored)
+docs/             # design + plan
+```
+
+## Configuration
+
+[`config.toml`](config.toml) is versioned and contains defaults; copy what you
+want to override into `config.local.toml` (gitignored — for the LM Studio
+token, or for tweaking the snapshot interval, etc.). Keys:
+
+- `[server]` — host, port, static dir
+- `[storage]` — `data_dir`, `snapshot_interval_secs`
+- `[lm_studio]` — base URL, model names, API token, parallel-op cap
+- `[galaxy_defaults]` — `major_radius`, `minor_radius`
+- `[token_caps]` — per-call input/output caps for each LLM use site
+  (note: `gemma-4-e4b` is a thinking model; output caps must budget for
+  hidden reasoning *plus* visible content)
+
+## Persistence
+
+Each galaxy snapshots to one row in a sled DB at `data/`, JSON-encoded, every
+60 s. Kill the server with `Ctrl-C` or `kill -9`; on restart it loads the
+latest snapshot and resumes — same engram positions, slates, manifests,
+memories, and synapses.
+
+## Stage map
+
+The implementation grew incrementally; each stage produced a runnable demo:
+
+0. Scaffolding — workspace + black canvas
+1. Engrams in space — physics, scheduler, position WS stream
+2. Slate + LM Studio — embed, ingest, manifest
+3. Gravity + clustering + synapses
+4. Birth introspection (taxonomy/goals) + memories + click-to-inspect
+5. Queries as conversations — broadcast, responders, integrator, SSE
+6. Persistence + replay-on-boot
+7. Polish — HUD (FPS, ticks/sec, LLM queue), `demo` subcommand
+
+Each is verifiable end-to-end and the full plan is in
+[`docs/system-analysis.md`](docs/system-analysis.md).