Socrates/docs/implementation-plan.md

# Socrata — Implementation Plan (Phase 0 + Phase 1 MVP)

## Context

**Why this exists.** Socrata is a structured-thinking platform for product managers. It uses SysML as a formal semantic backbone — taxonomy, ontology, constraints, traceability — but hides that backbone behind two synchronized surfaces (a narrative text canvas and a SysML diagram canvas) and a conversational AI character (Socrates) who mediates all structural change.

**Where we start.** The repo at `/Users/dtoro/Projects/Socrata` is empty. A design bundle was extracted to `docs/design-source/socrata/` containing three high-fidelity React/HTML prototype variants (Manuscript, Foundry, Atelier) of the dual-canvas editor and the Socrates-led seed onboarding screen. The prototype renders the same dataset (project "Aristotle" — an AI study companion) across all three themes via CSS variables.

**Two-phase plan.** Before committing 8 weeks to the editor build, we run a **2-week Phase 0 validation experiment** that isolates and tests the riskiest assumption: *can we actually translate a PM's idea into a useful SysML model, and can Socrates do meaningful analysis on it?* Phase 0 uses **PlantUML** for visualization (not React Flow) so we spend the budget on prompts and evaluation, not on canvas rendering. Phase 1 (MVP, 8 weeks) only begins if Phase 0 passes the rubric — see [phase-0-validation.md](phase-0-validation.md).

**Companion docs.** [idea.md](idea.md) — full vision and Phase 1–3 spec. [phase-0-validation.md](phase-0-validation.md) — the 2-week validation experiment that gates MVP. [sysml-modeling.md](sysml-modeling.md) — the metamodel, validation rules, dependency graph, and SE discipline this implementation enforces. [socrates.md](socrates.md) — character, surfaces, modes, prompts, lifecycle, failure modes for the agent that mediates everything. [sync.md](sync.md) — bidirectional text↔diagram sync engineering: ops alphabet, applyOps chokepoint, optimistic-local with tempIds, SSE protocol, conflict handling.

**Decisions locked.**
- **Scope:** Phase 1 MVP end-to-end (per spec): seed → dual-canvas editor → SysML model + validation → Socrates-mediated proposals → assumptions/risks → requirements traceability → web-search validation → versioning. Out of scope: branching, multi-user co-edit, downstream artifacts.
- **Aesthetic:** **Manuscript** only. Parchment + Newsreader serif + ink-blue. Best fit for document-first PM thinking tool. Drop Foundry/Atelier theme files.
- **Stack:** Next.js 15 (App Router) + TypeScript + React 18, single deployable unit.
- **Diagram lib:** React Flow with custom nodes/edges styled to match prototype's softened SysML look.
- **Backend:** Next.js full-stack — route handlers + server actions, Postgres via Prisma, Anthropic SDK called server-side.
- **LLM:**
  - **Phase 0:** local model via **LM Studio** (OpenAI-compatible endpoint at `http://localhost:1234/v1`). Zero LLM cost during validation; local capability is the bar Phase 0 must clear. Recommended models: Qwen 2.5 72B / Llama 3.3 70B (or smaller-quantized if hardware-constrained). See [phase-0-plan.md §9](phase-0-plan.md).
  - **Phase 1 MVP:** **user-configurable** — `LLMGateway` interface supports both LM Studio (local, OpenAI-compatible) and Anthropic (hosted: Sonnet 4.6 + Haiku 4.5 with prompt caching). User picks at project setup or via a settings panel. Self-hosters get the privacy story; users who want frontier capability get hosted.
  - **Cross-phase:** prompts are model-portable in shape but not necessarily in quality. M6 includes a regression run of the Phase 0 corpus against **both** providers to catch drift in either path.
- **Persistence:** Postgres + Prisma. **No auth, no users in MVP** — single-tenant assumption, project owner is implicit.

---

## Architecture overview

```
apps/web                              Next.js app (single package, Turborepo not needed yet)
  app/                                App Router
    (seed)/page.tsx                   Seed screen — Socrates interview, emerging-seed left rail
    (editor)/[projectId]/page.tsx     Main dual-canvas editor
    api/                              Route handlers for streaming LLM, web search
      socrates/route.ts               POST — streamed Socrates turn
      proposals/[id]/analyze/route.ts POST — impact analysis on a pending proposal
      research/route.ts               POST — web search for an assumption/risk
  components/
    editor/                           Shell, TopBar, LeftRail, CanvasHeader, StatusBar
    text-canvas/                      ProseMirror or Lexical editor + chip-rendering decorators
    diagram-canvas/                   React Flow wrapper + custom block/actor/constraint nodes
    socrates/                         Sigil, Dock, Bubble, NumberedOptions, MarginNote
    seed/                             SeedScreen, EmergingSeedRail, MiniGraph, Confidence
    ui/                               Buttons, pills, kbd shortcuts (shared primitives)
  lib/
    sysml/                            Metamodel types + validation engine
      model.ts                        Block, Property, Association, Constraint, Requirement
      validate.ts                     Pure validator → ValidationIssue[]
      diff.ts                         Structural diff between two model versions
    sync/                             Bidirectional text ↔ model sync
      narrative.ts                    Narrative AST (TipTap/ProseMirror doc shape)
      markup.ts                       Chip parse/serialize, model-element references
      reconcile.ts                    Apply text edits → model ops; model ops → narrative patches
    llm/
      gateway.ts                      LLMGateway interface (chat, stream, json mode)
      anthropic.ts                    Anthropic implementation
      prompts/                        Versioned prompt templates (socrates, impact, detect, seed)
    proposals/                        Change-proposal lifecycle + impact analysis orchestration
    research/                         Web-search integration (Brave or Tavily) + citation linking
    confidence/                       Confidence-scoring utilities for auto-generated elements
    db/                               Prisma client + repository functions
  prisma/schema.prisma                Event-sourced model + projection tables
  styles/
    theme-manuscript.css              Ported from design-source, the only theme
    base.css                          Ported from design-source/styles.css (cleaned)
```

### Data model (Prisma)

Event-sourced for the spec's "Activity Changelog with Reasoning" requirement. Every state change writes to `ChangelogEntry`; `SysMLModel`, `NarrativeDocument`, `Assumption`, etc. are projections rebuilt from the event log.

```prisma
model Project              { id, name, scope, tagline, ownerLabel, createdAt }
model SysMLModel           { id, projectId, version, json /* serialized graph */ }
model NarrativeDocument    { id, projectId, version, doc /* ProseMirror JSON */ }
model Block                { id, projectId, label, kind, x, y, w, h, properties Json }
model Association          { id, projectId, fromId, toId, label, kind }
model Constraint           { id, projectId, label, expression }
model Requirement          { id, projectId, tag, text, tracedToIds String[] }
model Assumption           { id, projectId, text, status, linkedElementIds String[], confidence }
model Risk                 { id, projectId, text, severity, linkedElementIds String[] }
model Experiment           { id, assumptionId?, riskId?, hypothesis, methodology, results, implications }
model SocratesThread       { id, projectId, anchorElementId?, status }
model SocratesMessage      { id, threadId, role, content Json /* text + options */ }
model Proposal             { id, projectId, threadId?, title, ops Json, status, iterationCount, impactSummary Json }
model ChangelogEntry       { id, projectId, ts, kind, payload Json, reasoning, proposalId? }
model ResearchFinding      { id, projectId, anchorElementId, query, url, snippet, stance }
```

`ops` on a Proposal is a list of model mutations (add-block, update-property, add-association, add-constraint…). Applying = appending to the changelog and recomputing projections.

### Bidirectional sync model

The narrative is a ProseMirror document with custom `chip` inline nodes that carry `{ kind, refId }`. The canonical SysML graph lives in Postgres; the narrative is a projection-with-prose: editing prose around a chip is free-form, chips are model references.

- **Text → Model:** ProseMirror transactions are inspected for chip insert/delete/edit. Each becomes a candidate model op fed through Socrates (or auto-applied for low-impact ops above a confidence threshold).
- **Model → Text:** When a model element is renamed or deleted, all chips referencing its id are updated/removed in a single ProseMirror transaction. New blocks created from the diagram are appended as a templated narrative paragraph the user can edit.
- **Conflict avoidance:** All structural mutations route through a single `applyOps(projectId, ops, source)` function that updates DB + emits patches over a Server-Sent Events stream consumed by both canvases.

### LLM orchestration

`LLMGateway` interface — `chat(messages, opts)`, `stream(messages, opts)`, `json(messages, schema)`. Anthropic implementation uses the official `@anthropic-ai/sdk` with prompt caching enabled on the system prompts for Socrates/impact templates.

Three model tiers:
- **Sonnet 4.6** — Socrates conversational turns, impact analysis on proposals, seed-interview synthesis. Streamed.
- **Haiku 4.5** — continuous low-cost detection passes (assumptions/risks/inconsistencies on document save), confidence scoring on auto-generated elements.
- **Web search** — Tavily API (cheaper than Brave for this volume), called from `lib/research`, results linked back to model elements with stance classification (supports / contradicts / neutral).

Prompt templates are versioned files under `lib/llm/prompts/` so we can A/B them without code changes.

---

## Build sequence — gated on Phase 0

The build runs **Phase 0 → gate decision → Phase 1 (M1–M8)**. Phase 0 is detailed in [phase-0-validation.md](phase-0-validation.md); the summary below covers what it produces and how it feeds into MVP.

### Phase 0 — Validation (~2 weeks)

A throwaway harness — CLI + PlantUML rendering, no editor, no DB — that runs a corpus of 10 diverse seed ideas through:
- Seed → SysML model generation (Sonnet)
- Detection (assumptions, risks, inconsistencies)
- Conversational Socrates loop on each generated model
- Mediation (Socrates proposes a change, accept/reject, regenerate)

Each seed is scored on a 7-dimension rubric (model coverage / accuracy / parsimony / constraint capture; Socrates assumption + risk detection quality; voice and character). Pass criteria: ≥8 of 10 seeds average ≥3.5 with no dimension below 3.0.

**Phase 0 gate:**
- **Pass** → promote prompts (`lib/llm/prompts/socrates/`), metamodel types (`lib/sysml/model.ts`), seed corpus (eval suite), and PlantUML renderer (kept as MVP "export" feature) into Phase 1. Begin M1 with confidence.
- **Soft fail** → +1 week iteration on prompts/metamodel/model selection, re-run.
- **Hard fail** → do not start MVP. Sit with user, decide whether to pivot the metamodel, try Opus, or rethink the product hypothesis.

### Phase 1 — MVP build sequence (8 milestones, ~8 weeks)

Each milestone is independently demoable. M1–M3 are visual. M4–M8 add engine + intelligence — and import the validated artifacts from Phase 0 rather than authoring from scratch.

**M1 — Repo + visual port (week 1).**
- `pnpm create next-app` (TypeScript, App Router, no Tailwind — we use the prototype's CSS).
- Port `styles.css` and `theme-manuscript.css` into `styles/`. Drop foundry + atelier.
- Convert prototype `.jsx` files (`editor-shell`, `text-canvas`, `diagram.jsx` for static SVG version, `socrates`, `seed-screen`) into typed React components under `components/`. Use static `data.js` content as a fixture.
- Wire `(editor)/[projectId]/page.tsx` and `(seed)/page.tsx` to render the static version pixel-faithful to the prototype.
- **Done when:** the rendered editor and seed screens match the Manuscript artboard at 1480×920.

**M2 — Real text editor (week 2).**
- Replace static narrative renderer with **TipTap** (ProseMirror under the hood) configured with a custom `Chip` node that renders identically to the prototype's chip styles (pill / color / underline / bracket variants).
- Add a TipTap command for inserting a chip (slash menu: `/block`, `/property`, `/req`, `/assoc`).
- Persist narrative as ProseMirror JSON in `NarrativeDocument`.
- **Done when:** users can type prose, insert chips, and the document round-trips through the DB.

**M3 — Real diagram (week 3).**
- Replace static SVG with **React Flow**. Custom nodes for `block` / `actor` / `constraint` mirroring the softened look (rounded rect, stereotype label, divider line, property compartment). Custom edges for `association` / `composition` / `constraint` (dashed for constraint).
- Drag-to-create blocks via a left-side palette. Click to edit label/properties in a sidebar. Drag to draw associations.
- Persist positions and structure to `Block` / `Association` tables.
- **Done when:** users can build the Aristotle example from scratch in the diagram and see it persist on reload.

**M4 — SysML metamodel + validation (week 4).**
- Import `lib/sysml/model.ts` from Phase 0 (validated). Implement `lib/sysml/validate.ts` and `lib/sysml/depgraph.ts` per the rule codes (S1–S5, M1–M5, T1–T3) in [sysml-modeling.md](sysml-modeling.md). Pure functions, fully unit-testable.
- Surface issues inline in all four places: rail entry dot, diagram node border, narrative margin note, dock summary of new high-severity issues.
- Build the dependency graph incrementally on each model mutation — this is the substrate for impact analysis in M7.
- **Done when:** breaking the Aristotle model in known ways (cyclic composition, untraced requirement, dangling association endpoint) produces correct issues with zero false positives.

**M5 — Bidirectional sync + change ops (week 5).**
- `lib/sync/reconcile.ts` — single chokepoint `applyOps(projectId, ops, source)` mutates DB + emits SSE patch.
- ProseMirror plugin watches transactions, derives candidate ops (add chip → resolve to existing element; rename chip → mutate referenced element; delete chip → mark referenced element for removal review).
- React Flow change handlers feed the same `applyOps` path.
- SSE consumer in both canvases re-renders without losing local cursor state.
- **Done when:** renaming a block in the diagram updates every chip in the narrative live, and editing chip text in the narrative updates the diagram block label live.

**M6 — Socrates dock + LLM gateway (week 6).**
- `lib/llm/gateway.ts` with Anthropic implementation, prompt caching on system prompts.
- Import `lib/llm/prompts/socrates/` from Phase 0 (character + interview + review + mediate prompts already validated).
- `api/socrates/route.ts` streams Socrates turns over SSE. State stored in `SocratesThread` + `SocratesMessage`.
- Dock UI with bubbles, numbered-options affordance (matching prototype), reply input, ⌘↵ to send.
- Seed screen wired to the Phase-0-validated interview prompt; field inferences stream into the left rail as they arrive.
- **Done when:** holding a real conversation with Socrates seeded by the project context produces relevant questions and the seed screen converges to a populated emerging-seed rail end-to-end.

**M7 — Proposal workflow + impact analysis (week 7).**
- When a structural edit lands (text or diagram), classify by impact heuristic: **low-impact** (cosmetic rename, position move, property add) auto-applies; **high-impact** (delete block, change association, add constraint, edit requirement) creates a `Proposal` and routes through Socrates.
- `api/proposals/[id]/analyze/route.ts` calls Sonnet with the model snapshot + the proposed ops + linked assumptions/risks/requirements; returns structured `impactSummary` (impacted blocks, affected requirements, broken assumptions, suggested follow-ups).
- Proposal card in the dock shows summary + Approve/Refine/Reject. Approve → `applyOps`. Refine → Socrates iteration. Iteration count caps at 3 before Socrates makes a recommendation.
- Every applied op writes a `ChangelogEntry` with proposal-derived reasoning.
- **Done when:** deleting a foundational block in the Aristotle model raises a proposal that correctly enumerates affected REQ-001/REQ-002 and produces a sensible recommendation in ≤3 iterations.

**M8 — Assumptions, risks, research (week 8).**
- Background detection job (debounced on document save) calls Haiku with the narrative + model — using the Phase-0-validated detection prompts (`detect-assumptions.md`, `detect-risks.md`). Returns candidate findings with confidence scores and linked element ids. Persist; surface in the rail with confidence indicators.
- Per-assumption/risk "Validate via web" action → `api/research/route.ts` runs Tavily, stores `ResearchFinding` rows with stance classification.
- Findings appear inline as margin notes on the linked element; clicking a finding opens a side panel with the snippet + source.
- Experiment logging UI (modal): hypothesis / methodology / results / implications, linked to assumption or risk.
- Run the Phase 0 corpus through the integrated MVP as a regression check — the detection quality should match or exceed the Phase 0 results.
- **Done when:** running the detector on a fresh Aristotle document surfaces ASM-1/ASM-2/ASM-3 and RSK-1/RSK-2 (or close equivalents) with reasonable confidence, and at least one assumption gets researched with a real-world citation.

After M8: the spec's Phase 1 success metrics are testable end-to-end. Stop, ship to 5–10 beta PMs, measure, then scope Phase 2 (branching).

---

## Critical files / functions to reuse from the design source

The following live in `docs/design-source/socrata/project/` and should be **read and ported**, not invented from scratch:

- `styles.css` (820 lines) → `apps/web/styles/base.css`. The shell, canvases, chips, dock, seed, and proposal styles are all production-grade and themeable via CSS variables.
- `theme-manuscript.css` (79 lines) → `apps/web/styles/theme-manuscript.css`. The full color palette + typography for the chosen aesthetic.
- `editor-shell.jsx` lines 31–129 → `components/editor/LeftRail.tsx`. Collapsible-section logic and block-count tooltip behavior.
- `editor-shell.jsx` lines 1–29 → `components/editor/TopBar.tsx`.
- `editor-shell.jsx` lines 143–208 → `components/editor/EditorShell.tsx`.
- `text-canvas.jsx` chip-style logic (`chip-style-pill | color | underline | bracket`) → `components/text-canvas/Chip.tsx` as a TipTap node-view.
- `diagram.jsx` → reference for visual fidelity only; the actual implementation is React Flow custom nodes/edges. Match `radius`, `stroke`, header-bar treatment, stereotype label position from this file.
- `socrates.jsx` SocratesSigil (lines 3–30) → `components/socrates/Sigil.tsx` verbatim. The numbered-options bubble (lines 79–95) → `components/socrates/NumberedOptions.tsx`.
- `seed-screen.jsx` whole file → `components/seed/*` split into `EmergingSeedRail`, `MiniGraph`, `Confidence`, `Thread`, `InputRow`.
- `data.js` → `lib/fixtures/aristotle.ts`. Used as the seed state for the demo project and as test fixture for SysML validation + LLM eval.

---

## Open implementation questions to resolve during build

These are listed as spec "open questions"; the plan defers them to the milestone where they bite:

- **Markup syntax** — TipTap chip insertion via slash menu in M2 (no user-facing brackets needed). The four prototype rendering styles (`pill | color | underline | bracket`) ship as a user preference.
- **Sync direction** — full bidirectional from M5; M2/M3 ship one-way (DB → canvas) only.
- **Confidence threshold for auto-apply** — start at `>= 0.85` for low-impact ops; tune in M7 based on PM feedback.
- **Proposal iteration cap** — 3 iterations, then Socrates recommends. Hardcoded in M7.

---

## Verification

End-to-end flow that exercises the whole stack:

1. `pnpm dev`, open `http://localhost:3000` — lands on seed screen.
2. Conduct a 4–5 turn interview with Socrates about the Aristotle idea; watch the emerging-seed rail populate.
3. Click "Open editor" — lands on dual-canvas with the seed-derived model.
4. Edit a heading in the narrative, insert a `[Block: Tutor]` chip via slash menu — verify it appears in the diagram.
5. Drag-rename a block in the diagram — verify all matching chips in the narrative update live.
6. Delete a foundational block (`Aristotle`) — verify a Proposal appears in the dock with impact analysis listing affected requirements; iterate once with Socrates; approve.
7. Open Assumptions in the rail, pick one, click "Validate via web" — verify a real Tavily citation lands as a margin note linked to the assumption.
8. Reload the page — verify everything persists.
9. Run `pnpm test` — Prisma migration smoke test, SysML validator unit tests (12+ rules), reconcile.ts round-trip property tests.

Local prereqs: `DATABASE_URL` (Neon dev branch fine), `ANTHROPIC_API_KEY`, `TAVILY_API_KEY`. All three documented in `.env.example`.