# Phase 0 — Validation Experiment A two-week, throwaway experiment to validate the riskiest assumption in the product before committing to the 8-week MVP build. **No editor. No persistence. No production polish.** Just the brain. --- ## 1. Why Phase 0 exists The MVP plan in [implementation-plan.md](implementation-plan.md) commits 8 weeks to building the dual-canvas editor (TipTap + React Flow + bidirectional sync + proposal UX). That work is well-understood territory — we know how to build editors. What we **do not yet know** is whether the load-bearing intelligence works: 1. **Can we translate a PM's idea into a useful SysML model?** Given a seed (problem, user, outcome, hypothesis, constraints), does an LLM produce a model that identifies the right entities, the right relationships, meaningful constraints, and a defensible System of Interest — across a range of idea types and quality? 2. **Can Socrates do his job?** Given a model, does he surface *genuinely important* assumptions and risks (not generic boilerplate), conduct *useful* clarifying conversations, and produce *trustworthy* impact analyses? If either answer is "no" or "barely," the editor is theater. We'd be polishing a UX around a brain that doesn't work. Better to discover this in week 2 than week 10. Phase 0 isolates these two questions. It uses **PlantUML** as a visualization shortcut so we can spend our budget on prompts and evaluation, not on canvas rendering. --- ## 2. What we're validating, what we're not ### Validating - **NL → SysML translation quality.** Across a diverse seed corpus, does the model-generation prompt produce models that pass the rubric in §6? - **Metamodel adequacy.** Are the types in [sysml-modeling.md §5](sysml-modeling.md) sufficient to represent real PM ideas? Or do we discover gaps? - **Socrates' analytical capability.** Detection (assumptions, risks, inconsistencies), mediation (impact analysis on a proposed change), and conversational quality. - **Socrates' character and voice.** Does he sound like the persona in [socrates.md §1](socrates.md)? Does he push back, question, observe — without becoming pretentious? - **Confidence calibration.** When the model is uncertain (low confidence), is it *correctly* uncertain? When confident, is it correctly confident? ### NOT validating (these are MVP concerns) - Bidirectional sync mechanics ([sync.md](sync.md)) - Editor ergonomics (chips, slash menus, drag-create) - Persistence at scale, versioning, branching - Performance under load - Visual polish (PlantUML output is utilitarian — reviewers must look past aesthetics) - Multi-user / multi-tab behavior - Production deployment, auth, billing If Phase 0 succeeds we know the brain works and we can build the body. If it fails we iterate on prompts/metamodel until it works, or pivot. --- ## 3. The harness — minimal build A single-page tool, deliberately spartan. Probably 800 LoC end-to-end. ``` socrata-phase0/ src/ cli.ts CLI entrypoint: `phase0 run ` web.ts Optional minimal Next.js page for live demos seed/ types.ts SeedPayload type interview.ts Optional 5-question Socrates interview to populate a SeedPayload generate/ prompt.ts The seed → SysMLModel prompt (versioned) schema.ts JSON schema for structured output validation generate.ts Calls Anthropic with the seed payload, returns SysMLModel socrates/ character.md Same character prompt that ships to MVP detect-assumptions.md detect-risks.md detect-inconsistencies.md mediate.md converse.ts Chat loop — user asks Socrates a question about the model propose.ts Socrates proposes a model change → JSON patch → regenerate render/ to-plantuml.ts SysMLModel → PlantUML class-diagram source render.ts Calls PlantUML server (or local jar), saves PNG eval/ rubric.ts Scoring rubric (§6) corpus.ts The test seed corpus (§5) run-corpus.ts Run all seeds, render outputs, save report judge.ts Optional LLM-as-judge for first-pass scoring seeds/ Test corpus as JSON files aristotle.json habit-tracker.json contract-redline.json skill-exchange.json cyclist-vague.json streaming-protocol.json pet-translator.json ... outputs/ Generated artifacts (gitignored) / model.json diagram.png assumptions.json risks.json socrates-conversation.md score.json reports/ Run reports — committed YYYY-MM-DD-run.md package.json ``` ### What "running" looks like ```bash # Run a single seed end-to-end: $ phase0 run seeds/aristotle.json # Output: # 1. Generates SysMLModel JSON # 2. Renders PlantUML diagram → PNG # 3. Runs assumption + risk detection # 4. Saves transcript of a 5-turn Socrates conversation about the model # 5. Saves all artifacts to outputs/aristotle/ # Run the full corpus: $ phase0 corpus # Output: # - Iterates over every seed in seeds/ # - Generates outputs for each # - Runs LLM-judge on each (first pass) # - Generates reports/2026-04-29-run.md with side-by-side scores ``` ### What's deliberately missing - **No database.** Everything is JSON files. - **No auth.** Local dev only. - **No editor.** The "edit" cycle is: Socrates proposes a JSON patch → user accepts → regenerate diagram. That's the only mutation path. - **No bidirectional sync.** There's no narrative document. The seed and Socrates' conversation are the only text surfaces. - **No React Flow, no TipTap, no Prisma, no SSE.** We will not write a line of those in Phase 0. --- ## 4. PlantUML as the visualization shortcut ### Why PlantUML - **Text-based.** Generate a string, render an image. No interactive editor. - **Free and offline-capable.** Public render server or a local jar. - **SysML-shaped via stereotypes.** PlantUML class diagrams support `<>`, `<>`, `<>`, `<>` stereotypes — visually close enough to SysML. - **Cheap iteration.** Tweak the metamodel → re-emit PlantUML → re-render. No custom-node debugging. ### What we render A single class diagram per project, with: - One `<>` block (the SoI), styled distinctly - `<>` blocks for everything inside the system - `<>` blocks for external participants - `<>` blocks with dashed borders - Associations as `-->` (label on edge) - Compositions as `*--` - Generalizations as `<|--` - Constraint applications as `..>` (dashed) - Properties as class attributes - Requirements as a separate boxed list (notes or a side rectangle) ### Sample output (Aristotle seed) ```plantuml @startuml skinparam backgroundColor #f5efe2 skinparam class { BackgroundColor #fdfaf0 BorderColor #b8a982 ArrowColor #6e5d3d } hide empty members class Aristotle <> { refusal_policy interaction_style scope_window } class Student <> { self_efficacy course_load prior_grade } class Course <> { syllabus prerequisites } class Assignment <> { due_at rubric graded } class Instructor <> { policy_set } class FERPA <> { tenancy = institutional } Student --> Aristotle : consults Aristotle ..> Assignment : scoped_to Course "1" *-- "*" Assignment : contains Instructor --> Aristotle : configures Aristotle ..> FERPA : obeys Student --> Course : enrolled_in note right REQ-001: never produces complete solutions REQ-002: <1.2s P50 latency REQ-003: FERPA tenancy end note @enduml ``` ### Limitations we accept - PlantUML's auto-layout is mediocre. We don't try to position blocks; we let it auto-flow. - It can't render the "softened" aesthetic of the prototype. **That's fine** — we're testing model content, not visual design. - Some SysML niceties (e.g. requirement diagram boxes with «satisfy» dashed lines) are clunky in PlantUML. Acceptable for Phase 0. --- ## 5. The test seed corpus Diversity is the whole point. We need ideas that fail in different ways. | # | Seed | Type | What it tests | |---|---|---|---| | 1 | **Aristotle** (existing) | AI study companion | Baseline — well-formed seed, known-good output | | 2 | **Habit Coach** | Consumer mobile app for parents tracking kid screen-time | Consumer SaaS pattern, multi-actor (parent/child) | | 3 | **Redline AI** | B2B contract-redlining assistant for legal teams | Domain-heavy, regulatory constraints | | 4 | **SkillSwap** | Peer-to-peer skill exchange marketplace | Two-sided market, transaction model | | 5 | **Cyclist Thing** | "I want to build something for cyclists" — vague, no problem named | Tests low-quality seeds — does Socrates push back? | | 6 | **EventStream** | A new event-streaming protocol with backpressure semantics | Tests where PM tool / engineer tool blurs | | 7 | **Pet Translator** | App that "translates" pet vocalizations to text | Wishful tech assumption — does Socrates surface feasibility risk? | | 8 | **Quiet Hours** | A workplace tool that auto-blocks meetings during deep-work blocks | Constraint-heavy (calendar, timezone), modest scope | | 9 | **Carbon Coach** | Personal carbon-footprint tracker with social leaderboards | Behavioral model, social mechanics | | 10 | **Telemetry Lite** | Open-source self-host alternative to PostHog | Technical, OSS distribution model | Each seed is a JSON file with the seed payload — problem, target user, desired outcome, initial hypothesis, constraints — at varying quality levels. ### Seeds 5–7 are deliberately failure-prone Seed 5 is too vague. Seed 6 may push the metamodel toward technical-system territory we're not optimized for. Seed 7 has a plausibility problem. We *want* Socrates to handle these badly-shaped inputs gracefully — that's what differentiates a thinking partner from a rubber stamp. --- ## 6. Evaluation rubric Each seed gets scored on seven dimensions, 1–5 each. Two passes: human + LLM-as-judge (sanity check). ### 6.1 Model dimensions 1. **Coverage** — did the model identify the major entities a real PM would name? (1: missed obvious ones; 5: comprehensive) 2. **Accuracy** — are the relationships correct? Compositions actually whole-part? Generalizations actually is-a? (1: many wrong; 5: all correct) 3. **Parsimony** — uncluttered, no fabricated entities? (1: hallucinated noise; 5: clean) 4. **Constraint capture** — meaningful non-functional constraints surfaced? (1: missed; 5: all the obvious ones) ### 6.2 Socrates dimensions 5. **Assumption detection quality** — surfaces real, non-obvious assumptions? (1: generic boilerplate; 5: sharp and specific) 6. **Risk detection quality** — surfaces real risks the PM should care about? (1: generic; 5: domain-specific and substantive) 7. **Voice and character** — sounds like Socrates per [socrates.md §1](socrates.md)? Question-led, skeptical, economical? (1: generic chatbot; 5: distinctly on-character) ### 6.3 Pass criteria - **Per-seed:** average score ≥ 3.5 across all 7 dimensions, with no individual dimension below 3.0. - **Across corpus:** at least 8 of 10 seeds pass. - **Failure-case seeds (5–7):** at minimum, Socrates must *surface* the issue (push back on vagueness; flag feasibility; note where the metamodel strains). He doesn't have to solve it, but he must not paper over it. If we hit those criteria, Phase 0 is a **go** for MVP. ### 6.4 LLM-as-judge for first-pass scoring We use a Sonnet 4.6 prompt that scores each dimension with reasoning. This is cheap and lets us iterate on prompts before involving human reviewers. **Important:** LLM-judge is a sanity check, not the final score. Anthropic's own research shows LLM judges drift toward agreement with the LLM-generated content. Final scoring is by humans (the user + 1–2 colleagues if available). --- ## 7. Iteration loop This is what most of the two weeks actually looks like: ``` 1. Run corpus → scores 2. Read failures 3. Identify pattern (prompt issue? metamodel gap? Socrates voice drift?) 4. Adjust: - Edit prompt template - OR adjust metamodel types - OR change model selection (Opus 4 for harder seeds?) 5. Re-run corpus 6. Compare scores against previous run 7. Repeat ``` We track every run in `reports/YYYY-MM-DD-run.md` so we can see whether we're improving or regressing. The reports are committed; the per-seed `outputs/` are gitignored (large PNGs). ### Things we expect to learn - **Which seed shapes work best.** Probably consumer SaaS with explicit constraints; probably worst on vague seeds. - **Where the metamodel strains.** Might discover we need `Goal` as a distinct kind, or `Stakeholder` as a richer Actor. We **must not** add types speculatively — only when a real seed forces it. - **Whether Sonnet is enough or we need Opus** for generation. Cost difference is ~3×; quality difference may justify it. - **Where Socrates' voice drifts.** Probably toward helpful-assistant register without explicit anti-patterns. We tighten `character.md` until it sticks. ### What we do NOT do during iteration - Add new dimensions to the rubric mid-experiment (game the score) - Cherry-pick seeds that look good and drop the others - Treat early high scores as success — calibrate by re-judging known cases --- ## 8. Wiring up Socrates — concretely Phase 0 implements four of Socrates' six modes from [socrates.md §3](socrates.md): | Mode | Phase 0 implementation | |---|---| | **Interview** | `seed/interview.ts` runs a 5-question CLI conversation that produces a SeedPayload. Tested on every corpus seed. | | **Detection** | `socrates/detect-*.md` prompts produce JSON arrays of assumptions, risks, inconsistencies. Run automatically after model generation. | | **Review (conversational)** | `socrates/converse.ts` is a CLI chat loop where the tester asks Socrates questions about the model and gets responses (with numbered options where appropriate). Transcripts saved per seed. | | **Mediation** | `socrates/propose.ts` — when Socrates proposes a model change, output is a structured JSON patch + reasoning. The tester accepts or rejects. Accepted patches re-run model generation; rejected patches log why. | Modes deferred to MVP: - **Synthesis** (rationale generation) — depends on changelog accumulation, which Phase 0 doesn't have - **Translation** (NL → ops on a narrative document) — depends on the editor surface That's still a robust Socrates experience — interview, surface findings, converse about the model, propose changes — exercised end-to-end across the corpus. --- ## 9. Success / failure → Phase 1 gate ### Pass If §6.3 criteria are met: - Promote `lib/llm/prompts/socrates/` files to MVP unchanged (or with minor refinement) - Promote `lib/sysml/model.ts` types to MVP - Promote test corpus to MVP eval suite - Begin MVP M1 with confidence the brain works ### Soft fail (most likely outcome) Some dimensions pass, others don't. Pattern emerges. Examples: - Model coverage 4.5, Socrates voice 2.5 → spend a week iterating `character.md` - Generation hallucinates entities for vague seeds → tighten generation prompt with "if seed is unclear, return fewer high-confidence entities + a clarifying question" - Metamodel strains on EventStream — discover we need `Capability` or `Service` as a distinct kind → add the type, re-run Soft fail extends Phase 0 by 1 week. Acceptable. ### Hard fail We cannot get above 3.0 average on any dimension after multiple iterations. This means: - The metamodel is wrong, OR - Sonnet 4.6 isn't capable enough, OR - The product idea has a deeper problem (PMs don't think SysML-shaped) In any of those cases, we **do not proceed to MVP**. We sit with the user and decide whether to: - Pivot the metamodel toward what does work (e.g., drop SysML pretense, use a flat entity-relationship model) - Try Opus or future-Anthropic-models - Reconsider the product hypothesis This is the whole point of Phase 0: making this discovery cheap. --- ## 10. Timeline — 2 weeks ### Week 1 — build harness, ship Aristotle | Day | Goal | |---|---| | Mon | Repo scaffold, Anthropic SDK wiring, PlantUML render of a hand-written model | | Tue | First-pass `generate/prompt.ts`, run on Aristotle seed end-to-end | | Wed | Detection prompts (assumptions, risks, inconsistencies) + integration | | Thu | Conversational Socrates loop + propose-and-apply flow | | Fri | Test corpus (10 seeds) authored as JSON; first full corpus run | ### Week 2 — iterate, evaluate, decide | Day | Goal | |---|---| | Mon | LLM-as-judge wired up; first scored run; identify top 3 failure patterns | | Tue | Iterate on the most-broken prompt(s); re-run corpus | | Wed | Iterate on metamodel if seeds force it; re-run | | Thu | Human scoring pass on best-run outputs | | Fri | Final report → go/soft-extend/hard-fail decision; promote artifacts to MVP if go | If iteration reveals deep issues, week 3 is allowed before triggering hard-fail. --- ## 11. Tech stack Deliberately tiny: - **Node 20+ / TypeScript** — same language as MVP, so prompts/types are portable - **`@anthropic-ai/sdk`** — Sonnet 4.6 + Haiku 4.5 - **`zod`** — JSON schema validation on LLM structured outputs - **`commander`** — CLI scaffolding - **`plantuml-encoder`** + public PlantUML server (or local `plantuml.jar` if we want offline) - **`fs` for JSON files** — no DB - *(Optional)* Next.js + a single page if we want a live-demo URL — not required No React Flow, no TipTap, no Prisma, no Postgres, no SSE, no auth. --- ## 12. Risks of Phase 0 itself - **Cherry-picking the corpus.** We'll be tempted to drop seeds that fail. *Mitigation:* corpus locked at end of week 1, day 5; seeds 5–7 (the hard ones) are mandatory. - **Self-grading bias.** The team scoring is the team building. *Mitigation:* LLM-judge is a baseline; the user does final scoring; if a colleague is available, double-blind a subset. - **PlantUML aesthetic ≠ MVP aesthetic.** Reviewers may unconsciously down-rate ugly output. *Mitigation:* explicit instruction to score on *content*, not visuals; brief reviewers on this before they score. - **Prompt drift.** We may iterate prompts to fit the corpus, then deploy and find they don't generalize. *Mitigation:* hold out 2 of the 10 seeds as "test set" — never iterate on them; only run them in the final round. - **MVP-confidence false positive.** Even if Phase 0 passes, MVP's editor UX may surface model-quality issues we didn't catch. *Acceptable:* the editor is well-understood; we accept that some MVP-time refinement is normal. --- ## 13. Deliverables → MVP hand-off If Phase 0 passes, these artifacts ship into MVP: 1. **`lib/llm/prompts/socrates/`** — all character + mode prompts, validated across the corpus 2. **`lib/sysml/model.ts`** — types confirmed adequate (or expanded based on corpus learnings) 3. **`seeds/`** — the 10-seed corpus, becomes the eval-suite for ongoing prompt regression in MVP 4. **`reports/`** — the run history, captured as evidence of validation 5. **`render/to-plantuml.ts`** — kept and shipped to MVP as an export format ("share as PlantUML") 6. **A short summary doc** — `phase-0-results.md` — appended to this file, outlining what worked, what didn't, what changed --- ## 14. Why this de-risks the whole project Phase 0 costs ~10% of the total project budget (2 weeks of 10) and cuts the largest unknown to a known-good or known-bad. The MVP is then either a confident build on a validated brain, or a pivot before we've spent 8 weeks building UX around something broken. This pattern is borrowed from product discovery: cheapest possible test of the riskiest assumption first. Everything else can be calibrated later.