Socrates/docs/phase-0-plan.md

# Phase 0 — Build Plan

The execution companion to [phase-0-validation.md](phase-0-validation.md). This doc is concrete enough that someone can sit down and build the harness from it.

- **Strategy / rationale:** [phase-0-validation.md](phase-0-validation.md)
- **Metamodel reference:** [sysml-modeling.md](sysml-modeling.md)
- **Character reference:** [socrates.md](socrates.md)

---

## 1. Repo layout

Phase 0 lives in a subfolder of the main repo so it can be deleted cleanly after promotion to MVP.

```
/Users/dtoro/Projects/Socrata/
├── docs/                       (existing)
└── phase-0/                    ← new
    ├── package.json
    ├── tsconfig.json
    ├── .env.example            LMSTUDIO_BASE_URL (default http://localhost:1234/v1), LMSTUDIO_MODEL, PLANTUML_SERVER (optional)
    ├── src/
    │   ├── cli.ts              entrypoint — `phase0 <subcommand>`
    │   ├── types.ts            shared TypeScript types
    │   ├── seed/
    │   │   ├── interview.ts    5-question CLI interview → SeedPayload
    │   │   └── load.ts         load a seed JSON file
    │   ├── generate/
    │   │   ├── generate.ts     SeedPayload → GeneratedModel (Sonnet)
    │   │   └── schema.ts       Zod schema for structured output
    │   ├── llm/
    │   │   ├── client.ts       OpenAI-SDK client pointed at LM Studio's local endpoint
    │   │   └── messages.ts     chat helper with retries; JSON-mode structured outputs
    │   ├── socrates/
    │   │   ├── detect.ts       runs all three detection prompts
    │   │   ├── converse.ts     interactive chat loop on a model
    │   │   └── propose.ts      Socrates suggests a model patch; user accepts/rejects
    │   ├── render/
    │   │   ├── to-plantuml.ts  GeneratedModel → PlantUML source string
    │   │   └── render.ts       PlantUML source → PNG via public server
    │   ├── eval/
    │   │   ├── rubric.ts       score one run against rubric
    │   │   ├── judge.ts        LLM-as-judge prompt
    │   │   ├── corpus.ts       loads ./seeds, enforces holdouts
    │   │   └── report.ts       writes ./reports/YYYY-MM-DD-run.md
    │   └── prompts/
    │       ├── character.md
    │       ├── generate.md
    │       ├── detect-assumptions.md
    │       ├── detect-risks.md
    │       ├── detect-inconsistencies.md
    │       ├── interview.md
    │       ├── mediate.md
    │       └── judge.md
    ├── seeds/                  10 corpus seeds as JSON
    │   ├── 01-aristotle.json
    │   ├── 02-habit-coach.json
    │   ├── 03-redline-ai.json
    │   ├── 04-skillswap.json
    │   ├── 05-cyclist-thing.json    ← deliberately vague
    │   ├── 06-eventstream.json      ← technical edge
    │   ├── 07-pet-translator.json   ← feasibility-suspect
    │   ├── 08-quiet-hours.json
    │   ├── 09-carbon-coach.json     ← HOLDOUT
    │   └── 10-telemetry-lite.json   ← HOLDOUT
    ├── outputs/                generated artifacts (gitignored)
    │   └── <seed-id>/
    │       ├── model.json
    │       ├── diagram.puml
    │       ├── diagram.png
    │       ├── findings.json   assumptions + risks + inconsistencies
    │       ├── conversation.md
    │       └── score.json
    └── reports/                committed run history
        ├── 2026-04-29-run-01.md
        ├── 2026-05-02-run-02.md
        └── ...
```

Two seeds (#9, #10) are flagged HOLDOUT in their filenames and the corpus loader enforces "never include in iteration runs, only in the final round." See §7.

---

## 2. Data types

```ts
// src/types.ts

export interface SeedPayload {
  id: string;                       // matches filename, e.g. "01-aristotle"
  title: string;                    // "Aristotle"
  problem: string;                  // 1–3 sentences
  targetUser: string;               // 1 sentence
  desiredOutcome: string;           // 1 sentence
  initialHypothesis?: string;       // optional, may be inferred
  constraints?: string[];           // optional explicit list
  notes?: string;                   // any extra prose context
  meta: {
    isHoldout: boolean;             // §7 enforcement
    expectedDifficulty: 'easy' | 'medium' | 'hard' | 'failure-prone';
    testsFor: string[];             // tags: ["consumer-saas", "vague-seed", "feasibility-suspect"]
  };
}

// Mirrors SysMLModel from sysml-modeling.md but with confidence per element
export interface GeneratedModel {
  systemOfInterestId?: string;
  blocks: Array<Block & { confidence: number }>;
  associations: Array<Association & { confidence: number }>;
  constraints: Array<Constraint & { confidence: number }>;
  requirements: Array<Requirement & { confidence: number }>;
  overallConfidence: number;        // 0..1, model-author's self-report
  notes?: string;                   // model-author's rationale, free text
}

export interface Finding {
  kind: 'assumption' | 'risk' | 'inconsistency';
  text: string;
  linkedElementIds: string[];
  confidence: number;               // 0..1
  // For risks only:
  severity?: 'low' | 'medium' | 'high';
  // For inconsistencies only:
  validationCode?: string;          // 'M2', 'T1', etc.
}

export interface ConversationTurn {
  who: 'socrates' | 'user';
  text: string;
  options?: Array<{ n: number; label: string; sub: string }>;
  ts: string;
}

export interface RubricScore {
  // each 1-5
  modelCoverage: number;
  modelAccuracy: number;
  modelParsimony: number;
  constraintCapture: number;
  assumptionDetectionQuality: number;
  riskDetectionQuality: number;
  voiceAndCharacter: number;
  // optional:
  comments?: string;
  scorer: 'human' | 'llm-judge';
  scorerName?: string;
}

export interface RunArtifacts {
  seedId: string;
  promptVersionHash: string;        // git rev or content hash of /prompts/
  model: GeneratedModel;
  findings: Finding[];
  conversation: ConversationTurn[];
  scores: RubricScore[];            // can have multiple scorers
  durationMs: number;
  tokenUsage: { input: number; output: number; cacheRead: number };
}
```

---

## 3. Module API

```ts
// src/seed/load.ts
loadSeed(path: string): SeedPayload
loadCorpus(seedsDir: string, opts: { includeHoldouts: boolean }): SeedPayload[]

// src/seed/interview.ts
runInterview(): Promise<SeedPayload>     // CLI prompts, returns populated SeedPayload

// src/generate/generate.ts
generateModel(seed: SeedPayload, opts?: { model?: 'sonnet' | 'opus' }): Promise<GeneratedModel>

// src/socrates/detect.ts
detectFindings(model: GeneratedModel, seed: SeedPayload): Promise<Finding[]>

// src/socrates/converse.ts
runConversation(model: GeneratedModel, seed: SeedPayload, opts: { maxTurns: number }): Promise<ConversationTurn[]>

// src/socrates/propose.ts
proposeChange(model: GeneratedModel, focusElementId?: string): Promise<{
  ops: ModelOp[];
  reasoning: string;
}>
applyProposal(model: GeneratedModel, ops: ModelOp[]): GeneratedModel

// src/render/to-plantuml.ts
toPlantUML(model: GeneratedModel): string

// src/render/render.ts
renderPNG(plantumlSource: string, outPath: string): Promise<void>

// src/eval/rubric.ts
scoreRun(artifacts: RunArtifacts, rubricInput: { scorer: 'human' | 'llm-judge' }): Promise<RubricScore>

// src/eval/judge.ts
llmJudge(artifacts: RunArtifacts): Promise<RubricScore>

// src/eval/corpus.ts
runCorpus(opts: { includeHoldouts: boolean }): Promise<RunArtifacts[]>

// src/eval/report.ts
writeReport(runs: RunArtifacts[], outPath: string): Promise<void>
```

Each module is independently testable. Glue lives in `cli.ts`.

---

## 4. CLI

```bash
phase0 interview                                    # run an interactive seed interview, save to seeds/
phase0 run <seed-id>                                # full pipeline on one seed
phase0 run <seed-id> --converse                     # also run a 5-turn conversation
phase0 run <seed-id> --propose                      # simulate one proposal cycle
phase0 corpus                                       # all seeds (excluding holdouts)
phase0 corpus --final                               # all seeds INCLUDING holdouts (final round only)
phase0 score <seed-id> [--judge]                    # human or LLM-judge scoring
phase0 report                                       # generate reports/YYYY-MM-DD-run-NN.md
phase0 diff <run-A> <run-B>                         # compare two runs by score deltas
```

Flags inherited everywhere:
- `--model <name>` — override the LM Studio model name (default: whatever's loaded; CLI reads `LMSTUDIO_MODEL` env)
- `--temperature N` — override (default per mode)
- `--prompt-version <hash>` — pin a specific prompt commit

---

## 5. The seed corpus

Below is the planned shape of each seed. Full JSON gets authored in week 1 day 5; outlines below are the thinking blueprint.

| # | Title | Difficulty | Tests for |
|---|---|---|---|
| 01 | **Aristotle** — AI study companion that refuses to give answers | easy | Baseline; well-formed seed; known-good output |
| 02 | **Habit Coach** — Mobile app for parents tracking kids' screen time | medium | Multi-actor (parent + child + content provider); consumer pattern |
| 03 | **Redline AI** — Contract-redlining assistant for in-house legal teams | medium | Domain expertise; regulatory constraints; B2B pricing implications |
| 04 | **SkillSwap** — P2P skill-exchange marketplace for remote workers | medium | Two-sided market; transaction model; trust mechanisms |
| 05 | **Cyclist Thing** — "I want to build something for cyclists, like an app or a tool, not sure" | failure-prone | Vague seed — does Socrates push back instead of fabricating? |
| 06 | **EventStream** — A new event-streaming protocol with stronger backpressure semantics than Kafka | hard | Where PM-tool / engineer-tool blurs; metamodel may strain on protocol-level concepts |
| 07 | **Pet Translator** — App that "translates" pet vocalizations to text via ML | failure-prone | Wishful tech assumption — does Socrates surface feasibility risk? |
| 08 | **Quiet Hours** — Workplace tool that auto-blocks meetings during deep-work focus blocks | medium | Constraint-heavy (calendar, timezones, integrations); modest scope |
| 09 | **Carbon Coach** *(HOLDOUT)* — Personal carbon-footprint tracker with social leaderboards | medium | Behavioral/social mechanics; held out from iteration |
| 10 | **Telemetry Lite** *(HOLDOUT)* — Open-source self-host alternative to PostHog | hard | Technical, OSS distribution model; held out from iteration |

### Example seed file

`seeds/01-aristotle.json`:

```json
{
  "id": "01-aristotle",
  "title": "Aristotle",
  "problem": "First-year STEM students at large public universities frequently disengage from coursework not because the material is intractable, but because they lack a low-stakes thinking partner during the long tail between lectures and office hours.",
  "targetUser": "Undergraduates at large public universities, weeks 3–10 of an intro course.",
  "desiredOutcome": "Students re-engage with material via a low-stakes thinking partner — without producing solutions.",
  "initialHypothesis": "Students will adopt a tool that explicitly refuses to solve their homework, because the market is saturated with answer-givers.",
  "constraints": [
    "Must never output a complete solution to a graded problem",
    "Response latency under 1.2s P50 to preserve flow",
    "FERPA tenancy — coursework never leaves institutional boundary"
  ],
  "meta": {
    "isHoldout": false,
    "expectedDifficulty": "easy",
    "testsFor": ["baseline", "consumer-edu", "explicit-constraints"]
  }
}
```

### Failure-prone seed example

`seeds/05-cyclist-thing.json`:

```json
{
  "id": "05-cyclist-thing",
  "title": "Cyclist Thing",
  "problem": "I want to build something for cyclists. Maybe an app, maybe a tool, not sure yet.",
  "targetUser": "Cyclists.",
  "desiredOutcome": "Something useful for them.",
  "meta": {
    "isHoldout": false,
    "expectedDifficulty": "failure-prone",
    "testsFor": ["vague-seed", "underspecified", "socrates-pushback"]
  }
}
```

For seed #5, "good output" doesn't mean "produces a coherent model." It means **Socrates pushes back, refuses to fabricate, and asks the right clarifying questions to get a real seed**. The rubric handles this in §7.

---

## 6. Prompt drafts

These are first-pass drafts. Iteration during week 2 is expected.

### 6.1 `prompts/character.md`

(Promoted to MVP unchanged if Phase 0 passes — see [socrates.md §7.1](socrates.md))

```markdown
# Socrates

You are Socrates, a thinking partner for a product manager designing a product idea inside Socrata. You speak with peerage — not as an assistant, as a colleague.

## Voice
- Question-led. Default to surfacing the right question rather than volunteering a solution.
- Economical. Sentences carry weight. No filler.
- Skeptical by default. Neutral or mildly contrarian, never optimistic.
- Concrete. Refer to specific model elements by name when possible.
- Decisive when threads run long. After 2–3 iterations on a point, recommend.

## Never
- Open with affirmations like "Great question" or "Sure".
- Recap what the user just said.
- Apologize for limitations.
- Ask permission to draft when you could just propose.
- Cheerlead a weak idea.
- Use bullet points for prose responses.
- Reference any element not in the current model.

## Pattern
When responding, follow this structure unless the user asked a direct factual question:
1. Observe what just happened or what's true now.
2. Name the underlying tension or implication.
3. Propose a concrete next move (with numbered options if a decision is wanted).
```

### 6.2 `prompts/generate.md` — the seed → model prompt

This is the highest-leverage prompt in Phase 0. First draft:

```markdown
# Generate a SysML-shaped product model from a seed idea

You are an analyst who turns a product manager's seed idea into a structured systems-engineering model.

## Inputs
A seed payload with:
- `problem` — the user-named problem
- `targetUser` — who experiences the problem
- `desiredOutcome` — what success looks like
- `initialHypothesis` — optional belief
- `constraints` — optional explicit list

## Output
A `GeneratedModel` JSON conforming to the schema. The model must include:
- Exactly one block with `kind: "system"` — the System of Interest, the thing being built
- Blocks with `kind: "actor"` for external participants (users, third parties)
- Blocks with `kind: "block"` for things inside or adjacent to the system that the system reasons about
- Blocks with `kind: "constraint"` for non-negotiable invariants
- Associations between blocks (kinds: association, composition, generalization, constraintApplies)
- Requirements (REQ-NNN tags) drawn from explicit constraints AND from the desired outcome

## Rules
- The System of Interest is named after what the user is building, not what they're solving.
- Actors are people or external systems that interact with the SoI; they are not inside it.
- A "constraint" block represents a non-negotiable rule (regulatory, technical, ethical). Distinct from a Requirement, which is a goal the system must satisfy.
- Compositions are whole-part. Use sparingly — only when X is *literally part of* Y.
- Generalizations (is-a) are rarely needed for product ideas; prefer associations.
- Per element, set a `confidence` in [0, 1] reflecting how strongly the seed supports its inclusion. The seed's explicit nouns get high confidence (0.85+); inferred elements get medium (0.5–0.8); speculative elements get low (<0.5) and should generally be omitted.
- If the seed is too vague to model, return a sparse model (just an SoI block + a single actor) and set `overallConfidence` below 0.3. Do NOT fabricate to fill the diagram.

## Critical
- Under-suggest rather than over-suggest. A clean, sparse, correct model is better than a dense fabricated one.
- Use the user's own nouns when possible. Don't rename "Aristotle" to "AI Tutor."
- Properties on a block are the noun's *attributes the system reasons about*, not exhaustive ontology.

Return only valid JSON conforming to the schema. No prose preamble.
```

The prompt is paired with a Zod schema that enforces structure on the output. Generation uses LM Studio's **JSON mode** (`response_format: { type: 'json_schema', json_schema: ... }`) so the model is constrained to emit valid JSON conforming to the Zod-derived schema. If the loaded local model doesn't support `json_schema`, we fall back to `response_format: { type: 'json_object' }` plus post-validation with retry on parse failure.

### 6.3 `prompts/detect-assumptions.md`

```markdown
# Detect implicit assumptions in a product seed and model

Read the seed payload and the generated SysML model. Return a JSON array of candidate assumptions per the schema.

## What is an assumption
An implicit belief the user is treating as true without explicit validation. Examples:
- "Students will accept a tool that refuses answers" — implicit belief about adoption
- "1.2s P50 latency is achievable on-prem with available models" — implicit belief about technical feasibility
- "Faculty will not classify Socratic prompts as academic dishonesty" — implicit belief about institutional acceptance

## What is NOT an assumption
- Stated requirements (those are explicit)
- Constraints (those are non-negotiables, not beliefs)
- Definitions of terms

## Output
For each candidate, return:
- `text` — the assumption restated cleanly, in the user's own register
- `linkedElementIds` — which model elements this assumption is about
- `confidence` — 0 to 1, your confidence this is genuinely an unstated assumption

Return only candidates with confidence ≥ 0.5. Cap at 8.

## Quality bar
- Surface assumptions that are SPECIFIC to this product, not generic startup truisms ("users will want this").
- Each assumption should name a measurable, falsifiable belief.
- If the seed is sparse and you cannot confidently surface assumptions, return fewer rather than padding with generic ones.
```

### 6.4 `prompts/detect-risks.md`

```markdown
# Detect risks in a product seed and model

Read the seed payload and the generated SysML model. Return a JSON array of candidate risks per the schema.

## Categories
- Technical — feasibility, performance, scaling
- Market — adoption, competitive, distribution
- Execution — team, timing, dependencies
- Regulatory — compliance, legal, privacy
- External — third-party reliance, geopolitical

## Output
Per candidate:
- `text` — the risk restated as a specific failure mode
- `linkedElementIds` — model elements implicated
- `severity` — low / medium / high
- `confidence` — 0 to 1

Cap at 6. Confidence ≥ 0.5 only.

## Quality bar
- A risk must name a SPECIFIC failure mode tied to a SPECIFIC element. "Won't work" is not a risk; "Latency target unachievable on consumer-grade hardware given 7B-param inference" is.
- Severity reflects impact-if-it-happens, not probability.
- Avoid fabricated risks for vague seeds; return [] if you can't surface a real one.
```

### 6.5 `prompts/detect-inconsistencies.md`

```markdown
# Detect inconsistencies in a generated model

Given the seed and the model, find:
- Internal contradictions (two requirements that can't both hold; a block whose properties contradict its kind; a constraint that's already violated by some property)
- Reference issues (an association whose endpoints don't make semantic sense — actor → constraint, etc.)
- Over-broad claims (a requirement that promises more than the system can deliver based on its blocks)

## Output
Per candidate:
- `text` — the inconsistency stated clearly
- `linkedElementIds` — affected elements
- `confidence` — 0 to 1
- `validationCode` — if it matches a structural rule from sysml-modeling.md (S1–S5, M1–M5, T1–T3), include the code. Otherwise leave null.

Cap at 6. Confidence ≥ 0.6 only — for inconsistencies, false positives are worse than misses.
```

### 6.6 `prompts/interview.md`

```markdown
# Seed interview mode

You are conducting an opening interview with a product manager. Goal: produce a complete SeedPayload (problem, targetUser, desiredOutcome, optional hypothesis and constraints).

## Constraints
- 5 questions maximum, plus closing.
- Question 1: the problem in one sentence — the smallest, most honest version.
- Question 2: target user, with a specificity probe.
- Question 3: desired outcome — what changes when this exists.
- Question 4: a tension probe — name a likely tension and ask which side they're on.
- Question 5: constraints — anything that's non-negotiable.

After answers, synthesize a SeedPayload and return as JSON. Use the user's own register.

## Voice
Per character.md. Question-led, economical, no filler. Press for specificity if an answer is vague.

## Closing
After question 5, you may either:
- (a) Synthesize and return the SeedPayload immediately, OR
- (b) Surface one final clarification (only if a critical gap remains), then synthesize.

Do not exceed 6 turns total.
```

### 6.7 `prompts/mediate.md`

```markdown
# Proposal mediation

You are mediating a proposed change to the model. The user (or you) has authored a `proposed_ops` set. Your job:

1. Compute the impact: which elements are affected, which assumptions/risks/requirements are touched, what gets validated or invalidated.
2. Surface the impact in 2–4 sentences of prose.
3. If the change has ambiguity or hidden cost, ask one clarifying question OR offer 2–3 numbered options for refinement.
4. After 2 rounds of refinement, render a final recommendation: APPROVE / REFINE-ONCE-MORE / REJECT, with one-paragraph reasoning.

Output `impactSummary` JSON + dialogue text.

## Quality bar
- Impact must cite specific element ids, not generic categories.
- Don't mediate cosmetic changes (label rename, position) — those should auto-apply, you should never see them.
- Prefer fewer, sharper questions over many small ones.
```

### 6.8 `prompts/judge.md` — LLM-as-judge

```markdown
# Score a Phase 0 run against the rubric

You receive a complete RunArtifacts: seed, generated model, findings, conversation transcript.

Score each of 7 dimensions on 1–5:
1. Model coverage — did the model identify the major entities a real PM would name?
2. Model accuracy — are the relationships correct?
3. Model parsimony — uncluttered, no fabricated entities?
4. Constraint capture — meaningful non-functional constraints surfaced?
5. Assumption detection quality — real and specific, not generic?
6. Risk detection quality — domain-specific and substantive?
7. Voice and character — sounds like Socrates per the character spec? Question-led, skeptical, economical, never sycophantic?

For each, write 1–2 sentences of reasoning.

## Anti-bias
You are scoring an LLM's output. Apply extra scrutiny on:
- Voice drift (sycophancy creep, "great question", excessive hedging)
- Fabricated entities for vague seeds (penalize heavily)
- Generic risks/assumptions ("user adoption" without specifics)

A score of 5 means: a senior PM colleague would approve this output without changes. A score of 3 means: useful but needs work. A 1 means: misleading or hallucinated.
```

---

## 7. Evaluation harness — concretely

### 7.1 Holdout enforcement

`src/eval/corpus.ts`:

```ts
export function loadCorpus({ includeHoldouts }: { includeHoldouts: boolean }): SeedPayload[] {
  const all = listSeeds();
  if (includeHoldouts) {
    if (!process.env.PHASE0_FINAL_ROUND) {
      throw new Error('Holdouts only allowed in --final mode AND with PHASE0_FINAL_ROUND=1');
    }
    return all;
  }
  return all.filter(s => !s.meta.isHoldout);
}
```

Two guards prevent accidental iteration on holdouts: the `--final` CLI flag and the env var. We commit a `.git/hooks/pre-commit` that fails the commit if reports/ touches holdout seeds outside the final-round commit.

### 7.2 Run-and-report workflow

A typical iteration day:

```bash
# 1. Edit a prompt
$ vim src/prompts/generate.md

# 2. Run corpus
$ phase0 corpus

# 3. LLM-judge a first pass
$ for seed in seeds/*.json; do phase0 score $(basename $seed .json) --judge; done

# 4. Generate report
$ phase0 report

# 5. Eyeball the worst seeds
$ open outputs/05-cyclist-thing/diagram.png outputs/05-cyclist-thing/conversation.md

# 6. Compare against last run
$ phase0 diff reports/2026-04-29-run-01.md reports/2026-04-30-run-02.md
```

### 7.3 Report format

`reports/2026-04-30-run-02.md`:

```markdown
# Phase 0 Run 02 — 2026-04-30

**Prompt version:** `4f3a8b1` (vs. `a1c0290` last run)
**Changed prompts:** `generate.md`
**Seeds run:** 8 (holdouts excluded)
**Total time:** 4m 12s
**Token usage:** 184k input / 23k output / 156k cache-read

## Score summary
| Seed | M.Cov | M.Acc | M.Par | Cons | Asm | Risk | Voice | Avg | Δ |
|------|-------|-------|-------|------|-----|------|-------|-----|---|
| 01-aristotle    | 4.5 | 4.5 | 4.0 | 4.5 | 4.0 | 3.5 | 4.5 | 4.21 | +0.14 |
| 02-habit-coach  | 4.0 | 4.0 | 3.5 | 4.0 | 4.0 | 4.0 | 4.5 | 4.00 | +0.29 |
| ...

## Per-seed notes
### 05-cyclist-thing (failure-prone)
- Generation produced 2 blocks (good — sparse, didn't fabricate)
- Socrates correctly pushed back on the vagueness in turn 1
- Voice scored 4.5 (consistent with character)
- ⚠ Risk detection scored 2.5 — generated risks were generic ("competition exists"). FIX: tighten detect-risks.md to refuse-on-low-confidence.

## Pass/fail status
- 7 of 8 seeds above threshold
- 1 of 8 (cyclist-thing) below on Risk dimension
- DECISION: iterate detect-risks.md, re-run tomorrow
```

### 7.4 Final round

After the team is satisfied with non-holdout corpus performance, run **once**:

```bash
$ PHASE0_FINAL_ROUND=1 phase0 corpus --final
$ phase0 score 09-carbon-coach
$ phase0 score 10-telemetry-lite
$ phase0 report
```

If holdouts pass: Phase 0 succeeds, promote artifacts to MVP. If holdouts fail (and non-holdouts don't): we overfit to the corpus — back to iteration with a fresh prompt baseline.

---

## 8. Day-by-day plan (refined)

### Week 1 — build the harness

| Day | Concrete deliverables |
|---|---|
| **Mon** | `pnpm init`, repo scaffold, env vars. **LM Studio install + load chosen model + verify `/v1/chat/completions` responds.** OpenAI SDK pointed at `LMSTUDIO_BASE_URL`. `phase0 hello` runs a chat round-trip against the local model. PlantUML render of a hand-written model JSON works end-to-end. |
| **Tue** | `generate.md` v1 + Zod schema. `phase0 run 01-aristotle` produces `model.json` + `diagram.png`. Eyeball: does Aristotle's model look reasonable? |
| **Wed** | `detect-assumptions.md` + `detect-risks.md` + `detect-inconsistencies.md`. `phase0 run` extends to produce `findings.json`. |
| **Thu** | `converse.ts` — interactive loop; user types replies, Socrates responds with character + numbered options. `propose.ts` — Socrates emits a model patch, user accepts → regenerate diagram. |
| **Fri** | Author all 10 corpus seeds as JSON. First full `phase0 corpus` run (excluding holdouts). Output dump committed to `outputs/`. No scoring yet — just visual review. |

### Week 2 — iterate, evaluate, decide

| Day | Concrete deliverables |
|---|---|
| **Mon** | `judge.md` + `phase0 score --judge` works. Run on all 8 non-holdout seeds. First scored report committed. Identify top 3 failure patterns. |
| **Tue** | Iterate on the 1–2 most-broken prompts. Re-run corpus. Compare reports — verify scores moved correctly. |
| **Wed** | Iterate on metamodel if seeds force it (e.g. discover need for `Goal` or `Capability` as a kind). If yes, update `sysml-modeling.md` AND the prompts to match. Re-run. |
| **Thu** | Human scoring pass on the latest run. User scores 4 seeds personally; if a colleague is available, double-blind 2 seeds. |
| **Fri** | **Final round.** Run holdouts. Generate final report. **Go / soft-extend / hard-fail decision** with the user. If go: write `phase-0-results.md` summarizing what worked, promote artifacts to MVP scope. |

Optional **Week 3** (only if soft-fail): one more iteration cycle. Hard cutoff — if Friday of week 3 is still soft-fail, escalate to the user as a probable hard-fail.

---

## 9. Time, throughput, and model selection

### LLM cost
**Effectively $0** — all inference runs locally via LM Studio. Iteration is unconstrained by API budget.

### Throughput considerations
Local inference is bound by the loaded model's tokens/sec on the user's hardware, not by API rate limits. A typical corpus run (8 seeds × ~5 LLM calls each = ~40 calls) at 30 tok/sec on a 70B-class model with ~3k tokens average output works out to ~70 minutes for a full corpus run. Budget for that:
- **Iteration cycle:** prompt edit → corpus run → review → repeat. ~90 min per cycle including review.
- **Daily iteration count:** ~3–4 cycles in a focused day.
- **Speed lever:** if a model is too slow, drop a quantization tier (Q5_K_M → Q4_K_M) or pick a smaller model — Phase 0 is about prompt iteration, not benchmark accuracy.

### Recommended local model

The harness should work with any chat-completion-capable model loaded in LM Studio. For the structured-output + agentic tasks Phase 0 exercises, models known to perform well:

- **Qwen 2.5 72B Instruct** — strong on JSON-mode structured output, good instruction-following
- **Llama 3.3 70B Instruct** — solid all-rounder, well-supported
- **Qwen 2.5 32B Instruct** — if RAM-constrained; surprisingly capable
- **DeepSeek V3 / R1** — strong reasoning, larger memory footprint

Decision criteria: pick the largest model that fits the user's hardware and runs at ≥20 tok/sec. Quality difference between 32B and 72B matters for Phase 0; quantization (Q4 vs Q6) matters less.

Document the chosen model in `reports/<run>.md` so iteration history is reproducible.

### Human time
2 weeks × 1 person ≈ 80 hours.

---

## 10. Definition of done — the gate

Phase 0 is **done** when these artifacts exist and the user has reviewed them:

1. `phase-0/` repo with full harness, all prompts, all 10 seeds
2. `phase-0/reports/` containing at minimum 4 run reports showing iteration history
3. A final report with **holdout seeds included**, showing pass/fail per the §7.3 rubric
4. `docs/phase-0-results.md` — a 1–2 page summary written at the end:
   - What worked (which prompts shipped well)
   - What didn't (which prompts/types changed)
   - What was promoted to MVP (specific files + reasoning)
   - What changed in the metamodel, if anything
   - Final go/soft-extend/hard-fail call

The gate decision is: with these artifacts in hand, the user looks at the holdout-included scores and says "yes, the brain works — start MVP" OR "no, something is off — let's iterate / pivot."

---

## 11. Risks and mitigations specific to this build

| Risk | Mitigation |
|---|---|
| Author too many seeds and don't have time to iterate | 10 seeds is the budget. No more. |
| Iteration burns through budget on prompt micro-tweaks | Track score deltas per change; if 3 consecutive iterations don't move scores by ≥0.2, stop and reconsider the approach. |
| LLM-judge over-fits to the corpus and gives overly generous scores | Holdouts catch this. Also: rotate the judge prompt's wording per run to detect surface-level gaming. |
| PlantUML rendering becomes a time sink | Cap budget at 1 day on the rendering. If `to-plantuml.ts` isn't shippable by Mon EOD, ship a textual-only output and skip diagrams. Score on JSON. |
| Conversation loop produces mediocre output because we cap turns | Cap at 5 turns per seed deliberately — production Socrates also operates in finite contexts. If 5-turn output looks bad, the bug is the prompt, not the cap. |
| LM Studio crashes / OOM mid-run | `messages.ts` retries on 5xx and connection errors with exponential backoff. Corpus runs checkpoint per-seed so a crash mid-run loses only one seed. |
| Local model doesn't support strict JSON-schema mode | Fall back to `json_object` mode + Zod post-validation; on parse failure, re-prompt with the validation error appended. Cap retries at 3 per call. |
| Local model under-performs Sonnet-class on structured output | Phase 0 results may underestimate MVP quality. Mitigation: validation rubric is set against absolute quality bars, not relative to a baseline; if Phase 0 passes locally, MVP-on-Sonnet should be ≥ that. If MVP also runs local, Phase 0 results are directly representative. |
| Tokens/sec too low to support 3+ iterations/day | Drop to a smaller / more quantized model OR shorten the corpus to 6 seeds for development iterations, run all 10 only on full validation runs. |
| User unavailable during week 2 to score | LLM-judge runs first; user does final scoring async. Can be batched. |

---

## 12. Hand-off contract — what enters MVP if Phase 0 passes

These specific artifacts get copied (or referenced) into MVP:

| Phase 0 artifact | MVP destination |
|---|---|
| `phase-0/src/prompts/character.md` | `apps/web/lib/llm/prompts/socrates/character.md` |
| `phase-0/src/prompts/generate.md` | `apps/web/lib/llm/prompts/socrates/generate.md` (used in seed-screen handoff M6) |
| `phase-0/src/prompts/detect-*.md` | `apps/web/lib/llm/prompts/socrates/detect-*.md` (M8) |
| `phase-0/src/prompts/interview.md` | `apps/web/lib/llm/prompts/socrates/interview.md` (M6) |
| `phase-0/src/prompts/mediate.md` | `apps/web/lib/llm/prompts/socrates/mediate.md` (M7) |
| `phase-0/src/types.ts` (Block/Property/etc. types) | `apps/web/lib/sysml/model.ts` |
| `phase-0/seeds/` | `apps/web/test/eval/seeds/` (regression suite) |
| `phase-0/src/render/to-plantuml.ts` | `apps/web/lib/export/plantuml.ts` (kept as MVP "export" feature) |
| `phase-0/src/eval/judge.md` | `apps/web/test/eval/judge.md` (CI eval) |

The handoff doc (§10) records exactly which versions of which prompts/types ship to MVP, so MVP starts on a known-good foundation that we can blame git for if anything goes wrong.

### Model-portability note

Phase 0 prompts are validated against the **local LM Studio model** chosen in week 1. If MVP runs against a different model (e.g. Anthropic Sonnet 4.6 in production, or a hosted provider), there is no guarantee the prompts transfer 1:1. Required sanity check at MVP M6:

1. Re-run the Phase 0 corpus through MVP's LLM gateway against the production model
2. Compare scores against the Phase 0 final-round baseline
3. If scores drop ≥0.5 on any dimension, treat as a regression and iterate the prompt against the production model before continuing M6

MVP is user-configurable across LM Studio (local) and Anthropic (hosted), so M6's regression check runs the Phase 0 corpus against **both** providers. The local path is a no-op (same model as Phase 0); the hosted path is where prompt drift may surface.

---

## 13. What this plan does NOT do

For clarity:

- Does not validate the editor UX (TipTap, React Flow, sync) — that's MVP territory
- Does not validate persistence at scale or branching — Phase 1.5/2
- Does not produce a deployable demo — local-only CLI + JSON files
- Does not validate the Manuscript visual aesthetic — PlantUML doesn't carry it
- Does not test multi-user dynamics — out of scope through Phase 1
- Does not produce final production prompts — the prompts here are first drafts intended to be iterated; the *outputs* of the iteration are what ships

If you want any of these covered, they belong in the MVP plan, not Phase 0.