Socrates/docs/phase-0-results.md

Phase 0 — Results

Closed 2026-04-28.

Verdict

GO for MVP.

The 10-seed corpus (8 working + 2 holdouts under final-round protocol) all pass the rubric defined in phase-0-validation.md §6.3. Pass rate 100%, corpus mean 4.32 / 5, no individual dimension below 3 on any seed. The brain works on a 4-billion-parameter local model — the assumption Phase 0 was designed to test.

Final round numbers

Run report: phase-0/reports/2026-04-28-FINAL-run.md. Model: google/gemma-4-e4b via LM Studio.

#	Seed	Difficulty	Avg	Min
1	01-aristotle	easy	4.43	4	✓
2	02-habit-coach	medium	4.29	3	✓
3	03-redline-ai	medium	4.43	4	✓
4	04-skillswap	medium	4.43	4	✓
5	05-cyclist-thing	failure-prone	4.29	3	✓
6	06-eventstream	hard	4.14	3	✓
7	07-pet-translator	failure-prone	4.14	3	✓
8	08-quiet-hours	medium	4.43	4	✓
9	09-carbon-coach (HOLDOUT)	medium	4.43	4	✓
10	10-telemetry-lite (HOLDOUT)	hard	4.14	3	✓

Holdouts both passed. They were never used during prompt iteration — both expectedDifficulty levels (medium and hard) cleared the rubric on first contact, ruling out corpus-overfitting as a confound.

What worked (shipping to MVP unchanged)

1. character.md — the Socrates voice prompt. Voice scored 4 across every seed. Anti-patterns ("never open with Great question", "no bullets in prose") held even on a 4B model. Promotes to apps/web/lib/llm/prompts/socrates/character.md verbatim.

2. generate.md — the seed → SysML model prompt, with two iterations applied during Phase 0:

   • Run-03 fix: explicit "constraint↔requirement separation" — items in seed.constraints go ONLY to the constraints array; requirements derive from seed.desiredOutcome. Eliminated content-duplication across all 8 seeds.
   • Run-05 fix: "vague desiredOutcome rule" — if the desired outcome is too vague to derive specific requirements, leave the requirements array empty rather than invent placeholders.

3. Detection prompts — detect-assumptions.md, detect-risks.md, detect-inconsistencies.md. Risk-detection averaged 4.50 across the corpus (highest of all dimensions). Promotes unchanged.

4. review.md — the conversational mode prompt. Numbered-options affordance works. Promotes unchanged.

5. propose.md + the oneOf JSON schema — the original "open kind: string" schema let Gemma cram all op fields into the kind name. Switching to oneOf per-op-kind made constrained decoding produce real ops. Critical lesson: small models need their JSON schema to enforce variant grammar, not just shapes.

6. judge.md — with the run-05 extension to invert coverage / accuracy / constraint-capture for failure-prone seeds (not just parsimony and risk). Survived a sanity check (deliberately broken Aristotle scored 1.29, real Aristotle 4.43+ — the judge isn't sycophantic).

What didn't, and what changed

• Constraint duplication (run-02 → run-03 fix). Every seed had REQ-NNN entries that literally restated seed.constraints text. Fixed with explicit prompt boundary.
• Vague-seed over-population (run-03 → run-05 fix). The "derive requirements from desiredOutcome" instruction nudged Gemma to invent placeholder requirements on Cyclist Thing's vague seed. Fixed with explicit empty-array rule.
• Hallucinated element refs in detection (existed throughout, mitigated by code). 4–5% of element ids in findings don't resolve to real model elements. Mitigated by lib/socrates/post-validate.ts — strips bad refs, drops findings with no remaining refs.
• Confidence > 1 emission (run-04 schema rejection). Gemma occasionally emits confidence: 1.05 etc despite the JSON Schema's maximum: 1. Fixed at the Zod layer with .transform(n => clamp(n, 0, 1)).
• Stochasticity (eventstream regression in run-02). At temperature 0.3 the same seed produces different fullness across runs. Variance can be ±1 on the average score. Mitigated by accepting that one bad single-run isn't dispositive.

What changed in the metamodel

Nothing. sysml-modeling.md types remained valid across the corpus. Gemma successfully generated all 10 models against the metamodel without forcing us to add or change types. Specifically:

• system | actor | block | constraint block kinds: sufficient for all 10 seeds
• association | composition | aggregation | generalization | constraintApplies association kinds: sufficient
• 4 property types (string | number | boolean | enum): sufficient
• The Constraint-Block separation (Section 4.5 in sysml-modeling.md) was the right call — once the prompt enforced it, every seed produced clean separation

One open question for MVP (deferred): EventStream and Telemetry Lite are technical/protocol seeds. They strain the metamodel because their natural vocabulary (Stream, Backpressure Signal, Producer client library) doesn't map cleanly to the PM-shaped Block/Actor distinction. Phase 0's metamodel handled them adequately (both passed) but a future "technical-systems profile" might add Service, Capability, or Protocol as block sub-kinds. Not needed for MVP.

What gets promoted to MVP (file-by-file)

Per the phase-0-plan.md §12 hand-off contract:

Phase 0 artifact	MVP destination
phase-0/src/prompts/character.md	apps/web/lib/llm/prompts/socrates/character.md (verbatim)
phase-0/src/prompts/generate.md	apps/web/lib/llm/prompts/socrates/generate.md (verbatim — used in seed-screen handoff M6)
phase-0/src/prompts/detect-*.md	apps/web/lib/llm/prompts/socrates/detect-*.md (verbatim, M8)
phase-0/src/prompts/review.md	apps/web/lib/llm/prompts/socrates/review.md (verbatim, M6 dock)
phase-0/src/prompts/propose.md	apps/web/lib/llm/prompts/socrates/propose.md (M7)
phase-0/src/prompts/judge.md	apps/web/test/eval/judge.md (CI eval)
phase-0/src/types.ts	apps/web/lib/sysml/model.ts (canonical types)
phase-0/src/generate/json-schema.ts	apps/web/lib/llm/schemas/sysml-model.ts (LLM JSON Schema for generation)
phase-0/src/socrates/post-validate.ts	apps/web/lib/socrates/post-validate.ts (the docs/socrates.md §6.3 reference-safety guard)
phase-0/src/socrates/propose.ts (oneOf schema)	apps/web/lib/socrates/propose.ts (op schema lessons preserved)
phase-0/seeds/	apps/web/test/eval/seeds/ (regression eval suite)
phase-0/src/render/to-plantuml.ts	apps/web/lib/export/plantuml.ts ("export model as PlantUML" feature)

Iteration history

5 corpus runs across 2 sessions. Reports committed to phase-0/reports/:

• 2026-04-28-1735-run.md — first corpus pass, no judge (visual-only review)
• 2026-04-28-1942-run.md — first scored run (run-02), 7/8 = 87.5% pass
• 2026-04-28-2115-run.md — run-03 with constraint↔requirement fix, 7/8, mean -0.05 but structure cleaner
• 2026-04-28-2129-run.md — run-04 added vague-outcome guard, 6/7 (one schema-validation failure)
• 2026-04-28-2144-run.md — run-05 with confidence-clamp + extended judge inversion, 8/8 = 100%, mean 4.39
• 2026-04-28-FINAL-run.md — 10/10 with holdouts, mean 4.32

Iteration notes per run in phase-0/reports/notes-run-0N.md.

Cost

LLM cost: $0. All inference was local via LM Studio (gemma-4-e4b on macOS). Wall-clock per scored corpus run: ~10–15 minutes for 8 seeds; ~17 minutes for 10 seeds (final round).

Constraints / things to know going into MVP

1. Hallucination floor. Even with the post-validate guard, expect ~4–5% of LLM-emitted element refs to need stripping. The MVP lib/socrates/post-validate.ts should preserve the guard from day one.
2. Variance. At temperature 0.3 a single seed can swing ±1 on average score across runs. MVP's Sonnet path may be more stable but should not be assumed to be deterministic.
3. Small-model JSON-schema lessons. Use oneOf for tagged unions, clamp ranges instead of rejecting, default-fill optional arrays. These survive into MVP for the local-LM-Studio user-config path; can be relaxed for the Sonnet path but cost nothing to keep.
4. Holdout discipline. Whenever MVP iterates a prompt, hold out at least 20% of the eval corpus from iteration. Prevents overfitting that the corpus alone can't detect.
5. The 4B-vs-Sonnet portability check (M6 task). Phase 0 validated against gemma-4-e4b. MVP M6 must re-run the corpus through the Anthropic provider to confirm scores hold. Prompts validated locally are not guaranteed to transfer 1:1.

Recommendation

Begin MVP M1 (visual port) per implementation-plan.md. The brain works. The risk that motivated Phase 0 is retired.