dtoro/Socrates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b55425cc68d818b3b22ad2f922cf8f2594265fa4
Socrates/docs/phase-0-results.md
dtoro f1c4566576 Initial commit — design docs + Phase 0 validation harness 
Sets up the Socrata project repo with:

docs/ — strategy and design documents
  - idea.md: full product vision
  - implementation-plan.md: Phase 0 + Phase 1 MVP plan
  - phase-0-validation.md: 2-week validation experiment strategy
  - phase-0-plan.md: concrete Phase 0 build plan
  - phase-0-results.md: Phase 0 gate outcome — GO for MVP
  - sysml-modeling.md: metamodel + SE discipline + validation rules
  - socrates.md: agent character, surfaces, modes, prompts, lifecycle
  - sync.md: bidirectional text↔diagram sync engineering
  - design-source/: HTML/CSS/JS handoff bundle from Claude Design

phase-0/ — validated harness (CLI, no UI, no DB)
  - LM Studio (local OpenAI-compatible) generation + detection + judge
  - PlantUML rendering for SysML model visualization
  - 10-seed corpus (8 working + 2 holdouts)
  - 5 corpus runs with iteration history in reports/
  - Final gate: 10/10 pass, mean 4.32/5, holdouts validated

Phase 1 MVP scope and milestones documented in implementation-plan.md.
2026-04-28 22:07:38 +02:00

112 lines
8.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Phase 0 — Results
_Closed 2026-04-28._
## Verdict
**GO for MVP.**
The 10-seed corpus (8 working + 2 holdouts under final-round protocol) all pass the rubric defined in [phase-0-validation.md §6.3](phase-0-validation.md). Pass rate **100%**, corpus mean **4.32 / 5**, no individual dimension below 3 on any seed. The brain works on a 4-billion-parameter local model — the assumption Phase 0 was designed to test.
## Final round numbers
Run report: [phase-0/reports/2026-04-28-FINAL-run.md](../phase-0/reports/2026-04-28-FINAL-run.md). Model: `google/gemma-4-e4b` via LM Studio.
| # | Seed | Difficulty | Avg | Min | |
|---|---|---|---|---|---|
| 1 | 01-aristotle | easy | 4.43 | 4 | ✓ |
| 2 | 02-habit-coach | medium | 4.29 | 3 | ✓ |
| 3 | 03-redline-ai | medium | 4.43 | 4 | ✓ |
| 4 | 04-skillswap | medium | 4.43 | 4 | ✓ |
| 5 | 05-cyclist-thing | failure-prone | 4.29 | 3 | ✓ |
| 6 | 06-eventstream | hard | 4.14 | 3 | ✓ |
| 7 | 07-pet-translator | failure-prone | 4.14 | 3 | ✓ |
| 8 | 08-quiet-hours | medium | 4.43 | 4 | ✓ |
| 9 | **09-carbon-coach** _(HOLDOUT)_ | medium | **4.43** | 4 | ✓ |
| 10 | **10-telemetry-lite** _(HOLDOUT)_ | hard | **4.14** | 3 | ✓ |
**Holdouts both passed.** They were never used during prompt iteration — both `expectedDifficulty` levels (medium and hard) cleared the rubric on first contact, ruling out corpus-overfitting as a confound.
## What worked (shipping to MVP unchanged)
1. **`character.md`** — the Socrates voice prompt. Voice scored 4 across every seed. Anti-patterns ("never open with Great question", "no bullets in prose") held even on a 4B model. Promotes to `apps/web/lib/llm/prompts/socrates/character.md` verbatim.
2. **`generate.md`** — the seed → SysML model prompt, with two iterations applied during Phase 0:
- **Run-03 fix:** explicit "constraint↔requirement separation" — items in `seed.constraints` go ONLY to the constraints array; requirements derive from `seed.desiredOutcome`. Eliminated content-duplication across all 8 seeds.
- **Run-05 fix:** "vague desiredOutcome rule" — if the desired outcome is too vague to derive specific requirements, leave the requirements array empty rather than invent placeholders.
3. **Detection prompts** — `detect-assumptions.md`, `detect-risks.md`, `detect-inconsistencies.md`. Risk-detection averaged 4.50 across the corpus (highest of all dimensions). Promotes unchanged.
4. **`review.md`** — the conversational mode prompt. Numbered-options affordance works. Promotes unchanged.
5. **`propose.md` + the `oneOf` JSON schema** — the original "open `kind: string`" schema let Gemma cram all op fields into the kind name. Switching to `oneOf` per-op-kind made constrained decoding produce real ops. **Critical lesson:** small models need their JSON schema to enforce variant grammar, not just shapes.
6. **`judge.md`** — with the run-05 extension to invert coverage / accuracy / constraint-capture for failure-prone seeds (not just parsimony and risk). Survived a sanity check (deliberately broken Aristotle scored 1.29, real Aristotle 4.43+ — the judge isn't sycophantic).
## What didn't, and what changed
- **Constraint duplication** (run-02 → run-03 fix). Every seed had REQ-NNN entries that literally restated `seed.constraints` text. Fixed with explicit prompt boundary.
- **Vague-seed over-population** (run-03 → run-05 fix). The "derive requirements from desiredOutcome" instruction nudged Gemma to invent placeholder requirements on Cyclist Thing's vague seed. Fixed with explicit empty-array rule.
- **Hallucinated element refs in detection** (existed throughout, mitigated by code). 4–5% of element ids in findings don't resolve to real model elements. Mitigated by `lib/socrates/post-validate.ts` — strips bad refs, drops findings with no remaining refs.
- **Confidence > 1 emission** (run-04 schema rejection). Gemma occasionally emits `confidence: 1.05` etc despite the JSON Schema's `maximum: 1`. Fixed at the Zod layer with `.transform(n => clamp(n, 0, 1))`.
- **Stochasticity** (eventstream regression in run-02). At temperature 0.3 the same seed produces different fullness across runs. Variance can be ±1 on the average score. Mitigated by accepting that one bad single-run isn't dispositive.
## What changed in the metamodel
**Nothing.** [sysml-modeling.md](sysml-modeling.md) types remained valid across the corpus. Gemma successfully generated all 10 models against the metamodel without forcing us to add or change types. Specifically:
- `system | actor | block | constraint` block kinds: sufficient for all 10 seeds
- `association | composition | aggregation | generalization | constraintApplies` association kinds: sufficient
- 4 property types (`string | number | boolean | enum`): sufficient
- The Constraint-Block separation (Section 4.5 in sysml-modeling.md) was the right call — once the prompt enforced it, every seed produced clean separation
**One open question for MVP** (deferred): EventStream and Telemetry Lite are technical/protocol seeds. They strain the metamodel because their natural vocabulary (`Stream`, `Backpressure Signal`, `Producer client library`) doesn't map cleanly to the PM-shaped Block/Actor distinction. Phase 0's metamodel handled them adequately (both passed) but a future "technical-systems profile" might add `Service`, `Capability`, or `Protocol` as block sub-kinds. **Not needed for MVP.**
## What gets promoted to MVP (file-by-file)
Per the [phase-0-plan.md §12 hand-off contract](phase-0-plan.md):
| Phase 0 artifact | MVP destination |
|---|---|
| `phase-0/src/prompts/character.md` | `apps/web/lib/llm/prompts/socrates/character.md` (verbatim) |
| `phase-0/src/prompts/generate.md` | `apps/web/lib/llm/prompts/socrates/generate.md` (verbatim — used in seed-screen handoff M6) |
| `phase-0/src/prompts/detect-*.md` | `apps/web/lib/llm/prompts/socrates/detect-*.md` (verbatim, M8) |
| `phase-0/src/prompts/review.md` | `apps/web/lib/llm/prompts/socrates/review.md` (verbatim, M6 dock) |
| `phase-0/src/prompts/propose.md` | `apps/web/lib/llm/prompts/socrates/propose.md` (M7) |
| `phase-0/src/prompts/judge.md` | `apps/web/test/eval/judge.md` (CI eval) |
| `phase-0/src/types.ts` | `apps/web/lib/sysml/model.ts` (canonical types) |
| `phase-0/src/generate/json-schema.ts` | `apps/web/lib/llm/schemas/sysml-model.ts` (LLM JSON Schema for generation) |
| `phase-0/src/socrates/post-validate.ts` | `apps/web/lib/socrates/post-validate.ts` (the [docs/socrates.md §6.3](socrates.md) reference-safety guard) |
| `phase-0/src/socrates/propose.ts` (oneOf schema) | `apps/web/lib/socrates/propose.ts` (op schema lessons preserved) |
| `phase-0/seeds/` | `apps/web/test/eval/seeds/` (regression eval suite) |
| `phase-0/src/render/to-plantuml.ts` | `apps/web/lib/export/plantuml.ts` ("export model as PlantUML" feature) |
## Iteration history
5 corpus runs across 2 sessions. Reports committed to `phase-0/reports/`:
- `2026-04-28-1735-run.md` — first corpus pass, no judge (visual-only review)
- `2026-04-28-1942-run.md` — first scored run (run-02), 7/8 = 87.5% pass
- `2026-04-28-2115-run.md` — run-03 with constraint↔requirement fix, 7/8, mean -0.05 but structure cleaner
- `2026-04-28-2129-run.md` — run-04 added vague-outcome guard, 6/7 (one schema-validation failure)
- `2026-04-28-2144-run.md` — run-05 with confidence-clamp + extended judge inversion, **8/8 = 100%**, mean 4.39
- `2026-04-28-FINAL-run.md` — **10/10 with holdouts, mean 4.32**
Iteration notes per run in `phase-0/reports/notes-run-0N.md`.
## Cost
LLM cost: **$0.** All inference was local via LM Studio (gemma-4-e4b on macOS). Wall-clock per scored corpus run: ~10–15 minutes for 8 seeds; ~17 minutes for 10 seeds (final round).
## Constraints / things to know going into MVP
1. **Hallucination floor.** Even with the post-validate guard, expect ~4–5% of LLM-emitted element refs to need stripping. The MVP `lib/socrates/post-validate.ts` should preserve the guard from day one.
2. **Variance.** At temperature 0.3 a single seed can swing ±1 on average score across runs. MVP's Sonnet path may be more stable but should not be assumed to be deterministic.
3. **Small-model JSON-schema lessons.** Use `oneOf` for tagged unions, clamp ranges instead of rejecting, default-fill optional arrays. These survive into MVP for the local-LM-Studio user-config path; can be relaxed for the Sonnet path but cost nothing to keep.
4. **Holdout discipline.** Whenever MVP iterates a prompt, hold out at least 20% of the eval corpus from iteration. Prevents overfitting that the corpus alone can't detect.
5. **The 4B-vs-Sonnet portability check (M6 task).** Phase 0 validated against gemma-4-e4b. MVP M6 must re-run the corpus through the Anthropic provider to confirm scores hold. Prompts validated locally are not guaranteed to transfer 1:1.
## Recommendation
Begin **MVP M1** (visual port) per [implementation-plan.md](implementation-plan.md). The brain works. The risk that motivated Phase 0 is retired.
Reference in New Issue View Git Blame Copy Permalink