dtoro
f1c4566576
Sets up the Socrata project repo with: docs/ — strategy and design documents - idea.md: full product vision - implementation-plan.md: Phase 0 + Phase 1 MVP plan - phase-0-validation.md: 2-week validation experiment strategy - phase-0-plan.md: concrete Phase 0 build plan - phase-0-results.md: Phase 0 gate outcome — GO for MVP - sysml-modeling.md: metamodel + SE discipline + validation rules - socrates.md: agent character, surfaces, modes, prompts, lifecycle - sync.md: bidirectional text↔diagram sync engineering - design-source/: HTML/CSS/JS handoff bundle from Claude Design phase-0/ — validated harness (CLI, no UI, no DB) - LM Studio (local OpenAI-compatible) generation + detection + judge - PlantUML rendering for SysML model visualization - 10-seed corpus (8 working + 2 holdouts) - 5 corpus runs with iteration history in reports/ - Final gate: 10/10 pass, mean 4.32/5, holdouts validated Phase 1 MVP scope and milestones documented in implementation-plan.md.
58 lines
5.0 KiB
Markdown
58 lines
5.0 KiB
Markdown
# Notes — first corpus run (2026-04-28)
|
||
|
||
Detailed report: `reports/2026-04-28-1735-run.md`
|
||
|
||
## Summary
|
||
|
||
| Seed | Diff | Blocks | Assoc | Constr | Req | A | R | I | Conf | Hallucinated refs |
|
||
|---|---|---|---|---|---|---|---|---|---|---|
|
||
| 01-aristotle | easy | 5 | 3 | 3 | 3 | 1 | 3 | 3 | 0.90 | **4** |
|
||
| 02-habit-coach | medium | 6 | 6 | 3 | 3 | 3 | 4 | 3 | 0.90 | 0 |
|
||
| 03-redline-ai | medium | 4 | 3 | 4 | 4 | 5 | 3 | 3 | 0.95 | 0 |
|
||
| 04-skillswap | medium | 5 | 5 | 3 | 3 | 3 | 5 | 3 | 0.90 | 0 |
|
||
| 05-cyclist-thing | failure-prone | **2** | 1 | 0 | 0 | 1 | 3 | 1 | **0.30** | 0 |
|
||
| 06-eventstream | hard | 5 | 4 | 3 | 4 | 4 | 4 | 3 | 0.90 | 0 |
|
||
| 07-pet-translator | failure-prone | 4 | 3 | 2 | 3 | 4 | 4 | 3 | 0.85 | 0 |
|
||
| 08-quiet-hours | medium | 6 | 5 | 3 | 3 | 4 | 3 | 4 | 0.95 | 0 |
|
||
|
||
8/8 seeds completed without errors. Total runtime: ~7 minutes.
|
||
|
||
## What worked
|
||
|
||
1. **The "under-suggest on vague seeds" rule fired correctly.** Cyclist Thing produced exactly 2 blocks (system + actor), 0 constraints, 0 requirements, confidence 0.30. The model didn't fabricate to fill the diagram — exactly the behavior we asked for in `generate.md`.
|
||
|
||
2. **Failure-prone seeds produced *the right kind of findings*.** Cyclist Thing's risks all flag the vagueness itself ("no measurable success criteria", "no use case defined", "actor too vague"). Pet Translator's #1 risk is "system's core value proposition 'translation' risks failing to meet user expectations if the output is merely an emotion classification" — which is the central feasibility critique we hoped Socrates would surface.
|
||
|
||
3. **Confidence calibration looks reasonable.** Range 0.30 → 0.95 across the corpus, correlating with seed quality / specificity. Vague → low; well-specified with explicit constraints → high.
|
||
|
||
4. **No lint warnings on any seed.** Structural integrity rules (S1–S5) hold across all 8 generated models.
|
||
|
||
5. **Hallucinated refs are concentrated on Aristotle.** 4 of 4 stripped refs were on the Aristotle run; the other 7 seeds produced zero. Worth understanding why — possibly the seed's narrative density triggers more inferred-but-non-existent element refs.
|
||
|
||
## Iteration candidates
|
||
|
||
In priority order:
|
||
|
||
1. **Constraint-vs-requirement duplication.** Across multiple seeds, Gemma renders the same content as both a constraint AND a requirement. The seed's `constraints` field is meant to flow into the `constraints` array; right now it's getting double-encoded. Fix: tighten `generate.md` with an explicit rule — "if a constraint is in `seed.constraints`, output it ONLY in the `constraints` array, never as a requirement."
|
||
|
||
2. **Aristotle hallucination cluster.** 4/4 stripped refs on this one seed. Hypothesis: the seed mentions narrative concepts (e.g., "lectures", "office hours") that the model includes as flavor in detection without modeling them as elements. Fix: tighten detection prompts — "every element id must appear verbatim in the model JSON shown to you."
|
||
|
||
3. **Property-vs-block confusion on Pet Translator.** Pet Translator's blocks `pet_vocalization` and `pet_body_language` have property names like `audio_input_stream`, `frequency_spectrum`, `pose_keypoints` — these get referenced in findings as if they were elements. Post-validate accepts them (we tolerate bare property names) but it's a UX smell. Fix: when surfacing findings, distinguish "linked block" from "linked property of block".
|
||
|
||
4. **Quiet Hours gen latency (49s).** 2× the median. Possibly because the seed has 3 detailed constraints producing a chunkier model. Acceptable but worth watching as we add more seeds.
|
||
|
||
5. **Cyclist Thing's `Cyclist Thing` block has no properties** and the only association is unlabeled (`cyclist→cyclist_thing`). The risks correctly flag this — not a bug, but evidence the model is willing to ship truly minimal structures when the seed warrants.
|
||
|
||
## Open questions
|
||
|
||
- **Should `phase0 corpus` render PNGs by default?** Currently `--skip-png` keeps it fast (~7 min); rendering all PNGs adds ~1 min total but lets us eyeball diagrams during review. Probably yes — flip the default once we're confident the harness is stable.
|
||
- **Threshold for stripping vs keeping a finding when refs are bad.** Currently we keep findings with zero refs (legitimate "missing element" findings) and drop only those whose every ref was hallucinated. This is the right default but worth re-examining once we have human scores.
|
||
- **Is the seed corpus diverse enough?** Of 8 non-holdouts: 1 consumer-edu, 1 consumer-mobile, 1 B2B legal, 1 marketplace, 1 vague, 1 technical-protocol, 1 wishful-tech, 1 productivity. Reasonable spread. The 2 holdouts (Carbon Coach, Telemetry Lite) add behavioral/social and OSS-positioning. We don't need more seeds yet.
|
||
|
||
## Next session — Day 6
|
||
|
||
1. Wire LLM-as-judge: implement `phase0 score <seedId> --judge` per the plan §6.4
|
||
2. Run judge over all 8 outputs; capture per-dimension scores in the report
|
||
3. Identify the top 1–2 failure patterns and iterate one prompt
|
||
4. Re-run corpus with the iterated prompt, compare scores against this baseline
|