# Notes — first corpus run (2026-04-28)

Detailed report: `reports/2026-04-28-1735-run.md`

## Summary

| Seed | Diff | Blocks | Assoc | Constr | Req | A | R | I | Conf | Hallucinated refs |
|---|---|---|---|---|---|---|---|---|---|---|
| 01-aristotle | easy | 5 | 3 | 3 | 3 | 1 | 3 | 3 | 0.90 | **4** |
| 02-habit-coach | medium | 6 | 6 | 3 | 3 | 3 | 4 | 3 | 0.90 | 0 |
| 03-redline-ai | medium | 4 | 3 | 4 | 4 | 5 | 3 | 3 | 0.95 | 0 |
| 04-skillswap | medium | 5 | 5 | 3 | 3 | 3 | 5 | 3 | 0.90 | 0 |
| 05-cyclist-thing | failure-prone | **2** | 1 | 0 | 0 | 1 | 3 | 1 | **0.30** | 0 |
| 06-eventstream | hard | 5 | 4 | 3 | 4 | 4 | 4 | 3 | 0.90 | 0 |
| 07-pet-translator | failure-prone | 4 | 3 | 2 | 3 | 4 | 4 | 3 | 0.85 | 0 |
| 08-quiet-hours | medium | 6 | 5 | 3 | 3 | 4 | 3 | 4 | 0.95 | 0 |

8/8 seeds completed without errors. Total runtime: ~7 minutes.

## What worked

1. **The "under-suggest on vague seeds" rule fired correctly.** Cyclist Thing produced exactly 2 blocks (system + actor), 0 constraints, 0 requirements, confidence 0.30. The model didn't fabricate to fill the diagram — exactly the behavior we asked for in `generate.md`.

2. **Failure-prone seeds produced *the right kind of findings*.** Cyclist Thing's risks all flag the vagueness itself ("no measurable success criteria", "no use case defined", "actor too vague"). Pet Translator's #1 risk is "system's core value proposition 'translation' risks failing to meet user expectations if the output is merely an emotion classification" — which is the central feasibility critique we hoped Socrates would surface.

3. **Confidence calibration looks reasonable.** Range 0.30 → 0.95 across the corpus, correlating with seed quality / specificity. Vague → low; well-specified with explicit constraints → high.

4. **No lint warnings on any seed.** Structural integrity rules (S1–S5) hold across all 8 generated models.

5. **Hallucinated refs are concentrated on Aristotle.** 4 of 4 stripped refs were on the Aristotle run; the other 7 seeds produced zero. Worth understanding why — possibly the seed's narrative density triggers more inferred-but-non-existent element refs.

## Iteration candidates

In priority order:

1. **Constraint-vs-requirement duplication.** Across multiple seeds, Gemma renders the same content as both a constraint AND a requirement. The seed's `constraints` field is meant to flow into the `constraints` array; right now it's getting double-encoded. Fix: tighten `generate.md` with an explicit rule — "if a constraint is in `seed.constraints`, output it ONLY in the `constraints` array, never as a requirement."

2. **Aristotle hallucination cluster.** 4/4 stripped refs on this one seed. Hypothesis: the seed mentions narrative concepts (e.g., "lectures", "office hours") that the model includes as flavor in detection without modeling them as elements. Fix: tighten detection prompts — "every element id must appear verbatim in the model JSON shown to you."

3. **Property-vs-block confusion on Pet Translator.** Pet Translator's blocks `pet_vocalization` and `pet_body_language` have property names like `audio_input_stream`, `frequency_spectrum`, `pose_keypoints` — these get referenced in findings as if they were elements. Post-validate accepts them (we tolerate bare property names) but it's a UX smell. Fix: when surfacing findings, distinguish "linked block" from "linked property of block".

4. **Quiet Hours gen latency (49s).** 2× the median. Possibly because the seed has 3 detailed constraints producing a chunkier model. Acceptable but worth watching as we add more seeds.

5. **Cyclist Thing's `Cyclist Thing` block has no properties** and the only association is unlabeled (`cyclist→cyclist_thing`). The risks correctly flag this — not a bug, but evidence the model is willing to ship truly minimal structures when the seed warrants.

## Open questions

- **Should `phase0 corpus` render PNGs by default?** Currently `--skip-png` keeps it fast (~7 min); rendering all PNGs adds ~1 min total but lets us eyeball diagrams during review. Probably yes — flip the default once we're confident the harness is stable.
- **Threshold for stripping vs keeping a finding when refs are bad.** Currently we keep findings with zero refs (legitimate "missing element" findings) and drop only those whose every ref was hallucinated. This is the right default but worth re-examining once we have human scores.
- **Is the seed corpus diverse enough?** Of 8 non-holdouts: 1 consumer-edu, 1 consumer-mobile, 1 B2B legal, 1 marketplace, 1 vague, 1 technical-protocol, 1 wishful-tech, 1 productivity. Reasonable spread. The 2 holdouts (Carbon Coach, Telemetry Lite) add behavioral/social and OSS-positioning. We don't need more seeds yet.

## Next session — Day 6

1. Wire LLM-as-judge: implement `phase0 score <seedId> --judge` per the plan §6.4
2. Run judge over all 8 outputs; capture per-dimension scores in the report
3. Identify the top 1–2 failure patterns and iterate one prompt
4. Re-run corpus with the iterated prompt, compare scores against this baseline