dtoro/Socrates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a0566ce64c7ee285bf4165d1232dd9b0f784e4d2
Socrates/phase-0/reports/notes-run-01.md
dtoro f1c4566576 Initial commit — design docs + Phase 0 validation harness 
Sets up the Socrata project repo with:

docs/ — strategy and design documents
  - idea.md: full product vision
  - implementation-plan.md: Phase 0 + Phase 1 MVP plan
  - phase-0-validation.md: 2-week validation experiment strategy
  - phase-0-plan.md: concrete Phase 0 build plan
  - phase-0-results.md: Phase 0 gate outcome — GO for MVP
  - sysml-modeling.md: metamodel + SE discipline + validation rules
  - socrates.md: agent character, surfaces, modes, prompts, lifecycle
  - sync.md: bidirectional text↔diagram sync engineering
  - design-source/: HTML/CSS/JS handoff bundle from Claude Design

phase-0/ — validated harness (CLI, no UI, no DB)
  - LM Studio (local OpenAI-compatible) generation + detection + judge
  - PlantUML rendering for SysML model visualization
  - 10-seed corpus (8 working + 2 holdouts)
  - 5 corpus runs with iteration history in reports/
  - Final gate: 10/10 pass, mean 4.32/5, holdouts validated

Phase 1 MVP scope and milestones documented in implementation-plan.md.
2026-04-28 22:07:38 +02:00

5.0 KiB
Raw Blame History

Notes — first corpus run (2026-04-28)

Detailed report: reports/2026-04-28-1735-run.md

Summary

Seed Diff Blocks Assoc Constr Req A R I Conf Hallucinated refs
01-aristotle easy 5 3 3 3 1 3 3 0.90 4
02-habit-coach medium 6 6 3 3 3 4 3 0.90 0
03-redline-ai medium 4 3 4 4 5 3 3 0.95 0
04-skillswap medium 5 5 3 3 3 5 3 0.90 0
05-cyclist-thing failure-prone 2 1 0 0 1 3 1 0.30 0
06-eventstream hard 5 4 3 4 4 4 3 0.90 0
07-pet-translator failure-prone 4 3 2 3 4 4 3 0.85 0
08-quiet-hours medium 6 5 3 3 4 3 4 0.95 0

8/8 seeds completed without errors. Total runtime: ~7 minutes.

What worked

  1. The "under-suggest on vague seeds" rule fired correctly. Cyclist Thing produced exactly 2 blocks (system + actor), 0 constraints, 0 requirements, confidence 0.30. The model didn't fabricate to fill the diagram — exactly the behavior we asked for in generate.md.

  2. Failure-prone seeds produced the right kind of findings. Cyclist Thing's risks all flag the vagueness itself ("no measurable success criteria", "no use case defined", "actor too vague"). Pet Translator's #1 risk is "system's core value proposition 'translation' risks failing to meet user expectations if the output is merely an emotion classification" — which is the central feasibility critique we hoped Socrates would surface.

  3. Confidence calibration looks reasonable. Range 0.30 → 0.95 across the corpus, correlating with seed quality / specificity. Vague → low; well-specified with explicit constraints → high.

  4. No lint warnings on any seed. Structural integrity rules (S1–S5) hold across all 8 generated models.

  5. Hallucinated refs are concentrated on Aristotle. 4 of 4 stripped refs were on the Aristotle run; the other 7 seeds produced zero. Worth understanding why — possibly the seed's narrative density triggers more inferred-but-non-existent element refs.

Iteration candidates

In priority order:

  1. Constraint-vs-requirement duplication. Across multiple seeds, Gemma renders the same content as both a constraint AND a requirement. The seed's constraints field is meant to flow into the constraints array; right now it's getting double-encoded. Fix: tighten generate.md with an explicit rule — "if a constraint is in seed.constraints, output it ONLY in the constraints array, never as a requirement."

  2. Aristotle hallucination cluster. 4/4 stripped refs on this one seed. Hypothesis: the seed mentions narrative concepts (e.g., "lectures", "office hours") that the model includes as flavor in detection without modeling them as elements. Fix: tighten detection prompts — "every element id must appear verbatim in the model JSON shown to you."

  3. Property-vs-block confusion on Pet Translator. Pet Translator's blocks pet_vocalization and pet_body_language have property names like audio_input_stream, frequency_spectrum, pose_keypoints — these get referenced in findings as if they were elements. Post-validate accepts them (we tolerate bare property names) but it's a UX smell. Fix: when surfacing findings, distinguish "linked block" from "linked property of block".

  4. Quiet Hours gen latency (49s). 2× the median. Possibly because the seed has 3 detailed constraints producing a chunkier model. Acceptable but worth watching as we add more seeds.

  5. Cyclist Thing's Cyclist Thing block has no properties and the only association is unlabeled (cyclist→cyclist_thing). The risks correctly flag this — not a bug, but evidence the model is willing to ship truly minimal structures when the seed warrants.

Open questions

  • Should phase0 corpus render PNGs by default? Currently --skip-png keeps it fast (~7 min); rendering all PNGs adds ~1 min total but lets us eyeball diagrams during review. Probably yes — flip the default once we're confident the harness is stable.
  • Threshold for stripping vs keeping a finding when refs are bad. Currently we keep findings with zero refs (legitimate "missing element" findings) and drop only those whose every ref was hallucinated. This is the right default but worth re-examining once we have human scores.
  • Is the seed corpus diverse enough? Of 8 non-holdouts: 1 consumer-edu, 1 consumer-mobile, 1 B2B legal, 1 marketplace, 1 vague, 1 technical-protocol, 1 wishful-tech, 1 productivity. Reasonable spread. The 2 holdouts (Carbon Coach, Telemetry Lite) add behavioral/social and OSS-positioning. We don't need more seeds yet.

Next session — Day 6

  1. Wire LLM-as-judge: implement phase0 score <seedId> --judge per the plan §6.4
  2. Run judge over all 8 outputs; capture per-dimension scores in the report
  3. Identify the top 1–2 failure patterns and iterate one prompt
  4. Re-run corpus with the iterated prompt, compare scores against this baseline
Reference in New Issue View Git Blame Copy Permalink