MVP M8: background detection — assumptions, risks, inconsistencies
The editor now runs Socrates' three Phase-0-validated detection prompts against the live model, persists the findings, and surfaces them in a new FindingsPanel beside the IssuesPanel. Click any finding to focus its linked element across rail + diagram. Re-detect after model edits to refresh against the new state. apps/web/lib/llm/prompts/socrates - detect-assumptions.md / detect-risks.md / detect-inconsistencies.md promoted verbatim from phase-0 (Phase 0 corpus validated them 10/10). apps/web/lib/llm/detect.ts - Three sequential detection passes (parallel was OOM-prone on 4B local models — Phase 0 lesson). Each pass uses the Phase 0 JSON schema with jsonObjectMode fallback + chatJSON repair-retry. Fail-soft per pass: one busted pass returns [] rather than blowing up the whole detect. - post-validate strips hallucinated element refs (drops findings whose refs ALL fail to resolve; keeps findings with zero refs since some inconsistencies are genuinely about absences). apps/web/prisma/schema.prisma - Finding table: kind / text / linkedElementIds (JSON) / confidence / severity / validationCode / status / modelVersion / provider / model. - ResearchFinding table reserved for the Tavily integration that comes next — schema in place so we don't have to migrate again. apps/web/lib/db/repo.ts - listOpenFindings(projectId), replaceFindings(...) — replaceFindings wipes prior open findings in a transaction and writes the new set so re-detect doesn't accumulate stale findings. apps/web/app/api/projects/[projectId]/findings/route.ts - GET returns persisted open findings. - POST runs detect, persists, returns findings + meta (provider, model, durationMs, strippedRefs, droppedFindings). apps/web/components/editor/FindingsPanel.tsx - New panel, anchored bottom-right just left of IssuesPanel. Shows count summary (asm / risk / inc), detect / re-detect button, list grouped by kind (inconsistencies first, then risks, then assumptions), per-finding glyph + tag + severity + confidence + linked refs. - Click a finding row → focus its first linked element via the same setFocusBlockId path the rail and IssuesPanel already use. - "stale" indicator when the model version has advanced past the one the findings were detected against. EditorShell wires version + projectId through to FindingsPanel. Smoke-tested end-to-end: 12 findings returned (5 asm / 4 risk / 3 inc), 0 hallucinated refs stripped, ~37s on local gemma-4-e4b. Sample assumption "students are willing to engage with an AI tutor that is programmed to refuse providing complete solutions" — specific to Aristotle's refusal_policy, not a generic startup truism. Deferred to follow-ups: inline rail/diagram badges from findings, auto-detect-on-save, Tavily research, experiment modal.
This commit is contained in:
50
apps/web/lib/llm/prompts/socrates/detect-assumptions.md
Normal file
50
apps/web/lib/llm/prompts/socrates/detect-assumptions.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Detect implicit assumptions in a product seed and model
|
||||
|
||||
You will receive a seed payload (JSON) and a generated model (JSON). Your job: surface the **implicit assumptions** the user is making — beliefs treated as true without explicit validation.
|
||||
|
||||
## What is an assumption
|
||||
|
||||
A measurable, falsifiable belief that underpins the idea but isn't stated as a requirement or constraint. Examples:
|
||||
|
||||
- "Students will accept a tool that refuses to answer" — assumes adoption willingness
|
||||
- "1.2s P50 latency is achievable on-prem with available models" — assumes technical feasibility
|
||||
- "Faculty will not classify Socratic prompts as academic dishonesty" — assumes institutional acceptance
|
||||
|
||||
## What is NOT an assumption
|
||||
|
||||
- Stated requirements (REQ-NNN entries) — those are explicit goals
|
||||
- Constraints — those are non-negotiables, not beliefs
|
||||
- Definitions of terms
|
||||
- Generic startup truisms ("users will want this") — too vague to be a useful assumption
|
||||
|
||||
## Output
|
||||
|
||||
Return a JSON object with a single field `findings` — an array of assumption candidates. Each candidate:
|
||||
|
||||
- `text` — the assumption restated cleanly, in one sentence, in the user's register
|
||||
- `linkedElementIds` — array of model element ids this assumption is about (block ids, requirement ids, or constraint ids — must match what's in the model)
|
||||
- `confidence` — 0.0 to 1.0, how confident you are this is genuinely an unstated assumption
|
||||
|
||||
## Rules
|
||||
|
||||
- Return only candidates with `confidence ≥ 0.5`
|
||||
- Cap at 8 findings
|
||||
- Each assumption must name a SPECIFIC, falsifiable belief — not a generic concern
|
||||
- Each must reference at least one real element id from the model
|
||||
- If the seed is sparse and you cannot surface real assumptions, return fewer (or none) rather than fabricating
|
||||
|
||||
## Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"findings": [
|
||||
{
|
||||
"text": "string",
|
||||
"linkedElementIds": ["string"],
|
||||
"confidence": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Return ONLY the JSON object. No prose, no code fences.
|
||||
Reference in New Issue
Block a user