Files

dtoro 4e725c0b2b MVP M8: background detection — assumptions, risks, inconsistencies

The editor now runs Socrates' three Phase-0-validated detection prompts
against the live model, persists the findings, and surfaces them in a
new FindingsPanel beside the IssuesPanel. Click any finding to focus its
linked element across rail + diagram. Re-detect after model edits to
refresh against the new state.

apps/web/lib/llm/prompts/socrates
- detect-assumptions.md / detect-risks.md / detect-inconsistencies.md
  promoted verbatim from phase-0 (Phase 0 corpus validated them 10/10).

apps/web/lib/llm/detect.ts
- Three sequential detection passes (parallel was OOM-prone on 4B local
  models — Phase 0 lesson). Each pass uses the Phase 0 JSON schema with
  jsonObjectMode fallback + chatJSON repair-retry. Fail-soft per pass:
  one busted pass returns [] rather than blowing up the whole detect.
- post-validate strips hallucinated element refs (drops findings whose
  refs ALL fail to resolve; keeps findings with zero refs since some
  inconsistencies are genuinely about absences).

apps/web/prisma/schema.prisma
- Finding table: kind / text / linkedElementIds (JSON) / confidence /
  severity / validationCode / status / modelVersion / provider / model.
- ResearchFinding table reserved for the Tavily integration that comes
  next — schema in place so we don't have to migrate again.

apps/web/lib/db/repo.ts
- listOpenFindings(projectId), replaceFindings(...) — replaceFindings
  wipes prior open findings in a transaction and writes the new set so
  re-detect doesn't accumulate stale findings.

apps/web/app/api/projects/[projectId]/findings/route.ts
- GET returns persisted open findings.
- POST runs detect, persists, returns findings + meta (provider, model,
  durationMs, strippedRefs, droppedFindings).

apps/web/components/editor/FindingsPanel.tsx
- New panel, anchored bottom-right just left of IssuesPanel. Shows
  count summary (asm / risk / inc), detect / re-detect button, list
  grouped by kind (inconsistencies first, then risks, then assumptions),
  per-finding glyph + tag + severity + confidence + linked refs.
- Click a finding row → focus its first linked element via the same
  setFocusBlockId path the rail and IssuesPanel already use.
- "stale" indicator when the model version has advanced past the one the
  findings were detected against.

EditorShell wires version + projectId through to FindingsPanel.

Smoke-tested end-to-end: 12 findings returned (5 asm / 4 risk / 3 inc),
0 hallucinated refs stripped, ~37s on local gemma-4-e4b. Sample
assumption "students are willing to engage with an AI tutor that is
programmed to refuse providing complete solutions" — specific to
Aristotle's refusal_policy, not a generic startup truism.

Deferred to follow-ups: inline rail/diagram badges from findings,
auto-detect-on-save, Tavily research, experiment modal.

2026-04-30 00:43:27 +02:00

1.9 KiB

Raw Permalink Blame History

Detect implicit assumptions in a product seed and model

You will receive a seed payload (JSON) and a generated model (JSON). Your job: surface the implicit assumptions the user is making — beliefs treated as true without explicit validation.

What is an assumption

A measurable, falsifiable belief that underpins the idea but isn't stated as a requirement or constraint. Examples:

"Students will accept a tool that refuses to answer" — assumes adoption willingness
"1.2s P50 latency is achievable on-prem with available models" — assumes technical feasibility
"Faculty will not classify Socratic prompts as academic dishonesty" — assumes institutional acceptance

What is NOT an assumption

Stated requirements (REQ-NNN entries) — those are explicit goals
Constraints — those are non-negotiables, not beliefs
Definitions of terms
Generic startup truisms ("users will want this") — too vague to be a useful assumption

Output

Return a JSON object with a single field findings — an array of assumption candidates. Each candidate:

text — the assumption restated cleanly, in one sentence, in the user's register
linkedElementIds — array of model element ids this assumption is about (block ids, requirement ids, or constraint ids — must match what's in the model)
confidence — 0.0 to 1.0, how confident you are this is genuinely an unstated assumption

Rules

Return only candidates with confidence ≥ 0.5
Cap at 8 findings
Each assumption must name a SPECIFIC, falsifiable belief — not a generic concern
Each must reference at least one real element id from the model
If the seed is sparse and you cannot surface real assumptions, return fewer (or none) rather than fabricating

Schema

{
  "findings": [
    {
      "text": "string",
      "linkedElementIds": ["string"],
      "confidence": 0.0
    }
  ]
}

Return ONLY the JSON object. No prose, no code fences.

1.9 KiB Raw Permalink Blame History