MVP M8: background detection — assumptions, risks, inconsistencies

The editor now runs Socrates' three Phase-0-validated detection prompts against the live model, persists the findings, and surfaces them in a new FindingsPanel beside the IssuesPanel. Click any finding to focus its linked element across rail + diagram. Re-detect after model edits to refresh against the new state. apps/web/lib/llm/prompts/socrates - detect-assumptions.md / detect-risks.md / detect-inconsistencies.md promoted verbatim from phase-0 (Phase 0 corpus validated them 10/10). apps/web/lib/llm/detect.ts - Three sequential detection passes (parallel was OOM-prone on 4B local models — Phase 0 lesson). Each pass uses the Phase 0 JSON schema with jsonObjectMode fallback + chatJSON repair-retry. Fail-soft per pass: one busted pass returns [] rather than blowing up the whole detect. - post-validate strips hallucinated element refs (drops findings whose refs ALL fail to resolve; keeps findings with zero refs since some inconsistencies are genuinely about absences). apps/web/prisma/schema.prisma - Finding table: kind / text / linkedElementIds (JSON) / confidence / severity / validationCode / status / modelVersion / provider / model. - ResearchFinding table reserved for the Tavily integration that comes next — schema in place so we don't have to migrate again. apps/web/lib/db/repo.ts - listOpenFindings(projectId), replaceFindings(...) — replaceFindings wipes prior open findings in a transaction and writes the new set so re-detect doesn't accumulate stale findings. apps/web/app/api/projects/[projectId]/findings/route.ts - GET returns persisted open findings. - POST runs detect, persists, returns findings + meta (provider, model, durationMs, strippedRefs, droppedFindings). apps/web/components/editor/FindingsPanel.tsx - New panel, anchored bottom-right just left of IssuesPanel. Shows count summary (asm / risk / inc), detect / re-detect button, list grouped by kind (inconsistencies first, then risks, then assumptions), per-finding glyph + tag + severity + confidence + linked refs. - Click a finding row → focus its first linked element via the same setFocusBlockId path the rail and IssuesPanel already use. - "stale" indicator when the model version has advanced past the one the findings were detected against. EditorShell wires version + projectId through to FindingsPanel. Smoke-tested end-to-end: 12 findings returned (5 asm / 4 risk / 3 inc), 0 hallucinated refs stripped, ~37s on local gemma-4-e4b. Sample assumption "students are willing to engage with an AI tutor that is programmed to refuse providing complete solutions" — specific to Aristotle's refusal_policy, not a generic startup truism. Deferred to follow-ups: inline rail/diagram badges from findings, auto-detect-on-save, Tavily research, experiment modal.
2026-04-30 00:43:27 +02:00
parent 4b8e3f04ee
commit 4e725c0b2b
10 changed files with 1010 additions and 1 deletions
--- a/apps/web/lib/db/repo.ts
+++ b/apps/web/lib/db/repo.ts
@@ -148,6 +148,114 @@ async function ensureSeeded(projectId: string): Promise<{ projectId: string }> {
  return { projectId };
 }

+// ─── Findings (M8) ───────────────────────────────────────────────────────
+
+import type { Finding as DetectedFinding } from "../llm/detect";
+
+export interface StoredFinding {
+  id: string;
+  kind: string;
+  text: string;
+  linkedElementIds: string[];
+  confidence: number;
+  severity: string | null;
+  validationCode: string | null;
+  status: string;
+  modelVersion: number;
+  provider: string | null;
+  llmModel: string | null;
+  createdAt: Date;
+}
+
+export async function listOpenFindings(projectId: string): Promise<StoredFinding[]> {
+  const rows = await prisma.finding.findMany({
+    where: { projectId, status: "open" },
+    orderBy: [{ kind: "asc" }, { createdAt: "desc" }],
+  });
+  return rows.map(rowToFinding);
+}
+
+export async function replaceFindings(
+  projectId: string,
+  findings: DetectedFinding[],
+  modelVersion: number,
+  provider: string,
+  llmModel: string
+): Promise<StoredFinding[]> {
+  return prisma.$transaction(async tx => {
+    // Wipe previous open findings — re-running detection supersedes them.
+    await tx.finding.deleteMany({ where: { projectId, status: "open" } });
+    if (findings.length === 0) return [];
+
+    const created = await Promise.all(
+      findings.map(f =>
+        tx.finding.create({
+          data: {
+            projectId,
+            kind: f.kind,
+            text: f.text,
+            linkedElementIds: JSON.stringify(f.linkedElementIds),
+            confidence: f.confidence,
+            severity: f.severity ?? null,
+            validationCode: f.validationCode ?? null,
+            status: "open",
+            modelVersion,
+            provider,
+            llmModel,
+          },
+        })
+      )
+    );
+    return created.map(rowToFinding);
+  });
+}
+
+export async function dismissFinding(findingId: string): Promise<void> {
+  await prisma.finding.update({
+    where: { id: findingId },
+    data: { status: "dismissed" },
+  });
+}
+
+interface FindingRow {
+  id: string;
+  kind: string;
+  text: string;
+  linkedElementIds: string;
+  confidence: number;
+  severity: string | null;
+  validationCode: string | null;
+  status: string;
+  modelVersion: number;
+  provider: string | null;
+  llmModel: string | null;
+  createdAt: Date;
+}
+
+function rowToFinding(row: FindingRow): StoredFinding {
+  let linked: string[] = [];
+  try {
+    const parsed = JSON.parse(row.linkedElementIds);
+    if (Array.isArray(parsed)) linked = parsed.filter((x): x is string => typeof x === "string");
+  } catch {
+    linked = [];
+  }
+  return {
+    id: row.id,
+    kind: row.kind,
+    text: row.text,
+    linkedElementIds: linked,
+    confidence: row.confidence,
+    severity: row.severity,
+    validationCode: row.validationCode,
+    status: row.status,
+    modelVersion: row.modelVersion,
+    provider: row.provider,
+    llmModel: row.llmModel,
+    createdAt: row.createdAt,
+  };
+}
+
 // ─── Socrates threads ────────────────────────────────────────────────────

 export async function getActiveThread(projectId: string): Promise<{ id: string; title: string | null; messages: Array<{ id: string; role: string; content: string; createdAt: Date }> } | null> {
--- a/apps/web/lib/llm/detect.ts
+++ b/apps/web/lib/llm/detect.ts
@@ -0,0 +1,301 @@
+// Background detection — runs three Phase-0-validated detection prompts
+// against the current model and returns structured findings.
+//
+// Sequential not parallel: small local models share a KV-cache budget; three
+// concurrent calls can OOM. Each call is independent so total wall time
+// roughly equals 3× per-call time, but stability is much higher.
+//
+// Post-validation strips hallucinated element ids — the same defensive step
+// that rescued ~5% of phase-0 detection runs.
+
+import "server-only";
+import { defaultGateway, chatJSON, type Message } from "./gateway";
+import { loadPrompt } from "./prompts";
+import type { SysMLModel } from "../sysml/model";
+
+// ─── Output types ────────────────────────────────────────────────────────
+
+export type FindingKind = "assumption" | "risk" | "inconsistency";
+
+export interface Finding {
+  kind: FindingKind;
+  text: string;
+  linkedElementIds: string[];
+  confidence: number;
+  severity?: "low" | "medium" | "high";
+  validationCode?: string;
+}
+
+export interface DetectResult {
+  findings: Finding[];
+  inputTokens: number;
+  outputTokens: number;
+  durationMs: number;
+  strippedRefs: number;
+  droppedFindings: number;
+  provider: string;
+  model: string;
+}
+
+// ─── Per-pass JSON schemas (oneOf-by-shape; small models cope) ──────────
+
+const baseProps = {
+  text: { type: "string", minLength: 1 },
+  linkedElementIds: { type: "array", items: { type: "string" } },
+  confidence: { type: "number", minimum: 0, maximum: 1 },
+};
+
+const assumptionsJsonSchema = {
+  type: "object",
+  additionalProperties: false,
+  required: ["findings"],
+  properties: {
+    findings: {
+      type: "array",
+      items: {
+        type: "object",
+        additionalProperties: false,
+        required: ["text", "confidence"],
+        properties: baseProps,
+      },
+    },
+  },
+} as const;
+
+const risksJsonSchema = {
+  type: "object",
+  additionalProperties: false,
+  required: ["findings"],
+  properties: {
+    findings: {
+      type: "array",
+      items: {
+        type: "object",
+        additionalProperties: false,
+        required: ["text", "confidence", "severity"],
+        properties: {
+          ...baseProps,
+          severity: { type: "string", enum: ["low", "medium", "high"] },
+        },
+      },
+    },
+  },
+} as const;
+
+const inconsistenciesJsonSchema = {
+  type: "object",
+  additionalProperties: false,
+  required: ["findings"],
+  properties: {
+    findings: {
+      type: "array",
+      items: {
+        type: "object",
+        additionalProperties: false,
+        required: ["text", "confidence"],
+        properties: {
+          ...baseProps,
+          validationCode: { type: "string" },
+        },
+      },
+    },
+  },
+} as const;
+
+interface RawFinding {
+  text?: string;
+  linkedElementIds?: string[];
+  confidence?: number;
+  severity?: "low" | "medium" | "high";
+  validationCode?: string;
+}
+
+// ─── Public API ──────────────────────────────────────────────────────────
+
+export async function detectFindings(model: SysMLModel): Promise<DetectResult> {
+  const gateway = defaultGateway();
+  const start = Date.now();
+  let totalIn = 0;
+  let totalOut = 0;
+
+  const userPayload = buildUserPayload(model);
+
+  const assumptions = await runPass(gateway, "detect-assumptions", userPayload, assumptionsJsonSchema);
+  totalIn += assumptions.inputTokens;
+  totalOut += assumptions.outputTokens;
+
+  const risks = await runPass(gateway, "detect-risks", userPayload, risksJsonSchema);
+  totalIn += risks.inputTokens;
+  totalOut += risks.outputTokens;
+
+  const incons = await runPass(gateway, "detect-inconsistencies", userPayload, inconsistenciesJsonSchema);
+  totalIn += incons.inputTokens;
+  totalOut += incons.outputTokens;
+
+  const raw: Finding[] = [
+    ...assumptions.findings.map((f): Finding => ({
+      kind: "assumption",
+      text: f.text ?? "",
+      linkedElementIds: f.linkedElementIds ?? [],
+      confidence: clamp01(f.confidence ?? 0.5),
+    })),
+    ...risks.findings.map((f): Finding => ({
+      kind: "risk",
+      text: f.text ?? "",
+      linkedElementIds: f.linkedElementIds ?? [],
+      confidence: clamp01(f.confidence ?? 0.5),
+      severity: f.severity ?? "medium",
+    })),
+    ...incons.findings.map((f): Finding => ({
+      kind: "inconsistency",
+      text: f.text ?? "",
+      linkedElementIds: f.linkedElementIds ?? [],
+      confidence: clamp01(f.confidence ?? 0.6),
+      validationCode: f.validationCode,
+    })),
+  ];
+
+  const validated = postValidate(raw, model);
+
+  return {
+    findings: validated.findings,
+    inputTokens: totalIn,
+    outputTokens: totalOut,
+    durationMs: Date.now() - start,
+    strippedRefs: validated.strippedRefs,
+    droppedFindings: validated.droppedFindings,
+    provider: gateway.provider,
+    model: gateway.model,
+  };
+}
+
+// ─── Per-pass runner ─────────────────────────────────────────────────────
+
+interface PassResult {
+  findings: RawFinding[];
+  inputTokens: number;
+  outputTokens: number;
+}
+
+async function runPass(
+  gateway: ReturnType<typeof defaultGateway>,
+  promptName: "detect-assumptions" | "detect-risks" | "detect-inconsistencies",
+  userPayload: string,
+  jsonSchema: object
+): Promise<PassResult> {
+  const prompt = loadPrompt(`socrates/${promptName}.md`);
+  const messages: Message[] = [
+    { role: "system", content: prompt },
+    { role: "user", content: userPayload },
+  ];
+
+  try {
+    const { value, result } = await chatJSON<{ findings?: RawFinding[] }>(gateway, messages, {
+      temperature: 0.2,
+      maxTokens: 768,
+      jsonSchema: { name: "findings", schema: jsonSchema as Record<string, unknown> },
+      jsonObjectMode: true,
+      maxRepairs: 2,
+    });
+    return {
+      findings: Array.isArray(value?.findings) ? value.findings : [],
+      inputTokens: result.inputTokens,
+      outputTokens: result.outputTokens,
+    };
+  } catch (err) {
+    // Fail-soft: a busted detection pass shouldn't take down the whole detect.
+    console.error(`[detect] ${promptName} pass failed:`, (err as Error).message);
+    return { findings: [], inputTokens: 0, outputTokens: 0 };
+  }
+}
+
+// ─── Trimmed payload (small models need tight context) ───────────────────
+
+function buildUserPayload(model: SysMLModel): string {
+  const compact = {
+    blocks: model.blocks.map(b => ({
+      id: b.id,
+      label: b.label,
+      kind: b.kind,
+      properties: b.properties.map(p => p.name),
+    })),
+    associations: model.associations.map(a => ({
+      id: a.id,
+      from: a.fromBlockId,
+      to: a.toBlockId,
+      label: a.label,
+      kind: a.kind,
+    })),
+    constraints: model.constraints.map(c => ({
+      id: c.id,
+      label: c.label,
+      appliesTo: c.appliesTo,
+    })),
+    requirements: model.requirements.map(r => ({
+      id: r.id,
+      tag: r.tag,
+      text: r.text,
+      satisfiedBy: r.relations
+        .filter(rel => rel.kind === "satisfy")
+        .map(rel => (rel as { kind: "satisfy"; blockId: string }).blockId),
+    })),
+  };
+  return `Model:\n\`\`\`json\n${JSON.stringify(compact, null, 2)}\n\`\`\`\n`;
+}
+
+// ─── Post-validation: strip hallucinated element refs ────────────────────
+
+interface ValidateOut {
+  findings: Finding[];
+  strippedRefs: number;
+  droppedFindings: number;
+}
+
+function postValidate(findings: Finding[], model: SysMLModel): ValidateOut {
+  const validIds = collectValidIds(model);
+  let stripped = 0;
+  let dropped = 0;
+  const out: Finding[] = [];
+
+  for (const f of findings) {
+    if (!f.text || f.text.trim().length === 0) {
+      dropped++;
+      continue;
+    }
+    const goodRefs: string[] = [];
+    for (const ref of f.linkedElementIds) {
+      if (validIds.has(ref)) goodRefs.push(ref);
+      else stripped++;
+    }
+    // Drop only if it had refs and ALL of them were hallucinated.
+    if (f.linkedElementIds.length > 0 && goodRefs.length === 0) {
+      dropped++;
+      continue;
+    }
+    out.push({ ...f, linkedElementIds: goodRefs });
+  }
+  return { findings: out, strippedRefs: stripped, droppedFindings: dropped };
+}
+
+function collectValidIds(model: SysMLModel): Set<string> {
+  const ids = new Set<string>();
+  for (const b of model.blocks) {
+    ids.add(b.id);
+    for (const p of b.properties) {
+      ids.add(`${b.id}.${p.id}`);
+      ids.add(p.name); // tolerate bare property names — common LLM pattern
+    }
+  }
+  for (const a of model.associations) ids.add(a.id);
+  for (const c of model.constraints) ids.add(c.id);
+  for (const r of model.requirements) {
+    ids.add(r.id);
+    ids.add(r.tag);
+  }
+  return ids;
+}
+
+function clamp01(n: number): number {
+  if (Number.isNaN(n)) return 0;
+  return Math.max(0, Math.min(1, n));
+}
--- a/apps/web/lib/llm/prompts/socrates/detect-assumptions.md
+++ b/apps/web/lib/llm/prompts/socrates/detect-assumptions.md
@@ -0,0 +1,50 @@
+# Detect implicit assumptions in a product seed and model
+
+You will receive a seed payload (JSON) and a generated model (JSON). Your job: surface the **implicit assumptions** the user is making — beliefs treated as true without explicit validation.
+
+## What is an assumption
+
+A measurable, falsifiable belief that underpins the idea but isn't stated as a requirement or constraint. Examples:
+
+- "Students will accept a tool that refuses to answer" — assumes adoption willingness
+- "1.2s P50 latency is achievable on-prem with available models" — assumes technical feasibility
+- "Faculty will not classify Socratic prompts as academic dishonesty" — assumes institutional acceptance
+
+## What is NOT an assumption
+
+- Stated requirements (REQ-NNN entries) — those are explicit goals
+- Constraints — those are non-negotiables, not beliefs
+- Definitions of terms
+- Generic startup truisms ("users will want this") — too vague to be a useful assumption
+
+## Output
+
+Return a JSON object with a single field `findings` — an array of assumption candidates. Each candidate:
+
+- `text` — the assumption restated cleanly, in one sentence, in the user's register
+- `linkedElementIds` — array of model element ids this assumption is about (block ids, requirement ids, or constraint ids — must match what's in the model)
+- `confidence` — 0.0 to 1.0, how confident you are this is genuinely an unstated assumption
+
+## Rules
+
+- Return only candidates with `confidence ≥ 0.5`
+- Cap at 8 findings
+- Each assumption must name a SPECIFIC, falsifiable belief — not a generic concern
+- Each must reference at least one real element id from the model
+- If the seed is sparse and you cannot surface real assumptions, return fewer (or none) rather than fabricating
+
+## Schema
+
+```json
+{
+  "findings": [
+    {
+      "text": "string",
+      "linkedElementIds": ["string"],
+      "confidence": 0.0
+    }
+  ]
+}
+```
+
+Return ONLY the JSON object. No prose, no code fences.
--- a/apps/web/lib/llm/prompts/socrates/detect-inconsistencies.md
+++ b/apps/web/lib/llm/prompts/socrates/detect-inconsistencies.md
@@ -0,0 +1,45 @@
+# Detect inconsistencies in a generated model
+
+You will receive a seed payload (JSON) and a generated model (JSON). Your job: find **inconsistencies** — internal contradictions or structural problems in the model.
+
+## Categories
+
+- **Internal contradictions** — two requirements that can't both hold simultaneously; a block whose properties contradict its kind; a constraint already violated by some property value
+- **Reference issues** — an association whose endpoints don't make semantic sense (e.g., actor → constraint, or system → external actor with the wrong direction)
+- **Over-broad claims** — a requirement that promises more than the system can deliver based on the blocks present
+- **Missing satisfiers** — a requirement with no plausible block to satisfy it
+- **Unused elements** — a block with no associations and no requirement satisfaction (may be dead)
+
+## Output
+
+Return a JSON object with a single field `findings`. Each candidate:
+
+- `text` — the inconsistency stated clearly in one sentence
+- `linkedElementIds` — array of element ids involved
+- `confidence` — 0.0 to 1.0
+- `validationCode` — optional. If the issue matches a structural rule, include the code: `M2` (cyclic composition), `T1` (untraced requirement), `T2` (unused element), `S1` (dangling association endpoint). Otherwise omit.
+
+## Rules
+
+- Return only candidates with `confidence ≥ 0.6` — for inconsistencies, false positives are worse than misses
+- Cap at 6 findings
+- An inconsistency must point to a SPECIFIC contradiction or structural defect, not a stylistic preference
+- "This block has too many properties" is NOT an inconsistency
+- "Requirement REQ-002 forbids what association A2 enables" IS an inconsistency
+
+## Schema
+
+```json
+{
+  "findings": [
+    {
+      "text": "string",
+      "linkedElementIds": ["string"],
+      "confidence": 0.0,
+      "validationCode": "string (optional)"
+    }
+  ]
+}
+```
+
+Return ONLY the JSON object. No prose, no code fences.
--- a/apps/web/lib/llm/prompts/socrates/detect-risks.md
+++ b/apps/web/lib/llm/prompts/socrates/detect-risks.md
@@ -0,0 +1,46 @@
+# Detect risks in a product seed and model
+
+You will receive a seed payload (JSON) and a generated model (JSON). Your job: surface **risks** — specific failure modes that could prevent the system from working as intended.
+
+## Risk categories
+
+- **Technical** — feasibility, performance, scaling
+- **Market** — adoption, competitive dynamics, distribution
+- **Execution** — team capability, timing, dependencies
+- **Regulatory** — compliance, legal, privacy
+- **External** — third-party reliance, geopolitical, supply
+
+## Output
+
+Return a JSON object with a single field `findings` — an array of risk candidates. Each candidate:
+
+- `text` — the risk restated as a specific failure mode in one sentence
+- `linkedElementIds` — array of model element ids this risk implicates
+- `severity` — `"low"`, `"medium"`, or `"high"` (impact-if-it-happens, NOT probability)
+- `confidence` — 0.0 to 1.0, how confident you are this is a real risk worth tracking
+
+## Rules
+
+- Return only candidates with `confidence ≥ 0.5`
+- Cap at 6 findings
+- A risk must name a SPECIFIC failure mode tied to SPECIFIC element(s). "Won't work" is not a risk; "Latency target unachievable on consumer-grade hardware given 7B-param inference" is.
+- Severity reflects what happens IF the risk materializes, not how likely it is.
+- Each finding must reference at least one real element id from the model
+- For vague seeds with weak models, return fewer findings rather than fabricated ones
+
+## Schema
+
+```json
+{
+  "findings": [
+    {
+      "text": "string",
+      "linkedElementIds": ["string"],
+      "severity": "low" | "medium" | "high",
+      "confidence": 0.0
+    }
+  ]
+}
+```
+
+Return ONLY the JSON object. No prose, no code fences.