EVALUATION_PROTOCOL

Verifies whether a Seed spec is of implementation-ready quality in two stages. Stage 1 is code. Stage 2 is one request to Jev (TypeSafe's judgment-only model). The authoring model never scores its own spec — it writes, Jev judges, the agent explains.

Stage 1  jev_judge.py check   (M-01..M-10, deterministic)
   │ PASS
Stage 2  jev_judge.py judge --set ambiguity,panel   (row-level noul questions, one request, counted by code)
   │ PASS
Gate PASS  →  stamp (model · date · spec_hash)  →  --lock | user confirmation  →  LOCKED

Any stage that is FAIL or UNVERIFIED stops the gate. UNVERIFIED is never a pass.

Structural and format completeness, computed by the parser (no LLM reads the checklist).

Verdict#

• PASS: all 10 items PASS
• FAIL: ≥ 1 item FAIL → the report lists each failed item with its detail; the agent writes remediation
• UNVERIFIED: a section heading exists but its table cannot be parsed (_unparseable) → blocks exactly like FAIL. Never silently score an unparseable table as passed — that is the mechanical fail-open (eval G1).

Per-item legend: ✅ PASS / ❌ FAIL / ⚠️ UNVERIFIED.

One request. Jev evaluates every question in parallel against the same state.spec, so a few dozen row questions cost about the same as one (one Seed spec ≈ 20–40 questions, under one cent). Jev judges one row at a time; the script counts. Question texts are the SoT in skills/jev-gate/questions/{ambiguity,panel}.json; the numbers below mirror config/thresholds.yaml.

Counts become rubric levels 0–4 (references/AMBIGUITY_RUBRIC.md), weighted 0.25 / 0.20 / 0.20 / 0.20 / 0.15:

ambiguity_score = Σ weight × (level / 4)

Rows answered between 0.25 and 0.75 are listed as uncertain_rows — a person reads them.

No role play: a persona asked to "approve as an architect" could not tell a clear spec from a vague one (every calibration fixture failed). Each seat is computed in code from narrow questions:

Otherwise the seat is REQUEST_CHANGES — including a seat with nothing to judge (no Must rows, no entities): an empty list is not an approval. A seat input without an answer makes the set UNVERIFIED — an empty seat is never filled with APPROVE. The panel needs 2a in the same request (--set ambiguity,panel).

blocking = REQUEST_CHANGES + REJECT

This panel is a cross-backend judgment: Jev is not the model that wrote the spec, and the seat rules are code, so the verdict is reproducible for the same answers.

Fail-closed rules (all stages)#

Integrated Verdict#

Worst result wins: UNVERIFIED > FAIL > CONDITIONAL > PASS.

Stage 1 PASS  and  2a PASS  and  2b PASS   →  ✅ Gate PASS   (lockable)
Stage 1 PASS  and  (2a or 2b CONDITIONAL), no FAIL  →  CONDITIONAL  (fix → re-run, ≤ 2 rounds)
any FAIL  →  FAIL
any UNVERIFIED  →  UNVERIFIED (not a gate outcome; fix the environment, re-run)

Stamp rule. Every verdict is bound to spec_version + spec_hash (git blob sha of the file, computed by jev_judge.py). If the file changes after a verdict, that verdict is stale: re-run both stages — they are cheap (Stage 1 is local; Stage 2 is one sub-cent request). Reusing a stale PASS is the null-as-pass form of plugins/cc-dev/rules/orchestration-graph.md §3.2. A missing stamp on an older report counts as a mismatch.

# Spec Evaluation: {project-name}

## Summary
- Seed Spec: {path} · {specVersion} · {spec_hash_short}
- Model: {model_reported} · Date: {YYYY-MM-DD} · Questions: {question_count} · Elapsed: {ms} ms
- **Final Verdict: {PASS | CONDITIONAL | FAIL | UNVERIFIED}**
- Caveats: {warnings, e.g. model differs from the calibrated one} · uncertain rows: {uncertain_rows}

## Stage 1: Mechanical — {PASS|FAIL}
| ID | Item | Result | Detail |
|----|------|--------|--------|
| M-01 | … | ✅ | |

## Stage 2a: Ambiguity — {score} ({grade}) → {verdict}
| Dimension | Level | Normalized | Weight | Rows |
|-----------|-------|-----------|--------|------|
| lexical_precision | 1 | 0.25 | 0.25 | AC-03, AC-05 |

Flagged Must/Must-Not rows: {flagged_must_rows}

## Stage 2b: Panel — blocking {n} → {verdict}
| Seat | Counted as | Basis |
|------|-----------|-------|
| analyst | REQUEST_CHANGES | AC-03 observable 0.10 |

## Remediation (written by the agent from the failed items — Jev supplies no prose)
- {row / seat}: {what to change, citing the Seed section and row id}

## Round table
| Round | Stamp | Stage 1 | 2a | 2b | Disposition |
|-------|-------|---------|----|----|-------------|
| 1 | v1.0.0 / 3ab91c | PASS | 0.10 WARNING (AC-03) | 1 | fixed S4 AC-03 wording |