3-Stage Spec Evaluation Protocol#

Verifies whether a spec document is of implementation-ready quality through 3 stages. All stages must pass to clear the specification gate.

Stage 1: Mechanical Verification#

Checks structural and format completeness. Automatable checks.

Checklist#

Verdict#

• PASS: All items pass
• FAIL: 1 or more failures → Provide failed item list and remediation guidance
• UNVERIFIED: An item's target section is present but unparseable, or absent-vs-empty cannot be determined (e.g. "S5 found but not a list — cannot count rows"). Record which item and why.

UNVERIFIED is NOT PASS (eval G1). Any UNVERIFIED item blocks the stage exactly like FAIL until the input is made parseable or the author confirms intent. Never silently score an unparseable check as passed — that is the mechanical fail-open. Per-item legend: ✅ PASS / ❌ FAIL / ⚠️ UNVERIFIED.

Stage 2: Semantic Verification#

Checks content clarity and logical soundness. Performed by AI agents.

2a. Ambiguity Score Evaluation#

Score calculated using the 5-dimension rubric from AMBIGUITY_RUBRIC.md.

• Pass criteria: ambiguity_score ≤ 0.2
• Warning: 0.2 < score ≤ 0.3 → Recommend fixing flagged items
• Fail: score > 0.3 → Fixes required

2b. Contrarian Review#

The Contrarian agent performs the following:

1. Assumption attack: Ask "What if this is wrong?" for each immutable constraint
2. Scale challenge: Ask "What if users grow 10x?"
3. Removal test: Ask "If we remove this feature, does the core value hold?"

Output: Challenge report (responses required for each challenge)

2c. Simplifier Review#

The Simplifier agent performs the following:

1. Complexity measurement: Entity count, relationship count, constraint count
2. Removal candidate identification: Items not contributing to core value
3. Simplification proposals: Before/after complexity comparison

Output: Simplification report (accept/reject decision required)

Verdict#

• PASS: Ambiguity ≤ 0.2 AND all Contrarian Critical/Major challenges Resolved with non-empty Evidence (a Seed-spec section/quote the resolution lives in) AND Simplifier report reviewed
• FAIL: Above conditions not met

Anti-reward-hacking (eval G2): A challenge marked Resolved with empty Evidence is invalid → counts as Unresolved (fail-closed). The presence of a response ≠ resolution; a hand-wave answer that does not engage the challenge is Unresolved. Each Resolved challenge MUST cite the concrete Seed-spec location its resolution lives in.

Stage 3: Consensus Verification#

A 3-agent panel independently evaluates and reaches consensus.

Panel mode: SINGLE-MODEL (eval G3). The Analyst/PM/Architect are prompt-only personas run on one workflow agent (not distinct backend models). Report the verdict as "3/3 APPROVE (single-model panel)" — a unanimous single-model result is one correlated opinion, NOT cross-model consensus. Each panelist MUST surface at least one concrete REQUEST_CHANGES candidate or explicitly state "attempted to find an objection and found none because X" (bare APPROVE is forbidden). For genuine cross-model consensus, escalate to a human or a multi-backend reviewer outside this skill.

Panel Composition#

Evaluation Process#

1. Each panel member independently reviews the Seed spec
2. Each submits their evaluation:
   • APPROVE: Implementation-ready
   • REQUEST_CHANGES: Changes needed (specific items listed)
   • REJECT: Fundamental issues (reasons stated)
3. Results aggregated

Verdict#

• PASS: All 3 APPROVE
• CONDITIONAL: 2 APPROVE + 1 REQUEST_CHANGES → Re-evaluate after fixes
• FAIL: 2+ REQUEST_CHANGES or 1+ REJECT

Integrated Verdict#

Stage 1 PASS → Stage 2 PASS → Stage 3 PASS → ✅ Specification Gate Passed

If any stage FAILs → Fix the relevant stage and re-evaluate. Sequential execution (next stage only after previous stage passes).

Result Report Format#

## Spec Evaluation Results

### Summary
- Seed Spec: {file path}
- Evaluation Date: {YYYY-MM-DD}
- **Final Verdict: {PASS|CONDITIONAL|FAIL}**

### Stage 1: Mechanical Verification — {PASS|FAIL}
{Checklist results table}

### Stage 2: Semantic Verification — {PASS|FAIL}
- Ambiguity Score: {score} ({verdict})
- Contrarian Challenges: {N} of {M} answered
- Simplifier Proposals: {N} of {M} reviewed

### Stage 3: Consensus Verification — {PASS|CONDITIONAL|FAIL}
| Role | Verdict | Key Comments |
|------|---------|-------------|
| Analyst | {APPROVE/REQUEST_CHANGES/REJECT} | {comments} |
| PM | {verdict} | {comments} |
| Architect | {verdict} | {comments} |