AMBIGUITY_RUBRIC
5-Dimension Ambiguity Rubric — the level definitions that jev_judge.py fills in by counting row-level Jev judgments from questions/ambiguity.json (lexical precision, reference clarity, completeness...
A 5-dimension rubric for quantifying ambiguity in a Seed spec.
Final score = weighted mean of level / 4 over the five dimensions (0.0 = perfectly clear, 1.0 = completely ambiguous).
The rubric is judged by Jev and counted by code, never scored by the authoring model. Jev cannot
count reliably, so it is never asked "how many vague terms are there?". Instead every row gets its own
narrow noul question (skills/jev-gate/questions/ambiguity.json
→ rows), and jev_judge.py counts
the flagged rows (probability > 0.5) and maps each count onto the levels below:
| Dimension | Row question (one per row) | Counted rows → level |
|---|---|---|
| Lexical precision |
vague::AC-xx
— does this criterion rely on a vague word instead of a value? ·
c_vague::C-xx
for constraints
|
vague criteria + vague constraints, over all S4 + S2 rows |
| Reference clarity | unclear_ref::AC-xx — does it point at something (이것·해당·기존 방식) without naming it? |
flagged S4 rows |
| Completeness |
covered::{Entity}
— does any criterion cover this entity? ·
has_failure_case
(one question)
|
1 if no failure / invalid-input / limit case + uncovered entities, max 4 |
| Consistency | conflict::C-xx::AC-yy — do this constraint and this Must/Must-Not contradict? |
2 × contradicting pairs, max 4 |
| Testability |
observable::AC-xx
(Must/Must-Not only) — does it state a trigger and an observable result?
|
rows without both |
Count → level: 0 → 0 · 1–2 → 1 · 3–5 → 2 · 6+ → 3 · ≥ 80 % of the rows → 4 (testability: ≥ 50 % → 3). The level tables below are what the counts mean; the JSON is what is sent and the script is what counts. Change them together.
Pass Criteria#
| Grade | Score Range | Verdict |
|---|---|---|
| CLEAR | 0.0 ~ 0.2 | PASS — Implementation-ready |
| WARNING | 0.2 ~ 0.3 | CONDITIONAL — Fix the weakest dimensions, re-judge (≤ 2 rounds) |
| AMBIGUOUS | 0.3 ~ 0.5 | FAIL — Major ambiguities must be resolved |
| UNCLEAR | 0.5 ~ 1.0 | FAIL — Spec rewrite required |
Row rule. A flagged Must/Must-Not row (vague, unclear reference, or not observable), any
contradiction, an entity no criterion covers, or a spec without any failure / invalid-input / limit case
makes the verdict at least CONDITIONAL, however low the weighted score — fix that row and re-judge. The report lists the rows (flagged_must_rows), so the fix is concrete.
If any row question has no answer, the score is UNVERIFIED — not 0.0, not a pass.
Level ↔ value#
| Level (counted) | Value | Meaning |
|---|---|---|
| 0 | 0.00 | none of the defect |
| 1 | 0.25 | 1–2 occurrences |
| 2 | 0.50 | 3–5 occurrences / partial |
| 3 | 0.75 | many |
| 4 | 1.00 | pervasive |
Dimension 1: Lexical Precision — Weight 0.25#
Whether ambiguous words/expressions have been replaced with specific values or definitions.
Vague-word list (given to Jev as examples — Jev judges meaning, not string matches)#
| English (ambiguous) | Korean (ambiguous) | Clear Replacement Example |
|---|---|---|
| fast, quickly | 빠르게, 신속히 | p95 response under 200ms |
| many, large amount | 대량, 다수, 많이 | 10,000 or more |
| appropriate, properly | 적절히, 알맞게 | {specify concrete criteria} |
| simple, easy | 간단히, 쉽게 | Complete within 3 steps |
| etc., and so on | 등등, 기타 | Enumerate complete list |
| if needed, depending on | 필요시, 상황에 따라 | {specify exact conditions} |
| as much as possible | 최대한 | {specify quantitative target} |
| generally | 대체로, 일반적으로 | {specify including exception conditions} |
| Level | Criteria |
|---|---|
| 0 | 0 vague terms, all values specified |
| 1 | 1-2 vague terms, most values specified |
| 2 | 3-5 vague terms, some values missing |
| 3 | 6+ vague terms, many values missing |
| 4 | Pervasive use of ambiguous expressions |
Dimension 2: Reference Clarity — Weight 0.20#
Whether pronouns, demonstratives, and abbreviations clearly point to specific entities.
- Pronouns: "this", "that", "said" / 이것, 그것, 해당, 위의 → Replace with specific entity names
- Abbreviations: Define full name + abbreviation on first occurrence
- Implicit subjects: "is processed", "is sent" / 처리된다, 전송된다 → Specify the agent
- External references: "as per existing approach" / 기존 방식대로 → Specify reference document + section
| Level | Criteria |
|---|---|
| 0 | All references have 1:1 mapping, all abbreviations defined |
| 1 | 1-2 ambiguous references, most abbreviations defined |
| 2 | 3-5 ambiguous references, some abbreviations undefined |
| 3 | Many ambiguous references, insufficient abbreviation definitions |
| 4 | Key entity references unclear |
Dimension 3: Completeness — Weight 0.20#
Whether every capability has acceptance criteria (AC) and edge/error cases have been identified.
| Level | Criteria |
|---|---|
| 0 | All capabilities have AC; edge cases and error scenarios covered |
| 1 | 1-2 capabilities lack AC or an edge case is missing |
| 2 | Several capabilities lack AC; edge cases partly covered |
| 3 | Most capabilities lack AC; error scenarios absent |
| 4 | Acceptance criteria essentially missing |
Dimension 4: Consistency — Weight 0.20#
Unified terminology, no numeric/logical contradictions between sections, consistent priorities.
| Level | Criteria |
|---|---|
| 0 | Fully consistent |
| 1 | A minor terminology inconsistency or one small contradiction |
| 2 | Several inconsistencies or a notable contradiction |
| 3 | Multiple contradictions between sections |
| 4 | Severe contradictions — sections describe different products |
Dimension 5: Testability — Weight 0.15#
Whether each requirement can be judged pass/fail objectively and reproducibly, preferably automated.
| Level | Criteria |
|---|---|
| 0 | Every criterion has an objective, automatable pass/fail condition |
| 1 | 1-2 criteria need clarification before testing |
| 2 | Several criteria only manually testable or partly subjective |
| 3 | Most criteria depend on subjective judgment |
| 4 | Criteria cannot be tested as written |
Language note (Korean-first)#
The spec is usually Korean; the question instructions are English (Jev's best-performing language)
and name the Korean vague terms explicitly. Jev judges meaning, so josa and missing word boundaries
are not a lexical-matching problem. Calibration was checked on five Korean fixtures
(skills/jev-gate/tests/fixtures/, see skills/jev-gate/SKILL.md) — a small sample, so rows reported
as uncertain_rows (probability 0.25–0.75) are still read by a person. Terminology consistency
(one name per concept) is not judged by Jev; read it yourself.