How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation
Elitsa Yotkova, Violeta Kastreva, Petar Velkov, Hristo Boyanov, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov
cs.CL
Quick verdict
Taxonomy + NLI-based CAP metric for Open QA correctness; strong gains on its own monotonicity protocol, but benchmark labels are mostly LLM-synthesized.
Summary
The paper tackles automatic evaluation of free-form question answering, where lexical and embedding metrics confuse paraphrases with errors. The authors propose an eight-class ordered taxonomy of semantic correctness (exact, equivalent, alternative-correct, overinclusive-valid, partial, overinclusive-invalid, invalid, contradictory) and release two resources: CAP-Correctness, 8,827 examples built from OpenBookQA, ARC, and MMLU whose candidate answers and labels were generated by Claude Haiku 4.5, with 1,638 examples human-validated (κ=0.779); and CAP-Statements, 11,000 question–answer–statement triples used to fine-tune an mT5-small reformulator. Their CAP metric converts question–answer pairs into declarative statements and scores them with bidirectional NLI (deBERTa-v3-large; α=0.85, λ=0.30). On 7,827 test examples CAP reaches Spearman ρ=60.37 versus 26.88 for the strongest baseline COMET, 77.70 pairwise accuracy, and 4/25 monotonicity violations versus 9/25; the ordering largely holds on human-labeled answers from GPT-4o, Gemini 2.0 Flash, and Qwen3-8B (2–6 violations). Main limitations: labels are largely synthetic and singly annotated with no inter-annotator agreement; CAP structurally inverts the overinclusive-valid versus partial distinction (16.34 pairwise accuracy) and undersells alternative-correct answers; the baseline set omits LLM judges and NLI-based metrics such as AlignScore; and coverage is limited to English short-form educational QA.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An eight-class ordered taxonomy distinguishes qualitatively different Open QA answer behaviors, including separating verbose-but-correct from hallucination-contaminated answers.
- CAP, which scores question-conditioned declarative statements with bidirectional NLI (α=0.85, λ=0.30), respects the taxonomy ordering far better than BLEU, ROUGE-L, METEOR, BERTScore and COMET (ρ=60.37 vs 26.88; 4/25 vs 9/25 violations).
- The class-mean ordering generalizes to human-labeled zero-shot answers from GPT-4o, Gemini 2.0 Flash and Qwen3-8B with only 2–6 monotonicity violations per model.
- CAP-Correctness (8,827 examples) and CAP-Statements (11,000 triples) are reusable resources, with human–LLM label agreement of κ=0.779 (quadratic-weighted 0.879) on the validated subset.
Strengths
Taxonomy fills a documented gap
Sec. 2.2 and Sec. 3: the explicit split of overinclusive-valid vs overinclusive-invalid (verbose-but-correct vs hallucination-contaminated) is absent from Bulian et al. 2022 and Yao & Barbosa 2024; Eq. (1) gives an explicit, inspectable monotonicity target that any metric can be scored against.
Large, CI-separated headline gains
Tab. 2/13: CAP Spearman ρ=60.37 [58.56, 62.11] vs COMET 26.88 [24.77, 28.99] on 7,827 test examples; pairwise accuracy 77.70 vs 61.10; monotonicity violations 4/25 vs 9/25 (Tab. 14); bootstrap CIs do not overlap any baseline.
Unusually candid error analysis
External validity check on real LLM outputs
Reproducibility detail
Weaknesses
Mostly synthetic, singly annotated ground truth
Structural failure on a central axis
Baseline set omits the strongest evaluators
Narrow domain
Closed evaluation loop
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real evaluation gap: free-form QA answers can be correct in many surface forms and fail in qualitatively different ways (incomplete, overinclusive, contradictory, false-premise), which lexical metrics and binary judgments collapse. Three contributions: (1) an eight-class ordered taxonomy (Eq. 1: exact ≈ equivalent ≈ alternative-correct ≥ overinclusive-valid > partial > overinclusive-invalid > invalid ≥ contradictory); (2) two released datasets — CAP-Correctness (8,827 examples from OpenBookQA/ARC/MMLU; 1,000 validation + 7,827 test) and CAP-Statements (11,000 question–answer–statement triples, 8,800/1,100/1,100); (3) CAP, a reference-based metric that scores question-conditioned declarative statements with bidirectional NLI. Central claims: CAP respects the taxonomy ordering better than BLEU/ROUGE-L/METEOR/BERTScore/COMET, and the framework exposes systematic failure modes of existing evaluators.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CAP metric for evaluating open-ended QA answers?
CAP (Context-Aware Precision) is a reference-based score in [0,1]. It first rewrites the question–gold and question–predicted pairs into declarative statements with a fine-tuned mT5 model, then scores how strongly the two statements entail each other in both directions using an NLI model (nli-deberta-v3-large), combining the two directional scores with weights α=0.85 and λ=0.30.
What is CAP-Correctness and where does its data come from?
A benchmark of 8,827 English [question, gold answer, candidate answer, correctness label] examples built on questions from OpenBookQA, AI2 ARC and MMLU. Candidate answers and their eight-class labels were generated by Claude Haiku 4.5 with class-specific prompts; 1,638 examples (18.56%) were re-labeled by human annotators, agreeing with the synthetic labels at Cohen's κ=0.779.
How much better is CAP than BLEU, BERTScore or COMET?
On the 7,827-example test split, CAP reaches Spearman ρ=60.37 against the taxonomy ordering versus 26.88 for COMET, the strongest baseline (BLEU 10.95, BERTScore 16.10); pairwise ranking accuracy 77.70 vs 61.10; monotonicity violations 4/25 vs 9/25. However, CAP scores below chance (16.34) on the overinclusive-valid vs partial pair and under-scores alternative-correct answers.
Is the code and data for this paper publicly available?
The paper states that code and data are available at a public GitHub repository linked in the abstract footnote, and the appendices document all prompts, model checkpoints and hyperparameters needed to rerun the experiments.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 are images not present in the extracted text; distribution and score-geometry claims (e.g., Fig. 3, Fig. 4) rely on the authors' in-text descriptions and tables.
- The GitHub repository and dataset release are stated in the paper but could not be verified without network access.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.