When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control
cs.CL · cs.AI
Quick verdict
CoSQ prompt gating cuts TruthfulQA wrong commitments 13.1%→8.9% across 11 models; strong paired stats, but missing CoT-abstain baseline weakens attribution.
Summary
The paper introduces Chain-of-Self-Questioning (CoSQ), a prompt-only selective answering framework in which a large language model first decomposes a question into required information units, elicits a 0–100 confidence score for each, and commits to an answer only if an aggregation rule passes a threshold τ; otherwise it abstains. Three variants (Grounded, Critical, Adaptive) are evaluated under seventeen conditions on the 817-item TruthfulQA multiple-choice validation set across eleven model families, with a secondary 300-question Natural Questions Short-Answer test on five models. Against forced-choice chain-of-thought, Grounded-CoSQ at τ=0.90 lowers the mean unconditional wrong-commitment rate from 13.1% to 8.9% (a 32.1% relative reduction) while raising answered accuracy from 86.9% to 89.7% at 87.6% coverage; the direction holds for every model and every threshold, supported by exact one-sided Wilcoxon tests (p=0.00049) and bootstrap intervals. Key limitations: the paper lacks a CoT-with-abstention baseline, so it cannot isolate whether multi-stage decomposition outperforms simple verbalized-confidence abstention; 68.7% of abstentions withhold answers the CoT baseline answered correctly; confidence scores concentrate so lower thresholds barely change behavior; hosted endpoint labels (e.g., GPT-5.5, Claude 5 Sonnet) are unverifiable; and TruthfulQA's high baseline accuracy limits headroom. Code, prompts, and reproducibility manifests are released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CoSQ is a prompt-only, model-agnostic framework that makes LLM answer commitment conditional on an explicit self-assessment of the information units required to answer a question.
- At τ=0.90, Grounded-CoSQ reduces the mean unconditional wrong-commitment rate from 13.1% (forced-choice CoT) to 8.9% — a 32.1% relative reduction — while raising answered accuracy from 86.9% to 89.7% at 87.6% coverage, with the direction holding for all eleven models and all fifteen selective configurations.
- Critical-CoSQ and Adaptive-CoSQ are neighboring operating points on the same risk–coverage frontier, not universally superior variants; τ is an empirical operating parameter, not a calibrated probability.
- A secondary 300-question NQ-Short evaluation on five models provides convergent open-form evidence that the selective risk reduction transfers beyond TruthfulQA-MC.
Strengths
Complete prespecified evaluation grid with anti-shortcut protocol
Seventeen conditions (Direct, CoT, 3 variants × τ∈{0.50–0.90}) × eleven model families on 817 TruthfulQA-MC items with deterministically balanced option positions (seed 1002) and forced-choice scoring of malformed outputs as wrong (§4.1, §4.3); the full sweep is reported so the τ=0.90 operating point cannot be accused of post-hoc selection (§5.3, Table 5).
Transparent accounting of abstention cost
Table 4 pairs every τ=0.90 abstention with the CoT outcome and reports that only 31.3% of Grounded-CoSQ abstentions coincide with CoT errors while 68.7% withhold CoT-correct answers — an unusually honest disclosure that most selective-prediction papers omit.
Paired statistics with full model-level disclosure
Strong reproducibility posture
Weaknesses
Missing CoT-with-abstention baseline undermines attribution
Net utility is modest and never cost-analyzed
Confidence distribution is degenerate below τ=0.90
Six of eleven endpoints are unverifiable
Ceiling and thin generalization evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes Chain-of-Self-Questioning (CoSQ), a prompt-only selective answering pipeline: decompose a question into required information units, self-score each unit 0–100, and commit to an answer only if an aggregation rule passes a threshold τ. Three variants are studied — Grounded (mean score ≥ τ), Critical (critical-unit mean), Adaptive (three fixed checks plus a post-answer contradiction check). The claims are explicitly scoped (§1.1): H1 (lower unconditional wrong-commitment rate than forced-choice CoT at τ=0.90), H2 (higher answered accuracy), RQ1 (threshold control), RQ2 (generalization). The paper is careful to frame the confidence gate as an empirical operating parameter rather than a calibrated probability, and to present abstention as a tunable operating characteristic — a refreshingly honest framing for this literature.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Chain-of-Self-Questioning (CoSQ)?
CoSQ is a prompt-only technique that makes a large language model decide whether to answer at all: it first lists the pieces of information it would need to answer a question, self-scores its confidence in each piece on a 0–100 scale, and only commits to an answer if the aggregated confidence passes a threshold; otherwise it explicitly replies that it does not know.
How much does CoSQ reduce wrong answers or hallucinations?
On the 817-question TruthfulQA multiple-choice benchmark across eleven model families, the rate of wrong committed answers fell from 13.1% with chain-of-thought prompting to 8.9% with Grounded-CoSQ at threshold 0.90 — a 32.1% relative reduction — while the accuracy of answered questions rose from 86.9% to 89.7% at 87.6% coverage. The improvement held for every model tested.
What is the trade-off when CoSQ abstains?
About 12.3% of questions receive an 'I don't know' instead of an answer, and roughly 69% of those abstentions are questions the chain-of-thought baseline actually answered correctly. The method therefore pays off mainly in settings where a wrong answer costs far more than a referral for human review.
Is the CoSQ code available?
Yes. The authors provide a GitHub repository (github.com/senolali/CoSQ) and a PyPI package named cosq, and the paper's Appendix A reproduces all prompt templates used in the experiments, including the abstention output.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 appear only as extracted captions and axis labels; error bars, curve shapes and plot layout were judged from captions and Tables 3–8, not from rendered plots.
- Hosted endpoint names (gpt5_5, claude5_sonnet, DeepSeek Flash, Gemma 4) follow the paper's run manifests and could not be externally verified under the no-network constraint.
- Per-question raw responses are not included in the extracted text; the 99.43% forced-choice parse-rate audit (§5.1) is taken from the authors' own report.
- arXiv id 2609.17516 and benchmark versions (e.g., TruthfulQA validation split selection) are as stated by the authors and were not independently checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.