When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain
Yunxiang Li, Xixin Wu, Helen Meng
cs.AI
Quick verdict
CIGAsk uses frozen-reference information gain + ambiguity bonus in GRPO to teach when/how to clarify; solid gains but single-seed and simulator-dependent.
Summary
The paper addresses a known failure of instruction-tuned LLMs: on underspecified queries they commit to one interpretation instead of asking a clarifying question. The authors argue that solving this requires jointly learning when to ask and how to ask, and propose CIGAsk, a multi-turn GRPO recipe with two rewards. Counterfactual Information Gain (CIG) scores each clarification turn by the change in gold-answer log-likelihood under a frozen reference model, with and without the user's reply, providing per-turn credit without a learned critic. An Asymmetric Ambiguity Bonus, keyed on gold ambiguity labels, adds terminal credit (+ρ for clarifying ambiguous queries, −ρ otherwise) that governs when to ask; a two-stream advantage combines both signals. Trained on Qwen2.5-3B/7B with a GPT-4o user simulator holding the gold intent, CIGAsk-7B reaches F1 0.795 on PACIFIC fullval, above reported ACT (0.681, Zephyr-7B) and SGP (0.726, Gemma-2-9B) anchors, and wins matched-backbone reruns (0.709 over ACT; 0.774 over deployable SGP, below its 0.787 oracle variant). It transfers with shared hyperparameters to AbgCoQA (F1 0.724) and AmbigNQ (EM 0.511) and preserves single-turn QA on TriviaQA/NQ Open. Limitations: all results are single-seed; comparisons with external methods cross evaluation protocols; training relies on an oracle cooperative simulator and per-instance ambiguity labels; no code release is mentioned.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Prompting alone cannot teach instruction-tuned LLMs both when and how to clarify; models either ask on every query or ask vague questions.
- Counterfactual Information Gain computed under a frozen reference provides per-turn credit that improves post-clarify F1, and freezing is essential (F1 .795 frozen vs .607 current-actor, Table 4).
- The Asymmetric Ambiguity Bonus governs when to ask: removing it collapses ambiguous-query clarify recall from .873 to .171 (Table 3).
- CIGAsk-7B reaches PACIFIC fullval F1 0.795, exceeding reported ACT and SGP results and winning matched-backbone reruns on Zephyr-7B and Gemma-2-9B.
- The recipe transfers across PACIFIC, AbgCoQA, and AmbigNQ with shared hyperparameters while preserving single-turn QA on TriviaQA and NQ Open.
Strengths
Frozen-reference per-turn credit without a learned critic
CIG (§3.2, Def. 1) scores each clarify turn as the log-likelihood ratio of the gold answer under a frozen π_ref with vs. without the user reply. Table 4 directly tests the core design: swapping in the current actor collapses F1 .795→.607 and Clr-amb .873→.233.
Component ablations cleanly dissociate the two skills
Table 3: removing the asymmetric bonus drops Clr-amb .873→.171 while F1post stays .733; removing CIG drops F1post .768→.700 with selectivity intact (.763/.167). 'When' and 'how' are separated, not confounded.
Honest framing of external baselines plus matched-backbone reruns
Simulator sensitivity is measured, not ignored
Selectivity reported as recall/FPR split
Weaknesses
Every result is single-seed
Metric and protocol mixing in Table 2
Training depends on an oracle simulator and gold labels
SFT warmstart under-described and behaviorally odd
Hyperparameter sensitivity only for ρ
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets the documented tendency of instruction-tuned LLMs to answer under-specified queries without clarifying (§1). It makes four claims: (1) prompting alone cannot teach the two coupled skills of when and how to ask; (2) Counterfactual Information Gain (CIG) under a *frozen* reference supplies per-turn credit for clarification quality, and freezing is essential; (3) an Asymmetric Ambiguity Bonus keyed on gold labels supplies the when-to-ask gate; (4) the combined recipe sets a new PACIFIC state of the art with a 7B backbone, transfers across three clarification benchmarks with shared hyperparameters, and preserves single-turn QA. The scope is a training-time recipe evaluated with a simulated cooperative user; no human-user study is attempted.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CIGAsk and what problem does it solve?
CIGAsk is a reinforcement-learning training recipe that teaches instruction-tuned LLMs two skills at once: when to ask a clarifying question instead of guessing on an ambiguous query, and how to phrase a question that actually recovers the missing information. It trains in a multi-turn GRPO loop with two rewards: Counterfactual Information Gain, which grades each clarification by how much the user's reply raises a frozen reference model's likelihood of the gold answer, and an asymmetric bonus based on gold ambiguity labels that gates when to ask.
How well does CIGAsk perform?
On the PACIFIC financial-QA benchmark (full validation set, 1,952 items), CIGAsk built on Qwen2.5-7B reaches F1 0.795, above the reported results of ACT (0.681 on Zephyr-7B) and SGP (0.726 on Gemma-2-9B); reruns on those backbones also win (0.709 and 0.774, the latter below SGP's 0.787 oracle variant). Retrained with the same hyperparameters, it reaches F1 0.724 on AbgCoQA and exact match 0.511 on AmbigNQ, while single-turn QA on TriviaQA and NQ Open stays close to the base model. All figures are single-seed.
Does CIGAsk require GPT-4o?
In the main experiments GPT-4o plays the user during training, given the true hidden intent. The authors show an open-weight Qwen2.5-7B-Instruct simulator works nearly as well (F1 0.783 vs 0.795), but a weaker 3B simulator degrades results sharply (F1 0.677), so simulator quality matters. At deployment no simulator is needed — the trained model decides on its own whether to ask the real user a clarifying question.
Is code or data available for CIGAsk?
The paper text does not mention a released code repository or model checkpoints. It does publish full training hyperparameters (Table 7), verbatim policy and simulator prompts (Appendix A.3), and dataset sources and licenses (Appendix A.2), all on public benchmarks (PACIFIC, AbgCoQA, AmbigNQ) — enough to reimplement, at the cost of an 8×H100 RL run and GPT-4o API access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear as mangled text fragments; quantitative reads of Fig. 3 (IG on clarify turns rising 0.17→0.47) rely on surviving axis labels.
- The SFT warmstart construction is only summarized in the provided text (referenced to App. A); its exact curation could not be assessed.
- App. A.7 comparisons inherit the original baseline evaluation protocols, as the paper itself notes; exact comparability could not be verified.
- arXiv IDs of concurrent works (SGP 2512.04068, InfoPO 2603.00656) could not be independently verified offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.