CompCQR: Compositional Query Generation for Training-Free Conversational Search
Yunah Jang, Kang-il Lee, Joongbo Shin, Kyomin Jung
cs.CL · cs.AI
Quick verdict
Claims SOTA training-free CQR via 325 permuted component queries + RRF + LLM selection; gains credible but hinge on rank-1 picking, single-run evals.
Summary
The paper proposes CompCQR, a training-free framework for conversational query reformulation (CQR) that avoids repeated LLM invocations. Motivated by the observation that dense retrievers are order-sensitive — reordering identical query content changes up to 28% of top-10 retrieved documents (Jaccard distance, ANCE) — the method makes two LLM calls to extract five atomic components (original query, rewritten query, rewritten answer, follow-up query, follow-up answer), then enumerates all 325 permutations as candidate queries. Retrieved lists are fused at document level with reciprocal rank fusion, and a final LLM call selects the single most answerable document, which is promoted to rank 1; total cost is three LLM calls per query. On QReCC, TopiOCQA, CAsT-19 and CAsT-20 with BM25 and ANCE (plus GTR/Contriever), CompCQR with GPT-4.1-mini reaches, e.g., 48.7 MRR on TopiOCQA (dense) versus 41.9 for LLM4CS, and up to 22.5% relative MRR gain over the best trained baseline (QReCC dense). Ablations show LLM selection adds +4.5 MRR and RRF beats mean aggregation. Limitations: main results are single-run without significance tests; CAsT recall@10 is slightly below LLM4CS; selection accounts for 62% of latency and total FLOPs (109T) exceed LLM4CS k=5 (69T); evaluation is English-only.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Dense retrievers are order-sensitive: reordering identical query components changes up to 28% of top-10 retrieved documents (ANCE top-10 Jaccard distance; GTR 22.6%, Contriever 23.2%).
- Compositional query generation from two LLM calls (325 permutations) outperforms 325-query IID and sequential sampling on average and upper-bound MRR at a small fraction of the cost (Figure 2).
- Document-level RRF aggregation increasingly outperforms query-embedding mean aggregation as candidates scale, and LLM best-document selection adds up to +5.0 NDCG@3 (Table 5, TopiOCQA, Llama-3.1-8B).
- Training-free CompCQR beats prior training-free methods across all tested settings and matches or exceeds trained baselines in several, with up to 22.5% relative MRR gain (QReCC dense: 61.5 vs 50.2) using 3 LLM calls per query.
Strengths
Cost-capped compositional expansion is well motivated and empirically validated
Section 2 shows identical component sets yield up to 28% top-10 Jaccard distance (ANCE; Table 16: GTR 22.6%, Contriever 23.2%), and Figure 2 shows 2-call compositional generation beating 325-call IID and sequential sampling on both average and upper-bound MRR.
Headline numbers trace arithmetically to the tables
22.5% relative = QReCC dense 61.5 vs ConvSearch-R1's 50.2 (+11.3 MRR, matching Section 4.2); 14.7% sparse = 67.0 vs ICR's 58.4. The paper also concedes ConvSearch-R1 stays ahead on TopiOCQA dense (50.5 vs 48.7), using 'comparable' honestly.
Broad, multi-axis empirical support
Both core components earn their place in ablations
Weaknesses
No statistical significance testing; single-run main results
Rank-1 promotion is metric-aligned rather than capability-aligned
Efficiency framing oversells in call-count terms
Internal numeric inconsistency in Section 6
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CompCQR (arXiv 2609.14646; Seoul National University + LG AI Research) targets conversational query reformulation (CQR): rewriting each dialogue turn into a standalone retrieval query without training any model. Four claims are made: (1) dense retrievers are order-sensitive, so permutations of identical content expose complementary retrieval sets; (2) compositional expansion — two LLM calls yielding 325 candidate queries — dominates IID and sequential sampling at equal candidate counts (Figure 2); (3) document-level RRF fusion plus an LLM best-document selection beat query-embedding aggregation (Tables 4-5); (4) the pipeline is state of the art among training-free methods and competitive with trained ones on QReCC, TopiOCQA, CAsT-19/20. The headline arithmetic checks out against Table 2: 'up to 22.5% relative MRR' corresponds to QReCC dense ANCE (61.5 vs ConvSearch-R1's 50.2, +11.3 MRR, as stated in Section 4.2), and '14.7%' sparse corresponds to QReCC BM25 (67.0 vs ICR's 58.4). Notably, on TopiOCQA dense the strongest trained baseline ConvSearch-R1 (50.5 MRR, R@100 86.3) still exceeds CompCQR with GPT-4.1-mini (48.7, 83.5); the paper's 'comparable' wording is accurate there.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CompCQR and how does it work?
CompCQR is a training-free method for conversational search query rewriting. It makes two LLM calls to extract five text components (original query, rewritten query, rewritten answer, follow-up query, follow-up answer), generates all 325 permutations of these as candidate queries, fuses the retrieved documents with reciprocal rank fusion (RRF), and finally uses one more LLM call to pick the single best document, which is moved to rank 1 — three LLM calls per query in total.
Does CompCQR require any training or fine-tuning?
No. It uses off-the-shelf LLMs (e.g., GPT-4.1-mini or Llama-3.1-8B-Instruct) and off-the-shelf retrievers (BM25, ANCE, GTR, Contriever) with no task-specific training; only a few hyperparameters (RRF smoothing, selection set size) are tuned on a small validation sample.
How much better is CompCQR than previous methods?
On four benchmarks (QReCC, TopiOCQA, CAsT-19, CAsT-20) it beats prior training-free baselines like LLM4CS on MRR — e.g., 48.7 vs 41.9 MRR on TopiOCQA with ANCE and GPT-4.1-mini — and reports up to 22.5% relative MRR improvement over the best trained baseline (on QReCC with dense retrieval). It uses 3 LLM calls per query versus 5-16 for comparable prompting baselines, though its total FLOPs (109T) exceed the 5-call baseline's (69T).
Is code available for CompCQR?
Yes, the paper links a public repository (github.com/YunahJang/CompCQR). Reproducing the open-model configuration requires serving Llama-3.1-8B (e.g., with vLLM), indexing the benchmark corpora with Pyserini/Faiss, and the public datasets; headline GPT-4.1-mini numbers additionally depend on the OpenAI API.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 2 and 4-8 appear only as extracted text fragments (axis ranges, scattered labels); per-figure data points could not be fully verified.
- Table formatting was flattened during extraction; a few cells (dashes for unreported metrics, footnote markers) were reconstructed from context.
- The Section 6 / Table 10 numeric mismatch might be an extraction artifact, though the phrasing suggests an authoring error.
- Appendix J prompts were partially extracted; exact prompt wording for Prewrite (inherited from LLM4CS RAR) could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.