ClusterFewshot: Improving Few-shot Optimization for LLMs workflow
Omri Bar Haim, Shahar Katz, Lior Wolf
cs.CL
Quick verdict
Claims cluster+one-shot-scored demo selection beats BFRS/MIPROv2 with big cost cuts; results broad but significance untested and several gaps sit within noise.
Summary
ClusterFewshot (Bar Haim, Katz, and Wolf, Tel Aviv University) targets the demonstration-selection stage of DSPy-style prompt optimization. Instead of the random sampling of BootstrapFewShotRS (BFRS) or the Bayesian search of MIPROv2, the method bootstraps successful traces once over the training set, embeds candidates with sentence encoders (gtr-t5-base selected via Silhouette grid search, K=3), scores each bootstrapped candidate by its accuracy as the sole one-shot demonstration on a small cluster-central validation subset (m=3 per cluster), and finally chooses between Global Top-k and per-cluster representatives by full-validation evaluation. Across GSM8K, HotPotQA, and Iris with Qwen2.5-7B and Llama-3.2-3B, embedded in five BetterTogether prompt/fine-tuning strategies, the paper reports higher mean test accuracy than both baselines in the majority of configurations (e.g., 87.78% vs 85.59% BFRS on GSM8K) alongside markedly lower cost (standalone HotPotQA: 25.5 minutes vs 42.9/59.1). Extensions cover Qwen2.5-14B/32B, a ReAct agent with ColBERTv2 retrieval (45.87% vs 44.80–44.95% accuracy at 7B; compile time cut about 60%), and wins over retrieval-based few-shot selection (RFS-kNN/MMR). Main limitations: three seeds with no significance tests, many margins within one standard deviation (HotPotQA/Llama standalone is nominally behind BFRS), very small optimization sets (100 training examples for GSM8K; 15/35 for Iris), and minor internal numeric inconsistencies.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Semantic clustering of bootstrapped candidates combined with one-shot utility scoring selects better few-shot demonstrations than BFRS random search or MIPROv2 Bayesian search within DSPy pipelines.
- ClusterFewshot substantially reduces optimization cost — e.g., standalone HotPotQA compile 25.5 min vs 42.9/59.1 min and ~60% ReAct compile-time cuts — while matching or improving accuracy in most configurations.
- The gains generalize across five BetterTogether prompt/fine-tuning strategies, model families and scales (Llama-3.2-3B to Qwen2.5-32B), and an agentic ReAct + ColBERTv2 setting.
- A fixed compiled few-shot context chosen by hybrid (Top-k vs cluster-representative) selection outperforms query-adaptive retrieval-based selection (RFS-kNN, RFS-MMR at several λ) from the same candidate pool.
Strengths
Unusually complete evaluation matrix
3 benchmarks × 2 models × 5 BetterTogether strategies × 3 seeds (Tables 1–2, 13–18), plus Qwen2.5-14B/32B extensions (Tables 4, 12) and a ReAct agentic setting with ColBERTv2 retrieval (Table 4) — far beyond a single-task demo.
Credible, well-isolated efficiency wins
Standalone HotPotQA compile: 25.5 min vs 42.9 (BFRS) / 59.1 (MIPROv2) and 6.31M tokens vs 8.75M/7.19M with +19.1pp over zero-shot (Tables 2–3); ReAct 7B compile 14.1 min = 63.8%/60.4% cuts (Table 4).
Thorough design ablations
Lower run-to-run variance than baselines in most cells
Reproducibility groundwork is solid
Weaknesses
No statistical significance testing anywhere
Abstract overclaims consistency
Baseline asymmetry confounds cost claims
Tiny optimization and evaluation sets
Internal numeric inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ClusterFewshot (Bar Haim, Katz & Wolf, Tel Aviv University; arXiv:2609.25939) targets demonstration selection in DSPy-style prompt optimization. The authors argue that prior bootstrap optimizers — BootstrapFewShotRS (BFRS, random search) and MIPROv2 (Bayesian joint search over instructions and demonstrations) — ignore the semantic structure of the task. The method (§3) bootstraps successful traces once over the training set (§3.1), embeds and clusters candidates with sentence encoders, selecting encoder and K by Silhouette grid search (§3.2: gtr-t5-base, K=3 for GSM8K), scores every bootstrapped candidate by its one-shot accuracy on a cluster-central validation subset (m=3 per cluster, §3.3), and finally chooses between Global Top-k and per-cluster representatives on the full validation set (§3.4–3.5). Central claims: (i) higher accuracy than BFRS/MIPROv2 in most configurations, (ii) substantially lower optimization cost, (iii) robustness across models (3B–32B), strategies, and an agentic ReAct setting.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ClusterFewshot and what problem does it solve?
ClusterFewshot is a prompt-optimization technique for LLM pipelines built on the DSPy framework. It decides which few-shot in-context examples to put in a prompt by (1) collecting successful model traces over the training set, (2) clustering those candidates semantically with sentence embeddings and K-means, (3) scoring each candidate by how well the model performs when it is the only example in the prompt, tested on a small validation subset sampled near cluster centers, and (4) keeping the best candidate set. It replaces the random sampling of BootstrapFewShotRS and the Bayesian search of MIPROv2.
How much faster and cheaper is ClusterFewshot than prior methods?
The paper reports that with Qwen2.5-7B-Instruct, standalone HotPotQA optimization took 25.5 minutes on average versus 42.9 minutes for BFRS and 59.1 minutes for MIPROv2 (a >40% cut), using 6.31M tokens versus 8.75M and 7.19M. In the ReAct agentic HotPotQA setting, compile time at 7B dropped 63.8% versus BFRS and 60.4% versus MIPROv2 (14.1 minutes). On GSM8K the token usage is roughly comparable to BFRS.
Does ClusterFewshot always beat the baselines?
No. It wins in the majority of the tested configurations — for example 87.78% vs 85.59% (BFRS) on GSM8K with Qwen2.5-7B — but on HotPotQA with Llama-3.2-3B in the standalone setting, BFRS is nominally ahead (36.96 vs 36.85), and many margins are smaller than one standard deviation of the three-run averages. No statistical significance tests are reported, so individual wins should be read as trends rather than proven differences.
Is the code and data available for ClusterFewshot?
The paper states that code is available at github.com/omrirh/clusterfewshot. All experiments use public datasets (GSM8K, HotPotQA, Iris) and open-weight models (Qwen2.5 7B/14B/32B, Llama-3.2-3B) served via SGLang within DSPy, and the paper documents prompt templates and a total compute budget of about 160 A100 GPU-hours, so replication is feasible for a well-resourced team.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–5 are visual artifacts; only their captions and surrounding claims could be assessed from the extracted text.
- Review is based on arXiv v1 (22 Sep 2026); peer-review status unknown.
- Extraction shows minor text-flow artifacts (e.g., broken citations 'Opsahl- Ong', duplicated reference entry), which do not affect table values but suggest light copyediting overall.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.