Data Efficient Sample Selection for In-Context Learning
V Venktesh, Cem Levi, Avishek Anand
cs.LG · cs.CL
Quick verdict
Gap-index bandit + differentiable sorting surrogate for ICL demo selection; strong gains on 3 tasks/LLMs, but theory's bias terms are empirically estimated.
Summary
The paper introduces DEARICL, a multi-armed bandit framework that selects in-context learning (ICL) demonstrations by treating k-sized example subsets as arms and training a non-linear ranking surrogate (an MLP scored with a differentiable, relaxed-NDCG sorting loss) from LLM feedback. Unlike prior task-level bandit selectors (GIFA, CASE, EXPLORA, BRIDGE), which pick one subset offline via linear surrogates, DEARICL uses top-m arm identification only as an auxiliary objective: gap-index computations steer sampling toward borderline arms, and instance-level rewards (per validation query, BertScore-based) continually retrain the ranker, which then ranks candidate subsets per test query. Theoretical contributions include a pairwise-gap concentration bound (Theorem 3.2) combining MC-dropout Bernstein widths, SGD stability, and surrogate bias terms, plus a sample-complexity upper bound (Theorem 3.3). On GSM8K, AquaRAT, and WMT19 en-zh with Llama3.2-3b, DEARICLdynamic reaches 75.66/55.11/78.89, beating CASEdynamic by 8.08-15.99% relative (t-test, 0.05) and all tested LTR baselines; the offline phase uses 63-75% fewer LLM calls than CASE and per-query inference adds only milliseconds over static selection. Results transfer to gpt-4o-mini and DeepSeek-R1-Distill-7B in a cross-model setting. Limitations: confidence-width bias and stability terms are estimated rather than bounded, no seed variances or CIs are reported, WMT19's reward and metric are both BertScore, and the ranker only scores subsets seen during bandit training.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Demonstration selection for ICL can be cast as instance-level subset ranking, with top-m arm identification in a gap-index bandit serving as an auxiliary objective that supplies borderline training samples for a non-linear ranking surrogate.
- DEARICLdynamic improves accuracy by 8.08-15.99% relative over CASEdynamic on GSM8K, AquaRAT, and WMT19 with Llama3.2-3b, and outperforms six learning-to-rank baselines (Table 1).
- The auxiliary top-m objective makes the offline phase 63-75% cheaper in LLM calls than CASE (Table 3) while adding only milliseconds of per-query inference overhead (Table 2).
- A sample-complexity guarantee for gap-index bandits with non-linear surrogates holds via a pairwise-gap concentration bound combining MC-dropout Bernstein terms, SGD stability, and surrogate bias (Theorems 3.2-3.3).
Strengths
Principled coupling of bandit exploration with LTR training
Using top-m gap-index identification as an auxiliary objective that steers sampling of borderline arms (Algorithm 1) is a principled way to manufacture hard negatives for the ranker; the random-exploration ablation (49.21 vs 55.11 on AquaRAT) and Appendix F's retraining of LTR baselines on DEARICL-logged data (Table 5: NDCGLoss2++ reaches 72.11/49.00/74.06 yet stays below 75.66/55.11/78.89) both support the attribution.
Strong and consistent empirical gains
Table 1 (llama3.2:3b): DEARICLdynamic 75.66/55.11/78.89 on GSM8K/AquaRAT/WMT19 vs CASEdynamic 70.00/47.51/71.70, marked significant at 0.05 on all three; also beats six LTR baselines and transfers cross-model to gpt-4o-mini (Table 6) and DeepSeek-R1-Distill-7B (Table 7) with exemplars selected by Llama3.2-3b.
Compelling efficiency story
End-to-end theoretical treatment attempt
Transparent specification
Weaknesses
Theoretical guarantees rest on unbounded empirical terms
Thin statistical reporting
Metric circularity and reward-eval mismatch
Restricted candidate space for dynamic selection
Attribution ambiguity and minor inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DEARICL targets demonstration (few-shot exemplar) selection for in-context learning with small open LLMs. The authors' claims: (1) prior bandit selectors (GIFA, CASE, EXPLORA, BRIDGE) pick a single subset offline with linear surrogates and fail on unseen queries; (2) recasting selection as instance-level subset ranking, trained with top-m gap-index identification as an auxiliary objective, yields a better ranker at lower sample cost; (3) concrete relative gains of 8.08-15.99% over CASEdynamic on GSM8K/AquaRAT/WMT19 with Llama3.2-3b (Table 1); (4) the first sample-complexity treatment for gap-index bandits with non-linear surrogates (Theorems 3.2-3.3). This is a method paper; no new datasets or benchmarks are introduced.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DEARICL?
DEARICL (Data Efficient Algorithm for Ranking ICL samples) is a bandit-based framework that selects which few-shot demonstration examples to place in an LLM's prompt. It trains a small neural ranker with a differentiable sorting loss while a gap-index bandit deliberately samples ambiguous candidate example subsets; at test time the ranker picks the best subset per query without rerunning the bandit.
How much does DEARICL improve ICL accuracy?
With Llama3.2-3b, the dynamic version reaches 75.66 on GSM8K, 55.11 on AquaRAT and 78.89 on WMT19 en-zh (BERTScore), which the authors report as 8.08-15.99% relative gains over the strongest prior bandit baseline (CASEdynamic), statistically significant at the 0.05 level.
Does DEARICL require fine-tuning the LLM?
No. The LLM stays frozen. DEARICL only learns to rank candidate demonstration subsets from LLM feedback collected during an offline bandit phase, adding only milliseconds of ranking overhead per query at inference time.
Is code available for DEARICL?
Yes. The paper points to https://github.com/VenkteshV/DearICL for code and data, and the experiments use public datasets (GSM8K, AquaRAT, WMT19) with open-weight models such as Llama3.2-3b and DeepSeek-R1-Distill-Qwen-7B.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Inline math is flattened by PDF extraction (fractions, square roots, subscripts), so Theorems 3.2-3.3 were assessed from their linearized statements; derivation-level judgments carry some uncertainty.
- Figures 1-2 are rendered plots; exact curve values are unavailable beyond the numbers quoted in the surrounding text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.