Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents
cs.CL
Quick verdict
Proposes RAE, a paired retrieval-conditioned metric showing aggregate skill-retrieval lift can mislead; sound but small-scale, tiny skill pools.
Summary
The paper argues that standard evaluations of retrieval-enabled LLM agents — whole-benchmark averages and retrieved-versus-skipped comparisons — carry task-selection bias and cannot tell whether retrieving a skill actually helps the task at hand. It formalizes Skill Following as the retrieval-to-answer chain and introduces RAE (Retrieval-Invoked Actual-Use Effect): the paired skill-enabled minus skill-disabled outcome difference, computed only on tasks where the agent invoked retrieval and received a skill. Together with OAE (the same pairing over all tasks), RAE yields an exact decomposition of the overall effect. Evaluating 17 LLMs on 80-task MBPP+ partitions (seeds 42/43/44), HumanEval+, and Math500, with BM25 retrieval over 9 coding and 8 math procedural skills, the authors find frequent sign reversals: Gemini-2.5-Flash-lite shows positive aggregate lift but negative RAE on all three MBPP+ partitions (e.g., +6.2/−15.6 on seed 42), and Llama-3.3-70B shows +14.2 lift versus −39.4 RAE on Math500. Filler/random/corrupted-skill controls leave RAE negative, and GPT-5.5 annotation plus a 50-case human audit finds most harmful transitions involve ignored skill content. Main limitations: only 80 tasks per partition with many sparse cells, tiny generic skill pools of arguable relevance, single-run pairing at temperature 0.7 without variance estimates, no multiple-comparison control, and no released code or logs.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 4
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Aggregate retrieval lift compares self-selected retrieved vs. non-retrieved task subsets and is confounded by task-selection bias (Δselect ≠ 0, Fig. 2).
- RAE, the paired skill-enabled minus skill-disabled outcome difference conditioned on retrieval-invoked tasks, isolates actual skill-use effect and decomposes OAE exactly (Eq. 4).
- Across 17 LLMs on MBPP+, HumanEval+, and Math500, aggregate lift and RAE frequently disagree in sign, including positive-lift-negative-RAE reversals (e.g., Gemini-2.5-Flash-lite on all three MBPP+ seeds).
- Negative RAE persists under filler, random, and corrupted skill content, so it is not solely an artifact of bad retrieved text; most harmful transitions show ignored skill content per GPT-5.5 annotation with human audit.
Strengths
Correct causal-identification core
Conditioning the paired SE/SD comparison on retrieval-invoked tasks (Scall) removes subset-comparison bias, and Eq. 4 gives an exact decomposition OAE = (ncond/npairs)·RAE + (nskip/npairs)·Δskip; §5.2 quantifies the selection mechanism via Δselect (Eq. 5, Fig. 2).
Statistically careful reporting
Pooled raw counts (App. A), exact McNemar tests and 20,000-resample paired bootstrap CIs for representative cells (Table 8, e.g., Mistral-S-24B partition 3: −45.5 [−57.6, −33.3], p=1.54e-8), and threshold sensitivity at ncond ∈ {5,10,15,20} (Table 5, stable 3–6/13 disagreements).
Honest diagnostic layering
Content controls bound the interpretation
Candid limitations section
Weaknesses
Small scale with sparse cells
Tiny, generic skill pools of questionable relevance
Single-run pairing at temperature 0.7
No multiple-comparison control
No artifact release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a narrow but real measurement problem: when an LLM agent retrieves a skill, does that retrieval help or hurt the same task? It formalizes Skill Following (SF) as the retrieval-to-answer chain and contrasts three metrics — aggregate retrieval lift (unpaired, within-SE subset comparison), OAE (paired SE−SD difference over all tasks), and RAE (the same pairing restricted to retrieval-invoked tasks Scall, Eq. 2) — connected by the exact decomposition OAE = (ncond/npairs)·RAE + (nskip/npairs)·Δskip (Eq. 4). Central claims: (1) aggregate lift is confounded by task-selection bias; (2) aggregate-vs-RAE sign reversals are common across 17 LLMs on coding and math benchmarks; (3) negative RAE is not merely an artifact of bad skill text and coincides with ignored skill content.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RAE in this paper?
RAE (Retrieval-Invoked Actual-Use Effect) is a paired metric that measures, only on tasks where an LLM agent actually invoked skill retrieval and received a skill, the difference in success rate between a skill-enabled run and a skill-disabled run of the same task. Because both runs solve the identical task, it removes the task-selection bias present when benchmarks compare retrieved vs. non-retrieved task subsets.
What is the 'evaluation paradox' the paper reports?
Several models look better with skill retrieval on benchmark-wide averages (positive aggregate lift), yet on the exact tasks where they retrieved a skill, their paired skill-on vs. skill-off comparison is negative. For example, Gemini-2.5-Flash-lite shows +6.2/+16.2/+4.4 aggregate lift but −15.6/−15.9/−22.2 RAE across the three MBPP+ partitions, and Llama-3.3-70B shows +14.2 lift vs. −39.4 RAE on Math500.
Which models and benchmarks were used?
17 LLMs — including Claude 3.5/4.5 Haiku, Gemini 2.0/2.5 Flash variants, GPT-4o-mini, DeepSeek-V3.2, Llama 3.1-8B/3.3-70B, Qwen 2.5-7B/3-8B/3-32B/3-235B/3.5-9B, Command-R, GLM4-32B, and Mistral NeMo/Small-24B — evaluated on 80-task partitions of MBPP+ (three seeds), HumanEval+, and Math500, using BM25 retrieval over 9 coding and 8 math procedural skills.
Does the paper release code or data?
No repository or artifact release is stated in the provided text. The protocol is described in detail (prompts, tool schema, skill-pool tables, model identifiers in the appendices), but code, skill files, and run logs are not stated as available, so exact numbers cannot currently be recomputed from published artifacts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction flattened multi-column table layouts (e.g., Table 2's outcome matrix and Tables 3/6 cells); all numeric values remained attributable to their rows and columns, and no end-of-text truncation was observed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.