EconSkills: Studying Skill Transfer and Retrieval for Web Agents on Live Economic Data
cs.AI · cs.CL
Quick verdict
Shows matched skills lift transfer 41→49.3% on held-out variants, but retrieval only ties the no-skill baseline; well-controlled though single-backbone.
Summary
This paper introduces EconSkills, a library of 50 reusable web-agent skills distilled from verified EconWebArena trajectories on live economic-data portals. Each skill is a seven-part standard operating procedure (name, purpose, preconditions, procedure, website guidance, verification, recovery) with instance-specific values replaced by placeholders. The evaluation separates two questions. In a controlled transfer study on 100 held-out task variants (BASE/TRAJ1/MATCH1 × 3 seeds = 900 episodes with gpt-5-mini in BrowserGym/AgentLab), the matched skill raises success from 41.0% to 49.3% over no-skill prompting (McNemar p=0.0031) and saves 2.37 steps on jointly successful runs, while replaying the raw source trajectory collapses to 7.0%. In a library-scale study across all 360 tasks (one run per task per condition), five retrieved skills (RETR5, 28.3%) merely tie the no-skill baseline (28.1%); the full library (ALL50, 11.4%) causes severe context interference, and an optional approximate hint lowers success on the 210 uncovered tasks from 16.7% to 11.4%. Coverage-stratified analysis shows gains concentrate on tasks directly covered by the library. Limitations include a single backbone, one run per deployment condition, a fixed 30-step budget, live-site drift, and no isolation of prompt-length effects.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Supplying a matched extracted skill raises success on 100 held-out EconWebArena variants from 41.0% to 49.3% (McNemar p=0.0031) and shortens jointly successful episodes by 2.37 steps on average.
- Parameterized abstraction is essential: replaying the raw verified source trajectory with only the answer removed collapses success to 7.0%.
- At library scale, retrieving five skills (RETR5, 28.3%) is statistically tied with no-skill prompting (28.1%) across all 360 tasks, with numerical gains concentrated on the 150 directly covered tasks (48.0% vs 44.0%).
- Delivering the full 50-skill library (11.4%) or an approximate one-skill hint on uncovered tasks (16.7% to 11.4%, p=0.043) degrades performance, evidencing context interference and a coverage-rejection gap.
- Coverage-aware selection — withholding skills whose preconditions do not match the task — is the primary design target for converting stored procedures into end-to-end gains.
Strengths
Two-level design cleanly decouples transfer from selection
Section 3.3 splits the evaluation into controlled matched-skill transfer (BASE/TRAJ1/MATCH1 on 100 held-out variants, 900 episodes) and library-scale delivery (BASE/ALL50/RETR5/RAND5/HINT30 on all 360 tasks), directly addressing the conflation of stored-procedure quality and retrieval quality the authors criticize in prior skill evaluations (Section 2).
Honest reporting of null and negative results
Table 4 prominently reports that RETR5 (28.3%) only ties BASE (28.1%) overall (McNemar p=1.0), that ALL50 (11.4%) is far worse (p<10^-12), and Table 6 shows an approximate hint lowers success on 210 uncovered tasks from 16.7% to 11.4% (p=0.043) — the coverage-stratified reporting is a model for skill-evaluation papers.
Statistically careful controlled study
Decisive abstraction ablation
Public artifacts enable reuse
Weaknesses
Library-scale conclusions rest on a single run per task per condition
Single backbone and single extraction pipeline
No external experience-reuse baselines
Unisolated confounds in library-scale conditions
Variant construction and matching are author-defined
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EconSkills (arXiv 2609.19523; Quan and Liu) targets a concrete inefficiency in LLM web-agent evaluation: agents revisit the same authoritative portals yet discard procedures learned from earlier successes. The authors distill 50 skills from verified EconWebArena trajectories — each a seven-part SOP (name, purpose, preconditions, ordered procedure, site-specific guidance, verification, recovery) with instance values replaced by placeholders — and pose two separate questions: does a known matched skill transfer to held-out instances, and does that benefit survive selection from a library?
The headline claims, all traceable to specific tables: (1) matched skills lift success from 41.0% to 49.3% on 100 held-out variants (Table 3; McNemar p=0.0031) and cut 2.37 steps on jointly successful runs; (2) abstraction is decisive, since replaying the raw verified trajectory collapses to 7.0%; (3) at library scale, five retrieved skills (RETR5, 28.3%) only tie no-skill prompting (BASE, 28.1%) across 360 tasks, leading only on the 150 directly covered tasks (48.0% vs 44.0%, Table 4); (4) full-library delivery (ALL50, 11.4%) and approximate hints on 210 uncovered tasks (16.7% to 11.4%, p=0.043) actively hurt. The framing as 'coverage-aware selection' as the remaining bottleneck is well supported qualitatively and stated with appropriate hedging.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EconSkills?
EconSkills is a library of 50 reusable skills for LLM web agents, distilled from successful runs on EconWebArena, a benchmark of 360 tasks on real economic-data websites (central banks, statistics offices, international organizations). Each skill is a seven-part standard operating procedure — purpose, preconditions, navigation steps, site-specific guidance, verification checks, and recovery steps — with task-specific values like country, date, or indicator replaced by placeholders so the same procedure can serve new questions on the same portal.
Do skills actually improve web-agent performance?
Only under specific conditions. When the correct skill is handed to the agent for a related held-out task, success rises from 41.0% to 49.3% and jointly successful runs take about 2.4 fewer steps. But when the agent must rely on retrieval from the library, five retrieved skills (28.3%) merely tie the no-skill baseline (28.1%) across all 360 tasks; gains appear only on tasks directly covered by the library (48.0% vs 44.0% on 150 tasks).
Why does giving the agent the entire skill library hurt?
Injecting all 50 skills drops success from 28.1% to 11.4%. The authors attribute this to context interference — many plausible-but-inapplicable procedures compete for the model's attention. A related harm appears when an approximate, non-matching skill is injected on tasks outside the library's coverage: success falls from 16.7% to 11.4%, which motivates coverage checks before inserting a skill into the prompt.
Is EconSkills publicly available?
Yes. The 50 skills are on Hugging Face (EconWebArena/EconSkills), and experiments can be rerun with the open-source BrowserGym and AgentLab tooling using gpt-5-mini under the described prompt conditions. Caveat: because tasks run on live websites, exact replication depends on the current state of those sites; the paper patched four gold values that changed at their sources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are browser screenshots; the qualitative case evidence (e.g., ONS task 225, USCIS task 264) is assessed from captions and prose only.
- The identity of the skill-extraction model and the full variant-construction protocol are not given in the extracted text, so matching validity cannot be independently checked.
- Live-benchmark numbers are time-sensitive; this review reflects the September 2026 preprint as written.
- Appendix content appears complete in the extraction, but minor truncation of tables cannot be fully ruled out.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.