AutoTailor: Automatic, User-Aligned Capability Selection and Adaptation for Web Agents
Xinyun Cao, Adriana Szekeres, Fazle Elahi Faisal
cs.AI · cs.SE
Quick verdict
AutoTailor cuts web-agent tool sets 1283→33 APIs, saving ~58% tokens at higher accuracy, but SDR is calibrated on the eval tasks and no variance is reported.
Summary
AutoTailor is a meta-agentic framework that builds and maintains a compact inventory of trajectory-derived MCP tools for web agents. Offline, web trajectories are converted into parameterized Playwright programs; a Quality Filter removes poor-granularity and duplicate APIs (embedding clustering at 0.75 similarity plus LLM duplicate verification), and a Usage Likelihood Filter keeps LLM-rated useful APIs while guarding cluster coverage, shrinking 1,283 initial APIs to 215 and then 87. Online, Dynamic Reselection logs usage, detects recurring coverage gaps, retrieves candidates from the quality-filtered pool, and prunes unused APIs, yielding a 33-API set. On 106 WebArena Postmill tasks with a Claude Opus 4.8 tooling agent, the reselected set with ReAct fallback reaches 90.6% correctness versus 87.5% for ReAct alone, cutting average total request tokens by 57.8% and latency by 29.4%; without fallback it retains 60.1% correctness (vs 60.4% unrefined) with 94.9% fewer request tokens. The paper honestly reports that aggressive static filtering drops no-fallback coverage to 50.2%. Main limitations: the 33-API set is calibrated on the same 106 tasks used for evaluation, results are single-run point estimates with no variance or significance testing, and evidence is limited to one forum benchmark and one backbone; no code release is mentioned.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Offline filtering (QF then ULF) compresses 1,283 trajectory-derived APIs to 215 then 87 while keeping ReAct-fallback correctness near the unrefined set (87.3% vs 88.0% on 106 WebArena Postmill tasks).
- Dynamic Reselection yields a 33-API set that is the best-performing condition with ReAct fallback (90.6% vs 87.5% ReAct-only), cutting average total request tokens by 57.8% and latency by 29.4%.
- Without fallback, the 33-API set nearly restores unrefined-set coverage (60.1% vs 60.4%) using 94.9% fewer request tokens, recovering most of the coverage lost by ULF (50.2%).
- Static filtering trades compactness against reliance on the ReAct fallback, and usage-driven online adaptation resolves that trade-off by tailoring the set to observed demand.
Strengths
Staged ablation design
Tables 1-2 evaluate every pipeline stage (Sinit → SQF → SULF → SDR) both with and without ReAct fallback, letting readers attribute each efficiency or coverage change to a specific filter rather than to the full system.
Transparent trade-off reporting
Section 4.4 explicitly reports that ULF's tooling savings (403.30k → 205.13k request tokens) are offset by fallback cost, raising total request tokens from 1.42m to 1.52m, and that no-fallback coverage drops from 59.9% to 50.2% — honesty uncommon in system papers.
Full prompt disclosure
Fully specified online parameters
Weaknesses
Calibration-evaluation circularity
No statistical uncertainty
Narrow external validity
Underspecified metric and cost accounting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AutoTailor (arXiv 2609.13548; Cao, Szekeres & Faisal, U. Michigan / Microsoft Research) addresses capability-inventory management for web agents: given thousands of automatically discovered candidate behaviors, which should become deployed MCP tools, and how should that set track user demand? The offline stage crawls trajectories (per Faisal et al. 2026), has an LLM convert each into a parameterized Playwright function, then applies a Quality Filter — LLM granularity scoring (A.1) plus greedy embedding clustering at cosine 0.75 with LLM duplicate verification above 0.80 (A.2) — and a Usage Likelihood Filter — LLM 1-5 usage ratings retaining ≥4 plus the best ≥3 API in any otherwise-uncovered cluster (A.3). The online Dynamic Reselection stage (3.1.1) logs usage, detects recurring coverage gaps over a 100-task rolling window (clustering at 0.82), retrieves up to 5 candidates per gap from the quality-filtered pool via LLM (A.4), and prunes APIs unused for 50 tasks (50-task grace, 10-API floor). Claims: 1,283 → 215 (SQF) → 87 (SULF) → 33 (SDR) APIs, with SDR plus ReAct fallback the best and cheapest condition on 106 WebArena Postmill tasks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AutoTailor in one sentence?
AutoTailor is a meta-agent framework that turns web-browsing trajectories into a compact, deduplicated set of MCP tool APIs (from 1,283 candidates down to 33) and adapts that set online to match observed user demand.
How much does AutoTailor improve accuracy and cost?
On 106 WebArena Postmill tasks with Claude Opus 4.8, the final 33-API set with ReAct fallback reached 90.6% correctness versus 87.5% for ReAct alone, cutting average total request tokens by 57.8% and latency by 29.4%; without fallback it kept 60.1% correctness with 94.9% fewer request tokens.
Is AutoTailor's code or data available?
The reviewed paper text does not mention a code or data release. The filtering and reselection prompts are fully printed in Appendices A.1-A.4, and the benchmark (WebArena Postmill) is public, but the trajectory corpus and final API list are not published.
What is the biggest limitation of the evaluation?
The final 33-API set was calibrated by replaying the same 106 tasks later used for scoring, and results are single runs without variance estimates — so the headline gains may partly reflect fitting to the test distribution rather than generalization.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 1-2 are flattened into runs of numbers by the PDF extraction; column-to-value alignment was cross-checked against the §4.4 narrative, but the original PDF should be consulted for exact table layout.
- Page numbers (e.g., '17') are embedded mid-text by the extraction (notably inside Appendix A.4); this is extraction noise, not missing content — the text runs complete through the final appendix.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.