From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents
Longtao Hu, Xiao Liang, Linchao Zhu
cs.AI
Quick verdict
Claims evolving skill libraries beat an empty-library control on OSWorld (+5.7–18.6pp); honest about single-run confounds, but causal evidence is descriptive.
Summary
The paper presents an online skill-evolution framework for computer-use agents: interaction trajectories and evaluator feedback are converted into a persistent, versioned library of textual skills, while the Executor (EvoCUA-32B) and GUI-grounding model (MVP) stay frozen. Each iteration runs a fixed OSWorld task set against a frozen library snapshot; a Kimi K2.5 Proposer diagnoses structured evidence and proposes skill creation, edit, deletion, or no-op, with accepted mutations Git-committed and exposed only at the next iteration. Against a configuration-matched empty-library control across four OSWorld domains (GIMP, VLC, LibreOffice Writer, Thunderbird), Full attains higher post-warm-up mean evaluator scores in all four observed runs: differences of 5.7–18.6 percentage points (GIMP +5.7, VLC +18.6, Writer +11.8, Thunderbird +12.4). GIMP provenance analysis (27 skills; 82.4% of rollouts retrieve skills) shows 43.3% of retrievals cross task-of-origin boundaries, while 'revision churn' leaves the origin task of repeatedly edited gimp-add-alpha-channel successful in only 2 of 35 iterations. The authors acknowledge the main limitations: one unreplicated run per condition–domain pair (no variance estimate or significance testing), pre-warm-up score divergence between conditions confounding causal attribution (Writer already +9.1 before skills exist), evaluation only on repeated fixed task sets with no unseen-task transfer, and detailed provenance restricted to GIMP.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An online skill-evolution framework can convert interaction trajectories and evaluator feedback into a persistent, versioned procedural library without changing model parameters.
- Against a configuration-matched empty-library control, Full attains higher post-warm-up mean evaluator scores in all four observed OSWorld domain runs, by 5.7–18.6 percentage points.
- In GIMP, 43.3% of skill retrievals cross task-of-origin boundaries, supporting the library's role as shared procedural memory.
- Repeatedly accepted skill edits do not guarantee task recovery: the repeatedly edited gimp-add-alpha-channel's origin task succeeds in only 2 of 35 iterations (revision churn).
Strengths
Configuration-matched longitudinal control design
§4.1: Full and Empty share Executor (EvoCUA-32B), Grounding (MVP), task sets, task order, iteration horizons, prompt template, catalog interface, and get_skill tool; the control keeps the identical skill interface with an empty catalog, isolating library content rather than interface overhead.
Unusual candor about evidence limits
§4.2 explicitly labels results 'a descriptive comparison... not a significance test', flags Writer's +9.1 pp pre-period offset as making 'most of its +11.8-point post-warm-up separation' unattributable, and §5 lists single runs, warm-up imbalance, and within-set adaptation as limitations.
Telemetry-verified provenance analysis
Mutation discipline mechanisms
Negative results reported alongside positives
Weaknesses
Single unreplicated run per condition–domain pair
Warm-up imbalance confounds the headline comparison
No static-library or evolution ablation
Within-set adaptation only; transfer untested
Detailed provenance limited to GIMP
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: From Interaction Traces to Persistent Skills
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much does the evolving skill library improve the agent compared to no library?
In the paper's single run per condition across four OSWorld domains, the full system scored 5.7–18.6 percentage points higher than the empty-library control after a five-iteration warm-up (GIMP +5.7, VLC +18.6, LibreOffice Writer +11.8, Thunderbird +12.4). The authors note these are unreplicated descriptive comparisons, and in Writer most of the gap already existed before any skills were created.
Does the agent learn or fine-tune any model parameters?
No. The action-generation model (EvoCUA-32B) and GUI-grounding model (MVP) stay frozen throughout. All adaptation happens through an external, Git-versioned library of textual SKILL.md procedures created and edited by a Kimi K2.5-based pipeline and exposed to the agent only at the start of the next iteration.
What is 'revision churn' in this paper?
It is the pattern where continued task failures keep triggering accepted skill edits without improving the task. In GIMP, the skill gimp-add-alpha-channel received repeated accepted edits, yet its originating background-transparency task succeeded in only 2 of 35 iterations — suggesting the bottleneck lies in action generation or GUI grounding rather than missing knowledge.
Is the code available?
Yes. The paper links to github.com/LongtaoHu/Skill-Evo4GUI, and experiments run on the public OSWorld benchmark (v0.1.16-934-g8f41f80a) in Docker. Reproduction requires the same model stack (EvoCUA-32B, MVP, Kimi K2.5) and substantial compute, and single runs are stochastic.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's per-iteration trajectory values are not recoverable from the extracted text; temporal-stability statements (e.g., 'repeated crossings' in GIMP) rely on the authors' captions, with Table 2 providing only pre/post aggregates.
- The code repository's contents were not verified (no network access); code availability reflects the paper's stated link.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.