M-SQE: Multilingual Skill Quality Estimation for Enhancing Language Equality in Agentic Skill Use
Yilun Liu, Shimin Tao, Minggui He, Chenxin Liu, Li Zhang, Chen Liu, Miao Zhang, Jiaxin Guo, et al.
cs.CL
Quick verdict
Proposes a two-view (intrinsic-quality plus task-fit) LLM scoring layer for multilingual agent-skill selection with unusually careful evaluation; gains are consistent and significance-tested, but scorer-solver same-model circularity and unvalidated design constants temper the headline claims.
Summary
The paper documents that community agent-skill libraries are English-dominated (about 84,700 audited entries; zero genuine in-language skills for Swahili and Hindi) and proposes M-SQE, a post-retrieval quality estimation layer. Each retrieved candidate is scored by an LLM on a Theory view (six intrinsic-quality dimensions built by a language-service expert team, with red-line/basic caps at 40/80) and an Action view (six query-grounded fit dimensions plus misleading risk); a 5-shot domain router fuses the scores differently for general (0.6/0.4 convex), tool-use (Theory>=65 lexicographic guard) and cultural (z-score fusion with retrieval) tasks. Evaluation spans 411 tasks across three domains and six-to-seven languages, with three-layer pools (ecological-style, machine-translated, self-generated; 1,750-5,285 skills) and three retrievers. M-SQE is first in all nine domain-by-retriever settings (69.1% average at Top-3, +3.5pp over SkillFlow, +6.3pp over retrieve-only), with paired sign-flip tests and bootstrap CIs reported; gains concentrate on Hindi (+12.9pp) and Swahili (+5.6pp) in tool use. Weaknesses: scorer, router and solver are the same LLM (Gemini-3-Flash) in headline runs; fusion constants were not tuned on held-out data; per-language/region gains lack CIs; the cultural domain has only 52 tasks; and compute cost of two LLM calls per candidate is unreported.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Community agent-skill ecosystems are overwhelmingly English: among ~84,700 audited entries, Swahili and Hindi have zero genuine in-language skill content, so cross-lingual retrieval degrades agent performance.
- Retrieval relevance does not imply skill usability, and a post-retrieval quality estimation layer scoring intrinsic quality (Theory) and query-grounded utility (Action) closes this gap.
- M-SQE achieves first or tied-first task success in all nine domain-by-retriever settings (69.1% average at Top-3), exceeding the strongest baseline by +3.5pp and retrieve-only by +6.3pp, with all differences statistically significant.
- Gains are largest for the lowest-resource languages (+12.9pp Hindi, +5.6pp Swahili in tool use) and hold across all six cultural regions, supporting claims of improved linguistic and cultural equality.
- Skills selected by M-SQE also make better fine-tuning material: a Qwen3.5-9B agent trained on M-SQE trajectories outperforms those trained on baseline selections (Table 5).
Strengths
Novel, well-motivated problem grounded in a quantitative ecosystem audit
Appendix A's audit of ~84,700 community skill entries (script-based in-language detection plus manual verification) quantifies the English dominance motivating the work, including zero in-language skills for Swahili and Hindi; the audit protocol, thresholds and precision caveats are disclosed rather than asserted.
Evaluation rigor above the field's norm
All headline outcomes use deterministic checkers (exact string/slot match, Appendix G) instead of LLM judges; leakage control is explicit (verbatim scans, 8-gram audits, anonymous skill bodies with provenance stripped, Section 5.2/Appendix E); Prompt-only anchors well below ceiling (39.9% average) show tasks genuinely require skills.
Consistent, significance-tested improvements across all nine settings
Unusually thorough robustness and mechanism analyses
Weaknesses
Same-model circularity between scorer, router and solver
Design constants not validated on held-out data
Headline per-language equity claims are cherry-picked cells without uncertainty
Cost and baseline strength under-reported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is M-SQE and what problem does it solve?
M-SQE (Multilingual Skill Quality Estimation) is a post-retrieval scoring layer for LLM-agent skill libraries. Because community skill collections are overwhelmingly English, queries in languages like Hindi or Swahili retrieve skills written in other languages, and translated or auto-generated skills are often relevant-looking but broken. M-SQE has an LLM score each retrieved candidate on intrinsic quality (a 'Theory' view) and on fit to the current task (an 'Action' view), then combines the scores differently for general, tool-use, and cultural tasks so the agent executes a skill that actually works.
How much does M-SQE improve agent task success?
Across 411 tasks in three domains (general procedures, tool/function calling, and cultural questions) and three retrievers, M-SQE ranked first in all nine domain-retriever settings, averaging 69.1% task success at Top-3: +3.5 points over the strongest baseline (SkillFlow) and +6.3 points over plain retrieve-only ranking. The lowest-resource languages benefited most, with +12.9 points on Hindi and +5.6 points on Swahili in tool-use tasks, and all aggregate improvements were statistically significant with bootstrap confidence intervals.
Why do Hindi and Swahili benefit most from M-SQE?
The paper's audit of roughly 84,700 community skill entries found zero genuine in-language skills for Hindi and Swahili, so any skill pool for those languages consists of machine-translated or model-generated material, which is error-prone. That makes post-retrieval quality filtering most valuable exactly there: usable translated or generated candidates exist but are buried among relevant-looking defective ones, and M-SQE's two-view scoring surfaces them.
Which view matters more, Theory or Action?
The ablation (Table 4) shows the Action view (task-grounded fit) carries most of the signal: Action-only reaches 71.3/38.5/88.5 on general/tool-use/cultural tasks versus Theory-only's 67.0/28.7/59.6, while the full M-SQE beats both (74.5/39.6/90.4). The Theory view acts mainly as a guard against superficially good-looking but catastrophically flawed skills, such as a note-style candidate that names the right function but cannot be executed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Fig. 3-6) were extracted as fragmented axis labels, so exact per-language and per-region values could not be independently verified; the review relies on the values reported in prose.
- Table 6's labels are garbled by extraction (column headers merged); its numeric deltas were cross-checked against Section 5.5 and Appendix B prose.
- Appendices D-I were partially truncated in extraction; rubrics and prompts were available but some construction details could not be fully checked.
- No network access was used; open-source artifacts at the stated GitHub repository were not inspected, so reproducibility is judged from the paper's own release claims.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.