Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agents
Tingyu Qu, Weigao Sun, Yuecheng Liu, Yucheng Zhao, Yi Zhu, Yifeng Ding, Qiyi Wang, Sihan Cao, et al.
cs.AI
Quick verdict
Closed-loop AI-for-AI pipeline trains a 27B mobile planner that tops MobilePA-Bench (77.05%), yet gains lack significance tests on a self-built benchmark.
Summary
This industrial technical report presents Qwen-Planner-Agent, a mobile-task agent coupling a trained planner model (35B-A3B MoE and 27B dense Qwen variants) with a runtime harness offering skills, persistent memory, and structured execution feedback, developed inside a closed-loop 'AI-for-AI' pipeline. Three stages interlock: AI-assisted task construction and trajectory curation feed a supervised cold start; hybrid online reinforcement learning across programmatic sandboxes, LLM-simulated environments, and real devices uses CARE, which shifts rewards from progress-shaping to success-consolidation to efficiency-refinement as group success rises, and floors the GRPO advantage denominator so tiny efficiency differences are not amplified once success saturates; diagnosed failures feed back into harness revisions. On the team-built MobilePA-Bench (1,700+ tasks), the 27B system scores 77.05% overall versus 76.84% for GPT 6 Astra, at an estimated $2.41 per 1,000 tasks in output tokens; co-evolution adds +5.83/+8.98 points on internal mobile and MCPMark sets, and the memory harness lifts BEAM-10M mean from 23.42% to 61.66%. Limitations: the headline benchmark is self-authored and baseline parity is unspecified, the 0.21-point lead has no significance testing or variance reporting, no code or weights are released, and human reviewers gate every update, making the loop less autonomous than the branding suggests.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Qwen-Planner-Agent 27B achieves the highest Overall score on MobilePA-Bench (77.05%) among evaluated frontier models and systems, at the lowest estimated output cost ($2.41 per 1,000 tasks).
- CARE (competence-aware reward scheduling plus advantage-calibration floor) preserves task accuracy while reducing output tokens by 32.5% relative to vanilla RL.
- Alternating model training and harness revision adds +5.83pp on MobilePA-Internal (4 iterations) and +8.98pp on MCPMark (3 iterations) over the model-only baseline.
- The persistent-memory harness lifts long-history performance across all matched Qwen pairs (BEAM-10M mean from 23.42% to 61.66%) while leaving short-history scores largely unchanged.
- Planner training transfers beyond mobile tasks, improving 6/7 (35B-A3B) and 5/7 (27B) non-mobile agentic benchmarks with only small drops on MMLU-Redux, C-Eval, and IFEval.
Strengths
Coherent system contract with clean attribution experiments
The shared action-feedback-verification formulation (Eqs. 1-3) unifies data collection, RL, and harness behavior; paired Table 1 rows hold the 27B checkpoint fixed and isolate the Harness effect (71.90 -> 77.05 Overall, +5.03 Skills), and §2.1.1 guarantees the held-out dev set is excluded from training.
CARE is mathematically grounded, not just tuned
Eqs. 7-8 derive that GRPO normalization drives the effective efficiency coefficient to ~1/sigma_e regardless of lambda_eff once success saturates; the sigma_anchor = sqrt(p_high(1-p_high)) denominator floor (Eq. 9) fixes this while preserving composed-reward rankings, and the Fig. 6 ablation confirms it (CARE w/o calibration: shorter outputs but substantially lower accuracy; full CARE: -32.5% tokens at comparable accuracy).
Honest cost scoping
Candid framing and negative results
Breadth of evaluation with failure-aware protocol
Weaknesses
Self-referential headline evaluation
No statistical treatment anywhere
Unverifiable end-to-end
Cost comparison structurally favors the harness system
Co-evolution baseline provenance unclear
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
An industrial technical report (arXiv v1, Alibaba MAI Team / Token Hub, ~40 contributors) presenting Qwen-Planner-Agent: Planner Models trained on Qwen backbones (35B-A3B MoE and 27B dense) coupled with a deployment-time Harness (Scenario Adapter, tool-conditioned Skills, persistent Memory, structured execution feedback), developed through a three-stage 'AI-for-AI' lifecycle. AI for Data (§2.3) runs an agentic task-construction/trajectory-curation flywheel with human-gated releases; AI for Training (§2.4) combines an SFT cold start (turn-level error masking, Eq. 4) with hybrid-environment online RL and the proposed CARE scheme; AI for Harness (§2.5) feeds failure diagnoses into versioned model and harness updates. Five headline claims: best Overall on MobilePA-Bench (77.05%, Table 1); lowest estimated output cost ($2.41/1K tasks); CARE cuts output tokens 32.5% at comparable accuracy (Fig. 6); co-evolution adds +5.83/+8.98 points on MobilePA-Internal/MCPMark (Table 2); memory harness lifts BEAM-10M mean from 23.42% to 61.66% (Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Qwen-Planner-Agent and what does it achieve?
It is an Alibaba system that couples a 27B trained planner model with a runtime harness (skills, persistent memory, structured execution feedback) for mobile-phone tasks. The paper reports the highest overall score on its MobilePA-Bench (77.05%), above GPT 6 Astra (76.84%) and Claude Opus 5 (75.71%), at an estimated $2.41 output-token cost per 1,000 tasks.
What is CARE in this paper?
CARE (Competence-Aware Reward-and-Advantage Engineering) is a reinforcement-learning technique: as a task group's success rate rises, rewards shift from progress-shaping to success-consolidation to efficiency-refinement, and the GRPO advantage denominator gets a floor so tiny efficiency differences are not amplified once nearly all rollouts succeed. It reportedly cut output tokens ~32.5% at comparable accuracy.
Is the code, model, or benchmark publicly available?
No release is stated in the paper. MobilePA-Bench is described in a companion paper by an overlapping team; MobilePA-Internal, the harness, and the model weights are internal. Reproducing the results currently requires proprietary infrastructure.
Is the 'AI-for-AI' loop fully autonomous?
No. AI agents propose tasks, data changes, and harness edits, but human reviewers gate every data release and update, and the LLM controller only tunes four bounded reward-schedule parameters. The authors themselves call it a 'development pathway' rather than established autonomous co-evolution.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text omits parts of §2.2.3, §2.5.2, the Table 4 body, and portions of §3.5/case studies; judgments on those areas rely on surrounding prose.
- Figures 1, 6, and 7 were read from OCR'd figure text; curve shapes and exact axis ranges could not be verified.
- arXiv v1 preprint, not peer-reviewed; the compared 2026 commercial systems cannot be independently checked from this text.
- No explicit code/data availability statement was found in the extracted text; codeAvailable=false reflects absence of any stated release.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.