EnvCraft: Synthesizing Executable Environments in Agentic RL for Claw-like Agent

Summary

EnvCraft is an automated pipeline that synthesizes executable, sandbox-isolated training environments and verifiable RL tasks for 'Claw-like' system agents (OpenClaw-style harnesses operating over files, shells, and databases). An environment-synthesis engine expands 83 curated scenarios (41 general tool-use, 42 Claw-specific) into specifications, generates Python sandboxes guided by nine interaction archetypes, and enforces static/dynamic audits with an LLM self-healing loop (324 raw environments reduced to 139, a 42.9% pass rate). A data engine builds bi-level tool/environment dependency graphs, samples mutation-terminating tool chains via weighted random walks, and uses LLMs to reverse-engineer information-hiding multi-turn user intents, initial states, and deterministic verification scripts; LLM rubric auditing plus Qwen3.5-27B difficulty calibration (keeping tasks whose pass rate over eight rollouts lies strictly between 0 and 1) retains 19,777 tasks. GRPO training on Qwen3-8B/32B and Qwen3.5-9B yields up to +11.91 points on PinchBench and +11.36 on ClawEval (both Qwen3-8B), with per-task token cost down 35% for that model, and transfers to BFCL-v3 (up to +8.00 on Qwen3.5-9B). Notably, the paper's own Table 4 shows ScaleEnv-trained models outperform EnvCraft-trained models on τ²-bench at both tested scales. Key limitations: rewards derive entirely from LLM-written, LLM-audited verifiers; statistical reporting is internally inconsistent (three vs. eight runs; 32 vs. 64 turns); and no human audit of verifier correctness or train/eval overlap analysis is reported.