Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation
Bingxin Xu, Yuzhang Shang, Zhen Dong, Emilio Ferrara
cs.RO · cs.AI · cs.CL · cs.CV
Quick verdict
Claims coding agents fail at safe manipulation and adds a plan-verify harness reaching 87.5% CAR on SafeLIBERO; convincing ablation, but stats are weak.
Summary
The paper asks whether coding agents that write robot controllers are safe when a task carries an obstacle constraint, and answers no: on SafeLIBERO, an unmodified Harness-VLA-style agent completes tasks but displaces the obstacle in most episodes, even though it names the obstacle in its traces and the prompt forbids contact. The authors attribute the failure to planning — the constraint never becomes a priority — and link it to long-context degradation of instruction adherence. SafeHarness adds two training-free components per manipulation phase: obstacle-aware route planning (SAM-3 bounding boxes, waypoint route proposal, out-of-model geometric verification, replanning at verification failure or mid-execution halt) and obstacle-aware contact execution (adjacency test choosing a far-side approach, tangent gripper orientation, or vertical lowering). With GPT-6 the system reports 71.9% task success and 87.5% collision avoidance versus 67.5/68.9 for the barrier-filter SOTA AEGIS; ablations show the harness, not model scale, drives safety, lifting GPT-6 from 31.0/59.0 to 71.9/87.5. Limitations: single axis-aligned obstacle per scene, parallel-jaw gripper tuning, simulation only, no variance or significance reporting, and internal arithmetic inconsistencies in Table 1.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Coding agents for manipulation complete tasks but displace the obstacle in most episodes; the failure is planning-side — the safety constraint never becomes a priority — not perception or instruction.
- Prompt-stated constraints decay over long-horizon episodes; stateless external verification tools re-apply the constraint more reliably than instructions held in context (Sec. 3.4).
- SafeHarness reaches 71.9% task success and 87.5% collision avoidance on SafeLIBERO with GPT-6, surpassing the prior SOTA AEGIS on both axes.
- Safety gains are a property of the harness rather than the model: harnessed GPT-5.5 and GPT-6 land within two points of each other on both metrics (Table 2).
Strengths
First safety-scoring of the coding-agent paradigm
Frames and measures a previously unasked question; Table 1 baseline rows quantify the success-safety gap sharply (π0.5: 57.8 TSR vs 17.1 CAR; OpenVLA-OFT: 26.2 vs 5.7), and the agent is evaluated on a collision-scored benchmark for the first time in this line.
Correct ablation design separating harness from model scale
Table 2 crosses two planners x three agent tiers: GPT-6 skills-only 31.0/59.0 becomes 71.9/87.5 with harnesses, and the harnessed planners land within two points of each other, so safety stops tracking model choice — the paper's central causal claim is properly tested.
Mechanistic diagnosis, not just a fix
Honest accounting of residual failure
Per-phase design makes the method practical
Weaknesses
Statistical reporting absent and internally inconsistent
No classical-planner control
Overclaim: safety 'guaranteed'
Ambiguous SOTA margins; modest TSR gain
Under-specified implementation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether the coding-agent paradigm for robot manipulation — an LLM that writes the controller as a program and calls tools — is safe when a task carries an obstacle constraint, and contributes SafeHarness, a training-free harness evaluated on SafeLIBERO (32 tasks, 10 seeds each). Central claims: (1) an unmodified Harness-VLA-style agent completes tasks but displaces the obstacle in most episodes (Sec. 1; Table 2 skills-only rows: 28.0-31.0 TSR vs 31.0-59.0 CAR); (2) the fault lies in planning, not perception or instruction — the agent names the obstacle in its traces and the prompt forbids contact — with Sec. 3.4 attributing it to long-context decay of instruction adherence; (3) two obstacle-aware components (route planning with out-of-model verification; contact-side selection) lift GPT-6 to 71.9% TSR / 87.5% CAR, beating AEGIS; (4) safety is a property of the harness, not the planner.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SafeHarness in robot manipulation?
SafeHarness is a training-free add-on for coding agents that control robots by writing programs. It adds obstacle-aware route planning — objects are segmented into boxes, routes are planned as waypoints and geometrically verified outside the language model, with replanning on failure — and obstacle-aware contact execution that picks grasp and place sides away from adjacent obstacles. This makes safety a checked property of each motion rather than an instruction the model must remember.
How well does SafeHarness perform?
On the SafeLIBERO simulation benchmark (32 tasks, 10 seeds each), SafeHarness with GPT-6 reaches 71.9% task success and 87.5% collision avoidance, versus 67.5%/68.9% for the prior best method AEGIS. An ablation shows the same agent without the harness scores 31.0%/59.0%, so the harness — not the model — drives the safety gain.
Was SafeHarness tested on a real robot?
No. The evaluation is simulation-only, on the LIBERO/SafeLIBERO simulator, with a parallel-jaw gripper and a single obstacle per scene. The authors list real-robot validation, multi-obstacle scenes, and grounding robustness as open limitations.
Why do LLM robot agents ignore safety rules written in the prompt?
The paper argues that over a long episode the safety sentence recedes into tens of thousands of tokens of tool calls and images, and instruction adherence degrades with context length, so the constraint is stated early but never re-applied. SafeHarness instead re-imposes the constraint at every phase via external, stateless verification tools.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only as captions; their content (e.g., Fig. 1 bar values) could not be cross-checked against tables
- No appendix appears in the extracted text; harness prompts, box margins, and replan budgets may exist in supplementary material that was not provided
- Reported baselines (AEGIS, SafeLIBERO protocol) are taken from the paper's description of external arXiv artifacts that could not be verified here
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.