CROCODIL: Cross-Model Code Editing with LLMs
Linghan Zhong, Aditya Thimmaiah, Jayanth Srinivasa, Milos Gligoric, Junyi Jessy Li
cs.CL · cs.SE
Quick verdict
LLMs over-edit foreign code; CROCODIL's GRPO reward halves Olmo3 7B edits while preserving success. Good corpus; gains lack statistical tests; one base model.
Summary
The paper asks whether LLMs edit code differently when the pre-edit code was authored by another LLM. Building a corpus of 8,235 functions from 4,514 Rust PRs across 356 repositories, the authors have five models (Qwen3.5 35B A3B, GPT-OSS 20B, Olmo3 7B, Olmo3.1 32B, Haiku 4.5) implement the pre-edit functions, verified by collected tests, then have every model apply each PR's edit to every implementation. Character-level Levenshtein analysis shows the four open-weight editors make smaller edits on their own code on 14 of 16 model pairings, and prompt-based minimal-edit instructions fail to close the gap. CROCODIL then post-trains Olmo3 7B with GRPO using a product reward: a similarity term normalized by the developer's original edit size, and an execution term for build and test outcomes. The trained model roughly halves edit distance on implementations by every model (e.g., 0.54 to 0.20 normalized on Qwen implementations) while improving build and test pass rates on every implementor except itself, and remains smaller on both-pass-restricted subsets. Limitations: Rust-only, function-level scope, only one trained model, training tasks use a single implementor (Qwen), no reward ablations, no significance testing of training gains; some restricted comparisons rest on as few as 4 tasks.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Four open-weight LLMs make smaller edits on code they authored themselves than on foreign code: self-edits are smaller on 14 of 16 implementor-editor pairings, up to 14% less.
- A system prompt demanding minimal edits cannot consistently reduce edit size or improve edit success.
- CROCODIL (GRPO with a similarity-times-execution reward) roughly halves Olmo3 7B's edit distance on implementations by every implementor while improving build and all-test pass rates on every implementor except Olmo3 7B itself.
- On tasks where both the base and the trained model pass all tests, the trained model still edits less, indicating genuine reduction of non-task edits rather than quality loss.
- The similarity penalty has a documented failure mode: the trained model can under-edit, for example failing to propagate an identifier rename.
Strengths
Novel, practically motivated research question
Multi-model development workflows motivate the study (Liang et al. 2024; Pyo et al. 2026); no prior benchmark varies the author of the pre-edit code, so the authors build the implementor-x-editor matrix themselves (Sections 1, 3).
Test-anchored corpus construction
Tests are harvested from base/head/max commits and filtered by a line-coverage collector; 2,458 functions retained with mean 87.9% and median 94.5% coverage of the developer's post-edit function (Section 3.1).
Deliberate style-leak suppression
Confound-aware evaluation of the RL results
Honest failure-mode reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether LLM editors behave differently on code they authored themselves (self-edit) versus code authored by another LLM (cross-edit), and proposes CROCODIL, a GRPO-based post-training framework that penalizes excessive edits. Three central claims: (1) four of the five studied models edit foreign code more, with self-edits smaller on 14 of 16 implementor-editor pairings, 'up to 14% less' (Section 3.4); (2) a strict minimal-edit prompt cannot consistently shrink edits or raise success (Section 6.1, Table 4); (3) CROCODIL roughly halves Olmo3 7B's edit distance on implementations by every implementor while improving build and test pass rates on every implementor except itself (Tables 4-5). The authors scope these claims honestly to Rust, function-level edits, and a single trained base model (Section 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CROCODIL?
CROCODIL is a post-training framework for code-editing LLMs that uses GRPO reinforcement learning with a combined reward: a similarity term that penalizes large edits relative to the developer's original change size, multiplied by an execution term scoring build and unit-test success. It trains a model to make fewer unnecessary changes while keeping edits functionally correct.
Do LLMs edit other models' code differently from their own?
Yes, according to this paper: four of the five studied models make smaller edits on code they generated themselves. Self-edits are smaller than cross-edits on 14 of 16 implementor-editor pairings, and Olmo3 7B edits foreign code roughly twice as aggressively. Telling the model via prompt to edit minimally did not reliably fix this.
Does CROCODIL reduce edit quality?
The paper reports the opposite on foreign code: after training, Olmo3 7B's build success and test pass rates improved on implementations by every model except itself, and on tasks where both the base and trained models pass all tests, the trained model still makes smaller edits. The main downside is occasional under-editing, such as failing to propagate an identifier rename.
What data and programming language does CROCODIL use?
Rust only. Functions were extracted from real GitHub pull requests of crates.io projects, with tests harvested from the PR's base, head, and later main-branch commits. Training used 355 tasks where Qwen3.5 35B A3B implemented the pre-edit code; evaluation spans five implementor models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal text (Figures 1 and 3) interleaves with body text in the extraction; the Figure 3 overlap counts could not be fully cross-checked against Table 4.
- The extracted Appendix Tables 11/12 contain no Haiku 4.5 implementor group although Table 4 includes one; whether this is an extraction loss or a paper omission cannot be determined from the provided text.
- Minor naming inconsistency visible in the extraction: Figure 1's caption reads 'Qwen 3.5 32b a3b' while the body text uses Qwen3.5 35B A3B.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.