When Edit Flows are Edit Jumps: replicating Edit Flows and EvoFlows
Gabriel B\'en\'edict, Melanie Buechler, Gerard Riera-Sol\`a, Chlo\'e de Ancos, Yves Gaetan Nana Teukam, Moritz Freidank
cs.LG · stat.ML
Quick verdict
A rigorous replication of Edit Flows and EvoFlows as one pure-jump process, releasing the first open implementation; evaluation baselines are self-implemented.
Summary
The paper replicates Edit Flows and EvoFlows, two edit-based generative models for proteins whose code and complete training specifications were never released. The authors recast both as the pure-jump case of generator matching — edits firing one at a time at learned rates in continuous time — and release EditJumps, the first open implementation, with checkpoints and tests. A single generalist antibody editor (ESM-2 35M trunk, FiLM-conditioned insertion/deletion/substitution rate heads) is trained on 1.66M Observed Antibody Space VH.VL homolog pairs and evaluated zero-shot on two held-out targets (Ty1 VHH, HER2-VH) at matched budgets of ~4.1–4.6 edits per sequence, outperforming forced evotuning and matching standard evotuning on MMD. The replication surfaces three findings: reconciling published edit statistics requires reverse-engineering an undocumented clock-normalization hyperparameter (~95 vs their 40) that dictates realized mutation counts; six of ten published metrics are not deterministically recomputable from the text; and distributional metrics are highly sensitive to reference sample size, sufficient to flip method rankings. The paper also documents a deletion-supervision indexing bug in the Edit Flows reference code and a text-versus-code schedule discrepancy. Limitations: strictly in silico evaluation, partially overlapping evaluation references, non-converged training, no per-target-retrained EvoFlows baseline, and published values recovered only by digitizing figures.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Edit Flows and EvoFlows are both instances of the pure-jump case of generator matching over finite sequences, requiring non-explosion and clock-normalization conditions the originals left unformalized.
- Reconciling published edit statistics with a faithful reimplementation requires an undocumented clock-normalization hyperparameter (≈95 versus the authors' operating value 40) that directly governs realized mutation counts.
- Six of the ten evaluation metrics reported in the EvoFlows paper cannot be deterministically recomputed from its text, and distributional metrics such as spectrum MMD flip method rankings as reference sample size changes (200 vs 800 homologs).
- A single generalist antibody editor trained on 1.66M OAS homolog pairs edits unseen targets zero-shot at matched budgets of 4.1–4.6 edits per sequence, outperforming forced evotuning and matching standard evotuning on MMD, without per-family retraining.
Strengths
First open implementation of an unreleased framework
Neither Edit Flows nor EvoFlows released code or complete specs; EditJumps (MIT-licensed code, checkpoints, configurations, automated test suite) fills this gap and unifies both methods under the pure-jump generator-matching formulation of Holderrieth et al. [17, Eqs. 80–86].
Principled formalization of previously unstated conditions
Section 2.3 identifies two regularity requirements the original papers left unformalized: non-explosion of the CTMC on unbounded sequence spaces (Edit Flows handles it via a hard length cap, EvoFlows leaves it unconstrained) and length-dependent clock normalization, which the authors show experimentally governs realized edit counts (Figure 2 sweep, clock ≈95 reproduces the published 0.43 edit ratio vs their operating point 40).
Unusually candid reporting of negative and sensitivity results
Systematic elimination of alternative explanations
Concrete bug and discrepancy documentation benefiting the original authors
Weaknesses
Headline comparisons rest entirely on self-implemented baselines
Evaluation references overlap the training corpus
Model is demonstrably under-converged
Cross-study quantitative claims rely on digitized figures
Asymmetric trunk sizes in the key comparison
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EditJumps?
EditJumps is an open-source PyTorch implementation, introduced by this paper, of edit-based continuous-time generative modeling for proteins. It unifies the previously code-less Edit Flows and EvoFlows methods as pure-jump processes: a neural network assigns rates to insertions, deletions, and substitutions of a seed sequence, and edits fire one at a time in continuous time. A single generalist antibody editor trained on 1.66M Observed Antibody Sequence pairs can propose homolog-like variants of unseen antibody leads without retraining per target.
Why couldn't the authors compare their results directly with the published EvoFlows numbers?
EvoFlows released no code or complete training specifications. The replication found that an unstated hyperparameter (a clock normalization of about 95) controls how many mutations are realized, that six of ten evaluation metrics were not defined precisely enough to recompute, and that published values existed only as figure graphics that had to be digitized from the PDF. Because the original experimental conditions could not be recovered, the authors treat published and replicated numbers as non-comparable and benchmark against their own grounded baselines instead.
Why do evaluation metrics in this area flip method rankings?
Distributional metrics such as spectrum MMD and covariance agreement are normalized against a set of reference sequences, and both the score and the baseline change with reference size. The paper shows covariance agreement between identical real-homolog sets rising from 0.790 at 20 references to 0.985 at 800, and the normalized MMD ratio changing by up to 50% when references grow from 200 to 800 — large enough to invert method rankings. The authors recommend standardized reference sets and ceiling normalization.
Is the code and data for this paper available?
Yes. The paper states that the full codebase, automated test suite, checkpoints, configurations, and data manifests are released under an MIT license at github.com/VisiumCH/editjumps, including the reimplementation of all ten evaluation metrics from the original EvoFlows paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 cannot be inspected directly; their interpretation relies on the authors' textual descriptions and their own digitization of published graphics.
- Appendix claims (e.g., audit percentages, disulfide retention counts) are taken as reported and could not be independently verified from the text.
- The linked GitHub repository and model cards were not examined; code-availability statements rest on the paper's assertions.
- The extracted text appears complete through Appendix E, but any supplementary material beyond it would not have been reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.