A Sim-to-Real Study of Surface-Code Decoder Benchmarking
Shay J. Manor, Leila S. Erhili, Yassine Jebbouri
quant-ph · cs.LG
Quick verdict
Finds decoder rankings hinge on noise-model structure not calibration fidelity, and Ising pre-decoder shows no hardware edge; rigorous, open benchmark.
Summary
The paper tests whether decoder rankings established under synthetic circuit-level noise transfer to real hardware, using the public below-threshold memory-experiment dataset from Google's Willow processor (code distances d=3/5/7; two bases; fifteen round counts; 420 configurations). Six decoders (eight entries) are ranked under a four-rung noise ladder of increasing fidelity: uniform depolarizing, SI1000, a per-patch 25-parameter fit, and a syndrome-estimated DEM. Rank agreement with hardware (Kendall's tau) is poor under uniform noise (tau=0.619-0.714) while all three circuit-level models reach tau>=0.810; yet the calibrated models do not beat the generic SI1000 template — at d=7 SI1000 attains tau=1.000 versus 0.810-0.905, plausibly because the single d=7 patch ill-conditions the estimates. Calibration improves absolute per-cycle error rates but not rank agreement. The authors also report the first independent hardware evaluation of NVIDIA's Ising pre-decoder, mapped onto its training lattice at distances below its receptive field: it is matched or beaten in both per-cycle error rate and decode latency in 278 of 280 evaluations, and is the only panel decoder whose error rate rises with distance. Limitations: the d=7 inversion cannot be separated from data scarcity; the Ising verdict applies only under these sub-receptive-field conditions; SI1000's d=7 tau=1.000 rests on few logical errors; latencies were measured on CPU nodes. The full pipeline and per-shot outcomes are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Rank agreement between synthetic benchmarks and hardware appears once the noise model gives each operation type its own error rate, and does not improve further with device calibration.
- Calibrating the noise model improves absolute per-cycle error rates but not rank agreement; at d=7 both calibrated models agree with hardware less well than the generic SI1000 template.
- NVIDIA's Ising pre-decoder, independently evaluated on hardware for the first time at d≤7 via a lattice mapping, holds no accuracy-latency advantage: another panel decoder matches or improves on it in 278 of 280 evaluations.
- Uniform depolarizing noise misranks decoders, with prior-sensitive belief-propagation decoders reaching 31.0× and 25.2× their real-hardware error at d=7 while MWPM, Tesseract and BeliefMatching stay within 2×.
Strengths
Hardware grounding with a validation gate
Appendix A verifies derived detection events reproduce the shipped ones, and measured error suppression rises from Λ=1.6–1.7 (uncorrelated MWPM) to 1.9 (BeliefMatching) and 2.1 (Tesseract), recovering the Λ≈2.14 of the Willow release [2] — the ground truth is demonstrably sane.
Clean controlled ablation design
The four-rung ladder (Section III-C) varies only the noise model; calibrated rungs are fit exclusively on detection-event statistics (logical outcomes never enter a fit) and scored on fresh samples, so no rung is evaluated on data that shaped it.
Ising adapter thoroughly checked
Exceptional reproducibility
Candid, quantified limitations
Weaknesses
Ising verdict measured outside its design envelope
Hardware reference ranking is prior-conditioned
Headline d=7 inversion is the least-supported cell
Latency methodology underdetermined
No correlated-noise rung
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: A Sim-to-Real Study of Surface-Code Decoder Benchmarking
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
That a synthetic noise model's fidelity to a device does not determine whether it ranks quantum error-correction decoders the same way real hardware does: what matters is giving each operation type its own error rate, not calibration to the device. The paper also finds NVIDIA's Ising pre-decoder holds no accuracy-latency advantage on Google Willow data at code distances 3–7, in the first independent hardware evaluation of it.
What data did the study use?
The public below-threshold memory-experiment release from Google's Willow processor: 420 configurations spanning code distances 3, 5, and 7 (14 physical patches), two bases, and fifteen round counts, with 50,000 shots per configuration. No proprietary or internal data was used.
What is NVIDIA's Ising pre-decoder?
A convolutional neural network that thins the detection-event lattice before handing a residual syndrome to PyMatching, aiming to improve the accuracy-latency frontier of surface-code decoding. Its original release was validated entirely in simulation, with speedups reported only at code distances of 13 and above; this paper tests it at distances 3–7 through an adapter mapping onto its training lattice.
Is the code and data from this study available?
Yes. The authors release the full pipeline, the four noise-model implementations, the decoder harness, and the per-shot outcome of every evaluation on GitHub (ShayManor/Sim2Real-Decoder-Benchmark), built on the public Willow data release, so new decoders or devices can be compared without hardware access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure graphics (Figs. 1–8) are not extractable as images from the PDF text; figure-dependent judgments rest on their captions and on the in-text numbers and tables that duplicate them.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.