Impact of Data Loss in Postprocessing on Training and Inference of Quantum Neural Networks
Soraya V. Panambalom, Edoardo Altamura, Nick Chancellor, Jonte R. Hance
quant-ph · cs.ET · cs.LG · physics.comp-ph
Quick verdict
Credible case study: SamplerQNN postprocessing silently discards 85–99.6% of hardware shots, distorting QNN training and inference; convincing but narrow.
Summary
This case study documents and fixes a silent data-loss bug in the Qiskit Machine Learning library's SamplerQNN postprocessing. The _postprocess routine converts each measurement bit-string to an integer and discards values at or above 2^n (n = number of virtual qubits). Simulator samplers return n-bit strings, so the filter is harmless there, but IBM hardware returns bit-strings over all 100+ physical qubits; depending on transpiler layout, either high qubit placement makes nearly all valid outcomes exceed the threshold, or ancilla readout noise pushes otherwise valid shots over it. Across five experiments (breast-cancer classification, 4-7 qubits) on ibm_kingston, ibm_torino and ibm_fez, 85-99.6% of shots were discarded. Because shot totals are computed before filtering, the resulting probability vector is unnormalised, silently distorting predictions and loss. On identical raw measurements, corrected marginalisation recovers accuracy of 0.94 versus 0.39 with the filter, and recomputed losses show the optimiser received a 22-27x compressed, near-constant signal resembling a barren plateau. The authors filed issue #1040 and merged a layout-based marginalisation fix (PR #1041), shipping in version 0.9.1. Main limitations: one classification task, few qubits, no uncertainty estimates from single hardware runs, and only five fine-tuning iterations in which COBYLA never completed its initial simplex, so the training impact is inferred from loss compression rather than demonstrated through actual optimisation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SamplerQNN's postprocessing filter discards 85-99.6% of valid measurement shots on 100+ qubit IBM hardware, depending on transpiler qubit placement (Table I survival rates 15.2% down to 0.2%).
- Because shot totals are counted before filtering, the resulting unnormalised probability vector silently distorts predictions and loss values without an API-level warning.
- On identical raw hardware measurements, correct layout marginalisation restores accuracy from 0.39 to 0.94 for the worst-affected model (Table IV).
- During hardware fine-tuning, the optimiser received a 22-27x compressed, near-constant loss signal (Table V) whose symptom mimics a barren plateau.
- A layout-based marginalisation fix was merged upstream as PR #1041 and ships in Qiskit ML 0.9.1; versions 0.8.4-0.9.0 are affected when using real hardware or SamplerV2.
Strengths
Clean causal attribution via controlled reanalysis
Section III-A applies three postprocessing methods (A: all-qubit parity, B: layout marginalisation, C: exact reproduction of the original filter) to identical archived raw counts; Method C reproduces not just accuracy but every per-class metric of the original SamplerQNN output across all five experiments, excluding hardware execution and circuit design as explanations and pinning the discrepancy on the filter (Table IV).
Root cause traced to exact code path with two characterised failure modes
Section II shows the integer filter (threshold 2^n) fails via high transpiler placement (Case 1: positions [108,109,110,118] make all outcomes except |0000> exceed 2^4) and via ancilla readout noise (Case 2: a single flip at position 20 pushes a valid shot over threshold), each with a worked numerical example.
Transparent handling of the underpowered training experiment
Real-world impact and reproducibility infrastructure
Weaknesses
No statistical uncertainty anywhere
Training impact inferred, not demonstrated
Narrow empirical scope
Residual gap left unexplained and fix edge cases untested
Minor numerical inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a software-reliability case study in quantum machine learning, centred on `SamplerQNN` in the Qiskit Machine Learning library (versions 0.8.4-0.9.0). The central claims are precise and falsifiable: (1) the `_postprocess` integer filter (Section II-A) discards any bit-string whose decimal value is >= 2^n — harmless for simulator output but silently removing 85-99.6% of valid shots on 100+ qubit IBM hardware, via two failure modes: high transpiler placement (Section II-B) and ancilla readout noise (Section II-C); (2) because `total_shots` is counted before filtering (Section II-D), the probability vector is unnormalised and distorts predictions and loss without an API warning; (3) on identical raw measurements, correcting the postprocessing raises accuracy from 0.39 to 0.94 for the most affected model, and during hardware fine-tuning the optimiser received a loss signal compressed 22-27x; (4) a layout-based marginalisation fix was merged upstream as PR #1041 (7 May 2026, shipping in 0.9.1). All four claims are directly supported by the presented data.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SamplerQNN bug in Qiskit Machine Learning?
In versions up to 0.9.0, SamplerQNN's postprocessing converted each measurement bit-string to an integer and kept only values below 2^n (n = number of user qubits). On IBM hardware, bit-strings span 100+ physical qubits, so 85-99.6% of valid shots could be silently discarded, leaving an unnormalised probability vector that distorted predictions and training loss.
How much did the bug affect results?
In the paper's breast-cancer classification experiments, the same raw hardware measurements gave 0.94 accuracy when correctly marginalised versus 0.39 with the buggy filter for the worst-affected model; during hardware fine-tuning, the optimiser received loss values compressed 22-27 times below their true magnitude.
How was the bug fixed and which versions are affected?
The authors replaced the integer filter with layout-based marginalisation that reads the transpiler's qubit placement and extracts only the virtual-qubit bits from each measurement. The fix was merged as PR #1041 (7 May 2026) and ships from version 0.9.1; versions 0.8.4 through 0.9.0 are affected, but only when running on real hardware or SamplerV2, not with default simulators.
Can previously published results affected by this bug be corrected?
Inference results can be recovered after the fact if the raw measurement counts were saved, by applying marginalisation to them. Models trained on hardware cannot be corrected retroactively, because corrupted loss values already shaped the weight updates at each optimisation step.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's plotted curves are not verifiable from the extracted text (only the caption and Table V are present, so curve-level claims rest on the table).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.