Accelerating Transfer-Learning-Based Autotuning with Predictive LLVM IR Performance Ranking
Md Arafat Hossain, Thomas Randall, Akash Dutta, Xingfu Wu, Rong Ge, Ali Jannesari
cs.PF · cs.LG
Quick verdict
NCS is a cheap IR-based ranker that cuts trials-to-best on Polybench kernels, but the headline saving rests on post-hoc reordering and collapses on three of six ECP benchmarks where the tuner never beats the default.
Summary
This paper attacks the cost of HPC autotuning by adding a discriminative ranker to a probabilistic transfer-learning autotuner. GCTLA (Gaussian Copula Transfer Learning Autotuner) can sample many high-performing configurations for an unseen input size without iterative search, but it emits them unordered, so identifying the best one still means empirically evaluating candidates one by one. The authors add NCS (Neural Configuration Scorer), an ensemble of five small convolutional neural networks that consume tokenized LLVM IR and each classify whether a configuration clears a speedup percentile threshold (50th, 60th, 70th, 80th, 90th relative to the default build). A candidate's score is the sum of the five binary predictions, so it ranges from 0 to 5, and GCTLA's oversampled candidate pool is sorted by that score before evaluation. The authors argue this only needs discrimination near the optimum, since GCTLA already culls bad regions, which lowers the data requirement. Setup: six benchmarks (Polybench 3MM, Heat3D, Syr2k; ECP AMG, RSBench, SW4Lite), 600 Ytopt-selected empirical observations per benchmark across three source input sizes, two transfer targets (interpolated SM, extrapolated XL), 300-evaluation or 24-hour budget, three seeds, three executions averaged. NCS training takes about four minutes per benchmark (Adam, 30 epochs) and the whole overhead is claimed to cost no more than ten evaluations. Reported wins are real but uneven: on Syr2k SM trials-to-best fall from 201.67 to 16, on Heat3D SM from 216.33 to 33, on SW4Lite SM from 210 to 28.33. But on Heat3D XL trials rise from 123.67 to 149.67, on RSBench SM from 51.33 to 162.33, and on AMG and RSBench both GCTLA and GC+NCS end below 1.0x speedup while OpenTuner reaches 1.509 on AMG SM and GPTune 3.290 on RSBench SM. The average saving quoted is 27.85% of budget, up to 61.67%.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GCTLA's sampled configurations are unordered, so a discriminative model over LLVM IR can prioritize them and cut the number of empirical evaluations needed.
- Because GCTLA already restricts candidates to high-performing regions, NCS only needs to discriminate near the optimum, which drastically lowers its data requirement.
- GCTLA+NCS requires up to 61.67% fewer evaluations and averages 27.85% fewer evaluations than prior autotuning techniques to reach the search optimum.
- NCS is orthogonal and can be applied to non-generative, iterative autotuners to reduce their trials-to-best as well.
- The combined method maintains competitive performance in longer search durations, extending probabilistic transfer autotuning beyond strict few-shot use.
Strengths
The design insight genuinely reduces the modeling burden
Section 4.1 argues NCS need not model the whole space because GCTLA has already culled it, so the CNN ensemble only has to discriminate near the optimum. That is a real reduction in problem difficulty, and it is the reason a five-CNN ensemble trained on 600 observations can be useful at all.
Large, concrete wins on the Polybench kernels
Table 4: trials-to-best on Syr2k SM drop from 201.67 (GCTLA) to 16 (GC+NCS), on Heat3D SM from 216.33 to 33, on Syr2k XL from 245.33 to 18, on 3MM SM from 150 to 95, on SW4Lite SM from 210 to 28.33. On Heat3D, the kernel the paper flags as 'previously performed the worst for the GCTLA', GC+NCS still finds the optimum in 33 trials versus 216.33.
Overhead is measured and small
The percentile-ensemble scoring scheme is simple and interpretable
Failures are reported rather than hidden
Weaknesses
The '+NCS' rows are post-hoc re-rankings, not live searches
On three of six benchmarks the method loses or never beats the default
NCS increases trials-to-best on some targets
Budget and stopping rules are not comparable across techniques
Training distribution and hardware generality are narrow
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a specific, well-identified inefficiency in HPC autotuning. GCTLA, a Gaussian-Copula transfer-learning autotuner, can sample high-performing configurations for an unseen input size without any iterative search - but it emits them unordered. Finding the best member of that sample still means measuring candidates one at a time, and because enlarging the sample dilutes it, plain GCTLA degrades toward random search as the budget grows. The proposed fix is NCS (Neural Configuration Scorer): an ensemble of five small CNNs over tokenized LLVM IR, one per speedup percentile (50/60/70/80/90), whose summed binary votes give each candidate a 0-5 score used to order GCTLA's over-sampled pool before evaluation.
The scoping argument in Section 4.1 is the paper's best idea: because GCTLA already restricts candidates to a good region, NCS only has to discriminate *near the optimum*, not across the whole space. That is a genuine reduction in modeling difficulty and explains why 600 observations per benchmark suffice.
The abstract claims the framework matches state-of-the-art improvements "with up to 61.67% fewer evaluations, averaging 27.85% fewer". Both halves of that sentence need qualification, examined below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is NCS in HPC autotuning?
NCS (Neural Configuration Scorer) is an ensemble of five small convolutional neural networks that read the LLVM intermediate representation of a program configuration and score how fast it is likely to run. Each network checks whether a configuration clears one speedup percentile (50th, 60th, 70th, 80th or 90th relative to the default build); the scores are summed into a 0-5 ranking used to order candidate configurations before they are measured on real hardware.
How many evaluations does GCTLA+NCS save?
The authors report an average of 27.85% fewer evaluations across their benchmarks, with a maximum of 61.67%. The largest single win is on the Syr2k SM target, where reaching GCTLA's best result took 201.67 trials on average versus 16 with NCS ranking. The saving is concentrated in the Polybench kernels; on Heat3D XL and both RSBench targets NCS needed more trials, not fewer.
Does the method beat other autotuners on every benchmark?
No. On the AMG and RSBench ECP proxy applications the combined method finishes below 1.0x speedup on all four target tasks, meaning slower than the untuned default, while OpenTuner reaches 1.509 on AMG SM and GPTune reaches 3.290 on RSBench SM. The reported advantage holds mainly on the smaller Polybench kernels.
Is the NCS ranking applied during the search or afterwards?
In the reported experiments the searches are run without NCS and the results are reordered afterwards, so the '+NCS' rows are post-hoc re-rankings of an already-measured candidate pool. This measures how good the ranking is, but it does not measure evaluations avoided in a live tuning run.
What are the main limitations of this approach?
Four stand out: the headline saving comes from post-hoc reordering rather than online tuning; two of six benchmarks end slower than the default build; the techniques being compared operate under unequal evaluation budgets (iterative methods hit the 24-hour cap before 200 evaluations while GCTLA used all 300); and everything is measured on a single machine with one compiler toolchain, so cross-architecture generality is untested.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-6 (search trajectories, Syr2k optimality plots, NCS ordering effects) are not analyzable from the extracted text; conclusions about them rely on captions and the surrounding narrative.
- Table 1's columns (GCTLA budget versus GCTLA few-shot unique configurations) are ambiguous in the extracted text; the search-space sizes and unique-configuration counts are cited but the mapping to the 300-evaluation budget is inferred.
- The '27.85% average' and '61.67% maximum' reductions are quoted as reported; the paper does not show the per-benchmark table from which they are computed, so they could not be recomputed from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.