LangSelect: Cost-Aware Target-Language Routing for LLM Code Generation
Son Ha Xuan, Phat T. Tran-Truong, Phat T. Tran-Truong, Xuan-Bach Le, Nghia Duong-Trung
cs.CL
Quick verdict
Target-language routing is a real code-gen cost lever; accounting is careful, but GPT-5-only evidence and a verbose Python baseline temper the 50.3% headline.
Summary
The paper argues that the choice of output programming language is an underused cost lever for unit-test-verifiable code generation. It introduces LangSelect, a router that scores an eight-language portfolio (C, Clojure, Go, Java, Julia, Python, Ruby, Rust) from pre-generation features (domain metadata, CodeBERT embeddings, Neural-LinUCB heads), generates in the top language with GPT-5, and falls back (up to three attempts) on verification failure. A new 3,000-task corpus, MultiLang-Bench, supplies harness-verified solutions in all eight languages. Two protocols are separated: verified-solution replay (100% pass by construction) isolates routing headroom, while live generation charges every failed and fallback attempt. Replay shows a 2.19x mean token spread between cheapest and costliest accepted solutions; live on 450 held-out tasks, a static train-split Domain heuristic cuts harness-proxy tokens 50.3% at 92.9% pass after fallback, within 1 pp of the hindsight oracle (+49.6%/92.7%), while CodeBERT+metadata reaches the highest pass (93.8%) with a 3.7% token increase. Strengths include conservative headline accounting, sensitivity checks, and honest downgrading of the bandit machinery; main limitations are a single generator model, supervised baselines tested on only 39 tasks, no terse-Python prompt baseline, and a corpus biased toward polyglot-solvable pure functions.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 2
● low 4
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Verified implementations of the same task differ by 2.19x in mean token length across eight languages (95% CI [2.17, 2.20]), making target language a measurable inference-time cost lever.
- In live GPT-5 generation on 450 held-out tasks, a static train-split Domain heuristic cuts harness-proxy tokens by 50.3% at 92.9% pass after fallback, within 1 pp of the hindsight oracle (+49.6%/92.7%).
- A learned CodeBERT+metadata selector attains the highest pass after fallback (93.8%) at a 3.7% token increase, defining a practical cost-correctness frontier.
- Routing overhead is negligible (39.65 ms mean, under 1.01% of GPT-5 generation latency), and a mainstream-only portfolio (Python, Java, Go, Ruby) still retains 15.15% replay headroom.
- Replay headroom and live recoverable savings answer different questions, and both must be reported to prevent a hindsight oracle from standing in for a deployable policy.
Strengths
Explicit protocol and accounting hygiene
Replay vs live protocols are cleanly separated (Section 5); live evaluation charges every failed and fallback attempt; three accounting boundaries are reported in parallel (Table 2) with the most conservative (harness-proxy) headlined. Table 9 shows the oracle itself flips with the boundary (Ruby oracle on 93.6% of tasks under function-body vs Python 45.1% under harness-proxy), demonstrating why boundary choice matters.
Honest downgrade of the method's own complexity
Section 7 concedes that a simple train-split Domain heuristic captures most of the live saving (+50.3% vs hindsight oracle +49.6%) and relegates the bandit to cold-start diagnostics; supervised baselines recover only 5.83% in replay (Section 6.2). The paper does not oversell its machinery, and the cost-correctness framing (Figure 4) presents Domain heuristic and CodeBERT+metadata as complementary operating points.
Broad sensitivity and audit trail
Verification-gated fallback with charged recovery
Weaknesses
Single generator model and possibly inflated Python baseline
Missing terse-prompt control
Tiny evaluation sets and no significance tests for headline deltas
Corpus bias and heterogeneous solution provenance confound the spread statistic
Scattered, inconsistently labeled replay numbers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LangSelect?
LangSelect is a routing framework that chooses which programming language an LLM should generate for a coding task, verifies the result with unit tests, and falls back to the next-ranked language (up to three attempts) on failure. The goal is to lower token cost while keeping pass rates comparable to defaulting to Python.
How much money or tokens can choosing the output language save?
In live GPT-5 experiments on 450 held-out tasks, a simple domain heuristic cut harness-proxy tokens (including wrapper and entrypoint overhead) by 50.3% while keeping 92.9% of tasks passing after fallback. Offline analysis of verified solutions shows a 2.19x average token spread between the cheapest and most expensive language per task, and a mainstream-only portfolio (Python, Java, Go, Ruby) still retains about 15% replay headroom.
What is MultiLang-Bench?
A 3,000-task corpus of pure-function coding tasks with one unit-test-verified solution in each of eight languages (C, Clojure, Go, Java, Julia, Python, Ruby, Rust), built to measure how much cost a language-routing policy can save for LLM code generation. It is split 2,100/450/450 for training/validation/test.
Does routing the output language hurt code correctness?
Not in the reported experiments: the cost-optimal Domain heuristic reached 92.9% pass after fallback versus 92.0% for Python-only, and a learned selector reached 93.8%. The catch is that non-Python first choices fail more often up front (87-92% strict pass@1), so fallback attempts - which are charged to the cost metric - are needed to recover those failures.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as embedded text; plotted curves (Figures 3-6) were verified from printed labels, not visual inspection.
- The anonymous artifact link could not be inspected, so repro-readiness reflects the paper's protocol descriptions (manifests, seeds, harness specs) rather than verified artifacts.
- Prompt templates and harness code are described at a high level only; exact prompt wording that drives the live token counts is not in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.