Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving
Muhammad Abdur Rab Siddiqui, Daniela Rojas, Chen Yang, Wenqi Cui, Yuanyuan Shi, Yize Chen
cs.PF · cs.DC · cs.LG · eess.SY
Quick verdict
Proposes measured-joule LLM routing (KL-GRPO, 60.9% acc, 1692 J); promising, but a fixed Qwen-7B arm (64.0%, 282 J) beats all learned routers on both metrics.
Summary
The paper trains a small language-model router (Qwen2.5-1.5B-Instruct) that selects, per query, one answer model from a fixed pool of nine locally served LLMs (Qwen2.5-0.5B to DeepSeek-R1-32B) to cut inference energy. Candidates are first profiled in an offline 'tournament': each model answers roughly 3,000 questions from seven benchmarks (MMLU, GSM8K, BBH, ARC Challenge, HellaSwag, MATH-500, HumanEval) on an A100, logging correctness, latency, and idle-subtracted NVML GPU energy. The router is initialized by SFT toward a cheapest-correct oracle, then optimized with GRPO and a KL-regularized variant under a gated reward (correct routes earn 2 minus lambda times energy-rank; failures score -1). Evaluation replays logged outcomes on 913 held-out questions. KL-GRPO reaches 60.9% accuracy at 1691.9 J mean answer energy, versus UniRoute's 61.8% at 2192.2 J (~23% less); GRPO step 750 reaches 58.4% at 811.8 J versus Smoothie's 59.9% at 1070.6 J. The authors also report an accuracy-energy 'phase transition' near 60%, MATH-500 driving 52-57% of energy, and reward settings that collapse routing onto one model. Key limitations: a fixed always-Qwen2.5-7B policy scores 64.0% at 281.6 J, beating every learned router on both aggregate metrics; router decode overhead (tens to hundreds of joules) is excluded from the headline comparison despite being comparable to claimed savings; and results are replay-based on a single GPU and serving stack without confidence intervals or seed variance.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A Qwen2.5-1.5B router trained with SFT followed by GRPO/KL-GRPO under a gated accuracy-energy-rank reward improves the accuracy-energy trade-off of multi-LLM serving.
- KL-GRPO (step 1500) matches UniRoute's accuracy (60.9% vs 61.8%) while using about 23% less mean answer energy (1691.9 J vs 2192.2 J) on 913 held-out questions.
- Routing policies exhibit a sharp accuracy-energy phase transition near 60% accuracy across a nine-model pool and seven benchmarks.
- RL post-training changes when extra answer-model cost is justified, shortening router decode from ~2.4 s (SFT) to ~0.8 s and preventing collapse onto the 0.5B draft model.
- MATH-500, only ~13% of held-out questions, accounts for roughly 52-57% of total answer energy in mid-band policies.
Strengths
Routing objective grounded in measured joules, not proxies
Section 2.2 defines idle-subtracted, per-call incremental GPU energy from 200 ms NVML board-power polling (Eqs. 2-4), directly addressing the paper's own critique (Sec. 1.1.1) that most routers optimize price/size/quality proxies.
Clean, transparent reward and oracle formalization
Eqs. 9-13 specify the gated rank-based reward, and Sec. 3.3 proves the hard oracle target (cheapest correct model) is invariant to any positive λ_SFT ≤ 1; invalid outputs get an explicit negative penalty. The paper also honestly states the surrogate does not guarantee the energy budget of Eq. 7.
Failure modes reported, not hidden
Unusually complete ablation tables
Router self-improvement quantified
Weaknesses
A trivial fixed policy Pareto-dominates all learned routers
Headline savings exclude the router's own energy
Replay-based evaluation with deterministic outcomes
No statistical treatment of headline differences
Narrow deployment envelope and re-implemented baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint (arXiv 2609.23085, ACM template, in submission) proposes training a small LLM router (Qwen2.5-1.5B-Instruct) to pick one of nine locally served answer models per query, optimizing **measured GPU energy** rather than price or parameter count. Three headline claims: (1) a formulation of routing around idle-subtracted NVML joules collected in an offline "tournament"; (2) an SFT→GRPO/KL-GRPO pipeline (TRL) under a gated accuracy-energy-rank reward (Eqs. 9-12); (3) on 913 held-out questions from seven benchmarks, KL-GRPO step 1500 matches UniRoute's accuracy (60.9% vs 61.8%) at ~23% less mean answer energy (1691.9 J vs 2192.2 J), with a "phase transition" in the accuracy-energy frontier near 60% accuracy.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much energy does the proposed LLM router save?
The paper reports that its KL-GRPO-trained router uses 1691.9 J of mean answer energy per query versus 2192.2 J for the UniRoute baseline (about 23% less) at nearly identical accuracy (60.9% vs 61.8%) on 913 held-out questions, and about 24% less than Smoothie at a slightly lower accuracy point (58.4% vs 59.9%). These are replay-based numbers on one A100 and exclude the router's own inference cost.
Which models are in the routing pool?
Nine open-weight models served locally with Ollama on an A100-80GB GPU: Qwen2.5 at 0.5B/1.5B/3B/7B, Phi-3 Mini (3.8B), Gemma2-2B, Llama-3.1-8B, and DeepSeek-R1 at 8B and 32B. Every model was run over ~3,000 questions from seven benchmarks (MMLU, GSM8K, BBH, ARC Challenge, HellaSwag, MATH-500, HumanEval) to build the offline correctness/energy logs used for training and replay evaluation.
Is the code or data available?
Not yet at the time of this preprint: the authors state they will open-source the router after the reviewing period. The tournament logs (per-question correctness and measured joules for all nine models), which are essential for exact reproduction, are described in detail (Tables 5-9) but not released in the paper itself.
What is the biggest weakness of the paper?
A trivial baseline undermines the headline: simply always routing to Qwen2.5-7B achieves higher accuracy (64.0%) at far lower mean energy (281.6 J) than every learned router in the main table. The authors argue the fixed model fails badly on the BBH benchmark (26.2%), which motivates routing, but this robustness argument is not formalized or optimized. Additionally, the router's own decode energy is excluded from the main comparison and results carry no statistical uncertainty estimates.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-10 (selection mixes, Pareto plot, per-query power traces, duration histograms) are not analyzable from extracted text; figure-dependent statements (e.g., controller energy of 'a few hundred joules' in Fig. 5 vs ~30 J in Fig. 10) rely on the prose captions.
- The paper is a v1 preprint in an ACM template with placeholder venue/DOI fields; content may change before peer review.
- Answer-evaluation correctness is delegated to benchmark graders whose implementations are not described, so correctness definitions could not be verified.
- 16-page text appears complete including appendices; no obvious truncation detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.