MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs
cs.DC · cs.AI
Quick verdict
Mean-field surrogate scales multi-model GPU profiling to ~linear samples; solid, but single-GPU, two-family validation limits generality claims.
Summary
The paper addresses runtime scheduling of multiple heterogeneous AI models (LLMs and vision) sharing one GPU, where profiling cost for joint performance predictors grows combinatorially with the number of co-running models. The authors propose a MeanField surrogate: instead of modeling all joint interactions, one lightweight MLP per model predicts throughput from that model's local configuration plus aggregate GPU state (VRAM usage, utilization, workload level), motivated by profiling evidence that interference is driven by aggregate resource pressure rather than specific pairings. On an RTX 3090 with Qwen2.5 LLMs (vLLM) and YOLO11 vision models (Ultralytics) at N=2..6, the surrogate reaches mean R²≈0.96 with a convergence sample budget growing approximately linearly (n*≈20N), versus O(k^N) for joint modeling. Integrated into a genetic-algorithm scheduler at N=5 (78,732 feasible configurations), GA decisions take 26 ms median versus 131 ms exhaustive search, staying within 0.10% utility gap with zero SLA violations across eight dynamic scenarios. Main limitations: validation uses a single consumer GPU and only two workload families; scheduling gaps are measured against the surrogate's own predicted landscape rather than ground-truth execution; the authors acknowledge limited adaptation benefit over static full-space configuration; and no code or data release is mentioned.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A per-model MeanField surrogate conditioned on local configuration and aggregate GPU state predicts concurrent throughput with mean R²≈0.96 across N=2..6 on an RTX 3090.
- The profiling budget to reach R²≥0.95 grows approximately linearly (n*≈20N: 40-120 samples for N=2..6), versus combinatorial O(k^N) cost for fully joint modeling.
- A GA scheduler using the surrogate stays within 0.10% of exhaustive surrogate search with zero SLA violations at N=5 (78,732 feasible configurations) across eight dynamic scenarios, deciding in 26 ms median (~5× faster than 131 ms exhaustive search).
- Interference among co-running models is primarily driven by aggregate resource pressure rather than specific model pairings (ANOVA, p<0.01).
Strengths
Quantified scalability protocol
Sample-complexity is measured rigorously: the convergence threshold n* at R²≥0.95 is computed over 25 evaluations (5 seeds × 5-fold stratified CV) per N (Sec. IV.A, Fig. 3), yielding n*=40/60/80/100/120 for N=2..6 — a falsifiable, practitioner-relevant number.
Honest negative result and calibrated framing
Sec. IV.C reports StaticFull within 0.05% of exhaustive and explicitly reframes the benefit as profiling-efficient near-optimal search 'rather than adaptation alone' — candor that calibrates the contribution correctly.
Above-average profiling hygiene
Thoughtful utility engineering
MeanField matches Joint MLP where comparison exists
Weaknesses
Surrogate-referenced scheduling evaluation
Narrow heterogeneity and single-GPU evidence
Thin statistics in Table II
Indirect validation of the core assumption
No artifacts, no system baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This IEEE Embedded Systems Letters paper (DOI 10.1109/LES.2026.3724013) tackles a real scaling pain in shared-GPU inference: training a *joint* performance surrogate over N co-running heterogeneous models requires profiling that grows combinatorially in N (O(k^N)). The authors propose a MeanField surrogate — one lightweight per-model MLP predicting throughput from local configuration plus aggregate GPU state (VRAM usage, utilization, workload level) — and advance three claims: (1) mean R² ≈ 0.96 across N ∈ {2..6}; (2) convergence sample budget n* ≈ 20N, approximately linear; (3) a GA scheduler at N = 5 (78,732 feasible configurations, three LLMs + two YOLO models, the 7B LLM AWQ-only) stays within 0.10% of exhaustive surrogate search with zero SLA violations across eight dynamic scenarios at 26 ms median decision latency (~5× faster than the 131 ms exhaustive search).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a MeanField surrogate for GPU scheduling?
It is a per-model predictor that estimates each model's throughput from its own configuration plus aggregate GPU state (VRAM usage, utilization, workload level), instead of one joint model over all co-running model combinations — cutting profiling cost from combinatorial to roughly linear in the number of co-running models.
How much profiling data does the MeanField surrogate need?
In the paper it reaches R² ≥ 0.95 after roughly 20 samples per co-running model (n* ≈ 20N: 40 samples at N=2 up to 120 at N=6), whereas a fully joint predictor needs profiling that grows combinatorially with the number of models.
What hardware and models were used in the paper?
A single NVIDIA RTX 3090 (24 GB); Qwen2.5-series LLMs served with vLLM (FP16/AWQ variants) and YOLO11 vision models (n/s/m) running under Ultralytics, with CUDA 12.6 on Ubuntu 20.04.
Does the scheduler beat existing serving systems like Orion or Clipper?
No head-to-head system comparison is reported. The baselines are exhaustive search over the same surrogate's predictions, static full-space configuration, sample-budgeted static configurations (K=5000 and K=500), and random selection; interference-aware systems like Orion and ElasticRoom are cited but not benchmarked.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's learning-curve values are not recoverable from text extraction (only axis labels and captions came through); the convergence narrative relies on the text's stated n* values of 40/60/80/100/120.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.