SCX Router: Streaming Zero-Shot Model Selection with a Decoder-KV Classifier and a Real-World Task Ontology
Ihor Stepanov, Aleksandr Smechov, Mykhailo Shtopko, Dmytro Vodianytskyi, Oleksandr Lukashov
cs.AI · cs.CL
Quick verdict
A 0.6B zero-shot LLM router tops the best fixed model by +0.012 on a self-selected 1,000-task subset; honest scoping, but gains are thin and lack CIs.
Summary
The paper introduces SCX Router, a 0.6B-parameter router that selects among LLM endpoints by treating candidate models as dynamic zero-shot labels for a GLiClass-style classifier built on a Qwen3-0.6B causal decoder. A persistent text-only KV cache carries dialogue state across turns while candidate-label tokens are scored transiently and discarded, so routing happens in a single non-generative forward pass. The same checkpoint predicts model suitability, task type, difficulty, reasoning mode, and expected output length; hard constraints plus cost- and cache-aware utilities live in a separate policy layer. Training combines roughly 104.5k benchmark-derived routing records with an ontology of 23 task families, 115 task types, and 345 subtypes used to generate 150,000 verifier-scored and 15,000 LLM-judged synthetic tasks. The headline result is a top-1 score of 0.707 versus 0.696 for the strongest fixed model on a selected 1,000-task subset, with mixed per-benchmark gains and a -0.013 deficit to the fixed baseline at k=3. Limitations are notable: the evaluation subset was chosen post hoc for positive router gains, no confidence intervals or significance tests are reported, the LiveBench comparison uses an earlier checkpoint, key training details are unavailable, and the full task corpus and outcome matrix remain private.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A 0.6B GLiClass router with a Qwen3 causal decoder scores a changeable roster of endpoint labels in a single non-generative forward pass while reusing a persistent text-only KV cache across dialogue turns (§4, Table 1).
- The released checkpoint predicts model suitability (macro F1 0.759), task type (0.837), difficulty (0.789), reasoning mode (0.897), and output length (0.788) from one set of weights (Table 6).
- On a selected 1,000-task subset, direct routing reaches top-1 0.707 versus 0.696 for the strongest fixed model, with benchmark-mixed gains and a -0.013 deficit at k=3 (§10.3, Table 9).
- A 23-family/115-type/345-subtype ontology with 30 domains structures 150,000 verifier-scored and 15,000 judge-scored synthetic tasks toward realistic and agentic workloads (§5-6).
- Routing gains are conditional on candidate disagreement: the router can underperform a strong fixed model, and unobserved outcomes are treated as missing data, not failures (§10.3, §6.3).
Strengths
Explicit evidence-tiering discipline
Table 5 and Section 8 cleanly separate the released-and-evaluated direct routing path from implemented-without-results hard profile aggregation and purely proposed posterior/hybrid/cascade/portfolio/agentic variants — rare candor that prevents overclaiming about three-quarters of the design space.
Principled missing-outcome handling
Equations 2-3 define positives as models tied within ε of the best observed score and mask unobserved task-model pairs out of the loss; Section 6.3 explicitly refuses to report an 11-way accuracy under unequal endpoint coverage, treating missing evaluations as missing data rather than failures.
Genuine architectural contribution
Failure-driven ontology refinement
Open checkpoint and harness
Weaknesses
Headline margin rests on a self-selected subset
No uncertainty quantification; sign flips at k=3
Checkpoint inconsistency across experiments
Reproducibility gaps in training and data
No latency or cost measurements of the core claim
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
SCX Router — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SCX Router?
SCX Router is a 0.6B-parameter, Apache-2.0 model-routing system that scores candidate LLM endpoints as dynamic zero-shot text labels using a Qwen3-0.6B causal decoder with a GLiClass-style bidirectional scorer. It runs in one non-generative forward pass, keeps a persistent text-only KV cache across dialogue turns, and also predicts task type, difficulty, reasoning mode, and expected output length; hard constraints and cost/cache policies are applied in a separate decision layer.
How much better is SCX Router than just always using one strong model?
On the paper's selected 1,000-task evaluation subset, the router scores 0.707 top-1 versus 0.696 for the strongest fixed model (+0.012). Gains vary by benchmark, and at k=3 the fixed baseline wins (0.837 vs 0.824). No statistical significance tests are reported, and the subset was selected partly because the router gained on it, so the real-world advantage should be treated as modest and unproven.
Is the SCX Router code and data available?
The checkpoint is public on Hugging Face (scx-admin/scx-router-v0.1, Apache 2.0) and the implementation and benchmark harness are on GitHub (Knowledgator/GLiClass). However, the full 165,000-task synthetic corpus and the model-outcome matrix are not yet public, and some training details (hardware, global batch size, focal-loss settings) are reported as unavailable.
How is SCX Router different from RouteLLM or FrugalGPT?
RouteLLM learns a binary strong-versus-weak gate and FrugalGPT learns cost-aware cascades; SCX Router instead performs multi-label suitability scoring over a changeable roster of natural-language endpoint labels, with no generative route token to decode. Its decoder-KV path reuses dialogue state between turns, and the same checkpoint outputs auxiliary signals (task type, difficulty, reasoning mode, output length) that a separate policy layer combines with price, latency, and cache state.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images (Fig. 1-6) are not present in the extracted text; only captions and in-text descriptions were available, so figure-internal content could not be checked.
- The metadata abstract is cut mid-sentence ('while explicitly s'), but the full abstract appears complete at the start of the paper text — no abstract content was lost for this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.