Register Bias in Complexity-Based Large Language Model Routing
cs.CL · cs.CY
Quick verdict
Novel, honest audit showing length-based routing downgrades non-standard-register queries; strong paired evidence, but no real deployed router is tested.
Summary
The paper audits complexity-based LLM query routing for register bias. Using tercile routers over four cheap complexity signals (token length, dependency-parse depth, Flesch-Kincaid grade, Gunning fog), the author tests whether meaning-equivalent queries in African American English (AAE) or learner English are routed to weaker tiers. On 37,704 authentic learner-correction pairs from W&I+LOCNESS, the length router downshifts learner originals far more than it upshifts (1,272 vs 474, p < 0.001); on 279 semantically gated Natural Questions parallel sets (AAE/Indian English via Multi-VALUE), downshifts again dominate (68 vs 42; 101 vs 34). The mechanism is function-word omission shortening text; readability and syntactic signals show no consistent effect and sometimes reverse. A quality ladder (Llama 3.2 1B, Llama 3.1 8B, Claude Opus) shows pervasive model bias: every tier, including Opus, is significantly less accurate on non-standard registers, with the gap widening with capability, while the routing step's own marginal quality cost is null on this benchmark. An on-device check (Llama 3.2 1B on a Galaxy S25+, 29.1 tokens/s, 37/40 agreement) supports deployment realism. Limitations: no real deployed router audited, rule-transformed variants, coarse unvalidated grading, no code release stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Meaning-equivalent queries in African American English and learner English are routed by token-length complexity routers to weaker model tiers than their standard-English counterparts (1,272 downshifts vs 474 upshifts over 37,704 authentic pairs, p < 0.001).
- The mechanism is function-word omission: non-standard registers yield shorter text that length-based routing misreads as simpler, while readability and syntactic-depth signals show no consistent register effect and can reverse.
- Every model tier, including the frontier Claude Opus cloud model, answers non-standard-register questions significantly less accurately, and the gap widens with capability (Path B).
- The marginal quality cost of the routing decision itself is not significant on this short-question benchmark (Path A null: AAE realized 0.430 vs counterfactual 0.423, p = 0.80).
- The weakest tier is genuinely deployable on-device: Llama 3.2 1B runs at 29.1 tokens/s on a Samsung Galaxy S25+ with accuracy matching the server-side tier (37/40 agreement).
Strengths
Novel audit target
Shifts dialect-bias auditing from single models (Hofmann et al., Nature 2024) to the routing layer above them; the claim to be the first such audit is plausible given the related-work coverage of cascades [1]-[3] and dialect stress-testing [4]-[6].
Mechanistic signal decomposition
Section IV cleanly isolates token length as the carrier of the disparity (1,272 down vs 474 up on authentic pairs) while Flesch-Kincaid reverses (1,147 vs 2,194), and Section IV.C narrows the claim honestly instead of generalizing to all complexity signals.
Honest null reporting
Triangulated evidence base
Appropriate paired statistics
Weaknesses
No deployed router audited
Judge-model contradiction
Construct slippage: register vs dialect vs interlanguage
Weak, unvalidated grader near the floor
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
If an LLM service routes queries to bigger or smaller models using text length as a complexity proxy, questions written in African American English or learner English — which omit function words and are therefore shorter — get sent to weaker models significantly more often than meaning-identical standard-English questions.
Did biased routing actually make answers worse in the study?
On the benchmark used (short Natural Questions), the routing step's own marginal quality cost was not statistically significant. The measured harm came from the models themselves: every tier, including the frontier cloud model, answered non-standard-register questions less accurately. Routing compounds this structurally by steering those users toward weaker tiers.
What data and models were used in the study?
37,704 learner-original vs corrected sentence pairs from the W&I+LOCNESS corpus, plus 279 Natural Questions transformed into African American and Indian English variants with Multi-VALUE and filtered by a semantic-equivalence judge. Models: Llama 3.2 1B (on-device), Llama 3.1 8B (edge), and Claude Opus (cloud), with the smallest model also validated on a physical Samsung Galaxy S25+ phone.
Is code or data available for this paper?
The reviewed text contains no code or data release statement. The underlying corpora (W&I+LOCNESS, Natural Questions, Multi-VALUE) are publicly available, but implementation details such as the tercile reference set, judge prompt, and grader normalization are not fully specified, so exact replication would require author input.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the extracted full text; Figures 1-2 are known only through their captions, though Table I's numbers are present in the text.
- Possible supplementary material (prompts, appendix tables, code links) was not visible in the extraction; the codeAvailable=false judgment relies on the visible text only.
- No access to the underlying corpora or model outputs; all statistical claims are taken as reported in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.