When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text
DongHyun Ryu, Jaehyeok Lee, YeongJun Hwang, JinYeong Bak
cs.CL · cs.LG
Quick verdict
Solid controlled study: noise on judged text inflates LLM-judge bias verdicts, fabrication over erasure up to 120×; headline ratio rests on 2 erasures.
Summary
This paper asks whether surface noise in judged passages distorts LLM-as-a-judge social-bias measurement. Building FABLE from CLEAR-Bias (42 stereotype-probing questions × 91 name pairs = 3,822 binary-choice items with Llama-3.1-generated responses), the authors inject five rule-based noise types (typos, informal spelling, punctuation collapse, a colloquial-particle variant, and a sequential combination) at five intensities (p=0.1–1.0) and compare four judges — Llama-3.1-8B, Qwen3-8B, Gemma-4-12B-it, GPT-5.4 — against each judge's own clean baseline. The central result is directional: noise turns neutral verdicts into biased ones far more often than the reverse — 240:2 (120×) for Llama under typos at realistic p=0.3 (p<0.001) — and combined noise amplifies fabrication up to 2.5× over any single type. Fabrication concentrates in religion, disability, sexual orientation, and socioeconomic status, holds across seeds and template-level sign tests (41/42 templates for Llama), and attenuates toward parity as judges grow more robust without ever reversing. Main limitations, largely acknowledged by the authors: 'fabricated bias' means neutral-to-directional instability without verifying stereotype alignment; noise is synthetic and meaning preservation was never human-verified; corrupted names may drive orthographic option-matching rather than stereotype reasoning; responses come from a single source, and most conditions ran under a single noise seed.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Surface noise shifts LLM judges from neutral to directional bias verdicts far more than the reverse — up to 240:2 (120×) for Llama-3.1-8B under typos at realistic intensity p=0.3 (p<0.001).
- The fabrication-over-erasure asymmetry attenuates toward parity as judges grow more robust and never significantly reverses across all 100 judge-condition pairs.
- Combining noise types amplifies fabrication by up to 2.5× over any single type (Qwen3-8B, p=1.0: 166→422).
- Fabrication concentrates in religion, disability, sexual orientation, and socioeconomic status, a ranking stable across noise types, intensities, and NONE-pool normalization.
Strengths
Directional asymmetry as the central quantity, not just degraded stability
Beyond raw flip rates (Table 4), the paper decomposes transitions and shows None→directional beats directional→None up to 240:2 (120×, Llama typo p=0.3, p<0.001), with parity tests per condition (Table 5) — a sharper quantity than 'accuracy drops'.
Self-baseline metric plus realized-edit-rate calibration
§3.4 compares each judge to its own clean decisions, correctly avoiding entanglement with differing clean tendencies (Table 3: GPT symmetric, Llama B-leaning); Appendix C (Table 12) reports realized edit rates, showing nominal p ≠ realized corruption (punct 26.1% vs typo 76.2% tokens at p=1.0).
Layered robustness checks targeting the obvious objections
Unusually transparent Limitations section
Releasable benchmark and seed-controlled noise toolkit
Weaknesses
'Fabricated bias' is not verified as stereotype-aligned
Headline 120× ratio rests on 2 erasures and was not seed-checked
GPT-5.4 per-item labels missing from key analyses
Name-matching confound left uncontrolled
Single synthetic response source limits ecological validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.11067) studies how surface noise — typos, informal spelling, punctuation collapse, discourse-particle insertion, and their sequential combination — affects LLM-as-a-judge measurement of social bias. Four central claims: (1) noise shifts judges from neutral to directional verdicts ('fabrication') far more than the reverse ('erasure'), peaking at 240:2 (120×) for Llama-3.1-8B under typos at realistic p=0.3 (p<0.001, Table 5); (2) the asymmetry attenuates toward parity as judges grow more robust and never significantly reverses (erasure exceeds fabrication in only 8 of 100 judge-condition pairs, none significant); (3) combined noise amplifies fabrication up to 2.5× over any single type (Qwen3, p=1.0: 166→422); (4) fabrication concentrates in religion, disability, sexual orientation and socioeconomic status. The abstract's framing — bias on noisy text is 'systematically overestimated' — modestly outruns the measured construct (see Limitations & Risks).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the FABLE benchmark?
FABLE is a set of 3,822 binary-choice bias items derived from CLEAR-Bias: 42 stereotype-probing questions crossed with 91 persona name pairs across seven categories (546 items each), each paired with a Llama-3.1-generated response labeled A/B/NONE, plus a seed-controlled toolkit for injecting five types of surface noise at five intensities.
Does noisy text make LLM judges overestimate bias?
According to the paper, yes directionally: neutral-to-biased flips exceeded biased-to-neutral flips in every tested condition, peaking at 240:2 (120×) for Llama-3.1-8B under typos at realistic intensity. Caveat: the flipped verdict's stereotype alignment was not verified, so this measures directional instability, not confirmed bias.
Which LLM judge was most robust to text noise?
Gemma-4-12B-it was most robust (1.2% flip rate under typos at p=0.3), followed by GPT-5.4, Qwen3-8B, and Llama-3.1-8B, which was most fragile at 10.7% — roughly a 9× spread.
Is the paper's code and data available?
Yes. The authors release FABLE and a seed-controlled, Python-stdlib-only noise-injection toolkit at github.com/dong4918-skku/Fable; FABLE itself derives from the public CLEAR-Bias benchmark.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal values (Fig 2–4 curve points, bar heights) are not recoverable from the extracted text; this review relies on the corresponding tables (4–7, 9–12), which report the same quantities.
- The extraction ends cleanly at the Table 12 caption (end of Appendix C); no truncation was observed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.