arXiv paper2026-09-07· computational linguistics / NLP (LLM evaluation)· empirical study

LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances

cs.CL · cs.AI

Quick verdict

Systematic LLM robustness study over 12 models and 5 benchmarks; breadth is real, but anomalous table jumps and weak statistics undercut the findings.

Summary

The paper introduces LANTERN, a systematic robustness evaluation of LLMs under controlled input perturbations. The authors perturb four multiple-choice benchmarks (MMLU, MMLU-Pro, GPQA, MUSR) at question level with three character-level corruption types (typos, letter duplication, letter removal) at 10/30/50% word rates, and at choice level by injecting 'None of These', 'All of the Above' and 'Both A and B' variants; instruction following is probed by rewriting IfEval prompts and their verifiable instructions. Twelve open-source models (1B–14B, base and instruction-tuned) are evaluated with lm-eval-harness on A100 GPUs. Reported findings: robustness is model- and dataset-dependent, with Phi-3.5-mini losing up to 10.6% on MMLU under typos and Phi-4 dropping 9.79% on MUSR; replacing the correct choice with 'None of These' is the most damaging perturbation on all benchmarks except MUSR, where accuracy paradoxically rises (e.g., Llama-3.2-1B 34.39→66.67); IfEval degrades 2.85% on average, with instruction-tuned models more resilient than base ones; scale does not confer robustness. A K-S test (p=0.975) and embedding PCA are used to argue the perturbed data preserves the original distribution, and three seeds give standard deviations of 0.47–1.01 on MMLU. Main limitations: no significance testing or confidence intervals on score deltas, unexplained anomalous jumps (gemma-2-2b MMLU-Pro 7.35→22.17 under typos), internally inconsistent baseline numbers across tables, no code or data release, and no per-model IfEval table.

Review complete

Assessment at a glance

computational linguistics / NLP (LLM evaluation)

risk flags identified

● high 1

● medium 3

● low 2

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

AI text signal4/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

4 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Character-level question perturbations degrade MCQ accuracy in a model- and dataset-dependent way (e.g., Phi-3.5-mini −10.6% on MMLU at v0-50%), with no clear correlation between model size and robustness.
Choice perturbations with 'None of These', 'All of the Above' and 'Both A and B' mostly reduce accuracy, and replacing the correct choice with 'None of These' is the most damaging except on MUSR, where accuracy paradoxically rises.
Instruction-tuned models are more robust than base models under IfEval perturbations (2.85% average degradation; base LLaMA models ~4%).
The synthetic perturbed datasets preserve the statistical distribution of the originals (K-S test p=0.975) and findings are stable across three random seeds.

Strengths

Systematic, broad perturbation taxonomy across two paradigms
Three character-level question perturbations (v0/v1/v2) at 10/30/50% word rates plus four choice-phrase variants and IfEval instruction rewrites are applied uniformly over MMLU, MMLU-Pro, GPQA, MUSR and 12 open models (1B–14B), with complete grids in Tables 2–9 (Sec. 3–4).
Full per-model score tables enable independent verification
Every model × perturbation cell is tabulated, so narrative numbers are checkable — e.g., Phi-3.5-mini MMLU 68.65→58.05 under v0-50% matches the quoted 10.6% drop (Tables 4, 2).
Choice-phrase probes extend known selection-bias work into new territory
Deliberate perturbation-validity checks

Weaknesses

Internally inconsistent baseline numbers across tables
No inferential statistics on the reported deltas
Unexplained anomalous jumps framed as robustness
Confounded and underspecified perturbation procedures
Reproducibility gaps and internal inconsistencies in setup description

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

[high] Tables 2 vs 6 (GPQA), Tables 3 vs 8, Tables 4 vs 9 — The same model's default accuracy on the same benchmark differs between question- and choice-perturbation tables: GPQA Phi-3.5-mini 44.31 (Table 2) vs 33.48 (Table 6), Llama-3.1-8B-Instruct 34.38 vs 31.92, Llama-2-13b-chat 25.94 vs 28.12; MMLU 68.02 vs 67.95 and MMLU-Pro 40.92 vs 40.82 show smaller mismatches.

Default GPQA baselines differ between Table 2 and Table 6 for several models (e.g., Phi-3.5-mini 44.31 vs 33.48; Llama-3.1-8B-Instruct 34.38 vs 31.92). Which values are correct, and why do baselines disagree across tables?

All 4 strengths & 5 weaknesses, analyzed in detail
6 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is the LANTERN paper about?

It is an empirical robustness study (arXiv 2609.07309) measuring how much large language models' accuracy drops when benchmark inputs are perturbed: typos and letter duplication/removal injected into multiple-choice questions (MMLU, MMLU-Pro, GPQA, MUSR), misleading answer options like 'None of These' or 'All of the Above', and rewritten instructions in IFEval. Twelve open-source models from 1B to 14B were tested.

What are the main findings of LANTERN?

Robustness is model- and dataset-dependent: Phi models were most sensitive (Phi-3.5-mini lost 10.6% on MMLU under 50% typos), replacing the correct option with 'None of These' was the most damaging perturbation on most benchmarks (but oddly raised accuracy on MUSR), instruction-tuned models handled perturbed IfEval prompts better than base models (2.85% average drop), and larger size did not guarantee robustness.

Is the LANTERN code or dataset publicly available?

The paper does not mention any public code, generated datasets, or release links. It describes generation via lm-eval-harness with custom perturbation functions, so the pipeline is conceptually reproducible from public benchmarks, but exact seeds, sample counts and configurations are not published.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figure plots themselves are not recoverable from the text extraction — only captions, table data, and in-text numbers were checkable (e.g., the distribution shapes in Figure 6 could not be visually verified).

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper