MAWILE: Multi-Axis Workbench for Inspecting LLM Evaluators
Jackson Hassell, Farima Fatahi Bayat, Pouya Pezeshkpour, Estevam Hruschka
cs.AI
Quick verdict
MAWILE unifies judge-audit perturbations across prompt, rubric, input, and output; a well-engineered system paper with broad but statistically thin evidence.
Summary
MAWILE is a developer-facing workbench for auditing LLM judges through controlled perturbations across four surfaces: the judge prompt, the judge rubric, the evaluated system's input, and its output. A planning agent proposes an editable suite of typed operators, each declaring whether the verdict should remain invariant or move in a specified direction; a validation stage filters variants that violate their own constraints before the judge is re-executed, and results are localized by surface, operator, and item without requiring gold labels. The authors run a 3x3x2 study over MT-Bench (ordinal), Search Arena (pairwise), and GSM8K (binary) with GPT-5.6 Luna, DeepSeek-V4-Flash, and Gemma 3 4B as judges and Qwen3.8-Max/Kimi K3 as perturbation generators. DeepSeek-V4-Flash is the most accurate judge in all six settings (e.g., 98.0-98.4% on GSM8K) and the best at degradation detection, but invariant robustness is task-dependent: Gemma 3 4B flips least on MT-Bench (2.4-2.8%) yet most on Search Arena (13.8-16.0%). Repeat flips (0.1-5.3%) underestimate perturbation sensitivity, and generator choice shifts degradation detection by about 7.5 points versus 0.5 for invariant flips. A walkthrough shows rubric reordering causing 15.9% flips against a 4.0% baseline. Limitations: sensitivity does not imply correctness, no statistical significance testing is reported, sample sizes are unstated for two of the three benchmarks, and no user study validates the workbench claims.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Auditing a configured judge requires perturbing both the judge instrument (prompt, rubric) and the evaluated items (input, output), and MAWILE unifies all four surfaces in one editable, validated workflow (Sections 1-3, Table 1).
- Typed invariant/directional operators with a validation stage enable label-free sensitivity audits of binary, ordinal, and pairwise judges (Section 3.2-3.3).
- Judge accuracy does not imply perturbation robustness: DeepSeek-V4-Flash is the most accurate judge in all six settings yet the least invariant-robust on MT-Bench (11.7-12.2% flips vs Gemma 3 4B's 2.4-2.8%) (Section 6, Table 2).
- Repeatability does not imply perturbation robustness: Gemma 3 4B shows 0.3-0.4% repeat flips but 13.8-16.0% invariant flips on Search Arena (Section 6).
- Perturbation generator choice matters far more for directional probes (~7.5 pp mean absolute difference in degradation detection) than for invariant probes (0.5 pp) (Section 6).
Strengths
Genuinely unified four-surface coverage
Table 1 shows prior systems each cover subsets (FBI: output only; JudgeSense: prompt only; JRH/EvalSense: output+paired; RobustJudge: rubric+prompt+output+coupled), while MAWILE covers rubric, prompt, input, output, and coupled pairs; the rubric and input surfaces are the real gaps it fills.
Typed operator contract with a validation gate
Section 3.2-3.3 gives each operator explicit applicability, expected verdict relation (invariant/directional with scale orientation), and a validator independent of the judge under test that rejects bad variants before execution - directly addressing the classic confound where incoherent perturbations get misread as judge failures.
Principled scoping of directional probes
Well-chosen empirical dissociations in the 3x3x2 grid
Candid limitations and released code
Weaknesses
No statistical inference anywhere
Accuracy metric is under-defined
Sample sizes and audit hyperparameters unstated
Invariant tolerance is a free hyperparameter with no sweep
Workbench actionability claims are unevaluated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MAWILE (Megagon Labs, arXiv 2609.22599) is a developer-facing workbench that audits configured LLM judges by re-executing them under controlled perturbations across four surfaces: the judge prompt, the judge rubric, the target-system input, and the target-system output(s). The paper makes three system claims - unification of item-side and judge-instrument reliability tests in one workflow (Section 1), a typed probe interface with explicit applicability, expected-relation, and validation semantics (Section 3.2), and an interactive audit that localizes failures to surface, operator, and item with inspectable evidence (Section 3.4) - plus four empirical findings drawn from a 3x3x2 grid (Section 6). It targets binary, ordinal, and pairwise judges without gold labels.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MAWILE and what does it do?
MAWILE is an open-source workbench from Megagon Labs for auditing LLM judges (LLM-as-a-judge setups). It applies controlled perturbations to four things - the judge's prompt, the judge's scoring rubric, the evaluated system's input, and its output - then re-runs the judge and reports where verdicts flip when they should not (invariant probes) or fail to move when they should (directional probes). No human gold labels are required.
What did the MAWILE paper find about LLM judges?
In a 3x3x2 study across MT-Bench, Search Arena, and GSM8K with three judges, the most accurate judge (DeepSeek-V4-Flash) was not the most robust: it flipped most often under meaning-preserving edits on MT-Bench (11.7-12.2%) while Gemma 3 4B flipped least there (2.4-2.8%) but most on Search Arena (13.8-16.0%). Low run-to-run randomness did not imply perturbation robustness, and which LLM generated the degradations shifted detection rates by about 7.5 points.
Does auditing with MAWILE need labeled data?
No. Expectations come from the perturbation type itself: invariant probes should not change the verdict, directional probes should move it in a known direction. Gold labels only add an accuracy column to the report when they are available.
Is MAWILE's code available?
Yes. The paper points to github.com/megagonlabs/mawile-judge; the built-in catalog ships 16 invariant and 7 directional perturbation operators and supports binary, ordinal, and pairwise judges.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-8 are screenshots (workflow, planner rationales, UI views); their content could not be verified from the extracted text.
- Table 2 best/second-best bold and underline markers were lost in extraction; per-metric rankings in this review were re-derived from the raw numbers.
- Sample sizes for Search Arena and GSM8K are absent from the extracted text; they may appear in figures or be genuinely unstated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.