Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening
cs.CL · cs.AI
Quick verdict
Controlled benchmark shows LLM resume screeners flip ~29-46% of matched decisions under competence-preserving rewrites; sound design, small scale limits reach.
Summary
The paper introduces a controlled audit framework separating candidate competence from resume presentation in LLM-based screening. From O*NET 30.3, the authors build 17 occupations × 6 synthetic profiles (102 candidates) at three competence tiers, render each into five resume forms (original bullet style plus verbosity, structure, AI-polish, and layout-artifact perturbations; 510 variants, 505 passing a deterministic fact-preservation gate), and score each resume independently against its occupation rubric, forming pairwise decisions only afterward. Across six open instruction-tuned LLMs plus BM25/TF-IDF baselines and a deterministic control, validity (recovery of known-superiority ordering) and presentation stability diverge: Llama-3.1-8B with native chat template reaches the best validity 0.781 yet flips 29.6% of matched pairwise decisions; Mistral-7B-v0.3 attains 0.644 validity with 41.4% flips. A chat-template ablation shows raw prompts understate validity (e.g., Llama-3.2-3B 0.321→0.551) but leave flip rates at 0.285–0.456. Lexical baselines flip little (0.040/0.052), which the authors attribute to the benchmark preserving rubric phrases. Main limitations: synthetic candidates and perturbations from unnamed generators, only 17 occupation clusters (bootstrap treated as pilot-scale), open-weight 2B–8B models only, no code/data release stated, and flip rates not decomposed by pair type or axis except one Qwen diagnostic slice.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Validity and presentation invariance are empirically distinct: the strongest condition (Llama-3.1-8B, native chat template, validity 0.781) still reverses 29.6% of matched pairwise decisions under competence-preserving presentation changes.
- Native chat templates substantially improve validity for chat-tuned models (e.g., Gemma-2-2B 0.287→0.576) but leave flip rates between 0.285 and 0.456, so format mismatch does not explain the instability.
- Low lexical-baseline flip rates (BM25 0.040, TF-IDF 0.052) are an artifact of the benchmark preserving rubric phrases, not evidence that lexical matchers are better screeners.
- Resume-screening evaluations should report presentation stability alongside validity as a complementary dimension.
Strengths
Clean causal separation of competence from presentation
§2.1–2.2 fixes competence at the profile level before rendering; the deterministic gate (§2.2) rejects variants that drop facts, add unsupported skills, or leak tier labels (505/510 pass), and same-tier pairs require exact evidence-vector agreement.
Independent scoring protocol removes pairwise confounds
§2.3 scores each resume independently against its rubric and forms decisions post hoc via sign(si − sj), excluding pair-order and comparative position effects that plague many pairwise LLM audits.
Prompt-format confound directly ablated
Honest metric interpretation
Candid limitations section
Weaknesses
Generator provenance unnamed
Flip-rate uncertainty absent
Flip metric conflates pair types
Axis analysis limited to one weak slice
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: when the qualification evidence in a resume is unchanged, does an LLM screener's pairwise decision survive changes in wording, structure, polish, and extraction noise? It claims (i) that validity and presentation invariance diverge sharply — Llama-3.1-8B under its native chat template attains the best known-superiority validity (0.781) yet reverses 29.6% of matched decisions, while Mistral-7B-v0.3 sits at 0.644 validity with 41.4% flips; (ii) that native chat templates fix part of the raw-prompt validity deficit (Table 2) but not the instability; and (iii) that hiring-screener evaluation should report presentation stability as a first-class metric. The claims are scoped honestly to six open instruction-tuned models (2B–8B) in a synthetic benchmark, and abstract, results, and conclusion are mutually consistent.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
LLM resume screeners can rank stronger candidates correctly yet still reverse roughly 29–46% of their pairwise decisions when the same competence evidence is merely reworded, restructured, AI-polished, or degraded by extraction noise. The paper argues validity and presentation stability are separate properties that hiring evaluations must both measure.
Which model performed best in the audit?
Llama-3.1-8B with its native chat template achieved the highest validity at 0.781 (95% CI [0.746, 0.814]) with a 29.6% flip rate. Mistral-7B-v0.3 reached 0.644 validity but flipped 41.4% of matched decisions. All six evaluated models were open-weight 2B–8B instruction-tuned LLMs.
Does using a chat template fix the instability?
No. Native chat templates substantially improve validity for chat-tuned models (Llama-3.1-8B 0.622→0.781, Llama-3.2-3B 0.321→0.551, Gemma-2-2B 0.287→0.576), but flip rates remain 0.296, 0.285, and 0.456 respectively, so presentation sensitivity persists after correcting prompt format.
Is the benchmark data or code publicly available?
The paper text does not mention any released codebase or dataset. The benchmark is synthetic, built from public O*NET 30.3 occupation data with unnamed generation models, a deterministic fact-preservation validation gate (505 of 510 variants accepted), and unpublished scoring prompts — so exact replication is currently not possible from the paper alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (pipeline diagram) is an image; the construction was reconstructed from prose only.
- Scoring prompts, rubric wording, decoding parameters, and generator model names are absent from the extracted text; reproducibility judgments rest on the described procedure.
- The equal-tier tie-rate table referenced in §2.4 does not appear in the provided text; possible minor truncation.
- Figure 2 axis decomposition exists only for the Qwen2.5-7B raw slice; per-axis flip rates for chat conditions are not in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.